STAR: Stability-Inducing Weight Perturbation for Continual Learning
Authors: Masih Eskandar
Tooba Imtiaz
Davin Hill
Zifeng Wang
Jennifer Dy
Venue: ICLR 2025 Poster (2025)
Keywords: Continual Learning
Deep Learning
Weight Perturbation
Representation Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In Continual Learning, we regularize the divergence of the model output distribution with regards to past data via worst-case weight perturbation, alleviating forgetting in future updates.

Abstract: Humans can naturally learn new and varying tasks in a sequential manner. 
  Continual learning is a class of learning algorithms that updates its learned model as it sees new data (on potentially new tasks) in a sequence.
  A key challenge in continual learning is that as the model is updated to learn new tasks, it becomes susceptible to \textit{catastrophic forgetting}, where knowledge of previously learned tasks is lost. A popular approach to mitigate forgetting during continual learning is to maintain a small buffer of previously-seen samples, and to replay them during training. However, this approach is limited by the small buffer size and, while forgetting is reduced, it is still present.  In this paper, we propose
a novel loss function STAR that exploits the worst-case parameter perturbation that reduces the KL-divergence of model predictions with that of its local parameter neighborhood to promote stability and alleviate forgetting. STAR can be combined with almost any existing rehearsal-based methods as a plug-and-play component. We empirically show that STAR consistently improves performance of existing methods by up to $\sim15\\%$ across varying baselines, and achieves superior or competitive accuracy to that of state-of-the-art methods aimed at improving rehearsal-based continual learning.  Our implementation is available at https://github.com/Gnomy17/STAR_CL.

Published as a conference paper at ICLR 2025
STAR: STABILITY-INDUCIN WEIGHT PERTURBATION FOR CONTINUAL LEARNING
Masih Eskandar', Tooba Imtiaz', Davin Hill', Zifeng Wang"; JJanifferr Dy'
"Department of Electrical & Computer Engineering, Northeastern University 2 Google Cloud Al Research
ABSTRACT
Humans can naturally learn new and varying tasks in sequential manner. Contin- ual learning a class of learning algorithms that updates its learned model as it sees new data (on potentially new tasks) in sequence. A key challenge in continual learning that as the model is updated to learn new tasks, becomes susceptible catastrophic forgetring. where knowledge of previously learned tasks is lost. A popular approach to mitigate forgetting during continual learning is to maintain small buffer of previously-seen samples, and replay them during training. However, this approach is limited by the small buffer size and, while forgetting is reduced, is still present. In this paper, we propose a novel loss function STAR that exploits the worst-case parameter perturbation that reduces the KL-divergence of model predictions with that of its local parameter neighborhood promote stability and alleviate forgetting. STAR can be combined with almost any existing rehearsal- based methods as a plug-and-plat and-play component. We empirically show that STAR consistently improves performance of existing methods by up to 15% across varying baselines, and achieves superior or competitive accuracy to that of state- of-the-art art methods aimed at improving rehearsal-based continual learning. Our implementation is available at wilhtts:// https:://gthhb com/ hamm//oommll7/ 1wtimmyy7//ARROOOC
INTRODUCTION
Humans are naturally continual learners. They are able to learn different sets of tasks sequentially without significant forgetting or loss of performance at previous tasks. This allows them to learn continuously from many sources of knowledge over time.
In contrast, modern machine learning models have been shown to suffer from catastrophic forgetting (McCloskey & Cohen 1989) during sequential learning of tasks, where the model's performance on previous tasks completely degrades after being trained on a new task. Continual Learning (CL) seeks to solve this problem and enable models to learn consecutively, akin to humans. A large body of work tackles this issue from varying perspectives; these are often inspired by balancing the stability-plastice dilemma (Mermillod al. 2013 Mirzadeh et al.) 2020a b), where plasticity is the ability to change and learn new knowledge, and stability is the ability to preserve previously learned knowledge. Intuitively, these concepts conflict. The goal of CL is to enforce stability without hindering model plasticity, with limited or no access to the previous training data.
Among different approaches mitigate catastrophic forgetting, rehearsal- based methods (Aljundi) Æt al. 2018 Chaudhry et al. 2019 Buzzega et al. 2020) have gained a lot of attention due to their simplicity and effectiveness. maintained and used in the training of new tasks to alleviate forgetting of previous tasks. Effectively. rehearsal- based CL. methods induce stability through replay of buffered samples to the model while training on new data. Many works focus on how to best utilize the limited buffered data effectively to retain the most information about previous data points with access to limited number of samples.
rehearsal-based CL. a relatively small buffer of previously seen data is
An under-explored area, however, is the overall stability of the local parameter space when comes to the model output distribution for previously seen samples. That is, if the
Conrespondence to eskandar.m northeasterm.edu Work done while the author was at Northeastem University.

---

Published as a conference paper at ICLR 2025
model outputs with respect to previous samples, stable and unchanging within local parameter neighborhood, parameter updates that neighborhood are not going to result in forgetting. Thus, in this work. attempt to bolster the stability of model predictions in local parameter neighborhoods, so that future parameter updates lead to less forgetting.
We propose a new surrogate measure of forget- ting, which is the change in distribution of cor- rect model predictions on buffered samples be- tween now and the future. As future parameters are not accessible in the present, we propose instead optimize worst-case version of this measure over local parameter neighborhood. We achieve this by first perturbing network pa- rameters by de-stabilizing perturbation, i.e., changing the network outputs as much as possi- ble, and then using the gradient of the perturbed parameters to optimize our objective. Intuitively, this results in leading model parameters into regions where model outputs are more stable locally, and thus gives more flexibility for pa- rameter updates in the future (Figure D Our method does not assume any task boundaries or any specific CL method and only requires memory buffer of previous samples. Thus, propose our STability Inducing PArameter- space Regularization (STAR R) as plug-and-play component to any existing rehearsal-based CL baseline.
0l2
Old Data Lasa
Naive Training no CL
Rebeanal CL
Rehamal STAR (ours)
(())
Figure 1: STAR improves rehearsal- based CL by considering the change in output distribution in local parameter neighborhood. Our regularization objective promotes convergence to regions of the parameter space where the loss is stable over local neighborhood ó around the parameters.
Finally, perform exhaustive experiments, that show that method leads to significant empirical improvements to various baselines, as well as exploratory studies into its different components. To the best of our knowledge, STAR is the first exploit worst-case parameter space perturbation for stability in CL. Our contributions are summarized below:
We present novel surrogate measure of forgetting, and propose to optimize this measure in local parameter neighborhood, to increase flexibility of future model updates and reduce forgetting during training.
STAR assumes no access to task boundaries or task labels during training and only requires a memory buffer. Our method can combined with existing CL baseline methods with minimal changes.
We empirically validate the benefits of STAR multiple existing CL baselines. STAR exhibits significant gains in performance for all evaluated methods across multiple datasets and buffer sizes. Furthermore, compare the performance improvement of our method with existing baseline-enhancemeen competitive performance gain compared existing methods. Overall achieve maximum 15.11% increase in average accuracy compared to the baseline across all baselines and datasets.
enhancement methods and show superior and occasionally closely
We perform exploratory experiments and ablation studies the different components of our methods and demonstrate their contributions, confirming the importance of our design choices.
RELATED WORKS
Continual Learning Existing CL methods can generally be categorized as regularization based. architecture-basee based, rehearsal-basee approaches. Architecture-bast based methods (Rusu al. 2016 Mallya Lazebnik] [0188] Wang et al. 2020) modify the model architecture by compartmentalizing and expanding the network parameters, keeping the information from new tasks from interfering
2

---

Published as a conference paper at ICLR 2025
catastrophically with the previously learned tasks. Rehearsal-bai -ased methods keep memory buffer of data during training and replay these buffered samples to prevent the model from forgetting previous task knowledge. Rehearsal offers simple and powerful framework to tackle CL and is leveraged in many state-of-the-art methods (Chaudhry et al. 2019, Buzzega et al. (020). Moreover, an alternative category of CL. methods called Regularization- based methods (Kirkpatrick et al.) 2017 Zenke et al.) 2017 & Hoiem 2017) seeks to ensure model stability through regularization on the weights during optimization. Such methods mainly focus on identifying model weights that are more important to the performance of previous tasks, and impose a penalty on changes to those weights.
In this paper, we focus on improving rehearsal- based methods through regularization using buffered samples. We present STAR, aang-ay loss term that can be combined with any general rehearsal- based CL baseline. Among other methods aimed at improving rehearsal-basee CL., DualHSIC (Wang
et al. 2023) adds a Kernel-based information bottleneck as well as feature alignment to motivate learning of more general and task-invariant features, LiDER (Bonicelli al. 2022) seeks to induce smoothness through optimization of layer-wise Lipschitz constants of the network, and OCM (Guo
ct al. 2022) use mutual information maximization to learn task-specific features while keeping the information from previous tasks. In contrast to STAR, OCM and DualHSIC do not consider the change in model outputs in the local parameter neighbourhood and LiDER only considers model Lipschitz constants which can be implicitly related to the landscape of model outputs in a local region but does not consider the input data.
Weight Perturbation Weight perturbation methods have been used in the past to perform variational inference (Khan et al. 2018), speed up training (Wen al., 2018). and increase generalization of neural networks (Wu et al. 2020; Foret et al. [202]] Lee et al. 2024). The generalization properties and convergence of worst-case weight perturbations has also been studied (Andriushchenko & Flammarion 2022). In this work, we adopt a worst-case weight perturbation approach in the context of CL as a means to regularize parameter updates across tasks and reduce forgetting. One more related work utilizing weight perturbation is DPCI. (Lee et al. 2024). which uses weight perturbation to perform online CL. However, in contrast to our work, DPCL uses random noise perturbations by way of stochastic classifiers and does not draw strict relationships to forgetting.
BACKGROUND AND PRELIMINARIES
Let fo denote classifier parameterized by 0 E R², where P is the number of network parameters, and 0 represents all network parameters concatenated together. Let ((1) denote the parameters of the 1th layer of the classifier, with a total of I. layers. As such, we denote so+s as a network whose parameters have been perturbed by some d E R Given an input belonging to one of K classes, fe( denotes the predicted label while qa (x) denotes the output probabilities of said classifier, i.e. fo(x)) arg maxke (1.....}} (x(x).. Subsequently, as (z()) represents categorical probability distribution, we can measure the KL-Divergence between two model output distributions
as (.)(go()) go (x:) = Σx (o(r) log go(x) We denote the parameters at training timestamp
(m)("))""
t tE [0. (T the paper, we conceptually consider time f to be the present timestamp and t + s to be representative of timestamp in the future. Finally, the notation [condition] denotes the indicator function, which
7 by O;: ft+ denotes the parameters at timestep t+ for E fO...., r-t Throughout
equal to 1 if the condition holds, and O otherwise.
3.1 CONTINUAL LEARNING AND REHEARSAL BASED METHODS
We assume general supervised CL scenario. Formally, at any given training timestamp / E {O....,T), we are given batch of input-label pairs (z,, V1) Xz. where X₂ is the training data distribution at time t. We do not make any assumptions on explicit task boundaries, therefore X, can change any time.
The objective is to minimize the classification error on all seen data streams by the end of training. As such. let Ny., denote uniform mixture of distributions (Xi)sst i.c. any sample drawn from X is equally likely to be drawn from any X. Intuitively, X is the distribution of all observed data at and before time I. Hence, we can write the CL. objective as follows:
3

---

Published as a conference paper at ICLR 2025
min X( - [1oo)))
The primary challenge of CL is that the training data distribution X may change over time. Modern learning methods which use stochastic gradient descent often assume no change in data distribution across time; such, they can encounter catastrophic forgetting when faced with the CL scenario, causing performance to deteriorate significantly on previous data distributions.
More formally, we can quantify forgetting as the difference in error between two timestamps, and t 4+4 8 during training (Chaudhry al. (2019)):
Forgetting(t, s) == ((a,y))Xoo [L (1))a) 1 Lo, (z)
Rehearsal-based Methods. Rehearsal-based methods (e.g. Chaudhry et al. (2019); Aljundi et al. (2018); Buzzega et al. (2020)) have been shown to help reduce the effects of catastrophic forgetting. These methods maintain relatively small buffer of previously-seen data samples, and then train jointly on new data samples and buffer samples. We denote the buffer at time as M,. is imperative to note that since the data distribution X, changes, so does the buffer, which is why, unlike most previous works, we index it with time t.
Thus, the training loss for most rehearsal based CL methods takes the following form
(C)(Ot) ffe. (zt), ve) + l.M (fe, (IM.). JM.))
(3)
where Te, is the current task data, T. MM, are drawn from the buffer, and E, lMM are typically loss functions, such as cross entropy mean squared error.
Buffer Management. The buffer M, serves as a surrogate for the distribution My. Since M, has limited capacity, sampling strategies are used to balance its distribution. In order to avoid the need for usage of task boundaries during training, mainstream methods such as [Aljundi et al. (2018) 8); Buzzega et al. (220)) use the reservoir sampling strategy (Vitter 1985) constantly update the buffer while keeping the distribution of data balanced. While we do not make any assumption about the buffer sampling strategy our method, most state of the art rehearsal methods use the reservoir sampling strategy. We reiterate that although this climinates the need for task boundaries, it does not result in a static buffer distribution M. Not only do new samples get added and old samples are removed, the distribution of the newly added samples changes with the incoming data distribution.
In this work, we improve the general rehearsal -aased loss function LCL (Equation (3) by adding a regularization term to improve the stability of previously-tearned parameters and further mitigate catastrophic forgetting. Our method can be combined with any rehearsal-based CL method to improve its performance; such assume LeL to be the loss function of some underlying baseline method.
4 STABILITY-NDUCCIG PARAMETER REGULARIZATION
In this section, we propose STAR, a general loss term that can be used to regularize any rehearsal- based CL loss function.
4.1 STABILITY LOSS FUNCTION
While Forgelting (eq. 2 is useful post-hoc evaluation metric for trained model, it is impractical to used during training. In particular, to evaluate Forgelting(t, 8) at time requires access to "future" model parameters at time t + s. Instead, in this section propose tractable surrogate for estimating forgetting without having access to future parameters or+a or data samples, by way of regularizing the local change in model output distribution with respect samples and parameters time t. That is, if the model output distribution is unchanging locally (in neighborhood around fj), then parameter updates in any arbitrary direction within this local region will not cause forgetting.
We start with the formulation in eq. 2 For samples on which both the classifiers are incorrect (i.e. fo, (x) # y fo (ar) o y) the indicator functions cancel out. Hence, are left with
4

---

Published as a conference paper at ICLR 2025
Space
sample RM
comreclly classiied
sample Xaa classifted
K.c Nee
Bes
AV Af 8
(a) Generie CL Gradient Update (Hypothetical)
(b) STAR Gradient
(c) Improved Parameter Update
(d) Output Distribution
Divergence Across Parameter Space
Figure 2: Conceptual illustration of STAR. Vanilla rehearsal- based CL approaches do consider consistency of output distribution in the local parameter neighborhood (a). By leveraging the gradient of our proposed loss LSTAR AR (b), we navigate towards parameter regions where the model prediction of previously seen data is consistent (c), reducing forgetting during training due to future parameter updates leading to lower divergence of model output distribution (d).
Forgetting(t, s) = E(x,))~aor ((fo(())= (e)1(2)
1 (o, (
Forgatten samples
Newly-learned samples
The difference in error is comprised of forgotten samples, minus the newly-learned samples. This why the empirical measure of forgetting in eq. (2 technically be negative. In this work, we focus on reducing the first term, i.e. the eтror due to the forgotten samples.
Inspired by this decomposition, we propose the change in the distribution of network output probabil- ities (xx) over correctly classified samples x s.t. fo() = y as a surrogate measure for forgotten samples (first term in eq. (4). In order to measure the change in distribution, propose measuring the KL-divergence between output probabilities at time & and t s.
Be [x)a()))=)] KLL(qv. (x)]ge. (x))
In order to optimize eq. (5). we first need access to samples from all previous data distributions. However, we can use the buffer M, surrogate for No.t. Therefore, the empirical loss function
corresponding to eq. (5) is as follows
Σ (x))90..) (x))
(6)
(n.))M;;
= [ E Melfe. =y}
The loss function LFG seeks to promote the prediction of correctly classified samples to stay the same in the future. However, the main issue remains: we do not have access to future parameters 0its at time We propose to approximate lit. with local, worst-case perturbation 0, + E R R to reduce forgetting regardless of the direction of future parameter updates. We present our novel stability loss function STAR as follows
LSTAR(()) == max LFG( 6, +
s.t. |8112 < d
where d an arbitrary positive number that controls the magnitude of the perturbation neighborhood.
Intuitively, the STAR loss function promotes consistency of model prediction distribution w.r.t correctly classified samples in a local parameter neighborhood. This gives future model updates more flexibility and robustness in the parameter space, making forgetting in the future less likely. conceptual illustration of this is present in Figure 2 a generic CL gradient update may lead to still correctly classified, but forgotten easily in
point where some current buffer samples future parameter updates due to the noise in SGD or samples being replaced in the buffer. In contrast, STAR promotes parameter regions where the output w.r.t. correctly classified samples is consistent.
5

---

Published as a conference paper at ICLR 2025
4.2 COMBINED OBJECTIVE FUNCTION
The STAR loss function (eq. (7)) can be combined with any CL method that uses a rehearsal-based loss function in order reduce forgetting and improve performance. Given some rehearsal-bace -based loss LCL. such as ER (Chaudhry et al. 2019) or its variants, we can use LSTAR as a plug-and-play component and optimize the combined final objective:
Lfinal (0) + ACSTAR(O.)
(8)
where À is a hyperparameter to control the importance of each objective.
4.3 STAR IMPLEMENTATION
We propose to optimize eq. 7 using Stochastic Gradient Descent (SGD). However, optimizing LSTAR is not trivial due to the maximization over ö. In particular, there are two main challenges: 1) calculating the weight perturbation 6, and 2) calculating the gradient with respect to the original parameters 0, while the loss depends on 6. Fortunately, these challenges can be addressed by adapting
existing works, which leverage worst case weight perturbations robustness (Wu et al. 2020) and generalization (Foret et al. 2021)
Optimizing for the Perturbation 6. Calculating eq. requires computation of 8, which is the locally-maxine maximizigg perturbation with respect the parameters A. We propose calculate ó using gradient ascent, i.e., taking gradient steps to increase the KL--Divereence in eq. (5). Calculating the exact local maximum can be challenging. as go is typically non-convex. However, as in the case with existing works (Foret al., 202 Wu et al. 2020), step is sufficient to improve performance in practice. We experiment with multiple gradient steps and present the details in appendix D Following Li et al. (2018); Wu et al. (2020), instead of using single constant 1' as a norm constraint (as shown in eq. ( we normalize the gradients of the model styer-by-layer This is due to the different numeric distributions of the weights of each layer, well as the scale invariance of the weights; i.e., multiplying by 10 at one layer and dividing by 10 the next layer will lead to the same network.
find that taking a single maximizing gradient
Therefore, to calculate 8, we take normalized gradient step (detailed below) towards increasing the KL-Divergence term in eq. (
Since 8 = 0 is the global minimizer of the KL term, and thus has zero gradient, we initialize as do with small noise proportional to the layer norm as follows:
NO (100( [I]
(9)
where the (th layer index, e is a small, positive scalar, and / is the identity matrix with appropriate dimensions.
Let g the gradient of the inner KL term eq.
==
Veot,,Caal 6n)
(10)
Subsequently, we calculate å for each layer in the following manner:
|000 112
800
(11)
(g
where 1 < / < L, g(1) is the perturbation to layer / parameters 001) and 0 S 7 is hyper-parameter
to control the perturbation ratio (i.c., 5()) is normalized such that 18
[ Rs 7).
Updating the Parameters 0. To perform SGD using LSTAR with the 8 calculated in eq. we must now compute the gradient descent step of minimizing LSTAR w.r.t. the original parameters 0,- With the formulation in eq. (11), calculating the gradient of LFG in eq. (7) with respect to model parameters requires computing the Hessian, or at the very least Hessian- -essian-vector products (Foret et al. approximate the gradient via the gradient measured at the perturbed parameter point, l.e. VoLra((,6 5) xe VotsCra(e 0 + 8). to accelerate computation. It is important to note that this is an approximation, and that L is in
2021). However, following Wu et al. (2020); Foret et al. (2021),

---

Published as a conference paper at ICLR 2025
practice non-linear. Otherwise, the minimizing gradient step would be equal to the negative of the maximizing gradient step in eq.
Intuitively, model parameters first perturbed to parameter point which results in outputs with high KL-Divergence with outputs of the unperturbed parameters, with respect to correctly classified samples. Subsequently, the value of the STÄR loss is equal to the KL--Divereence of the outputs of the perturbed model with the outputs of the unperturbed model. We then compute the gradient of the STAR loss the perturbed parameter point and use it for SGD. A detailed pseudocode of our training algorithm can be found in Algorithm
Algorithm STAR Enhanced Rehearsal Based CL Pseudocode
Require: Data stream Xr, parameters with layers, scalar perturbation coefficient 7, learning rate a, memory buffer M, CL loss function CCL
Mo f O
Initialize Oo
for 0 S t << T do
It. Bt Xi
JM. MM, + Me
o² L.M.Ioo (IMM)) (f
JM.3
Correctly classified buffer samples Calculate perturbation for each layer
for 1 < 1 do
6% c(00cc11(()) ((22)
g0 Vo.to K.((g, (x*) 00,+ (x)))
Lrg Gradient
80) 00 7 Tg (())
10(0 ll
end for
ur t AV 0 Aoot&KKll KKL (x=" (x"),, 90, 1000000622 (x()))
STAR gradient
Ur t ut + Ve, CoL (Ft, yes fe., Mi) an4
>> Generic CL method gradient
0(+1 + M t sample(M, Zt, 21,
Update parameters with both gradients
end for return 0,
EXPERIMENTS
To evaluate the effectiveness of method, conduct comprehensive experiments various CL benchmarks. We adhere to the challenging stass-incremental CL scenario, where the task identity of samples is not known during inference, and the model uses single classification head for all tasks. We incorporate STAR as a plug-and-play component into various SOTA CL frameworks to demonstrate significant improvement, and also compare against existing plug-and and-play SOTA CL methods. Additionally, we perform exploratory experiments and an ablation study on the different components of our method.
5.1 EXPERIMENTAL SETTING
Datasets. We evaluate STAR three mainstream CL benchmark datasets. Split-CIFAR10 and Split-CIFAR100 are the CIFAR10/106 datasets (Krizhevsky et al., 2009). split into 5 disjoint tasks of 2 and 20 classes respectively. Split-minilmaggnet (Deng et al. 2009), split into 20 disjoint tasks of classes each. Both CIFARIO and CIFAR 100 are comprised of 32x32 input images, while Minilmagenet uses 84x84 input sizes. Following previous works (Bonicelli et al. 2022 Wang et al. 2023), for CIFARIO 1EFAR10-100 100 we use batch size of 32 and 50 epochs per task. For minilmagenet we use batch size of 128 and 80 epochs per task. For full list of hyperparameters, see appendix
subsampled version of the Imagenet dataset
Comparing Baselines. STAR is a plug-and and-play CL framework that can be combined with any
rehearsal-baut based CL method. We combine STAR with ER ER(Chaudhry et al.) 2019), ER-ACE Caccial ett al. (2022). X-DER-RPC (Boschini et al.] 2022) and DER++ (Buzzega et al. 2020) as SOTA rehearsal- based methods, to demonstrate the gains in performance.
7

---

Published as a conference paper at ICLR 2025
Additionally, following Bonicelli al. methods applied as enhancements to rehearsal-busee methods. We use ER-ACE and DER++ as representative basclines and add each method to both and compare the performance improvements. We compare to sSGD (Mirzadeh et al. (020b), oFwC (Schwarz. et al. 2018), OCM,Guo et al. 2022), LiDER (Bonicelli et al.) 2022) and DualHSIC (Wang al. 2023) Naturally, we would prefer to also compare against DPCL (Lee et al., (022). However, the experimental setting in their paper the online CL setting (which is different than ours). Furthermore, they do not open-source their implementation, thus we cannot compare against DPCL in a fair and reasonable fashion.
(2022). compare with existing SOTA regularization-
Finally, for reference, we include the Sequential baseline which consists of performing CL training without any enhancements to facilitate CL, such as rehearsal or regularization. We also include potential upper bound on performance, termed Joint, which consists of training with access the all of the training data in an i.idd. manner.
Evaluation Metrics. Following previous works Chaudhry et al. (2019); Lopez-Paz Pazzz-Pzz & Ranzato (2017); Wang et al. (2023), we report average accuracy in all tables, and refer readers to appendix B for final forgetting values. The formal definitions of these metrics, as well as information about error bars and number of runs presented in appendix A
5.2 INTEGRATING STAR WITH REHEARSAL BASED CL BASELINES
In this section, we empirically evaluate the effect of integrating STAR with various rehearsal based CL baselines. STAR can be combined with most SOTA baselines to improve their performance. We present the results in Table [ and observe that STAR consistently improves performance across different methods and rehearsal buffer sizes. Notably, even simple baselines like ER benefit from integrating STAR, as well as more complex methods like X-DER, which incorporates numerous enhancements to the baseline rehearsal method. Therefore, including the STAR regularization improves performance for wide range rehearsal-basee methods.
Table 1: Comparison of adding STAR to baseline rehearsal-based CL methods in terms of average accuracy.
Method
Split-Cifar10
Split-Cifar100
Split-minilmageNet
Sequential Joint
19.67
9.29
4.51
92,38
73.29
53,55
Buffer Size
100
200
500
200
500
2000
1000
2000
5000
ER
36.39
44.79
57.74
14.35
19.66
36.76
8.37
16.49
24.17
+ STAR (ours)
51.5
59.3
70.70
19.64
29.64
44.65
11.83
16.64
25.83
ER-ACE
52.95
61.25
71.16
29.22
38.01
49.95
17.95
22.60
27.92
+ STAR (ours)
60.69
67.58
75.44
30.38
40.20
51.67
21.06
24.9
31.01
DER++
57.65
64.88
72.70
25.11
37.13
52.08
18.02
23.44
30.43
+ STAR (ours)
61.76
68.60
76.52
27.64
39.77
53.24
22.4
28.19
33.36
X-DER (RPC)
52.75
58.48
64.77
37.23
48.53
57.00
23.19
26.38
29.91
+ STAR (ours)
58.85
65.94
69.19
38.15
47.56
57.55
24.6
27.95
32.6
5.3 COMPARISON WITH OTHER ENHANCEMENTS OF REHEARSAL- METHODS
Having established that STAR can be combined with various baselines to improve performance, we now evaluate STAR against existing approaches that seek to enhance rehearsal- -asse baselines. The results are presented in Table STAR achieves superior performance in most scenarios and is closely competitive in all other settings. In particular, STAR performs exceptionally well with smaller buffer sizes. This is in line with our intuition: With smaller buffer sizes, fewer samples are retained in the buffer and replayed in future time steps. However, STAR promotes stability in the output distribution across future updates regardless whether or not a sample is still in the buffer in the future. Hence, it is able achieve considerable performance boosts.
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with other regularization methods in average accuracy, our results are high- lighted. The best results are bold, and second-best are underlined.
Method
Split-Cifar10
Split-Cifar
Split-minilmageNet
Buffer Size
100
200
500
200
500
2000
1000
2000
5000
ER-ACE
53.90
63.41
70.53
26.28
36.48
48.41
17.95
22.60
27.92
+sSGD
56.26
64.73
71.45
28.07
39.59
49.70
18.11
22.43
24.12
+ oEWC
52.36
61.09
68.70
24.93
35.06
45.59
19.04
24.32
29.46
+OCM
57.18
64.65
70.86
28.18
37.74
49.03
20.32
24.32
28.57
+LiDER
56.08
65.32
71.75
27.94
38.43
50.32
19.69
24.13
30.00
DDualHSIC
57.03
64.05
72.35
30.97
39.73
49.94
20.75
25.65
31.16
STAR (ours)
60.69
67.58
75.44
30.38
40.20
51.67
21.06
24.9
31.01
+ER+
57.65
64.88
72.70
25.11
37.13
52.08
18.02
23.44
30.43
sSGD
55.81
64.44
72.05
24.76
38.48
50.74
16.31
19.29
24.24
+ OEWC
55.78
63.02
71.64
24.51
35.22
51.53
18.87
24.53
31.91
+OCM
59.25
65.81
73.53
27.46
38.94
52.25
20.93
24.75
31.16
+LiDER
58.43
66.02
73.39
27.32
39.25
53.27
21.58
28.33
35.04
DDaaHSSIC
58.90
67.11
74.34
22.44
39.16
54.07
21.73
27.42
33.74
STAR (ours)
61.76
68.60
76.52
27.64
39.77
53.24
22.4
28.19
33,36
5.4 ABLATION AND EXPLORATORY EXPERIMENTS
Ablation Study. We explore the contributions of different components of STAR towards the overall performance. Specifically, we study the effect of two key components our loss function eq. (
x" vs. xt Using only correctly classified buffer samples x: versus using all buffer samples x in the STAR loss function.
vs. 2: Perturbing the weights by the gradient of our forgetting surrogate in eq. 5 versus random perturbation of the same magnitude by replacing in eq. (1I) with Gaussian noise z, i.e., g(()) = =
|
20), 20 ~ (O, T).
We study the contribution of each component towards the final performance by conducting series of experiments adding one component at a time while keeping other ayper-parameters fixed. The results on the S-CIFARI0 dataset are presented in Table and show that each component is important to the final performance increase across different buffer sizes.
Notably, the biggest gain in performance corresponds to using the gradient of the stability loss for weight perturbation, as opposed to random perturbation the same magnitude (TableB]rows two and three vs the last two rows). This aligns with our intuition, as using the gradient corresponds to optimizing the worst-case scenario which acts as an upper bound on the local neighborhood. This increases the flexibility for future parameter updates. On the other hand, random perturbation only improves the average in the local neighborhood.
Visualizing our Forgetting Surrogate. In this section perform an experiment to validate claim that our proposed STAR loss function eq. 7 is indeed optimizing our forgetting surrogate eq. (5) in the local neighborhood. In section 4 we noted that eq. (5) relies on parameters from the future and thus cannot be optimized directly during training. However, we can conduct post-hoc exploratory experiment and directly measure LFG (eq. (5)) empirically after training is complete as we have access to parameters across training. For this purpose, let 6,, be the model parameters immediately after training on the ith task in the S-CIFARIO dataset. For each task i, we measure (o. 02) for /. >> I; on the test set for task i. Moreover, to verify we are optimizing the worst-case of our surrogate as well, we measure (Fa (0,,,0)) as well, where 0 is 0, after gradient ascent steps as detailed in section 4.3 We repeat the experiment with and without including STAR and plot the results in Figure 3 The results show a significant decrease in divergence of future model output distributions when using STAR, expected.
Choice of Buffer Data or Current Task Data. In eq. (6). we use data from the buffer as a surrogate for all past distributions. However, can argue that adding incoming current task data the mix

---

Published as a conference paper at ICLR 2025
improve the consistency of model predictions for the current task and further reduce forgetting. In Table 4 we experiment with using current task data in the STAR loss function. We find that combining both buffer and current task data does not lead to any significant improvements, and using only current task data causes significant increase in forgetting. We hypothesize two reasons as to why. Firstly, if we only use current task data and not buffer data, once the data distribution shift occurs, the consistency of predictions for previous data is no longer considered. This shift in objective leads shift in the corresponding minima for the rehearsal loss, reducing stability significantly. Secondly, our objective seeks to keep the overall output distribution of already learned samples consistent. However, contrast to buffered samples, the output distribution of current samples are not fully trained and therefore keeping them consistent does not necessarily benefit performance.
STIR
(Werei
Task
Task
Taak
Task
25.
1.50 200
252
100
1.50 200
Tima
Figure 3: KL Divergence of the correctly classified samples of the test set each task for the S-CIFARI0 10 Dataset, between model predictions immediately after training on that task, and future parameter points during training. "Worst" indicates the KL Divergence after multiple gradient ascent steps as detailed in section 4.3
Table 3: Effect of method components on av- erage accuracy. The second column represents whether only correctly classified samples (x" or all samples (z) were used in STAR, the third column represents whether the weight perturba- tion for calculating the loss is VLr or random are fixed.
Table 4: Effect of using current task data for
ASTAR AR as measured in average accuracy. The highlighted row represents the proposed setting for STAR.
Method
Buffer
Current
100
200
500
noise (z). All other
DER++
57.52
65.07
73.05
STAR
X
/
24.25
30.29
34.06
Method
x=" vs, 27
Vo vs.
100
200
500
/
61.76
68.60
76.52
DER++
55.36
62.97
70.48
/
58.88
52.55
75.48
+ STAR
56.28
63.15
72.38
56.20
66.82
72.51
V
67.03
75.11
V
76.52
CONCLUSION
The principal challenge of Continual Learning is overcoming catastrophic forgetting. Inspired by suubilit-plasstinn trade-off, we propose using the divergence model output distribution for correctly classified samples as novel surrogate for forgetting. Subsequently, we introduce STAR, which is regularizer for boosting the performance of rehearsal-based continual learners. STAR exploits the worst-case parameter perturbation that reduces the KL-divergence of model predictions with that of its local parameter neighborhood to promote stability and alleviate forgetting. Our loss function can be used as plug-and-play component to improve any rehearsal-based CL method. We demonstrate the superior performance of STAR through extensive experiments on various CL benchmarks against state-of-the-att continual learners. Our novel proposed formulation and methodology provides new perspective for CL research via optimizing the change in model prediction distribution across the local parameter-space.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
STAR is Blug-and-play enhancement that can be integrated into almost any rehearsal -based CL method. Therefore, we should address the baseline CL method used for STAR terms of fairness and bias (Mehrabi al. 2021 as it could extend into the resulting model from STAR. Moreover, extra measures should be taken when it comes to user data privacy (AI-Rubaie & Chang 2019) or robustness to adversaries (Madry et al. 2018) in safety-critical applications.
REPRODUCIBILITY STATEMENT
We present thorough explainations of implementation details for STAR section 4.3 as well as detailed pseudo-code in algorithm!] [ Moreover, we present details of our experimental settings in section and appendix A as well all the used haper-parameters table g Furthermore, we have made implementation publicly available Githut
ACKNOWLEDGEMENTS
This work was partially supported by NIH/NHLBI IROIHL NIH/NCI U24CA264369, NIH/NCI RO1CA240771, NIH 2T32HL007427- 41, and DoD/USAMRAA HT94252410553, We want to thank all ICLR reviewers and arca-chairs for their efforts toward improving this work.
REFERENCES
Mohammad Al-Rubaie and Morris Chang, Privacy-preserving machine learning: Threats and solutions. IEEE Security & Privacy, 17(2):49-58, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference on Machine Learning, pp. 639-668. PMIR, 2022.
Lorenzo Bonicelli. Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems, 35:31886-31901 2022.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE transactions on pattern analysis and machine intelligence, 45(5):5497-5512 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 5920--159,, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference Learning Representations, 2022.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 1902. 10486, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icve, 2009.
https:://gtthub.ccmm/mnm

---

Published as a conference paper at ICLR 2025
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awart minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International conference on machine learning, pp. 8109-8126. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEER conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International conference on machine learning, pp. 2611 1620. PMIR, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Ban Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Byung Hyun Lee, Min-hwan Oh, and Se Young Chun, Doubly perturbed task free continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13346-13354, 2024.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-22) 2944,, 2017.
David Lopez-Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Arun Mallya and Svetlana Lazebnik. Packnet: pruning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Adding multiple tasks to single network by iterative
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:54654, 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh. Dropout as implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 232-233, 2020a.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understand- ing the role of training regimes in continual learning. Advances in Neural Information Processing Systems,
2020b.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 2016.
12

---

Published as a conference paper at ICLR 2025
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinsk,, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference machine learning, pp. 4528 4537. PMLR, 2018.
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks, arXiv preprint arXiv: 1946, 2019.
Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):37-57, 1985.
Zifeng Wang, Tong Jian, Kaushik Chowdhury, Yanzhi Wang, Jennifer Dy, and Stratis Ioannidis. Learn-prune-shan for lifelong learning. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 641-650. IEEE, 2020.
Zifeng Wang, Zheng Zhan, Yifan Gong, Yucai Shao, Stratis loannidis, Yanzhi Wang, and Jennifer Dy. Dualhsic: Hsic-bottleneck and alignment for continual learning. In International Conference on Machine Learning, pp.
PMLR, 2023.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations mini-batches. In International Conference on Learning Representations, 2018
Dongxian Wu, Shu-Shuuaa Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in neural information processing systems, 33:2958-2998 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pp. 3987-3995, PMLR, 2017.
SUPPLEMENTARY MATERIAL
A ADDITIONAL EXPERIMENTAL DETAILS
Definition of Evaluation Metries Assuming the dataset has & disjoint tasks, average accuracy A, and final forgetting F, final training timestep 7 can be computed as follows
A
ai.
(12)
F
max jE(I....R-1)
[ai,t,
ai,7l
(13)
k:
where a,, is the accuracy on the ith task test set evaluated on model parameterized by O and is the timestamp after training on data from the jth task.
For each experiment, averaged over 5 runs with different seeds. The error bars are presented in T 6 All experiments were run on a single Nvidia RTX A6000 GPU.
Following previous works (Bonicelli al., 2022).
the ResNet18 (He et al. 2016) archi- tecture for CIFARIO and CIFAR datasets and the efficient-net b2 (Tan 2019) architecture for minilmagenet.
B FORGETTING VALUES
We present forgetting values in Table 5 Consistent with the increase in average accuracy when using STAR, we observe that the forgetting is comparable or significantly reduced when STAR is integrated with the baseline CL objectives.
13

---

Published as a conference paper at ICLR 2025
Method
Split-Cifar10
Split-Cifar100
Split-minilmageNer
Buffer Size
100
200
500
200
500
2000
1000
2000
5000
ER
55.90
44.46
38.15
70.17
63.92
46.56
63.55
54.14
41.32
STAR (ours)
42.03
36.22
20.39
72.54
61.76
41.97
69.41
62.97
49.76
ER-ACE
22.76
18.30
14.96
50.63
38.21
27.90
29.82
23.74
19.72
STAR (ours)
21.01
19.32
12.03
39.50
34.45
24.62
26.70
25.11
20.38
DER+
40.25
30.06
21.85
62.92
49.80
31.10
63.40
46.69
37.11
STAR (ours)
33.87
21.29
17.91
52.94
40.46
27.64
36.59
26.77
27.47
X- DER (RPC)
18.72
15.16
12.29
41.58
31.84
17.01
48.67
38.33
28.29
STAR (ours)
17.38
13.36
10.46
29.79
26.10
14.46
27.36
22.37
16.78
Table 5: Final Forgetting (FF) values for Table
|
Lower is better.
Method
Split-Cifart
Split-Cifar100
Split-minilmageNett
Buffer Size
100
200
500
200
500
2000
1000
2000
5000
ER 8
1.23
1.64
0.97
0.64
1.56
0.94
1.11
0.93
1.46
+ STAR (ours)
6.28
4.90
2.16
1.13
3.52
0.28
0.56
0.16
ER-ACE
0.88
0.79
0.99
1.12
0.43
+ STAR (ours)
1.64
1.41
0.54
1.69
0.57
0.14
0.42
0.53
0.14
DER+
1.90
0.91
0.53
1.32
1.42
+ STAR (ours)
1.95
1.10
0.94
0.51
0.59
1.22
1.77
1.07
0.55
X-DER (RPC)*
0.76
1.49
2.07
2.26
0.73
+ STAR (ours)
1.66
1.15
1.22
0.61
0.46
0.27
0.40
0.93
0.57
Table 6: Standard deviation of the average accuracies reported in Table 1
means the result was
taken from existing works, "-" means the error bar was not provided the original work
C RUNNING TIME
STAR uses worst-case weight perturbation to improve rehearsal-bart continual learning. While STAR requires additional computation, namely two extra optimization steps per batch, we believe this cost is not prohibitive and it is natural to have trade-off of performance and computation cost. For the sake of completeness, we report the running time of the experiments of Table T in Table below.
Method
Split-Cifar10
Split-Cifar100
Split-minilmageNet
ER *
47
52
142
+ STAR (ours)
106
77
305
ER-ACE"
49
57
146
+ STAR (ours)
87
112
296
DER++
71
76
259
+ STAR (ours)
124
109
372
X-DER (RPC)*
64
77
241
+ STAR (ours)
107
130
396
Table 7: Running time of methods (mins) for Table
D NUMBER OF STEPS
We experiment with taking multiple gradient steps as opposed to one, for calculating the weight perturbation in eq. (7). We present the results in fig. 4 and show that taking three gradient steps can
14

---

Published as a conference paper at ICLR 2025
Butfer Size
100
Butfer Size 200
Bufffer Size 500
76322
75
72,70
72.48
15.60
69 23
$4.50
$4.09
64.2
13.75
60-
50,23
tro 14811
20
Fr 11481
10
tr 51481
30
Figure 4: Effect of Number of Gradient Steps in the Maximization Objective Performance as Measured in Average Accuracy
2,042
0,1122
50106
5-7003
33339
0.964
5.731
5.6745
5.0003
23326
03356
2,455
5.5003
5.7023
22332
2.289
25400
5.5032
5.5973
52378
0.211
5,665
5.4124
5.1524
0.1333
5.3300
5.3317
23320
011366
22255
5.2459
5.201C
20226
2,150
0.1379
5.2002
5.2002
3,032
3,150
2,0004
5,6765
Figure 5: Contour lines of the entropy loss values, the set of tasks far (darker is lower). STAR leads to larger low loss regions.
sometimes lead to slight increase in performance, at the cost of extra computation. Additionally, we see that beyond that leads to reduction in the performance gain. We hypothesize that this is due to the gradient approximation discussed in section 4.3 As we take more and more optimization steps, the gradient approximation becomes less accurate, leading to less improvements.
E EMPIRICAL VISUALIZATION OF THE LOss LANDSCAPE
Figure and Figure are conceptual illustrations meant to convey the intuition of our methodology. For the sake of completeness, we conduct an empirical visualization the cross-entropy loss landscape. While training the S-CIFAR 10 dataset, training on each task and visualize the loss on the test set of the tasks seen so far. To visualize the loss, perturb the network along two random (multivariate standard Gaussian) directions, normalized in the same scheme as detailed in section 4.3 We repeat the same experiment on ER with and without applying STAR while starting from the same seed, and present the visualization in Figure 5 We can see that applying STAR leads to loss minima with a larger low-loss region, leading to more "flexibility" for future parameter updates.
record the weights of the network after
F HYPER-PARAMETERS FOR STAR
We find our hyperparameters using grid-search validation set. We present our hyperparameters in table 8 For any missing hyperparameters, we refer the readers to the works of Buzzega et al. (2020); Bonicelli et al. (2022); Boschini et al. (2022) as we used the same values.
15

---

Published as a conference paper at ICLR 2025
Dataset
Seq- CIFAR10
Seq-CIFARION
Seq meegimiilmaeeNet
Buffer Size
100
200
500
200
500
2000
1000
2000
5000
ER STAR
Ir
0.1
0.1
0.1
0.1
0.1
0.05
0.1
0.1
0.1
ASTAR
0.01
0.01
0.01
0.05
0.05
0.05
0.01
0.05
0.01
ASTAR
0.1
0.1
0.1
0.05
0.05
0.05
0.1
0.1
0.1
ER-ACE + STAR
Ir
0.03
0.03
0.03
0.05
0.05
0.05
0.1
0.1
0.1
ASTAR
0.01
0.01
0.01
0.05
0.05
0.05
0.01
0.05
0.01
ASTAR
0.1
0.05
0.1
0.05
0.05
0.05
0.1
0.1
0.1
DER+ STAR
Ir
0.1
0.1
0.1
0.03
0.03
0.03
0.1
0.1
YSTAR
0.05
0.05
0.05
0.03
0.03
0.03
0.01
0.01
0.005
ASTAR
0.05
0.05
0.05
0.1
0.1
0.1
0.1
0.15
0.15
0.15
0.2
0.4
0.4
0.3
0.2
0.3
0.15
0.15
0.15
0.3
0.2
0.3
0.1
0.1
0.4
XDER-RPC STAR
Ir
0.03
0.03
0.03
0.03
0.03
0.03
0.1
0.1
0.1
ASTAR
0.001
0.001
0.001
0.05
0.01
0.01
0.001
0.001 0.01
0.001
ASTAR
0.01
0.01
0.01
0.05
0.05
0.05
0.01
0.01
Table Choice of hyperparameters for combination STAR and CL baselines.
16


Reviews:
[{"review_id": "174fzfKh14", "reviewer": "Reviewer_yANV", "summary": "The paper introduces STAR, a plug-and-play loss component designed to enhance rehearsal baselines by addressing potential forgetting from future parameter updates. Since future parameters are unknown, STAR estimates forgetting through a surrogate measure: i.e., capturing the worst-case perturbation of current parameters within their local neighborhood. Substantially, the authors argue that making the model resilient to perturbations (with a dedicated loss components) helps in reducing the future forgetting. Ideally, STAR is evaluated across three datasets, comparing rehearsal baselines i) with and without its application, and ii) against other state-of-the-art plug-and-play components.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) Tackling the problem of future forgetting by acting on the current task is novel and interesting;\n2) the ablations and exploratory experiments are concise yet to the point;\n3) leveraging straight weight perturbation as a regularizer when training in continual learning is compelling, although similar in spirit to [1].\n\n[1] Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems, 35:31886–31901, 2022.", "weaknesses": "1) While an improvement over existing rehearsal baselines is interesting, its appeal is limited as these baselines have largely been surpassed by prompting approaches. Indeed, some of these techniques [1, 2] now represent the state of the art in Continual Learning;\n2) while the improvements on Split-CIFAR10 are solid, those on Split-CIFAR100 and Split-miniImageNet (Table 2) are far less noticeable and sometimes absent;\n3) the results in Table 1 for Split-CIFAR100 seems to be different for X-DER [3] to what reported in the original paper. This hinders a good evaluation, as the original results (reported in [3]) surpass those of X-DER equipped with the proposed methodology.\n\nGenerally, I feel this work is incremental w.r.t. LiDER [4] in its idea. Also, the improvement w.r.t. other plug-and-play techniques appears not significant enough.\n\nSome minor issues that did **not** affect my evaluation:\n - In the related works section, regularization-based methods are listed twice.\n - In the line preceding Eq. 4, “f” should be “f(x).”\n - For the gradient ascent step, eta seems to be used in place of gamma (as in Figure 2).\n - In the explanation of the gradient ascent, the equivalence of eta within the i.e. parentheses appears incorrect.\n - In Algorithm 1, “f” is used instead of “q” in the STAR gradient.\n\n\n[1] Smith, James Seale, et al. \"Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Wang, Liyuan, et al. \"Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.\" Advances in Neural Information Processing Systems (2024).\n\n[3] Boschini, Matteo, et al. \"Class-incremental continual learning into the extended der-verse.\" IEEE transactions on pattern analysis and machine intelligence (2022).\n\n[4] Bonicelli, Lorenzo, et al. \"On the effectiveness of lipschitz-driven rehearsal in continual learning.\" Advances in Neural Information Processing Systems (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9GyR2gdLg", "reviewer": "Reviewer_UhZb", "summary": "In this paper, the authors mainly focused on maintaining the output distribution of previous models to prevent the catastrophic forgetting in rehearsal-based CL. To maintain the output distribution, the proposed method adopts not only the regularization between the future output and the output of the current model, but also minimizing the worst case version of this regularization. By doing so, the models can be updated toward the region in which the model outputs are well preserved. In the experiment, the authors show that the proposed method can strengthen the baselines, and also extensively conducted the ablation analysis.", "questions": "Already mentioned in the weaknesses section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths\n\n1. The viewpoint that the model should preserve the output distribution may be similar to the methods using the knowledge distillation in CL, the approach minimizing the worst case version of the regularization is novel. I think the critical difference between STAR and previous methods lies on the optimization scheme.", "weaknesses": "Weaknesses\n\n1. I think the proposed method highly focuses on the stability of the model. If the number of incoming tasks is quite large (e.g. 50 tasks in split Omniglot), I wonder the proposed approach can still strengthen the baselines in the settings containing large number of tasks.\n\n2. The authors said that this method does not assume any information on the task boundary. However, in the experiment, is it possible to consider the notion of epoch without the assumption on the task boundary? I know there is no terms on the task identifier in the formula, but I think the experiment setting is not consistent to the authors' argument. If the proposed method can cover any scenario in CL, does this methods also can work in single epoch setting?\n\n3. The computational cost on optimizing the min-max loss is not negligible. I think it would be better to show the running time of this algorithm.\n\n4. There is no experiments on large-scale dataset. Since the optimization procedure is much complex than previous methods, I wonder the proposed approach can be applied to much larger networks with large datasets\n\n5. In terms of computing the gradient of Eq.8, the authors said that they assume the Hessian is identity matrix. However, I wonder using this gradient can find the minimum of the worst case loss function. In the all procedure, there are too many approximations to optimize the loss function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag0HBqgdfb", "reviewer": "Reviewer_LJc1", "summary": "The paper focuses on improving rehearsal-based continual learning by stabilizing future predictions across the local neighborhood of the parameters. Specifically, it proposes a plug-and-play loss function, STAR, which applies parameter perturbation and reduces the KL-divergence between the model's predictions and those of its local parameters neighborhood. For each forward pass during training, a local neighbor of the current parameters is sampled, and this neighbor is perturbed by a single step of normalized gradient ascent to maximize the KL-divergence between the predictions of the model and the neighbor. Then, by combining the gradient of the KL-divergence between predictions with respect to the perturbed neighbor and the gradients of the rehearsal method's log-likelihood, the model parameters are updated cumulatively. This approach allows the models to learn a flat loss landscape, making the learned local parameter space less sensitive to future updates.", "questions": "1. Why is the perturbation ratio defined as the ratio of two norms in line 316 even though it is called and actually used as a hyper-parameter, as shown in table 7.\n\n2. What is the essence for scaling gradients by the norm of weights in equation 11?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is easy to follow and provides extensive experiments to show its plug-and-play effectiveness across different replay-based methods on small to large-scale datasets.\n\n2. The authors have thorough experiments including ablation study, choice of buffer or current data for STAR loss and demonstration of distribution shift for seen tasks.", "weaknesses": "1. The paper lacks theoretical justification for why the method works, which could have further strenghtened the proposed method. \n\n1. Minor Inconsistencies in Algorithm 1: (i) does not use epochs, (ii) two different hyper-parameters $\\gamma$ and $\\eta$ in equation (11) for perturbation coefficient (iii) $f$ used instead of $q$ in 345.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VUGOWAIzWo", "reviewer": "Reviewer_Rtfe", "summary": "The paper proposes a new method ‘STAR’ to prevent catastrophic forgetting in continual learning, by minimizing the KL divergence between the output of the current parameters and the worst case parameters in a neighborhood of the current ones. The hypothesis is that if the parameters in the neighborhood of the current solution don’t change the output much for past tasks, then it will be easier to find a solution within that region that also performs well on new tasks. Since the exact computation of this method is intractable, a practical approximation is proposed by first taking a gradient ascent step to find the worst case parameters and then it is assumed that the gradient at the worst case parameters is approximately equal to the gradient of the actual current parameters. The approach is tested by combining this idea with several state of the art rehearsal methods. STAR consistently improves other rehearsal baselines across different datasets and memory sizes.", "questions": "* Is it possible that the results are explained by a different hypothesis, e.g. reduced overfitting on the memory?\n* Is there any empirical evidence for the loss landscapes in Figures 1 and 2?\n* Can the mathematical derivation in 4.3 be simplified, or the importance of non-linearities be highlighted?\n* Is local stability always sufficient in a class incremental settings, as is claimed in line 051?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and clearly explains the followed methodology\n* The hypothesis and solution are plausible\n* The results are well tested with regard to improvement of typical CL baselines and adequately compared to other similar approaches.\n* Section 3 is especially clear, which makes the remainder of the paper a lot easier to understand.", "weaknesses": "* The main hypothesis of this paper is that it is important to reduce the difference in output with the worst case parameters in the neighborhood of the current solution. A loss function is proposed to avoid the worst case parameters, but I am not convinced that is sufficiently shown that this works as intended. Figure 3 does show that the final KL divergence between the current model and the final model are reduced, but that doesn’t imply anything about the worst case situation, only that one specific instance in the neighborhood is closer. To test this, an experiment could be done were a gradient ascent step is taken as in Equation 10 for both the proposed solution and one of the other replay benchmarks to directly compare the worst case parameters. An alternative explanation for the current results may be that the additional loss function acts as a good regularizer to prevent overfitting on the memory samples.  \n\n* The drawing in Figure 1 and 2 are solely hypothetical. There is no evidence that the actual loss landscape looks like this nor that the proposed method actually follows the path that is indicated in these figures. Without evidence that this is the problem, it is hard to accept a solution as long as the problem is not clearly identified.\n\n* The mathematical derivation in lines 288:321 is confusing. First a gradient ascent step is taken to maximize the KL divergence (Eq. 10). Then at those parameters a new gradient is calculated to minimize the same KL divergence (line 323), which should be equal to the negative gradient of Eq. 10, if linearity is assumed (which is done in line 323). Applying this gradient at parameters $\\theta$ is then simply a gradient descent step at the initial parameters. So either this derivation could be simplified, or it could be shown that because of the non-linearity of the loss surface, these extra steps actually make a difference (but then the assumption in line 323 is no longer accurate). Figure 2 shows this differently, but only because non-linearity is assumed there.\n\n* Line 051: in a class incremental setting stability is sometimes not sufficient; if new classes are similar to representation of old classes may need to change too. (E.g. if a model had only learned a color representation of a past object and later a new yellow object is added an old yellow object cannot be represented as only being yellow).\n\n * Line 109: repeated sentence from earlier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)