Predicting the Susceptibility of Examples to Catastrophic Forgetting
Authors: Guy Hacohen
Tinne Tuytelaars
Venue: ICML 2025 poster (2025)
Keywords: continual learning
catastrophic forgetting
replay buffer
simplicity bias
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: A behavioral study of catastrophic forgetting, revealing a link between the learning speed of an example and its susceptibility to being forgotten by the model when new data is introduced.

Abstract: Catastrophic forgetting -- the tendency of neural networks to forget previously learned data when learning new information -- remains a central challenge in continual learning. In this work, we adopt a behavioral approach, observing a connection between learning speed and forgetting: examples learned more quickly are less prone to forgetting. Focusing on replay-based continual learning, we show that the composition of the replay buffer -- specifically, whether it contains quickly or slowly learned examples -- has a significant effect on forgetting. Motivated by this insight, we introduce Speed-Based Sampling (SBS), a simple yet general strategy that selects replay examples based on their learning speed. SBS integrates easily into existing buffer-based methods and improves performance across a wide range of competitive continual learning benchmarks, advancing state-of-the-art results. Our findings underscore the value of accounting for the forgetting dynamics when designing continual learning algorithms.

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Guy Hacohen Tinne Tuytelaars
Abstract
ing of this behavior is still lacking.
Catastrophic forgetting the tendency of neural networks to forget previously learned data when learning new information remains a central chal- lenge in continual learning. In this work, we adopt a behavioral approach, observing a connection be- tween learning speed and forgetting: examples learned more quickly are less prone to forgetting. Focusing on replay-basee based continual learning, we show that the composition of the replay buffer- specifically, whether it contains quickly or slowly learned examples - has a significant effect on for- getting. Motivated by this insight, we introduce Speed-Based Sampling (SBS), simple yet gen- eral strategy that selects replay examples based on their learning speed. SBS integrates easily into existing buffer-based methods and improves performance across a wide range of competitive continual learning benchmarks, advancing state- of-the-art results. Our findings underscore the value of accounting for the forgetting dynamics when designing continual learning algorithms.
In this work, we adopt a behavioral perspective, inspired by psychology and neuroscience, where internal mechanisms are often inaccessible. We train deep networks in various CL settings, observe their forgetting behavior, and infer insights from the resulting patterns.
Our key observation is last-in-first-out forgetting pattern: examples learned later are more prone to forgetting, while earlier-learned ones are preserved. This aligns with the sim- plicity bias of neural networks (Shah et al., 2020; Szegedy et al., 2014), where simpler examples are learned first. As a result, simple examples are consistently remembered, while more complex ones are forgotten as new data arrives. This pattern holds across a wide range of architectures, datasets, and training configurations including variations in learn- ing rates, optimizers, schedulers, epochs, and regularization strategies (see $2, App. D). Fig. visualizes remembered vs. forgotten examples in CIFAR-100.
We further explore how this effect changes as the model's 's ability to remember examples improves. This improvement is achieved by either expanding the buffer size ($2.4), in- creasing the network size (App.C.3), reducing dataset complexity 11). As memory improves, even slower- learned examples are remembered, leaving only the most complex ones susceptible to forgetting. Nonetheless, the core principle remains consistent: the slower an example is learned, the more likely it is to be forgotten.
1. Introduction
Recent years have seen a surge in the practical applications of deep learning, yet our mathematical and theoretical un- derstanding of these models remains limited. While many advances are supported by some theoretical insights, the mechanisms behind key phenomena often remain elusive.
Next, we examine how selective replay affects catastrophic forgetting. We rank examples based on their learning speed, and construct different buffer compositions by focusing on examples from a specific range of speeds. We find that selecting moderately fast-learning examples - the simplest and quickest to learn, which are still complex enough to ben- efit from replay - is particularly effective. As the model's ability remember improves, so does the optimal buffer focus shift progressively toward slower-to-learn examples.
One such phenomenon is catastrophic forgetting (French, 1999; Kemker et al., 2018; McCloskey & Cohen, 1989), a core challenge in continual learning (CL) (see reviews: De Lange et al., 2021; Hadsell et al., 2020; Parisi et al., 2019). When models are exposed to new data, their perfor- mance on previously learned tasks often degrades. Despite extensive theoretical efforts, full mathematical understand-
Most competitive buffer- based methods rely on random sam- pling to fill the replay buffer, treating all examples from the old task equally. To demonstrate the potential of accounting for the susceptibility of examples to forgetting, we intro- duce a simple sampling strategy for replay buffers called speed-based sampling (SBS). SBS tracks the speed at which
'ESAT-PSI, KU Leuven, Belgium. Correspondence to: Guy Hacohen gguyhhacohenee @achhen@@@mallll huj.. Tinne Tuytelaars <Tinne. Tuytelaars esat kuleuvenluu.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Example learned quickly tend to remembered
Examples learned slowly tend to be forgotten
forgetting from model-centric perspective: Nguyen et al. (2020); Ramasesh et al. (2021) showed differences in for- getting patterns across model layers, while Mirzadeh et al. (2020); Pfülb & Gepperth (2019) demonstrated the impact of training hyper-parameters. Nguyen et al. (2019) identi- fied task complexity as a factor influencing forgetting rates. In contrast, our approach is data-centric and behavioral - we link forgetting to the speed at which individual examples are learned
Bees Bicycles
Figure 1: Test examples that networks tend to either re- member or forget. Networks were trained on tasks from CIFAR-100. We show the examples that were learned quick- est (left) and slowest (right). Slowest examples were often forgotten, while the quickest were rarely forgotten.
Simplicity Bias and Selective replay. Previous studies have shown that neural networks tend to learn simple patterns before more complex ones (Cao et al., 2021; Gissin et al., 2020; Gunasekar et al., 2018; Heckel & Soltanolkotabi, 2020; Hu et al., 2020; Jin & Montúfar, 2023; Kalimeris et al., 2019; Pérez et al., 2019; Soudry et al., 2018; Ulyanov et al., 2018) (Dingle et al., 2018; Shah et al., 2020). Our work extends this concept to catastrophic forgetting, revealing a reverse simplicity bias - complex examples are more likely to be forgotten than simple ones.
each example is learned during the old task training and prioritizes sampling the quickest-to-learn examples that are still prone to forgetting (details in $3.1). This approach is computationally and memory-efficient, requiring just one float per example computed during the forward pass, and integrates easily with any replay-based CL method. De- spite its simplicity, SBS consistently improves performance across datasets and buffer sizes, demonstrating the practical value of our insights.
phenomenon known as simplicity bias
Our method leverages this insight by selectively replaying examples based on learning speed, indirectly controlling for complexity. Related paradigms such as curriculum learn- ing (Bengio et al., 2009; Hacohen & Weinshall, 2019) and self-paced learning (Kumar et al., 2010), structure train- ing around example complexity, demonstrating that such structure can improve generalization. However, unlike these approaches, we operate in a continual learning setting. We do not modify the original data or training sequence. In- stead, we influence learning solely through selective replay, using example complexity as the criterion and observing its effect on catastrophic forgetting.
1.1. Our Contribution
Observing connection between an example's learning speed and its susceptibility to catastrophic forgetting.
We show that this observed relationship also persists across CL settings, even as models become more capa- ble of remembering.
Demonstrating that this phenomenon is robust across diverse datasets and hyperparameters.
Our work also aligns with insights from sumi-supervised learning, transfer learning, and active learning, where the complexity of the limited labeled data significantly affects algorithm design and performance (Hacohen et al., 2022; Ychuda et al., 2022; Hacohen & Weinshall, 2023a;b: Fluss et al., 2023; Chen et al., 2024). In all these settings, under- standing and leveraging the distribution of example com- plexity can be key to effective learning.
Showing that restricting the replay buffer to examples of specific learning speeds significantly affects forget- ting across different continual learning scenarios.
Proposing practical application of the observed phe- nomena, by introducing a novel sampling strategy for replay buffers, called speed-basse sampling (SBS). SBS integrates with existing continual learning al- gorithms, significantly improving their performance across wide range of methods and advancing the state of the art.
Forgetting Dynamics. Several studies investigate forget- ting dynamics during training. Maini et al. (2022); Toneva et al. (2019) show that, in non-continual learning settings, certain examples are inherently less prone to forgetting dur- ing training. Millunzi et al. (2023) highlighted distinct forgetting behaviors for noisy versus clean labels, proposing adjustments to the rehearsal process account for these differences. Our findings build on this body of work by further identifying which examples are most susceptible to forgetting, linking catastrophic forgetting to simplicity bias. We propose a method to proactively identify examples susceptible to forgetting and demonstrate how these insights
1.2. Related Work
Catastrophic forgetting. Most existing work on catas- trophic forgetting focuses on mitigation (Kirkpatrick et al., 2017; Lee et al., 2017; Li et al., 2019; Ritter et al., 2018; Serra et al., 2018; Verwimp et al.. 2025), rather than its characterization. Some works have explored catastrophic
2

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
can be applied to optimize replay buffer sampling across various CL methods.
1 E 2
Sharning_speediie (2))) y)
(z) = vl (I)
Replay buffer sampling functions. Replay-based methods mitigate forgetting by storing past examples in a fixed-size buffer. While numerous sampling strategies have been pro- posed (Aljundi et al., 2019; Benkō, 2024; Buzzega et al., 2021; Rebuffi et al., 2017; Tiwari et al., 2022; Wiewel & Yang, 2021), many are limited to specific conditions or methods. As result, uniform sampling remains common (Buzzega et al., 2020; Guo et al., 2020; Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017; Prabhu et al., 2020; Ramesh & Chaudhari. 2022: Rolnick et al., 2019). In con- trast, our proposed SBS offers a lightweight, general ap- proach that consistently enhances replay-based methods across tasks, datasets, and buffer sizes.
Here, ft denotes the intermediate model f after training for E [F]] epochs task D,. This definition is based on the accessibilityy score introduced by Hacohen et al. (2020), which we adapt for computation using a single model instead of an ensemble, facilitating efficient eval- uation throughout the training of f. Intuitively, this score correlates with how quickly the model learns an example: if an example is correctly classified from the early stages of learning, it will sustain S (xr y for more epochs, result- ing in a higher learning speed. A discussion of alternative metrics to learning speed can be found in App. G. A de- tailed explanation of the relationship between the example's learning speed and this score is provided in Hacohen et al. (2020), and thus is not repeated here.
2. Catastrophic Forgetting Vs. Learning Speed
In this section, we observe the behavior of neural models when they exhibit catastrophic forgetting under different settings, drawing a connection between the speed at which a model learns examples, and its likelihood to forget them.
Computing learning speed. To calculate the learning speed of each example, we maintain a boolean matrix, called the epoch-wise classification matrix, M E {o, 13 ExIei throughout the training on task D,, indicating whether the model correctly classified each example during learning. Specifically, for example (Ia Vi) E Dr, we have M 1 [ (z;) y1]. An example's 's learning speed is the average of the M across epochs. This matrix can be computed during the forward pass of the network, incurring minimal computation overhead. Furthermore, memory usage can be reduced by directly computing the matrix's 's mean during training, maintaining single float for each example.
2.1. Definitions
We examine the CL classification setting, where the training dataset D -xX,)) consists of examples x X and their corresponding labels y y. This dataset is divided into T= 1 tasks, with each task [T] represented as D. = )i,, and D U D.. There is no overlap between data of different tasks. A deep model f: X' + is sequentially trained each task D, from = 1 to / = T, with E' N epochs per task. Each task also has separate test set from the same distribution. During training on a task Dr, the model f is provided with replay buffer B of fixed-memory size, containing examples from previous tasks 1, ...,, / - 1. Notably, [B] << [D], meaning only a fraction of the data can be stored in the replay buffer. Note that the buffer may be empty (B- 0), representing training f without replay buffer.
2.2. Preliminaries
Class Incremental vs. Task Incremental Learning, Al- though similar, task incremental learning differs (TIL) from class incremental learning (CIL) by providing the task iden- tity for each example during testing, allowing the use of the specific classifier for the classes within the task. We con- ducted experiments for both TIL and CIL. Our qualitative results across all experiments were consistent for both TIL and CIL. Therefore, to avoid redundancy, we present in the main paper results only for TIL. setting, and repeat all the figures and experiments for CIL in App. B.
Learning speed. Neural networks demonstrate trend in learning examples, where certain examples are consistently learned before others across various deep neural models (Baldock et al., 2021; Choshen et al., 2022; Hacohen et al., 2020). This orderliness is often connected the simplicity bias phenomenon (Shah et al., 2020; Hacohen & Weinshall, 2022), examples learned earlier are generally simpler than those learned later. Here, we measure the speed at which an example is learned, which also serves as a proxy for the example's s complexity. Formally, given a model f and an example (x, y) from either the training or test data, we define the learning speed of the example (xr, 3) as:
Datasets. We investigated various image continual learn- ing classification tasks using split versions of several im- age datasets, including CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Tiny data is split into T tasks by partitioning the classes into T cqual-sized subsets. This partitioning is denoted as dataset- T. For example, splitting CIFAR- into 5 classes is denoted as CIFAR- 10-5, comprising 5 tasks, each with 2 distinct classes. Unless otherwise specified, classes are divided into
(Le & Yang, 2015). The
3

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
tasks according to the original order they appeared in the dataset (often alphahetically).
relation (r 0.995, S 10-1"), indicating that networks are more likely remember quickly learned examples.
For detailed implementation details, including the architec- tures and spper-prramttrss used, please see App. A.
While the results presented here pertain to a simplified case, this phenomenon is robust across diverse datasets, archi- tectures, and hyperparameters. For additional experiments, please refer to App. C, D.
2.3. Learning Speed vs. Catastrophic Forgetting
We begin by exploring catastrophic forgetting in simplistic case, without using replay buffer (B = 0), and with only T= 2 tasks. In Fig. 2a, we plot the mean test accuracy of each task when training 10 networks on CIFAR-100- The dashed black line marks the transition from the first to the second task. Consistent with prior research, the networks exhibit significant catastrophic forgetting, evidenced by sharp decline in the accuracy of the first task during training on the second task. Our objective is to characterize those examples where the network successfully classified after the first task but failed after the second task.
2.4. Adding a Replay Buffer
of Different Sizes
To evaluate replay-based methods, we incorporate a replay buffer in our experiments. Similarly to $2.3, we trained 10 networks on CIFAR-100-2, with replay buffers of varying sizes (1k; 3k: 10k). In Figs. 3(a-c), we plot the epoch-wise classification matrix of the test data for each case. The ex- amples appear in each matrix in the order of their learning speed, from slow (bottom) to quick (top). As expected, inte- grating a replay buffer mitigates some catastrophic forget- ting, boosting the model performance on the first task, with larger buffers yielding better results. Nevertheless, similar to the scenario without a replay buffer B = 0, a relationship exists between learning speed and catastrophic forgetting: networks tend to remember fast-learned examples while for- getting those learned later even when using replay buffer. Notably, models with larger buffer sizes remember most of the examples that models with smaller buffer sizes do, while additionally remembering gradually slower-learned examples. Similar results occur when changing the network architecture, and when decreasing the dataset complexity (see App. C, D).
To simultaneously track learning speed and forgetting, we maintain the epoch-wise boolean classification matrix M for each network (formally defined in $2.2). Throughout the training of both tasks, we record, for each epoch, whether the network correctly classified each example from the first task. The initial 100 epochs represent classification during the first task, enabling us to observe the learning speed of the different examples, while the subsequent 100 epochs allow us to monitor the forgetting of the first task during the training of the second task.
In Figs. 2(b-c), we examine the epoch-wise classification matrices for the train and test sets, respectively. To aid visualization, instead of plotting the epoch-wise classifica- tion matrix for each network, we aggregate the networks from all runs using majority voting. The examples in the matrix are sorted by their learning speed. We observe a cor- relation between learning speed and catastrophic forgetting: examples learned more quickly during the first task tend to remain correctly classified throughout the second task, whereas slower-learned examples are prone to immediate misclassification upon task transition, making them suscep- tible to forgetting. Essentially, slower learning speeds of examples correlate with higher probability of forgetting by deep models. Additional classification matrices, for dif- ferent datasets and different amounts of tasks, can be found in App. C and Fig. 15.
We quantify the relationship between the buffer size and re- membered examples' learning speed. Training 10 networks on CIFAR- 100-2 with various buffer sizes, we plot the mean learning speed of the remembered examples for each buffer size. A strong correlation emerges (r 0.966, p S 10- 10:;); models with smaller buffer sizes remember quickly-learned examples, whereas larger models with larger buffers en- able the model also to remember slower-learned These results are plotted in Fig. 3d. Similar results were achieved across diverse datasets, architectures, hyperparameters, and continual learning algorithms, see App. C, D.
2.5. Different Buffer Compositions and Sizes
Replay-based continual learning methods often sample the replay buffer uniformly, treating quickly and slowly learned examples equally. Here, we analyze different re- play buffer compositions, focused on examples learned specific speeds. To focus the buffer on examples learned at a certain speed, we set 2 threshold, quick and slow, and sample the buffer uniformly from all the examples that were learned slower than the quick threshold and faster and the slow threshold. This allows us to bias the replay buffer toward examples learned at certain speeds.
To quantify this relationship, we define an example in the first task as "remembered" by the network if it is classified correctly at the conclusion of both the first and second tasks. In Fig. 2d, we plot, for each example in the test set, the % of networks that remembered it vs. its mean learning speed across all 10 networks. For visualization, we group exam- ples remembered by similar percentage of networks and average their mean learning speeds. We observe a strong cor-
4

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Wrong
Comeec
Wiorg Lorral
6a
Exammp ls (slow + feett
Exampier (slow fest)
031
*
Tisk accurscy
0.3
13k
Main
0.7
00
swith tasks
Epoch
Epoch
<nntworkss
(a) Test accuracy
(b) M train
(c) M test
(d) Speed vs. forgetting
Forgetting as function of the learning speed. We trained 10 networks on CIFAR-100-2 without replay buffer. (a) Mean test accuracy of each task, where the dashed line marks the task switch. The models forget much of the first task after the switch. (b-c), the first task's ' binary epoch- epoch-wise classification matrices M for train and test data. The y-axis corresponds to different examples, and the x-axis corresponds to different epochs, showing if the examples were classified correctly in each epoch. The order in which the examples are plotted is sorted by the examples' learning speed. Faster-learned examples from the first task are less likely to be forgotten at the end of the second task. (d) The mean learning speed of examples in the first task vs. the % of networks that remember them at the end of the second task. Networks forget more examples learned slowly.
Figure 2:
Vhong
Whoig
Gorrect
Wroig
Correct
Exmples fslow - fest
Examples (slow - fasti
0190
0.09
kaminn
0.7
Examples
Epocn
Epoch
Epoch
Butter
(a) Small buffer
(b) Medium buffer
(c) Large huffer
(d) Buffer size vs. speed
Figure 3: Impact of the buffer size on forgetting dynamics. (a-c) Similarly to Fig. 2c, the first task's binary matrices M, when adding replay buffer of varying buffer sizes: small (1k), medium (3k), and large (12k). The y-axes denote examples, and the x-axes denote epochs, indicating if the example has been classified correctly by the model at that epoch. The order in which examples appear is sorted by the examples' learning speed. In all cases, examples that were learned faster are more likely to be remembered. With bigger buffers, the networks can remember gradually slower-to-learn examples. (d) The mean learning speed of remembered examples of models with different buffer sizes. Models with bigger replay buffers remember slower-to-learn examples.
We compare replay buffer compositions for CIFAR-100-2
in the buffer shifts towards harder, slowly- examples, where quickly-learned examples removed.
across different buffer sizes. Fags.4(a-b) show the mean final accuracy difference relative uniform sampling, eraged over 10 runs. Compositions outperforming uniform sampling are in red, while those reducing performance are in blue. The black-boxed point (guick, slow) = (0, rep- resents the uniform baseline. Standard errors are plotted separately (App.E). In both cases, a wide range of buffer compositions (marked in red) significantly improves the fi- nal accuracy, suggesting that uniform sampling of the replay buffer is suboptimal.
In Fig. 4c, we replicate this experiment on Tiny with a large buffer of size 20k. Similar to CIFAR-100-1 focusing on examples learned midway through learning yields the best performances. Notably, across all cases, the range of thresholds that enhance performance is broad and continuous, indicating that improvements not limited to specific hyperparameter sets. Similar results achieved in the multi-task case, see App. C.5.
To assess the robustness of these results, we repeat the exper- iment from Fig. under various learning settings, including changes to optimizers, architectures, learning rates, training durations, fine-tuning parameters, regularization techniques, and data splits. Across all cases, we observe consistent qual-
While many buffer compositions are beneficial for both cases, we see that when the buffer size is smaller, it is more beneficial to remove slowly-learned examples, focus- ing mainly on the easier examples learned quickly by the models. In contrast, with larger buffers, the optimal focus
5

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
itative results. These experiments can be found in App. D.
strategy for replay buffers that replaces traditional random sampling which is common in various CL methods. SBS prioritizes examples with specific learning speeds, allowing the replay buffer to account for the susceptibility of exam- ples to forgetting. This allows the underlying CL algorithm to focus on the most relevant examples, effectively reducing catastrophic forgetting.
2.6. How Subsequent Tasks Affect Buffer Composition
We empirically examine how subsequent tasks impact the ideal composition of replay buffer for given task. We find that regardless of the similarity or dissimilarity between subsequent tasks and the original task, the optimal replay buffer composition remains largely independent and con- sistent. We conduct experiments with three distinct tasks denoted A, B, and C. By training the model on task A and subsequently introducing either task B or task C while keep- ing the original task unchanged, we analyze the ideal replay buffer composition under varying subsequent tasks.
3.1. Method Definition
Speed-Bascd Sampling (SBS) is a sampling strategy for the replay buffers in buffer-based CL at the end of each task. is compatible with any buffer-basee CL algorithm.
During training on task f, SBS calculates the classification matrix (defined in $2.1) of task t, computing at the end of the task the learning speed of each example in D,. Once train- ing on task t is completed, SBS creates a filtered dataset 'D; by removing the g quickest-learned examples and & slowest- learned examples from D, Examples for the replay buffer are then sampled uniformly from D/--
We initially focus on tasks from the same datasets. We pick task A to be the first 25 classes of CIFAR-100 task B to be classes 26 to 50, and task C to be classes 51 to 75. In Fig. 5, we train various replay buffer compositions when either task or task C is after task A. We plot the difference between the mean final accuracy of each buffer composition to a uniformly sampled buffer. Each composition was repeated 10 times. The consistent qualitative results indicate that good buffer compositions for one case are also good for the other, and vice versa. These results suggest that the ideal replay buffer composition of task A is robust to differences between tasks B and C in this context.
The hyperparameters q and can be adjusted based on the user's requirements (see below). In scenarios where the buffer size is limited and examples need to be removed (e.g.. when handling multiple tasks), examples from previous tasks can be removed from the buffer at random.
Pseudocode for integrating SBS with any busfer-based CL method is provided in Algorithm 1.
Further, we explore scenarios involving either changing the classification task or replacing it with memorization by train- ing on random labels. We adopt the RotNet approach (Gi- daris et al., 2018) to change the classification task. We pick tasks A and B to be different subsets of CIFAR-100, while task C involves a rotation classification task, where each example in task A is randomly rotated between 90"; 180", or 270" degrees, with the task being to classify these rota- tions regardless of the original label. For the random case, tasks A and B remain unchanged, while task C involves the same subset of task B, but with labels chosen uniformly at random, as suggested in Zhang et al. (2021). In both scenarios, we observe consistent results: the ideal replay buffer composition of task A remains robust to differences between tasks B and C in both contexts. Further details and heatmaps for the results are available in App. F.
Choosing and s. The choice of q and 8 is crucial for the performance of SBS. In our experiments, we determined these hyperparameters by training a RotNet auxiliary task on the same dataset, without using additional data points. Specifically, after training the CL method on D., we created rotated version of the dataset by randomly rotating each image by 09, 90P, 180 or 270". Then we trained the network to classify the rotation of each example. A coarse grid search was performed over g and 8 values, selecting the ones that yielded the best performance on this task.
While effective in practice, this approach requires training the model multiple times, which can be computationally expensive for larger datasets. However, the smoothness of the parameter space (see Fig.4) enables a coarse grid search to be sufficient. Moreover, since the auxiliary task is rela- tively simple, each training run is significantly faster than training the model from scratch on the full dataset Dr, as fewer epochs are needed. In addition, the method robust to hyperparameter choices: a broad range of values for q and s tend to yield improvements, making heuristic settings often adequate. For example, setting 4 = 8 = 20% consis- tently enhances performance across all evaluated datasets and generalizes well to unseen ones. This is evident in Figs. 4(a-c), 9(--b), 16(a-b). 17(a-b), 18(a-c), 19(a-c), 20(a-c), 21 and 24(a-c), where the point correspond-
3. Speed--Basee Sampling (SBS)
Above, we observed a distinctive behavior of neural net- works trained in continual learning setting: when exposed to new data, networks tend to forget previously learned data, with examples that were learned more quickly being the least likely to be forgotten. This section demonstrates that this behavior has practical implications and can be leveraged to mitigate catastrophic forgetting in CL.
We propose Speed- Speed-Based Sampling (SBS), a novel sampling
6

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
4:3
292
0.00
0.93
2.20
10%
229
1.06
1.41
2.42
Remove quickest
124
Rermove quickest
Accuracy difference
Remnve quickest
11.31
Accurscy difference
20%
7.72 10.6 10.0133006
7.55 250 74
282
24%-
0.72
0.01
100
2:45
0.00
6%
20%
1.00
0.41032
0.2
1:22 1.00 2.20 2:32 2.50
0.0
0.79
100
0.66
1.01
1:54 1.00
2:30
12%
0.40
0.36
0.00
0.71
2:30
06:
1.35
1.52
0.25
0.40
1.54 1. 195 285 2M0 1.72 1.00
-.42
4%
0.41
0.51
0:11
0.10
3,50
2,42
0%
2.00
0.69
116
1.55
12% 201
5a%
Remove slowest
12% 20%
66
MP%
22,82
Remove slowest
Rernove slowest
(a) CIFAR-I08 2, small buffer
(b) CIFAR-I00 100-2, big buffer
(c) TinylmageNet-22 2, big buffer
Figure 4: Comparison of buffer compositions for various buffer sizes, removing slowly and quickly learned examples (see $3.1). The mean final accuracy difference between each composition and randomly sampled buffer is shown, averaged over 10 runs. Red shades indicate improved performance over uniform sampling, while blue indicates worse performance. The baseline (uniformly sampled buffer) appears at (0%, 0%), marked with black box. (a) CIFAR-100-2 with small 1k buffer: removing flower-to-lean examples is more beneficial. (b) CIFAR- 100-2 with large 10k buffer: removing faster-to-learn examples is more beneficial. (c) Tene with a 20k buffer. Across all scenarios, diverse buffer compositions are beneficial, with smaller buffers benefiting more from foster-to-learn examples.
Petrove quickett
Dernoos quickeet
Algorithm 1 Training Cl. method with SBS.
Input: D., |B], E, amount of quick/slow to remove 4, 8, Output: buffer of size [B]
A34
021
6:41
611
630
332
Classification Matrix 0 for e 1, F. do
Refrove slowest
Remove slowest
Train the model f one epoch using the C'L method
(a) (B)
(b) (A)~(C))
for (I,, ) E DD do Vaasificatio__Matiijee, i] < [[f(x;) = y:l
Figure 5: Buffer compositions when training on the same initial task, followed by different subsequent tasks. We use three subsets of CIFAR- 100: A, B, and C (see $2.6). (a) Task A followed by B. (b) A followed by C. Despite the differences in subsequent tasks, the buffer compositions for the original task similarly mitigate catastrophic forgetting.
end for end for Speed +- Mcaa(CClssificalltt Matrix, axis=0)
D, t- remove 4% quickest and s% slowest from D: return [B] examples sampled uniformly from Dj
ing to q = s = 20% consistently significantly outperforms random sampling, even if it does not always achieve the optimal result. Table further compares the performance of SBS with and without hyperparameter tuning. Even in the fixed setting (SBS-fix, with g = s 20%), the method consistently improves performance across all datasets and buffer sizes tested.
introduces minimal computational overhead to the underly- ing CL method. The classification matrix required by SBS is computed using the classifications of each example dur- ing each epoch, which are already part of the forward pass. While the matrix has a size of E' x [Dr], potentially large for many epochs, this memory cost can be reduced by main- taining only running mean for each example, lowering the storage requirement to [Pr]. Additionally, as SBS samples examples randomly from a smaller data pool, the sampling
Further refinements can also be guided by task-specific rules. Based on the results of $2.5, if the task is relatively easy for the model (resulting in less catastrophic forgetting), it beneficial to focus more on slower-to-leam examples, increasing q and decreasing s. Conversely, for more chal- lenging tasks where forgetting is pronounced, decreasing q and increasing is more appropriate.
process has a similar runtime to standard random sampling.
3.2. SBS Improves Various CL Algorithms
Many replay-based based CL algorithms rely on uniform sampling for their replay buffers. Here, we evaluate the impact of re- placing uniform sampling with SBS across such algorithms,
Running time, Given set of q and s parameters, SBS
7

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Table I: Comparison of replay buffer sampling methods. Each entry represents the mean test accuracy across all tasks for 10 networks trained with experience replay using dif- ferent sampling methods. The highest accuracy in each scenario is in bold. For visualization, standard errors are reported separately in Table 4. While certain methods, such as Herding (small buffers) and GSS (large buffers), perform well in specific cases, they fail to consistently outperform random sampling across all scenarios. In contrast, SBS con- sistently achieves the highest accuracy across all scenarios.
EAT 01
Mastoc
(a) LIFAR-10-33
(b) CIFAR-IO-
(c) TinylmageNet- 10
Figure 6: SBS vs. uniform sampling with various continual learning methods. Bars show the mean final test accuracy across all tasks, with random sampling in blue, and SBS in orange. Error bars denote the standard error. (a) CIFAR- 100-5 (b) Tiny 10, with buffer of 500 examples. SBS improves performance across all methods and datasets.
CIFAR-
CIFAR-106
TinylmageN-2
Buffer
1k
10k
1k
10k
1k
10k
Random
51.75
71.25
79.48
82.4
49,81
61.48
Max Ent
48.3
69.83
77.28
81.72
44.49
57.88
including DER, DER+ (Buzzega et al., 2020), X-DER
IPM
49.91
71.7
79.3
80.57
48.58
61.97
(Boschini et al., 2022), AGEM (Chaudhry et al., 2019), ER (Rolnick et al., 2019), GEM (Lopez-Paz & Ranzato, 2017), RPC (Pernici et al., 2021), and FDR (Titsias et al., 2020). Our results show that SBS consistently improves the per- formance of all tested continual algorithms, advancing the state of the art in multiple image classification tasks.
GSS
48.13
72.9
76.71
84.13
49.36
62.89
Herding
54.0
73.51
81.8
81.03
51.17
62.44
LARS
51.82
71.41
79.17
84.93
50.22
62.78
SBS-fix
54.65
73.72
82.24
86.15
52.15
63.12
SBS
55.43
74.80
83.59
89.17
52.15
63.16
We compared the mean final test accuracy across tasks for each method trained with either uniform sampling (as in the original works) or SBS. While not all methods perform cqually well in every scenario, SBS significantly improves accuracy compared to random sampling in all cases. We focused on small buffer sizes, as these are the primary use cases addressed in the original works.
lent in practical implementation within CL algorithms. This preference may stem from the specific settings or algorith- mic modifications required for other sampling methods to work well. These methods do not consistently improve dif- ferent CL algorithms across datasets and buffer sizes. In contrast, SBS improves a wide range of continual learning algorithms across various buffer sizes and datasets.
Fig. 6 shows results on CIFAR-10-5, CIFAR-100-5 and TinylmageNet-10 10 with a fixed buffer size of 500. Each task was trained for 50 epochs. As the buffer size is fixed, examples were removed starting from the second task to accommodate new data. For SBS, examples were removed random, while uniform sampling employed either random or reservoir sampling, following the strategies outlined in each method's 's original implementation. The q and hy- perparameters for SBS were determined using the RotNet auxiliary task, as detailed in $3.1. When training with multiple tasks, different q and 8 hy- perparameters could be used for each task. However, to reduce computational cost and for simplicity, we apply the same q and values across all tasks in multi-task scenarios. While choosing separate and for each task would likely improve performance further, we observe that SBS's perfor- mance boost is especially pronounced in multi-task settings (sec App. C.5), indicating its particular effectiivnnes in such scenarios.
We explored various replay buffer sampling strategies, in- cluding uniform sampling, max entropy, IPM (Zaeemzadeh et al., 2019), GSS (Aljundi et al., 2019), Herding (Re- buffi et al., 2017), and LARS (Buzzega et al., 2021). We trained 10 networks on TinylmageNet.
(IFAR-100-20 and CIFAR- 10-5 datasets with buffer sizes of 1 and 10k using experience replay. Table 1 presents the mean test ac- curacy across all tasks for each case. Additional results on CIFAR-10-2 and CIFAR-100-2 can be found in App. C.5.
SBS consistently outperforms all other sampling strategies across all evaluated scenarios. In contrast, alternative meth- ods show effectiveness only in specific settings. For exam- ple, Herding favoring examples whose features are close to the class mean - surpasses uniform sampling at smaller buffer sizes but underperforms with larger ones. This likely reflects its bias toward typical examples, which are learned early and thus more useful when the replay buffer is small. Conversely, GSS, which prioritizes examples with large gradient distances, excels with larger buffers but lags with smaller ones, as its preference for slowly learned exam- ples becomes more beneficial when a larger replay buffer is available. Table I also includes SBS-fix, a variant of SBS that uses default hyperparameters (g = 8 = 20%, as
3.3. SBS Vs. Other Replay Buffer Sampling Methods
While numerous replay buffer sampling methods have been proposed over the years, uniform sampling remains preva-
8

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
recommended in $3.1) without fine-tuning. While SBS-fix underperforms SBS, it still consistently outperforms other sampling methods, indicating that extensive hyperparameter tuning is often unnecessary.
A Acknowledgment
This work was supported by ERC-2020-AdG grant 101021347, KeepOnLeaming.
3.4. Limitations
References
One limitation of SBS is that the learning speed score is based on the number of training epochs. With small num- ber of epochs, the score becomes overly discrete, which reduces the precision of the evaluation. However, as shown in App. D and Fig. 20, SBS remains effective even with a small number of epochs, as long as the network converges, although with increased noise. Nonetheless, many continual learning scenarios emphasize stream-like settings, which typically involve only single epoch. These scenarios are not well-supported by SBS, as the learning speed fails to adequately capture how different examples are learned over time. Alternative methods for approximating example com- plexity are needed, which we leave as future work.
Aljundi, R., Lin, M., Goujaud, B., and Bengio, Gradi- ent based sample selection for online continual learning, Advances in neural information processing systems, 32, 2019.
Baldock, R., Maennel, H., and Neyshabur, B. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34: 108866 2021.
Bang. J., Kim, H., Yoo, Y., Ha, J.-W., and Choi, J. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2021.
3.5. Discussion
Bengio, Y., Louradour, J., Collobert, R., and Weston, J.
Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
This paper takes an observational approach to catastrophic forgetting, identifying link between the speed at which examples are learned and their likelihood of being forgotten -examples learned more quickly are less prone to forgetting when new data is introduced. SBS was proposed as simple sampling function that demonstrates the practical utility of this observation. By modifying the contents of the replay buffer, SBS encourages algorithms to focus on examples learned at varying speeds. Indeed, integrating SBS into a wide range of continual learning algorithms across diverse settings yields significant improvements, suggesting that this link is an important factor in mitigating catastrophic for- getting. This insight opens avenues for future work, such as designing more sophisticated sampling strategies or incorpo- rating learning speed more directly into continual learning methods.
Benkö, B. Example forgetting and rehearsal in continual learning. Pattern Recognition Letters, 179:65-72, 2024.
Boschini, M., Bonicelli, L.., Buzzega, P., Porrello, A., and
Calderara, S. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Buzzega, P., Boschini, M.. Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learn ing: strong, simple baseline. In Larochelle, H., Ran- zato, M., Hadsell, R., Balcan, M. F,, and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 15920 1590001559....00 Curran Associates, Inc., 2020.
The simplicity of SBS also brings practical advantages: it is computationally efficient, easy to implement, and compati- ble with any buffer-based continual learning approach. As continual learning methods evolve, SBS remains relevant due to its ease of integration and broad applicability.
Buzzega, P.,, Boschini, M., Porrello, A., and Calderara, S. Rethinking experience replay: bag of tricks for contin- ual learning. In 2020 25th International Conference on Patlern Recognition (ICPR), pp. 21 2180-2187. IEEE, 2021.
Cao, Y., Fang, Z., Wu, Y., Zhou, D., and Gu, Q. Towards understanding the spectral bias of deep learning. In Zhou, Z. (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 2205- 2211. ijcai.org, 2021. doi: 10. 10.24963/jjcai.222211
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM. In 7th Inter- national Conference on Learning Representations, ICLR

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
2019, New Orleans, LA, USA, May 6-9, 20/9. OpenRe- view.net, 2019. URL pttsstps://pperevvvie net./ SrrummiddHkf2ssc555
Hacohen, G. and Weinshall, D. On the power of curriculum learning in training deep networks. In International Con- ference Machine Learning, pp. 2535-2544. PMLR, 2019.
Chen, L., Bai, Y., Huang, S., Lu, Y., Wen, B., Yuille, A., and Zhou, Z. Making your first choice: to address cold start problem in medical active learning. In Medical Imaging with Deep Learning, pp. 496-525, PMLR, 2024.
Hacohen, G. and Weinshall, D. Principal components bias in wee-paaraeetrrize linear models, and its manifestation in deep neural networks. Journal of Machine I Learning Research, 23(155):1 2022.
Choshen, L., Hacohen, G., Weinshall, D., and Abend, O. The hrammar-leaningg trajectories of neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8281 5281-8297, 2022.
Hacohen, G. and Weinshall, D. Pruning the unlabeled data to improve semi-supervised learning. arXiv preprint arXiv:2308. 14058, 2023a.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars. T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366 3385, 2021.
Hacohen, G. and Weinshall, D. How to select which active learning strategy is best suited for your specific problem and budget. Advances Neural Information Processing Systems, 0113335--3409 2023b.
Hacohen, G., Choshen, I.., and Weinshall, D. Let's agree agree: Neural networks share classification order real datasets. In International Conference Machine Learning, pp. 3950-3960, PMLR, 2020.
Dingle, K., Camargo, C. Q.. and Louis, A. A. onput- maps are strongly biased towards simple outputs. Nature communications, 9()):1-7, 2018.
Fluss, N., Hacohen, G., and Weinshall, D. Semi-supervised learning in the few-shot zero-shot scenario. arXiv preprint arXiv:2308. /4/ 19, 2023.
Hacohen, G., Dekel, A., and Weinshall, D. Active learning on budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:220..02794 2022.
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4):128-135, 1999.
Hadsell, R., Rao, D.,, Rusu, A. A., and Pascanu, Embrac- ing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-104) 2020.
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised rep- resentation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: /ttp::://ppeeeeee hwtp///eeeeeeeeeeeeeeeeeee net/ forum? id=$1v4N210-
Heckel, R. and Soltanolkotabi, M. Denoising and regular- ization via exploiting the structural bias of convolutional generators. In 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Gissin, D.,
S., and Daniely, A. The im- plicit bias of depth: How incremental learning drives generalization. In Sth International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OeenReview.net, 2020.
Hu, W., Xiao, L.., Adlam, B., and Pennington, J. The prising simplicity of the early-time learning dynamics of neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33, 2020.
Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Im- plicit bias of gradient descent on linear convolutional networks. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 9482-9491, 2018.
Jiang, Z., Zhang, C., Talwar, K., and Mozer, M. C. Charac- terizing structural regularities of labeled data in overpa- rameterized models. arXiv preprint rrXiv:2002.03206 2020.
Jin, H. and Montúfar, G. Implicit bias of gradient de- scent for mean squared error regression with two-layer wide neural networks. J. Mach. Learn. Res., 24:137:1 137:97, 2023. URL http: / jmlr.. wwwla..rrgpppppapp v24/21-0832. html.
Guo, Y., Liu, M., Yang, T., and Rosing, T. Improved schemes for episodic memory-based lifelong learning. Advances in Neural Information Processing Systems, 33: 1023-1035, 2020.
10

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Kalimeris, D., Kaplun, G., Nakkiran, P., Edelman, B. L., Yang, T., Barak, B., and Zhang, H. SGD on neural net- works learns functions of increasing complexity. In Wal- lach, H. M., Larochelle, H., Beygelzimer, A., d' Alché- Buc, F., Fox, F. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32. pp. 3491- 3501, 2019.
Mirzadeh, S. I,, Farajtabar, M., Pascanu, R., and Ghasemzadeh, H. Understanding the role of training regimes in continual learning. Advances in Neural Infor- mation Processing Systems, 13:7308-7320) 2020.
Nguyen, C. V., Achille, A., Lam, M., Hassner, T., Ma- hadevan, V., and Soatto, S. Toward understanding catas- trophic forgetting in continual learning, arXiv preprint arXiv:/908.01000 2019.
Kemker, R., McClure, M., Abitino, A., Hayes, T., and Kanan, C. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Nguyen, G., Chen, S., Do, T., Jun, T. J., Choi, H.-J.. and Kim, D. Dissecting catastrophic forgetting in con- tinual learning by deep visualization. arXiv preprint wrXiv:2001.01577 2020.
Kirkpatrick, J., Pascanu, R., Rabimowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Bannkkkkaaiiikk,,,... A., et Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):52113322) 2017.
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: review. Neural networks, 113:54-71, 2019.
Pérez, G. V., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In 7th International Conference on Leurning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Online, 2009.
Kumar, M. P., Packer, B., and Koller, D. Self-paced learning for latent variable models. In NIPS, volume pp. 2, 2010.
Pernici, F., Bruni, M., Baecchi, C., Turchini, F., and Del Bimbo, A. Class-incremental learning with pre- allocated fixed classifiers. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6259- 6266. IEEE, 2021.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., and Zhang, B.- T. Overcoming catastrophic forgetting by incremental moment matching. Advances neural information pro- cessing systems, 30, 2017.
Pfülb, B. and Gepperth, A. A comprchensive, application- oriented study of catastrophic forgetting in dnns. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.ne,, 2019. URL /atttsss//opeeeeee /pwpneevee......e//m net / forum? BedddBkklosssooo
Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong, C. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International con- ference on machine learning, pp. 3925-3934. PMLR, 2019.
Prabhu, A., Torr, P. H., and Dokania, P. K. Gdumb: A simple approach that questions our progress in continual learning. Computer Vision-ECCC 2020: 16th Fu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540. Springer, 2020.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Maini, P., Garg, S., Lipton, Z., and Kolter, J. Z. Charac- terizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044- 30057, 2022.
Ramasesh, V. V., Dyer, E.,, and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: / //operrevi ew net forum? ho-hmmmsddddddhhh0gGc
McCloskey, M. and Cohen, N. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 165- Elsevier, 1989.
Ramesh, R. and Chaudhari, P. Model zoo: A growing brain that learns continually. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https::////........ http::///ppeeeevieee net/ forum? á L=WEvgGRcgbE77
Millunzi, M., Bonicelli, L., Porrello, A., Credi, J., Kolm, P. N., and Calderara, S. May the forgetting be with you: Alternate replay for learning with noisy labels. 2023.

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
OpenReview.net, 2019. URL htLpstt:::/ /owpr//v/eeerriieeeeoomo net/ wtt.:/oruumuud//// xm30cKm.
Ulyanov, D., Vedaldi, A., and Lempitsky, V. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454, 2018.
Ritter, H., Botev, A., and Barber, D. Online structured laplace approximations for overcoming catastrophic for- getting. Advances in Neural Information Processing Sys- tems, 31, 2018.
Verwimp, E., Hacohen, G., and Tuytelaars, T. Same ac- curacy, twice as fast: continuous training surpasses re- training from scratch. arXiv preprint arXiv:2502.21147, 2025.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Wiewel, F. and Yang, B. Entropy-based sample selection for online continual learning. In 2020 28th European Sig- nal Processing Conference (EUSIPCO), pp. 1477-1481. IEEE, 2021.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp. 4548 4557. PMLR, 2018
Ychuda, O.,, Dekel, A., Hacohen, G., and Weinshall, D. Ac- tive learning through a covering lens. Advances in Neural Information Processing Systems, 35:22354-22367, 2022.
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netra- palli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33: 9573-9585, 2020.
Zaeemzadeh, A., Joneidi, M., Rahnavard, N., and Shah, M. Iterative projection and matching: Finding structure- preserving representatives and its application to computer vision. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. $414-5423, 2019.
Soudry, D., Hoffer, E.,, Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent sepa- rable data. The Journal of Machine Learning Research, 19(1))2822-2878, 2018.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Szegedy, C., Zaremba, W., Sutskever, 1., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing prop- erties of neural networks. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Rep- resentations, ICLR 2014, Banff, AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014. URL http: / arxiv. org/ abs aww/aass111 1112 6199.
Titsias, M. K., Schwarz, J.. de G. Matthews, A. G., Pascanu, R., and Teh, Y. W. Functional regularisation for continual learning with gaussian processes. In 8th International Conference on Learning Representations, ICLR 2020, Ad- dis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:tt////ttt.ttt http:///ppeeeeeeeeeeeeeeeeee n forum? id=HkxCzeHFDB.
Tiwari, R., Killamsetty, K., Iyer, R., and Shenoy, P. Ger: Gradient coreset based replay buffer selection for contin- ual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 99-108, 2022.
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
12

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Appendix
A. Implementation Details
Architectures and hyper-parametee In our experiments, unless stated otherwise, we trained ResNet-18 for E' = 100 epochs per task. We employed base learning rate of 0.1 with cosine scheduler, SGD optimizer, momentum of 0.9, and weight decay of 0.0005. All networks were trained on NVIDIA TITAN X. These hyperparameters were chosen arbitrarily based on their performance during joint dataset training and were consistent across all experiments. We anticipate consistent qualitative results despite potential variations in these hyperparameters. When introducing a replay buffer, we employed an experience replay strategy (Rolnick et al., 2019), alternating batches of data from the new task and the replay buffer. When integrating with other continual learning algorithms, 2022; Buzzega et al., 2020), using the hyperparameters suggested in the original papers, changing only the replay buffer sampling to SBS, keeping the rest of the method intact.
evaluated all methods within the framework of (Boschini et al.,
B. Class Incremental Learning
There are two primary paradigms in continual learning: class incremental learning (CIL) and task incremental learning (TIL). The distinction lies in how tasks handled. In TIL, task identities known during inference, enabling the use of specialized classifiers within each task, whereas in CIL, the model must infer both the task and class identity without prior knowledge, often leading to lower performance. These paradigms address different real- world scenarios, and methods optimized for one sometimes underperform on the other, necessitating separate evaluations in continual learning research.
Our work examines the relationship between the speed at which examples are learned and their susceptibility to forgetting in continual learning. Despite the differences between CIL and TIL, our study reveals that the same qualitative results and the same conclusions can be made in both cases. Therefore, to avoid repetition in the main text, focus there on TIL only, it isolates forgetting the model level rather than conflating with final-layer interference. To complement this, reproduce all key experiments under CIL settings in this appendix, highlighting similar conclusions despite the inherent challenges of CIL.
Figs. 7, 8, 9, 10, 17 present CII. counterparts to the main text's results, Figs. 2, 3, 4, 6, 16 respectively. While CIL results exhibit lower overall performance, the trends remain consistent: examples learned faster are less likely to be forgotten, and the correlations between the buffer size and forgetting and the correlation between the learning speed and forgetting are strong. Notably, optimal buffer composition in CIL scenarios (Figs. 9, 17) slightly favors keeping more quickly learned examples compared to TIL, reflecting the added complexity of task inference in CIL.
Fig. 10 reproduces results from Fig. 6, demonstrating the impact SBS sampling strategy across CIL datasets and algorithms. SBS consistently yields significant performance gains, reaffirming its effectiveness across diverse continual learning challenges.
Finally, Table shows the results of the experiments in Table the CIL settings, showing similar qualitative behavior.
Table 2: The same table as Table 1, when doing class-incremental learning instead of task-incremental. Each entry represents the mean test accuracy across all tasks for 10 networks trained with experience replay using different sampling methods. The highest accuracy in each scenario is in bold.
CIFAR- 100-20
10-AR-10-5
TinyImageN-2
Buffer
1k
10k
1k
10k
1/:
10%
Random
42.9
48.5
57.74
79.47
9.99
27.4
Max Ent
41.76
47.82
57.11
69.13
9.47
27.2
IPM
42.94
49.51
57.6
80.87
9.96
27.48
GSS
41.63
49.12
$7.73
71.27
9.19
27.73
Herding
44.65
48.47
58.02
79.02
9.38
28.43
LARS
43.55
49.73
57.32
79.96
10.21
28.1
SBS fix
44.87
50.39
58.89
81.16
12.78
29.37
SBS
45.64
53.71
61.13
82.22
12.78
29.41
13

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Wrong
Comecc
Wiorg
Lorrell
Exaimppe les (Slow) + fast)
Examples (slow fext)
*
Mein leaming Spaaed
Tusk accurscy
10.11
13k
00
switch sasks
0.75
Epoch
Epoch
(a) Test accuracy
(b) train
(c) M test
(d) Speed vs. forgetting
Figure 7: Repeating Fig. 2 in class incremental settings. All results are trained on CIFAR-100-2, without a replay buffer. (a) Mean test accuracy of each task. (b-c) first task's binary epoch-wise classification matrices M for train and test data. (d) The mean learning speed of examples in the first task vs. the % of networks that remember them at the end of the second task.
Whong
Do
Wronn
Correct
Whoong
Cerrect
Exampler es (slow) - faxt3
$4
Dxamples (stowo feet3
0.92
32
bc
Examples es (slow)
0.00
1x
Mean
0:00
0.80
100 Epoch
87%
10%
18/
(a) Small buffer
(b) Medium buffer
(c) Large buffer
(d) Buffer size vs. speed
Figure 8: Repeating Fig. 3 for class incremental learning. Impact of the buffer size on forgetting dynamics. (a-c) The first
task's 's binary matrices M, when adding replay buffers of varying buffer sizes. (d) The mean learning speed of remembered examples of models with different buffer sizes. Models with bigger replay buffers remember slower-to-leam examples.
C. Correlations Between Learning Speed and Catastrophic Forgetting in Different Datasets
In Section established a link between the learning speed of examples and catastrophic forgetting, primarily utilizing ResNet- 18 networks trained on CIFAR-100-2. Here, expand upon these findings by demonstrating similar phenomena across different datasets, architectures, and task numbers.
C.1. Other Datasets
We first explore other datasets using the experimental setup detailed in Fig. 2. We extend analysis to CIFAR-10-2, CIFAR-100-2, and a subset of TinylmageNet-22 (comprising the initial 40 classes). The extended epoch-wise classification matrices for each task and the correlation between learning speed and the percentage of networks remembering each example are plotted in Fig. 11. Consistently across all three datasets, we observe a robust correlation between learning speed and the likelihood of example retention during continual training. Notably, faster-leamned examples exhibit lower rates of catastrophic forgetting, as evidenced by both quantitative correlations and visual inspection of the extended epoch-wise classification matrices.
Further, in Fig. 12, expand upon Fig. 3 to include CIFAR-10-2. This extension reveals that as the buffer size increases, continual models tend to retain slower-to-learn examples. Analogous to the results observed in (IFAR-100-22 depicted in Fig. 3, we also observe a near-perfect negative correlation between the buffer size and the mean learning speed of remembered examples in CIFAR-10-2.
C.2. Classification Matrices of Different Datasets
In Section 2, we analyze epoch-wise classification matrices, which indicate whether each example was classified correctly or incorrectly at each epoch during training. Figs. and 3 plot these matrices for CIFAR-100-2 and CIFAR-10-2. revealing a clear correlation between learning speed and catastrophic forgetting: examples learned quickly are less likely to be forgotten during task switches.
14

---

Predicting the Susceptibility of Examples to Catastrophie Forgetting
2.88
44%
-0.68
0.84
2.52
Remove quickest
1.99
11.44-1.994000
0.85
2.40
0.10
2.82 1.33 1.81
8.42
Accuracy difference
Remove quickest
36%
-0.45
1.67
2.24
2.88
1.44
Accuracy difference
28%
2.15
0.10
-1.03
0.32
0.31 0.60
2.01 2.02
1.84
4.21
28%
0.57
0.68
1.41
2.71
2.61
20%
+1.13-1.01
0.13-0.04
0.85
1.35
2.10
2.60
4.84
20%
0.39
0.96
1.48
2.41
2.36
0.00
12%
+..58
-0.300.90000
0.09
1.18
2.31
5.49 3.03 5.19
7.47
12%
0.26
0.87
1.52
1.98
1.42
4%
00.31
0.82
1.09
2.16
4.43 3.55
5.76
8.42
7.24
44.21
-1.44
4%
0.10
1.67
0%
0.00
0.29
1.23
2.07 1.50 3.40 4.86 5.58
6.48
0%
12%
28%
44%
52% 60%
0.00
0.35
1.12
1.28
0.94
Remove slowest
-288
0%
4%
12%
20%
28%
Remove slowest
(a) CIFAR- LIFAR-100-2, small buffer
(b) CIFAR-100 2 big buffer
Figure 9: Repeating Fig. 4 for class incremental learning. Comparison of buffer compositions for various buffer sizes. (a) CIFAR-100-2 with small 1k buffer. (b) CIFAR-100-2 with large 10k buffer. Note that since class incremental learning harder than task incremental learning, it is beneficial to remove fewer "quick" examples in this case.
Uniform
54.3
72,1 12 3972
Uniform
27.5
Uniform
Mean accuracy
Mean accuracy
Mean accuracy
19
20.08
46.23
48.16
60
17.5
17.75
42.99
41.75
46,64
12.78
12.5
11.63
31.4011
33
30
22 22.49.12
22.691226,,0
28.58877
10.0
7.5
9.99
8.0
5.0
DER DER++
GEM
RPC
X-DER
DER DER+ AGEM
GEM
FDR
DER
DER++
AGEM
FDR
Method
Method
Method
(a) CIFAR-100-5
(b) CIFAR-10-5
(c) TinylmageNet-10
Figure 10: SBS vs. uniform sampling with various continual learning methods, in caassincrrementl settings. Each bar shows the mean final test accuracy across all tasks for each method, with a uniform sampled buffer (traditionally done) in blue, and SBS in orange. Error bars denote the standard error.
In Fig. 15, we extend this analysis to the classification matrices of the first task of TinyImageNet-2, CIFAR-100-20 and CIFAR-10-5, For these datasets, which involve more than two tasks, the x-axis includes multiple task switches, marked by vertical dashed black lines. The same correlation is observed and becomes even more pronounced: examples from the first task are increasingly forgotten additional tasks are introduced. With each new task, the newly forgotten examples increasingly those learned earlier in training. Notably, examples learned fastest remain robust forgetting, even after long sequence of tasks, while those learned more slowly are more vulnerable.
C.3. Other Architectures
We extend the results from Fig. architectures beyond ResNet-18 by creating two new, smaller variants: Small-ResNet and ResNet, which can be found in Fig. 13. These formed by reducing both the width and depth of ResNet- 18 by factor of 2 and 4 respectively. We replicate the experimental setup from Fig. 2, training 10 models of each architecture on CIFAR-100-2 without a replay buffer, and storing the extended epoch-wise classification matrix. The order examples appear in the matrix sorted by the learning speed of the examples. All matrices depict the classification of the test dataset. In all architectures, learning speed is highly correlated with catastrophic forgetting: networks forget more examples learned later in training while retaining an almost perfect recollection of those learned early on. Additionally, stronger architectures.
15

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Wrond
Correct
Wrong
Correct
Wrong
Correct
Examples fastt
5k-
Examples (slow fost)
5k-
Examples (slow fast)
14
3k
3k
500
100
Epoch
Epoch
Epoch
(a) CIFAR-100-2
(b) CIFAR- 10-2
(c) TinylmageNet2
Mean learning speed
0.8
Mean learning speed
Mean learning speed
0.8
0.6
0.4
0.4
0.2
0.2 0.5
1.0
0.2
0.6
10
0.2 0.4
% networks remembered
% networks remembered
% networks remembered
(d) CIFAR-100-2
(e) CIFAR-10-2
(l) TinyImageNet-2
Figure 11: Extendeing Fig. 2 to CIFAR-100-2, Cifar-10-2 and a subset of TinylmageNet-2. (top) The first task's binary epoch-wise classification matrices M for the test data of each dataset. The y-axis denotes examples, and the x-axis denotes epochs, indicating if an example is correctly classified by the model at the given epoch. The order in which examples appear sorted by learning speed. Faster-learned examples from the first task are less likely be forgotten at the end of the second task. (bottom) The mean learning speed of examples in the first task the percentage of networks that remember them the end of the second task. Networks forget more examples learned slowly across all 3 datascts.
similar larger buffers, remember klower-to-leane examples. This reinforces the motivation for SBS, as it shows that the examples selected by SBS are those the model could learn independently if it had stronger architecture.
C.4. Other Continual Learning Algorithms
In $3.2, demonstrated the benefits of non-uniform buffer compositions for various continual learning methods. We extend these findings to additional algorithms including DER, DER++, GEM, A-GEM, RPC, and X-DER. For each algorithm, we trained 10 networks with different buffer compositions by varying the quick and slow hyperparameters of SBS. For each algorithm, we trained 10 networks on different buffer compositions, achieved by varying the quick and slow hyperparameters of SBS. In all cases, we used the hyperparameters and architectures suggested in the original works. We used a buffer of size 500, due to popularity previous works. Consistent with the GEM results, we found that replay buffers focused on examples learned mid-way through the training process were most beneficial. Additionally, in all cases, removing slower-to-leane examples proved advantageous, given the small buffer size. These results can be found in Fig. 14.
C.5. Two Tasks Vs. Multi-Task
In the behavioral analysis, we focused on two-task setting provides a simple and controlled environment. However, SBS was introduced in multi-task scenarios, which standard in the CL literature and allow for more meaningful comparisons. For completeness, we now extend our SBS results to the two-task setting and the behavioral results to multi-task scenarios, aligning them with each other.
Fig.4 expands on 6ig.6 by showcasing a broader range of SBS hyperparameters, hence including the results of SBS on these datasets. Using the RotNet auxiliary task to pick the hyperparameter in those cases obtains the hyperparameters that maximize performance across all three cases.
16

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
0.96
Mean learning speed
0.94
0.92
0.90
0.88
20%
40%
60%
80%
Buffer size
(a) CIFAR-1O-2
Figure 12: Extending Fig. 3d to CIFAR-1O-2. We plot the mean learning speed of remembered examples by 10 models trained with different buffer sizes. Models with bigger replay buffers remember slower-to-learn examples.
Table extends Table with results CIFAR-1O-2 and CIFAR-100-2, showing trends consistent with the multi-task setting. Additionally, Figs.16 16 and 17 extend Fig.4 4 to CIFAR-100-20 and CIFAR-10-5 (buffer size 5k) in both class- and task-incremental settings, showing similar qualitative trends.
Notably, while SBS improves performance in both two- and multi-task scenarios, its impact is even more pronounced in multi-task settings, further highlighting its practical value.
D. The Effects of Different Learning Hyper-Paramette
In this section, we investigate how varying learning hyperparameters influences the optimal composition of examples in the replay buffer. We replicate the experiment shown in Fig. 4a, training networks on CIFAR-100-2 with buffer size of 1000, while modifying factors such as network architecture, optimizer choice, learning rate, regularization strength, and the number training epochs.
Across all configurations, we observe consistent qualitative trends similar those in Fig. 4a: a wide range of buffer compositions significantly improves performance and alleviates catastrophic forgetting. These findings suggest that the conclusions drawn in the main paper are robust and generalizable, extending to continual learning scenarios under different hyperparameter settings.
Optimizers. In Fig. 18, compare the performance of different buffer compositions when training ResNet-18 CIFAR- 100-2 using three optimizers: SGD, Adam, and Adagrad. For SGD, we used a momentum of 0.9, weight decay of 0.0005, and a learning rate of 0.1, as is common for CIFAR-100 training. For Adagrad, we employed a learning rate of 0.01 and a weight decay of 0.0001. For Adam, we used a learning rate of 0.001, a weight decay of 0.0001, and (0.9, 0.999) betas. These hyperparameters were tuned optimize ResNet- 18's performance on CIFAR- 100 without considering continual learning, as the optimal settings vary by optimizer due their differing update mechanisms.
Our results indicate that similar buffer compositions consistently enhance performance across all three optimizers. This con- sistency suggests that the benefits of our analysis are robust to the choice of optimizer, further supporting the generalizability of our findings.
Architectures. In Fig. 19a, we evaluate the impact of buffer composition on the performance of three different architectures: R8, VGG-16, and a smaller version of ResNet, tiny-ResNet. All models were trained on CIFAR-100-2 under identical conditions. Our results show that similar buffer compositions consistently improve learning performance across these diverse architectures. This finding suggests that our analysis is not specific to any single architecture.
17

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Wrong
Correct
Wrong
Correct
Wrong
Correct
Examples (slow fastt
5k-
Examples (slow fost)
5k-
Examples (slow fastt
5k
3k
3k
3K
Epoch
Epoch
Epoch
(a) ResNet-18
(b) Small ResNet
(c) Tiny ResNet
Comparing the extended epoch-wise classification matrix for different architectures, the y-axis represents examples, and the x-axis represents epochs, indicating if an example is correctly classified by the model in given epoch. In (a), train ResNet-18. In (b), we train ResNet-18 18 with both width and depth reduced by half. In (c), train ResNet-18 with both width and depth reduced by a factor of four. In all cases, the order examples appear is sorted by learning speed. Examples learned quickly in the first task are less likely to be forgotten after the second task.
Figure 13:
Training epochs. Fig. 20, we compare the performance of models trained for different numbers of epochs per task, all using cosine learning rate scheduler to ensure exposure to both large and small learning rates throughout training. While reducing the number of epochs leads a general drop in performance, we observe that similar buffer compositions consistently yield strong results, even under shorter training schedules. is worth noting that the accuracy of determining the learning speed of each example may decrease with fewer epochs, there is less time for the model adapt. Nonetheless, the qualitative trends remain robust, with specific buffer compositions consistently outperforming or underperforming the random baseline across all tested epoch counts.
Learning Rates. We examine the impact of learning rates on model performance using SGD with rates of 0.05 and 0.2 (Figs. 21a, 21b). While the final accuracies differ across these configurations, the qualitative behavior remains consistent: the same buffer composition consistently achieves strong results.
Fine- Sine-Tuning. In multi-task training, it is common to reduce the learning rate for subsequent tasks to fine-tune the network on the next task. To investigate this, we replicate the experiment in Fig. 4a, lowering the learning rate for the second task by factor of 10. The results align closely with those Fig. 4a, indicating that our findings are robust and not solely attributable to the specific learning rates used in the original experiments. These results can be found in Fig. 21c.
Regularization. All ResNet-18 experiments in the main paper were conducted with a small weight decay of 0.0005. To assess the impact of this regularization, we replicate the experiment from Fig. 4a in Fig. 19b where remove the weight decay entirely. The results remain qualitatively consistent, indicating that the conclusions from our analysis are not sensitive to this specific regularization choice.
Different splits of CIFAR. We repeated the experiment from Fig. 4a using a different split of CIFAR-100, where classes were randomly assigned to tasks. The results, shown in Fig. 19c, remain consistent with those Fig. 4a. Importantly, all task splits our experiments were across the paper chosen arbitrarily, ensuring unbiased evaluations.
E. Standard Errors
In the main paper, we omitted the standard errors for visualization porpuses from Figs. 4,5 and Table 1. In all cases, these were usually very small and did not affect the qualitative results. For completeness, we report the omitted values this section. The standard error for Figs. 4,5 can be found in Figs. 22,23 respectively. The standard errors of Table can be found in Table 4.
18

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
Nemovve quickest
Accuracy difforc lifoorecc
Remove quickest
Remove quickest
114
Accuracy diffrrecc
22%6
0.80
4.35
+2.77
4.33
0.87
-05
23%-
3.81
0.32
40.1
0.09
4:00
23%-
-0.31
..22
0.9
040
0.53
055
ats-
0.61
0.36
0.30
006
0.16
0.55
9.00
4%
2.54
4.00
037
0.75
0.4
136
Bifferc
0.23
0.25
537
070
0.07
101
0.00
0.10
0.34
0.45
0.54
0.00
0.54
0.74
0.64
0.80
0.00
0.09
0.76
0.82
0.57
656
0%
12%
20%
285
103
86
6%
30%
386
144
6%
30%
185
1:14
Remove slowest
Remove slowest
Remove slowest
(a) DER
(b) DER++
(c) GEM
Remove quickest
Remove quickket
Remove quickest
Accuracy diferrncc
22%
046
0.87
1.
22%
0:22
0.33
0.45
0.74
1.00
1:5
24%
8:00
0.13
0.44
0.44
0.50
0.90
0.00
4%c-
0.30
0.61
1.18
1.50
1770
0.00
4%
0.24
0.3
0.24
122
4.00
4%s-
0.82
0.00
0.54
0.49
0.7
0.90
4.80
0.21
20%
20%
3.42
3,60
20%
20%
0.38
Remove slowest
Remove slowest
Remove slowest
(d) AGEM
(e) RPC
(f) X-DER
Figure 14: Comparing different replay buffer compositions for various continual learning methods, with buffer of size 500. The experimental setup replicates the one done in Fig. 4. Like the experience replay case, focusing on examples learned midway through the learning process is most beneficial. Due to the smaller buffer, removing slower examples is better.
F. Independency of the Optimal Buffer Composition When Changing the Subsequent Tasks
We empirically examine how subsequent tasks impact the ideal composition of replay buffer for given task. We find that regardless of the similarity or dissimilarity between subsequent tasks and the original task, the optimal replay buffer composition remains largely independent and consistent. To show this, we conduct experiments with three distinct tasks denoted A, B, and C. By training the model on task A and subsequently introducing either task B or task while keeping the original task unchanged, we analyze the ideal replay buffer composition under varying subsequent tasks.
In Section 2.6, we demonstrate that when selecting tasks A, B, and C from the same dataset, the optimal buffer composition of task A remains unaffected by tasks or C. We observe consistent quantitative behavior across different quick and slow parameters. We further explore task variations by setting A as the first 50 classes of CIFAR- B as the last 50 classes of CIFAR-100, and C as rotation classification (Gidaris et al., 2018) task. In the rotation classification task, examples from are randomly rotated by angles of (909, 180 270"), ), with labels adjusted accordingly. We evaluate the performance of 10 networks trained continuously with different buffer compositions first on task A and then on task B (Fig. 24a), and on task A followed by task C (rotation) (Fig. 24b). Consistent with previous findings, the performance of various buffer compositions remains consistent, independent of the choice of B or C. This property enables us to utilize the rotation task for barar-parameter grid search without requiring additional label data.
We further modify task C to comprise examples from the last 50 classes of CIFAR-100 but with random labels (Zhang et al., 2021). Training on random labels forces the network to memorize the examples, as no generalization is possible, eliminating any potential overlap between the subsequent tasks. The results of training on task A followed by the random label task are depicted in Fig. 24c, Analogous to the rotation classification scenario, we observe that the same buffer compositions of task remain effective, further suggesting that the optimal buffer composition of A does not depend on the subsequent task.
19

---

Predicting the Susceptibility of Examples to Catastrophie Forgetting
Wrong
Correct
Switch tasks
Wroog
Correct
Switch tasks
Wrong
Correct
Switch tasks
Examples (slow - fast)
Examples (slow + fast)
1000
100
Examples (slow
1000
3000
300
1500
4000
400
Epoch
Epoch
Epoch
(a) TinylmageNet-2
(b) CIFAR-I0- 10-5
(c) CIFAR-100-20
with tasks, (b) CIFAR-10 with tasks, and (c) CIFAR-100 with 20 tasks. Dashed black lines indicate the introduction of new task. Consistent with Fig. 2c, examples learned more slowly examples learned relatively quickly become susceptible to forgetting.
Figure 15: Repeating Fig. 2c with different datasets and task counts: (a)
prone to forgetting after task transitions. As the number of tasks increases,
Table 3:
Extended results from Table for CIFAR-10-2 and CIFAR-100- 100-2. For convenience, the original results from Table I are included. Standard errors are reported separately in Table 4.
CIFAR-100 10-2
CIFAR-100
CIFAR-100 100-20
CIFAR CUFAR-10-5
TinyImageNet-2
Buffer size
1k
10k
1k
10k
1k
10k
1k
10k
1k
10k
Random
87.67
95.03
69.03
79.24
51.75
71.25
79.48
82.4
49.81
61.48
Max entropy
84,44
94.63
64.27
77.91
48.3
69.83
77.28
81.72
44,49
57.88
IPM
85.9
94.76
64.16
75.73
49.91
71.7
79.3
80.57
48.58
61.97
GSS
85.28
95.55
64.15
80.11
48.13
72.9
76.71
84.13
49.36
62.89
Herding
88.41
94.53
71.07
78.97
54.0
73.51
81.8
81.03
51.17
62.44
LARS
87.43
95.43
69.53
79.92
51.82
71.41
79.17
84.93
50.22
62.78
SBS-fix
88.11
95.56
70.29
80.2
54.65
73.72
82.24
86.15
52.15
63.12
SBS
88.99
96.03
71.71
80.62
55.43
74.80
83.59
89.17
52.15
63.16
G. Relationship Between Other Scoring Methods and the Learning Speed
To analyze catastrophic forgetting, we focus on its relationship with learning speed (Eq.1). While other metrics exist such as uncertainty score (Bang al., 2021), which evaluates classification consistency under augmentations, and c-score (Jiang et al., 2020), which measures expected accuracy held-out instances they ineur higher computational costs. Using per-example loss as score is computationally simpler but shows weak correlation with catastrophic forgetting. As shown in Fig.25, examples sorted by uncertainty and c-score show moderate correlation with forgetting, while learning speed provides the strongest correlation. Therefore, we adopt learning speed throughout this work for its computational efficiency and strong alignment with catastrophic forgetting.
20

---

Predicting the Susceptibility of Examples to Catastrophie Forgetting
4.22
4,63
48%
-2.00
-0.37
0.72
0.97
40%
-0.67
-0.48
0.13
1.03
2.81
Accuracy difference
Accuracy difference
Remove quickest
2.11
Remove quickest
-2.31
36%
0..41
0.16
1.64
3.22
3.40
30%
00.5
0..0
0.23
2.14
3.13
-0.14
1.54
3.08
3.89
4.01
0.30
1.18
1.30
2.61
3.91
0.00
12%
10%
2211
-2.11
0%
0.00
0.84
2.57
3.58
3.93
0%
0.00
0.05
0.50
3.02
3.90
..22
.463
0%
48%
10%
20%
40%
Remove slowest
Remove slowest
(a) CIFAR-100-20
(b) CIFAR-10-5
Figure 16: Comparison of buffer compositions in multi-task training scenarios: (a) CIFAR- 100-20, (b) CIFAR-I0-5, both trained with a 5k buffer. Similarly to the 2-task case, wide range of buffer compositions significantly improves performance, helping to mitigate catastrophic forgetting. The performance gains of Goldilocks tend to get bigger in the multi-task case.
48%
3.38
0.78
0.24
0.71
40%
0.68
0.53
0.02
0.57
1.38
Accuracy difference
Accuracy difference
Remove quickest
2.61
Remove quickest
1.75
35%
30%
24%
11.49
1.09
1.59
3.16
4.01
20%
.0.23
0.12
1.25
1.39
1.80
0.00
12%
40.27
1.22
2,12
5.21
10%-
-0.11
0.23
1.24
1.72
0.30
4.80
1.79
-5.21
-3.49
0%
12%
24%
30%
48%
10%
20%
30%
40%
Remove slowest
Remove slowest
(a) CIFAR-100-20
(b) CIFAR-10-5
Figure 17: Repeating Fig, 16 for class incremental learning. (a) CIFAR- 100-20, (b) CIFAR-10-S both with 5k buffer. Similar to the 2-task case, wide range of buffer compositions significantly improves performance, helping to mitigate catastrophic forgetting.
Remove quickest
Accuracy
Rembbe quickesst
1:00
Remoove quickest
Accuracy difference
2074
3.09
2.43
235
230
1.00
1..-0.02..11 1.31
1.0
1.72
3.51
5.98
20%
2,20
3,00
180
125
3.50
240
-113
12%
00.23
101
180
12%
0.05
044 us0
1.00
1.97
32%
1.45
120
200 640
814
131
1.22
-1.2
0.00
0.56
0.000.00
3.00
1.33
1.35
1.06
225
30% 12%
-1.52
52%
Remove
(a) SGD
(b) Adam
(c) AdaGrad
Figure 18: Repeating Fig. 4a using different optimizers: (a) SGD, (b) Adam, and (c) AdaGrad. Across all optimizers, a wide range of buffer compositions consistently enhances performance, yielding similar qualitative results the experiment in the main text.
21

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
3.39
342
310
Remove quickest
36%
3:00 2:06 1:1
Accuracy difference
Demmoo quickess
354:
1.56 0.2.002
17:
7.0
Remove quickest
36%
0:75.030
3:31
1.00
Accuracy difference
20%
22.48
0.50-0.03
111
1.40
3.00
2.52
3.53 2.5
20%
0..4
0.26
052
0.00
2:33 2:23
5.42 3:30 2.53
20%
0.00
0.63
0.0
0.55
3.75 1.4
153
1.00
1.5
12%
221
2.
9.00
12%
0.00
difee Pifference
12%
0.00
194
0.24
101
270
2AT 324
233 240
2.79
0.30
202
177 1.5 100
-1.31
4%
0.23
1.2
0.01
2.00 170 1.2 127
-1.5
0%
1.40 2.73 2:77 2:42
0.7 1:40 1.75
0%
1.01
1/41
310
0%
12%
2016 1856 a0% 44% 50% 00%
0%
52% 20% 25% 10% diss 32% c0%
0%
4/
12%
20% 78% 20% 44%, 50% 60%
Removo slowest
Romove slowest
Romove slowest
(a) VGG-16
(b) No regularization
(c) Different CIFAR Split
Figure 19: Repeating Fig. 4a under different learning settings: (a) replacing the ResNer-18 18 architecture with VGG-16, (b) removing weight decay from ResNet-18, and (c) using a random class-to-task split of CIFAR- 100-2. In all cases, the results remain qualitatively similar, highlighting the robustness of the analysis.
Remove quickest
30%
1.51
0.600.0
0:5
081 1.18 0.7
Remoue quickeest
35%
0.66
080
029
127 1:30
Remove quickest
20%
2.04
0.) 0.5
1.2
2.10
250
Accuracy differncce
20%--0
0.50
0.00
0.00
0.00 0.0 380 1.47
1:30
2.15
020020
107
1,53
1.44 3.35
1.48
58%
6.33
3.09
2.34 2.3
235 250
18-6
6. 1:30
11011
0.5
104 170
146
745
0.00
0.00
12%
3.00
0.50
0.53
033
0.00
1.38 1-1 1:15 0.00
12%
4.49
0.12
0.70
084
1.b5 1.50 1.2 007
32%
0.28
0.40
1.01
1.0
1.20
2.44 2.54
2:38 181
0:0
332
0.44 1.04 3.35 0.74 1.04 5:15 0.00
3.00
008
0.55
1.17 1.05 0.73 040 0:32
0.0
1.59
1.83
145 1.50 1.68 1.54
1.35
0.82
12% 20% 30% 60% 52% 00%
2.00
32% 20% 10% 40% 52% 02%
3,56
45%
12% 70% 30% 44% 52% 00%
3244
Remove stowest
Remove stowest
Remove stowest
(a) 20 epochs
(b) 50 epachs
(c) 100 epochs
Figure 20: Repeating Fig. 4a with varying training epochs: (a) 20 epochs, (b) 50 epochs, and (c) 100 epochs per task. Despite shorter training durations, the qualitative results remain consistent, with a wide range of buffer compositions significantly improving performance.
3.08
4.3
Remove quickest
Accurace Lurrcyy difference
Remove quickest
Soli Pruu FOR AlCutt
Remove quickest
Accuracy difference
28%
0.00
1:63
0.38
0.1 1.55 3.68 3.50 1.00
1.54
28%
1.90-1.23 0.0065
2:30 3.57
3.13 2.00
3.30
2.32
28%
0.20
0.05
0.35
0.70
1.24
3.38
131
0.59
20%
1.0 1.70 1.5
25%
1.11 0.5L
101
211 281
0:11
1.00
12%
00.18
0.20
0.36
121
3.40 1.75 1.37 1:3
0.00
0.35
7. 2.44 3.36 3.04
2.74
12%
0.48
0:71 0.89 3.01
0.7
0.01
2.50
49
0.00
0.09
045 1: 3.36 1.67 1:41 095 0.27
-154
4%-00.0 0.9 16 288 2883
2.38
2.32
4%
0:30
06)
0:32 867 3.04 0:00 0.72
0.89
397
28
52%
12t
30% 52%
52%
Remove slowest
Remove slowest
Remove slowest
(a) Smaller learning rate
(b) Larger learning rate
(c) Finetune
Figure 21: Repeating Fig. 4a with varying learning rates: (a) doubling the learning rate, (b) halving the learning rate, and (c) keeping the first task's learning rate unchanged while reducing the second task's by a factor of 10. These scenarios, common in continual learning research, yield consistent qualitative results, indicating that our analysis is not dependent on specific learning rate.
22

---

Predicting the Susceptibility of Examples Catastrophie Forgetting
Remove Quickest
30%
0,250
36%
0.0
0.03
-0.12
Remove quickest
20%
00.9
0.15
0.18
6.13
532
0.12
314
0.5
0,223
0.08
0.33
0.1
0.10
008
0.34
20%
0.3
01:
212
0.00
0.12
01.9
Remove
0.12
5.5
0.03
210
0.150
12%
0.09
0.09
0.00
0.07
.%
0.05
0.12
0.37
0.00
4..0
474
0.22
0:30
0.00
6.14
0.19
000
011
0.00
61000
011
0.07
0.00
010
0.8
0.00
0%
0.13
0.00
12%
32%
0.06
Remove slowest
0%
4%
12%
27%
20%
085
26
6%
10%
Remove slowest
Remove slowest
(a) CIFAR-100-2, 1A buffer
(b) CIFAR-100-, 10k buffer
(c) TinylmageNet-2, 1k huffer
Figure 22: Standard errors for Fig. 4. The error taken over 10 repetitions in each experiment.
0.07
0.08
0.04
0.05
0.03
0.04
0.04
0.06
0.04
0.03
-0.06
-0.07
Remove quickest
Remove quickest
Final Accuracy
Final Accuracy
8%
0.05
8%
0.06
4%
0.05
0.05
0.05
0.04
0.04
4%
0.05
0.03
0.06
0.04
0.05
0.03
0..4
0.04
0.02
0.02
0.04
0.08
0.06
0.05
0.06
0.02
0.03
0%
4%
8%
12%
0%
4%
8%
12%
Remove slowest
Remove slowest
(a) (A)->(B)
(b) X(C)
Figure 23: Standard errors for Fig. 5. The error taken over 10 repetitions in each experiment.
Table 4: Standard error of Table 1.
CIFAR-10-2
CIFAR-100-2
CIFAR-100-20
CIFAR-104 10-5
TinyImageNet-2
Buffer size
1k
10k
10k
10k:
1K
10k
10k
Random
0.06
0.05
0.22
0.12
0.18
0.04
0.1
0.11
0.15
0.07
Max entropy
0.23
0.06
0.23
0.15
0.12
0.02
0.1
0.03
0.13
IPM
0.16
0.13
0.29
0.16
0.17
0.06
0.15
0.06
0.15
0.13
GSS
0.13
0.08
0.29
0.16
0.15
0.01
0.11
0.04
0.13
0.07
Herding
0.09
0.12
0.24
0.14
0.11
0.1
0.14
0.06
0.16
0.11
LARS
0.15
0.12
0.04
0.11
0.05
0.13
0.08
SBS-fix
0.08
0.1
0.25
0.11
0.08
0.07
0.13
0.11
0.19
0.1
SBS
0.03
0.1
0.22
0.1
0.1
0.02
0.1
0.12
0.16
0.1
23

---

Predicting the Susceptibility of Examples to Catastrophic Forgetting
133
1.57
163
68%
0:00
0.02
44%
0.12
0462
1.90
44%
0.51
0.73
1.50
Remove quickest
Accuracy diffcrence
Remove cuickest
Accuracy difference
Remove cuickest
0.52
Accuracy difference
2076
0.72
0.91
1.00
0.09
229
0.68
1.18
1.
1.56
20%
0.28
22
1.55
1.0
0.38
0.79
1.00
0.00
1.02
+300
0.68
1.32
1.32
0.00
12%
0.50
0.0
12%6
22%
026
1.01
0:08
0.07
0.53
0.31
-0.9
49%
0.00
0:38
0.53
0.15
-0.9
4%
0.30
0.05
0.58
0.60
-0.2
066
.6
12%
20%
06:
46.
12%
2016
0%
4%
12%
20%
Remove slowest
Remove slowest
Remove slowest
(a) CIFAR-100 ~> CIF
(b) CIFAR-100 -> rotated
(c) CIFAR-100 -> rrandmm labels
Figure 24: Comparing replay buffer compositions when training the same original task, followed by different subsequent tasks. Each entry in the matrices denotes the difference in final accuracy between each buffer composition and a buffer sampled uniformly at random. Each composition is repeated 10 times. Networks are trained using experience replay 2 tasks, with buffer size of 10k. This Figure replicated the same experimental setup as Fig. 5. In (a) we train the models on the 50 first classes of CIFAR-100 100, followed by the last 50 classes of CIFAR- 100. In (b) we train the models on the 50 first classes CIFAR-100, followed by a rotation classification task on the same examples (see text). In (c) we train the models on the 50 first classes of CIFAR-IO0 100, followed by the last 50 classes of CIFAR-I00, but with random labels. In all cases, the same buffer compositions of the first task give the same qualitative results, suggesting that the optimal composition of the buffer of the first task is independent of the subsequent task.
Examples (low + high score)
Wrong
Correct
Examples (low + high score)
Wrong
Correct
1.0k
1.0k
5.0k
Epoch
Epoch
(a) learning speed
(b) uncertainty score
Examples (low high score)
Wrong
Correct
Examples (low high score)
Wrong
Correct
1.0k
1.0k
3.0k
5.0k
5.0k
Epoch
Epoch
(c) loss score
(d) c-score
Figure 25: Comparison of the extended epoch-wise classification matrix for CIFAR-100-2, sorted by different example-level scoring functions (see App- G). (a) Learning speed (used throughout the paper), (b) Uncertainty measure from Bang et al. (2021), (c) Final loss, (d) c-score from Jiang et al. (2020). While all scores show some correlation with forgetting, as more complex examples tend to forgotten more, learning speed shows the strongest, suggesting it is most suitable for our analysis.
24


Reviews:
[{"review_id": "6QV4HaHj2q", "reviewer": "Reviewer_98Km", "summary": "In this work, the authors investigate catastrophic forgetting from a behavioral perspective,  observing the connection between learning speed and forgetting: examples learned more quickly tend to be more resistant to forgetting. Motivated by the observation, this paper introduces Speed-Based Sampling (SBS), a simple yet general strategy that selects replay examples based on their learning speed, for replay-based continual learning. Experiments show the advantages of the proposed SBS over uniform sampling and other sampling baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate that you found our analysis comprehensive and our results well-supported. Below, we address your comments in detail:\n\n**Comparison with non-uniform sampling in continual learning methods**\n\nMost competitive continual learning methods rely on random sampling [1] because non-uniform sampling strategies generally fail to provide consistent improvements across continual learning methods and settings. SBS is, to our knowledge, the first sampling method that consistently enhances different continual learning methods, making this a key contribution of our work.\n\nSome methods, like iCaRL [2], incorporate specific non-random sampling strategies (herding for iCaRL), but these are tightly coupled to the method itself and do not generalize well. For instance, herding does not work effectively with most competitive continual learning methods, while iCaRL is designed around herding and performs poorly with other sampling methods, including random sampling. However, for completeness, we will follow your suggestion and include additional results on continual learning methods with different sampling functions in the supplementary material and briefly discuss them in the main paper.\n\n**Max entropy and IPM performance**\n\nMax entropy (Max Ent) selects examples with the highest entropy in network logits. While a common baseline, it is rather a naive baseline, and it is known to perform poorly in practice, as also observed in our results.\n\nIPM [3] selects a diverse subset of examples from a dataset, and tries to maximize the information that this set has. The original IPM paper was not focused on continual learning, and showed advantages of this subset in other fields, such as active learning, representation learning, and GAN training. However, prior work [4] indicates that its effectiveness in continual learning is inconsistent, which aligns with our findings. A possible explanation is that IPM prioritizes \"informative\" examples, which often overlap with slower-to-learn examples. Since our analysis shows that slower-to-learn examples are more prone to forgetting, this selection bias may explain IPM’s weaker performance. Interestingly, IPM performs better in settings where learning is easier or buffer sizes are large (e.g., CIFAR-100-20 with a 10k buffer), which is consistent with this hypothesis.\n\n**Choice of hyper-parameters**\n\nAs noted in our response to reviewer hXyj, the performance of $q=s=20\\%$ is already present in multiple figures throughout the paper, including Figs. 4(a-b), 16(a-b), 17(a-b), 18(a-c), 19(a-c), 20(a-c), 21(a-c), and 24(a-c), showing its advantages over random sampling. However, to make this clearer for future readers, we will add a dedicated paragraph consolidating these results in the camera-ready version. Additionally, we will include SBS results with these fixed hyper-parameters in Table 1, demonstrating that they still significantly outperform random sampling without requiring additional tuning or additional computation. The exact numeric results for $q=s=20\\%$ that will appear in Table 1 are also provided in our response to reviewer hXyj.\n\n--------------------------------------\n\n[1] Wang, Liyuan, et al. \"A comprehensive survey of continual learning: Theory, method and application.\" (PAMI 2024)\n\n[2] Rebuffi, Sylvestre-Alvise, et al. \"icarl: Incremental classifier and representation learning.\" (CVPR 2017)\n\n[3] Zaeemzadeh, Alireza, et al. \"Iterative projection and matching: Finding structure-preserving representatives and its application to computer vision.\"  (CVPR 2019)\n\n[4] Brignac, Daniel, Niels Lobo, and Abhijit Mahalanobis. \"Improving replay sample selection and storage for less forgetting in continual learning.\" (ICCV 2023)", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, as no theoretical claims are included.", "experimental_designs_or_analyses": "Note that the paper highlights the benefits of SBS to existing replay-based continual learning methods. However, the authors only compare their method with uniform sampling when utilizing continual learning methods. To make the claim sounder, the authors should also compare it with other sampling methods when utilizing replay-based continual learning methods.", "relation_to_broader_scientific_literature": "The proposed sampling method is tailored for continual learning tasks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength:\n\n1.\tThe phenomenon that examples learned later are more likely to be forgotten while earlier learned examples are not is well studied and supported by extensive experiments.\n\nWeaknesses:\n\n1.\tThe proposed method may be invalid for prevailing large-scale training. First, when training with large-scale data, people usually train on the dataset one or two epochs, making the proposed ‘learning speed score’ inaccurate as mentioned by the authors. Second, with large-scale data set, it would be computationally expensive to run the RotNet auxiliary task on the same dataset for selecting hyperparameters q and s to get the best performance shown in the paper. Although the authors suggest that simply setting q = s = 20% is generally good choice, the performance of the choice is not clear in the paper.\n\n2.\tThe paper emphasizes the benefits of SBS to existing replay-based continual learning methods, but the authors only compare their method with uniform sampling when utilizing continual learning methods.", "other_comments_or_suggestions": "Refer to Weaknesses part.", "questions_for_authors": "1.\tIn Table 1, methods ‘Max Ent’ and ‘IPM’ are almost consistently worse than random sampling, which seems unusual. Do you have any insights into this phenomenon?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZdP9vqgihz", "reviewer": "Reviewer_hXyj", "summary": "The manuscript addresses the challenge of selecting the most relevant examples to store in a memory buffer for rehearsal-based continual learning. Based on a preliminary analysis on the speed at which examples are learned and forgotten, the manuscript finds that the most complex samples are the fastest to be forgotten. Based on this, it introduces Speed-Based Sampling (SBS), a sampling strategy that excludes the $q$ fastest and $s$ slowest learned samples from the selection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for your willingness to consider raising your score. Below, we address your concerns in detail.\n\n**concern 1**\n\nThe advantages of picking $q=s=20\\%$ across different settings are scattered throughout the figures of the paper: see Figs 4(a-b), 16(a-b), 17(a-b), 18(a-c), 19(a-c), 20(a-c), 21(a-c) and 24(a-c). However, we will add a dedicated appendix section explicitly showcasing results for picking $q=s=20\\%$, hopefully making this point clearer.\n\nAdditionally, we will add a new row to Table 1, reporting the results of SBS with these fixed hyperparameters, showing the advantages that can be gained by SBS with and without the hyper-parameter search. The results that will be added are:\n\n|buffer|1k|10k|\n|-|:-:|:-:|\n|CIFAR-100-20|54.65|73.72|\n|CIFAR-10-5|82.24|86.15|\n|TinyImageNet-2|52.15|63.12|\n\nto compare, the baseline results of random sampling (Table 1 in the original manuscript) are:\n|buffer|1k|10k|\n|-|:-:|:-:|\n|CIFAR-100-20|51.75|71.25|\n|CIFAR-10-5|79.48|82.4|\n|TinyImageNet-2|49.81|61.48|\n\nWhile removing hyperparameter tuning slightly reduces performance, SBS still consistently outperforms other methods across all settings. This demonstrates that SBS is effective even without extensive tuning.\n\nWe will also add computational cost for both the RotNet-based version and the fixed-hyperparameter version in the appendix to provide a complete picture.\n\n**concern 2**\n\nThank you for catching this oversight. You are correct that Table 1 (main paper) and Table 2 (supplementary) both report results for TIL. We also conducted experiments for CIL, and while they were qualitatively similar (though lower in absolute performance due to the increased difficulty of CIL), they were mistakenly omitted from App. B.\n\nWe will correct this in the camera-ready version by including the CIL results in App. B.\n\n**concern 3**\n\nThis is an interesting question. While we do not have a formal theoretical proof, we can provide an intuitive explanation based on our observations.\n\nWhile the training error of networks tends to go to $0$, meaning that every example in the dataset is going to be learned at some speed, the test error often does not. The training examples that are learned slowest are often those that correspond to points near test points that the model simply did not learn, meaning they are inherently difficult for the model to generalize to, even in non-continual settings. Therefore, in the CL settings, where the replay buffers are limited, we want to focus on examples that the model can generalize from, and removing the slowest-learned examples helps allocate space to more useful ones, improving overall performance. However, we note that the better the model can perform on the original task, the smaller the number of these unhelpful slower-to-learn examples, allowing us to remove less of them to get better performance.\n\n**comparison to other methods**\n\nYou suggested comparisons with methods that retain complex or diverse samples, specifically Rainbow Memory (RM) [1] and LARS [2]. Below, we summarize how our method relates to them and provide additional quantitative comparisons.\n\n**Rainbow Memory (RM)**\n\nThe original RM paper focuses on blurry-CIL settings, whereas our study considers disjoint settings. The RM paper itself notes that RM does not consistently improve performance in disjoint settings, often performing similarly to random sampling. Since SBS outperforms random sampling, this suggests that SBS is superior to RM in these settings.\n\nThat said, we acknowledge RM’s potential value and have already discussed its relation to SBS in App. G. We will make this connection more explicit in the main paper and include RM results in Table 1 for direct comparison.\n\n**LARS (from BAGS of Tricks)**\n\nLARS is presented in [2] as one of several \"tricks\" to improve CL and was not presented as a stand-alone sampling method, which is expected to improve performance across different settings. LARS suggests to sample the buffer randomly, but to remove examples from it in a suggests loss-aware, where examples with low loss will be removed more frequently. We agree that adding a comparison to LARS can strengthen our work, and we added such a comparison to the paper. To compare to SBS, we isolated LARS from the rest of the \"tricks\" suggested in [1], and evaluated it on the different datasets and buffer sizes in Table 1. Below are the results:\n\n|buffer|1k|10k|\n|-|:-:|:-:|\n|CIFAR-100-20|51.82|71.41|\n|CIFAR-10-5|79.17|84.93|\n|TinyImageNet-2|50.22|62.78|\n\nWhile LARS improves over random sampling in most cases, its gains are smaller than those achieved by SBS. Following your suggestion, we will add both RM and LARS results to Table 1 in the camera-ready version for a clearer comparison.\n\n---\n[1] Bang, Jihwan, et al. Rainbow memory: Continual learning with a memory of diverse samples. (CVPR 2021)\n\n[2] He, Tong, et al. Bag of tricks for image classification with convolutional neural networks. (CVPR 2019)", "claims_and_evidence": "The claims are supported by a thorough analysis.", "methods_and_evaluation_criteria": "The benchmarks are adequate.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "While the design of the benchmarks is adequate, I believe the manuscript should compare against other methods that seek to retain complex or diverse samples from the stream, such as [1,2]. Indeed, while the proposal addresses this by measuring the \"learning speed\" of each sample, [1] uses the loss of the examples as a proxy for complexity.\n\n[1]: Buzzega, Pietro, et al. \"Rethinking experience replay: a bag of tricks for continual learning.\" ICPR 2021.  \n[2]: Bang, Jihwan, et al. \"Rainbow memory: Continual learning with a memory of diverse samples.\"  CVPR 2021.", "relation_to_broader_scientific_literature": "The considerations regarding the speed at which the examples are learned are based on existing literature and the results are known in literature [1,2]. However, the proposed SBS seems novel and relevant for the CL field.\n\n[1]: Maini, Pratyush, et al. \"Characterizing datapoints via second-split forgetting.\" aNeurIps 2022.  \n[2]: Millunzi, Monica, et al. \"May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labels.\" BMVC 2024.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1: As mentioned in lines 389-391, the results presented in Tab. 1 for the proposed SBS are obtained with the RotNet auxiliary task. However, given the high computational cost of training the network multiple times, a more fair comparison should compare with the fixed values for q and s, as mentioned in l379. Indeed, while the manuscript mentions that \"q=s=20% consistently enhances performance across all evaluated datasets\", I did not find results to support such a claim. On the same note, I suggest including the computational cost (in terms of compute time) of SBS with the RotNet auxiliary task.\n\n2: It was not clear to me if the experiments of Tab. 1 (main paper) and 2 (supplementary) were conducted on the TIL or CIL setting. From Sec 2.2 it seems to me that both figures and tables in the main paper are for TIL and that results for CIL are in the supplementary. However, App. B only mentions results for the qualitatives and the figures and I could not find the results for the main experiments in the CIL setting. \n\n3: I did not understand the reasoning behind removing the $s$ slowest-learned examples from the dataset. According to the preliminary analysis, if the motivation is to retain the most complex samples, why would we need to discard the ones that would be the first to be forgotten?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Overall the manuscript is well written and mostly easy to read and understand. I would consider raising my score upon addressing the concerns in “Other Strengths And Weaknesses\" and the lack of similar methods that seek to retain complex or diverse samples from the stream.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nBWvkBqeSg", "reviewer": "Reviewer_f954", "summary": "This paper reports on a large volume of observations regarding the learning speed and catastrophic forgetting in the context of continual learning, and proposes a new sampling strategy called SBS to improve the replay-based continual learning methods. The experiments consistently show the effectiveness of the proposed speed-based sampling, which highlights the practical value of the proposed strategy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our extensive empirical study and the practical value of our findings.\n\nRegarding the connection between learning speed and catastrophic forgetting, our primary contribution is identifying and characterizing this phenomenon. While a formal theoretical analysis is an exciting future direction, many fundamental insights in machine learning (e.g., double descent, simplicity bias) were first observed empirically before theory followed. Despite the lack of theory, our results have several more intuitive explanations: later-learned examples likely depend on more complex or composite features, making them more susceptible to forgetting when new tasks are introduced, as those features are more likely to break first. Another intuition is that a similar pattern occurs in human learning, where foundational skills persist longer than more complex ones.\n\nOn the choice of hyperparameters $q$ and $s$, our goal with SBS is to demonstrate that leveraging learning speed can improve continual learning across different algorithms and settings. While a more theoretically derived selection method could be valuable, it is non-trivial and beyond this paper’s scope. Instead, we propose a practical heuristic -- using a related self-supervised task (e.g., RotNet) to tune these parameters without additional labels. While RotNet was chosen arbitrarily, our experiments show that this approach consistently finds the best hyperparameters across all the setups we evaluated.", "claims_and_evidence": "Most of the claims are based on empirical study and I don't have any objections on them. However, as they are based on the empirical observation, the academic value of the work is clumsy. Especially, the sizes of q and s should be more theoretically derived for the method.", "methods_and_evaluation_criteria": "The datasets used are popular in the field of continual learning, and all the settings follow the standard convention. However, in some experiments, the hyperparameters need to be justified objectively.", "theoretical_claims": "There's no theoretical claim in this paper.", "experimental_designs_or_analyses": "The overall design of the experiments and analyses follows the standard convention.", "relation_to_broader_scientific_literature": "This is specific to the continual learning and less room for broader scientific literature.", "essential_references_not_discussed": "Most of the important references are discussed in the paper.", "other_strengths_and_weaknesses": "The strongest point of the paper is an extensive empirical study and interesting observation. However, despite the extensive efforts, the academic value is limited to the empirical study. The authors should investigate more in depth on the proposed method, and need to provide a theoretical justification on the proposed strategy.", "other_comments_or_suggestions": "None", "questions_for_authors": "- How can you determine the optimal values for q and s without such a arbitrary task?\n\n- What is the reason of the learning speed for the catastrophic forgetting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: This paper uses the learning speed of examples to decide whether or not to store them in replay-based continual learning, finding improvements. 

Reviewers agree that the method is well-motivated empirically. Reviewer 98Km asks for baselines beyond uniform sampling, and although I mostly agree with the authors' point that random uniform sampling performs very well (and often better than other methods), I think the paper would be stronger by including a couple of the better-performing sampling methods as baselines. I do note that the authors have Table 1 already. 

All reviewers mention the issue of hyperparameter tuning, specifically the cost of the Rotnet task. The authors rebutted that q and s are usually set at the same values. I think this paper would be stronger if the authors also used another, cheaper, hyperparameter tuning strategy and showed that their method still performs well. This is especially important given that this paper is a more empirical and less theoretical work, and therefore it is important to show the benefits empirically. 

It is extremely important to add class-incremental results. The authors promised to add these results in a future version but they are not there right now (or in the rebuttal text explicitly).
Decision: Accept (poster)