How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning
Authors: Giuseppe Serra
Ben Werner
Florian Buettner
Venue: Accepted by TMLR ()

Abstract: Many real-world applications require machine-learning models to be able to deal with non-stationary data distributions and thus learn autonomously over an extended period of time, often in an online setting. One of the main challenges in this scenario is the so-called catastrophic forgetting (CF) for which the learning model tends to focus on the most recent tasks while experiencing predictive degradation on older ones. In the online setting, the most effective solutions employ a fixed-size memory buffer to store old samples used for replay when training on new tasks. Many approaches have been presented to tackle this problem and conflicting strategies are proposed to populate the memory. Are the easiest-to-forget or the easiest-to-remember samples more effective in combating CF? Furthermore, it is not clear how predictive uncertainty information for memory management can be leveraged in the most effective manner. Starting from the intuition that predictive uncertainty provides an idea of the samples' location in the decision space, this work presents an in-depth analysis of different uncertainty estimates and strategies for populating the memory. The investigation provides a better understanding of the characteristics data points should have for alleviating CF. Then, we propose an alternative method for estimating predictive uncertainty via the generalised variance induced by the negative log-likelihood. Finally, we demonstrate that the use of predictive uncertainty measures helps in reducing CF in different settings.

Published in Transactions on Machine Learning Research (03/2025)
How to Leverage Predictive Uncertainty Estimates for Reducing Catastrophic Forgetting in Online Continual Learning
Giuseppe Serra
Serra@med uni-fruuuuui.f
Goethe University Frankfurt, Frankfurt, Germany German Cancer Consortium (DKTK)", Frankfurt, Germany
Ben Werner
buerner.sci@gmail. com
Goethe University Frankfurt, Fronkfurt, Germany
Florian Buettner
fordan.bucther@@uffhcheeeeee
Goethe University Frankfurt, Prankfurt, Germany
German Cancer Consortium, (DKTK)", Frankfurt, Germany
Germon Cancer Research Center (DKFZ), Heidelberg, Germany
Reviewed on OpenReview: https: //
/ttps:///penreview openreview. net/ forum? id= dcaXe0S1o1.
Abstract
Many real-world applications require machine-learning models be able deal with non- stationary data distributions and thus learn autonomously over an extended period of time, often in an online sctting. One of the main challenges in this scenario is the so-called catastrophic forgetting (CF) for which the learning model tends to focus on the most recent. tasks while experiencing predictive degradation on older ones. In the online setting, the most effective solutions employ a fixed-size memory buffer to store old samples used for replay when training on new tasks. Many approaches have been presented to tackle this problem and conflicting strategies are proposed populate the memory. Are the easiest-to-forget or not clear
the easiest-to-remember samples more effective in combating CF? Furthermore, how predictive uncertainty information for memory management. can be leveraged in the most effective manner. Starting from the intuition that predictive uncertainty provides idea of the samples' location in the decision space, this work presents an in-depth analysis of different uncertainly estimates and strategies for populating the memory. The investigation provides a better understanding of the characteristics data points should have for alleviating CF. Then, we propose an alternative method for estimating predictive uncertainty via the generalised variance induced by the negative log liceliood Finally, we demonstrate that the use of predictive uncertainly measures helps in reducing CF in different settings.
Introduction
Typical machine learning models assume to work in a single-task static scenario where multiple epochs are performed over the same data until convergence. In many real-world situations, however, this setting is too limiting. As example, let consider the problem of product recommendation. Since trends may change and new types of products may arrive, new product categories need to be classified. In this context, typical learning model would fail because the standard setup does not account for the continuous addition of new classes, For this reason, online Continual Learning (online-CI) CLL has been constantly more explored. In online-CL, a single model is required to learn continuously from a sequence of tasks that comes as a stream of tiny batches which can be processed only once (Aljundi et al. 2019b 2017 Mai et al. 2022 This is to reflect
partner Frankfurt, a partnership between DKFZ and UCT Tranffurt-Marbug

---

Published in Transactions on Machine Learning Research (03/2025)
realistic conditions where, for example, new personal data arrive with high-frequency to limited-resouress devices (e.g., wearable smart devices) and the model needs to be updated with the incoming data to adjust the model on the fly. In this context, given the overlap between old and new information, the model tends to forget about the past knowledge (which still needs to be classified) to focus more on the newest tasks, leading to a performance degradation on previous tasks. This challenge is usually referred to as calastrophic forgetting (CF) (McCloskey & Cohen 1989 Ratcliff 1990).
Many approaches have been developed in online-CI. to prevent catastrophic forgetting and the most successful ones fall in the memory- bosed category. These approaches employ memory buffer (Chaudhry et al 2019 Soutif-Cormerais et al. 2023) to a. store samples of past tasks and b) tackle CF by training the model on both current samples and some old samples stored in such limited-size memory (replay or rehearsal). In this context, what differentiates each memory-base based approach are the update (or population) strategy and the retrieval (or sampling) strategy (Mai et al., 2022) i.c., how to populate and update the memory with meaningful and reprosentative samples, and how to efficiently sample from the memory respectively.
Although many approaches have been developed in this direction ranging from using random approaches (Chaudhry et al.] 2019) to exploiting the gradient (Lopez-Paz & Ranzato 2017] Chaudhry et al. 2018b)
Aljundi et al.) 2019b) or the loss (Belilovsky et al.) 2019) information it is difficult to identify a clear strategy to exploit the memory at its best. In fact, contrasting strategies can be found in the literature. For instance, Kumari et al. (2022) suggest working on instances close to the decision boundary, thus considering the marginal samples as the most important for alleviating CF. Contrarily, in Hurtado et al. (2023) and Yoon et al.] (2021). the proposed strategies focus on populating the memory with the most representative samples for each class. Hence, it is unclear which type of data points would reduce CF in consistent manner: Are the least or the most representative samples more effective in combating CF?
Starting from this question, our investigation seeks to understand the contribution of the most or least rep- resentative samples in combating CF through an uncertainty lens. In detail, we will focus on the uncertainty of the model in its predictions, iee, predictive uncertainby. Intuitively, using measures of uncertainty to populate the memory represents a solution for identifying the location of the instances in the decision space. However, predictive uncertainty can be seen as composition of aleatoric (data-inherent and irreducible) and epistemic (model-centric and reducible by gathering more data) uncertainty (Malinin & Gales 2018 Muesányi et al. 2024), and different uncertainty scores focus more on one source or another leading to different results accordingly. The most common confidence scores mostly capture the (irreducible) aleatoric uncertainty (Wimmer et al., 2023) which is inherent in the data and cannot be reduced by gathering more data. In this case, samples with high confidence from an alcatoric perspective reflect data points that are distant from the decision boundary irrespective of the fact that they may be outliers or not. In the epistemic case, instead, samples with high confidence (i.e., low epistemic uncertainty) indicate that the observed data sufficienlly support the inferred patterns (or, in other words, that the samples are representative of the data distribution). Thus, although representativeness is an intrinsic property of the data, we can interpret (low) epistemic uncertainty as a way to infer this property from a predictive model (sce Section 4.2.3 for further details). For this reason, we hypothesize it would be beneficial to focus on samples with a low epistemic uncertainty. To empirically validate this hypothesis, we propose a new population strat- egy based on recent theoretical contributions in uncertainty quantification via the Bregman decomposition. More specifically, we introduce a memory management strategy based on the Bregman Information (BI) as generalised variance measure, which stems directly from hias-variance decomposition of the model loss (Gruber& & Buettner 2023). Based on this insight, we interpret BI a measure of epistemic uncertainty that is statistically well grounded in a bias-variance decomposition, and propose to use it for populating the memory.
In the first part of this work, we evaluate and compare different combinations of uncertainty scores and sorting on CIFAR-10 and CIFAR-100 (Krizhevsky et al. 2009), two datasets commonly used in CL. The objective is to understand how different uncertainty estimates behave when used in different ways under same conditions. In order to achieve the intended goal, following a recent trend research transparency and comparability (Mundt et al. 2021), the evaluation framework will be freed from any other 'trick' that could create ambiguity in the assessment. The investigation will focus on describing the characteristics of the considered uncertainty scores, while providing an in-depth analysis of the effect of the metrics under memory- based regime. In
2

---

Published in Transactions on Machine Learning Research (03/2025)
CIFARLO0
CIFAR10OLT
0.4
0/2
0.3
Relative Improvement
0.3
6.1
0.0
Top-k
Bottom-k
-0.3
-02
MS
RM
(ours)
RC
EN
BI (ours)
Figure 1: Relative CF improvement of different uncertainty scores over ER when using the easiest-to-forget (lop-k) and the casiest-to-remennte (bollom-k) samples. The bottom-k strategy (in darker colour) reduces CF in all cases. Furthermore, the proposed BI-based uncertainty estimate (in green) further diminishes CF compared to common uncertainty scores.
the second part of our work, we evaluate our findings on more challenging and realistic scenarios. realistic setups, recent tasks have less data points available than older tasks as the time to collect instances is considerably shorter than for previous tasks. Considering this behaviour, we will focus on long-tailed (LT) datasets with this characteristic. We test findings two datasets with controllable degrees of data imbalance, well real-world imbalanced dataset for classification of biomedical images (Yang et al. 2023). The goal of this work is not to propose new method that outperforms the slate-of-the-art, but rather to present practical insights from an uncertainty-aware perspective on the desirable characteristics samples should have to alleviate catastrophic forgetting both standard and rcalistic sconarios.
Thus, the main contributions of this paper can be summarised as follows:
We conduct systematic evaluation of established predictive uncertainty scores to assess their effec- tiveness in mitigating catastrophic forgetting (CF) in online continual learning for image classification tasks.
We propose the use of Bregman Information (BI) as measure of epistemic uncertainty to populate the memory in online-CL.
As summarised in Figure [ we demonstrate that a) selecting the most representative samples (bottom-k) to populate the memory consistently reduces CF, as compared choosing marginal samples (top-k); and b) focusing on estimates of epistemic uncertainty (BI) provides an advantage compared to aleatoric uncertainty estimates.
Different from conventional evaluation pipelines, we validate findings both standard bench- marks and realistic scenarios, including challenging settings like medical image classification with imbalanced data.
2 Related Work
2.1 Continual Learning
According to van de Ven et al. 2022 three different scenarios of incremental learning (IL) can be identified; 1) Task-IL where the task ID information is available at both training and testing time; 2) Domain-ll ILL in which the lcarning task remains unchanged (c.g., binary classification) but there is shift in the input distribution; and 3) Class-IL where the number of classes to discriminale can grow over time. Additionally, CL problems can be further categorised depending on whether the data can be accessed and processed multiple times (offline) or singlo-pass through the data expected (online) (Mai al.) 2022). In this work, following a popular trend in the literature, we focus on the most challenging scenario, i.c., online class-IT..
3

---

Published in Transactions on Machine Learning Research (03/2025)
2.2 Class- Incremenaee Learning
Following the classification proposed by Mai et al (2022). class-IL approaches can be grouped into the following categories: a) Regularization techniques that adjust the model parameter updates by incorporating penalty terms in the loss function (Aljundi et al. 2018 Lee et al. 2017). modifying parameter gradients during optimization (Chaudhry et al.; 2018b; He & Jaeger 2018). or employing knowledge distillation (Wu
jtt al. 2019; Rannen et al.] 2017): : b) Memory-based techniques in which a fixed-size subset of past samples
is stored for replay (Aljundi ct al. 2019a Chaudhry et al. 2019) or for regularization purposes (Nguyen
jet al., 2018 Tao et al., 2020) ; c) Generalive-based Lechniques which involve training generalive models to
produce pseudo-samples that replicate the information from past tasks (Lesort et al 2019 Shin et al. 2017) and d) Parameter-isolation--ee
techniques that allocate distinct model parameters to each task, either by activating only the relevant parameters for each task (Fixed Architecture) (Mallya & Lazebnik 2018 Serra let al. 2018) or by adding new parameters while keeping the existing ones unchanged (Dynamic Architecture)
(Yoon et al. 2018 Aljundi et al. 2017).
In online-CL, where data are received in mini-batches and the model is updated with high frequency, rehearsal-based methods are favoured over more complex solutions like generative methods due to their flexibility and reduced training time (Mai et al. 2022).
2.2.1 Population Strategies in Rehearsal-based CL
As already anticipated, there exist methods that use different metrics for deciding the samples to store in the memory. In [ljundi et al. (2019b), the gradient information is used as a feature to maximize the diversity of the samples in the memory. In Chaudhry al. (2018b), the gradient of the current mini-batch is compared with with the gradient of randomly sampled set of the same size from the memory. If the dot product between the current gradient and the memory gradient is negative, the current gradient is projected. Otherwise, the gradient is used normally. Similarly, in Yoon et al. (2021), gradient vectors are used to select the most representative and informative coreset at each iteration. Other methods exploit the loss information to select samples based on their interference on the loss function (Aljundi et al. 2019a) or to analyze the loss region for improving generalization and minimizing CF (Verwimp et al. 2021 J. Sun et al. (2021) and Wiewel & Yangl (2021) address the problem of memory diversity from an information-basee perspective via entropy-based functions, As previously mentioned, the characteristics samples should have to be stored in the memory and alleviate CF are not clear. For instance, Hurtado et al. (2023) propose a method to eliminate outliers from the memory such that only the most representative samples for each class are kept via a label-homogeneit score. Intuitively, their approach inspect the neighborhood of a given sample to check whether it is surrounded by samples from different classes or not. If the label homogeneity score is low (i.e., most of the nearest samples do not share the same class label), it means we are in the presence of an outlier or noisy sample and the considered sample should be discarded from memory sampling. Contrarily, in Kumari al. (2022), the authors claim that the replay-phase should focus on marginal samples and propose method to synthesize samples near the forgetting boundary that are confused with the current task. The approach consists on identifying the easiest-to- samples in the memory and then on moving those samples closer to the data points of the current task. Finally, the new perturbed samples are used for replay to cover marginal samples more consistently.
In both cases, the idea is similar to using uncertainty estimates to detect data points with specific character- istics i.c., outliers or representative samples. Differently from their approaches, which require the samples' vector representation, our method exploits predictive uncertainty for the same purpose. It can be seen as more principled way of identifying whether sample is representative of its class or not. Nevertheless, despite the use of predictive uncertainty looks appealing, its usage for populating the memory with samples with desired properties is largely unexplored.
2.2.2 Uncertainty-aware Memory Management in CL
The idea of employing predictive uncertainty for memory management is borrowed from Active Learning (AL). In this context, uncertainty sampling (Lewis & Gale 1994) is used to decide which samples from a pool of unlabeled data will be considered for labeling. The main assumption is that the most uncertain
4

---

Published in Transactions on Machine Learning Research (03/2025)
samples represent the most informative data points. improve the overall performance. Popular examples of uncertainty scores include, to name a few, entropy, smallest margin, and least confidence (Shannon 1948 Campbell et al. 2000; Lewis & Gale 1994 Culotta & McCallum 2005).
Thus, by including them in the training set, we
Analogously, we can use estimates of predictive uncertainty for populating memory buffers in replay-basod methods for Online-CL. In [ang et al. (202I), the authors argue that samples stored in the memory should be representative of their own class, but also discriminative towards the remaining ones. Starting from this assumption, they evaluate the relative position of the samples in the decision space by estimating the predictive uncertainty of perturbed samples; exemplars with high uncertainty should be closer to the decision boundary, while the most certain ones should be located closer to the center of the corresponding class distribution. To ensure diversity, they use a step-sized sampling which guarantees to keep in the memory samples that span from the most uncertain to the most certain ones. Given a list P of perturbations, the authors propose to use the following uncertainty score URM (z):
(2) = P
max S.
(1)
where S. = -EP= Le argmax, p(y = a...). Eq. represents an agreement score with respect to the perturbations. If the predicted classes è of all the perturbed versions of x (i.c., 5;) are prodicted with the same label e, then uRas(x) will be 0. Otherwise, the higher is uRA (r) the most uncertain is the model about the considered sample.
The authors claim to work in the online setting. Nevertheless, the definition of online setting in the paper differs from the popular one. In their case, online means that the training stream is only processed once and the memory is updated at the end of each task. In the standard scenario, instead, online implies that only a mini-batch of data points is available to be processed (Mai et al.) 2022), and the model and the memory are updated with high frequency (Soutif- Cormerais et al.) 2023). In addition to this, to further enhance diversity, they employ data augmentation on the memory. Thus, the contribution of Eq. (1) on reducing CF is unclear.
In the remainder of this work, we systematically investigate the effect of established uncertainty scores including unM as well as the newly proposed Bregman Information on combating catastrophic forgetting in rcalistic online-CL setting.
3 Preliminaries and Notation
Following the notation proposed in Bang et al. 20021,, we assume to have a set C = fci...., Cn of different classes. Each class can be randomly assigned to a task t. T; represents a subset of classes determined by an assign function vb()) such that Тe = (c/p/c) = t}. For each task t, we have an associated dataset D. = {(x,, y: )i with H; an input sample, Y: the corresponding class label, and ni the number of training samples. In the proposed online setting, we assume that samples for each task arrive sequentially in a stream of mini-batches b, {(x,, vi) D with each mini-batch available for single pass. It is important to notice that, for each task t, n. can vary depending on whether or not. Finally, for replay, we introduce a fixed-size memory bulfer M to store a portion of samples from past tasks. The memory is updated with fighfrreuencyy whenever a mini-batch from the stream of data is processed. Following the standard procedure in online- CL, for roplay, we assume to extract from the memory a number of samples equal in size to the batch size.
working
class- balance setting
4 Methodology
To facilitate a fair evaluation of the framework, we focus on the assessment of the uncertainty metrics under same conditions. In this way, we climinate any ambiguous effect which could be inherited from other methodological choices. As anticipated in Section I there are two main objectives when dealing with memory-based approaches: 1) memory populating, and 2) memory sampling.
5

---

Published in Transactions on Machine Learning Research (03/2025)
In the following subsections, we describe the available strategies for each of the two objectives, and the uncertainty scores considered in our evaluation. Finally, we propose an alternative uncertainty estimate that overcomes the limitations of the most popular uncertainty metrics.
4.1 Memory Management
Memory Population. Apart from the criteria-based population strategics presented in Scction 2.2.1
random stralegies have demonstrated to work surprisingly well in Online-CL. Reservoir [Vitter 1 1985) is a random sampling technique without replacement to select n samples from pool of N samples where N is unknown and N > n. This strategy is used in Chaudhry et al. (2019) as a way to populate a memory of size M from a stream of data points with unknown length. In this way, each data point has a probability of being included in the memory equal to M. N. Class-bulanced Reserwoir, instead, based on a class-balanced random sampling strategy (Chrysakis Moens 2020). This is important to consider the case where the stream of data highly imbalanced. Indeed, class-imbalance may further deteriorale the predictive performance of the framework.
Different from the population strategy proposed in Chrysakis & Moens 2020p, where each class-memory set is populated and updated randomly, we introduce a class-balanced memory management based on predictive uncertainty estimates. This similar to [ang et al. (2021). ; where the intent is to keep in the memory samples with desired charactoristics for cach class.
Memory Sampling. Starting from the second task, we need a strategy to sample data points from the memory. The easiest way is random sampling, which selects at random a subset from the available data points in the memory buffer excluding the ones from the current lask. The size of the replay set is equal to the batch size. Another option would be to extract past samples with specific characteristics. For this purpose, we can exploit uncertainty estimates. However, this solution represents an effective strategy only in the case the memory is populated at random. Furthermore, given the size of the memory buffer and the frequency of memory updates, computing uncertainty estimates for the whole buffer may be computationally expensive and not feasible during training.
In our case, since the memory is populated with samples with specific characteristios, the random sampling is sufficient.
4.2 Uncertainty Metrics
4.2.1 Predictive Uncertainty Estimation
Following previous works (Bang et al. 2021 Wang et al. 2022 for a given input image z, we employ the ensembling technique Timt-Time Augmentation (TTA) (Wang et al. 2019). The set of transformations applied are the ones usually employed for image classification. A detailed list of the transfornations used in our evaluation can be found in the supplementary. After selecting the set of transformations, we apply them on the selected input. In this way, we create a set P perturbed inputs 0 which are used at test-time to compute their corresponding logits é. Finally, the generated logits are used by the uncertainty estimates in different ways to compute the uncertainty score u(x))
to compute predictive uncertainty estimations
4.2.2 Predictive Uncertainty Scores
In our investigation, we consider the following popular uncertainty metrics:
Least Confidence (LC) Culottta & McCallum 2005) determines the level of predictive uncertainty by examining the samples with the smallest predicted class probability. A model is less certain about the sample if it associates the most probable class U(1) with a low probability.
P p(uC) = ala:)
1 p
u()) = 1
(2)
6

---

Published in Transactions on Machine Learning Research (03/2025)
Margin Sampling (MS) (Campbell et al. 2000) calculates predictive uncertainty by computing the difference between the most probable predicted class 7(1) and the second largest one y(2)- If the difference is low, the model is uncertain.
u(z)Ms= = |
Σ (p(vc)) 8/2+) - p(y(2) = -(3.))
Ratio of Confidence (RC) (Campbell et al. 2000 is similar to MS. In this case, the ratio between the probabilities the two most probable classes is computed. The closer the ratio to 1, the more uncertain the model is about the considered sample.
plu() = -(;))) uxx) u(x)RC RC = P ==1 p(y(1) = c(,)))
1
(4)
differently from the above scores, estimates the uncertainty consid- ering the whole probability distribution. If the predicted probabilities are similar to each other (i.c., tend to an uniform distribution), the model is not certain about the sample. The formulation reads as follows:
Entropy (EN) (Shanoo 1948
P
M())EN
Σ Σ Er(us = log(P(y = e(z.))
(5)
P
In addition to these metrics, will also consider the agreement score u(z) reported in Eq. 1
4.2.3 Uncertainty Quantification via Bregman Decomposition
Inspecting the metrics described above, we can make two main observations: 1) all the metrics need a normalization step to squash the logits the [0, 1] range of the probability space; 2) the majority them compute the uncertainty relying on the largest predicted probabilities only. In both cases, with different magnitudes, there loss of information compared to using the logit space. Importantly, such confidence scores reliable mcasure of prodictive uncortainty only when the model is well calibrated, which is not the case in many real-world scenarios (Gruber & Bueltner 2021).
2023
Ovadia et al. 2019 Tomani & Buellner
Furthermore, the confidence scores detailed Section 4.2.2 mostly capture the alcatoric uncertainty which is inherent in the data and irveducible (Wimmer al. 2023). For this reason, we believe would be favorable focus on samples with low epistemic uncertainty which is model-centric and can be reduced by gathering more data. To provide an intuitive explanation behind this decision, let us consider Figure where we have graphical representation of the behaviour of the two different sources of uncertainty. With commonly used confidence scores, regions with low confidence correspond to areas close to the decision boundary irrespective of the presence or not of outliers. Contrarily, when exploiting measures mostly based on epistemic uncertainty, low confidence regions correspond to areas which the data density is low, i.c., where the uncertainty could be reduced by gathering more data. Consequently, we are interested in the high confidence samples, this would result in selecting diffcrent samples depending on the employed score. For confidence scores, given their distance from the decision boundary, we may obtain outliers not being representative of the class of interest.
In light of these considerations, following recent advances for computing uncertainty estimates in the logit space for classification lasks, we suggest to employ an uncertainty estimator based on the Bregman Informa- tion (BI) (Gruber & Buettner. 2023). The authors demonstrate that BI can be used to quantify the variance of the model at the sample level through deep ensembles or TTA. This approach particularly appealing from theoretical standpoint, since it stems from a general bias-variance decomposition of proper scoring rules which gives rise to the Bregman Information as the variance term of the loss.
7

---

Published in Transactions on Machine Learning Research (03/2025)
Data
Predicted Score
Bregman Information
10gh
Figure 2:
Illustration of the behaviour of the Bregman Information for uncertainty estimation in classification tasks. Given the focus on epistemic uncertainty, BI is low in the presence of high data density areas i.e., where the uncertainty could be reduced by gathering more data.
For the commonly used cross-entropy loss, this variance term that can be directly associated the epis- temic uncertainty be estimated via p TTA perturbations follows:
1 E LSE(E) LSE u(z) = P
1
P
(6)
where 2 E I2 and LSE( xn) In Σm= et represent the logits generated by the model and the LogSumErp (LSE) function respectively. Intuitively, a large value of u(r) means that the logits predicted across the perturbations are very different and, thus, the model not confident about the considered sample.
4.2.4 Uncertainty Scores Ranking
Depending on the characteristies we want to extract from the images, we can use the uncertainty estimates in different ways. In particular, in our experiments we consider three different strategies to select the set images for populating the memory depending on the uncertainty scores. By sampling images with the highest uncertainty (top-k), we aim at extracting the easiest-to-forget samples which are closer to the decision boundaries and/or represent outliers. With the slep-sized strategy, we aim at sampling diverse set of images, spanning from the most representative to the most uncertain ones. Finally, with the bottom-k approach we aim at finding the most representative set of images for cach of the classes scen in the past tasks. Note that, as explained above, different uncertainty scores select different images based on their greater focus on epistemic or aleatoric uncertainty.
5 Experiments
5.1 Datasets and Settings
5.1.1
Datasets
To understand the change in the performance when using different uncertainty strategies, we employ two datasets commonly used in online-CL for image classification, CIFAR-10 and CIFAR-100 (Krizhevsky et all, 2009). To configure the maline-CIL setup, randomly assign with different random seeds set of classes to 5 tasks. Thus, each task T, has 2 (CIFAR-10) or 20 (CIFAR-100) classes designated. In the second part of our experiments, once the best stralegy is identified, we evaluate our findings on class-imballnced sce- narios to assess the performance under more realistic conditions. First, we employ two artificially controlled imbalanced datascts, CIFARI0-LT and CIFAR100-LT (Cao et al.] (201)). As explained in Section [ with this experiment we want to replicate realistic scenario in which recent tasks contain less data than the older ones. For this, the most appropriate type of imbalance is the long-tailed (I.T) one (Cui et. al. 2019)
8

---

Published in Transactions on Machine Learning Research (03/2025)
which follows exponential decay to choose the sample size of each class. Specifically, for both datasets, we decided to use an imbalance factor p equal to 0.1. This means that, if the largest class contains, e.g., 500 data points, then the smallest one consists of 50 instances. Finally, we decide to focus on biomedical image analysis using the microscopic periphcral blood cell images dataset (BloodCell) which consists of 8 classes annotated by expert clinical pathologists (Acevedo et al. 2020 Yang et al.) 2023). To reflect. the intended realistic conditions, we assign classes for each tasks by increasing size. In this way, older tasks have more data points than the most recent ones. This simulates realistic scenario where most common subtypes are prevalent when collecting data for ML-model and, over time, less common disease subtypes appear in the clinic and the model needs to be updated with the new incoming classes. We provide more details about the motivations behind the choice of this dataset in Appendix A.4
5.1.2 Experimental Settings
The baseline for assessing the goodness of the proposed strategies will be Experience Replay (ER) (Chaudhry jt al. 2019). ER consists of reservoir approach for the memory management part, and a random sampling for replay. We decide to use ER because, despite its simplicity, is surprisingly competitive compared to more sophisticated and newer approaches. This corroborated by recent empirical surveys finding that newly introduced methods perform very similarly to the common Experience Replay (ER) method (Soutif- Cormerais et al. 2023). In addition to the standard strategy, we will also consider the class-balanced version (CBR) proposed in Chrysakis Moens (2020) and previously described in Section 4.1 and gradient-based method (A-GEM Chaudhry al. (2018b)) for comparison. Finally,
assess the results in comparison with Monte Carlo Dropout (MC) Gal & Ghahramani] 2016). As reported in Ovadia et al. (2019), MC can be used to estimate predictive uncertainty and considered as competitive bascline under distribution shifts. Since online-CL, can be seen as a special case of data under distribution shifts, we include MC as an additional baseline to estimate predictive uncertainty from a Bayesian perspective.
In all the experiments, we employ a slim version of Resnet18 HHe et al.) 2016 as done previous works (Hurtado et al.] 2023 Lopez-Paz Ranzato, 2017 Soutil-Cormerah et al. 2023 Kumari et al., 2022 and use the SGD optimizer with learning rate of 0.1. Following standard practice we set the batch size equal to 10. We set the memory size to different values for each dataset to evaluate a variety of memory configurations considering both large or small buffers.
For ovaluation, following previous works, we employ the Last Accuracy (A) and Last Forgetting (F) 19)-des defined in Chaudhry el al.] (2018a). Last refers to the calculation of the metrics at the end of the training on all tasks. Let denote with a² and T, the accuracy of task after learning task and the total number of tasks respectively.
The last accuracy A is then defined as follows:
T A
T
(7)
T
1=1
Last Forgetting (F) is defined as the difference between the peak knowledge (i.c., the maximum accuracy) captured about a particular task during the learning process and its accuracy at the end of the learning process. This provides an idea of the information retained about a certain task after learning new tasks. The last forgetting (F) is computed as:
T T-1
Σ le
d/d 11)
max
aTs
Yj<
(8)
T-1
All the experiments are run on three different random seeds. The code to conduct the experiments is available at http::///tttaac...ooooo/////atts

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Comparison of last accuracy (A) and last forgetting (F) on CIFARIO.
Score
M=500
M 1000
26.38 2:27
36.50
ER
74.92 4.93
57.42 #: 5.93
24.65
30.88
CBR
76.70 70 +: 5.29
66.40 :: 5.58
28.81 1.36
30.55 :: 1.54
A-GEM
56.56 0.18
56.02 + 4.49
Top
Step
Bottom
Top
Step
Bottom
A(t)
25.27
30.51
34.49
30.70
38.03
38.30
LC
F(L)
72.88 + 2.76
58.07 :: 6.23
53.11
6.85
64.48 +: 4.00
48.04 6.64
41.71 +: 6.68
A(t)
26.33
30.16
32.50 2.31
33.29
38.85
39.36
MS
F(L)
71.08 + 5.00
61.19 +: 5.74
56.03 4.50
59.75 + 1.87
48.89 8.57
43.84
+ 3.42
RC
A(t)
27.84 + 2.33
30.57 :: 2.43
32.49
3.86
36.35 + 1:35
37.68 1.17
39.63
F(L)
66.13 + 7.78
59.70 à: 6.71
54.73
7.06
54.53 4: 4.54
48.08
3.09
41.9
3.53
EN
A(t)
23.72 # 1.50
28.76
+ 1.7G
35.20
1.52
26.57
+ 0.98
36.13
1.56
35.78
+: 4.99
F(L)
77.04 + 3.78
61.20
3.33
46.12
3.02
71.66 + 3.78
50.37 3.44
42.65
8.07
RM
A(t)
26.92
+ 1.75
28.57
+: 0.80
35.71
+ 4.12
32.87
+ 2.56
38.23
0.95
38.57
#: 2.09
F(L)
71.45
2.92
61.16 4.40
51.01
6.60
62.01
+: 4.96
47
3.68
42.59
+: 2.79
MC
A(T)
28
1: 2.05
28. 72
1: 2.57
27.52
4.84
28.96
1: 2.69
32.87
3.31
28.04
+ 1.58
F(L)
14.97
5.56
47.05
5.57
38.01
3.38
45.10 +
39.18 6.23
40.85
A(t)
26.41
# 1.17
28.88 1.96
31.10
6.24
28.62
+ 2.14
37.44 2.87
40.44
+ 3.09
F(I)
72.68
61.21
43.03
69.02
47.68
37.79
Table 2: Comparison of last accuracy (A) and last forgetting (F) on CIFAR100.
Score
M=1000
M=2000
14.23 i: 1.00
16.84 +: 0.96
ER
29.32 0.16
22.67 1.35
13.38 0.58
16.59 + 0.41
CBR
31.66 0.30
23.05 1.29
14.06 0.46
16.52 + 1.05
A-GEM
33.23 1.81
28.80
Top
Step
Bottom
Top
Step
Bottom
A(t)
13.33 + 0.16
12.52 + 0.65
14.07 = 0.36
15.63 + 1.28
15.06 0.69
15.44 + 1.31
LC
F(L)
25.18 + 1.51
27.50 1.02
22.67 3.02
19.71 4: 2.82
20.56 1.93
17.50 a 3.48
MS
A(T)
13.69
+ 1:25
11.69 + 1.07
13.34
0.22
15.76
+ 0.98
14.73
0.92
16.32 # 1.11
F(L)
24.45
1.82
27.05
25.64
20.23
18.14 1.28
17.50
RC
A(t)
13.65
+: 0.97
12.65 0.07
13.94
0.78
16.69 +: 1.77
14.30 0.80
16.03 # 0.88
F(L)
25.71
27.55
24.99
17.15 +
20.72
17.68
A(t)
13,00
1: 0.64
12.79 0.30
13.42
0.87
14.18 1: 1.77
14.49 0.73
14.42 1: 0.78
EN
F(L)
25.30
27.82
23.20
4.00
20.93
19.59
17.43
RM
A(t)
12.99
1.35
12.24
13.65
0.75
15.62
14.33
0.64
16.39
F(L)
26.01 1 3.18
0.98
24.27
3.52
19.49 1: 4.25
1.55
16.68
1: 3.67
A(t)
9.48
10.49
10.77
10.99
12.36
10.68
MC
F(L)
16.77
4: 2.42
15.09 1.41
7.50
2.53
15.98 1: 1.45
0.90
9.08
4.36
A(t)
13.38
12.27
13.81
14.52
14.33
17.03
BI
F(L)
25.43
+ 3:23
25.50 +: 1.16
19.41
1.74
20.87 + 3.51
17.68
1.63
13.30 + 3.65
10

---

Published in Transactions on Machine Learning Research (03/2025)
CIFAR10 10
CIFAR1 00
ER
0.40
CBR
Last Accuracy (A)
MC
0.35
Bi-topk
0.30
Bl (ours)
0.4
0.20
0.15
0.10
0.05
Task
Task
Figure 3: Average task accuracy at the end of the learning procedure for different strategies and datasets. One can observe how the proposed strategy based on the Bl perform consistently across tasks and helps maintaining competitive performance on old tasks and thus reducing catastrophic forgetting.
5.2 Empirical Results
show the results for all the uncertainty metrics and sorting strategies employed for populating the memory. From the values reported in the tables, can observe that selecting the most representative samples for each class ("Bottom' column, in green) consistently improves the results in terms of both accuracy (A) and forgetting (F) across all the considered uncertainty scores and memory sizes. some cases, ER is able to outperform our proposed approach in terms of predictive accuracy. However, taking closer look at the single mumbers, we can observe that the higher accuracy not accompanied by a smaller forgetting value. Thus, our results indicate that the most representative samples are beneficial in improving the predictive performance of the learning model and in reducing considerably CF.
Tables and 2
The comparison among the different uncortainty strategics suggests that BI outperforms the other metrics in most cases. While the difference among the accuracy values less pronounced, BI stands out in its effectiveness at mitigating CF. Given the reduced information loss compared to the most popular uncertainty metrics, we think that BI is more sensitive to changes the predictive space and thus able detect changes when subtle differences are present. We can notice that MC delivers the smallest forgetting value (F) in most of the cases. However, this is because MC performs poorly in the predictive task (see Figure 3 and considering how forgetting is computed, the resulting value is inevitably smaller. The poor performance of MC compared to other strategics is probably given by the online nature of the setting since, as reported in the original paper, dropout takes longer to converge (Gal & Ghahramani 2016). Thus, a single-epoch setup may be insufficient. for training convergence. All these findings are evident Figure 3 where we can observe that BI able to maintain competitive predictive performance on all tasks at the end the training compared to other strategies.
Bottom-k
Top-k
Figure 4: Class-spocific samples stored the memory according different sorting strategies: bottom-k se- lects prototypical examples of the considered class facilitating the recollection of the class characteristics and reducing CF; lop-k selects the most. uncertain and hard samples making difficult to learn clear discriminative patterns and to reduce CF.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Comparison of last accuracy (A) and last forgetting (F) on long-tailed imbalanced datasets.
Score
CIFARIO-LT LT
CIFARIOO-LT
BloodCell
M-200
M-500
M=500
M=1000
M-40
M-160
A(t)
22.06 + 1.13
31.62 + 1.65
8.22 0.36
9.30 + 0.22
58.82 + 6.62
64.08 + 11.98
ER
F(L)
76.19
+ 3.17
32.09 = 3.21
20.16 0.9
15.78 0.62
49.60 + 8:23
28.50 = 5.75
CBR
A(1)
22.38
+ 2.84
27.34 = 5.84
8.10 = 0.70
8.75 + 0.33
55.74 + 2:21
70.11
= 2.06
F(L)
77.41 4: 5.76
62.59 7.66
21.82 1.98
18.32 1.84
57.53 4: 2.76
29.56
à: 5.80
A-GEM
A(1)
21.79
1,30
23.10
1.69
8.39 1.65
9.95 :: 1.83
49.44 4: 3.92
55.24
à: 3.37
F(L)
63.75
+ 6:20
68.19 + 1.38
28.60 0.96
25.76 + 0.84
54.89 + 4.37
24.96
= 5.96
LC
A(T) 31.16
4: 1.24
32.5 + 1.05
7.15 0.73
8.93 # 1.05
51.19 + 5.49
66.08 # 1.78
F(L)
42.31
+: 2.81
37.59 + 2.01
17.53 1.71
13.98 2.66
27.33 + U.64
17.03
= 3.05
MS
A(1) F(L)
29.62 + 1.70
33.48
1.23
7.20 = 0.67
9.11 + 0.66
55.28 +: 5.04
68.34
+ 3.38
47.31
+ 2.04
39.46 2.45
19.36 1.79
14.43 + 1.85
34.41 + 5.90
14.21
# 2.80
A(1)
31.09
+ 0.79
32.48
1.21
7,86 0.45
9.47 + 0.59
51.88 +: 3.59
63.92 =: 4.31
RC
F(L)
44.38 + 1.21
40.65 + 1.75
18.51 1.38
15.33 +: 2.04
28.43 +: 2.91
21.20 + 7.00
EN
A(1) 31.34
+ 0.19
32.53
+ 1.89
7.02 = 0.49
8.48 # 0.71
49.32 + 3.86
69.38 + 1.13
F(L)
41.98
+: 2.06
32.70
17.79 2.18
14.34 2.99
32.37 4: 2.22
11.41 + 1.36
A(1)
28.50 +: 2.04
33.98
7.89 = 0.90
9.90 +
57.13 4: 3.18
67.97
+ 4.19
RM
F(L)
51.20 + 5.82
39.17 + 3.95
18.36 1:27
13.54 + 1.64
31.93 + 4.18
14.29 + 7.72
A(1)
25.84
+ 4.35
23.16 + 1.03
7.42 2.12
7.76 + 2.23
55.90 + 0.18
51.90 = 5.25
MC
F(L)
43.03 4: 10.72
28.75 1.55
7.84 :: 2.62
8.74 2.69
16.96 4: 0.61
9.9 :: 6.08
BI
A(1)
30.35
+ 1.01
34.37
1.26
8.93 + 1.02
10.35 + 0.58
58.08 + 7.08
71.88 +: 2.33
F(L)
40.66 1.06
34.93
16.31
13.27 1.10
18.27 7.25
8.71
To provide a graphical intuition of the differences between the sorting strategies and their plausible effect on the learning capabilities of the model, Figure | depicts a subset of class-specific samples stored in the memory at the end of the training for the two opposite population strategies, i.e. bottom-k and top-k. By inspecting the images, we note that the ones selected via the bottom-k strategy show some consistency and are easily identifiable as birds. On the opposite side, the lop-k samples present images with different perspective, zooming, or background making it more difficult, even for the human eye, to identify bird. For this reason, we believe that the casiest-to-remember images are more useful to the model for recalling the past information while the easiest-to-forget data points may hamper the generalization of the model since we could be in the presence of noisy images or outliers.
Finally, in Table we evaluate our findings in the more realistic long-tailod data imbalance scenario. In both cases (artificial and real data imbalance), the results support our findings from the standard balanced setting: BI substantially reduces CF while delivering competitive or improved predictive performance.
6 Conclusion
Given the conflicting nature of the strategies available in the literature for populating the memory, our investigation seeks under uncertainty lens to a) clarify which characteristics of the samples alleviate CF in consistent manner, and b) assist practitioners in selecting the appropriate memory management stratogy. Starting from the examination of the propertics and behaviours of popular uncertainty estimates, we identify that they mostly capture the irreducible aleatoric uncertainty and hypothesize that a better strategy should focus on the epistemic uncertainty instead. For this, we propose to use an uncertainty estimate based on the Bregman Information which, via general bias-variance decomposition for strictly proper scores, identifies the variance term as an estimate for the epistemic uncertainty. Our findings indicate that the most representative samples ie., the rasiebbrtorremember or the least uncertain ones are the
12

---

Published in Transactions on Machine Learning Research (03/2025)
most effective at mitigating CF (Figure 3) while maintaining competitive performance in classification tasks (Tables [ and 2 Furthermore, the newly proposed BI-based uncertainty score for memory management shows its superiority in reducing CF compared to well-established uncertainty scores and other memory- based strategics (Figure D Due to its emphasis on epistemic uncertainty, the memory contains samples that are representative of their respective class distributions (Figure 4 while being close enough to the decision boundary, helping the learning model to distinctly shape the boundaries between the classes during training. The results on various realistic setups (Table 3) not addressed in related work further support our findings and demonstrate that the use of BI-based predictive uncertainty estimates for reducing CF in online-CI. is appealing and well-motivated even in more challenging scenarios.
Acknowledgments
This work was supported by the The Federal Ministry for Economic Affairs and Climate Action of Ger- many (BMWK, Project OpenFLAAS 01MD23001E). Co-funded by the European Union (ERC, TAIPO, 101088594). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
References
Andrea Acevedo, Anna Merino, Santiago Alférez, Angel Molina, Laura Boldú, and José Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in brief, 30, 2020.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3366- 3375, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytclaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference compuler vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advonces in neural information processing systems, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019b.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8218 8227, 2021.
Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, Tinne Tuytelaars, Rahaf Aljundi, et al. Online continual learning with maximally interfered retrieval. Advances in Neural Information Processing Systems, (NewrIPS), 2019.
Colin Campbell, Nello Cristianini, Alex Smola, et al. Query learning with large margin classifiers. In 1CML, volume 20, pp. 0, 2000.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with dibablddsstrbutionnawwar margin loss. Advances in neural information processing systems, 32, 2019.
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. 532 547,5 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong
learning with a-gem. In International Conference Learning Represendations, 2018b.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhosciny, Thalatyasingam Ajanthan, P Dokania, P Torr, and M Ranzato. Continual learning with tiny episodic memories. Tn Workshop on Mulli- Mulki-Task and Lifelong Reinforcement Learning, 2019.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In Inter- national Conference on Machine Learning, pp. 1952-1961. PMLR, 2020.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongic. Class-balanced loss based on effec- live number of samples. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 9268 0977,, 2019.
Aron Culouta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pp. 746-751, 2005.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Scbastian Gruber and Florian Buettner. Uncertainty estimates of predictions via a general bias-variance decomposition. In International Conference on Artificial Intelligence and Stalistics, pp. 11331 11354. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770- 0007778--0 2016.
Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backpropagation. In International Conference on Learning Representations, 2018.
Julio Hurtado, Alain Raymond-Sácz, Vladimir Araujo, Vincenzo Lomonaco, Alvaro Soto, and Davide Bac- ciu. Memory population in continual learning via outlier elimination. In Proceedings of the IEEE/CVP International Conference on Compuler Vision, pp. 3481 3481-3490, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. online, 2009.
Lilly Kumari, Shengjie Wang, Tianyi Zhou, and Jeff Bilmes. Retrospective adversarial replay for continual learning. Advonces in Neural Information Processing Systems, 35:28530-28544 2022.
Sang- Woo Lee, Jin-Hwe Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. In 2019 International Joint Conference Newral Networks (IJCNN), pp. 1-8. IEEE, 2019.
D Gale Lewis and William A Gale. A sequential algorithm for training text classifiers. In Proceedings of SIGIR-94, pp. 3 3-12, 1994.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing. 469:28 51, 2022.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural
information processing systems, 31, 2018
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pp. 109-165. Elsevier, 1989.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Bálint Muesányi, Michael Kirchhof, and Scong Joon Oh. Benchmarking uncertainty disentanglement: Spe- cialized uncertainties for specialized tasks. Advonces in Neural Information Processing Systems, 2024.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. Cleva-compass: A continual learning evaluation assessment compass to promote research transparency and comparability. In International Conference on Leurning Represenlations, 2021.
Cuong Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. International Conference on Learning Representations, 2018.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's unccrtainty? evaluating predictive uncertainly under dataset shift. Advances in neural informalion processing systems, 32, 2019.
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proceedings of the IEEE international conference on computer vision, pp. 1320-13200 2017.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting
functions. Psychological review, 97(2))285, 1990.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp. 4548 4666-4557.00 PMT.R, 2018.
Claude Elwood Shannon. A mathematical thcory of communication. The Bell system technical journal, 27 (3):379 423, 1948.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Albin Soutif- Corimrais,,,,,s,, Antonio Carta, Andrea Cossu, Julio Hurtado, Vincenzo Lomonaco, Joost Van de Weijer, and Hamed Hemati. A comprehensive empirical evaluation on online continual lcarning. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 3518 3528, 2023.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. nntematiin-theoreffe online memory selection for continual learning. International Conference on Learning Representations, 2021.
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology-preserving class- incremental learning. In Compuler Vision ECCV 2020: 16th European Conference, Glasgow, UK, Augusl 28-28, 2020, Proceedings, Port XIX 16, pp. 254-270. Springer, 2020.
Christian Tomani and Florian Buettner. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.
2021.
Gido M van Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185 1197, 2022.
Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9385-9394, 2021.
Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
Guotai Wang, Wenqi Li, Michael Acrtsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren. Alcatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocompuling, 338:34-44 45, 2019.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sebastien Ourselin, and Tom Vercauteren. Test-time augmentation with uncertainty estimation for deep learning-based medical image segmentation. 2022.
Felix Wiewel and Bin Yang. Entropy-based sample selection for online continual learning. In 2020 28th European Signal Processing Conference (EUSIPCO), pp. 1477-1481. IEFE, 2021.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pp. 2282-2292. PMLR, 2023.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognition, pp. 174-382, 2019.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Dala, 1011):41, 2023.
Jachong Yoon, Eunho Yang, Jcongtae Lec, and Sung Ju Hwang. Lifelong learning with dynamically expand- able networks. In International Conference on Learning Represenlations, 2018.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal- based continual learning. In International Conference on Leorning Representations, 2021.
16

---

Published in Transactions on Machine Learning Research (03/2025)
A Appendix
A.1 List of Augmentations
The list of augmentations used for estimating the variance of the classification loss via TTA is reported in Figure
traneforn catoorrnneede_eeeeeeeeeeeeeeeeee
Outoutsktte:Tooessor (arge,
(arge, 1, 10),
OutoortAfterrooonnoor 1, 20),
v2. Randonniorizznttal/P1..)
v2 RendonVertiiclFFLip))
v2. Bandoonfottatiioo((ggrrre1000001
v2. RendonRotationn(45)
v2. RandonfRotatiom(00)
v2. color:Jsttrr
v2. SandomParsppcctivae)
Bandonntrrineeddegeeee-22) (ramslate=:0.1,, 0.3), (0.le=(0.5, 0.75)),
V2 RaddofferieeeCrrggllge...ppueu [1:], 80ale=(0.8, 1.0), ratio-(0.9, 1.1), antiallse-True),
v2. RandonInvert()
Figure 5: List of the augmentations applied to the images for computing the uncertainty via TTA.
A.2 Time Complexity Analysis
Table reports the average runtime per batch and the total runtime in seconds on CIEAR1O. The increase in runtime from CBR (class-balanced, random) to BI (clase-blaaced,, uncertainty-based) stemns from the requirement to generate TTA images but remains fast in absolute terms. fact, considering that in online- CL interested in updating the model every time mini-batch of new data arrives, the average time for processing it remains reasonable with less than second per batch. Since the batch size is the for all datasets, this estimate reflects the average runtime per batch for all the considered datasets.
Table 4: Average runtime per batch and total runtime (in seconds) on CIFAR10.
Runtime per batch
Total runtime
ER
0.22s
42s
CBR
0.31s
60s
BI (ours)
0.69s
132s
A.3 Memory Samples Analysis
In Figure 0 we analyse the composition and characteristics of the samples stored in the memory according to different strategies (random, top-k, and bottom-k). Figure 6a depicts the distribution of the confidence scores when samples are being stored in the memory. As one can see, although employing different strategies, the distributions look quite similar. Differently, Figure [b] the distributions of the BI-based uncortainty scores for the same samples differ significantly across the strategies. This shows the ability of BI to capture the epistemic uncertainty of the samples irrespective of the corresponding confidence score.
17

---

Published in Transactions on Machine Learning Research (03/2025)
80
Random
Top-k
Bottom-K
Prequency
60
40
20
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.6
0.8
1.0
Confidence score
(a)
Distributions of confidence scores.
60
Frequency
Random
Top-k
Bottom-k
40
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
2.0
0.0
0.5
1.0
1.5
2.0
Uncertainty value (BI)
(b) Distributions of uncertainty values (BI).
Figure 6: Distributions of confidence scores (a) and uncertainty values (b) for samples stored in the memory according to different strategies (random, top-k, and bottom-k).
A.4 Relevance of Online-CL CL for Al-aided Medicine
In our experiments, we include BloodCell as an example of a realistic online-CL scenario that reflects a common challenge in Al-aided medicine: (i) classifying colls from blood smears is an important component in diagnostic hematolooy and (ii) its imbalanced nature reflects the common setting where different disease subtypes (that require differential treatment) occur with vastly different frequencies (class imbalance); with common diseases being common, hospitals tend first see patients with the most common subtypes when collecting data for ML-model and over time also patients with less common discase subtypes will present in the clinic. If the model is trained in the online-CT. scenario it is not necessary to wait potentially a long time until sufficient patients of all subtypes have presented. Instead the model can be updated continually over time (online) whenever patients with a new subtype present. It is then crucial to mitigate CF e.g. via a memory based approach, to make sure all subtypes can be predicted well.
18

---

Published in Transactions on Machine Learning Research (03/2025)
A.5 Pseudocode of online- CL with BI
Algorithm Online-CL learning scheme
Input T: number of tasks, bs: batch size, D;: task dataset.
Notation O: model parametters, M: memory buller, &: current lask, oe current batch of task i m: memory sample (size equal to bs).
3: while training not completed do
4:
8' + GETNEXTBATCH( Di)
b Get one batch from
if = 0 then
0 TRAIN(6
else
SAMPLEFROMMENORY(M
Random sampling from memory
0 +- TRAIN(ÚS, m.)
Train with replay
10: end if
11: M + UPDATEMEMORY(M, 8*)
12: end while
13:
14: function UPDATEMEMORY B9)
15: for e b' do
For each label c in the current batch
16:
m. t MMy cl
b Extract samples with label from memory
17:
a - m U by c
b Get candidale samples for label e
18:
tr + "BI(TTA((a))
Compute uncertainty with Eq. (6) Update M based on ue
19: M + REPLACESAMPLESS (u.)
20: end for
21: return updated M
22: end function
19


Reviews:
[{"review_id": "ERWFpAukhL", "reviewer": "Reviewer_PuAH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths] \n1. The paper studies a critical challenge in online continual learning—catastrophic forgetting—by leveraging uncertainty estimation, which is both novel and applicable to dynamic real-world environments.\n\n2.  Applying BI-based uncertainty as a method for selecting representative samples is a theoretically grounded and practical approach.\n\n3. Experiments on imbalanced datasets replicate real-world scenarios, showcasing the practical utility of the proposed approach.\n\n[Weaknesses]  \n 1. This paper does not thoroughly address related works. For instance, it is crucial to evaluate the proposed method against Average Gradient Episodic Memory (A-GEM) [1]. Additionally, the study overlooks relevant work such as Bioslam [2], which also explores memory-replay-based continual learning. Bioslam incorporates an internal reward mechanism (Equation 12) to guide sample selection, which aligns with the concept of uncertainty estimation.\n\n2. The claim that \"least uncertain samples are the most effective at mitigating CF\" is not rigorously supported. For example, random sampling (ER) achieves competitive or better results in some cases (e.g., Table 2). This claim should be rephrased with appropriate conditions or clarified to emphasize relative effectiveness of least uncertain samples over most uncertain samples.\n\n3. The paper would benefit from analyzing the uncertainty spectrum of samples stored in the buffer. For instance, it should investigate whether most samples belong to low-uncertainty regions and compare the uncertainty distributions of ER (random sampling)  and the proposed method.\n\n4. While the proposed method involves additional computation for uncertainty estimation, the paper does not provide a comparison of time or memory complexity with simpler methods like ER.\n\n[1]  A. Chaudhry, et al. “Efficient lifelong learning with a-gem. In International Conference on Learning Representations,” 2018.  \n[2] Yin, Peng, et al. \"Bioslam: A bioinspired lifelong memory system for general place recognition.\" IEEE Transactions on Robotics (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the problem of catastrophic forgetting in online continual learning (CL) using uncertainty estimation. It evaluates several predictive uncertainty metrics for classification loss functions, with a focus on Bregman Information (BI). The authors propose that selecting the most representative samples—characterized by low uncertainty according to BI—is an effective strategy to reduce catastrophic forgetting. The findings are tested on the experiments of class-balanced and imbalanced datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Expand the discussion to include [1] (A-GEM) and [2] (Bioslam) and conduct experiments comparing the proposed method with these baselines.  \n2. Reassess the claim about the effectiveness of least uncertain samples in reducing CF and make it more precise.\n3. Include an analysis of the uncertainty spectrum of samples stored in the memory buffer and compare it to random sampling (ER) to provide deeper insights into the proposed approach.  \n4. Provide time and memory complexity analysis for the proposed method and compare it with ER.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b1wnDBJt41", "reviewer": "Reviewer_pj6v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe topic is highly relevant to the audience of TMLR. The paper evaluates several existing methods from the literature across diverse experimental setups and also introduces a new method. This work has the potential to provide valuable insights to the community.\n\n# Weakness\n\nOne of the biggest weaknesses of this paper is its lack of clarity throughout the text. Specifically, 1) it excessively uses technical terms without clear explanations, 2) presents vague arguments and reasoning about the literature, and 3) fails to clearly differentiate between existing methods and the new method proposed in the paper. Another major issue is that the proposed method is weak in both its reasoning and the results.\n\nWhile the clarity issues may seem minor individually, they accumulate and significantly impact the readability of the paper. I will highlight some of these issues in a general manner and provide a few examples, but I strongly urge the authors to carefully proofread and thoroughly revise the paper.\n\n## 1. Terminology Issues\n\nThe paper contains a large number of terminologies, which can be acceptable if handled carefully. However, some terms feel unnecessary, and the essential ones are not introduced clearly, creating unnecessary obstacles for readers.\n\nFor instance, a quick search for words like “-based,” “approach,” “strategy,” and “technique” reveals a large number of technical terms. Some terms, like “retrieval strategy” and “update strategy,” are only used once, while others, such as “rehearsal” and “replay,” appear to be used interchangeably. While it’s acceptable to use synonyms, the lack of proper definitions for these terms can make the text overwhelming and confusing.\n\nAnother issue is the introduction of terms without sufficient explanation, as if assuming the readers are already familiar with them. For instance, “predictive uncertainty” is used extensively without clarification—predictive uncertainty of what? Uncertainty could refer to several things, such as the uncertainty of samples in the training set, the uncertainty of potentially unseen samples within the valid domain, etc. Additionally, is the uncertainty being discussed only irreducible data uncertainty (aleatoric uncertainty), or does it also include the reducible model uncertainty (epistemic uncertainty)? These terminology issues are significant because terms can have different meanings across domains. Additionally, explanations should avoid introducing further technical jargon, as this can create layers of nested terms without proper clarification.\n\n## 2. Vague arguments and reasoning\n\nI want to start with a general comment about the concept of “predictive uncertainty.” The paper does not provide a convincing explanation of why the effect of predictive uncertainty estimates on management is worth exploring, nor does it justify the approach of using Bregman Information (BI) being reasonable. To clarify, I am not opposed to exploring this direction; my concern is that the paper does not explain or motivate it well enough.\n\nThe paper also attempts to connect predictive uncertainty to several aspects, including 1) whether data points are “sitting close to the boundary” or “sitting at the center” 2) whether they are representative or not, and 3) whether they are easy-to-forget or easy-to-remember. However, these connections are presented in a disjointed and unclear manner, making the reasoning feel vague and unstructured.\n\n## 3. Unclear differentiation between existing methods and the new method\n\nThe last paragraph of Section 2.2.1 does not successfully convey why using predictive uncertainty is an appealing approach. Additionally, the sentence “Different from their approaches, which …, our method exploits predictive uncertainty for the same purpose” is unclear in its implications. You dedicate several paragraphs to linking previous approaches to the idea of selecting samples based on uncertainty, but this sentence fails to clarify how your method differs from those approaches. Furthermore, your method, which is not described in detail at this point, does not appear to be presented as a more principled solution compared to the existing methods.\n\nSimilarly, in Section 4, you frequently claim that your approach differs from others but rarely describe how your method actually works. This lack of detail makes the distinctions between methods unclear. For example, you mention, “We introduce a class-balanced memory management … This is similar to Bang et al. (2021).” If they are similar, what exactly are the differences? Without clearer explanations, the novelty and uniqueness of your approach remain ambiguous.\n\n## 4. The proposed method is weak in its idea and result\n\nThe authors state: “The goal of this work is not to propose a new method that outperforms the SOTA, but rather to present practical insights from an uncertainty-aware perspective on the desirable characteristics samples should have to alleviate CF in …” I want to emphasize that it is perfectly acceptable for the proposed method not to outperform state-of-the-art (SOTA) methods. However, a method is worth introducing only if the idea behind it follows strong reasoning—such that, before conducting experiments, we could reasonably expect it to achieve meaningful outcomes and potentially perform well.\n\nIn this paper, it remains unclear 1) what exactly is meant by “predictive uncertainty,” 2) why this concept is worth exploring, 3) why the interpretation of “uncertainty” is framed in the way described, and 4) why the use of Bregman Information (BI) was chosen as the foundation for the proposed method.\n\nThe proposed method, Bregman Information, is only mentioned sparsely throughout the paper, with the only explanation being a brief paragraph in Section 4.2.3. While some reasoning is provided earlier, it is not sufficiently clear. For example, the discussion of epistemic and aleatoric uncertainty includes the statement: \"capture the aleatoric uncertainty which is inherent in the data, ..., the uncertainty about the data generating process (i.e., the epistemic uncertainty) is low.\" However, the \"uncertainty about the data in the data-generating process\" is precisely the aleatoric uncertainty inherent in the data. And hence, the reasoning that follows appears shaky.\n\n## Questions and More Comments\n\n### Section 1\n\nIn the statement, “Given the overlap between old and new information, the model tends to forget about the past knowledge to focus more on the newest tasks, …” I find the argument unclear.\n\nConsider an extreme case where we are in a single-task scenario, and the samples are independently and identically distributed (i.i.d.) from a data-generating process, with the data arriving sequentially in batches. In this situation, the “overlap of the old and new information” would be expected to be substantial. However, the reasoning provided regarding forgetting past knowledge and focusing on new tasks does not seem to align with this scenario as described. The connection between overlap and forgetting requires further explanation and justification.\n\n### Section 2\n\nI find the argument about “forgetting” rather hand-wavy. The reasoning would be more convincing if it focused on whether the samples are close to the decision boundaries or not. Ultimately, what matters is the model’s ability to retain relevant information from the samples it has seen, but at the end of the process, all the samples are discarded.\n\nIn Section 2.2.2, you attempt to define the “uncertainty” of each sample by measuring how much it stays or shifts into another class after introducing noise. However, you do not specify how the labels for these perturbed samples are assigned. If you rely on classic data augmentation methods, the perturbed samples would inherit the same labels as the original ones, making the uncertainty of every sample effectively zero. If this is not the case, are the labels assigned by another fixed predictor?\n\nYou claim that the contribution of Eq. (1) on reducing CF is unclear, and you argue by saying how the “online” data was presented differently. However, Eq. (1) says nothing about how data comes. Therefore, I don’t understand your argument.\n\n### Section 3\n\nThe notations are poorly defined and inconsistent. For example, n is used for the number of classes but later reused for a different purpose. Domains and codomains are not specified, b_t​ denotes a set while b_s​ represents a number, and terms like “class-balanced” and “high-frequency” are undefined. These issues make the paper hard to follow and need to be addressed.\n\n### Section 4\n\nBefore Section 4, you repeatedly emphasize the importance of sample uncertainty in memory population, yet here you introduce a parallel discussion on maintaining class balance in the memory. If this aspect is central to your method, it should be discussed in the related works, or you should expand the related works section to include relevant literature.\n\nFor memory sampling, the descriptions provided feel arbitrary and lack focus. You do not specify any methods requiring sampling beyond uniform or i.i.d. sampling. As it stands, this discussion is more distracting than informative.\n\nIn Section 4.2, the description of logits is insufficient. Please explain more on what they represent and what you intend to do with them. Additionally, you do not specify how labels for perturbed samples are assigned. The variable “c” in your formulas is also undefined, making the meaning of these equations unclear. Please make sure the consistency of equations with Eq. (1), as it appears some formulas may be incorrect. It is also unclear whether you deviated Eq. (1) in an earlier section.\n\n### Section 5\n\nIn Section 1, you mentioned there is a first part, where you will just use plain CIFAR-s and a second part, where you will use the -LT version of them. It’s better to make it clear also in the “Datasets” section.\n\nIn Section 5.1.2, introducing ER feels abrupt. Instead of “reservoir approach,” can you simply say data is randomly selected for memory and sampled randomly for replay? The mention of Dropout also lacks context and should be explained or removed.\n\nThe section needs thorough proofreading based on the earlier feedback.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies Online Continual Learning, with a focus on memory-based methods - approaches that maintain a fixed-size memory to enable replay such that the learned model is less likely to suffer from catastrophic forgetting. Such methods involve two key components: 1) memory population: decide which samples to include in the fixed-size memory 2) memory sampling: decide how to sample from the memory for the learning algorithm to replay.\n\nThe authors evaluate several existing memory population methods and propose a new method based on Bregman Information. The evaluation is conducted across several setups: 1) a more standard setup where CIFAR-10 and CIFAR-100 are divided into a sequence of tasks, with each task containing only a subset of classes, 2) an imbalanced setup using CIFAR-10 and CIFAR-100, but the number of samples for classes can be very different, and 3) a biomedical image dataset, BloodCell.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the section \"Weakness\" as well.\n\n## More on Figure 1\n\nFigure 1 is not properly explained. The labels are not defined until Section 4.2.2, and the meaning of “relative improvement” is not clarified. Additionally, the modification of a dataset to “LT” is introduced only in Section 5. Overall, this makes the placement of Figure 1 and the message it aims to convey very unclear.\n\n## More on Figure 2\n\nThe term “predicted score” is not defined anywhere in the paper, and the label “Conf” is not explained. Moreover, the choice to represent the upper bar as “low” and the lower bar as “high” seems counterintuitive.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XDNF4NcI4P", "reviewer": "Reviewer_3Hfb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The empirical study is systematic.\n- The relevant related work is surveyed and discussed in detail.\n- The presentation is somewhat hard to follow due to organizational issues and missing details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper conducts an empirical study of different memory population strategies used in online continual learning. The empirical study considers popular image classification benchmark (CIFAR100) as well as a medical image classification task.\n- These strategies revolve around different uncertainty metrics used for populating the replay memory. Bregman information is proposed as another uncertainty metric.\n- Analysis is conducted on the qualities of the samples that should be included in the memory to alleviate catastrophic forgetting efficiently.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The introduction is hard to follow. Please update.\n\t- Motivate why the data can be processed only once.\n\t- Motivate why there is only limited memory available for replay.\n\t- Include a high-level definition of predictive uncertainty.\n\t- The first paragraph of the introduction is almost a full page long. Break it up for improved readability.\n\t- Fix grammar in the sentence \"Furthermore, considering the quickly expanding landscape of recently rehearsal-based methods proposed, it is difficult to identify a clear strategy to exploit the memory at its best\"\n- Equation 1 comes before the \"Preliminaries and Notation\" section, yet it uses notation introduced in that section. Consider moving or removing the equation. If you keep it, please add the definitions of $c$, $\\hat{c}$, and $\\tilde{x}$ in the explanation for equation 1.\n- Please include details for what model is used for computing the predicted score and logits in figure 2.\n- There are many numbers in tables 1 and 2 but the discussion and analysis is lacking.\n\t- The error estimates in tables 1 and 2 are not explained. The bolding strategy is not explained. The step and bottom accuracies for BI seem to have error bars overlapping the mean. Some of the other methods, such as MS and LC also have error bars overlapping the mean of the bolded method.\n\t- What is the takeaway from the different ordering of the methods based on M?\n- The BloodCell dataset it claimed to be realistic, yet very little details are provided. Please include a short description how the online continual learning setting is relevant to the realism.", "broader_impact_concerns": "While the paper includes some results on real-world inspired data, most of the experiments are conducted on small scale problems. Therefore, it does not seem like this work is necessarily going to have immediate impact on real world tasks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the initial submission had some important flaws, especially around some technical language, it seems that those have mostly been addressed. Overall the work is an interesting scientific contribution. Even though the novelty and immediate impact of this work are not particularly high, the work is correct and belongs in TMLR.
Decision: Accept (poster)