Federated Continual Learning Goes Online: Uncertainty-Aware Memory Management for Vision Tasks and Beyond

## Abstract

Abstract Given the ability to model more realistic and dynamic problems, Federated Continual Learning (FCL) has been increasingly investigated recently. A well-known problem encountered in this setting is the so-called catastrophic forgetting , for which the learning model is inclined to focus on more recent tasks while forgetting the previously learned knowledge. The majority of the current approaches in FCL propose generative-based solutions to solve said problem. However, this setting requires multiple training epochs over the data, implying an offline setting where datasets are stored locally and remain unchanged over time. Furthermore, the proposed solutions are tailored for vision tasks solely. To overcome these limitations, we propose a new approach to deal with different modalities in the online scenario where new data arrive in streams of mini-batches that can only be processed once. To solve catastrophic forgetting, we propose an uncertainty-aware memory-based approach. Specifically, we suggest using an estimator based on the Bregman Information (BI) to compute the model’s variance at the sample level. Through measures of predictive uncertainty, we retrieve samples with specific characteristics, and – by retraining the model on such samples – we demonstrate the potential of this approach to reduce the forgetting effect in realistic settings while maintaining data confidentiality and competitive communication efficiency compared to state-of-the-art approaches.

## 1 Introduction

In recent years, federated learning (FL) (McMahan et al., ) has received increasing attention for its ability to allow local clients work towards the same objective without the need for sharing limited and sensitive information. However, despite the benefits provided by the privacy-preserving collaborative training, the assumptions of the standard scenario are far from realistic (Babakniya et al., ). For example, the standard static-single-task framework has very limited practical applicability in real-world cases where local clients often need to continuously learn new tasks. Let us consider the problem of classifying new COVID-19 variants as illustrative use-case. Given the evolving nature of the virus, new variants (i.e., new classes) arise over time. In this context, healthcare facilities which collaborate using FL would fail since the standard setting does not consider the dynamic increment of new classes. For this reason, a new paradigm was recently introduced to model more complex dynamics: federated continual learning (FCL) (Yoon et al., ; Dong et al., ). This new paradigm takes the global-local communication and privacy-preserving abilities enabled by FL and combines them with the Continual Learning (CL) ability of learning different tasks sequentially over time. However, FCL not only shares the characteristics at the intersection of FL and CL, but also inherits their respective challenges. In particular one of the most prominent problems in CL, the so-called catastrophic forgetting for which the model is prone to suffer from significant performance degradation on older tasks. In CL, this problem is addressed in different ways ranging from memory-based approaches (Yoon et al., ; Kumari et al., ; Hurtado et al., ) to generative methods (Shin et al., ; Lesort et al., ) or to regularization techniques (Lee et al., ; Aljundi et al., ; Chaudhry et al., ; He and Jaeger, ). In the context of FCL, instead, the majority of the proposed approaches exploit generative models at both local and global level to trace and encode the past information and generate synthetic instances which faithfully mimic previous history whilst preserving data privacy. A substantial drawback of such generative approaches is that they are tailored to image data and it is not clear how they could translate to other data modalities.
Importantly, such generator-based approaches also imply an offline setting where task data are collected before training and remain unchanged over time: local generators require large training datasets of complete tasks and similarly, more recently proposed generative models based on data distillation can only be trained at the end of each task (Babakniya et al., ).

In realistic conditions however, all data from the current task may not be available for collection at the same time but may rather arrive in small chunks sequentially. Furthermore, given the ubiquity of smart and edge devices with limited capabilities (e.g., wearable devices), there is a need for updating the learning model with the new incoming data to minimize memory overload and communication bandwidth (Ma et al., ). This problem of learning from an online stream of data is well investigated in CL while it remains largely unexplored in FCL.

Inspired by the problem of training models over online streams of data, we first formalise a new scenario to tackle the online problem in FCL (online-FCL). In this new scenario, we assume each client to learn from a stream of data where new data arrive in mini-batches which can only be processed once. As such, in line with the definition of online-CL, the model is updated with high frequency (Soutif-Cormerais et al., ). Then, taking inspiration from the most popular solutions in online-CL, we introduce memory buffers to alleviate CF at local level. More in detail, we propose an uncertainty-based memory management where data points are stored in local buffers according to their predictive uncertainty. Intuitively, predictive uncertainty provides a glimpse of the samples’ location in the decision space; samples with low uncertainty are the most representative ones for the respective class, while samples with high uncertainty represent data points that are close to the decision boundary and/or outliers. Thus, via estimates of predictive uncertainty, we can store in the memory samples with desired properties. Here, we propose to quantify uncertainty by directly estimating a generalized variance term from the loss function, which can be interpreted as a measure of epistemic uncertainty. To this end, we leverage a recently proposed bias-variance decomposition of the cross-entropy loss (Gruber and Buettner, ) and estimate the Bregman Information (BI) as the variance term in logit space.

The uncertainty-based memory management makes our approach flexible in terms of data modality. In fact, independent of the data modality (e.g., images, texts), our solution allows us to compute uncertainty estimates and thus populate the memory accordingly. This is in contrast to most of the current solutions in FCL which are limited to vision tasks and the offline setting.

In the last part of the paper, we demonstrate the ability of the proposed approach to reduce the forgetting effect in different scenarios. In the first part of the experiments, we evaluate our approach on CIFAR-10 and CIFAR-100 (Krizhevsky et al., ), two standard datasets used in this context. The goal is to understand how the proposed (epistemic) uncertainty estimate based on the Bregman Information performs in comparison with other standard estimates of overall model uncertainty under a memory-based regime. Then, departing from the common evaluation pipeline that would involve datasets like EMNIST (as in Qi et al. (); Wuerkaixi et al. ()) or larger datasets from the same naturalistic domain (e.g., ImageNet – as in Qi et al. (); Babakniya et al. ()), we validate our results on more probing real-world datasets from the medical domain. Finally, to showcase the ability of our approach to work with different data modalities, we test our findings on text classification tasks.

The contributions of this work can be summarized as follows:

- •
We propose and formalise a novel framework to tackle the FCL problem in the online setting which is largely overlooked in the literature.
- •
We highlight the limitations of current state-of-the-art generative-based solutions to work in the online setting and show empirically their inefficiency in the experimental section.
- •
We propose a memory-based solution that employs an alternative estimate for predictive uncertainty – which stems directly from a bias-variance decomposition of the cross-entropy loss for classification tasks – to populate the memory.
- •
We demonstrate the efficacy of our method in more realistic scenarios including datasets from different domains, with imbalance, and with different modalities.

Figure: Figure 1: Schematic overview of the proposed online-FCL scenario. Each client $\mathcal{C}_{k}$ receives their data as a stream of batches. Each batch $b_{k}^{t}$ is used for training and for updating the local memory $\mathcal{M}_{k}$. Once the current batch is processed, the client receives a new batch and continues their training without the possibility of retraining on previously seen batches.
Refer to caption: x1.png

## 2 Related Work

##### Continual Learning.

There are three types of incremental learning (IL) (van de Ven et al., ); task-, domain-, and class-IL. In task-IL, training and testing data include explicitly task IDs. In this unrealistic scenario, the learning algorithm knows which task needs to be completed. In the domain-IL setup, instead, the classification problem remains the same while the input distribution (or domain) shifts over time. For instance, an example of domain drift appears when a model for detecting even/odd digits in images learns first $1s$ and $2s$ and then $4s$ and $5s$. Lastly, class-IL scenarios represent the most realistic setting where the model continuously learns from an increasing number of classes. CL problems can also be classified based on whether the data can be stored and viewed multiple times (offline) or whether they can be processed once (online) (Mai et al., ).

##### Class-Incremental Learning.

Methods for class-IL can be divided into several categories (Mai et al., ) as follows; a) Regularization techniques alter the model parameter updates by adding penalty terms to the loss function (Lee et al., ; Aljundi et al., ), adjusting parameter gradients during optimization (Chaudhry et al., ; He and Jaeger, ), or employing knowledge distillation (Rannen et al., ; Wu et al., ); b) Memory-based techniques exploit a fixed-size buffer containing samples from past tasks for replay (Aljundi et al., ; Chaudhry et al., ) or regularization (Nguyen et al., ; Tao et al., ); c) Generative-based techniques involve training generative models that can produce pseudo-samples mimicking data from the past (Shin et al., ; Lesort et al., ); d) Parameter-isolation-based techniques assign different model parameters to each task. This can be done either by activating only the relevant parameters for each task (Fixed Architecture) (Mallya and Lazebnik, ; Serra et al., ) or by adding new parameters and keeping the existing ones unchanged (Dynamic Architecture) (Aljundi et al., ; Yoon et al., ). In online-CL, where data arrive in single mini-batches and the model is updated frequently, rehearsal-based methods are favoured over more complex solutions, like generative methods, because they offer greater flexibility and require less training data and computational time (Mai et al., ).

In the context of memory-based techniques, the main question is how to optimally manage the memory. In order to reduce catastrophic forgetting, samples in the memory should be representative of their own class, discriminative towards the other classes, and informative enough for the model to recall the information about the old classes. In the literature, many conflicting strategies can be found. Some of them suggest to use the most representative samples (Yoon et al., ; Hurtado et al., ), while others consider the samples near the decision boundary as the most useful to reduce CF (Kumari et al., ).

##### Federated Continual Learning.

The intersection of federated learning and continual learning has only been investigated very recently. One of the first papers in this direction is Yoon et al. (). The approach focuses on the less challenging task-IL scenario, where the task ID information is required during inference and testing. In Dong et al. (), the authors introduce the federated class-IL (FCIL) problem which tends to model more realistic scenarios. The clients have access to a large memory buffer and can share perturbed prototype samples with the global server, which differs with the standard FL setting where only the parameters of the local models are shared with the server (Babakniya et al., ). Ma et al. () uses knowledge distillation on both local and global levels via unlabeled surrogate datasets. Other works (Hendryx et al., ; Qi et al., ; Babakniya et al., ; Wuerkaixi et al., ) pose their attention on the FCIL problem without the use of memory replay data. Hendryx et al. () focus on few-shot learning and allows overlapping classes between tasks. Qi et al. () instead, introduces the constraint of non-overlapping classes for intra-client tasks. The work is based on generative replay where clients train a discriminator and a generator locally. At the same time, at each communication round, the server performs a consolidation step and generates synthetic data using the locally trained generators. Following a similar principle, Babakniya et al. () presents a generative model which is trained by the server in a data-free manner. Another FCIL scenario is considered in Shenaj et al. (), where tasks can arrive asynchronously at each client. The problem is tackled via a combination of prototype-based learning, representation loss, and stabilization of server aggregation.

Here we briefly summarise the identified limitations of such generator-based approaches. As anticipated before, they assume to work in the offline setting where static datasets are collected in advance and stored locally. This allows them to be trained over the same task dataset many times which is needed to reach convergence and to learn meaningful patterns in order to generate informative synthetic images. Finally, the storage of the whole task datasets and the generator models may be unfeasible in resource-limited devices.

## 3 Our Approach: O-FCL

### 3.1 Online-FCL: Problem Formulation

Following the notation proposed in Qi et al. (), we assume to have a set $\bm{\mathcal{C}}=\left\{\mathcal{C}_{1},\dots,\mathcal{C}_{n}\right\}$ of n different clients. Each client $\mathcal{C}_{k}$ keeps its private data $\bm{\mathcal{D}}_{k}=\{D_{k}^{1},\dots,D_{k}^{t}\}$ with its corresponding sequence of tasks $\bm{\mathcal{T}}_{k}=\{t_{k}^{1},\dots,t_{k}^{t}\}$. For each client $k$ and task $t$, we have an associated dataset $D_{k}^{t}=\{(x_{i},y_{i})\}_{i=1}^{n_{k}^{t}}$ with $x_{i}$ an input sample, $y_{i}$ the corresponding class label, and $n_{k}^{t}$ the number of training samples. In the proposed online setting, we assume that samples for each task $t$ come gradually in a stream of mini-batches $b_{k}^{t}=\{{(x_{i},y_{i})\in D_{k}^{t}\}_{i=1}^{bs}}$ which can only be seen once. Similar to Qi et al. (), we assume non-overlapping classes for intra-client tasks. In other words, the clients can see a specific class only once during the training.
At each communication round $r$, the client trains the local model on its own data and shares the local parameters $\bm{\theta}_{k}^{r}$ to update the global model parameters $\bm{\theta}_{g}^{r}$.
Since data points can be processed only once, in order to exploit the information from the incoming data points at its best, communications with the server are performed when a single mini-batch or multiple consecutive mini-batches are processed. This is considerably different from the standard FCL scenario, where multiple iterations and communication rounds are performed for the whole task dataset.
Following a popular trend in FCL, we also focus on the FCIL scenario which represents the most realistic and challenging case.

### 3.2 Methodology

As briefly anticipated above, the online nature of the newly introduced problem poses new challenges compared to the standard case. For this, we propose a memory-based approach on the client-side to alleviate catastrophic forgetting. The motivation is threefold: 1) the most effective solutions in online-CL are approaches with memory buffers; 2) given the online nature of the problem, generative-based approaches would be limited as they require many iterations over the same dataset; 3) compared with generative-based solutions, we only store a small amount of data reducing the overall overhead on the local device.

In the following subsections, we describe in more detail the characteristics of our approach at the client and server levels. For the client side, we outline our uncertainty-aware memory management detailing the properties of the employed uncertainty estimate and its advantages compared to standard scores. For the server side, we detail the adjustments done to improve the communication and parameter averaging effectiveness in the online scenario. A summary of the complete training procedure is reported in Algorithm 1 (see Appendix A.9).

#### 3.2.1 Client-level

##### Memory management.

For each client $k$, we introduce a fixed-size memory buffer $\mathcal{M}_{k}$. Similar to Chrysakis and Moens (), the memory population strategy is based on a class-balanced update, which is crucial to consider in case the stream of data is highly imbalanced. In fact, if classes are not equally represented in the memory, sampling from the memory may further deteriorate the predictive performance of the framework for under-represented classes. Differently from the populating strategy proposed in Chrysakis and Moens (), the criteria to decide which samples to keep in the memory is not random, but based on predictive uncertainty estimates in order to intentionally store samples with desired characteristics for each class. There are different ways to select the samples. We can decide to store a) the class-representative data points by selecting the least uncertain ones for each class (bottom-k); b) the easiest-to-forget samples by sampling the ones with high uncertainty (top-k).

From the second task on, we need a strategy for sampling from the memory a subset of data points used for replay (replay set). Since the memory represents a prototypical set of data points for each class, we assume that a random sampling is sufficient to extract informative data points from the memory. Following the standard practice, the number of samples in the replay set is equal to the batch size. It is important to note that the memory may already contain samples from the current task. In such cases, samples from the current task are excluded from the sampling process to ensure that the focus remains solely on those belonging to past tasks.

Figure: Figure 2: Illustration of the different aspects of uncertainty captured by model confidence scores and Bregman Information (BI). Close to the decision boundary we can notice how, due to the high aleatoric uncertainty, data points have low-confidence scores. Contrarily, due to high density of observed data, there is a low uncertainty about the data generating process resulting in a low BI.
Refer to caption: x2.png

##### Predictive uncertainty estimation in the logit space.

Different measure of predictive uncertainty can capture distinct aspects of a model’s irreducible aleatoric uncertainty (inherent in the data) and its epistemic uncertainty (that stems from the finite training data and can be reduced by gathering more data).
In online CL, the most commonly used measures are derived directly from the confidence scores of the model and mostly capture the irreducible aleatoric uncertainty (Wimmer et al., ). Here, we hypothesize it may be more beneficial for the model to replay instances which are representative in the sense that there is low uncertainty about the data generating process (we refer to this as low epistemic uncertainty, while acknowledging that there exist varying definitions). To not rely on specialized Bayesian models, we leverage a recently proposed bias-variance decomposition of the cross-entropy loss and compute a generalized variance term directly from the loss (Gruber and Buettner, ). Such bias-variance decomposition decomposes the expected prediction error (loss) into the model’s bias, variance, and an irreducible error (noise term). The latter is related to aleatoric uncertainty, whereas the variance term can directly be related to epistemic uncertainty (Gruber et al., ). This is the first attempt of using this decomposition for memory management in FCL contexts.

Gruber and Buettner () have recently shown that a bias-variance decomposition of cross-entropy loss gives rise to the Bregman Information as the variance term and measures the variability of the prediction in the logit space. We illustrate the different aspects of uncertainty captured by confidence scores and BI respectively in Figure 2. For example, data points close to the decision boundary have a low confidence score due to the inherently high aleatoric uncertainty; in contrast, due to the high density of observed data, there is actually a low uncertainty about the data generating process (DGP), resulting in a low BI (low epistemic uncertainty). Outliers far away from the decision boundary can have a high confidence score, but will have a high BI due to the high uncertainty regarding the DGP. We hypothesize that it is samples with a low BI that are most useful for replay in online-FCL.
To populate the memory with representative samples, we therefore propose using an uncertainty estimator based on Bregman Information (BI) (Gruber and Buettner, ). The authors demonstrate that BI at the sample level can be estimated through deep ensembles or test-time augmentation (TTA). However, to reduce the computational overhead at the local level, we employ TTA for computing the estimations. Let us consider the problem of multi-class classification (as in our case), where the standard loss is represented by the cross-entropy. Considering a set $P$ of perturbations for a given data point $x$, we can compute the variance term of the classification loss $u(x)_{BI}$ as follows:

$$ $u(x)_{BI}=\frac{1}{P}\sum_{i=1}^{P}\text{LSE}(\hat{z}_{i})-\text{LSE}\left( \frac{1}{P}\sum_{i=1}^{p}\hat{z}_{i}\right),$ (1) $$

where $\hat{z}_{i}\in\mathbb{R}^{c}$ represent the logit predictions and $\text{LSE}(x_{1},\dots,x_{n})=\ln\sum_{i=1}^{n}e^{x_{i}}$ the LogSumExp (LSE) function respectively.
Intuitively, a large value of $u(x)_{BI}$ means that the logits predicted across the perturbations vary significantly, suggesting a high uncertainty of the prediction and the DGP at this point of the input space. The use of this estimator is motivated by the fact that, in comparison with other uncertainty scores such as entropy, smallest margin, or least confidence, there is no information loss in the estimation step. In fact, if we inspect these alternative metrics (see Appendix A.1 for the equations), we can notice that they either need a normalization step to move the logits in the probability space or rest on the largest activation value only. Furthermore, as reported in Gruber and Buettner (); Ovadia et al. (), and Tomani and Buettner (), common confidence scores are reliable only in case of well-calibrated models.
In contrast, the BI-based estimation of the epistemic uncertainty is meaningful also under distribution shift and able to identify robust and representative samples (Gruber and Buettner, ).

#### 3.2.2 Global-level

##### Communication rounds.

Following the standard assumptions of the federated scenario, we allow local clients to share the model parameters solely. However, the proposed online framework poses additional challenges compared to the standard case. In the standard scenario, the communication round is performed at the end of several iterations over the same task dataset. This implies that the model has probably reached convergence and can effectively share the learned information during the communication round. In our case, a few new samples are available at each step and, to keep the model up-to-date, the communication round cannot be performed at the end of the task only. For this reason, as anticipated in Section 3.1, communications with the server are performed when a single mini-batch or multiple consecutive mini-batches are processed. Given the instability of the model when a new task starts, to ensure an effective and efficient information sharing, we propose to set a burn-in period. During this period, the local model learns independently for a certain number of batches without sharing and receiving information from the others. This guarantees that, when the local client starts to be involved in the communication rounds, the model has effectively learned relevant information on the current task. Additionally, since every parameter update degrades the predictive performance at local level (Qi et al., ), we propose to limit the number of communications rounds per task. Instead of performing a model averaging every time a batch is processed, we let the local models learn for $q$ consecutive mini-batches before actively participating to the communication round.

##### Parameter averaging.

Although clients work on the same task, they may receive data coming from different classes where some of them are over-represented compared to others. In this case, the local parameters collected for parameter averaging can be biased towards the most common classes. For this, we suggest to first create an aggregated model for each class available in the current round, and then compute $\bm{\theta}_{g}^{r}$ using the class-based models just created. This way, the current classes contribute equally during the updates of the global parameters. In case all the clients receive data containing classes equally distributed, the parameter averaging results in the standard computation, like e.g., FedAvg (McMahan et al., ). Importantly, if sharing the class information is not possible, we can rely on standard averaging strategies (e.g., FedAvg or FedProx (Li et al., )) without hampering the performance since our methodology is flexible on this matter (see Table 8). Finally, once the new global parameters $\bm{\theta}_{g}^{r}$ are computed, to avoid a drastic change between the old parameters and the new ones, following Shenaj et al. (), we propose to average the newly computed parameters at round $r$ with the previous parameters computed at round $r-1$.

## 4 Experiments

### 4.1 Datasets and Settings

##### Datasets.

To understand the behaviour of different algorithms in the online-FCL scenario, we use two datasets commonly used in the literature, namely, CIFAR10 and CIFAR100 (Krizhevsky et al., ). We randomly assign a set of classes to 5 tasks and we split the task data evenly among the clients such that the task sequence $T_{k}^{t}$ for task $t$ and client $k$ does not share any data with the other clients. In this way, since the class assignment per task is different every time, we can identify the strategy providing greater flexibility and effectiveness irrespective of the composition of the tasks. Then, as anticipated in the introductory part, we assess the performance under more difficult and realistic conditions. Instead of using datasets that, to some extent, share similar characteristics with CIFAR (i.e., ImageNet) or represent unrealistic tasks (such as EMNIST), we validate our results on datasets for biomedical image analysis. Apart from the change in the domain (which in turn means different backgrounds in the images, different statistics, etc.), these datasets pose an additional challenge compared to the standard benchmark datasets, i.e., data imbalance. To reflect realistic conditions where recent tasks contain fewer data points than the older ones since the time to collect them is shorter, we assign classes to tasks based on the class size. We decide to focus on two biomedical datasets annotated by expert clinical pathologists; the colorectal cancer histology (CRC-Tissue) dataset (Kather et al., ; Yang et al., ) containing images divided in 8 classes of hematoxylin–eosin (HE)–stained slides taken from patients with colorectal cancer (CRC), and the kidney cortex cells (KC-Cell) dataset containing 8 classes of human kidney cortex tissue sections (Ljosa et al., ; Yang et al., ). Finally, to evaluate our approach on text classification tasks, we employ three different datasets: the 20NewsGroups dataset (Lang, ), DBPedia (Zhang et al., ), and Yahoo Answers (Zhang et al., ). As for vision tasks, we randomly assign classes to tasks every run. Statistics and more details about the used datasets can be found in the supplementary material (Appendix A.3).

##### Experimental settings.

Given the novelty of the online setting, there are no direct competitors which we could refer to. We therefore investigate the most successful algorithms from the online-CL and the FCL literature. We compare our approach with standard baselines for FL, i.e., FedAvg (McMahan et al., ) and FedProx (Li et al., ), a standard memory-based approach for online-CL, namely Experience Replay (ER) (Chaudhry et al., ), and two state-of-the-art approaches for FCL with generative replay, namely MFCL (Babakniya et al., ) and FedCIL (Qi et al., ). Our decision to use ER is based on the fact that, despite its simplicity, it is surprisingly competitive when compared with more sophisticated and newer approaches as shown in recently conducted empirical surveys (Soutif-Cormerais et al., ). For FCL, we are particularly interested in MFCL because its data-free solution trains the generator on the server side. As such, we believe it may be potentially feasible in the online scenario, in contrast to other approaches such as FedCIL that train local generators at the client-side (Qi et al., ). In addition to the standard ER, we also include the class-balanced version (CBR) presented in Chrysakis and Moens (). Finally, to show the competitiveness of $u(x)_{BI}$, we also consider other uncertainty scores. In particular, least confidence (LC), margin sampling (MS), ratio of confidence (RC), and entropy (EN) (Shannon, ; Campbell et al., ; Culotta and McCallum, ).

In all the experiments for image classification, we employ a slim version of Resnet18 (He et al., ) – as done in previous works in online-CL (Kumari et al., ; Hurtado et al., ; Soutif-Cormerais et al., ) –, and use the SGD optimizer with a learning rate of $0.1$. For text classification, we employ a simple MLP with a single fully connected hidden layer (512 units) and Adam optimizer with a learning rate of $0.01$. To generate the perturbed inputs for estimating predictive uncertainty, we use two different strategies according to the different data modalities. For vision tasks, the perturbations are generated via standard augmentation. The list of augmentations used in our experiments is provided in Appendix A.2. For the natural language experiments, we leverage recent progress in foundation models and first create general-purpose latent representations of the input texts via a pre-trained sentence embedder. In particular, we employ e5-small-v2 (Wang et al., ) (384 dimensions) via HuggingFace (Wolf et al., ). Then, for perturbing the generated vector representations, we add Gaussian noise from $\mathcal{N}(0,0.1)$ to each latent dimension.

Following standard practice, we set the batch size equal to $10$. The memory size was set to different values for each dataset in order to examine the performance of various memory configurations, including both large and small buffers. The burn-in period is set to $30$. A communication round with the server is performed after $q=5$ mini-batches. The number of clients is set to $5$. For parameter averaging, we employ FedAvg (McMahan et al., ). Given the flexibility of the approach, as mentioned before, the model averaging strategy can be changed as desired. For evaluation, following recent works (Yoon et al., ; Wuerkaixi et al., ), we use the average last accuracy (A) and average forgetting (F) (see appendix A.4 for the definitions). All the experiments are run on three different random seeds. For each dataset, experiments were run on a Linux machine using a single Quadro RTX 5000 and 16 GB RAM.

Figure: Figure 3: (left) Average accuracy per task in comparison with FL (FedAvg) and generative-based FCL baselines (MFCL); (right) Relative CF improvement over the CL baseline (ER).
Refer to caption: x3.png

**Table 1: Comparison of average last accuracy (A) and last forgetting (F) on CIFAR10 (5 tasks).**
| Score |  | M=200 | M=500 | M=1000 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ER |  | 20.74 $\pm$ 2.53 | 27.90 $\pm$ 2.03 | 33.64 $\pm$ 0.72 |  |  |  |
|  | 48.48 $\pm$ 3.95 | 30.18 $\pm$ 2.69 | 24.3 $\pm$ 1.73 |  |  |  |  |
| CBR |  | 22.82 $\pm$ 1.55 | 24.17 $\pm$ 3.23 | 32.67 $\pm$ 1.84 |  |  |  |
|  | 46.79 $\pm$ 2.92 | 28.54 $\pm$ 2.22 | 23.62 $\pm$ 2.63 |  |  |  |  |
|  |  | Top | Bottom | Top | Bottom | Top | Bottom |
| LC | A$(\uparrow)$ | 21.58 $\pm$ 1.56 | 20.92 $\pm$ 0.70 | 29.49 $\pm$ 1.46 | 27.38 $\pm$ 1.68 | 36.14 $\pm$ 2.44 | 31.94 $\pm$ 2.44 |
| F$(\downarrow)$ | 50.78 $\pm$ 1.96 | 49.78 $\pm$ 3.16 | 36.13 $\pm$ 2.54 | 30.17 $\pm$ 2.52 | 21.97 $\pm$ 4.06 | 29.53 $\pm$ 4.46 |  |
| MS | A$(\uparrow)$ | 22.34 $\pm$ 1.04 | 21.88 $\pm$ 1.28 | 29.72 $\pm$ 1.98 | 28.96 $\pm$ 1.54 | 34.42 $\pm$ 0.91 | 32.28 $\pm$ 1.20 |
| F$(\downarrow)$ | 49.38 $\pm$ 3.09 | 47.20 $\pm$ 4.97 | 35.16 $\pm$ 3.20 | 28.07 $\pm$ 2.71 | 29.58 $\pm$ 2.73 | 28.64 $\pm$ 4.34 |  |
| RC | A$(\uparrow)$ | 21.80 $\pm$ 1.65 | 22.75 $\pm$ 1.84 | 28.36 $\pm$ 2.07 | 31.17 $\pm$ 1.75 | 35.08 $\pm$ 1.11 | 28.98 $\pm$ 2.26 |
| F$(\downarrow)$ | 56.91 $\pm$ 1.90 | 45.05 $\pm$ 2.11 | 36.47 $\pm$ 2.84 | 31.86 $\pm$ 4.16 | 26.10 $\pm$ 1.76 | 28.39 $\pm$ 2.95 |  |
| EN | A$(\uparrow)$ | 21.15 $\pm$ 1.37 | 20.14 $\pm$ 1.41 | 28.63 $\pm$ 0.86 | 26.48 $\pm$ 1.59 | 36.25 $\pm$ 1.22 | 26.42 $\pm$ 1.49 |
| F$(\downarrow)$ | 53.30 $\pm$ 2.97 | 55.58 $\pm$ 3.49 | 36.70 $\pm$ 2.21 | 35.85 $\pm$ 3.17 | 25.52 $\pm$ 3.01 | 34.64 $\pm$ 3.40 |  |
| BI | A$(\uparrow)$ | 21.31 $\pm$ 1.46 | 24.89 $\pm$ 0.83 | 26.57 $\pm$ 0.83 | 27.84 $\pm$ 2.31 | 35.12 $\pm$ 2.51 | 35.83 $\pm$ 2.60 |
| F$(\downarrow)$ | 54.65 $\pm$ 1.83 | 35.77 $\pm$ 4.13 | 42.86 $\pm$ 3.81 | 24.59 $\pm$ 3.16 | 27.00 $\pm$ 2.99 | 19.07 $\pm$ 2.17 |  |

### 4.2 Empirical Results

From the experiments conducted on the CIFAR datasets, we can observe that the proposed approach is able to reduce CF consistently across different memory sizes and class-per-task assignments. The results in Tables 1 and 2 suggest that storing the class-representative samples (i.e., the least uncertain data points for each class) provides a benefit in terms of predictive performance gain and forgetting reduction. Our findings are substantiated in the classification tasks for biomedical images. Although the images originate from another domain, Table 3 confirms that in terms of CF, BI outperforms the considered memory-based baselines in most of the cases. Crucially, we also show the ability of our approach to work with imbalanced real-world data. Table 5 and Figure 3 summarise our findings; in comparison with both memory-based and generative-based methods, our simple-yet-elegant approach is able to perform consistently across tasks (left plot) and reduce CF on different datasets (right plot). Finally, we demonstrate that our method outperforms baselines also on non-vision data. Table 4 reports the results on the textual classification datasets. Here we find that the difference in terms of predictive accuracy is less marked. This is because the baselines have higher accuracy on the last tasks and poor performance on the first ones, while in our case the accuracy is kept more uniform across the tasks (reflected in a lower average forgetting) as shown in the left plot in Figure 3. Note that the use of generative baselines is restricted to image data only.

**Table 2: Comparison of last accuracy (A) and last forgetting (F) on CIFAR100 (5 tasks).**
| Score |  | M=1000 | M=2000 |  |  |
| --- | --- | --- | --- | --- | --- |
| ER |  | 10.33 $\pm$ 0.58 | 10.95 $\pm$ 0.4 |  |  |
|  | 6.89 $\pm$ 1.53 | 9.12 $\pm$ 0.96 |  |  |  |
| CBR |  | 13.38 $\pm$ 0.58 | 11.41 $\pm$ 0.36 |  |  |
|  | 31.66 $\pm$ 0.30 | 9.37 $\pm$ 1.56 |  |  |  |
|  |  | Top | Bottom | Top | Bottom |
| LC | A$(\uparrow)$ | 12.71 $\pm$ 0.96 | 13.43 $\pm$ 1.00 | 13.50 $\pm$ 0.81 | 14.09 $\pm$ 0.47 |
| F$(\downarrow)$ | 8.03 $\pm$ 1.88 | 6.95 $\pm$ 0.91 | 7.99 $\pm$ 1.34 | 7.75 $\pm$ 1.87 |  |
| MS | A$(\uparrow)$ | 13.34 $\pm$ 1.71 | 13.77 $\pm$ 1.31 | 13.83 $\pm$ 0.56 | 14.26 $\pm$ 0.72 |
| F$(\downarrow)$ | 7.48 $\pm$ 2.04 | 6.84 $\pm$ 1.60 | 8.04 $\pm$ 0.99 | 8.01 $\pm$ 1.76 |  |
| RC | A$(\uparrow)$ | 13.30 $\pm$ 1.47 | 13.76 $\pm$ 1.24 | 13.28 $\pm$ 1.23 | 13.83 $\pm$ 0.69 |
| F$(\downarrow)$ | 8.20 $\pm$ 1.91 | 6.97 $\pm$ 1.12 | 7.52 $\pm$ 1.70 | 7.70 $\pm$ 1.58 |  |
| EN | A$(\uparrow)$ | 12.77 $\pm$ 1.00 | 12.77 $\pm$ 0.71 | 13.81 $\pm$ 1.00 | 13.93 $\pm$ 0.76 |
| F$(\downarrow)$ | 8.11 $\pm$ 1.14 | 7.90 $\pm$ 1.20 | 8.10 $\pm$ 1.51 | 7.50 $\pm$ 1.17 |  |
| BI | A$(\uparrow)$ | 12.89 $\pm$ 0.97 | 14.04 $\pm$ 0.62 | 13.72 $\pm$ 0.70 | 14.31 $\pm$ 0.76 |
| F$(\downarrow)$ | 7.54 $\pm$ 1.43 | 6.73 $\pm$ 1.76 | 8.06 $\pm$ 1.49 | 6.96 $\pm$ 1.40 |  |

**Table 3: Comparison of average last accuracy (A) and last forgetting (F) on (left) CRC-Tissue (4 tasks), and (right) KC-Cell (4 tasks).**
| CRC-Tissue |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Score |  | M=80 | M=120 |  |  |
| ER |  | 49.79 $\pm$ 1.93 | 50.68 $\pm$ 2.69 |  |  |
|  | 20.31 $\pm$ 8.76 | 23.96 $\pm$ 4.78 |  |  |  |
| CBR |  | 46.70 $\pm$ 1.34 | 49.38 $\pm$ 1.01 |  |  |
|  | 33.69 $\pm$ 3.16 | 19.19 $\pm$ 6.86 |  |  |  |
|  |  | Top | Bottom | Top | Bottom |
| LC | A$(\uparrow)$ | 47.66 $\pm$ 2.52 | 56.02 $\pm$ 4.89 | 56.49 $\pm$ 2.21 | 58.12 $\pm$ 2.94 |
| F$(\downarrow)$ | 41.42 $\pm$ 3.64 | 13.25 $\pm$ 4.91 | 26.65 $\pm$ 6.30 | 11.13 $\pm$ 3.12 |  |
| MS | A$(\uparrow)$ | 47.87 $\pm$ 1.95 | 54.92 $\pm$ 2.93 | 52.82 $\pm$ 3.51 | 55.64 $\pm$ 4.42 |
| F$(\downarrow)$ | 36.96 $\pm$ 4.74 | 17.24 $\pm$ 8.46 | 30.42 $\pm$ 4.05 | 9.59 $\pm$ 6.93 |  |
| RC | A$(\uparrow)$ | 50.42 $\pm$ 3.92 | 57.50 $\pm$ 3.01 | 55.93 $\pm$ 5.85 | 58.81 $\pm$ 3.34 |
| F$(\downarrow)$ | 32.95 $\pm$ 4.04 | 12.91 $\pm$ 5.00 | 26.57 $\pm$ 6.35 | 8.71 $\pm$ 5.86 |  |
| EN | A$(\uparrow)$ | 49.18 $\pm$ 2.36 | 55.22 $\pm$ 4.05 | 53.80 $\pm$ 4.56 | 58.48 $\pm$ 3.06 |
| F$(\downarrow)$ | 29.55 $\pm$ 6.39 | 14.95 $\pm$ 8.85 | 29.48 $\pm$ 12.65 | 8.16 $\pm$ 3.79 |  |
| BI | A$(\uparrow)$ | 49.18 $\pm$ 2.61 | 57.05 $\pm$ 3.65 | 50.16 $\pm$ 2.79 | 62.33 $\pm$ 1.93 |
| F$(\downarrow)$ | 36.01 $\pm$ 7.81 | 13.63 $\pm$ 8.04 | 37.28 $\pm$ 9.50 | 7.99 $\pm$ 4.08 |  |

**Table 4: Comparison of average last accuracy (A) and last forgetting (F) on text classification tasks.**
| Dataset |  |  | ER | CBR | LC | MS | RC | EN | BI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 20NewsGroup | M=60 | A$(\uparrow)$ | 42.33 $\pm$ 1.66 | 45.21 $\pm$ 1.86 | 42.90 $\pm$ 1.23 | 44.14 $\pm$ 1.63 | 44.15 $\pm$ 1.32 | 41.65 $\pm$ 1.82 | 44.92 $\pm$ 1.63 |
| F$(\downarrow)$ | 31.12 $\pm$ 2.18 | 30.39 $\pm$ 0.80 | 32.05 $\pm$ 2.27 | 31.67 $\pm$ 2.57 | 31.61 $\pm$ 1.77 | 33.77 $\pm$ 1.43 | 29.98 $\pm$ 1.37 |  |  |
| M=500 | A$(\uparrow)$ | 45.95 $\pm$ 1.76 | 46.22 $\pm$ 1.55 | 45.72 $\pm$ 2.28 | 46.46 $\pm$ 2.53 | 46.58 $\pm$ 2.83 | 44.92 $\pm$ 2.71 | 46.72 $\pm$ 2.11 |  |
| F$(\downarrow)$ | 27.43 $\pm$ 1.12 | 28.00 $\pm$ 0.98 | 28.58 $\pm$ 1.61 | 27.97 $\pm$ 2.36 | 28.02 $\pm$ 2.53 | 29.63 $\pm$ 2.45 | 26.97 $\pm$ 2.36 |  |  |
| DBPedia | M=60 | A$(\uparrow)$ | 75.99 $\pm$ 2.10 | 76.24 $\pm$ 1.21 | 74.73 $\pm$ 2.04 | 75.21 $\pm$ 1.92 | 75.36 $\pm$ 1.61 | 74.58 $\pm$ 1.88 | 77.86 $\pm$ 1.95 |
| F$(\downarrow)$ | 24.73 $\pm$ 2.70 | 23.06 $\pm$ 1.58 | 27.06 $\pm$ 2.30 | 26.44 $\pm$ 2.28 | 26.52 $\pm$ 1.77 | 27.14 $\pm$ 2.15 | 23.06 $\pm$ 2.10 |  |  |
| M=100 | A$(\uparrow)$ | 76.78 $\pm$ 1.04 | 79.78 $\pm$ 0.83 | 76.21 $\pm$ 1.58 | 76.18 $\pm$ 1.70 | 76.18 $\pm$ 1.50 | 76.33 $\pm$ 1.45 | 78.68 $\pm$ 1.08 |  |
| F$(\downarrow)$ | 24.14 $\pm$ 1.18 | 21.45 $\pm$ 1.49 | 25.15 $\pm$ 1.60 | 25.19 $\pm$ 1.83 | 25.29 $\pm$ 1.78 | 24.87 $\pm$ 1.83 | 21.79 $\pm$ 1.41 |  |  |
| Yahoo Answers | M=500 | A$(\uparrow)$ | 45.80 $\pm$ 1.77 | 47.45 $\pm$ 2.86 | 47.87 $\pm$ 3.38 | 47.60 $\pm$ 3.38 | 48.25 $\pm$ 2.61 | 49.32 $\pm$ 3.61 | 50.13 $\pm$ 1.69 |
| F$(\downarrow)$ | 40.15 $\pm$ 2.91 | 36.38 $\pm$ 4.29 | 36.02 $\pm$ 6.81 | 35.69 $\pm$ 4.61 | 34.60 $\pm$ 3.49 | 33.67 $\pm$ 5.12 | 32.65 $\pm$ 3.21 |  |  |
| M=1000 | A$(\uparrow)$ | 45.48 $\pm$ 2.11 | 44.62 $\pm$ 3.23 | 46.40 $\pm$ 3.07 | 47.00 $\pm$ 2.75 | 46.33 $\pm$ 3.68 | 46.70 $\pm$ 1.11 | 47.03 $\pm$ 3.36 |  |
| F$(\downarrow)$ | 39.40 $\pm$ 3.47 | 39.17 $\pm$ 4.16 | 37.15 $\pm$ 5.78 | 37.15 $\pm$ 4.35 | 38.08 $\pm$ 5.26 | 37.85 $\pm$ 1.78 | 37.25 $\pm$ 4.17 |  |  |

**Table 5: Comparison of average last accuracy (A) and last forgetting (F) on vision and textual tasks for standard FL, generative-based FCL, and our method.**
| Method |  | CIFAR10 | CIFAR100 | CRC-Tis. | KC-Cell | 20News | DBPedia | Yahoo |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FedAvg | A$(\uparrow)$ | 16.90 $\pm$ 0.55 | 3.57 $\pm$ 0.33 | 22.26 $\pm$ 2.29 | 15.04 $\pm$ 0.83 | 6.34 $\pm$ 0.58 | 19.57 $\pm$ 0.30 | 14.18 $\pm$ 1.59 |
| F$(\downarrow)$ | 78.29 $\pm$ 2.60 | 19.73 $\pm$ 0.99 | 87.39 $\pm$ 4.06 | 74.08 $\pm$ 3.70 | 58.84 $\pm$ 1.91 | 99.02 $\pm$ 0.32 | 70.90 $\pm$ 2.88 |  |
| Weighted FedAvg | A$(\uparrow)$ | 16.51 $\pm$ 0.77 | 4.22 $\pm$ 0.20 | 22.00 $\pm$ 2.40 | 14.85 $\pm$ 0.92 | 6.36 $\pm$ 0.56 | 19.23 $\pm$ 0.44 | 12.52 $\pm$ 1.60 |
| F$(\downarrow)$ | 70.08 $\pm$ 1.57 | 20.18 $\pm$ 0.23 | 80.40 $\pm$ 5.15 | 76.21 $\pm$ 2.72 | 50.22 $\pm$ 2.41 | 97.64 $\pm$ 1.25 | 69.00 $\pm$ 2.11 |  |
| FedProx | A$(\uparrow)$ | 16.57 $\pm$ 0.50 | 3.73 $\pm$ 0.15 | 22.12 $\pm$ 1.26 | 14.79 $\pm$ 1.01 | 12.25 $\pm$ 0.40 | 19.58 $\pm$ 0.26 | 17.35$\pm$ 1.34 |
| F$(\downarrow)$ | 76.74 $\pm$ 1.12 | 20.26 $\pm$ 0.58 | 87.86 $\pm$ 4.78 | 73.38 $\pm$ 3.66 | 67.31 $\pm$ 3.14 | 99.06 $\pm$ 0.32 | 82.54 $\pm$ 0.55 |  |
| MFCL | A$(\uparrow)$ | 16.42 $\pm$ 0.32 | 4.60 $\pm$ 0.09 | 27.62 $\pm$ 7.99 | 16.32 $\pm$ 1.21 | ✗ | ✗ | ✗ |
| F$(\downarrow)$ | 55.64 $\pm$ 3.03 | 13.69 $\pm$ 0.33 | 76.34 $\pm$ 12.68 | 70.88 $\pm$ 1.94 |  |  |  |  |
| FedCIL | A$(\uparrow)$ | 16.26 $\pm$ 1.16 | 2.45 $\pm$ 0.28 | 22.44 $\pm$ 1.23 | 13.14 $\pm$ 1.31 | ✗ | ✗ | ✗ |
| F$(\downarrow)$ | 49.87 $\pm$ 0.61 | 4.99 $\pm$ 0.81 | 62.47 $\pm$ 1.35 | 55.82$\pm$ 1.60 |  |  |  |  |
| BI (best) | A$(\uparrow)$ | 35.83 $\pm$ 2.60 | 14.31 $\pm$ 0.76 | 62.33 $\pm$ 1.93 | 21.61 $\pm$ 1.23 | 46.72 $\pm$ 2.11 | 78.68 $\pm$ 1.08 | 50.13 $\pm$ 1.69 |
| F$(\downarrow)$ | 19.07 $\pm$ 2.17 | 6.96 $\pm$ 1.40 | 7.99 $\pm$ 4.08 | 59.05 $\pm$ 4.42 | 26.97 $\pm$ 2.36 | 21.79 $\pm$ 1.41 | 32.65 $\pm$ 3.21 |  |

## 5 Discussion and Conclusion

The novelty of the online scenario for FCL tasks raises questions on different aspects of the proposed approach and we next provide an in-depth discussion of key methodological choices and findings.

Ablation studies - Appendix A.5. We conduct several ablation studies to assess the impact of different hyperparameters (i.e., burn-in period, jump parameter, parameter averaging strategy, and augmentation set) on the learning performance of the proposed approach while also providing insights and explanations for usage best practices in this newly introduced context. One key aspect is the set of augmentations. From the results, the number of augmentations is important to improve the uncertainty estimation and, consequently, the performance of the approach. In terms of the choice of modality-specific augmentations, we rely on the large body of literature on TTA and show that standard augmentations work well both in the medical domain and for standard benchmarks of natural images. For other modalities, we propose to perform the augmentation on the latent representation level by simply adding Gaussian noise.

Practicality and computational time - Appendices A.6 and A.7. In terms of practicality, for standard datasets like CIFAR10 and CIFAR100, the number of communication rounds per task with a jump parameter $q=5$ is comparable to the one reported in FedCIL (40 communication rounds per task) and considerably smaller than MFCL (100 communication rounds per task). The same consideration is also valid for the training computational time; our approach has a comparable computational time compared to FedCIL and a considerably smaller one compared to MFCL.

Performance of generative-based approaches - Appendix A.8. The performance gap of generative-based solutions in our experiments stems from the different assumptions of the online scenario compared to the original setting. For instance, MFCL on CIFAR100 assumes 100 communication rounds for each task. This implies that the whole task dataset is processed 100 times, allowing an effective synthetic image generation. In our online scenario, instead, we have access to one batch of 10 images at every iteration. In this case, the effectiveness of generative-based solutions is limited by their need for large amounts of data. This is in line with the online-CL literature where the most successful solutions rely on memory-based approaches (Ma et al., ; Soutif-Cormerais et al., ). To illustrate this point, we generate some images using MFCL trained in the online setting (see Figure 5 in Appendix A.8). The synthetic images provide limited information about past tasks resulting in MFCL being better than standard FL approaches without memory, but not competitive compared to our solution (see Table 5).

Limitations. A limitation of the proposed approach is the need to use an ensembling technique (in our case, test-time augmentation) in order to compute the BI-based estimates. This may result in a reduced efficiency compared to, e.g., ER. Furthermore, the TTA-based estimation of the Bregman Information is only an estimate of the true unknown uncertainty; the estimator used throughout the experiments (Eq. 1) is only asymptotically unbiased and underestimates the theoretical quantity (Gruber and Buettner, ).

Conclusions. In this work, we address the challenges of real-world FCL, where new data arrive in streams of small chunks that cannot be fully retained after being processed during training. While current research in FCL focuses on generative-based solutions that imply an offline setting where generators are trained at the end of each task, we advocate that in realistic conditions complete task datasets may be unavailable and local models should be updated every time new data are received. For this, we devise and formalise a new scenario for the online problem in FCL. To solve CF, we introduce an effective memory-based baseline that combines uncertainty-aware updates with random replay, outperforming state-of-the-art methods. Unlike generative solutions, the newly proposed BI-based estimation for epistemic uncertainty is simple to implement and applicable across different data modalities. Furthermore, compared to standard uncertainty metrics, the empirical results show its superiority in reducing CF in standard settings as well as imbalanced and probing real-world scenarios taken from the biomedical domain. This confirms the ability of the proposed uncertainty-aware strategy to sample more robust and representative samples in challenging tasks and opens new frontiers in FCL, particularly for domains where data scarcity and imbalance are prevalent.

## Acknowledgments and Disclosure of Funding

This work was supported by the The Federal Ministry for Economic Affairs and Climate Action of Germany (BMWK, Project OpenFLAAS 01MD23001E). Co-funded by the European Union (ERC, TAIPO, 101088594). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

## Appendix A Appendix

### A.1 Predictive Uncertainty Scores

The uncertainty metrics considered in our assessment are the followings:

- -
Least Confidence (LC) [Culotta and McCallum, ] measures the predictive uncertainty by looking at the samples with the smallest predicted class probability. If the probability associated with the most probable class $y_{(1)}$ is low, the model is less certain about the given sample.
$u(x)_{LC}=1-\frac{1}{P}\sum_{i=1}^{P}p(y_{(1)}=c|\tilde{x}_{i})$
(2)
- -
Margin Sampling (MS) [Campbell et al., ] measures the predictive uncertainty by looking at the difference between the most probable predicted class $y_{(1)}$ and the second largest one $y_{(2)}$. If the two probabilities are similar, the model is uncertain.
$u(x)_{MS}=1-\frac{1}{P}\sum_{i=1}^{P}\left(p(y_{(1)}=c|\tilde{x}_{i})-p(y_{(2)
}=c|\tilde{x}_{i})\right)$
(3)
- -
Ratio of Confidence (RC) [Campbell et al., ] is similar to MS. In this case, instead of computing the difference, the ratio between the probabilities of the two most probable classes is considered.
$u(x)_{RC}=\frac{1}{P}\sum_{i=1}^{P}\left(\frac{p(y_{(2)}=c|\tilde{x}_{i}))}{p(
y_{(1)}=c|\tilde{x}_{i}))}\right)$
(4)
- -
Entropy (EN) [Shannon, ] considers, differently from the previously introduced metrics, the whole probability distribution. The entropy is computed as follows:
$u(x)_{EN}=-\frac{1}{P}\sum_{i=1}^{P}\left(\sum_{j}p(y_{j}=c|\tilde{x}_{i})\log
(p(y_{j}=c|\tilde{x}_{i}))\right)$
(5)

### A.2 Set of Augmentations

As described in the main paper, we employ TTA to measure the epistemic uncertainty via BI and to compute all the other uncertainty estimates. The set of augmentations used in our experiments is presented in Figure 4. Each augmentation is applied singularly on the data points of interest.

Figure: Figure 4: Augmentation set used sequentially for the calculation of uncertainty in the experiments.

### A.3 Dataset Statistics

In Table 6, we report the statistics of the datasets used for the main experiments. We use dataset from different domains (CIFAR10, CIFAR100, CRC-Tissue, KC-Cell) and with different modalities (20NewsGroups, Yahoo Answers, DBPedia). For CRC-Tissue, in order to create tasks with decreasing size and equal number of classes, we remove the smallest class from our evaluation. For DBPedia and Yahoo Answers Zhang et al. [], we randomly select a subset of 25000 and 33600 samples respectively.

**Table 6: Statistics of the datasets.**
| Number of | Classes | Samples | Tasks |
| --- | --- | --- | --- |
| CIFAR10 | 10 | 60000 | 5 |
| CIFAR100 | 100 | 60000 | 5 |
| CRC-Tissue | 9 | 107180 | 4 |
| KC-Cell | 8 | 236386 | 4 |
| 20NewsGroups | 20 | 18828 | 5 |
| DBPedia | 14 | 33600 | 5 |
| Yahoo Answers | 10 | 25000 | 5 |

### A.4 Evaluation Metrics

In line with Yoon et al. [] and Wuerkaixi et al. [], we employ the average Last Accuracy (A) and average Last Forgetting (F) – a federated adaptation of the metrics defined in Chaudhry et al. []. Last refers to the measurement of the value at the end of the stream for all the clients. Suppose $a_{k}^{t,i}$ represents the accuracy of task $i$ after learning task $t$ on client $k$. The last accuracy $A_{k}$ at task $T$ on client $k$ is defined as $A_{k}=\frac{1}{T}\sum_{i}a_{k}^{T,i}$.
Let $K$ represent the total number of clients. The average last accuracy is then defined as:

$$ $A=\frac{1}{K}\sum_{k=1}^{K}A_{k}$ (6) $$

The forgetting measures the difference between the peak accuracy and the final accuracy of each task. Usually, the peak accuracy is reached when the considered task is trained. After that, since the model is prone to focus more on the upcoming tasks, we observe a performance degradation on the previous tasks. For this, we want to keep the forgetting as low as possible. The average last forgetting is defined as follows:

$$ $F=\frac{1}{K}\sum_{k=1}^{K}F_{k}$ (7) $$

where $F_{k}$ is defined as

$$ $F_{k}=\frac{1}{T-1}\sum_{t=1}^{T-1}\max_{l\in\{1,\dots,T-1\}}a_{k}^{l,j}-a_{k} ^{t,j},\qquad\forall j<t.$ (8) $$

### A.5 Ablation Study

In this section, we investigate the effect of various hyperparameters (i.e., burn-in period, jump parameter, parameter averaging strategy, and set of augmentations) on the learning capabilities of the proposed approach while also providing insights and explanations for best practices to use them in this newly proposed context.

Burn-in period. From the results reported in Table 7, the burn-in period seems to provide some benefit compared to the standard case (burn-in equal to 1) although its contribution is not crucial for improving the overall performance of the approach.

**Table 7: Effect of the burn-in period on accuracy (A) and forgetting (F).**
| Burn-in | A($\uparrow$) | F($\downarrow$) |
| --- | --- | --- |
| 1 | 23.77 | 29.05 |
| 10 | 25.64 | 29.64 |
| 30 | 24.89 | 35.77 |
| 50 | 22.43 | 34.71 |
| 100 | 24.48 | 33.20 |

Parameter averaging strategy. The parameter averaging strategy can be changed without affecting particularly the performance of the approach as can be seen in Table 8. Overall, taking the average between the current and the previous global parameters (as proposed in Shenaj et al. []) helps in reducing CF compared to the vanilla FedAvg.

**Table 8: Effect of the parameter averaging strategy on accuracy (A) and forgetting (F).**
| Strategy | A($\uparrow$) | F($\downarrow$) |
| --- | --- | --- |
| FedAvg | 26.73 | 44.38 |
| Weighted FedAvg | 24.89 | 37.77 |
| FedProx | 26.67 | 38.32 |

Jump parameter. A value between 3 and 5 is a good trade-off for communication efficiency and good predictive performance. If the communication frequency is troublesome due to hardware constraints, one might use higher values at the price of losing pure predictive performance.

**Table 9: Effect of the jump parameter $q$ on accuracy (A) and forgetting (F).**
| $q$ | A($\uparrow$) | F($\downarrow$) |
| --- | --- | --- |
| 1 | 21.15 | 16.55 |
| 3 | 37.83 | 34.84 |
| 5 | 24.89 | 35.77 |
| 10 | 22.75 | 46.46 |
| 50 | 21.99 | 37.86 |

Set of augmentations. To ablate the influence of the augmentations on the performance of our approach, starting from the set of augmentations listed in Appendix A.2, we randomly remove 4 and 2 augmentations from the list. The results reported in Table 10 suggest that removing augmentations decreases the performance of the proposed approach. This is probably because the higher is the number of augmentations used, the more precise is the uncertainty estimation resulting in better samples stored in the memory.

**Table 10: Effect of the augmentation set on accuracy (A) and forgetting (F).**
| Nr. of augmentations | A($\uparrow$) | F($\downarrow$) |
| --- | --- | --- |
| 8 | 20.84 | 40.96 |
| 10 | 22.18 | 38.03 |
| 12 | 24.89 | 35.77 |

### A.6 Practicality Analysis - Number of Communication Rounds

For standard datasets like CIFAR10 and CIFAR100, the number of communication rounds per task with a jump parameter $q=5$ (as used in the main experiments) is comparable to the one proposed in FedCIL (40 communication rounds per task) and considerably smaller than MFCL (100 communication rounds per task). More in details, for each of the aforementioned datasets, each client possesses roughly 2000 images. With a batch size equal to 10, this results in 200 batches per task. Thus, removing the burn-in period of 30 batches, we are left with 170 batches. This means that, for a given task, the number of communication rounds with $q=5$ is approximately 35. The frequency can be further reduced by increasing the jump parameter with a slight degradation in the predictive performance of the model (see results in the ablation study reported in Appendix A.5 Table 9.).

**Table 11: Communication rounds for FCL approaches on CIFAR10 and CIFAR100.**
|  | Communication rounds |
| --- | --- |
| MFCL | 100 |
| FedCIL | 40 |
| O-FCL (ours) | 35 |

### A.7 Time Complexity Analysis

Table 12 reports the runtime in seconds at the end of the training procedure on CIFAR10 for FedAvg (no replay mechanism), ER (random, memory-based), CBR (class-balanced, memory-based), MFCL (generative-based), FedCIL (generative-based), and BI (uncertainty-based, memory-based). From the results, the computational time of BI is in line with most of other uncertainty-based approaches and much shorter than generative approaches. The increase in runtime from CBR to BI stems from the requirement to generate TTA images. In comparison to generative approaches, the runtime of BI is similar or, compared to MFCL, considerably shorter.

**Table 12: Runtime (in seconds) at the end of the learning procedure on CIFAR10.**
|  | Runtime (in seconds) |
| --- | --- |
| FedAvg | $\sim$ 190 |
| ER | $\sim$ 210 |
| CBR | $\sim$ 300 |
| MFCL | $\sim$ 3840 |
| FedCIL | $\sim$ 600 |
| BI (ours) | $\sim$ 660 |

### A.8 Performance Analysis of Generative-based Models

The proposed online-FCL (O-FCL) scenario is very different from the standard FCL scenario presented in, e.g., MFCL [Babakniya et al., ] and related works based on generative-based solutions. In the standard scenario, they assume to work on task datasets that can be iterated multiple times. For instance, let us consider MFCL on CIFAR100 in which the authors assume to have 100 communication rounds for each task. This implies that the whole task dataset is processed 100 times. Thus, after many iterations over the same task dataset, the generative model can effectively generate synthetic images for the given task. In our online scenario, instead, we only have access to one batch of 10 images at every iteration. In this setting, generative-based solutions are limited in their capabilities as they require many data points and many iterations over the training data for being effective. For this reason, we believe that the performance gap compared to the standard FCL setting is given by the inefficiency of generative models in online settings where we do not have the possibility to retrain the model on the same data multiple times. This is in line with the literature in online-CL where the most successful solutions rely on memory-based solutions [Ma et al., , Soutif-Cormerais et al., ]. To prove this point, we generated some images using MFCL at the end of the training procedure in the online setting (see Figure 5). Differently from the images reported in the original paper, we can see that no informative pattern has been discovered; the synthetic images provide limited information about the past information. This is also corroborated by the results of MFCL on the datasets for vision tasks reported in the main paper. From the results, we can see that MFCL is better than standard approaches without memory, but it is not competitive compared to memory-based approaches.

Figure: Figure 5: Synthetic images generated by MFCL [Babakniya et al., ] in the online setting.
Refer to caption: x5.png

### A.9 Pseudocode of O-FCL

Figure: Algorithm 1 O-FCL learning scheme

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see the weakness above.
soundness: 3
strengths: - A real-world online federated continual learning setting is proposed to adapt the practical scenario where new data arrive in streams of mini-batches that can only be processed once.
- Using a bias-variance decomposition of the cross-entropy loss for classification tasks has some merits.
- Experiments are comprehensive and well-designed.
summary: This paper presents a new federated continual learning setting to deal with different modalities in the online scenario. The proposed uncertainty-aware memory-based approach uses an estimate to measure of predictive uncertainty of samples, The extensive experiments demonstrate the effectiveness of reducing the forgetting effect in realistic settings while maintaining data confidentiality and competitive communication efficiency.
weaknesses: - Some advanced memory replay methods are lacking.
- Why do not consider some uncertainty-aware continual learning methods, such as [1] for comparison?
- Why the number of $M$ is different from different datasets?
- The improvement on the KC-Cell dataset is incremental compared to the other datasets. why?
- On the left of Fig. 3, it is better to discuss the notable performance gap between the proposed method and the others on the last task.

[1] NPCL: Neural Processes for Uncertainty-Aware Continual Learning.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: In table 5, for BI method, why do you think the last forgetting(F) score is lower than FedCIL for CIFAR100? Is it related to the increased number of classes?

Please refer to weaknesses section.
soundness: 3
strengths: 1.	The problem statement is meaningful and not widely talked about.

2.	The proposed method works considering memory management in an uncertainty sampling setting, and handles class imbalance and data scarcity well. 

3.	The performance looks promising and overall the paper is easy to read.
summary: The paper proposes an uncertainty aware-memory-based approach for federated continual learning in an online setting where an estimator based on Bregman Information is employed to compute model’s variance at sample level. The proposed method includes predictive uncertainty-aware updated coupled with random replays.
weaknesses: 1.	The fundamental idea of memory management is based on predictive uncertainty which is highly model-dependent and widely used.

2.	While federated continual learning in online settings is not very common, the paper proposed uncertainty estimator and random sampling for replay sets are not novel.

3.	A scalability issue may arise for large datasets such as ImageNet.

4.	It’d be interesting to see how the proposed method would work if task numbers were increased (>20).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to my comments in the weakness section.

======== Justification for my final rating after the discussion with authors ================
The authors argue the technical novelty of the method of using BI for online continual learning and I appreciate/agree to it. While the technical novelty is acknowledged, the empirical validation with a small scale dataset still limits the value of the work. But with the novelty issue has been resolved, I raised my rating to borderline accept as my final rating.
soundness: 3
strengths: * Proposing a new federated learning with **online** continual learning.
* Proposed method improves the accuracy significantly over the state of the arts in multi-modal (vision-and-textual task) setup
summary: The paper proposes a federated learning with end points doing online continual learning (CL), called online-FCL. To address the new task, the paper proposes a memory based CL method that manage samples based on its uncertainty score using Bregman Information (BI). The proposed method is evaluated on CIFAR10/100. On vision, medical and textual datasets, the proposed method does not show much of gain but in multi-modal data, the proposed method improves the performance over the state of the arts by significant gains.
weaknesses: * The proposed method is a straightforward application of Gruber & Buettner (2023), thus the technical novelty is limited.
* Empirical gain seems marginal (also considering the standard deviation) compared to prior arts (Table 1-4). But in Table 5, the empirical gain in vision and textual task seems significant. Any reasoning for this?
* Empirical validation is limited due to the size of the dataset. Although CIFAR-10/100 are popularly used datasets in CL literature, they are quite small sized. ImageNet-1K would be the minimum large scale dataset to validate the value of the proposed method.
* Some results are not clear 
  * In Fig. 3, why the BI and MFCL improve the last accuracy significantly only at the last accuracy?
* In L291-292, it is claimed that "the BI-based estimation of the epistemic uncertainty is meaningful also under distribu- tion shift and able to identify robust and representative samples". But is this fact used in your method? If so, can you analyze that this is true in the Fed CL context?
* Presentation can be improved.
  * In several lines, line break has been removed -- L056, L271, L290 to name a few
  * Some abbreviation is not used without definition -- L072 CF -> catastrophic forgetting (this should be defined in L048)
  * Figure 1 could be larger and clearer. Currently it is drawn with too small fonts, making visibilty bad.
* Misc.
  * Similar previous work that uses CE loss for uncertainty of a sample is missing: Koh et al., **Online Boundary-Free Continual Learning by Scheduled Data Prior**, *ICLR 2023* (the method called CLIB)

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The performance improvements on the CRC-Tissue and KC-Cell datasets are marginal. Given that these datasets contain more samples than others, could you provide a justification for this outcome?

2. How does the Bregman Information approach to uncertainty estimation compare with recent probabilistic methods in Federated Continual Learning, particularly regarding computational efficiency and predictive accuracy?

3. Most results are primarily compared to other uncertainty measurements, while comparisons with previous continual learning or federated learning methods appear limited.
soundness: 3
strengths: 1. The paper presents a novel framework for Federated Continual Learning in an online setting, effectively addressing the practical limitations faced in scenarios where clients continuously receive new data without the ability to revisit previous data.

2. The innovative use of Bregman Information for uncertainty estimation allows for selective sample retention based on epistemic uncertainty.

3. The authors conduct extensive experiments with detailed analyses, demonstrating the effectiveness and depth of the proposed approach across various data types and scenarios.
summary: The paper addresses Federated Continual Learning under the challenges of catastrophic forgetting and online learning, where new data arrives in sequential mini-batches and can only be processed once. The authors introduce an uncertainty-aware memory management strategy that leverages Bregman Information to measure predictive uncertainty. This approach selectively retains samples in memory based on their uncertainty, aiming to improve learning retention while handling various data modalities beyond vision tasks. The paper also showcases the method’s performance on vision and text data, demonstrating its capability to mitigate forgetting across diverse, real-world settings.
weaknesses: see questions.

Meta-review: This paper proposes a new framework for Federated Continual Learning in an online setting. The authors introduce a mechanism to selectively retain representative samples in memory, emphasizing epistemic uncertainty. The method demonstrates competitive performance across multiple modalities, including vision and text, and effectively balances learning retention and communication efficiency. 

After the author-reviewer discussion period, this paper received mixed final ratings. Two reviewers are positive towards this submission, while the other two reviewers still remain negative. Specifically, Reviewer o8MC didn't actively participate in the discussions, while Reviewer BinS confirmed they still have remaining concerns. 

After checking the paper, reviews, and authors' responses, I think the overall quality of this paper is impressive. Extensive results and detailed responses are provided. Therefore, I recommend acceptance of the paper. The authors should include the additional discussions and results in the final revision. They should also revise their paper to address Reviewer BinS's remaining concerns.

Decision: Accept (Poster)