Continual Adaptation of Vision Transformers for Federated Learning
Authors: Shaunak Halbe
James Seale Smith
Junjiao Tian
Zsolt Kira
Venue: Accepted by TMLR ()

Abstract: In this paper, we focus on the important yet understudied problem of Continual Federated Learning (CFL), where a server communicates with a set of clients to incrementally learn new concepts over time without sharing or storing any data. The complexity of this problem is compounded by challenges from both the Continual and Federated Learning perspectives. Specifically, models trained in a CFL setup suffer from catastrophic forgetting which is exacerbated by data heterogeneity across clients.  Existing attempts at this problem tend to impose large overheads on clients and communication channels or require access to stored data which renders them unsuitable for real-world use due to privacy. In this paper, we attempt to tackle forgetting and heterogeneity while minimizing overhead costs and without requiring access to any stored data. We study this problem in the context of Vision Transformers and explore parameter-efficient approaches to adapt to dynamic distributions while minimizing forgetting. We achieve this by leveraging a prompting based approach (such that only prompts and classifier heads have to be communicated) and proposing a novel and lightweight generation and distillation scheme to consolidate client models at the server. We formulate this problem for image classification and establish strong baselines for comparison, conduct experiments on CIFAR-100 as well as challenging, large-scale datasets like ImageNet-R and DomainNet. Our approach outperforms both existing methods and our own baselines by as much as 7% while significantly reducing communication and client-level computation costs. Code available at https://github.com/shaunak27/hepco-fed.

Published in Transactions on Machine Learning Research (09/2024)
Continual Adaptation of Vision Transformers for Federated Learning
Shaunak Halbe, James Seale Smith, Junjiao Tian, Zsolt Kira Georgia Institute of Technology
Correspondence: shalbe9@gatech.co edu
Reviewed on OpenReview: https: openreview. net/ forun? id= ua25A3Zayr
Abstract
In this paper, focus on the important yet understudied problem of Continual Federated Learning (CFL), where server communicates with a set of clients to incrementally learn new concepts over time without sharing or storing any data. The complexity of this problem compounded by challenges from both the Continual and Federated Learning perspectiven Specifically, models trained in a CFL setup suffer from catastrophic forgetting which is exacerbated by data heterogeneity across clients. Existing attempts at this problem tend to impose large overheads clients and communication channels or require access to stored data which renders them unsuitable for real-world use due to privacy. In this paper, we attempt to tackle forgetting and heterogeneity while minimizing overhead costs and without requiring access to any stored data. We study this problem in the context of Vision Transformers and explore marameter-efficien approaches to adapt to dynamic distributions while minimizing forgetting. We achieve this by leveraging a prompting based approach (such that only prompts and classifier heads have be communicated) and proposing novel and lightweight generation and distillation scheme to consolidate client models at the server. We formulate this problem for image classification and establish strong baselines for comparison, conduct experiments on CIFAR-100 as well challenging, largo-scale datascts like ImageNet-R and Nom Our approach out purperorss both existing methods and our own baselines by as much as 7% while significantly reducing communication and client-level computation costs. Code available at https:://gi
com/ wwt./oooo/aaaaeee (ahauunk227hhecoottee
1 Introduction
Federated Learning (FL) is a srivacy-preserving learning paradigm that enables learning global model through communicatin with distributed set of clients. These clients have exclusive access to private data, and collaborate with central server to learn shared task by communicating parameters such model weights, gradients, learning statistics. For example, the popular Fed McMahan et al. (2023) method works by iteratively aggregating client models by averaging their model weights. Classical FL methods such as FedAvg have garnered significant attention due to the increasing demand for user privacy and the growth of edge computing.
However, currently most federated learning methods focus on learning statically, that across a fixed set of categories determined a-priorí. In non-federated works, the other hand, there has been great deal of progress on learning increasing number of categories incrementally, referred to continual learning (and more specifically dass--nncrementl learning) Hsu et al. (2018); van de Ven & Tolias (2019). In addition to the problem of catastrophic forgelting, incremental learning roduces challenges to typical federated learning (FL) scenarios by inherently involving non- non-Indepen -on-Indeeendent and Identically Distributed (non-lID) data, which has been shown to cause issues of model divergence Zhao et al. (2018); [i al.) (2020b). While heterogeneous federated learning Li et al. (2020b) approaches have been developed, they do not support the dynamic data distributions that occur in continual learning and the real-world. For example, such setting has immense

---

Published in Transactions on Machine Learning Research (09/2024)
Client
Client
Pve/
Tash
Server Model
Task
Preiinenedd
Transformer
Prompt
send prompts + task head
receive updated prompts + task headd
Prompt
Pool
Client 1
Client
...
Client
Client 1 - learn continual stream, updating prompt pool and task head
Server Aggregation without Private Data using Pseudo-Latent Knowledge Distiliation
Client $ same as client 1, but sees unique and different continual stream
Figure 1: In Continual Federated Learning (CFL), clients learn from unique, continual data. We propose a prompt-based CFL approach, paired with a lightweight generation and distillation scheme, to consolidate client models al the server a communicationn-efiiieni manner.
practical impact to applications such healthcare, autonomous vehicles, and chat-bots Rieke et al. (2020 Nguyen et al. 2022
Therefore, in this paper we look at the understudied problem of Continual Federated Learning (CFL) Yoon et al. ) Ma et al. (2022); Qi et al. (202)). While a few CFL methods exist, they address the issue of forgetting using approaches such as: reducing inter-cliem interference, knowledge distillation, and image-level generative replay Yoon et al. (202T); Dong et al. (2022); Zhang et al. (2023). Specifically, they often communicate full model weights, rcal/s hsythsszd image-level data, or gradients. Additionally, some methods store old data in memory buffers train generative model to mimic local data; at the very least, all methods share complete models parameters with the server which can lead to privacy leaks with advancements in model inversion and other extraction techniques Carlini et al. (2023). As result, many of these methods fail to effectively uphold the principles of CFL, such as communication efficiency, compulational efficiency, and privacy.
To mitigate forgetting while adhering to the core principles of CFL, we propose HePCo: Heterogeneous Prompt Consolidation (Fig, D.. Our method is driven by the goals of (i) minimizing communication costs, (ii) improving client privacy, and (iii) client-level computation efficiency. We first propose to leverage prompting- based methods, which have shown successful results in the rehearsal-free continual learning setting Wang et al. 2022c b). This also has the benefit of utilizing frozen Vision Transformer backbones, meaning that only prompts and classifiers have to be transmitted, reducing communication. The key contribution of our approach is then to answer the question of how to merge prompts from different clients in a scalable manner. Towards this end, we propose lightweight method for generating pscudo-data in the latent space and distilling client model information. Importantly, we distill data from both the past task label distribution as well as the current task label distribution, preventing both catastrophic forgetting and performance degradation due to client heterogeneity. In summary, make the following key contributions:
1. We modily and extend popular continual prompting based approaches to the selling of Continual Federated Learning (CFL), implementing range of methods that we will open-source.
2. We introduce a light-weight pseudo-latent knowledge distillation mechanism to aggregate client mod- els trained non-lID data partitions. Importantly, our approach incurs no additional client-side computation and does not store generate training data.
3. We outperform both existing methods and contributed baselines by much 7% while drastically reducing communication costs by only sharing a small set of model parameters, which constitutes only -9.5% of the total parameters.
2 Related Work
Prompting for Continual Learning. Continual learning algorithms fall into several primary groups.
Some methods involve expanding the model's architecture as new tasks arise Ebrahimi et al. (2020); Lee ct al. (2020); Lomonaco Maltoni (2017); Maltoni & Lomonaco (2019): Rusu et al. (2016), while others regularize the model with prior task knowledge either the weight space or the prediction space Ahn et al.
(2021) Aljundi et al. (2018): Hou et al. (2018); Kirkpatrick et al. 2217 Li & Hoiem (017); Zenke et al.
2

---

Published in Transactions on Machine Learning Research (09/2024)
Additionally, rehearsal- methods leverage stored data or samples from generative model Bang et al. 2021p | Chaudhry et al. (b]9; Hayes et. al. (2019): Hou et al. (2019); Lopez-Paz & Ranzato (2017
2017
Ostapenko et al. ; Rebuffi et al. (2017b); Shin et al. 2017); von Oswald et al. (2019) van de Ven et al. (2020). Despite their efficacy, these methods might compromise data privacy and impose substantial memory costs, which justifies the need for rehewrsal-free stmleegies. In the realm of rehearsal-free continual learning, some studies focus on an online learning perspective utilizing a pre-trained model Hayes & Kanan (2019); Lomonaco et al. (2020), or explore prototype- based techniques to avert catastrophic forgetting Yu al.
(2020); Wu et al. (021];
); Zhu et al. (2021). Other work proposes deep-model model inversion to produce images for rehearsal, yet the computational expense and data-privacy issues render this approach challenging Choi et al.
(201 ;; Kaissis et al. (202Tp; Smith et al. (2021); Yin et al.] (2020). NCDwF Joseph et al. (2022) performs
classifier inversion and pseudo-replay in the latent-space to reduce forgetting in a novel class-discovery setting. Recent works have demonstrated that prompting within a static, pre-trained transformer model for
continual learning achieves state of the art performance without any rehearsal data /mith ct al. (2022); Wang
et al. [002dda) a). Our work builds on the foundations of these prompting methods, but in lederated selting.
Heterogeneous Federated Learning. Federated learning involves number of clients learning over local data, to be aggregated by a global server. One of the carliest methods for this is Fed.Ag McMMahan et al.] (003)), which simply averages the parameters of the clients weighted by the relative amount of data they were trained on. The most investigated challenge in federated learning (FI.) is client heterogeneity, which can cause local models to diverge and the aggregated global model to perform sub-optimally Li et al. (2020b). In this work, we discuss FL works tackling this issue in two categories: parameter-averaging methods and knowledge distillation methods. To combat high heterogeneity in local data, methods like FedProx T.i et. al. (2020a), FedPD Zhang ct al. (2220); FedDyn Acar et al. (2021), and SCAFFOLD Karimireddy al. (2019) are used, out of which FedProx is a stateless algorithm useful in settings where the number of clients is very large which prevents the server from keeping a track of all participating clients. The rest are stateful algorithms that maintain client states. nnoowvegg-dissilation--eee
methods Mora et al.] (2022); ; Lin et al.
2220); Sattler et al. (202I) usually use additional data to perform distillation on the server or client side to robustly aggregate local models. FedFTG Zhang et al. (2022a) operates in a data-free setting by generating pseudo-data through inversion in the image space. In contrast, we achieve knowledge distillation using pscudo-data. generated in latent space to 1) reduce computation overhead stemming from inversion in a higher-<limensional space and 2) fine-tune both the classifier and the prompt components simultaneously which is essential to mitigate forgetting.
Continual Fedorated Learning. Most of the current CFL methods suffer from various limitations in terms of performance, efficiency and privacy. FedWeiT Yoon el al. (202T) aims to learn better client models by leveraging indirect experience of other client models. The objective introduced in this work minimizes interference across client weights. A knowledge base of previous task parameters for all seen clients is maintained at both the server and client side and is updated during each round through client-server communication. Each client selectively utilizes the parameters in this knowledge base through attention masks to gain indirect experience. FedWeiT incurs considcrable overheads in terms of communication, computation and storage stemming from maintaining and updating this knowledge base. GLFC Dong el al. (2022) uses a prototype based approach with a memory buffer to store old data. This poses a threat to privacy of client data. CFed Ma et al.] (2022) proposes distillation based approach that makes use of an unlabelled datasct to aggregate client models as well as to rehearse old tasks. CFeD performs distillation server side. However the requirement for a curated dataset can severely impact real-world applicability. TARGET Zhang et al. (2023) attempts to combat forgetting through replay of old tasks driven by generated images. FedCIL Qi et al. (2023) leverages an Auxiliary Classifier GAN (ACGAN) to alleviate forgetting by synthesizing old images for replay. However, generating images to mimic local datasets can be viewed as a privacy risk especially in light of recent research on model inversion attacks Carlini et al. (0023). In contrast, our approach prioritizes client privacy by generating in the latent space, thereby eliminating the need to generate in the image space. Further, this benefits communication and compute efficiency. GAL Wang et al. (2023) introduces the task of Federated Continual Novel Class Learning, where FI. methods are expected to discover and learn unlabelled novel class data. In contrast, our work explores a supervised continual learning setup where new labeled instances are incrementally introduced. The recent work Fed- Ped-CPrompt Bagwe et al.) (023). investigates the use of prompting for federated class- incrementtt learning. To combat client divergence,
both client and
3

---

Published in Transactions on Machine Learning Research (09/2024)
their approach introduces contrastive loss at the client side. Client aggregation is performed at the server using the standard Avg algorithm. Our work differs in that we do not introduce additional objectives at the client side. Instead, we propose a novel method for combining client models at the server to mitigate forgetting.
3 Problem Formulation
In this section, we describe the formulation by introducing the class-ncremennal learning and heterogeneous federated learning aspects in our CFL setting.
Incremeancll Federated Learning. We focus on the class-incrementta learning scenario, where a model is tasked with learning new classes over time. Under this setting, a global model is learned through a sequence of N global tasks T [T" T2, I. Following the standard assumption in continual learning, the sets of categorics seen across distinct global tasks are mutually exclusive. As this is done in federated setup, each task is learned through R independent rounds by randomly sampling set of stateless clients C= (cI, 02 C3; .... Cg) } in each round. In a stateless setting, the total number of clients is kept very large to simulate woal-world FL applications (like mobile devices). The server does not keep track of clients as new clients are visited in each round. Further, previously seen data cannot be accessed at any time during the training.
Heterogencous Federated Learning. To simulate a real-world heterogencous foderated learning scenario, we use three configuration parameters to control the level of heterogeneity with increasing granularity: splil ratio, category ratio, and imbolance ratio. At the top level, the most common heterogeneity is varying local dataset sizes across the clients. A specific client can be exposed to a subset of the current task dataset D' as their local dataset D and the size IDi varies from each other. We denote this as the split ratio 7 7= |DE///DD' At a lower level, the local dataset. DÝ consists of a subset of the categories from those in the current global task. Specifically, a global task I consists of categories K', where | denotes the number of unique categorics. In a given round r, cach client e, soes data containing KỤ E K" categorics. We denote = | as the calegory ralão which is a value wetween O and 1. At the lowest level, each category can have a different amount of data. We follow [ao et al. (0119) to also create long-tail distribution for local data which is different for each client. This distribution is governed by an imbalance ratio B. If 3 1, each client C is allocated samples uniformly from Kÿ categories. In summary, a smaller split ratio ,, a smaller category ratio K, or a smaller imbalance ratio ,3 increases heterogeneity thereby increasing the complexity of the task. Formalizing the setting in this manner enables us to methodically vary the parameters, thereby simulating a heterogeneous setting. This formalization unifics the setups previously employed separately across federated and continual learning landscapes Li et al. (2020a): Rebufh et al. (2017a) combining their distinct characteristics.
As discussed before, combining client models in a heterogeneous setting causes the obtained model to forget global knowledge from the previous rounds as shown by Lee et al. (2022): Hsu et al. (2019). Also, training clients on locally-available datascts induces forgetting of global knowledge outside the local distributions. Intru-lask forgelling Ma et al.] (2022) measures such performance drops induced by the data heterogeneity (non-IIDness)) across clients. Inter-lask forgetting measures the drop in performance on old tasks {T1 ,7 r2....,?'-" after learning new task T
4 Background: L2P
L2P (Learning to Prompt) Wang et al. (2022d) is a continual learning method that learns set of model embeddings (prompts) that can be dynamically inserted into a pretrained vision transformer. (ViT) Dosovitskiy et al. (2020). Prompts hold task-specific information that instructs the model to solve the corresponding tasks. L2P maintains prompt pool P = (P1, P2, ,,r of size M, where P. E BR,DD are prompt parameters with Lp as the prompt length (chosen as hyperparameter) and D the embedding dimension. Each prompt.
We the terms 'category', 'class', and 'label' interchangeably to refer the target classification label.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Client Side (Section 4.2)
prompt weighting
query
p R L
K E R MxD
P PER M
y
gen
Ugen
0 gen
gen
concat
P1:ma
1in
p Psersos
Resrver
& 1-1 Pservar
P c-1
Paervar
Cors (Eg.2)
Cors (E)
11:32
Peervor
P
Pacrver
Secrver
Curo (Eq.4b)
DMsE (Ey 1b)
Cprompt (Eq.6)
LCE
Cicz (Eq.4a)
Cic. (Eg.4u)
Server Latent Generation (Section 4.3)
Server Side: Latent Space Distillation (Section 4.4)
Figure 2: Latent generation and distillation with underlying decomposed prompting scheme.
P. has associated key k RD, An input image is converted into visual query ((x) E R by passing it through the frozen vision transformer encoder Opt. Prompts are selected from the pool by measuring the cosine similarity between associated keys and the visual query. Top N prompts that have the highest key-query similarity are chosen to be inserted into the transformer. Here, the ViT encoder is frozen and the prompts and keys are learned using two separate optimization losses.
5 Method
In this section, describe novel approach called HePCo (Heterogenous Prompt Consolidation) which tackles forgetting and heterogeneity using data-free distillation strategy applied in the model's latent space. Knowing the limitations of FedAvg under non-lID data partitioning, we propose to fine-tune the global model at server side by distilling knowledge from client models. However, unlike prior CFL works, we first propose to leverage the current state of art prompting methods continual learning. Such methods optimize lcarnable parameters that augment the input to transformer model (prompl. tuning) or underlying atlention mechanism (prefix tuning). These methods have been shown to obtain strong performance in traditional rehearsal-free continual lcarning scttings. These prompting-baser approaches Wang al. (2022d) Smith et al. (0222) can be thought of implicit mechanisms isolate model parameters across tasks. Implementing prompting scheme at the client level can thus prevent forgetting of past tasks while efficiently adapting to
new tasks.
Despite these advantages, there a key challenge in applying prompting to a federated setting: not obvious how the server should combine prompts learned by the individual clients on heterogeneous sources of data. Naively averaging the prompt weights is suboptimal (as we show in Sec. 6 and simply maintaining growing prompt pool scales poorly with the number of clients. Our key novelty is therefore to propose lightweight distillation method, applied to the latent-space of the model, which greatly mitigates intra-task and inter-task forgetting. Crucially, rather than averaging different client prompts, we perform distillation of these prompts in data-free manner by generaling pseudo-dala. Importantly, the generation and distillation operations are computationally cheap as they are carried out in the latent space of the model. This design prioritizes privacy and efficiency, which are crucial for federated learning. In summary, with our mothod, communication costs between clients and the server are low, heterogencous information is effectively aggregated, and the method achieves state-of-art performance. Below we detail our method and depict it in Fig.
5

---

Published in Transactions on Machine Learning Research (09/2024)
5.1 Client Side: Decomposed Prompting
While L,P is quite successful in protecting against lorgetting, its performance is limited by certain design choices, as noted by Smith et al. (2022). We therefore adapt L2P to side-step these issues, including for our basclines, resulting in better accuracy across the board. Specifically, rather than using discrete prompts obtained via a hard maximum function which restricts capacity and introduces an additional hyperparameter (N, corresponding
top-N prompts), we form our final prompt p by taking sum of the P; weighted by their cosine scores. Such a prompt is inserted into subset of the self attention layers of the ViT encoder. This subset of layers is determined by hand as in Smith et al. (2022).
In the federated learning setting, each client hosts the aforementioned prompting scheme and learns the key and prompt matrices along with the classifier in an end-to-end fashion while keeping the ViT backbone frozen. After learning the local task, in our method, the clients transfer the key, prompt and classifier weights to the server. By sending only a limited number of parameters to the server, the overall communication load is significantly reduced compared to sharing the entire model. Our approach requires clients to share only the prompt and classifier weights. The server lacks knowledge of the exact locations for inserting the prompts, this information is known only to the respective clients. To reconstruct the exact client model, the server would need to conduct an exhaustive search over the number of layers in the transformer and try various combinations to determine the precise insertion positions. This method represents a positive step towards enhancing client. privacy compared to approaches that share complete model weights.
5.2 Server Side: Latent Generation
At the end of each round, the server receives |C| prompt and classifier weights collected from the active clients. Let We indicate a weight parameter that encompasses key, prompt and classifier weights. In the first stage, we obtain the server model by averaging client weights as: w = Ene We call these as the provisional server weights.
Due to data heterogeneity, the local model weights diverge which degrades the performance of this aggregated model. We therefore propose to fine-tune the server model using data-free distillation in the latent space to prevent this degradation. We generate pseudo data in the latent space of the visual query q(x) R which is essentially the output space of the vision encoder. The advantage of generating in this space is that it allows us to fine-tune both the classifier and the key-prompt weights without needing a forward-pass through the encoder. We use a lightweight feedforward neural network as our conditional generator with a D dimensional output. This generator takes as input a class label (from categorics in current task t) and a noise vector of dimension Dnoise sampled from the standard normal distribution (00,1) 1). We encode the class label using an embodding layer and concatenate the obtained class embedding with the noise vector to form the input of the generator. From the generator, we obtain pseudo latent of dimension D conditioned on the class label as follows:
2 = G(c, y: Ogen).
where: E R is the generated pseudo latent and E Raase ~ NO, 1) is the noise vector. For effective knowledge distillation, pscudo data should conform to the latent space of the client models. We optimize for a classification loss which is a weighted sum of classification losses for each individual client, similar to Zhang et al. (2022b). The total classification loss can be given as:
Lasa Σ C.. and, CEC
(2)
C3 Σ (cn(b(2z.....))
(3)
BEC
where CG is the cross-entropy loss between the prediction of local model e given latent z and sampled class label y. Here, & denotes the classifer (last layer). However, optimizing for just the classification loss encourages the generator to produce pscudo latents which are casy to be classified and hence less effective for distillation. Our goal to generate latents that create a discrepancy between the provisional server model and clients, thereby providing learning opportunity for the server. To promote the generation of such hard samples, we
6

---

Published in Transactions on Machine Learning Research (09/2024)
maximize two disagreement losses (one for prompts and one for classifier) between server and client models. As stated earlier, latents in this space can be forwarded through both prompting and classifier mechanisms. Hence, to measure the disagreement with respect to the classifier, we compute the Kullback-Leibe (KL) divergence between the predictions of the classifier corresponding to the provisional server model and cach individual client. Next, to measure the disagreement with respect to the prompting module, we introduce a Mean-Squared Error (MSE) loss between the final prompts generated by the server and all clients. By training the generator to maximize these losses, we increase the potency of pseudo-latents for distillation in both parts of the network.
CKL Σ
(da)
LMSE ELmssoooesss)
(1b)
CEC
where a denotes the softmax function and p denotes the prompting mechanism described in the generator by optimizing for these losses jointly as:
5.1 We train
min (( (Lass -OI--KKLKE---- AMSELMSE) ( Byon
(5)
The trained generator is then used to perform data-free knowledge distillation which helps combat intra-task forgetting and allows the server model to achieve a high accuracy on the global data distribution of the current task. However, as the generator is trained to generate pscudo-data corresponding to the current task distribution only, a model fine-tuned with this pseudo-data suffers from inter-task forgetting as shown in our ablation experiments in Scction 6.2.1 To prevent this, we train a scparate copy of the generator (dgen) to generate latents corresponding to the previously seen tasks. At the server side, we assumee access to the key, prompt and classifier weights corresponding to the previous task's global model which is the fine-tunes global model after the R round of the task 1. Similar to above, we optimize for the classification and disagreement losses jointly. Here the classification loss Lais is computed for the previous task server model and Lx and LMSE are computed between this and the provisional server model. We empirically show that using pseudo- data corresponding to past tasks for knowledge distillation helps mitigate inter-task forgetting to a great extent.
5.3 Server Side: Latent Space Knowledge Distillation
Once the generator is trained to generate pseudo-latents corresponding to the current and previous tasks, use it to efficiently finc-tune the provisional server model w. We use the pscudo-latents obtained from the generator to fine-tune both the classifier head and key-prompt weights (K and P) without requiring a forward pass through the full model. We use the key, prompt and classifier weights corresponding to the current round client models and the last-task server model to fine-tune the server model. As it operates in a low dimensional latent space and updates a small subset of parameters, this distillation process is much cheaper in terms of computation compared to training the entire model. Also, this design does not require clients to share entire models with the server which reduces the client-server communication costs to great extent and improves privacy of the client model. While introduce additional server overhead, it is important to note that, in the context of CFL, clients are typically edge devices with limited computing power, while servers have ample computational resources. We prioritize client-level efficiency while making efficient use of the server's resources. In Section 6.2.2 we compute this overhead and show that it is competitive to that incurred by existing state-of-the-art (SOTA) methods.
To perform knowledge distillation, we first generate a batch of pseudo-data from the gencrators corresponding to the current. round and previous task. We mix the current and previous task batches form a single composite batch according to hyperparameter named replay ratio which determines the size of the previous task batch relative to the current round batch. We use this composite batch of pseudo-latents to finc-tune the key-prompt weights and the classifier weights separately.
To fine-tune the key-prompt weights, we first obtain distillation targets in the form of final prompts p by passing the pscudo latents through the prompting mechanism of the teacher models (clients and previous-task server model). Notably, we do not require full models to generate these targets, as having only the key-prompt and classifier weights is sufficient. Now, to fine-tune the key-prompt weights of server model, we optimize
7

---

Published in Transactions on Machine Learning Research (09/2024)
for the Mean Squared Error (MSE) loss between the final prompts predicted by the provisional server model and each individual teacher model (clients and previous-task server).
Cprompt CirsE + 1i-L
(6)
eEC
where LMSE denotes the MSE loss belween client e and the provisional server model and LMSE denoles the MSE loss between the provisional model and the previous task server model. Further, Çp-1 is an indicator variable which is set to 1 if y was seen in previous tasks and 0 if present in current task.
Next, we fine-tune the classifier layer of the provisional server model. As discussed in Section 5.2 pseudo-latents were obtained from the generator by conditioning on randomly sampled class labels. The composite batch of pseudo-latent used in the previous step is employed here again as input to the classifier, and the cross-cntropy loss is computed between the predictions of the provisional server and the class labels upon which the pseudo-latents were conditioned. This approach allows us to fine-tune the classifier weights using the same batch of pseudo-latents used to fine-tune the key-prompt weights. Operating in the latent space allows us to efficiently fine-tune the key-prompt and classifier modules without requiring forward passes through the entire model. Our ablation studies in Section 6.2.1 highlight the efficacy of this approach.
6 Experiments
Model Architecture. We use the ViT-B/16 backbone Dosovitskiy el al. (2020) pretrained on Imagenet-1K Russakovsky et al. (215)) as the encoder for our method and all baselines. We use a prompt pool size (M) of 100 and a prompt length (I,)) of 20 with dimension (D) being 768 and insort prompts into 1-5 Multi-head Self Attention (MSA) layers of the VIT encoder following the standard practice Wang al. (2022b) and perform prefix-tuning as done in Smith et al. (2022), by prepending prompts to the keys and values of the MSA layers. The classifier (Ø) is fully-connected layer with input dimension D and output dimension equal to the number of classes. We implement the generator fgen using three layer fully-connected network and train it for 100 epochs. We encode the class label using an embedding matrix and concatenate the class embedding to the sampled noise vector before feeding into the generator.
Datasets. We conduct our experiments on three image classification datasets. First, we adapt CIFAR-100 Krizhevsky al. (2009) to our formulation as it is a commonly used benchmark in CFL. Additionally, evaluate our methods on the larger-scale ImageNet-R [Hendrycks et al.. (2021) and DomainNet Peng et al. (2019) which have been used in recent continual learning works Smith et al. (2022) but haven't 't been explored in a continual federated learning setting. These datasets capture real-world distribution shifts that can be challenging for models pre-trained on ImageNet to generalize to. The total number of classes for CIFAR-100, ImageNet-R and DomainNet are 100, 200 and 345 respectively. We divide these datascts into 10-task (CIFAR-100, TmageNet-R) and 5-task (DomainNet) benchmarks. The 10-task setups contain a longer task sequence with small number of classes per task whereas the 5-task setup has a shorter task sequence with more classes per task. CIFAR and ImageNet have 10 and 20 classes, while DomainNet has validation dataset
69 classes per task. Following Wang et al. (202aa), use 20% of the training set to determine hyperparameters for our approach and all competing baselines.
Configuration. We learn each task through It = 10 communication rounds by selecting C' = 5 stateless clients per round. Thus, we have 100 total rounds for a 10-task setup and 50 for a 5-task setup. For all experiments reported Tables 1 we use a category ratio x = 0.6 which means that a task contains 10 categories, each active client randomly assigned 6 of these categories. We analyze the affect of different category ratios in Sec. 6.2.1 Overall, HePCo outperforms existing methods across all category ratios, particularly excelling in scenarios with smaller category ratios, which signify higher heterogeneity and, consequently, higher level of task complexity. Further, we use a split ratio = 0.1 which allows client to be assigned 10% of the images corresponding to the subset of calegories. We train local models for 10 epochs per round.
We include additional implementation details in Appendir
Metrics. We evaluate all methods using the standard continual learning metries of (1) final average accuracy An which is the accuracy averaged over all N tasks after learning the No task and (2) average forgetting
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Results (%) for the class-balanced setups reported over 3 indeppndentt trials. AN gives the accuracy averagod over tasks and Fv gives the average forgetting.
Dutasets (3
CIFAR-100
ImngeNot-RR
Domainn
Method
AN (t)
FN (C)
Ay (1)
EN (1)
Ay (T)
Fy (4)
Prompting (Centralized)
85.35
72.28
71.33
FedAvg-FT FT
10.23 = 1.10
31.74
+ 0.80
12.03 0.75
29.07 0.66
18.76 = 0.44
32.81 + 1.22
FodLwF.MM Rebuffi et al.
2017a
59.08 + 1.06
12.39
0.76
52.87 0.61
13.34
62.39 1 1.12
10.76 0.550
Fod/rrg- Prompt
67.34
1.42
8.38
0.42
51.15 0.68
8.84
51.03 1 2.23
12.03
Fed-C Ved-CPromptt//l Bagwe et al. 2023
69.38 = 1.39
7.66.
52.24 0.66
8.21 4.21:0055
60.38 0.78
8.22 0.46
CFed
Ma et
72.26 1.56
8.82
0.64
45.64 1.32
11.74 1174++1.0
63.32 0.78
7.12 + 0.66
TARGET Zhang ot
2023
73.56 + 1.42
6.83
52.38 1.16
8.88 + 0.96
61.88
1.66
7.94 0.52
HePCo (Ours)
76.54 1 1.14
6.61 + 0.73
59.96 1 0.94
7.08 1 0.40
64.01 + 0.36
6.83 1 0.31
Table 2: Results (%) for class-imbalanced setup reported over 3 independent trials. AN gives the accuracy averaged over tasks.
Datasets
CIFAR-10
ImageNet-R
DomainNet
Method
Av (1)
AN (t)
AN (t)
Iibaalance ratio (,))
8 = 0.05
8 0.01
5 0.05
8 8- 0.01
8 0.05
8 = 0.01
FodAvg-FTT
8.81 = 1.53
9.18 = 1.26
9.26 #: 1.02
8.88 +: 1.24
13.02
: 1.29
11.65
+ 1.84
FedLwF.MM MC Rebuffi et al
2017a
50.40 + 0.88
40.30 + 1.06
19.94 + 0.78
13:34 1.41
57.34
+ 0.84
52.46 +: 0.72
FedAAvg Prompi
62.72 + 1.79
54.43 + 1.57
36.51 + 0.86
28.16 - 1.12
47.73 + 1.25
43.23
+ 1.03
Fed-CPrompt Bagge et al. 2023
65.42 + 1.12
56.85 + 1.79
39.03 + 1.04
30.14 = 1.16
54.44 + 0.80
49.96 + U.74
CFed Ma el al.
2022
70.26 = 1.20
62.04 + 1.62
34.62 + 1.41
25.74 1.08
59.89 + 0.68
55.22 + 0.80
TARGET Zhang et, al. 202
66.47 = 1.22
58.13
1.54
30.20 + 1.35
19.84 = 1.41
56.44
+ 0.45
51.82
+ 0.58
HePCo (Ours)
70.34 +: 1.08
61.70 + 1.48
45.45 + 0.98
41.68 + 1.44
61.10 + 0.76
58.82 + 0.84
Chaudhry et al. (2019a) Lopes et al. (0117 Fy which measures the drop in performance previous tasks after learning new task averaged over all N tasks. As noted by (mmith et al. (0022), AN is the more informative metric as it encompasses both forgetting and plasticity (new task performance). Our approach provides parameter efficiency which is reflected by reduced communication and local computation costs; albeit at the expense of a small overhead at the server. We quantify this communication efficiency by specifying the number of parameters shared by our approach relative to the original model size in 6.2.2
Baselines. We compare our method against existing state-of-the-art approaches: TARGET Zhang et al. (2023). CFed Ma et al. (022) and Fed-CPrompt as well two strong baselines that introduce. TARGET and CFeD mitigate forgetting by replaying old lask data through generative methods or by assuming access to surrogate datasets. Fed-CPr -roomtt uses prompting based approach similar to ours but introduces a contrastive loss at the client side. This additional loss aims to mitigate forgetting by alleviating hetorogenoity across clients and tasks. At the server side, Fed-CPrompt aggregates clients using the standard FedAvg algorithm. Tn contrast, our approach does not. introduce any additional computation at the client's end. Instead, we foeus on altering the aggregation procedure at the server to combat forgetting. For fair comparison, we adapt Fed- Coo-Cpr our experimental setup and use the same Vision Transformer (ViT) architecture in our method. Similarly for all other methods, we adapt their implementations to use the same ViT backbone with proper modifications. We tune hyperparameters for our proposed method and all compared baselines. We observe that the FedLwF method Ma et al. (022) used in prior CFL work performs poorly owing to the complexity of our setting. As the heterogeneity across clients increases, coupled with an increase in the length of the global task sequence, the performance of FedLwF deteriorates catastrophically. We instcad adapt LwF.MC with sigmoid binary cross-entropy loss as described in /Smith et al. (2023) which is a strong continual learning baseline to our setting. We call this method FedLFF.MC which achieves much better performance than its vanilla counterpart. Additionally, we introduce a simple yet strong prompting based methods which we call Prd.Avt-rrompt. TedAAv--rrompl differs from our method in the model aggregation part at the server side where the clients are simply averaged to obtain the server model. Additionally for completeness, we report the performance of FedAvg-F'T where the entire client models are sequentially finetuned on new task data (as opposed to learning only prompts and classifier) and aggregated using FedAvg. Finally, we report the performance of our decomposed prompting scheme in a centralized, traditional continual learning setting. This can be thought of an upper bound performance for all prompt-based methods included here.
9

---

Published in Transactions on Machine Learning Research (09/2024)
90
Method
Aw (t)
Fy (L)
80
HePCo (Ours)
76.54 + 1.14
6.61 = 0.73
70
Ablate previous server model
61.15 2.13
11.18 + 0.76
60
Ablate LKL. & CMSE
70.22 1 1.45
8.86 0.66
Ablate CKL
71.39 + 1.34
8.28 = 0.74
50
Ablate CuSE
74.11 1.31
6,96 0.68
Ablate prompt distillation
74.42 1.22
7.02 0.88
HelCo
TARGET
Ablate classifier distillation
68.46 +: 0.91
8.28 = 0.52
30
CFed
MC
Table 3: Ablation Results (%) on 10-task CIFAR
Feoteg-hhampt
20
100. AN gives the accuracy averaged over tasks and FN gives the average forgetting.
0.2
0.3
0.4
0.6
0.7
0.8
0.9
1.0
Category Ratio
Figure 3: Comparison of the methods under different category ratios.
6.1 Main Results
The results presented in Tables D and ] demonstrate the dominant performance of our method in terms of average accuracy and forgetting across across all datasots and setups. The gains achievod by our method are more pronounced in the ImageNet-R setup which has longer lask sequences and offer siguificant shift. from the pretrained distribution. Our approach achieves absolute improvements of up to 7% in average accuracy compared to the best bascline. All basclines that fine-tune the entire model are seen to struggle with longer sequences (CIFAR, Imagenet-R), showing significant forgetting. Under the class-balances setting of Table 1 our approach achieves absolute improvements of more than 7% on ImageNet-R in average accuracy compared to TARGET Zhang et al. (2023), which is the current SOTA. For the
settings in Table 2 our approach outperforms the competition by even wider margins. The notable performance drops observed across all methods highlight the complexity of this setting. Most importantly, HePCo achieves these solid results while enjoying low communication costs and without introducing any additional costs at the client-side. Furthermore, our approach faithfully aligns with the principles of Federated Learning (FL) by not assuming access to any storage, be it surrogate datasets or generated images, in contrast to methods like CFed Ma et al. )022), TARGET Zhang et al. (2023) and GLFC Dong et al. (2022).
6.2
Additional Analysis
6.2.1
Ablation Studies
We perform ablations experiments on CIFAR-100 in the No Imbalance setting from Table I
A Ablating distillation of previous server model. By removing the previous task server model from the distillation and generation steps, we highlight its importance in alleviating forgetting. By ablating this component, we observe significant drop in performance indicated by a rise in forgetting (Fx) and a drop in average accuracy (An). The underlying intuition is that without the replay of past task data, the method strongly prioritizes learning of the current task leading to a loss of knowledge from previously seen tasks. In other words, using past task latents for replay mitigates inter-task forgetting.
Ablating disagreement losses in generation. To demonstrate the effectiveness of disagreement losses in generation, we set the lambda coefficients λKL and AMSE to zero and observe 6% drop in accuracy. As discussed before, the intuition here is that in absence of the disagreement losses, the generator is prone to generate easily discriminable examples that lead to low classification loss but are less effective in distillation. To further highlight the importance of the individual losses, i.e LMSE and LKI, we individually ablate them and observe performance drops.
A Ablating distillation sites. Our approach uses pseudo-latents to fine-tune the key-prompt and classifier weights of the server model. In this experiment, we ablate the decision of fine-tuning the prompt components
10

---

Published in Transactions on Machine Learning Research (09/2024)
Method
Communication Cost (4)
Computation Overhead (4)
Storage Overhead (4)
Server-side
Client-side
Server-side
Client-side
CFed
Ma et al. 20222
330.3 MB
98s
68s
1433.6 MB
1433.6 MB
TARGET Zhang al. 2023
714.7 MB
128s
62s
384.1 MB
384.1 MB
HePCo (Ours)
31.37 MB
220s
N/A
31.37 MB
N/A
Table 4: Overhead costs incurred by our method against top performing baselines.
and the classifier scparately and observe a decline in accuracy in both cases. The drop in performance is more pronounced when we do not perform distillation for the classifier. This experiment highlights our decision to fine-tune both prompt components and classifiers by operating in the latent space.
Varying the category ratio. Figure g shows the performance of all methods for different values of calegory ralio. We observe that HePCo consistently out performss competing methods without requiring any hyperparameter or design changes. The performance gap between HePCo and the competing methods widens with decreasing category ratio, indicating its effectiveness in high heterogencity scttings.
6.2.2 Overhead Cost Analysis
Memory & Communication Overhead. Our method introduces additional parameters forming the prompting mechanism. The additional paramcters amount to -9.4% of the original size of the ViT encoder. Our method only needs to communicate the learnable parameters in the model which are the classifier and key-prompt components amounting to -9.5% of the original model size. Methods that finetune the entire model need to learn and communicate all parameters the encoder and classifier. Hence, our approach requires only 9.5% of the communication costs compared to all other methods that share complete models. Furthermore, the current state-of-the-art methods like CFed and TARGET require communicating dataset. of images (obtained from the surrogate dataset generative mechanism) after every round task which significantly increases the communication overhead in addition to sharing complete models! For ViT-B/ 16 architect ure, sharing a complete model amounts to 330.3 MB of information. In addition lo complete model weights, TARGET sends a buffer of 8k synthesized image-level data from server to clients to perform distillation leading to a communication volume of 714.7 MB. In contrast, our approach, only sends 31.37 MB of data in each client-server exchange. We report this information shared (in MB) during each client-server communication in Table 4
Computation Overhead. Our method does not require any extra computation at the client side but introduces an overhead at the server side. This overhead includes the time required to train the generators and perform knowledge distillation. To quantify this overhead, we conducted benchmarking using 2 NVIDIA TITAN RTX GPUs in a 5 client setup, as described in the experiments section. We report results in Table I in terms of overhead time in seconds (s) per round. Our method adds an extra 220 seconds of computational time at the server side per round, in contrast to the 98 seconds introduced by CFed and the 128 seconds incurred by TARGET. It is crucial to emphasize that our method does not impose any additional overhead the client side, unlike CFed and TARGET which incur 68 seconds and 62 seconds per client respectively. In those methods, the client is responsible for learning the current task as well as distilling knowledge from past tasks. By transferring the computation load from the client to the server, we prioritize client-level efficiency. In most practical federated learning scenarios, edge devices have limited computational capacity compared to the server. Our approach prioritizes client-level efficiency, even entails a slight trade-off' in server-level efficiency.
Storage Overhead. As our method operates in stateless FL setup, we do not require clients to maintain any state information additional storage. Our approach requires the server model to store the classifier and prompt components corresponding to the last task model which is used in distillation resulting into storage cost equal to -9.5% of the base encoder model size. Other baselines Rebulli et al. (2017a) incur extra storage costs at the client side equal to the size of entire encoder and classifier i. -86M parameters. Additionally, CFed and TARGET incur costs oquivalent to storing an entire image dataset at both server and individual client levels. We report the storage costs incurred by these methods (in MB) at both server and client side in Table 4 The storage cost does not include the space needed to store the current round model itself. For
11

---

Published in Transactions on Machine Learning Research (09/2024)
CFed, we use the Caltech-256 Griffin et al. (2007) as the surrogate dataset preseribed in Ma et al. 2022 The storage overhead in Table accounts for storing this dataset at both client and server sides.
In summary, our approach attains state-of-the-art performance while imposing lower overheads compared to ezisting methods.
7 Conclusion
In conclusion, we propose HePCo (Heterogeneous Prompt Consolidation) for continual federated learning. We formalize the setting and provide a methodical approach to simulate real-world conditions combining perspectives from continual and federated landscapes. Our method harnesses the prompt learning capabilities of foundation models to facilitate an efficient distillation framework for consolidating heterogeneous clients. By generating pseudo-data in low-dimensional latent space, our approach enables parameter-efficiett and data-free distillation of information from clients to server. We demonstrate the superior performance of method compared to existing srate-of-the-ae methods through series of experiments that emulate challenging real-world scenarios. By requiring clients to share parts of their models, we significantly reduce communication costs and enhance privacy. Importantly, our approach does not impose any additional overheads on the client side, making it highly valuable for real-world deployment.
8 Discussion
Limitations. It is worth noting that prompting-based methods are still relatively new and not extensively studied, making the explainability of these prompts challenging. Therefore, future work should focus on testing the robustness of these methods in diverse setups to ensure their effectiveness in different scenarios. Considering the typical asymmetry in the availability of computational resources across servers and clients, our approach prioritizes client-level efficiency. Yet, the computation overhcad introducod at the server, may be an issue for some use-cases. Although the generation and distillation procedures relatively lightweight, they still rely on availability of server-side compute resources, which may not be universally accessible in all scenarios. Additionally, our approach necessitates clients to use pretrained vision transformers, lcaving open the question of how this framework can be extended to accommodate other architectures. These are interesting avemues for future research.
Broader Impact. The machine learning community is increasingly leaning towards the adoption of large- scale models for various applications. However, updating these models with new data poses significant challenge. Retraining models from scratch each time new data arrives is computationally expensive and can have substantial financial Justus et al. (2018) and environmental Patterson et. al. (2021) Lacoste et al. (2019) implications. Our approach offers solution by enabling incremental learning on new data without the need for complete model retraining, Additionally, our use of prompting techniques allows for significant reductions in communication and local computation costs while enhancing privacy, which is especially critical for on-device edge computing applications.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 2239292
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. ar preprint ar 1rXix:211.004 11004263 2021.
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyconsu Bang, Hyojun Kim, and Tacsup Moon. Ss-il: Separated softmax for incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (TCCV), pp. 844 853, October 2021.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Gaurav Bagwe, Xiaoyong Yuan, Miao Pan, and Lan Zhang, ced-oprompt: Contrastive prompt for rehearsal-free federated continual learning, 2023. URL http:::/aax.v..or///aaaaaa
04889
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung- Lung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218-8227, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datascts with labelldlssributtion-awaee margin loss, 2019.
Nicholas Carlini, Jamic Hayes, Milad Nasr, Matthew Jagielski, Vikash Schwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models, 2023.
Arslan Chaudhry, Marc' Aurelio Ranzalo, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memorics. arXie preprint arXiv:1902.104 2019b.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lec. Dual-teacher free generative replay. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5543-3552 2021.
learning with data-
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class- incremental learning, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint orXiv:2010.11922 2020.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. arXiv preprinl ar arXie:2003.095
2020.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-250 256 Object Category Dataset. Mar 2007.
Tyler Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. Xie preprint or
2019.
Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9769-9776. IEEE, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis distr--diit generalization. ICCV, 2021.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 437-452 2018
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification, 2019.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Yen- Con-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating contimual learning scenarios: A categorization and case for strong baselines. arxiv preprinl or 1ato:1880..288 2018 8.
KJ J Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and Vineeth N Balasubrama- nian. Novel class discovery without forgetting, 2022.
Daniel Justus, John Brennan, Stephen Bonner, and Andrew Stephen McGough. Predicting the computational cost of doep learning models, 2018.
Georgios Kaissis, Alexander Ziller, Jonathan Passorat-Palmbach, Théo Ryffel, Dmitri Usynin, Andrew Trask, lonésio Lima, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al. End-to-end privacy preserving deep learning on multi-institutional medical imaging. Nature Machine Intelligence, 3(6):473 ((6):473-484, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning. 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsker et al. Overcoming calastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning, 2019.
Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se- Young Yun. Preservation of the global knowledge by not-true distillation in federated learning, 2022.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. neural dirichlet process mixture model for task-free continual learning. arXiv preprint orXiv:2001.0068) 2020.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020a.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data, 2020b.
Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2900 2947, 2017.
Tao Lin, Linging Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. ariiv preprint or Xiv: 1ll:1705.0000 705.05550 2017.
Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. Rehearsal-free continual learning over small non-iid batches. In CVPR Workshops, pp. 989 998, 2020.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv: 0.000.00555.. 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for contimual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6470-6479, USA, 2017. Curran Associates Inc. ISBN 978-1-5108-6096-4
14

---

Published in Transactions on Machine Learning Research (09/2024)
Yuhang Ma, Zhongle Xie, Jue Wang, Ke Chen, and Lidan Shou. Contimual federated learning based on knowledge distillation. In Lud De Raedt (ed.), Proceedings of the Thirty- First International Joinl Conference on Artificial Intelligence, IJCA1-22, pp. 2182 2188. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.24963/ (jcai.2022, /303. URL pttps://doi.orgg/ 24963/1jcai ottp:://oo....rr....22///....... 2022/3333 Main Track.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-incremental -tak scenarios. Neural Networks, 116:56-73, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commuiicttion-effcient learning of deep networks from decentralized data, 2023.
Alessio Mora, Trene Tenison, Paolo Bellavista, and Irina Rish. Knowledge distillation for federated learning: a practical guide. arXiv preprint ar XxXx:2211.04742, 2022.
Anh Nguyen, Tuong Do, Minh Tran, Binh X. Nguyen, Chien Duong, Tu Phan, Erman Tjiputra, and Quang D. Tran. Deep federated learning for autonomous driving. In 2022 IEFE Intelligent Vehicles Symposium (TV), pp. 1824 1830, 2022. doi: 1LLL09//VV1197..22228877700
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11321-11322 2019.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation, 2019.
Daiqing Qi, Handong Zhao, and Sheng Li. Better generative replay for continual federated learning, 2023.
Sylvestre- Alvise Rebulli, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in newral information processing systems, 30, 2017a.
Sylvestre-Alvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremen- tal classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR'17, pp. 5533-5542, 2017b.
Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu Galtier, Bennett Landman, Klaus Maier-Heein, Sébastien Ourselin, Micah Sheller, Ronald Summers, Andrew Trask, Daguang Xu, Maximilian Baust, and Manuel Jorge Cardoso. The future of digital health with federated learning. npj Digital Medicine, 3, 12 2020. doi: 10.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of compuler vision, 115(3):211 000:222,,,22211 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsoll. Progressive noural networks. arXiu preprint orXiv:1606.04000 2016.
Felix Sattler, Tim Korjakow, Roman Rischke, and Wojciech Samek. Fedaux: Leveraging unlabeled auxiliary data federated learning. IEFE Transactions on Neural Networks and Learning Systems, 2021.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), A Advances in Newral Information Processing Systems 30, pp. 2990 2999. Curran Associates, Inc., 2017.
15

---

Published in Transactions on Machine Learning Research (09/2024)
James Smith, Yen- Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. Tn Proceedings of the TERE/CVF International Conference on Computer Vision (ICCV), pp. 9374 9384, October 2021.
James Scale Smith, Lconid Karlinsky, Vyshmavi Gutta, Paola Cascanto-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompl: Continual decomposed attention-based prompting for rehearsal-free continual learning. arXiv preprint arXiv:2911.13918 2022. Accepted for
publication at CVPR 2023.
James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning, 2023.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprinl Xix::19040077555 2019.
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. irain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):1-14, 2020.
Johannes von Oswald, Christian Hemning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXit:1906.0000 2019.
Lixu Wang, Chenxi Liu, Junfeng Guo, Jiahua Dong, Xiao Wang, Heng Huang, and Qi Zhu. Federated continual novel class learning, 2023. URL https://arxix: http://aaxxi..ogggbbbb.b//// orgg/bs/////////// 13500
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Y Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Dualprompt: Complementary prompling for rehearsal-free continual learning, 2022a.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jemnifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. Xr preprint arXiv:2204.00 7999 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for contimual learning. In Proceedings of the IEFE/CVE /CFF Conference on Computer Vision and Pattern Recognition, pp. 139 149, 2022c.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning, 2022d.
Ross Wightman. Pytorch image models. httpe:///ttttbbboooorrrgggarrrrr
2019.
Guile Wu, Shaogang Gong, and Pan Li. Striking a balance between stability and plasticity for class-incremental learning. In Proceedings of the IEEE/CVF' International Conference on Computer Vision, pp. 1124 1133, 2021.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8715-8724, 2020.
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang, Federated continual learning with weighted inter-client transfer, 2021.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recogrition, pp. 6982 6991, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In
International Conference on Machine Learning, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lv. Addressing catastrophic forgetting in federated class-continual learning, 2023.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the TEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 10174-10183, 2022a.
Lin Zhangg Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, 2022b.
Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with optimal rales and adaptivity to non-iid data. arXiv preprint urXiv:2005.1143 11 11418, 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning URL
with non-iid data. 2018. doi: 10.48550/
wrwp:///.../bbbb//...... 00582
Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self- supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5871-5880, 2021.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Appendix
A Algorithm
To better illustrate our proposed method, we present a whole picture of the method in Algorithm | The algorithm describes our complete procedure for a global task T", where i [1.
B Experimental Details
Implementation Details. For fair comparison, we use the ViT-B/16 backbone pretrained on Imagenet-1K as the encoder for all methods. We implement our methods in PyTorch and use the PyTorrh Image Models library WWighmman (2019) to obtain pretrained checkpoints. We use NVIDIA A40 GPUs for all experiments. For each result reported in this paper, we calculate the mean and standard deviation over separale runs.
Training Details. For all methods, we use the Adam Kingma & Ba (017)) optimizer with B1 = 0.9 and 82 = 0.999. We resize images to 224 x 224 and normalize to [0,1].
Hyperparameter Search. Following DualPrompt Wang et al. (202bb), we use 20% of the training datasot as our validation data and conduct a hyperparameter search. We tune hyperparameters for both our approach and all competing baselines. We use batch size of 64 for both local and server-side training, determined after searching over values of 16, 32, 64, 128. For our method and the prompting based baselines, we use learning rate of 1c-3, while for basclines that tune the entire model (FedAvg, FodLwF.MCC) we use 5c-5. We 2, 1) Our method employs
search for learning rates among (1e-6, [1e-6,5e-3, 5, 1e--, 1,, 1,, -e 1, a layer fully-con fully-comnected network as the generator. We encode class labels using an embedding matrix of length 64 and concatenate it with a 64-ddimensional noise vector. This dimension was chosen after scarching over 32, 64, 128, 256. While our approach is robust various values, 64 performs best. The generator architecture has input sizes of [128, 256, 1024) per layer, with output size of 768 which is the dimension of the visual query. We opt for a three-layer architecture to maintain lightweight generation and training. We train the generator for 100 epochs using a batch size of 64 and a lcarning rate of le-4 using the Adam optimizer. After a logarithmic scale search, we find a learning rate in the range [5e-5, 1e-4] to provide optimal results.
We fine-tune the server model using a learning rate of 14-4 for 200 epochs. Through grid search, we find that the model is robust to various learning rate and epoch combinations, with these values providing the best average accuracy the validation set. We use a replay ratio of 0.5 for method, which means we mix 50 mheudo-latente correspouding to previous tasks for every 100 pseudo-latents corresponding to the current task. We conduct a search over values like [0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1] and find 0.5 to result into the best average accuracy AN. We observe a stability- trade-off controlled by this hyperparameter with larger values lcading to lower forgetting (Fx) but lower current task accuracies (plaslicily) and smaller values yielding the opposite effect. Through a hyperparameter search within [0, 5] at 0.1 increments, we choose λKL and ÄMSE values to be and 0.1 respectively. We find these values work best across all datasets reported in the paper. Overall, Aey values between 00.6, 3] yield similarly good results, with too low (close to 0) and too high (close to 5) values leading to poor performance.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm HePCo:
Heterogeneous Prompl Consolidation
Task: For each task T" the server is trained through R. communication rounds
Input: Set of C clients with trainable parameters we {Pe, Ke), Pe, pretrained ViT parameters Opt, trainable server parameters w, = {Ps, Kahs S,o, Øs. local epochs E, generator training epochs Eg. distillation epochs Ed
Output: Ws:
Server executes:
Send the pretrained model parameters Pee to clients.
for E {1, R) do
4: for cach client E C do
5: We - W.
6:
Oc 0=
7:
We: th f Client Update(We; 6c)
end for
ggen Train Generator
10: Obtain provisional server model weights ws and @e by averaging client weights.
11: W. Ox f Latent Distillationww,, Os. 0 Bgen)
12: end for
1: Train Generator:
2: for e E Eg do
Cets Calculate Cross Entropy loss in equation
LKL Calculate Kullback-Leibler divergence in equation 4a.
LMSE + Calculate Mean-Squared Error in equation 4b
6: Update 8gen using equation
7: end for
8: return 8gen
Latent Distillation (Ws: the, ggen):
for Eg do
e~ 10,,1)
Sample class labels y randomly from current and past task categories
5: Obtain pseudo-latents = G(e, yy 6 gen) from generator
6:
Obtain distillation target prompts from clients and previous server model
7: Cprompt + Calculate Mean-Squaree Error in equation
Update w.
Calculate cross-entropy loss between sampled class labels y and predictions (x)
10: Update
11: end for
12:
return w..
1: Client Update (We, Φ):
2: Freeze VIT backbone Opt
for e do
Calculate visual query feature q(x) for image x
Obtain final prompt p from ((x) using Pa and K. Insert into selected layers of ViT 0pt.
7: LCE + Calculate cross entropy loss Lce
8:
Update We, &c
end for
10:
return We: de
19


Reviews:
[{"review_id": "GQSNZM47H3", "reviewer": "Reviewer_iEXM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is clearly written and the proposed method shows a strong numerical performance.\n\nWeaknesses: See my comments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the continual federated learning problem, where a server communicates with a set of clients to incrementally learn new concepts over time without sharing or storing any data. The authors study this problem in the context of Foundation Models and explore parameter-efficient approaches to adapt to dynamic distributions while minimizing forgetting. They propose to leverage prompting-based methods and distill data from both the past task label distribution as well as the current task label distribution. Numerically, their proposed approach outperforms both existing methods and contributed baselines by as much as 7\\% while reducing communication costs by only sharing a small set of model parameters. The paper is in general interesting and well-written, but I have the following comments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Comments:\n\n- Definition of $L_{cls}^c$: $\\mathcal{L}_{cls}^c$ is interpreted as the cross-entropy loss between the prediction of local model $c$ given latent $z$ and sampled class label $y$. It is defined for the local model $c$. However, in the right-hand-side of Equation (3), it sums over all local models in $C$. Please define precisely.\n\n- What is $\\mathcal{L}_{dis}$ defined on page 7 line 5?\n\n- In Equation (6), why choosing $\\zeta_{t-1}^y$ as an indicator variable instead of a weighted indicator variable? That is, $L_{prompt}= \\sum_{c\\in C} L_{MSE}^c + w*\\zeta_{t-1}^y L_{MSE}^{t-1}$ where $w$ is a parameter?\n\n- For the ablation study, could you also report the metric of average forgetting?\n\n- In Table 2, $\\beta$, which is an imbalance ratio, is chosen as 0.05 and 0.01. Could you report the experiment results for a wider range of $\\beta$ (e.g., 0 to 1)?\n\n- How do you choose $\\lambda_{KL}$ and $\\lambda_{MSE}$ in the numerical experiments on different datasets?\n\nMinor comments: \n\n-  The sentence below (3) \"where... and sampled class label y\": y should be in math mode $y$. \n\n- The sentence before Equation (5) \"for these these losses\": delete \"these\"", "broader_impact_concerns": "The robustness of the prompting-based methods need to be tested across diverse setups to ensure their effectiveness in different scenarios.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sIqBXXRWeV", "reviewer": "Reviewer_TAtt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* By and large, the proposed method seems to make sense.\n* The authors present some empirical evidence that the proposed method improves on the existing baselines.\n* The paper includes some ablation results pointing to the relative importance of the various parts of the proposed method.\n\n### Weaknesses\n* The paper is a bit hard to follow, and the writing could be improved.\n* The experiments do not seem to include all existing methods as baselines.\n* Some of the claims are either unsupported or their status is hard to assess.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at the problem of continual federated learning (CFL) in a class-incremental cross-device setting, i.e., with many stateless clients and with new classes appearing with new tasks in a sequential fashion. The proposed method is based on client-side prompt-learning, that closely builds on existing CL methods, and a novel server-side knowledge distillation approach to aggregate the heterogeneous client-contributions. The authors empirically show that their approach can improve on existing baselines in several settings for image recognition.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In decreasing order of importance:\n\n1) It is hard for me to get the exact flow of what is happening on the clients and on the server in the proposed method and how exactly these link together, even given Figs 1 & 2. I would propose you add explicit server and client side algorithms to make this clearer.\n\n2) It is generally a bit hard to see which parts are meant as novel contributions and which are simply restating background information from existing methods, as the background is covered here and there in Sec4 while introducing the proposed method. I would propose that you clearly separate the background from the proposed method on the level of sections to make this clear.\n\n3) Please explain why you do not include some existing methods in any of the comparison and discussion (FedWeiT, FedCIL mentioned in Sec2 related work, GAL from Wang et al. 2023, Fed-CPromp from Bagwe et al. 2023; also add Wang et al. and Bagwe et al. to the discussion about existing work).\n\n4) In several places (see pages 1-5, 7, 11) you claim in so many ways that the presented method is privacy-preserving, yet there is no actual definition of privacy given in the paper, either formally or informally. While some of the claims make sense on an intuitive level, e.g., not requiring a server-side replay buffer holding actual client data is better than requiring it, in many places the method is claimed to be privacy-preserving without comparing it to such an obvious baseline (e.g. p2: goal is to \"ensur[e] client privacy\"; I do not see how the proposed method would guarantee privacy in any meaningful sense). I find such claims unsupported, as communicating fewer parameters does not necessarily mean that the method is any more private than a baseline sending more parameters, or that the proposed method could not leak client data in the clear to an adversary.\n\n5) Based on the title and abstract I would assume you look at different foundation models (FM) or at least present a method which is more or less directly applicable to any FM. However, the actual method and all the experiments look solely at visual transformer (ViT) models, and seem to rely on having attention layers in the model. I would propose you change the title and abstract to better fit the actual work described in the main paper.\n\n6) In several places there hard-to-verify claims (e.g. , achieving high accuracy in describing generator on p6, efficiently fine-tuning at the start of Sec4.4). Please clarify in which sense these claims are true, and point to the supporting evidence in the text.\n\n7) How sensitive the results are to the hyperparameter tuning? Please state (at least in the appendix) if you have done some other hyperparameter search (e.g. as currently stated for lr), and whether you have also tuned all the baseline hyperparams.\n\n8) Please include some deviation measure in Fig3.\n\n9) In the results (e.g. Tables 1 & 2), consider (half-)highlighting methods that overlap with the best mean given the reported deviation.\n\n10) abstract: \"understudied problem\": there are plenty of papers on the topic, so how exactly is this understudied?\n\n11) p1: claiming that standard FL assumes iid data is misleading; as you state next, the problem of heterogeneous data is a well-studied in existing FL literature. Please refine this claim.\n\n12) p2, related to contributions: please also state how you measure communication costs (eg, number of parameters sent vs number of communication rounds) and computational efficiency when claiming that the proposed method is efficient in these respects.\n\n13) p4: \"Formalizing the setting in this manner enables us to methodically vary the parameters, thereby simulating a heterogeneous setting that closely resembles the complexities encountered in the real world\": do you actually have some real-world data or settings that the data splits you use closely mimic? If not, please do not claim close resemblance.\n\n14) Table 3: $F_N$ is not included in the results although mentioned in the caption and in the text.\n\n### References:\n\nBagwe et al. 2023: Fed-CPrompt: Contrastive Prompt for Rehearsal-Free Federated Continual Learning.\n\nWang et al. 2023: Federated Continual Novel Class Learning.", "broader_impact_concerns": "I have no broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rThICesckn", "reviewer": "Reviewer_aaiv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nS1: Transferring the client prompts and classifiers is more efficient than transferring the client models.\n\nS2: Generating pseudo images in the latent space is cheaper than generating the original images.\n\nS3: The experiment results show that the proposed method achieves good accuracy.\n\n\nWeakness\n\nW1: It is unclear why the proposed method achieves better data privacy than methods that transfer client models. I understand that if the server has client models, it can use model inversion techniques to reconstruct the original images. In the proposed method, the server can also use model inversion if it knows the frozen vision encoder. This is necessary because the server needs to know the vision encoder for model serving. \n\nW2: The efficiency advantage is analyzed over many places but not properly established. It will be better if the authors can provide concrete numbers (e.g., via a table) for the communication and computation volume of the clients in the training process. \n\nW3: The discussions of related work can be enhanced, especially the continual federated learning part. It is difficult to understand why existing methods are not efficient from the current discussions. For instance, the authors say “FedWeiT Yoon et al. (2021) aims to learn better client models by minimizing interference across client weights. FedWeiT incurs considerable overheads in terms of communication, computation and storage.” However, where do these overheads come from? The authors should describe how FedWeiT works in addition to the high-level ideas.\n\nW4: The method and experiment parts consider image datasets. The paper can be enhanced if the authors can discuss how the proposed methods may be used for other types of data (e.g., tabular).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies continual learning in the setting of federated learning. On the client side, it uses a frozen vision encoder and learns prompts and classifiers. On the server side, it reconstructs images in the latent space and fine-tunes the prompts and classifier according to the latent images. The proposed method is shown to achieve higher accuracy than existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adress all comments in the weakness part.\n\nR1: Clarify the privacy benefits of the proposed method over existing methods.\n\nR2: Enhance the efficiency analysis with concrete experiment results.\n\nR3: Clarify how existing researches work such that readers can understand their limitations.\n\nR4: Discuss whether and how the proposed method can work with other types of data.", "broader_impact_concerns": "I do not see any ethical concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the overall positive reviews, the authors' comprehensive response to the reviewers' comments, and the technical merits of the proposed method, I recommend accepting this paper. The authors have well addressed the major weaknesses and provided a clear and detailed presentation of their work.
Decision: Accept (poster)