Federated Continual Learning via Orchestrating Multi-Scale Expertise
Authors: Xiaoyang Yi
Yang Liu
Binhan Yang
Jian Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Continual Learning
Primary area: deep_learning
TL;DR: We propose MultiFCL, a novel FCL framework that fine-tunes PTMs to adapt to FCL while preserving their strong generalization capabilities.

Abstract: Federated continual learning (FCL) aims to maintain the model's performance on old tasks (i.e., stability) while enhancing its ability to acquire knowledge from current tasks (i.e., plasticity). With the development of pre-trained models (PTMs), fine-tuning PTMs on clients has become a promising approach to leveraging their extensive knowledge in FCL. In this paper, we propose MultiFCL, a novel FCL framework that fine-tunes PTMs to adapt to FCL while preserving their strong generalization capabilities. Specifically, to ensure the stability, MultiFCL introduces lightweight adapters for task adaption, which are subsequently frozen to prevent catastrophic forgetting. Moreover, by utilizing the semantic features of old tasks, MultiFCL performs multi-modal initialization of new task class prototypes. To enhance the plasticity, MultiFCL employs a multi-expert training mechanism that integrates multi-scale feature learning with multi-teacher dynamic self-distillation. Through intra-client and inter-client expert communication, MultiFCL facilitates cross-task and cross-client knowledge fusion. Experimental results demonstrate that MultiFCL achieves state-of-the-art performance across multiple datasets and settings, showcasing its effectiveness in FCL scenarios.

Federated Continual Learning via Orchestrating Multi-Scale Expertise
Xiaoyang Yi1.3,4
Yang Liu? Binhan Yang²
Jian Zhang 1.23,44 1,2,444*
College of Cryptology and Cyber Science, Nankai University, China College of Computer Science, Nankai University, China Tianjin Key Laboratory of Network and Data Security Technology, Tianjin, China Key Laboratory of Data and Intelligent System Security, Ministry of Education, Tianjin, China [xiaoyangyi, liuyang99, yangbinhan| | @ @mail.nan www.kai.eu..co zhang. jian @ nankaii.edu.om edu.cn
Abstract
Federated continual learning (FCL) aims maintain the model's performance old tasks (i.e., stability) while enhancing its ability to acquire knowledge from current tasks (i.e., plasticity). With the development of pre-trained models (PTMs), fine-tuning PTMs on clients has become promising approach to leveraging their extensive knowledge FCL. In this paper, we propose MultiFCL, novel FCL framework that fine-tunes PTMs to adapt to FCL while preserving their strong gen- eralization capabilities. Specifically, to ensure the stability, MultiFCL introduces lightweight adapters for task adaption, which are subsequently frozen to prevent catastrophic forgetting. Moreover, by utilizing the semantic features of old tasks, MultiFCI. performs multi-modal initialization of new task class prototypes. To enhance the plasticity, MultiFCL. employs multi-expert training mechanism that integrates multi-scale feature learning with multi-teachs dynamic self aff-distillation. Through intra-client and inter-client expert communication, MultiFCL facilitates cross-task and cross-client knowledge fusion. Experimental results demonstrate that MultiFCL achieves state-of-the-ar performance across multiple datasets and settings, showcasing its effectiveness in FCL scenarios.
Introduction
Federated learning (FL) is distributed machine learning paradigm that allows clients to collabora-
tively train models without sharing sensitive data (McMahan et al., 2017). In recent years, FL has
gained significant attention across various domains, such as healthcare (Yang et al. 2021 Ogier du
Terrail et al. 2022 Chen et al. 2024b Xie et al. (022)) and the internet of things (IoT) (Nguyen
al. 2023) Cui al. 2022 Yu et al. 2024a Jia et al. 2024). Traditional Fl. approaches typically assume that client data distributions do not change over time. However, this assumption is unrealistic
in practical applications, client data usually evolves continuously (Ma et al. 2022] Lu et. 2024).
For example, healthcare models must adapt to emerging diseases and their variants (Babakniya et al. 2023), while IoT models need to keep pace with changes user behaviors (Zhang al. 2024). When there are such shifts in data distributions, directly retraining existing models on new data can lead
catastrophic forgetting (Smith al. 2023b Li et al. 2024 Yu et al. 2024b), which significantly impairs the model's 's performance on previously learned tasks.
To address this, federated continual learning (FCL) integrates continual learning (CL) techniques into FL. (Yoon et al. 2021), aiming to seek a balance hetween stability (i.e., maintaining performance on
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

old tasks) and plasticity (i.e., acquiring knowledge from new tasks). Unlike traditional CL methods that rely on extensive data retention to perform experience replay, many FCL approaches instead opt for generative replay (Qi et al. 2023 Wuerkaixi et al. 2024 Tran et al. 2024) model decomposition (Bakman et al., 2024), due to privacy concerns and resource limitations. These approaches train models from scratch, trying to continuously absorb knowledge from new tasks while reviewing knowledge from previous ones (Zhou et al. 2024b). However, the heterogeneity among client data presents challenges in maintaining both stability and plasticity during the process of training a model from scratch. Fortunately, the emergence of pre-trained models (PTMs) has offered a promising alternative. owing to their remarkable generalization capabilities (Dosovitskiy et al., [2021 Radford ct al. 2021). ). Trained on large-scale datasets, PTMs are better equipped to acquire optimal knowledge compared to incrementally trained model, thereby significantly alleviating the learning burden (Zhou
et al. 2024c).
Although PTMs exhibit strong generalization capabilities, how to fine-tune them for downstream tasks still dilemma: fully fine-tuning risks diminishing their generalizable features, while freezing the backbone can prevent the integration of downstream information (Zhou et al., 2024a). To this end, some FCL approaches incorporate trainable prompt parameters into PTMs, training prompts locally on clients and updating the global prompt pool on the server by aggregating local prompt pools (Halbe et al. 2024| Yu et al. 2024c). A query mechanism is then employed to dynamically select subset of prompts from the global prompt pool, guiding PTMs in adapting to different data distributions (Wang et al. 2022 Smith et al. 2023a). By keeping the pre-trate pre-trained weights frozen during this process, these prompt-basee based approaches effectively preserve the generalizability of PTMs while allowing task-specific adaptions. However, prompt selection often focuses on specific subsets, limiting the adaptability to broader data distributions (Moon et al. 2023 Moreover, new data accumulates, the prompt pool size expands, leading to increased communication overhead, whereas restricting the prompt pool size constrains its representational capacity (Piao et al.. 2024).
Inspired by these insights, we propose a novel FCL framework, MultiFCL, which effectively fine- tunes PTMs while preserving their strong generalization capabilities to adapt to FCL. Specifically, to maintain the stability of PTMs, MultiFCL employs lightweight adapters to quickly adapt PTMs to downstream image classification tasks, and then freezzes them to prevent catastrophic forgetting. Furthermore, MultiFCL leverages semantic features from old tasks to perform multi-modal initializa- tion of new class prototypes, thereby incorporating prior knowledge. To enhance the plasticity of PTMs, MultiFCL introduces a multi-expert training mechanism, which combines multi-scale feature learning and multi-teacher dynamic self-distillation to strengthen feature representation for current tasks and achieve effective knowledge integration. Through both intra-client and inter-client expert communication, MultiFCL facilitates cross-task and cross-client knowledge fusion, addressing the dynamic and heterogeneous data distribution challenges inherent in FCL.
Our contributions are as follows:
We propose MultiFCL, a novel FCL framework that preserves the strong generalization capabilities of PTMs while fine-tuning them, enabling PTMs to maintain both stability and plasticity.
To ensure stability, we employ lightweight adapters for task-specific fine-tuning and utilize multi- modal initialization of new task class prototypes based on semantic features from old tasks.
To enhance plasticity, we design multi-expert training method that combines multi-scale feature learning and multi-teacher dynamic self-distillation, achieving effective knowledge fusion.
Experimental results demonstrate that MultiFCL achieves state-of-the-art performance across multiple datasets and configurations, highlighting its effectiveness and the robustness of PTMs.
2 Related Work
FCL can be broadly categorized into inseemnnreettal learning (TIL) and class incremetal learning (CIL). The former focuses on learning new tasks sequentially, while the latter involves learning classes in each task incrementally. Specifically, Federated TIL was first introduced in FedWelT (Yoon (et al. 2021]. To prevent forgetting old tasks while learning new ones, FedWelT decomposes network weights into global parameters and sparse task-specific parameters, each client selectively transfers knowledge from others by their task-specific parameters. CFeD (Ma et al.) 2022) employs continual
distillation using unlabeled proxy dataset on both clients and the server, assigning different learning

---

1 Leal training
Adapted Image Encoder
Image Blocks 1 *
Imuge Blocks *
Image Blocks M *
Multisdaispier Fine- Taming
Adapters
Adapters
Adapters
Tabel
* Text
Avóm
lee
quary
key
quen
key
quers
hgg
Encoder
value
Protunye inttializaiion
Frozen
Expert
Expert
Expert M
Fine-tuned
KL
Liceperte
Frozen After
HI. Multi-Sall
XL
The First Task
Feature Leaming
W. Muitt Teacher
Ld
Dynamic Self- Diiillatiooonn
2 Global Update
Client I
Client
Client K
Server
Upload
Expert
Expert
Expert
Expert
Download
Figure 1: The overall framework of MultFCLL. MultiFCL employs adapters to fine-tune the PTM and leverages the semantic features old tasks to initialize new class prototypes. Then, MultiFCL establishes multiple experts, employing feature learning loss Lespens and the multi-teacher dynamic self-distillation Cyeli-ka to transfer knowledge to the final expert.
objectives to clients for learning new tasks and revisiting old ones. AF-FCL (Wuerkaixi al. 2024 selectively leverages prior knowledge in the federated network and uses probabilistic framework based on normalized flow models to quantify the reliability of this prior knowledge. FOT (Bakman
al. 2024) extracts global input subspaces for old tasks at each layer and modifies the aggregated updates for new tasks to ensure orthogonality with these global subspaces. Powder (Piao al. 2024)
introduces novel prompt generation and aggregation method to effectively facilitate knowledge transfer encapsulated
prompts across sequential tasks and clients. FedMGP (Yu et al. 2024c)
employs coarse- grained global prompts for efficient shared knowledge transfer and fine-graine ain-ggrined local prompts for personalized learning. mitigating spatial and temporal forgetting.
Class-increment FCL., also known as federated class-incremental learning (FCIL), was first intro- duced by GLFC (Dong al., 2022). To mitigate forgetting of old classes while learning new ones, GLFC proposes Class-aware gradient compensation loss and class-semantic relation distillation loss balance old-class retention. FedCIL (Qi al. 2023) employs ACGAN (Odena et al. 2017) to synthesize data from old distributions for replay, alleviating catastrophic forgetting. Similarly, MFCL (Babakniya et al. 2023) uses a generative model to synthesize samples from old data distributions and trains on the server using data-free method. FedET (Liu et al. 2023) introduces a small Enhancer module to absorb and communicate new knowledge and proposes an enhancer distillation method to address the imbalance between old and new knowledge. LANDER (Tran al. 2024) uses label text embeddings as anchors to constrain the feature embeddings of corresponding training samples, generating meaningful samples for replay. FedCBB (Yu et al.) 2024b) constructs class-specific binary classifier for each class, extracting prior knowledge from the global model into multiple local models for selective knowledge fusion. LGA (Dong et al.) (024) introduces class-balance balanced gradient adaptive compensation loss and a glass-gradient-indie semantic distillation loss to balance the heterogeneous forgetting rates of old classes.
3 Method
The MultiFCL framework is shown in Figure [ Specifically, to adapt the powerful PTM to down- stream image classification tasks, MultiFCL introduces adapters into the PTM for task-specific fine-tuning (Section 3.1). These adapters remain frozen during the training process of subsequent tasks, effectively mitigating catastrophic forgetting. By integrating semantic features from old tasks with those from new tasks, MultiFCL performs multi-modal initialization of new task prototypes (Sec-

---

which enhances the representation of new class features while preserving old knowledge. During the training process, multi-scale features and new task prototypes are used for multi-scale feature learning to ensure the plasticity (Section 3.3) Additionally, MultiFCI. employs multi-teacher self-distillation to transfer knowledge to the final expert, and dynamically adjusts the distillation loss using uncertainty- weigttee perception in the knowledge transfer process (Section 3.4..
tion 3.2).
3.1 Multi-Adapter Fine-Tuning
In FCL, each client & (1 < K) trains a local model using its local dataset DJ =
Adapter
{I,,, y:)) to sequentially deal with tasks (Ti, T2...., ,, where n? the dataset size for task T on client k. For the same task T, data distributions across clients are non non- -oniideependent and identically distributed (Non-IID). Mcan- while, within the same client k. the class sets across different tasks are disjoint. The overall optimization objective for clients is two fold: (1)
MLP
W down
x(-)
W
Fine-tuned
Frozen
minimizing the loss function [(-) on the current task's ' dataset to achieve high performance (i.e., plasticity), (2) and retaining knowledge from previously learned tasks' datasets to mitigate catastrophic forgetting (i.e., stability).
Figure 2: The structure of a adapter in the PTM.
To bridge the domain knowledge gap between the PTM and downstream image classification tasks, we introduce trainable adapter parameters W'k into the frozen PTM, enabling adaptation to FCL tasks for each client k. Adapter introduces small neural network modules independently into various layers of the model, separate from its main structure. Only these adapter parameters are updated during fine-tuning. In contrast, LoRA (Hu et al. 2022) introduces low-rank matrices into the model's weight matrices. These low-rank matrices act as modifications applied to the original weight matrices, adjusting them during computation. Specifically, (Radford et al., (2021)) as the backbone model, which consists of multiple repeated blocks with inserted into each layer of the image encoder facilitate fine-tuning. As illustrated in Figure [ each adapter is a bottleneck module, which is composed of a down-sampling layer W down for reducing feature dimensions, nonlinear activation function (C() (c.g.. ReLU (Glorot ct al. 2011)), and an up-sampling layer Wap up for restoring the original dimensions (Chen et al. 2024a).
employ transformer-basee basedd PTM (e.g.. CLIP
several layers. Adapters
This process can be formulated as:
h' = h + 6(hW down W up
where h and h' are the input and output feature representations through adapter, respectively.
Since the local dataset of client & constantly evolves with changing tasks, training adapters for each task can easily lead to catastrophic forgetting. To fully preserve the generalization ability of the PTM, we fine-tune adapters only on the first task and then freeze them to prevent excessive updates, as follows:
(x)x) L()(),, Di, wx)
Here, f()) represents the original PTM. f(() is the PTM with adapters, and D is the dataset for the first task of client k: By fine-tuning with adapters on the first task, the PTM can rapidly adapt to downstream image classification tasks.
During each communication round i of the first task training, each client uploads its locally trained adapters to the server for aggregation, enabling the model to acquire more comprehensive knowledge as follows:
K
X
(3)
A=
where w² denotes the global adapters, and |X is the total number of samples owned by client k for the first task. Once the first task is completed, all adapters will be frozen and no longer updated during model training, preventing performance from degrading due to excessive fine-tuning.

---

3.2 Multi-Modal Prototype Initialization
Adapter fine-tuning enables the PTM to quickly adapt to downstream image classification tasks, while for classification tasks, classifier is also needed to ensure its generalizability. Since the adapter is only trained in the first task, focuses more on the knowledge related to that task and may overlook some irrelevant high-level features (Zhou et al. (024b). Therefore, instead of using a conventional linear classifier for classification, we use prototypes as weights of the classifier, performing classification by measuring the distance between class prototypes and sample features. Specifically, we continuously update the optimal class prototype pd (c E ') through training, as follows:
Pio)
Pini
(- (x(
P
P
Vj (1, .n
(5)
2
where Pinit is the initialization prototype of class c, P is the current prototype from class c to sample x5. and n! is the number of class in the task
common initialization method is random initialization, which makes the training of the classifier independent learning around the task. For subsequent tasks, it may lead to the classifier of the new task lacking knowledge from old tasks, resulting in lower discriminability (Kurniawan et al. (002)). To address this, the bias-adjustment heuristic (Kahneman et al.) 1982) is proposed, which uses available related knowledge to estimate unknowns. Therefore, we introduce relevant semantic features from old tasks as available knowledge to estimate the new class prototypes, and use multi-modal fusion to form their initialized prototypes.
In order to fuse semantic features with sample features of the new class for initializing the new class prototype, we first calculate the similarity between the feature of the first sample of the new class and the semantic features Tau of the old task labels to establish the basic relationship assumption between the old and new classes, as follows:
f'(11) fiea (Ca) (1 fl (CTua)]
sim (J'( Stear (Tau))
where sim(-) is similarity calculation function, and Steat (-) the text encoder of the PTM.
Subsequently, the most similar label semantic features are selected and merged with the sample features of the new class through a cross-attention mechanism to initialize the new class prototype. thereby incorporating prior implicit bias as follows:
Piit Attention (f'xz1), Steee (Gam Pols
where Sim and Paaa is the semantic feature and the class prototype of the old task that is most similar to the current class, respectively. Attention(-) is the cross- toos-attenntio. The initialized prototypes obtained in this way can enhance the feature representation of the new class while preventing the forgetting of old class knowledge, thereby ensuring the stability of the model.
3.3 Multi-Scale Feature Learning
While ensuring the stability of the model, it is also crucial to maintain its plasticity, which refers to the model's ability to acquire knowledge from the current task. Research (Jung et al. (202)) has shown that multi-scale features can help the model distinguish between different samples by providing multi-level information, ranging from shallow to deep knowledge. Therefore, we divide the PTM into M modules, each consisting of multiple blocks, which can be treated as experts with different levels of knowledge.
Specifically, the feature output from the last layer of each module serves as the input for the next module. Additionally, the features output by these modules will be used to initialize and update the new task class prototypes, as described in Section 3.2 By utilizing these prototypes, the output features at different scales from each module are continuously trained to move closer to their corresponding class centers, allowing the experts to effectively consolidate the class knowledge
5

---

of each new task. Subsequently. the experts optimize the acquired knowledge through mutual communication, with the optimization objective as:
M Σ
expprts
Σ Z,ET
yi log Pa
(8)
m==1
where .M is the number of experts, and Pa, m represents the prediction probability that sample a; belongs to label Yi expert m. Through multi-scale feature learning, the multi-level experts in clients able make decisions from different perspectives the same sample, thus capturing the diversity and complexity of the sample more comprehensively. As the data distribution continues to evolve, clients become more flexible in adapting to tasks.
In addition the collaboration of experts within each client to jointly classify samples, coordination and communication between clients are also necessary. To achieve this, during each communication round I, each client uploads the class prototypes obtained after local training to the server for aggregation as follows:
D
c.t Pk.m
where p represents the global class prototype, 1 denotes the total samples of class c in client k and Pk,m is the class prototype in expert of client k. As result, the server collects the multi-scale prototypes from different clients form global class prototypes, which are then sent back to the clients as classifier weights for each module in the next round of training. By continuously sharing and updating class prototypes, clients are able to better learn representative multi-scale knowledge during the federated learning process. This provides more precise task guidance for subsequent feature learning.
3.4 Multi-Teacher Dynamic Seff-Distillation
Multi-scale feature learning enables better predictions by allowing different experts to make judg- on the samples from different levels of knowledge (Aljundi al.] (017). However, using multiple experts to generate predictions introduces additional computational overhead, which becomes particularly redundant during testing (Yan et al.) 2024). Therefore, we propose novel delf-distillation method that transfers knowledge from multi-level experts to the final high-level expert. Among them, the former experts act as teachers, while the latter serves the learner to integrate the knowledge from all the experts, as follows:
Ceell-ad Σ (o( a (Ts,naa)
(10)
where KL(-) the Kullback-Leibler divergence function, and (+-) represents the softmax func- tion. Moreover, Zi,m and Ii,final denote the output of expert m and the final expert, respectively.
calculates the distance between the output of each expert and the output of the final expert, thereby consolidating the knowledge from multi-
14
dist-train
12
dist-test
level experts into the final expert and reducing the number of experts that need to be computed during testing. During training, the distillation loss encourages the final expert to learn from the knowledge of all the experts, including the fine ine-grained features from lower-level experts and the abstract decisions from higher-level ex- perts. In this way, the final expert is able to independently perform the task while avoiding excessive reliance on the features from the final layer, thereby obtaining more comprehensive decision- masiing capabilities.
Distance
8
2-
10
12
14
Block
Figure 3: The distance between feature outputs at each block.

---

During self-distillation, the output quality of different experts may vary. Figure B shows the distance between the feature outputs at each block of the PTM, with a sigmificant difference between deep and shallow features. Simply adding the distance between each expert's output and that of the final expert does not fully reflect each expert's contribution to the task, as it tends to emphasize shallow knowledge. Therefore, we propose an umcertainty-weighting method to dynamically adjust the importance of each expert in the self-distillation process.
Specifically, the output quality of each expert is measured not only by the distance between its output and the final expert's 's output but also by considering the uncertainty of its output features. To quantify uncertainty. we use the entropy of the output from each expert. A higher entropy indicates that the expert's output more ambiguous or uncertain, and its contribution should be correspondingly reduced. Conversely, an expert with lower entropy represents more certain outputs, which can provide more stable knowledge, and thus should be assigned higher weight, as follows:
Aexpert
(11)
KL (xxport)))) o (Firfinaa) +
where is small positive constant used to prevent the denominator from becoming zero and to avoid the weights from growing infinitely large.
To ensure fairness among the weights of different experts, we normalize these weights as follows:
Aexport
expprt
(12)
Σ M mm=1 Am
By suppressing the weights of shallow experts with high uncertainty and enhancing the weights of deep experts with low uncertainty, the knowledge from experts various layers can be more effectively integrated. It not only prevents excessive interference from shallow knowledge during the self-distillation process, but also ensures that deep knowledge is fully utilized during learning. thereby enhancing the plasticity of the model.
Ultimately, the overall loss with multi-scale feature learning and multi-teacher dynamic self- distillation is as follows:
L Cepprts + Nexpert Caelf-kd
(13)
Through cross-task and cross-client modll knowledge fusion, MultiFCL effectively handles changes in data distribution in the FCL setting, preventing catastrophic forgetting while maintaining the powerful capabilities of the PTM.
4 Experiments
4.1 Experiment Setting
We evaluate CIL and TIL experiments using CLIP (Radford et al. 202[) as our base model, with the image encoder being ViT (Dosovitskiy et al. 2021). In addition, we use CIFARI00 100 (Krizhevsky & Hinton 2009), TinylmageNet (mnmoustafa & 2017), ImageNet-R (Hendryeks et al.) 2021). and CUB-200 2011 (Wah et al. 2011). which are not learned by CLIP during pre- tre-training. We divide all datasets into 5 and 10 tasks for incremental learning, using accuracy in the last task (Last), average accuracy in all tasks (Avg.), and the forgetting rate F as the metric for evaluation, where F is defined as the difference in accuracy between the first task and the final task. We present partial results for CIL on CIFAR100, with the remaining experiments in Appendix A Following the setting of existing FCL methods, we employ two different data partition strategies, including distribution- and quality- -based partitioning, where the former fixes the number of samples per class and specifies the number of classes owned by the client through a, while the latter uses 3 to specify the degree of data heterogeneity based the Dirichlet distribution. Each partition is further subdivided into three scenarios, with specific settings provided in Appendix B
partitioning
To demonstrate the capabilities of PTMs, we compare ViT and CLIP with FL implementations of several FCL methods such as MFCL (Babakniya et al. 2023), LANDER (Tran et al. 2024) and Fe YYu et al. 2024b). Furthermore, to validate the effectiveness of MultiFCI., we compare it with paM--based FCI. methods such as FedMGP (Yu et al.) 2024c) and PoR (Guo al. (024).
which also use VIT. We also replace the base models of MFCL and PiLoRA with CLIP to maintain
7

---

Table 1: Main results of performance comparison CIFARI00 with 10 tasks in distribution-based partitioning, where -A means using CLIP as base model. The best results are shown in bold.
B =
0.5
B = 0.1
B = 0.05
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
ViT
ICLR 2021
65.0
68.2
22.9
63.3
66.7
23.7
60.3
64.4
24.2
CLIP
ICML 2021
65.0
68.2
25.6
63.3
66.7
27.8
60.3
64.4
28.5
MFCL
NeurIPS 2023
32.9
44.1
25.9
31,4
43.2
26.4
30.7
42.8
28.8
MFCL-A
NeurIPS 2023
52.8
61.3
31.9
52.1
60.4
32.8
51.9
59.1
33.2
FedCBC
MM 2024
17.6
22.9
34.7
17.4
23.1
35.3
15.9
21.0
36.6
LANDER
CVPR 2024
34.8
48.5
27.6
33.4
45.2
28.5
32.1
43.5
30.5
FedMGP
KDD 2024
23.4
33.1
34.9
21.8
32.7
35.6
20.9
32.4
36.9
PiLoRA
ECCV 2024
69.6
72.1
26.7
63.7
65.4
27.8
63.0
68.0
29.4
PiLoRA-A
ECCV 2024
68.7
70.3
26.6
63.8
66.2
28.0
63.0
67.6
29.8
MultiFCL
This Paper
70.7
76.1
16.4
68.9
74.6
18.5
67.9
72.8
17.5
Table 2: Main results of performance comparison CIFARIO0 with 10 tasks in quality-based partitioning, where -A means using CLIP as base model. The best results are shown in bold.
6
4
= 2
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
ViT
ICLR 2021
64.6
67.8
20.9
63.5
66.4
22.0
63.0
66.5
21.6
CLIP
ICML 2021
64.3
67.7
21.4
63.6
66.7
21.6
62.7
64.4
22.5
MFCL
NeurIPS 2023
30.4
39.8
33.6
24.6
33.1
34.5
15.9
21.2
35.9
MFCL-A
NeurIPS 2023
49.1
57.9
29.4
48.7
56.2
29.9
47.1
55.3
31.2
FedCBC
MM 2024
17.5
21.8
27.4
15.5
20.6
27.9
15.1
19.5
28.5
LANDER
CVPR 2024
20.3
26.7
28.9
19.8
24.8
28.6
18.7
24.4
29.3
FedMGP
KDD 2024
19.8
23.5
30.3
13.6
17.3
31.6
12.6
16.5
32.5
PiLoRA
ECCV 2024
70.5
74.8
22.8
63.8
68.2
24.4
63.6
69.5
25.7
PiLoRA-A
ECCV 2024
69.8
73.2
23.4
64.6
69.2
24.6
64.5
70.1
25.3
MultiFCL
This Paper
71.0
76.1
15.9
70.3
75.3
15.9
69.7
75.2
17.1
consistency with our base model for comparison. Specifically, we set up 10 clients, with each client owning 4 experts and performing epochs of local training. For non-PTM methods, use learning rate le-2 and conduct 100 communication rounds per task. For PTM methods, we use a learning rate of le-5 and perform communication rounds per task.
4.2 Main Results
We compare MultiFCL with other baselines, the results are shown in Table [ad Table For FCL methods trained from scratch, MFCL and LANDER use generators deployed on the server to synthesize data and apply knowledge distillation transfer knowledge to the global model, yet without guidance from real samples the generators struggle to produce high-quality data. FedCBC builds a binary classifier for each class and performs selective knowledge fusion between local and global models an attempt to balance new and old knowledge, yet these classifiers focus only single-class decision boundaries and cannot capture the complex semantic relationships among multiple classes. In addition, their accuracy is much lower than that of pre-trained ViT and CLIP, which demonstrates the powerful generalization ability of PTMs.
For PTM-based FCL methods, FedMGP has lower accuracy under quality-based -bssed data partitioning due to difficulties in finding task similarities. Its poor performance stems from the limitations of its prompt pool mechanism in FCL., especially when tasks differ greatly. Furthermore, its data split differs substantially from ours because the first task contains more classes giving the model an opportunity to overfit early categories, so performs poorly in our experiments. Pil PiL learns independent low-rank LoRA parameters for each task and adds regularization to the loss to mitigate

---

forgetting, but it lacks multi-scale cross-task knowledge fusion and dynamic integration mechanism, its forgetting rate reaches 29.4% under distribution heterogeneity (3 0.05) and its accuracy drops to only 63.6% under quality heterogeneity (a 2).
Meanwhile, also replace the base model in MFCL and PiLoRA with CLIP create MFCL-CLIP and FLoRA-CLIP. CLIP's strong alignment of vision languages improves the accuracy of MFCL,it still relies on the generator synthesize old samples of limited quality. And PiLoRA-CLIP still lacks cross-task knowledge integration. In contrast, direct fine-tuning of VIT and CLIP does not include any anti- forgettingg design, whereas MultiFCL achieve sustained and robust performance gains across all settings and experimental results confirm its effectiveness and expressiveness.
4.3 Ablation Study
We perform the ablation study of MultiFCL CIFAR100 with 10 Tasks, the results shown in Table 3 The remaining experiments can be found in the appendix. Specifically, when no modules added, the results correspond to the PTM itself, which establishes clear baseline for comparison. When multiple adapters are added for fine-tuning on the first task, the average accuracy increases, demonstrating that task-specific adaptation allows the PTM to quickly adapt downstream image classification tasks and exhibit its strong generalization ability.
When multi-modal features added to initialize prototypes, MultiFCL supplements old class knowledge, thereby mitigating catastrophic forgetting in the current task. This addition yields a further gain in accuracy, confirming the benefit of richer prototype representations. Furthermore, when multi-experts are incorporated for training using Lexpents each expert specializes in a different scale of features and together provide multi-scale knowledge, helping the model better distinguish differences between samples, resulting in higher accuracy. After incorporating multi-teacher self-distillation LuatEdd MultiFCL transfers knowledge from all experts the final expert, reducing computational overhead during testing while consolidating expert knowledge, ultimately improving classification accuracy. The cumulative results of these experiments validate the design of MultiFCL and illustrate how each component contributes to stable and effective continual learning under federated and heterogeneous conditions.
Table Ablation results of performance comparison on CIFARIO0 with tasks. The best results are shown in bold.
Adapter
Init.
Lexports
Leetf-kd
6
4
2
8 0.5
8 0.1
3 0.05
64.3
63.5
63.0
65.0
63.3
60.3
67.6
64.9
65.2
66.0
64.5
63.1
68.5
67.0
66.9
67.1
65.3
64.7
70.1
68.7
68.6
68.9
67.8
66.3
71.0
70.3
69.7
70.0
68.9
67.9
4.4 Adapter Frozen Study
Figure shows the experimental results for freezing adapters versus keeping them trainable on CIFAR-I00 with 3 0.05 and = 2. When adapters continue to be trained in later tasks, the model's performance drops rapidly because prolonged training leads to catastrophic forgetting, which undermines generalization and compromises plasticity. Furthermore, due to client data heterogeneity, the adapters tend to drift during training. resulting in a severe loss of plasticity. Under these conditions of extreme heterogeneity and limited class samples, the model experiences pronounced forgetting as adapter updates quickly diverge from the task's optimal direction, causing the PTM's strong generalization ability to be lost. Therefore, freezing the adapters after the first task not only reduces computational cost and communication overhead but also effectively preserves the model's generalization capacity, facilitating the transfer of previous knowledge to tasks.
4.5
Expert Study
We use uniform hierarchical strategy to divide the 12 Transformer layers into modules, each containing 3 layers that correspond to shallow texture features, mid-level semantic features, and

---

100
100
85
86.8
80.4
82.4
84.0
72.7
72.2
71.8
70.8
69.3
69.0
68.5 67.9
67.9
84.1
76.3
75.5
73.8
73.5
72.1
71.4
70.6
76.3
Accuracy (%)
Accuracy (%)
60
60
65.3
58.8
53.1
40
40
45.6
43.0
40.7
33.4 29.5 28.6
35.9
32.3
29.9 27.6
20-
Non-Frozen
23.1 21.2 19.6
20-
Non-Frozen
Frozen
17.4
Frozen
0
0
5
5
(a) 10 tasks with
tasks with B 0.05
Figure 4: Adapter frozen results in the last task on CIFARI00, the x-axis represents which task it is.
Table 4: Expert number results of performance comparison ImageNet-R with 10 Tasks. The best results are shown in bold.
8 = 0.5
β = 0.1
B = 0.05
Last
Avg.
F
Last
Avg.
JF
Last
Avg.
F
12 Experts
72.5
78.3
14.7
72.5
78.2
15.3
72.2
78.1
15.8
Experts
73.2
78.7
14.3
73.2
78.7
14.6
72.7
78.3
15.6
Experts
74.0
79.5
13.9
73.5
79.2
14.3
72.9
79.1
15.4
Experts
72.6
78.4
14.9
72.3
78.4
15.5
72.2
78.1
16.1
deep abstract features, and insert an expert into each module. It is important to note that this expert allocation strategy is not unique. We investigate how the number of experts affects model performance on distribution-based splits of ImageNet-R, R, and the results are shown in Table A With 12 experts, there is one per layer, with experts, there is one every layer. And with experts, there every 5 layers. The results show that having too many experts prevents the final expert from fully learning the relevant knowledge, while having too few experts limits the amount of knowledge that can be transferred. Therefore, it is essential to choose appropriate number of experts to ensure that sufficient and important knowledge passed on to the final expert.
Conclusion
In this paper, we introduce MultiFCL, a novel FCL framework to address the stability-plastiche trade- off in FCL while leveraging the strengths of PTMs. Specifically, to maintain the stability of PTMs, MultiFCL incorporates lightweight adapters for efficient task adaption, and prevents catastrophic forgetting by freezing them during fine-tuning. To further preserve knowledge retention, it employs multi-modal prototype initialization, incorporating semantic features and sample features to enhance class representations. On the other hand, to strengthen the plasticity of PTMs, MultiFCL employs multi-expert training mechanism, which combines multi-scale feature learning with multi-teacher self-distillation improve knowledge transfer and fusion. Experimental results demonstrate that our proposed MultiFCI. achieves state-of-the-arr performance across multiple datasets and configurations, highlighting its effectiveness FCL.
Acknowledgment
This work is supported by the National Key Research and 0022YBB1132002 of China.
Development Program (No.
10

---

References
Aljundi, R., Chakravarty, P., and Tuytelaars, T. Expert gate: Lifelong learning with a network of experts. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HII USA, July 21-26, 2017, pp. 7120-7129. IEEE Computer Society, 2017. doi: 10.1109/CVPR.2017..55
Babakniya, S., Fabian, Z., He, C., Soltanolkotabi, M., and Avestimehr, S. data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks. In Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Bakman, Y. F., Yaldiz, D. N., Ezzeldin, Y. H., and Avestimehr, S. Federated orthogonal training: Mitigating global catastrophic forgetting in continual federated learning. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024. URL
wwt. http://oommiidnnnssiaooo
Chen, H., Zhang, Y., Krompass, D., Gu, J., and Tresp, V. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Thirly-Eighth AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixih Conference on Innovative Applications of Artificial Intelligence. IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 11285-11293 AAAI Press, 2024a doi: A1100//AAALLLSS...........
Chen, J., Ma, B., Cui, H., and Xia, Think twice before selection: Federated evidential active learning for medical image analysis with domain shifts. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11439-1 11449, 2024b. CVPR52733.20244.0008
doi: 10.1109/
Cui, Y.. Cao, K., Cao. G.. Qiu, M., and Wei. T. Client scheduling and resource management for efficient training in heterogeneous iot-edge federated learning. IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems, 41(8):2407-2 2022. doi: 10.1109/. 2021.3110743.
Dong, J., Wang. L., Fang, Z., Sun, G., Xu, S., Wang. X., and Zhu, Q. Federated class-incremental learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10154-10163. IEEE, 2022. doi: 10.1 199////528825888 2022.00992.
Dong, J., Li, H., Cong, Y., Sun, G., Zhang, Y., and Gool, L. V. No one left behind: Real-world federated class-incremental learning. IEEE Trans. Pattern Anal. Mach. Intell., 46(4):2054-2070, 2024. doi: 10. 10.11///MM..2203333332222
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://opennreveew net. wow.mmid=diicdaaia
Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA. April 11-13, 2011, volume 15 of JMLR Proceedings, pp. 315-323 JMLR.org, 2011.
Guo, H., Zhu, F., Liu, W., Zhang, X., and Liu, C. Pilora: Prototype guided incremental lora for federated insremnncemettll learning. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 2024, Proceedings, Part LXV. volume 15123 of Lecture Notes in Computer Science, pp. 141-159. Springer, 2024, doi: 10. 11-1007797783--01-7256-----
Halbe, S., Smith, J. S., Tian, J., and Kira, Z. Continual adaptation of vision transformers for federated learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL. htttps://openreviex. http://rwww...eeeeoooooooeeeeeea

---

Hendrycks, D., Basart, S., Mu, N., Kadavath, S.. Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 8320-8329.. IEEE, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL atttps///openrevice net/ DerrumiidnZZeWKeeF11
Jia, Z., Zhou, T., Yan, Z., Hu, J., and Shi, Y. Personalized meta-federated learning for iot-enabled health monitoring. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(10):3157-3170, 2024. doi: 10.1 LLL10/CCDD.222.3338338
Jung. D., Lee, D.. Hong. S.. Jang, H., Bae, H., and Yoon, S. New insights for the stability-plasticee dilemma in online continual learning. In The Eleventh International Conference on Learning Representutions, ICLR 2023, Kiguli, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httts://openreviis wttp:/nee//eee..ee ierumiid=ExC7kJJah...
Kahneman, D., Slovic, P., and Tversky, A. Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press, 1982.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report O, University of Toronto, Toronto, Ontario, 2009.
Kurniawan, M. R., Song, X., Ma, Z., He, Y., Gong, Y., Qi, Y., and Wei, X. Evolving parameterized prompt memory for continual learning. In Thirty-Eighth AAAI Conference on Artificial Intelligence,
AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 13301-13309. AAAI Press, 2024. doi: 10.1609/ AAAL
Li, Y., Xu, W., Qi, Y., Wang, H., Li, R., and Guo, S. SR-FDIL: synergistic replay for federated learning. IEEE Trans. Parallel Distributed Syst., 35(11)11879-1890 2024. doi: 10.1 10.1/09T 202..34372224444774
Liu, C., Qu, X., Wang, J., and Xiao, J. Fedet: A communication-effici federated class-incremental learning framework based enhanced transformer. In Proceedings of the Thirty-Scoond Inter- national Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pp. 3984-3992. ijcai.org, 2023. doi: 10.2496//JCAAI.2223444
Lu, Y., Yang. L., Chen, H. Cao, J., Lin, W., and Long. S. Federated class-incremental learning with dynamic feature extractor fusion. IEEE Trans. Mob. Comput., 10.11099TMMC..224444199999
2024. doi:
Ma, Y., Xie, 7.,, Wang, J., Chen, K., and Shou, L. Continual federated learning based on knowl- edge distillation. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 2182-2188. ijcai.org, 2022. doi: 10.24963/ICCA1.22223333
McMahan, B., Moore, E.,, Ramage, D.,, Hampson, S., and y Arcas, B. A. Communication-efficiase learning of deep networks from decentralized data. In Proceedings of the 20th International Con- ference Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pp. 1282. PMLR, 2017.
mnmoustafa and Ali, M. Tiny 2017. Kaggle.
htt.aaggle..cmmmmmmmiitiiiooo/os -mmgenntt
Moon, J., Park, K., Kim, J. U., and Park, G. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In IEEF/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11697-11707. IEFF, 2023. doi: 2001099//CCS1100022200.00
12

---

Nguyen, M.-D., Lee, S.-M., Pham, Q.-V. Hoang, D. T., Nguyen, D. N., and Hwang, W.-J. Hefl: A high compression approach for communication-efficiee federated learning in very large scale iot networks. IEEE Transactions on Mobile Computing, 22(1 1)::695566555 2023. doi: 10.1109/MMC. 2022.3190510.
Odena, A.,
Olah, C.,, and Shlens, J. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651. PMLR, 2017.
Ogier du Terrail, J., Ayed, S.-S., Cyffers, E., Grimberg, F., He, C., Loeb, R., Mangold, P., Marchand, T., Marfoq. O., Mushtaq, E., Muzellec, B., Philippenko, C., Silva, S., Teleñczuk, M., Albarqouni, S., Avestimehr, S., Bellet, A., Diculeveut, A., Jaggi, M., Karimireddy, S. P., Lorenzi, M., Neglia, G., Tommasi, M., and Andreux, M. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. In Advances in Neural Information Processing Systems, volume 35, pp.
Curran Associates, Inc., 2022.
Piao, H., Wu, Y., Wu, D., and Wei, Federated continual learning via prompt-based dual knowledge transfer. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL WRRLitttps:/ppeereeviee ntt//iiewwneeeteeeee formm?id Kqa5JakTjB
Qi, D., Zhao, H., and Li, S. Better generative replay for continual federated learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OvenReview.net, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021.
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. FLAVA: foundational language and vision alignment model. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pp. 15617- 15629. IEEE, 2022. doi: 10. 10.1009/CVVR2228822222..82 CVPR52688. 2022.01519
URL httpp:///0o org 10. 1109/
Smith, J. S., Karlinsky, L., Gutta, V., Liscante-Bonill,, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-basee prompting for rehearsal-free continual learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11909-11919, 2023a. doi: 10. 10.1109//VPP52229..22 202022552229.22.011444
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal- free continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 11909-11919, IEEE, 2023b. doi: 10.1 10.11000CVPS55272.2200..11
Tran, M., Le, T., Le, X., Harandi. M., and Phung, Text-enhanced data-free approach for federated class-incrementee learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2024, Seattle, WA. USA, June 16-22, 2024, pp. 23870-2333 23880. IEEE, 2024. doi: 10. 109/ CVPR52733..0224.022
Wah, C., Branson, S.. Welinder, P., Perona, P., and Belongie, S. The caltech-uesd dataset. 2011. URL htttps://app www.nticschollaatee......c wtt.c//cchh..a..oorroooooooliia. wwwp/oorrssDDi.......oo : 16119123
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C., Ren, X., Su, G., Perot, V., Dy, G., and Pfister, T. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, volume 13686 of Lecture Notes Computer Science, pp. 631-66600 -648. Springer, 2022. doi: 00.1007/978-3-003 1-033-198009-00
13

---

Wu, K., Peng, H., Zhou, Z. Xiao. B., Liu, M.. Yuan, L., Xuan, H., Valenzuela, M., Chen, X. S., Wang, X., Chao, H., and Hu, H. Tinyclip: CLIP distillation via affinity mimicking and weight inheritance. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 21913-21923, IEEE, 2023, doi: 10.1100/CCCV55000000000.0 0200800 URL https: /wt..//oi...o org/ 10. 1109/ ICCV51070. 2023. 02008
Wuerkaixi, A., Cui, S., Zhang. J., Yan, K., Han, B., Niu, G., Fang. L.., Zhang, C.,, and Sugiyama, M. Accurate forgetting for heterogeneous federated continual learning. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024. URL htttp:::/ppeeeeeiiieee./ net wttp://oooooriiindhdaaaaaaaaa
Xie, L., Lin, M., Luan, T., Li, C., Fang, Y., Shen, Q. and Wu, Z. MH-pFLID: Model heterogeneous personalized federated learning via injection and distillation for medical data analysis. In Forty-first International Conference Machine Learning, 2024. ptttps://openreviiii JVhMHMY
Yan, H., Wang, L.. Ma, K., and Zhong, Y. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 53670-23688 IEEE, 2024. doi: 111119/CYYR27733222200000
Yang. Q.. Zhang, J., Hao, W., Spell, G.P. P., and Carin, L. FLOP: federated learning on medical datasets using partial networks. In KDD '21: The 17thACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pp. 3845-3853. ACM, 2021. doi: 10.1145/444544.344777 344.775
Yoon, J., Jeong, W., Lee, G., Yang, E., and Hwang, S. Federated continual learning with weighted inter-client transfer. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning PMLR, 2021.
Research, pp.
Yu, H., Xu, R., Zhang, H.. Yang. Z., and Liu. H. Ev-fl: Efficient verifiable federated learning with weighted aggregation for industrial iot networks. IEEE/ACM Transactions on Networking, 32(2): 1723-1737, 2024a. doi: 10.1 N0.110//NEE..22233322222
Yu, H., Yang. X., Gao, X., Feng. Y., Wang. H., Kang, Y., and Li, T. Overcoming spatial-temports catastrophic forgetting for federated inssemnacremtt learning. In Proceedings of the 32nd ACM International Conference Multimedia, MM 2024, Melbourne, VIC, Australia, 28 October 2024-
November 2024, pp. 5280-5288. ACM, 2024b. doi: 10.1 145/3664647.36666 1384.
Yu, H., Yang, X., Gao, X., Kang, Y., Wang. H., Zhang, J., and Li. T. Personalized federated continual learning via multi-granularity prompt. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pp. 4023 4034. ACM, 2024c. doi: 10. 145/3637528..671999
Zhang, 7., Guo, B., Sun, W., Liu, Y., and Yu, Z. Cross-fel: Toward a cross-edge federated continual learning framework in mobile edge computing systems, IEEE Trans. Mob. Comput., 23(1):313-326. 2024. doi: 10. LU9/TMCC.2022.2223944
Zhou, D., Sun, H., Ning, J., Ye, H., and Zhan, D. Continual learning with pre-trained models: A survey. In Proceedings of the Thirty- -hird International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 8363-8371. ijcai.org, 2024a.
Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., and Liu, Z. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 2024b.
Zhou, D.W.., Wang. Q.. Qi. Z.-H,, Ye, H.J.,, Zhan, D.-C., and Liu, Z. Class clasemnnnremlttal learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851-987 2024c.
14

---

A Supplemental Experiments
A.1 Main Results Supplement
Table and Table 6prreeets the results on CIFARI00 with 5 tasks. It can be observed that MultiFCL. consistently achieves the best performance across data partitioning settings, demonstrating its superior ability to adapt to new tasks while maintaining strong generalization.
Specifically, as a decreases, MultiFCL exhibits the smallest performance drop, indicating that it effectively leverages PTM pre-training characteristics along with multi-scale experts and self- distillation mitigate catastrophic forgetting and enhance model stability.
Moreover, as 3 decreases, the data distribution becomes increasingly heterogeneous, leading to an overall decline in model accuracy. However, MultiFCL still demonstrates the highest robustness. For instance, under the most extreme partitioning condition (3 0.05), MultiFCL achieves 68.1% (Last) and 72.4% (Avg.), significantly outperforming PiLoRA (63.5% and 68.3%) and FedViT (60.3% and 64.4%). This result highlights the strong learning capability of MultiFCL in low-data and highly heterogeneous federated environments, effectively preventing severe knowledge forgetting during the FCL process.
Table 5: Main results of performance comparison on CIFAR 100 with tasks in distribution-based partitioning. where -A means using CLIP base model. The best results are shown in bold.
8 = 0.5
β = 0.1
B = 0.05
Methods
Last
Avg.
FF
Last
Avg.
F
Last
Avg.
F
ViT
ICLR 2021
65.0
68.2
25.1
63.3
66.7
25.7
60.3
64.4
26.0
CLIP
ICML 2021
64.5
67.6
25.8
63.6
67.3
25.9
60.5
64.1
26.3
MFCL
NeurlPS 2023
36.9
47.2
25.7
32.5
44.8
26.4
30.7
41.9
27.2
FedCBC
MM 2024
23.9
23.0
30.5
20.4
24.2
31.1
17.8
22.9
32.5
LANDER
CVPR 2024
25.8
26.6
28.7
24.6
26.3
29.4
24.7
26.0
30.7
FedMGP
KDD 2024
24.4
35.2
31.2
24.4
33.5
32.1
24.7
31.4
32.6
PiLoRA
ECCV 2024
64.2
68.6
26.9
66.8
69.5
27.3
63.5
68.3
27.4
PiLoRA-A
ECCV 2024
66.3
68.2
26.8
66.4
68.8
27.1
63.4
68.0
27.6
MultiFCL
This Paper
72.4
75.5
12.6
70.0
73.1
12.7
68.1
72.4
12.6
Table 6: Main results of performance comparison CIFAR100 with 5 tasks in quality- partitioning, where -A means using CLIP as base model. The best results are shown in bold.
6
4
2
Methods
Last
Avg.
Last
Avg.
F
Last
Avg.
F
VIT
ICLR 2021
63.4
66.9
27.4
62.8
66.4
28.1
62.3
65.3
28.7
CLIP
ICML 2021
63.4
67.2
27.7
63.1
66.7
28.0
62.6
65.8
28.5
MFCL
NeurIPS 2023
35.4
43.7
30.6
31.3
35.8
31.4
21.9
26.4
32.8
FedCBC
MM 2024
20.3
25.7
28.9
18.5
22.4
29.5
17.1
22.5
31.0
LANDER
CVPR 2024
36.8
45.5
27.8
32.6
37.8
28.0
27.6
31.2
29.1
FedMGP
KDD 2024
19.8
24.3
30.6
16.6
19.1
31.2
14.7
18.9
32.0
PiLoRA
ECCV 2024
66.4
72.9
28.4
64.4
68.9
29.1
64.2
69.3
29.6
PiLoRA-A
ECCV 2024
65.5
72.1
28.9
64.1
68.8
29.3
63.9
68.7
29.8
MultiFCL
This Paper
69.2
74.1
10.4
69.9
74.6
11.0
68.5
73.5
12.3
A.2 Ablation Results Supplement
To further analyze the contributions of key components in MultiFCL, we conduct ablation studies on 100 Table T the impact of each module on the final model performance, including adapter, prototype initialization, Lexpens and Lself-kd-
15

---

Table 7: Ablation results of performance comparison CIFARI00 with Tasks, The best results shown in bold.
Adapter
Init.
Lexports
Leclf-kd
6
2
8
0.5
0.1
3
0.05
63.4
62.8
62.3
65.5
63.8
61.2
65.7
64.2
62.8
67.4
65.0
63.2
66.5
65.7
63.8
68.2
66.9
64.6
68.1
68.6
67.3
70.9
68.7
66.7
69.2
69.9
68.5
72.4
70.0
68.1
Table 8:
Few-shot results of performance comparison on CUB-200-2011 with 10 tasks distribution- based partitioning, where -A means using CLIP base model. The best results are shown in bold.
0.5
0.1
3
0.05
Methods
Last
Avg.
Last
Avg.
Last
Avg.
VIT
ICLR 2021
73.5
75.6
29.7
71.2
74.9
30.3
70.9
73.2
31.7
CLIP
ICML 2021
73.0
74.9
30.0
71.4
75.0
30.8
70.9
73.3
32.0
MFCL
NeurIPS 2023
22.8
27.6
33.6
21.1
25.7
34.1
20.6
25.9
35.3
FedCBC
MM 2024
20.2
29.5
30.5
19.9
27.4
31.7
17.1
26.3
32.8
PiLoRA
ECCV 2024
63.0
73.5
29.6
53,9
68.0
30.7
57.3
69.3
31.5
PiLoRA-A
ECCV 2024
62.6
72.2
30.2
53.6
67.3
31.6
57.5
69.4
31.6
MultiFCL
This Paper
78.5
84.5
16.7
77.7
84.9
18.2
77.2
84.9
18.0
Introducing adapters improves performance across all settings, with 5 tasks improving by 2.3%. And with the addition of prototype initialization, shows further accuracy gains. After incorporating Lexpents accuracy for 5 tasks improves by 1.6% 3.5%, confirming that it facilitates knowledge sharing across tasks and enhances the model's s ability to handle imbalanced data distributions.
Finally, with Csetr-dd, the model achieves the highest accuracy. Overall, these results validate that MultiFCL, by integrating multiple critical modules, significantly boosts model performance FCL scenarios across various data partitioning settings.
A.3 Few-Shot Study
In FL., resource-constrained clients may have only a limited number of samples for training, and this data scarcity exacerbates the challenge of adapting new tasks the long-term learning process. To address this, we conduct the few-shot study, exploring the performance existing methods experiment
under few-shot conditions. Specifically, Under the distribution- based data partitioning, each client has a few dozen samples, while under the quality-based partitioning, each client only has three samples per class. The results shown in Table and Table
the few-shot dataset CUB-200-201
It can be observed that MultiFCL demonstrates the most significant advantage in the few-shot setting. Since PTMs are pre-trained on large-scale datasets, they can achieve expressive results with only small number of samples. In contrast, FCL methods that train models from scratch tend to over-fit in the few-shot setting, lcading to a significant performance decline. Mcanwhile, MultiFCL enhances the model's ability to adapt new tasks through multi-teacher self-distillation and multi-scale feature learning, preventing excessive reliance on the limited sample information. Experimental results show that MultiFCL effectively alleviates adaptation challenges in few-shot settings, maintaining its performance even data-scards scarce scenarios.
What's more, Table 10 and Table [] presents the results on CUB-200 with 5 tasks. MultiFCL consistently achieves the best performance across all data partitioning settings. For instance, when = 6, MultiFCL. attains 66.6% (Last) and 75.9% (Avg.) accuracy, outperforming PiLoRA by 27.5% and 24.8%, respectively, and surpassing FedViT by 5.6% and 6.8%. Even under extremely low-data conditions, MultiFCL effectively leverages pre-trained knowledge while enhancing model
16

---

Table 9: Few-shot results of performance comparison CUB-200-2011 with 10 tasks in quality- based partitioning, where -A means using CLIP as base model. The best results are shown in bold.
6
4
2
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
VIT
ICLR 2021
61.0
69.1
33.9
57.9
64.3
34.5
53.1
56.5
35.2
CLIP
ICML 2021
61.1
69.3
34.0
57.6
64.5
34.4
53.2
56.6
35.5
MFCL
NeurIPS 2023
18.8
21.5
34.4
13.6
19.9
35.0
10.8
17.3
35.7
FedCBC
MM 2024
17.4
21.4
34.3
15.9
21.0
35.2
15.2
20.5
35.6
PiLoRA
ECCV 2024
39.1
51.1
35.7
43,9
57.0
36.4
53.2
65.9
37.2
PiLoRA-A
ECCV 2024
40.3
51.6
36.1
44.0
57.2
37.3
53.2
65.8
37.0
MultiFCL
This Paper
66.6
75.9
24.0
64.4
73.5
25.2
57.0
69.0
26.0
Table 10: Few-shot results of performance comparison CUB-200-2011 with tasks in distribution- based partitioning, where -A means using CLIP as base model. The best results are shown in bold.
8 =
0.5
β = 0.1
B = 0.05
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
ViT
ICLR 2021
73.2
75.0
29.9
72.0
74.9
30.5
70.4
73.6
31.1
CLIP
ICML 2021
73.0
75.2
30.4
72.1
75.0
30.7
71.3
73.9
32.0
MFCL
NeurIPS 2023
23.9
28.6
30.3
21.2
26.7
32.6
20.5
26.0
32.1
FedCBC
MM 2024
21.5
30.7
30.2
19.5
27.3
30.8
18.3
27.0
31.9
PiLoRA
ECCV 2024
61.0
70.8
27.6
55.6
66.0
28.3
58.4
68.3
29.0
PiLoRA-A
ECCV 2024
60.7
70.2
28.0
55.7
66.4
28.3
58.3
68.0
29.3
MultiFCL
This Paper
77.4
83.7
15.0
77.8
83.8
14.8
77.8
84.1
15.6
generalization through multi-scale learning and self-distillation mechanisms. As result, it maintains stable and competitive performance even under highly heterogeneous data distributions.
CIFAR100 Task 10
CIFAR100 Task
CUB-200-2011 Task 10
CUB-200-2011 Task 5
90
Accurasy
80-
Accurazy
Accuracy
Accuracy
80
25
70
70-
70
60
2
4
0
2
4
0
Task
Task
Task
Task
TinylmageNet Task
Tiny ImageNet Task
ImageNet-R Task
ImageNet-R Task
Accuncy
80
Accurcyy
80
Accunsy
80
Accunce
80
70
70
70
4
0
Task
Tusk
Task
Task
6
4
a=2
B=0.5
P=0.1
6=0.05
Figure 5; The performance of MultiFCL per task in the 5 and 10 tasks on all datasets.
A.4 Incremental Trend Study
Figure 5
presents the FCIL results of MultiFCI. across all datasets and data splits, where fine- tuning the adapters the first task enables the PTM to achieve highly expressive performance. As
17

---

Table 11: Few-shot results of performance comparison CUB-200-2011 with 5 tasks in quality- based partitioning, where -A means using CLIP as base model. The best results are shown in bold.
6
4
2
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
VIT
ICLR 2021
60.3
68.9
30.2
58.4
65.5
31.0
54.6
58.7
31.7
CLIP
ICML 2021
60.4
68.2
30.4
58.6
65.7
31.1
55.0
59.2
32.0
MFCL
NeurIPS 2023
20.4
22.5
33.3
18.7
20.3
33.8
15.2
18.6
35.1
FedCBC
MM 2024
19.5
21.6
32.3
16.9
22.1
33.9
15.2
21.2
34.5
PiLoRA
ECCV 2024
49.8
59.3
30,9
50,9
61.9
31.6
62.5
72.9
32.7
PiLoRA-A
ECCV 2024
50.0
59.9
30.1
51.0
66.3
31.7
62.4
67.6
32.4
MultiFCL
This Paper
66.4
75.9
19.2
62.5
69.6
22.4
59.6
69.0
24.9
number of tasks increases and the model must recognize all previously seen classes, accuracy gradually declines. However, from the second through fourth tasks, by using multiple experts to help the model quickly understand the knowledge new and old tasks, PTM is able to distinguish old classes from new ones, maintaining balance between stability and plasticity.
A.5 Extreme Heterogeneity Study
In experiments, distribution- based partition- ing corresponds to Non-IID splitting, and the value of / determines the degree of hetero- geneity, with smaller values producing more heterogeneous data. We evaluate MultiFCL's performance on the final ImageNet-R task un- der extreme client distribution heterogeneity, shown in Figure 6 The results show that Mul- tiFCL maintains expressive performance even at 3 = 0.05. In addition, we test the model at values of 0.01, 0.005, and 0.001 and found that MultFCCL handles all of these scenarios effectively. Even in the most extreme cases the model's s accuracy fluctuates by only 1.4%. This robustness comes from clients learning di- verse perspectives through experts and sharing knowledge internally, while FL enables external knowledge exchange from multiple angles, al- lowing the model to quickly adapt to new tasks and remain highly resilient.
100
80
74.0
73.5
72.9
73.0
72.7
72.6
Accuracy (%)
60
40
20
0
0.5
0.1
0.05
0.01
0.005
0.001
Figure 6: Extreme heterogeneity results
ImageNet-R with 10 tasks, the x-axis represents the value of 3.
A.6 Loss Weight Study
We explore the relationship between weights of two loss functions during client local training, with
CIFAR100 results shown Table [2, and the CUB-200-2011 results are shown Table [3 Specifically, we conduct experiments by selecting values for Xexpert from 10.1, 0.3, 0.5, 0.7, 0.9). As the value decreases, the importance of multi-teacher self-distillation Lsett-kd becomes smaller. From the results, it is evident that the performance of MultiFCL CIFAR 100 experiences only minor fluctuations, and different loss function weights do not significantly impact the overall performance of the model. This indicates that the coordination between multi-teacher self-distillation and multi-scale feature learning losses is quite robust during training. allowing for flexible adjustments within certain range without affecting the model's performance.
Additionally, when the value of Nexpert is around 0.5, MultiFCL achieves optimal performance. This suggests that learning knowledge from different experts helps the final expert better distinguish different samples, but the knowledge learned from former experts should not too important, as this could cause the final expert to neglect the features it has extracted. By appropriately adjusting the
18

---

Table 12:
Loss weight results of performance comparison CIFARI00 with 10 tasks. The best
results are shown in bold.
10 Tasks
5 Tasks
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
a =
68.6
69.2
69.7
69.6
69.5
67.9
68.2
68.1
68.0
67.7
4
69.7
70.0
70.3
70.3
70.2
68.8
69.6
69.9
69.5
69.1
69.0
69.3
71.0
69.6
68.5
68.3
68.9
69.2
68.9
68.7
0.5
69.1
70.7
70.6
69.9
70.1
71.9
72.4
72.0
71.8
71.5
0.1
68.3
68.7
68.9
68.8
68.7
69.3
69.7
70.0
69.9
69.5
8 0.05
66.5
66.9
67.2
67.9
67.8
67.7
68.0
68.1
68.1
68.1
Table 13: Loss weight results of performance comparison on CUB-200-2011 results are shown in bold.
with 10 tasks. The best
Tasks
5 Tasks
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
2
56.4
56.8
57.0
56.7
56.5
69.0
59.
58.4
59.6
59,4
64.1
64.2
64.4
64.3
64.0
61.9
62.1
62.4
62.5
62.1
Q
66.1
66.3
66.4
66.6
66.4
65.8
66.0
66.4
66.3
66.4
8 0.5
78.0
78.3
78.3
78.5
78.1
77.3
77.4
77.0
76.9
76.6
8 0.1
77.3
77.6
77.7
77.6
77.4
77.3
77.6
77.8
77.6
77.5
0.05
76.5
76.9
77.2
77.0
76.8
77.4
77.5
77.8
77.8
77.7
value, MultiFCL can be flexibly configured across different tasks and datasets, further enhancing the adaptability of the model.
A.7 Long Tasks Study
We evaluate MultiFCL's performance on long task sequences by extending ImageNet-R 20, 30. 40, and 50 tasks using distribution-buu splits with B = 0.05, and the results are shown in Figure 7 It can be seen that the model re- mains stable across long sequences, even though each task contributes less new knowledge. The client experts effectively retain previous knowl- edge while assimilating new information, allow- ing the model to enhance plasticity without sac- rificing stability. As result, MultiFCL exhibits only a 1.3% fluctuation in accuracy over these extended task sequences, demonstrating its ef- fectiveness and robustness.
100
80
72.8
72.9
73.0
73.1
72.7
71.7
Accuracy (%)
60
40
20
0
10
20
30
40
50
Figure 7: Long task results on ImageNet-R with
A.8 TIL Study
0.05, the x-axis the number of tasks.
We analyze the TIL results of MultiFCL CIFAR-100 under both 10-task and 5-task set- tings, as shown in Tables 14 and [5] MultiFCL consistently achieves the best performance. The tables show that TIL tasks are easier than CIL tasks since they not require evaluating model's performance on previous tasks and focus only on the current task. MultiFCL. is designed to enhance plasticity while preserving stability, and its performance on the final task far surpasses that of other PTM-based methods, demonstrating its superior plasticity and validating its effectiveness.
19

---

Table 14: TIL results of performance comparison CIFAR100 with 10 tasks in distribution- partitioning, where -A means using CLIP as base model. The best results are shown in bold.
B
0.5
0.1
0.05
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
F
VIT
ICLR 2021
70.6
75.7
25.4
69.2
72.9
25.9
68.8
72.7
26.4
CLIP
ICML 2021
70.7
76.0
25.9
69.8
72.4
26.1
68.2
72.4
25.9
PiLoRA
ECCV 2024
74.4
75.4
24.3
67.3
72.9
24.1
74.8
72.8
25.7
PiLoRA-A
ECCV 2024
74
75.5
24.1
67.0
73.3
25.0
74.7
64.3
26.6
MultiFCL
This Paper
75.3
79.5
10.6
74.0
77.8
12.8
72.5
76.5
11.9
Table 15: TIL results of performance comparison on CIFAR-100 with tasks in distribution- partitioning, where -A means using CLIP as base model. The best results are shown in bold.
B
0.5
β
0.1
B
0.05
Methods
Last
Avg.
F
Last
Avg.
F
Last
Avg.
ViT
ICLR 2021
71.2
74.7
15.5
70.8
72.6
15.0
70.0
71.8
16.8
CLIP
ICML 2021
72.0
75.0
15.3
71.1
72.7
74.9
69.8
71.9
17.5
PiLoRA
ECCV 2024
72.0
72.9
13.6
71.7
72.7
14.0
71.9
74.7
14.5
PiLoRA-A
ECCV 2024
72.6
73.0
13.0
71.7
72.7
14.2
72.1
74.3
13.9
MultiFCI.
This Paper
77.8
78.8
1.0
75.5
74.9
1.5
76.0
76.9
2.4
A.9
PTM Study
MultiFCL relies solely on CLIP's image-text alignment capability and can be extended to any PTM with such alignment. For example, we supplement Flava (Singh et al. 2022) and TinyCLIP VIT- 39M/16 19M (Wu et al., (203)) as our backbone model and report its last accuracy results in Table 16 MultiFCL maintains relatively stable performance across different PTMs. Beyond this, image-lext alignment can also be achieved by adding projection layers, offering pathway to extend compatibility with more PTMs. It can be seen that using TinyCLIP does not degrade MultiFCL's performance. Conversely, it remains highly stable, which demonstrates the potential for balancing resource constraints and performance requirements in resource- limited scenarios (e.g., edge devices).
Table 16: Comparison results of last accuracy on CIFAR100 with 10 tasks in both quality- based partitioning and distribution-bas based partitioning.
Model
=
6
a
4
= 2
8 = 0.5
8 = 0.1
8 = 0.05
CLIP--aseed
71.0
70.3
69.7
70.7
68.9
67.9
Flava-based
70.4
69.9
69.7
69.5
68.2
67.5
TinyCLIP ViT-39M-based
70.8
70.2
69.5
70.9
70.0
69.2
B Data Partition Details
Referring the data partitioning strategies described in PiLoRA (Guo et 2024), greater numbers of classes possessed by client indicate higher data quality. Thus, in quality-based partitioning, each client is assigned data from randomly selected classes. Specifically, the number of samples for each class on a client is determined by dividing the total number of samples for that class by the number of clients, ensuring that data across clients do overlap. Therefore, larger a corresponds to higher data quality for the clients. Additionally, In the distribution-baat distribution is determined by the Dirichlet distribution, with smaller values of /3 indicating higher data heterogeneity. Figure shows the data distribution under different datasets and partitioning strategies in the last task, which the ID of clients and labels are only used to distinguish between different clients and labels. The code is available https: : ///ggtttubb... com wwt.c//wwwccc////..oocoo
data partitioning, each client's s data
20

---

311
397
1
180
59
321
699
0
0
0
50
50
0
0
0
0
50
0
3
0
3
21
0
50
50
0
0
Client
141
0
32
59
Client
0
50
50
477
344
50
86
0
0
0
23
264
50
0
0
0
0
50
231
13
24
o
201
0
50
0
0
50
0
2
0
123
3
50
0
0
30
7
Label
bel
(a) 10 tasks with 3 = 0.05
(b) 10 tasks with 2
Figure 8: The Data Partition in the Last Task on CIFAR100.
C
Method Analysis
Communication Cost. The model parame- ters consist of the PTM, multiple experts, and adapters. Notably, the PTM does not require communication, and adapters only need to be exchanged during the first task. For instance, in scenario with 10 tasks, the total number of expert parameters is 384,000, while the adapters account for 156,768 parameters. Approximately 0.54 million parameters are communicated dur- ing the first task, and only about 0.38 million in subsequent tasks. The following comparison of parameters in Table with with others demonstrates that MultiFCL is highly communication-efficcne
Table 17: Parameter Comparison.
Model
Parameters
ResNet-
11M
ResNet-50
23M
ResNet-
43M
ViT-B/16-INIK
86M
ViT-B/I6-CLIP
86M
MultiFCL.
0.54/ 0.38M
Computational Efficiency. In clients, for forward propagation, the multi-expert feature learning requires extracting features {Aim m=1 from M modules. For a PTM with L total layers divided into M modules, the computational complexity remains: Ctorward = O (tr (d2 (s) x M = O(LPS)) where S represents the sequence length, is the dimension of prototype. For backward propagation, it is equivalent to forward propagation, which is (). In the server, for the first task, the server needs to aggregate experts and adapters uploaded by clients, with a computational complexity of O(K x M x O x N x D), where D is the dimension of an adapter, and N is the total number of adapters in a client. For the subsequent tasks, the server only needs to aggregate the prototype experts and adapters are frozen, with computational complexity of O(K M x d).
We compare communication parameters and wall-clock per round for MultiFCL, PiLoRA, MFCL, and LANDER on CIFAR 100. Each round includes 5 local epochs across 10 clients plus one server update, as shown in Table T8 In MFCL, the server must distribute three components to clients, including the current-task global model, previous-task global model, and generator. Although the global model uses R8 (11.2M), this still incurs substantial communication overhead. Meanwhile, despite its wall-clock per round remains low, it requires 100 communication rounds per task to achieve competitive results, whereas MultiFCL and PiLoRA attain expressive outcomes within merely rounds. LANDER transmits the ResNet-18 18 global model each round. Its integration of pre- trained CLIP and generator yields wall-clock per round comparable to PTM-based methods while requiring 100 rounds per task. PiLoRA communicates LoRA parameters and prototypes per round.
21

---

Although parameter volume stays low, continu- ous LoRA tuning prevents effective knowledge retention and increases computation time, result- ing suboptimal performance versus MultiFCI. MultiFCL. freezes adapters after initial tasks, reducing communication parameters (0.54M to 0.38M) and wall-clock per round (228.2) f228.29s to 150.24s) while achieving competitive per- formance within 5 communication rounds per task. Collectively, MultiFCL maintains compu- tational efficiency while reducing communica- tion costs versus existing methods.
Table 18: Comparison results of communication parameters and wall-clock per round on CIFAR with 10 tasks in distribution-base partitioning.
Model
Parameters
Wall-clock
MFCL
24.27M
35.20s
LANDER
11.20M
169.65s
PiLoRA
0.0584M
207.10s
MultiFCL
0.54/ 0.38M
228.29/ 150.24s
D Algorithm
Algorithm displays the detailed process in MultiFCL. It employs adapters to fine-tune the PTM and leverages the semantic features old tasks to initialize new class protolypes. Then, establishes multiple experts, employing feature learning loss and the multi-teacher dynamic self-distillation to transfer knowledge to the final expert.
Algorithm MultiFCL Framework
Require: Pre-trained model f, clients K, communication rounds T, tasks (Ti, T2,... : Tr}, local
epochs E
1: for each task T. (TI...., T do
2: if then
3: for round r = 1-110 to T do
4: for each client k to in parallel do
5:
Fine-tune adapters on D/
Upload wiz and sample count [X] to server 7: end for
6:
8:
Server aggregates: w" -
K
[X]
Σ
9: Server broadcasts global adapter w" to clients
end for
11: Freeze global adapters w' + n
else
13: for each new class Te do
Compute semantic similarity: sim ('(x)) (C
15:
Initialize prototype: P < Attention (friiion((((()) (f'(x)), frea Tas ), P end for
17:
end if
18:
for round r 1 to T' do
19:
for each client &o = 1 to K in parallel do E do
for local epoch
21:
Extract multi-scale features: {hm 15
Jm=1 &- fl( w²)
22:
Compute expert loss: Lexperts + Nexpert LretEkal
Update parameters: <- 17 Vo (C)
Update prototypes: P(G) +- P ( end for
25:
26:
Upload prototypes {pf.. - and sample counts ( to server
27:
end for
28:
ix Server aggregates prototypes: par Σk ΣixET
Σm Pk.m
29:
Server broadcasts global prototypes p" to clients
30:
end for
31:
end for
22

---

E Limitations
Although the proposed MultiFCL framework delivers strong performance across variety of hetero- geneous data partitions and CL scenarios, it still has several limitations that call for further study. MultiFCL depends on large scale PTMs and their multi- scale feature modules, which may impose considerable computational and storage demands on devices with limited resourees. The mechanisms for prototype initialization and semantic feature matching need refinement extreme class imbalance or few sample scenarios to ensure that both new and existing classes are represented effectively. The current evaluation focuses mainly on image classification tasks and remains necessary to investigate how MultiFCL can be applied to other modalities such text and speech extended to more complex tasks.
Code of Ethics
This paper is guided by commitment privacy, performs all model updates and feature extrac- tion entirely in clients without collecting or storing any raw data. We provide a detailed description of each algorithmic component including multi-adapter fine-tuning, prototype initialization, multi-scale feature learning and multi-teacher adaptive self-distillation along with network architectures and hy- perparameter settings ensure that others can reproduce and validate our findings without accessing private data. During evaluation, we use publicly available datasets to emulate realistic distributions and prevent exposure of any individual-level information. All original data remains on local devices under the full control of each participant throughout the entire process.
G Societal Impacts
This paper presents work that aims to advance the field of deep learning. Our research explores how to leverage pre-trained models to enhance performance in FCL. By introducing mechanisms such as multi- multi-adapter fine-tuning, modal prototype initialization, multi-scale feature learning, and multi-teacher dynamic self-distillation, improve the model's adaptability and generalization, ensuring performance is maintained while preventing catastrophic forgetting.
Since this work provides novel perspective for deep learning from a technical standpoint, we recognize that with the broader application of such technologies, there may be some impacts to consider. For instance, FCL has the potential to play critical role in fields like healthcare and the internet of things. This requires us to design methods that avoid bias or misleading decisions, and we are actively working towards that goal.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly state in the abstract and introduction that this paper is research on FCL. Please refer to Section for contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4 Appendix A and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Please refer to Section Appendix
B and our code in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http///bbb////iiiiodoeee/eaaaaaa cassiicnncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Please refer to Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the priority given to the repeatability of the dataset, these results obtained with fixed hyperparameters and fixed sequence test segmentation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Please refer Appendix
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes)
Justification:
We referenced the open-source datasets and models used in Section 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
30


Reviews:
[{"review_id": "Xh5tbfFGJg", "reviewer": "Reviewer_aSUp", "summary": "The paper proposes MultiFCL, an FCL framework that (i) trains task-specific adapters on the first task and then freezes them, (ii) initialises class prototypes by fusing CLIP visual embeddings with label text embeddings, and (iii) introduces several “experts” (different depth slices of the ViT backbone) whose predictions are merged into a single head via uncertainty-weighted self-distillation.", "questions": "1. Can you report true communication (MB) and wall-clock per round on a standard GPU cluster? Currently efficiency claims are speculative.\n\n1. Prototype sharing may leak class information. Can you provide an empirical or theoretical privacy analysis, or add differential-privacy noise?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Implementation details (optimiser, epochs, learning rates) are mostly disclosed and code link is promised.\n1. High-level intuition is understandable.\n1. Shows that freezing CLIP with adapters can outperform training-from-scratch FCL baselines.\n\nWeaknesses\n1. No statistical significance: all results are single runs, no error bars.\n1. Communication/time claims lack real wall-clock or FLOP evidence.\n1. Gains largely attributable to CLIP pre-training; incremental benefit of the expert/distillation machinery is small.\n1.Datasets are moderate-scale; no real FL use-case (health, IoT) included.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aSUp,\n\nWe sincerely appreciate your recognition of our framework's high-level intuition and the effectiveness of adapter freezing. Your insights on statistical significance and communication efficiency have helped us deepen our analysis. We've carefully addressed your technical questions about empirical validation and privacy protection in the following responses, which we hope will resolve your concerns.\n\n> **W1: No statistical significance: all results are single runs, no error bars.**\n\nFollowing your suggestion, we conduct experiments on CIFAR100 under distribution-based partitioning using five distinct random seeds. We supplement standard deviation for the last-task accuracy of MultiFCL, PiLoRA, and MFCL-CLIP. The results are as follows:\n\n||0.5|0.1|0.05|\n|-|-|-|-|\n|MFCL-CLIP|52.3±0.4|51.8±0.4|50.6±1.0|\n|PiLoRA|69.2±0.4|63.2±0.5|62.7±0.5|\n|MultiFCL|**70.7±0.3**|**68.7±0.2**|**67.7±0.2**|\n\nTable 1: Comparison results of last accuracy on CIFAR100 with 10 tasks in distribution-based partitioning $\\beta=0.05$.\n\nThe p-values for MultiFCL versus PiLoRA and MultiFCL versus MFCL-CLIP are both substantially below 0.001. This demonstrates that MultiFCL **achieves stable performance with statistically significant improvements**.\n\n> **W2 (Q1): Communication/time claims lack real wall-clock or FLOP evidence.**\n\nWe compare communication parameters and wall-clock per round for MultiFCL, PiLoRA, MFCL, and LANDER on CIFAR100. Each round includes 5 local epochs across 10 clients plus one server update. The results are as follows:\n\n||Communication|Wall-clock|\n|-|-|-|\n|MFCL|24.27M|35.20s|\n|LANDER|11.20M|169.65s|\n|PiLoRA|0.0584M|207.10s|\n|MultiFCL|0.54/0.38M|228.29s/150.24s|\n\nTable 2: Comparison results of communication parameters and wall-clock per round on CIFAR100 with 10 tasks in distribution-based partitioning.\n\nIn MFCL, the server must distribute three components to clients, including the current-task global model, previous-task global model, and generator. Although the global model uses ResNet-18 (11.2M), this still incurs substantial communication overhead. Meanwhile, despite its wall-clock per round remains low, it requires 100 communication rounds per task to achieve competitive results, whereas MultiFCL and PiLoRA attain expressive outcomes within merely 5 rounds.\n\nLANDER transmits the ResNet-18 as global model each round. Its integration of pre-trained CLIP and generator yields wall-clock per round comparable to PTM-based methods while requiring 100 rounds per task.\n\nPiLoRA communicates LoRA parameters and prototypes per round. Although parameter volume stays low, continuous LoRA tuning prevents effective knowledge retention and increases computation time, resulting in suboptimal performance versus MultiFCL.\n\nMultiFCL freezes adapters after initial tasks, reducing communication parameters (0.54M to 0.38M) and wall-clock per round (228.29s to 150.24s) while achieving competitive performance within 5 communication rounds per task. Collectively, MultiFCL **maintains computational efficiency while reducing communication costs versus existing methods**.\n\n> **W3: Gains largely attributable to CLIP pre-training; incremental benefit of the expert/distillation machinery is small. Datasets are moderate-scale; no real FL use-case (health, IoT) included.**\n\nIn Table 1 of our paper, we present results for ViT, CLIP, MFCL-CLIP, FedMGP, PiLoRA, and PiLoRA-CLIP, which are all PTM-based methods. Among them, MultiFCL consistently outperforms these approaches across all CIFAR100 partitioning configurations, achieving 3.8\\%-19.5\\% higher average last-task accuracy. Simultaneously, Table 3 of our paper demonstrates MultiFCL delivers 6.4\\% higher average accuracy than directly using pre-trained CLIP, validating the efficacy of MultiFCL’s components.\n\nAdditionally, following your suggestion, we supplement comparative experiments of MultiFCL, PiLoRA, and CLIP on MedMNIST [1] and IP102 [2]. MedMNIST constitutes a large-scale 2D and 3D medical image dataset incorporating multiple modalities (X-ray images, retinal OCT, ultrasound, CT scans), while IP102 is an agricultural pest/disease dataset exhibiting high inter-class similarity, significant intra-class variance (e.g., identical pests at different life stages), and imbalanced distribution. The results are as follows:\n\n[1] MedMNIST: https://medmnist.com\n\n[2] IP102: https://github.com/xpwu95/IP102\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP|31.2|30.9|20.3|28.4|24.3|21.5|\n|PiLora|33.4|32.5|22.9|30.0|25.3|21.8|\n|MultiFCL|**40.3**|**40.0**|**34.1**|**43.8**|**42.2**|**43.2**|\n\nTable 3: Comparison results of last accuracy on MedMNIST with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP|35.3|34.6|33.9|40.6|38.9|37.2|\n|PiLora|36.4|35.6|35.3|40.3|38.3|36.5|\n|MultiFCL|**44.9**|**43.8**|**42.6**|**44.8**|**43.8**|**42.5**|\n\nTable 4: Comparison results of last accuracy on IP102 with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\nIt can be seen that MultiFCL maintains robust adaptability compared with other methods in these FL application fields.\n\n**Q2: Prototype sharing may leak class information. Can you provide an empirical or theoretical privacy analysis, or add differential-privacy noise?**\n\nThe prototype features are ultimately obtained through mean aggregation, a process that is irreversible. This can be regarded as a linear transformation of the data. Due to the nonlinearity of the feature extraction function, the features of a single sample cannot be separated from the mean. Furthermore, we perform feature inversion on both the communicated prototypes and the features extracted from individual image samples. We observe that these prototypes **cannot reconstruct the approximate outlines or sensitive information of the original images**. Only the features extracted from a single sample are able to reconstruct sensitive information from the original image. In the future, we will explore adding Gaussian noise to the communicated prototypes to enhance privacy protection, as you suggested, which may require balancing privacy protection against model performance."}, {"review_id": "0rwW8vSrjO", "reviewer": "Reviewer_6opu", "summary": "In federated continual learning (FCL), achieving both stability (to preserve performance on old tasks) and plasticity (to learn new tasks effectively) is important. This paper uses pre-trained models (like CLIP/ViT) with lightweight adaptations to improve both stability and plasticity in continual federated settings. It first introduces adapters to quickly adapt PTMs to downstream tasks, which are then frozen to prevent catastrophic forgetting. Then multi-modal prototype initialization, multi-scale feature learning, multi-teacher dynamic self-distillation strategies have been proposed to effectively preserve the knowledge of prior tasks while learning new knowledge in federated continual learning. Experiment is conducted using CIFAR10, CIFAR100, TinyImageNet datasets on CLIP model.", "questions": "Please see the weakness 1, 2, 3 above, and try to address the issues.\n\nIn addition to the above weaknesses, I believe the paper could benefit from providing the overall pseudo code of the algorithm. It's difficult to know when model aggregation is conducted, when specific losses are used locally, etc.", "limitations": "Yes the limitation is well addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. This paper appropriately integrates new techniques to handle the continual federated learning problem. Multi-modal prototype initialization, multi-scale feature learning, multi-teacher dynamic self-distillation strategies are relatively novel to tackle this problem.\n\n2. Experiments are conducted using various datasets and the effect of each component, including self distillation, multi-modal prototype initialization, have been well studied.\n\nWeakness\n\n1. In practice, there could be many clients in the system but the paper considers only 10 clients.\n\n2. The intuition behind each component is less illustrated. For example, the adapters are only updated based on the first task and is frozen. So this means that this approach can preserve the knowledge of the first task, but not the knowledge of subsequent tasks. So how are the knowledge of subsequent tasks are preserved? In general I feel that the advantage/effect of each scheme should be better illustrated instead of just describing how they work.\n\n3. The computational/communication costs are analyzed, but they are not compared with the ones of the baselines. How much additional burden does this approach require compared to baselines?", "was_revised": true, "final_justification": "The authors have addressed my concerns, including results with more clients, communication/computation, and pseudo code, better clarifications. The authors also have addressed other reviewers' comments. I am leaning towards acceptance for this paper, so I keep my original score borderline accept.", "rebuttal": "Dear Reviewer 6opu,\n\nThank you for acknowledging our novel integration of multi-modal prototypes and dynamic distillation. We're also particularly grateful your attention to the thorough component ablation studies. Your perspective on client scalability, the intuition behind each component, computational trade-offs, and pseudo code display has enriched our discussion. Detailed responses to your questions about them are provided below.\n\n> **W1: In practice, there could be many clients in the system but the paper considers only 10 clients.**\n\nPer your suggestion, we investigate the last-task accuracy of MultiFCL, PiLoRA, and CLIP under varying the number of clients on CIFAR100. The results are as follows:\n\n||10|30|50|70|90|\n|-|-|-|-|-|-|\n|CLIP|60.3|59.8|59.9|60.1|60.0|\n|PiLora|63.0|63.6|63.2|63.7|63.3|\n|MultiFCL|**67.9**|**67.9**|**66.9**|**68.1**|**67.9**|\n\nTable 1: Comparison results of different number of clients on CIFAR100 with 10 tasks in distribution-based partitioning $\\beta=0.05$.\n\nWith fixed dataset size and numerous incremental tasks, increasing the number of clients reduces per-client data size and intensifies extreme heterogeneity in data distribution across clients. Nevertheless, MultiFCL consistently **maintains superior performance versus other methods, demonstrating its robustness**.\n\n> **W2: The intuition behind each component is less illustrated. For example, the adapters are only updated based on the first task and is frozen. So this means that this approach can preserve the knowledge of the first task, but not the knowledge of subsequent tasks. So how are the knowledge of subsequent tasks are preserved? In general I feel that the advantage/effect of each scheme should be better illustrated instead of just describing how they work.**\n\nBased on your suggestion, we provide the following supplementary explanation for the intuition behind each component:\n\n- **Multi-Adapter Fine-Tuning**:\n\nThe core objective of the adapters is to **bridge the domain gap** between the PTM and downstream tasks while avoiding catastrophic forgetting. Traditional full-parameter fine-tuning would disrupt the PTM's **generalization capabilities**, whereas completely freezing parameters prevents adaptation to new tasks. Therefore, by fine-tuning lightweight adapters on the first task and then freezing them, we lock the PTM's adaptation capability for that initial task. This prevents subsequent task updates from overwriting learned knowledge. Knowledge retention for subsequent tasks no longer relies on adapter updates but is instead achieved through later components (e.g., prototype initialization), **thus preventing generalization degradation caused by excessive fine-tuning**.\n\n- **Multi-Modal Prototype Initialization**:\n\nThis component addresses the **initialization bias** problem for new class prototypes in FCL. Traditional random initialization causes the new class classifier to lack historical task knowledge, leading to confusion between new and old classes. This component enables the fusion of features from new class samples with semantic features from similar old classes to initialize the new class prototype. Its essence is to **introduce semantic priors from old tasks**, positioning the initial location of the new class prototype near the feature space of semantically related old classes, which **enhances inter-class discriminability and reduces forgetting**.\n\n- **Multi-Scale Feature Learning**:\n\nTo address the insufficient plasticity of the model under dynamic data distributions, we partition the PTM into multi-level experts. Shallow experts extract textural features, while deep experts capture semantic features, collectively providing diverse representations of samples. Each expert independently optimizes its class prototypes, driving samples of the same class towards a unified class center within their respective feature spaces. Simultaneously, aggregating multi-scale prototypes across clients forms global class centers, addressing the feature shift problem caused by data heterogeneity. This multi-expert mechanism allows clients to learn new tasks at different granularities while **implicitly fusing historical task knowledge through prototype sharing**.\n\n- **Multi-Teacher Dynamic Self-Distillation**:\n\nTo balance **inference efficiency** and **knowledge integration** within the multi-expert system, we distill knowledge from all experts into the final expert, reducing computational overhead during testing. Since features from deeper experts are generally more reliable, we design an uncertainty weighting scheme. This down-weights contributions from shallow experts with high uncertainty to **prevent them from interfering with the final decision**. This component maintains multi-scale information while enhancing inference efficiency. The weighting mechanism also strengthens the transfer of reliable knowledge.\n\n> **W3: The computational/communication costs are analyzed, but they are not compared with the ones of the baselines. How much additional burden does this approach require compared to baselines?**\n\nWe compare communication parameters and wall-clock per round for MultiFCL, PiLoRA, MFCL, and LANDER on CIFAR100. Each round includes 5 local epochs across 10 clients plus one server update. The results are as follows:\n\n||Communication|Wall-clock|\n|-|-|-|\n|MFCL|24.27M|35.20s|\n|LANDER|11.20M|169.65s|\n|PiLoRA|0.0584M|207.10s|\n|MultiFCL|0.54/0.38M|228.29s/150.24s|\n\nTable 2: Comparison results of communication parameters and wall-clock per round on CIFAR100 with 10 tasks in distribution-based partitioning.\n\nIn MFCL, the server must distribute three components to clients, including the current-task global model, previous-task global model, and generator. Although the global model uses ResNet-18 (11.2M), this still incurs substantial communication overhead. Meanwhile, despite its wall-clock per round remains low, it requires 100 communication rounds per task to achieve competitive results, whereas MultiFCL and PiLoRA attain expressive outcomes within merely 5 rounds.\n\nLANDER transmits the ResNet-18 as global model each round. Its integration of pre-trained CLIP and generator yields wall-clock per round comparable to PTM-based methods while requiring 100 rounds per task.\n\nPiLoRA communicates LoRA parameters and prototypes per round. Although parameter volume stays low, continuous LoRA tuning prevents effective knowledge retention and increases computation time, resulting in suboptimal performance versus MultiFCL.\n\nMultiFCL freezes adapters after initial tasks, reducing communication parameters (0.54M to 0.38M) and wall-clock per round (228.29s to 150.24s) while achieving competitive performance within 5 communication rounds per task. Collectively, MultiFCL **maintains computational efficiency while reducing communication costs versus existing methods**.\n\n> **Q1: I believe the paper could benefit from providing the overall pseudo code of the algorithm. It's difficult to know when model aggregation is conducted, when specific losses are used locally, etc.**\n\nAccording to your suggestion, the following is the pseudo code of MultiFCL. We will update the more detailed version to the paper:\n\n**Require**: Pre‑trained model $f$, clients $K$, communication rounds $T$, tasks $\\mathcal{T}_{\\tau}$, local epochs $E$\n\n> **for** each task:  \n> &nbsp;&nbsp;**if** t = 1:  \n> &nbsp;&nbsp;&nbsp;&nbsp;**for** round $r = 1$ **to** $T$:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for** each client $k = 1$ **to** $K$ *(in parallel)*:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Fine‑tune adapters on $D_k^1$  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Upload $w_k^r$ and sample count $|X_k^1|$ to server  \n> &nbsp;&nbsp;&nbsp;&nbsp;**Server aggregates**:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$w^r \\leftarrow \\sum_{k=1}^K \\frac{|X_k^1|}{\\sum |X_k^1|} w_k^r$  \n> &nbsp;&nbsp;&nbsp;&nbsp;**Server broadcasts** global adapter $w^r$ to clients  \n> &nbsp;&nbsp;Freeze global adapters: $w^* \\leftarrow w^T$  \n> &nbsp;&nbsp;**else**:  \n> &nbsp;&nbsp;&nbsp;&nbsp;**for** each new class $c$ in new task:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute semantic similarity  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Initialize prototype $\\mathbf{p}_{\\mathrm{init}}^c$  \n>  \n> &nbsp;&nbsp;**for** round $r = 1$ **to** $T$:  \n> &nbsp;&nbsp;&nbsp;&nbsp;**for** each client $k = 1$ **to** $K$ *(in parallel)*:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for** local epoch $e = 1$ **to** $E$:  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Extract multi‑scale features  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute expert loss  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update parameters $\\theta \\leftarrow \\theta - \\eta \\nabla_\\theta (\\mathcal{L})$  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update prototypes  \n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Upload prototypes and sample counts to server  \n> &nbsp;&nbsp;**Server aggregates prototypes**:  \n> &nbsp;&nbsp;&nbsp;&nbsp;$\\mathbf{p}^{c,r} \\gets \\sum_{k=1}^K \\frac{|X_k^{c,t}|}{\\sum |X_k^{c,t}|} \\sum_m \\mathbf{p}_{k,m}^{c,r}$  \n> &nbsp;&nbsp;**Server broadcasts** global prototypes $\\mathbf{p}^r$ to clients"}, {"review_id": "KZj1Cm6sC0", "reviewer": "Reviewer_3UFP", "summary": "This paper proposes MultiFCL, a federated continual learning framework leveraging pre-trained models such as CLIP with ViT backbones. MultiFCL introduces a combination of lightweight adapters, multi-modal prototype initialization, multi-scale feature learning, and multi-teacher dynamic self-distillation to balance stability (preventing catastrophic forgetting) and plasticity (adapting to new tasks) in continual learning. The authors conduct extensive experiments on CIFAR100 and CUB-200-2011 under various partitioning and few-shot scenarios, demonstrating performance improvements over several recent baselines.", "questions": "1. The authors mention using TinyImageNet and ImageNet-R in your experimental setup (Section 4.1), but I could not locate any quantitative results. Could you clarify whether you conducted experiments on these datasets and include the results?\n\n2. All experiments appear to rely on CLIP ViT. Have the authors tested MultiFCL with other pre-trained models (e.g., standard ViT models, ResNet-based PTMs) to validate its general applicability?\n\n3. To what extent does MultiFCL’s success depend on the quality of the underlying PTM? Could the authors provide insights or experiments showing performance with weaker PTMs?", "limitations": "The paper provided a discussion with limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n+ The proposed method looks novel. The combination of adapters frozen after the first task, multi-scale experts, and dynamic self-distillation is well-motivated and original.\n+ The paper is clearly written overall, with good diagrams and thorough ablations.\n+ The experiments are extensive and cover multiple data heterogeneity scenarios, few-shot learning, and ablation studies.\n\nWeaknesses\n- Dependency on CLIP ViT: The approach heavily relies on CLIP as the backbone. It is unclear how well the method generalizes to other architectures or less powerful pre-trained models.\n- The authors state that they use TinyImageNet and ImageNet-R for evaluation (Sec 4.1), but I could not find any reported results for these datasets in the main text or appendix, even though they showed some heterogeneity study on ImageNet-R dataset. This omission weakens the empirical validation.\n- While the experiments are extensive, they only explore a single PTM architecture (CLIP ViT). Evaluating with other PTMs or even smaller ViT models would strengthen claims of generalizability.\n- Some implementation details, for example, adapter size, hyperparameter settings for all datasets, are scattered or only briefly mentioned.", "was_revised": true, "final_justification": "The authors provided the detailed response with additional experiments. I keep my initial positive rating.", "rebuttal": "Dear Reviewer 3UFP,\n\nWe're honored by your praise for our method's originality and the comprehensive ablation studies. Your perceptive questions about model applicability have significantly broadened our perspective. In the responses below, we've included additional experimental insights from TinyImageNet/ImageNet-R evaluations and different PTM architectures to clarify your technical inquiries.\n\n> **W1 (Q2): All experiments appear to rely on CLIP ViT. Have the authors tested MultiFCL with other pre-trained models (e.g., standard ViT models, ResNet-based PTMs) to validate its general applicability?**\n\nMultiFCL relies solely on CLIP’s image-text alignment capability and can be extended to any PTM with such alignment. For example, we supplement Flava [1] as our backbone model and report its last accuracy results on CIFAR100 below:\n\n[1] Amanpreet Singh et al., FLAVA: A Foundational Language And Vision Alignment Model\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP-based|71.0|70.3|69.7|70.7|68.9|67.9|\n|Flava-based|70.4|69.9|69.7|69.5|68.2|67.5|\n\nTable 1: Comparison results of different backbone models on CIFAR100 with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\nAs shown in Table 1, MultiFCL **maintains relatively stable performance across different PTMs**. Beyond this, image-text alignment can also be achieved by adding projection layers, offering a pathway to extend compatibility with more PTMs. In the future, we will explore novel architectures to enhance MultiFCL’s scalability.\n\n> **W2 (Q1): The authors mention using TinyImageNet and ImageNet-R in your experimental setup (Section 4.1), but I could not locate any quantitative results. Could you clarify whether you conducted experiments on these datasets and include the results?**\n\nFigure 4 in the paper (Appendix A.5) shows the experimental results of MultiFCL on CIFAR100, CUB-200-2011, TinyImageNet, and ImageNet-R under six different data partitions and two distinct task partitioning strategies. These results include detailed performance of MultiFCL across incremental tasks. Additionally, as per your suggestion, we supplement the experiments with PiLoRA, CLIP on TinyImageNet and ImageNet-R. The results are as follows:\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP|57.6|57.4|57.4|57.9|58.0|57.3|\n|PiLora|64.4|62.2|61.8|61.9|60.6|58.2|\n|MultiFCL|**66.1**|**65.0**|**64.8**|**64.4**|**63.0**|**61.6**|\n\nTable 2: Comparison results of last accuracy on TinyImageNet with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP|58.7|58.2|58.1|59.9|59.3|58.4|\n|PiLora|63.2|62.6|61.2|67.7|67.8|66.9|\n|MultiFCL|**69.3**|**67.7**|**65.4**|**74.0**|**73.5**|**72.9**|\n\nTable 3: Comparison results of last accuracy on ImageNet-R with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\nIt can be observed that MultiFCL demonstrates impressive results across these datasets.\n\n> **W3 (Q3): To what extent does MultiFCL’s success depend on the quality of the underlying PTM? Could the authors provide insights or experiments showing performance with weaker PTMs?**\n\nFollowing your suggestion, we supplement the results using TinyCLIP ViT-39M/16 Text-19M [2] on CIFAR100. This is a distilled version of CLIP with significantly fewer pre-training parameters than CLIP. The results are as follows:\n\n[2] Kan Wu et al., TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP-based|71.0|70.3|69.7|70.7|68.9|67.9|\n|TinyCLIP ViT-39M-based|70.8|70.2|69.5|70.9|70.0|69.2|\n\nTable 3: Comparison results of last accuracy on CIFAR100 with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\nIt can be seen that using TinyCLIP does not degrade MultiFCL's performance. Conversely, it remains highly stable, which demonstrates **the potential for balancing resource constraints and performance requirements in resource-limited scenarios** (e.g., edge devices). Moreover, we note that multiple distilled versions of TinyCLIP exist. In the future, we will experiment with other variants to further assess MultiFCL’s adaptability across diverse model scales.\n\n> **W4: Some implementation details, for example, adapter size, hyperparameter settings for all datasets, are scattered or only briefly mentioned.**\n\nIn response to your suggestions, we supplement specific details regarding the adapter size: An adapter consists of $W_{down}=768\\times 8+8$ parameters and $W_{up}=768\\times 8+768$ parameters, totaling 13,064 parameters. Additionally, the positions mentioning hyper-parameters for all datasets and other experimental settings will be adjusted to ensure they are consolidated together."}, {"review_id": "cOn7CiMQf4", "reviewer": "Reviewer_Bnjx", "summary": "This paper proposes MultiFCL, a novel framework that leverages pre-trained models (PTMs) to address the challenges in Federated Continual Learning (FCL). To ensure stability, MultiFCL introduces lightweight adapters, which are frozen after the initial task training, to enhance PTM performance on downstream tasks. MultiFCL further utilizes prototypes as classifier weights, enabling classification by measuring the distance between class prototypes and sample features. Concurrently, to enhance plasticity, the framework integrates multi-scale feature learning with multi-teacher dynamic self-distillation. Extensive experiments are conducted to validate MultiFCL's state-of-the-art performance.", "questions": "as above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-structured, logical, and innovative. However, there are still areas that need improvement and further clarification for the paper to be accepted.\n\nWeakness:\n1.\tThe paper's exploration of unique challenges inherent to FL is not enough. Furthermore, the integration and handling of various components primarily rely on basic weighted averaging, without delving into more FL-specific mechanisms. Consequently, the paper more focuses on Continual Learning, with less emphasis on novel contributions directly addressing the unique complexities of FCL.\n2.\tFigure 1 does not sufficiently illustrate the overall workflow of the proposed framework, which may lead to confusion. It is suggested that the authors further revise this figure to enhance its readability and facilitate a comprehensive understanding of the method's working process.\n3.\tWhile the framework comprises four components, the clarity of the authors' description for each part could be further improved. Specifically, a more detailed and structured explanation of their individual functions and interconnections within the overall workflow would significantly enhance reader comprehension.\n4.\tRegarding the Multi-Adapter Fine-Tuning: Firstly, this fine-tuning mechanism bears a strong resemblance to LoRA (Low-Rank Adaptation). It is recommended that the authors include relevant citations for LoRA and similar parameter-efficient fine-tuning techniques to provide appropriate context. Secondly, the rationale behind freezing these adapters immediately after the initial task training warrants further clarification. This design choice might potentially limit the method's knowledge transfer capability, especially when executing cross-domain tasks that could benefit from continuous adaptation of these modules. In summary, the overall motivation and implications behind the design of this specific component require more comprehensive elucidation, potentially including a discussion of trade-offs or supporting experimental justifications.\n5.\tRegarding the Multi-Modal Prototype Initialization component, my primary concern is: what is the source of the textual information associated with the images? Furthermore, a question arises regarding the fairness of using this multi-modal technique, which introduces additional knowledge for classification. Could the authors clarify if and how this extra knowledge source impacts the comparison with baselines that might not leverage such external information?\n6.\tRegarding the Multi-Scale Feature Learning component: Does every block need to perform a classification task? If so, is it required that each block calculates a prototype for its current layer for subsequent operations? These design choices raise concerns about their potential computational implications: would this significantly increase the client's local computational burden, thereby limiting the method's applicability in resource-constrained edge computing scenarios?\n7.\tRegarding the Multi-Teacher Dynamic Self-Distillation component: There is a concern that distilling all expert knowledge into a single expert might not necessarily translate into significant performance gains. Simultaneously, this component appears to introduce substantial computational overhead, which could limit the method's practicality. It would be beneficial if the authors could provide further justification for this design choice, perhaps through an ablation study on its precise contribution to performance versus its computational cost.\n\nOverall, while these components demonstrate innovativeness, the motivation behind their design and enablement remains insufficiently articulated. The authors are strongly encouraged to provide more in-depth explanations regarding these aspects.", "was_revised": true, "final_justification": "Thank you for your detailed and comprehensive rebuttal. After carefully reviewing your responses, most of my previous concerns have been addressed. However, I still have some concerns regarding the computational resources of this method, and therefore, I have decided to maintain my favorable rating.", "rebuttal": "Dear Reviewer Bnjx,\n\nWe sincerely appreciate your recognition of our framework’s structural innovation and logical design, and we’re grateful you acknowledged the originality of our multi-expert integration. Your thoughtful examination of the adapter design and multi-expert distillation has strengthened our presentation. We've prepared detailed explanations below addressing your workflow description and component motivation questions.\n\n> **W1: Research on FL mechanisms**\n\nIn most FL research, weighted averaging of model parameters or weighted averaging of prototypes are common techniques for integration and processing. To address the unique challenge of data heterogeneity in FL, we enable knowledge sharing among clients by aggregating multi-scale prototypes, while addressing task heterogeneity through internal expert collaboration. Furthermore, we propose **uncertainty-weighted multi-teacher self-distillation**, which dynamically adjusts knowledge transfer weights based on the entropy of expert outputs, avoiding the limitations of simple prototype averaging.\n\nAdditionally, in our experiments, we address the core issue of **client data distributions evolving over time** and being **cross-task non-IID**. We simulate unique FL scenarios using two data partitioning strategies, and validate MultiFCL's performance under extreme heterogeneity conditions. In the future, we will explore more FL-specific mechanisms based on your suggestions.\n\n> **W2: Figure 1 modification**\n\nThank you for your correction. We will further optimize Figure 1 to fully illustrate the workflow of MultiFCL and improve readability.\n\n> **W3: Component workflow**\n\nBelow is our supplementary description for these four components:\n\n**Multi-Adapter Fine-Tuning:**\n\n- **Functionality:** Inserts lightweight adapters into each layer of the PTM, fine-tuning them locally only on the first task. The adapters enable the PTM to rapidly adapt to downstream tasks, and are frozen for subsequent tasks to prevent catastrophic forgetting.\n\n- **Connection:** Serves as the PTM fine-tuning module for the entire framework. Its output features are reused by all subsequent components. The freezing mechanism preserves the PTM's generalization capability, providing stable representations for subsequent cross-task knowledge fusion.\n\n**Multi-Modal Prototype Initialization:**\n\n- **Functionality:** Selects the most relevant old-class semantic features and fuses them with the features of the new class sample using cross-attention to initialize the new class prototype. This process incorporates implicit bias from historical tasks, enhancing the discriminative power of the new class representation.\n\n- **Connection:** Relies on features extracted by the PTM after adapter fine-tuning. Injects knowledge from old tasks into new class initialization, avoiding old knowledge forgetting caused by random initialization. Works together with the adapter freezing mechanism to ensure model stability.\n\n**Multi-Scale Feature Learning:**\n\n- **Functionality:** Divides the PTM into $M$ modules, with each module outputting features at different scales. Enforces experts to pull sample features closer to their corresponding class prototypes. Clients upload locally computed multi-scale prototypes to the server for aggregation, forming global class prototypes used in the next round of training.\n\n- **Connection:** Input relies on features extracted by the PTM after adapter fine-tuning. Class prototype updates are based on the multi-modal initialization results. The aggregated prototypes from the server fuse hierarchical features from different clients and are downloaded back to clients for use in multi-teacher self-distillation training.\n\n**Multi-Teacher Dynamic Self-Distillation:**\n\n- **Functionality:** Treats the first $M-1$ experts as teachers. Introduces an uncertainty weighting mechanism that dynamically adjusts distillation weights, suppressing interference from high-uncertainty experts and enhancing the contribution of low-uncertainty experts.\n\n- **Connection:** Takes the outputs of the experts from multi-scale feature learning as input. Solves inference efficiency issues through knowledge compression. The dynamic weighting mechanism prioritizes fusing deep, abstract knowledge, avoiding the influence of shallow noise, further enhancing the discriminative power of the final expert.\n\nThe overall workflow can be summarized as:\n\n- **Phase 1: Task Initialization**: Clients train adapters on the first task and upload them to the server for aggregation. Aggregated adapters are distributed back to clients and frozen.\n\n- **Phase 2: New Task Processing**: Clients extract features using the frozen adapters, initialize new class prototypes using semantic features from old tasks. Multi-scale experts are trained using extracted multi-scale features for multi-teacher self-distillation. After local training, clients upload their locally computed prototypes to the server for aggregation.\n\n- **Phase 3: Continual Learning Loop**: Global prototypes are fed back to clients to guide learning on new tasks. This forms a closed loop: Local Training $\\to$ Prototype Aggregation $\\to$ Knowledge Feedback.\n\n> **W4: Adapter fine-tuning**\n\nWe will supplement references related to LoRA. Although similar, the fine-tuning parameters of adapter and LoRA are different. Adapter introduces small neural network modules *independently* into various layers of the model, separate from its main structure. Only these adapter parameters are updated during fine-tuning. In contrast, LoRA introduces low-rank matrices into the model's weight matrices. These low-rank matrices act as modifications applied to the original weight matrices, adjusting them during computation.\n\nRegarding the issue of adapter freezing, as demonstrated in Figure 6 of the paper (Appendix A.8), continuously training the adapters leads to a rapid and severe decline in accuracy. This is because the PTM lies in its generalizable features acquired through large-scale pre-training. Fully fine-tuning the PTM causes the learned feature space to shift towards new tasks, damaging performance on old tasks and resulting in catastrophic forgetting.\n\nAlthough freezing the adapter might potentially limit cross-domain adaptation capability, this challenge is effectively addressed through the synergistic combination of **multi-modal prototype initialization** and **multi-scale expert distillation**. Specifically, we leverage semantic features to generate the initialization values for new class prototypes through cross-attention mechanisms, thereby transferring abstract knowledge from old domains to new ones. Concurrently, the PTM is partitioned into multi-level expert modules. Uncertainty-weighted dynamic distillation facilitates the reorganization of features across these experts, and the final expert integrates knowledge from all scales. This coordinated approach significantly enhances both generalization capability under domain shifts and task-specific adaptability.\n\n> **W5: Multi-modal knowledge introduction**\n\nThe textual information used in the paper to initialize new class prototypes is derived directly from the PTM’s own text encoder. It generates semantic features of old-task class labels through this encoder, without introducing any additional external corpora or knowledge bases. This ensures **the text prototypes solely carry category semantic information directly relevant to the visual task**.\n\nAdditionally, we conduct ablation experiments by removing multi-modal initialization (`w/o init.`), as follows:\n\n||6|4|2|0.5|0.1|0.05|\n|-|-|-|-|-|-|-|\n|CLIP|64.3|63.6|62.7|65.0|63.3|60.3|\n|MFCL|30.4|24.6|15.9|32.9|31.4|30.7|\n|LANDER|20.3|19.8|18.7|34.8|33.4|32.1|\n|PiLora|70.5|63.8|63.6|69.6|63.8|63.0|\n|MultiFCL-w/o init.|69.7|68.9|68.2|69.3|67.6|65.8|\n|MultiFCL|71.0|70.3|69.7|70.7|68.9|67.9|\n\nTable 1: Comparison results of last accuracy on CIFAR100 with 10 tasks in both quality-based partitioning and distribution-based partitioning.\n\nIt can be observed that although removing multi-modal initialization leads to performance degradation, the interaction of other components in MultiFCL still enables the model to maintain strong performance, consistently outperforming other methods.\n\n> **W6: Multi-scale feature learning**\n\nEach divided module of the model acts as an independent expert, outputting multi-scale prototypes and **does not perform classification tasks**. Classification tasks are performed **by the final layer expert** after the model is trained via multi-teacher self-distillation. Simultaneously, class prototypes are obtained through iterative updates (Eq. 5) rather than repeated training, with the computation involving only the weighted averaging of feature vectors.\n\nFurthermore, MultiFCL does not impose restrictions on the number of experts established. In our experiments, we set up 4 experts, and this computational overhead is manageable. In scenarios with extreme resource constraints, clients can choose an appropriate number of experts by balancing performance and computational resources.\n\n> **W7: Multi-teacher dynamic self-distillation**\n\nAs shown in the ablation experiments of the paper (Tables 3 and 6), after introducing this component, the model achieves the highest accuracy across all data partitioning settings on CIFAR100. This proves its effectiveness in consolidating knowledge from multiple experts and improving final classification performance.\n\nRegarding computational overhead, multi-teacher self-distillation can reduce the number of experts that need to be computed during testing. As described in W6, this avoids the need for every expert to perform classification tasks. During the testing phase, **only the final single-expert model needs to be executed, significantly reducing inference overhead**. Overall, this design achieves a balance between inference efficiency and classification performance through knowledge consolidation during the training phase."}]

Author rebuttal:

Meta-review: This paper introduces MultiFCL, a federated continual learning framework that leverages pre-trained models with lightweight adapters, multi-modal prototype initialization, multi-scale feature learning, and multi-teacher dynamic self-distillation to balance stability and plasticity. Reviewers agreed that the method is novel, well-motivated, and supported by extensive experiments across diverse datasets, including federated and cross-domain settings. Concerns were raised about reliance on CLIP backbones, missing efficiency evidence, and unclear component motivations, but the rebuttal provided new experiments, statistical analyses, computational cost measurements, and detailed clarifications that addressed most issues. Some reviewers still noted heavy dependence on PTMs and computational overhead, but these were judged to be acceptable trade-offs given the strong empirical gains. Overall, the paper offers a technically sound and innovative contribution to FCL, with convincing results and thorough ablations
Decision: Accept (poster)