Fed-Duet: Dual Expert-Orchestrated Framework for Continual Federated Vision-Language Learning
Authors: Tao GUO
Junwei Chen
Laizhong Cui
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Federated Continual Learning
Prompt learning
Vision-Language model
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Pretrained vision-language models (VLMs), such as CLIP, have shown promise in federated learning (FL) by bringing strong multimodal representations to edge devices. However, continual adaptation remains a core challenge in practical federated settings, where task distributions evolve over time and data remain non-IID across clients. In this emerging area, recent works adopt parameter-efficient fine-tuning (PEFT) as a lightweight way to reduce communication overhead, yet they fail to preserve satisfactory performance under continual learning conditions. Meanwhile, traditional federated continual learning (FCL) methods lack the capacity to maintain cross-modal alignment crucial to VLM performance. We introduce Fed-Duet, a novel Dual Expert-orchestrated framework for efficient federated continual learning in vision-language models. Fed-Duet features a dual-expert adaptation mechanism, combining server-coordinated semantic prompts with client-personalized modular adapters. These pathways are dynamically fused via a cross-attention mechanism, enabling effective knowledge transfer while preserving multimodal alignment and mitigating forgetting. We evaluate Fed-Duet across multiple challenging continual learning tasks in federated vision-language settings and demonstrate that it achieves superior performance and stability compared to existing approaches. Our work highlights the importance of coordinated expert composition in enabling scalable and robust multimodal continual learning. The code is available at https://github.com/cocogt96/Fed-Duet.

Published as a conference paper at ICLR 2026
FED-DUET: DUAL EXPERT-ORCHESTRATED FRAME- WORK FOR CONTINUAL FEDERATED VISION-
LANGUAGE LEARNING
Tao Guo', Junwei Chen & Laizhong Cui²
1 Cooleege of Computer Science and Software Engineering, Shenzhen University
WeeBank Instituite of Financial Technology, Shenzhen University
cocogtêszu, www.ttessuueuuu.c..mmm edu.cn, 2023290205Bemai www...s..e.................... www.szzu..uc.... cuilz0szu. eetesssuueduu... en
ABSTRACT
Pretrained vision-language models (VLMs), such as CLIP, have shown promise in federated learning (F1.) by bringing strong multimodal representations edge devices. However, continual adaptation remains a core challenge in practical fed- erated settings, where task distributions evolve over time and data remain non- IID across clients. In this emerging area, recent works adopt parameter-effiiiet fine-tuning (PEFT) as a lightweight way reduce communication overhead, yet they fail to preserve satisfactory performance under continual learning conditions. Meanwhile, traditional federated continual learning (FCL) methods lack the ca- pacity to maintain cross-modal alignment crucial to VLM performance. We in- troduce Fed-Duet, a novel Dual Expert-orchestrateed framework for efficient fed- erated continual learning in vision-language models. Fed-Duet features dual- expert adaptation mechanism, combining server- server-coordinated semantic prompts with pereonaleeso modular adapters. These pathways dynamically fused via cross-attentee mechanism, enabling effective knowledge transfer while pre- serving multimodal alignment and mitigating forgetting. We evaluate Fed-Duet across multiple challenging continual learning tasks in federated vision-language settings and demonstrate that achieves superior performance and stability com- pared to existing approaches. Our work highlights the importance of coordinated expert composition enabling scalable and robust multimodal continual learning. The code is available at FedDuet.
INTRODUCTION
Federated Learning (FL) has emerged as a key paradigm for learning from sensitive and siloed data by enabling collaborative model training across decentralized data sources while preserving privacy (McMahan et al., / Kairouz et al. 2021 Li et al. 2020 Zhao et al.) 2018). Mcanwhile, Large-scale Vision-Language Models (VLMs), pretrained on web-scale data, such as CLIP (Radford et al., 2021), have revolutionized multimodal AI by demonstrating unprecedented capabilities in zero-shot generalization and cross-modal understanding (Jia et al.) 2021 et al.) 2022. Du et al.) 2022). Integrating VLMs into the FL paradigm offers great potential for empowering edge devices with advanced multimodal intelligence (Ren et al. 2025). while also introducing new challenges in downstream learning and adaptation.
The most immediate challenge in realizing this paradigm is the large-scale of modern VLMs. Their massive size makes the full-model fine-tuning required in each federated round prohibitively pensive due to immense communication costs. To surmount this bottleneck, Parameter-Ffffiiia Fine-Tuning (PEFT) techniques (Houlsby et (2019) Lester Lester al., [021] Feng et. 2023) Li al., Yu et al., 2024), such as adapter-based and prompt-based methods, have emerged as an es- sential solution. By freezing the vast VLM backbone and only training and communicating a small fraction of parameters, these methods drastically reduces the communication overhead (Lu et al. 2023
2024
Guo et
2023b 2024).
*Corresponding author.

---

Published as a conference paper at ICLR 2026
Despite recent advances in efficiency, real-world edge environments remain highly challenging: tasks evolve continuously and client data exhibit non-IID distributions. (Kirkpatrick et al. 2017] Aljundi et al. 2019: Belouadah & Popescu 2019 Waan et al. 2024). These dynamics give rise to the paradigm of Federated Continual Learning (FCL., which seeks to continually adapt models to new
knowledge without catastrophic forgetring, while also addressing the inherent data heterogeneity of
federated networks (Lopez-Paz & Ranzato 2017; Zhao et al., [020] Dong et al. 2022 So et al.,
2022). A solution is therefore needed address these challenges effectively.
However, directly applying traditional FCL methods to this new context is problematic because they are fundamentally ill-suited for VLMs. First, their reliance on full-model update paradigm is computationally prohibitive for large-scale models and risks catastrophically overwriting pre- trained knowledge. Second, their uni-model perspective ignores the specialized, dual-stream nature of VLMs, thereby threatening the delicate cross-modal alignment that is central to their capabilities.
Furthermore, in the emerging field of federated vision-language model learning, most existing works have focused applying PEFT techniques as lightweight solution to reduce communication over- head, yet these methods introduce new challenges when applied under continual learning settings. 1) First, applying a singular PEFT strategy leads to adaptation imbalance. Relying solely on high-level prompts may fail to capture client-specific nuances, while using only low-level adapters can weaken global semantic consistency. 2) Second, aggregating sparse and heterogeneous PEFT updates across clients risks disrupting the VLMS' intrinsic cross-modal alignment, which is essential for unified vision- language understanding. These challenges highlight the need for coordinated chestration mechanism that can effectively integrate diverse client adaptations while preserving the model's semantic integrity. This leads our core research question:
How can we design an orchestrated framework that efficiently addresses adaptation imbalance and cross-modal misalignment federated dision-languge learning?
To overcome the above challenges, we propose Fed-Duet, a novel Dual-Exper orchestrated frame- work for FCL in VLMs. Fed-Duet employs a dual-expert architecture that orchestrates two comple- mentary guiddng prompts for semantic guidance and modular adapters for fine-grained task specialization- whose outputs are dynamically integrated ensure coherence and adaptability. Our main contributions are summarized as follows:
We pioneer a paradigm shift in federated VLM learning by identifying a critical gap be- tween the latest PEFT- based approaches and the need for continual adaptation as tasks evolve. By filling this gap, we unlock greater potential for retaining old knowledge while acquiring new insights, paving the way for more sustainable federated VLM learning.
We propose Fed-Duet, a novel framework designed specifically for federated continual learning in CLIP-like VI.Ms under data heterogeneity. Fed-Duet enables effective knowl- edge transfer by synergizing two complementary expert pathways, while simultaneously maintaining multimodal alignment and mitigating forgetting.
We validate the effectiveness of Fed-Duet through extensive experiments on challenging federated VLMs and FCL benchmarks. The superiority of our results offering a scalable solution for lifelong adaptation in federated vision-language settings.
RELATED WORK
2.1 CLIP-LIKE VLMS IN FEDERATED LEARNING
Pre-trained Vision-Language Models (VLMs) like CLIP (Radford et al. 2021) offer promising foundation for Federated Learning. For instance, CLIP2FÍ. (Shi et al. 2024) demonstrates this by leveraging frozen CLIP model as a "teacher" to guide smaller models, specifically tackling data heterogeneity and long-tailed distributions. However, their massive size makes full-model aggrega- tion infeasible due to prohibitive communication costs. To overcome this bottleneck, the community has adopted PEFT techniques, prominently featuring
and adapter-tuning (Houlsby
et al. 2019 Lester al. 1 Li & Liang] 2021 Hu al. 2022).
Prompt-tuning has emerged as the dominant PEFT paradigm for VLMs. The concept was popu- larized in centralized learning by methods like CoOp (Zhou al. 2022bla), which demonstrated
2

---

Published as a conference paper at ICLR 2026
that optimizing few learnable prompt vectors could effectively steer frozen VLM. The commu- nication efficiency of this approach made a natural fit for FL. Building on this idea, subsequent works such PromptFL (Guo et al., 2023b) and pFedPrompt (Guo et al. 2023a have proposed federated prompt-tuning strategies, leveraging global aggregation and personalizate lizatio,,z respectively. In parallei, adapter-tuning has shown promising results in federated settings, with methods like Fed- CLIP (Lu et al. 2023) demonstrating that communicating small, trainable adapter modules can also successfully adapt VLMs in a federated manner.
While these PEFT paradigms are effective in alleviating training or communicating costs, they have primarily focused on static FL scenarios. This remains an open and challenging question of how to leverage and potentially synergize them in a continually evolving, non-stationary FCL environment.
2.2 FEDERATED CONTINUAL LEARNING
Early efforts in Federated Continual Learning (FCL) often involved directly adapting central- ized Continual Learning (CL) strategies, such as rehearsal- -assed iCaRL (Rebuffi et al. (017]) or regularization-base -baee LwF (Li & Hoiem] (017) and EWC (Lee et al., 2017). However, as these methods were not designed to handle the statistical heterogeneity inherent to federated networks, they typically resulted in compromised global model performance.
This limitation motivated the development of native FCL frameworks with more sophisticated mech- anisms. Foundational works like FedWeIT (Yoon et al. 2021 focused on parameter-level transfer, introducing decomposition scheme where clients selectively weight sparse parameters from others to mitigate negative interference. Alternatively, FedKNOW (Luopan et al. (023) proposed client-centric gradient-level solution that extracts critical model weights and integrates restored gradients to guide learning without interference. More recently, the paradigm has shifted towards nompptbbseed adaptation. Methods like Fed-CPrompt (Bagwe al. 2023) demonstrate that com- municating only lightweight prompts, guided by a dedicated contrastive loss to tackle heterogeneity and asynchronicity, can achieve excellent performance with high efficiency.
While these native FCL methods represent significant advances, their fundamentally uni-modal de- signs are ill-suited for VLMs. They risk disrupting the delicate cross- modal alignment that essen- tial to VLM capabilities. This creates critical for new FCL framewwor designed specifically to learn continually while preserving this vital cross-modal integrity.
2.3 MIXTURE-OF-EXPERTS
The sparse Mfxture-of- (MoE) paradigm, matured through foundational works like the Switch Transformer (Fedus et al. 2022) and state-of-the-art models like GPT-4 (Achiam et al.) 2023), has recently been applied to enhance centralized Continual Learning. For instance, the work by Yu et al. (Yu et al., 2024) successfully uses MoE-Adapters to learn sequential tasks in VLMs, employing selector preserve zero-shot capabilities. However, such centralized MoE-for-CL ap- proaches are fundamentally ill-suited for the Federated Continual Learning setting. They would re- quire communicating entire expert modules complex selector mechanisms, leading prohibitive communication costs and failing to address client data heterogeneity.
While some reseraches have attempted to incorporate MoE paradigm into FCL setting, to bridge the gap, such as MoAFCL (Zhang & Liu 2025), they focus solely on applying MoE to server-side adapters, overlooking the crucial role of semantic guidance, limiting their adaptability in Non-IIID scenarios. This presents critical research opportunity for unified architecture that synergizes both semantic guidance and parametric specialization to create robust, communication-ffficenn FCL framewor,, which our work aims to address.
3 METHODOLOGY
Preliminaries and Problem Formulation. We consider Federated Continual Learning setting composed of a central server and C' clients, C = (ct,...,cc). Each client observes a private, sequential stream of tasks {D The objective in this challenging setting is for each client to leam an effective model, that performs well on its cumulative history of tasks by leveraging both shared knowledge from the federation and its own evolving, local experience.
3

---

Published as a conference paper at ICLR 2026
Federtted Knowledge Orohestrator
Local Training
Knowledge Repository
Smardi
T Shared
Server
Top selection
Sahured
Satured2
C thertion
Multi Head
Lihared
Sharedd!
Client
Sshureel
Tirrll
shuredi
sharedt
Stccat
Frmape
Iataat
Semanfic Erparts
Semamic Ergrrts
Parametric! Experts
Parametricl
Experts
Client
Client
Duall-Channd Adaptation Duet
Attention
Gating
Specialization
Parametric
Parametric Exparts
Task sequences
Pshrred
Pannal
Plucal
Plncad
Task
Task 2
Task
Stecel
Schozed
Sehared
Shared
FedDuet Training overview
Semantic Experts
Guidance
Semontic
Router
Upload & Downlood
Image Encoder
Text Encoder
Adaptive
Gating
Frozen parameter
Trainable parameteer
(a) FedDuet Training Overview
(b) Client Local Training
The architecture of Fed-Duet. (a). FedDuet Overview, Interaction between Clients and Central Server. The server-side Federated Knowledge Orchestrator maintains Knowledge Repos- itory and employs an adaptive gate to dispatch Shared Semantic Experts based on client features. (b). Detailed Local Training. The client-side Dual- Duet adapts via two complementary pathways: the Semantic Pathway fuses Local and Shared experts via Cross-Attention Gating for semantic guidance, while the Parametric Pathway fine-tunes adapters for feature specialization.
Figure I:
Framework Overview. As illustrated in Figure I1 Fed-Duet orchestrates continual adaptation through two synergistic modules. 1) The server-side Federated Knowledge Orchestrator (Figure Tap addresses knowledge coordination by leveraging global repository and adaptíve gating mecha- nism to dispatch tailored Shared Semantic Experts to clients. 2) The client-side Dual-Exper Duet (Figure [b) resolves adaptation imbalance via two parallel pathways: Semantic Pathway dynami- cally fuses local and shared prompts through Cross-Attention Gating to provide semantic guidance, while a Parametric Pathway adapts modular experts for fine one-grained feature specialization. This ar- chitecture is explicitly designed to preserve cross-modal alignment and is further regularized against catastrophic forgetting by two auxiliary losses, modil and Lxtaiility
3.1 FEDERATED KNOWLEDGE ORCHESTRATOR
A key innovation of Fed-Duet lies in redefining the server's role from simple aggregator to an intelligent knowledge orchestrator.
The Orchestrated Knowledge Repository. To ensure the Global Prompt Pool, P = (pI..... Pk a semantically diverse repository from the outset, we forgo random initialization. Instead, we first derive K conceptual anchors by performing K-Means clustering the word embeddings of large vocabulary (e.g., ImageNet- class names), yielding set of centroids (cI...., where each C₁ E
Each prompt ps E RLXD then constructed using textual template "a photo [CLS]", The embeddings for the static context tokens ("a", "photo", "of"" use their pre-trained values, while the embedding for the learnable [CLS] ] token is initialized directly with its corresponding centroid C₁- This approach grounds each prompt in meaningful semantic concept while providing natural linguistic structure.
Adaptive Dispatch Mechanism. To enable efficient knowledge transfer without broadcasting the full repository, we introduce an Adaptive Gating network, ge. This module learns to dispatch opti- mal experts based on a client's crivacy-preservin feature summary, f.. The summary is computed global statistic derived through batch averaging, ensuring that individual data is exposed.
4

---

Published as a conference paper at ICLR 2026
Furthermore, demonstrated in Sec. 4.2, this mechanism effectively preserves privacy while main- taining robustness under Differential Privacy (DP) noise injection. Guided by the client's training feedback, the gating is optimized via loss- weighted binary cross-entropy (BCE) objective:
Cgaate Σ
(.CCCC ye).
(1)
OES,
where S, is the participant set in round r. The weight 1/(C +é) prioritizes expert selections that yield lower client losses, ensuring the server learns distribute the most effective knowledge.
3.2 DUAL-EXPERS DUET FOR CLIENT ADAPTATION
Our client-side module is built upon a Dual- Dual-Expert principle, consisting of two complementary adaptation pathways that disentangle semantic alignment from parametric feature transformation.
Semantic Experts Pathway. This pathway enables semantic adaptation through learnable prompts, which function as semantic experts fused via a Dual-Stream Cross-Attention module. Given input image feature, the model simultancously attends to private Local Semantic Ex- pert that captures client-specific semantics and a set of server-provider Shared Semantic Experts representing generalizable semantics. The two sources yield separate expert logits (logitS and shrea)) which are combined through a weighted fusion:
Logits
jmnl logits + (1 X) logits
This mechanism allows the model to dynamically balance personalized and shared semantic guid- ance per-sample basis (details in Appendix A.4).
Parametric Experts Pathway. While prompt-based experts adjust semantic alignment, they do not directly transform internal features. To provide this complementary capability, we introduce an adapter-based pathway offering parametric experts for fine-graing aiue-grained representation adaptation. A server-aggregated Shared Adapter remains active to supply stable and generalizable feature backbone, whereas private Local Adapters are selectively activated via Top-k router for efficient, on-demand personalization. This dual-path path design forms dedicated parametric routes for robust generalization and targeted specialization, achieving balanced adaptation.
Synergy of Dual Expert Pathways. The two expert pathways reinforce each other in bidirectional loop. The stable feature backbone established by the parametric pathway enables more precise semantic guidance from the semantic pathway, while the refined semantic signals provide clearer contextual cues that improve the routing of parametric experts. This continuous interplay between foundational stability and semantic refnnmment underpins the effectiveness of our framework.
3.3 OVERALL TRAINING STRATEGY
Progressive Decoupled Optimization. To resolve optimization conflicts and realize the bidirec- tional synergy between the two expert pathways, we introduce a Progressive Decoupled Optimiza- tion strategy. It first trains parametric experts exclusively to establish stable feature foundation, then freezes them while training semantic experts to provide precise semantic guidance. This pro- gressive scheme ensures that stable parametric base empowers semantic guidance, which in turn improves subsequent parametric specialization.
3.4 SYNERGISTIC MULTI-OBJJECTIVE LOsS
The client-side training is guided by composite loss function, Letiens designed to pursue multiple objectives simultaneously. The overall objective is defined as:
Leien = LCE + a& + T) .nodal + Saability
where the primary objective is standard cross-entropy loss (LCE), supplemented by a balancinng loss (Lmms) for the experts, well two auxiliary losses, moxil and Luability to address key FCL challenges. Specifically, we detail the two loss components as follows:
Routing Consistency Loss (Ceross modal) for Alignment. To counteract the tendency of standard MoE layers to disrupt a VLM's inherent vision-language alignment, we introduce routing con- sistency loss inspired by CLIP's contrastive objective. This loss enforces multimodal alignment by
5

---

Published as a conference paper at ICLR 2026
ensuring the expert routing for an image is consistent with that of its corresponding text. The loss is formulated as:
Leross modal
+
where S is a similarity matrix computed from the routing distributions of image-text pairs within batch, r is temperature hyperparamete that controls the sharpness of the distribution, and y represents the ground-truit truth labels indicating the correct image-text pairings. The symmetric cross- entropy objective guides the experts toward learning modality-invarianl representations.
Expert Stability Loss (Cstabilitt) for Anti-Forgeetin ploy an expert stability loss that acts as form of knowledge distillation on the routing policy. This loss preserves learned routing behaviors by regularizing the current policy to remain close to histor- ical ones. defined using KL divergence as:
To mitigate catastrophic forgetting, we em-
(5) where (t) is the expert routing distribution for the current task and p(t-1) the historical policy from previous tasks, maintained on a per-layer basis using an exponential moving average. This regularization provides a robust mechanism against forgetting at the expert level.
Cauaiility = [KL(P 1)
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets and Task Settings. We evaluate our framework on three benchmarks: CIFAR- 100 (Krizhevsky & Hinton| 2009) and Tiny-ImageNet (Le & Yang 2015) for Class-nncremental Learning (partitioned into 5 and 10 tasks), and DomainNet (Peng et al., 2019) for Domain- Incremental Learning. The DomainNet setup, where each domain constitutes new task, rigorously tests the model's 's robustness against distribution shifts.
Evaluation Metries. Overall performance is measured by Average Accuracy and Last Accuracy, while knowledge retention is captured by the Forgetting Measure. To assess the plabiltty-plastiityy trade-off, we report Stability, Plasticity, and Continual Utility. Cross-modaa aligmment is evaluated via Recall@ K and Alignment Score. Finally, to rigorously assess defense against privacy attacks, we employ Peak Signal-to-No -ooisee Ratio (PSNR) and Structural Similarity Index (SSIM) quantify image reconstruction quality. Formal definitions are provided in Appendix A.3.1
Baselines. To comprehensively evaluate our framework, we compare against a range of bench- marks from three main categories: 1) Standard FI. Methods, foundational approaches not de- signed for continual learning, used to establish a performance baseline (e.g.. FedAvg (McMahan et al. 2017) and FedProx (Li et al. 2020)); 2) Representative FCL Benchmarks, which include regularization-baat based approaches (Fe (eee et al., 2017). Fed-LwF (Li & Hoiem 2017)), a parameter-transfe method WelT (Yoon et al. (021)). D), and gradient-integration approach (Fed- KNOW (Luopan et al., 2023 ); 3) SOTA PEFT-based FCL Approaches for VLMs, methods that also leverage Parameter-Effiiiee Fine-Tuning method for FCL with large-scale models. This cate- gory includes prompt- based methods like Fed-CPrompt (Bagwe et al. 2023). PFedMoAP (Luo et al.) framework where clients download multiple prompts as experts, and Powder (Piao et al., 2024). which facilitates dual knowledge transfer across clients and tasks using a task correlation matrix to guide prompt aggregation. This group is rounded out by the Mixture- of-AAapter based MoAFCL Zhang & Liu 2025). In addition to the FCL methods, our benchmark includes FedCLIP (Lu et al. 2023), strong PEFT-based method for static Federated VLMs setting.
2024), Mixture-of-Expeeee
Furthermore, to ensure fair comparison against our VLM-native approach, we enhance key base- lines by uniformly equipping them with a CLIP backbone, following the protocol of MoAFCI. This includes methods like FedWelT, FedKNOW, and other ViT-based approaches such as Fed-CPrompt and Powder. This creates a level playing field for all comparisons.
Implementation Details. Following previous research, we employ pre-trained CLIP model as the VLM backbone across all experiments. Our federated system consists one central server and five clients. Each experiment is repeated three times with different random seeds (42, 2005, 2026) and we report the averaged outcomes. We use the Adam optimizer with a learning rate of 3e-5. All experiments were conducted single NVIDIA RTX 4090 GPU with 24GB of VRAM.

---

Published as a conference paper at ICLR 2026
Table 1: The performance of our method against SOTA benchmarks on representative CIFAR-100 and Tiny-ImageNet datasets under various FCL settings. Non-IID (3) indicates the Dirichlet pa- rameter is set to 3. Avg denotes the average accuracy over all learned tasks, while Last denotes the accuracy on the final task. Best results are in bold. The results show the superiority of our method.
Data partition
IID
Non-IID (0
0.5)
Non-IID (3 0.1)
Tasks
T=5
T=10
Tw5
T=10
Dataset
Method
Avg 1
Last
Avg
Last
Avg
Last T
Avg
Last
Avg
Last
Avg
Last
FodAvg
46.56
21.65
32.86
10.67
35.57
15.67
20.54
5,86
16:90
5.71
10.58
2.06
FedProx
48.33
23.54
37.17
14.18
38.84
17.68
29.17
12.74
19:46
10.23
12.49
2.58
Fed-EWC
49.17
23.54
38.18
13.18
38.56
18.48
26.78
10.22
28.34
22.48
11.98
3.18
Fed-I.wF
$3.69
27.21
47.17
17.88
39.41
19.77
30.92
13.24
21.25
12.19
16. 18
6.94
CIFAR- 100
FodWeIT
72.52
57.52
72.53
62.67
71.54
56.98
72.06
61.55
70.43
54.59
71.58
61.58
FedKNOW
78.47
74.19
79.27
73.59
77.34
73.07
78.76
72.56
77.25
72.95
77.55
72.16
FedCLIP
76.41
77.61
75.97
Fed-CPrompt
73.41
65.87
74.58
65.71
73.07
65.69
74.39
65.03
73.00
65.27
74.20
65.16
Powder
73.32
74.45
66.04
73.11
65.79
74.42
65.42
72.71
65.75
73.79
65.81
pFedMoAP
76.61
68.65
76.80
67. 86
70.02
62.79
70.19
60.16
52.63
47.57
58.46
50.61
MoAFCL
77.93
69.05
77.72
65.84
73.21
64.72
75.51
65,27
65.96
59.78
68.47
60.73
Fed-Duet (Ours)
86.21
79.11
86.22
78.56
84.77
78:48
84.55
77.97
84.58
75.97
84.22
75.88
FodAvg FedProx
40.34
17.80
27.84
9.08
29.83
12.40
19.76
5.83
14,52
5.54
9.44
2.67
43.41
20.34
31.05
11.47
33.75
15.36
22.79
6.87
17:57
8.64
12.84
3.51
Fed.FWC
43.07
21.16
29.85
9.87
31.57
13.59
27.37
9.57
22.73
10.71
16.57
4.64
Fed-LwF
42.76
17.58
51.28
27.18
32.27
14.21
24.08
7,86
19.59
9.39
15.27
5.47
Tiny-
FedWeeT
72.06
62.98
72.03
62. 84
71.38
62.31
71.51
62.78
71.06
62.25
70.45
61.73
ImageNet
FedKNOW
76.24
71.79
77.68
71.58
75.32
70.77
76.82
70,86
74.61
70.01
75.68
70.18
FedCLIP
76.14
68.59
76.67
67.91
74.53
66.79
75.59
66.14
71.94
63,76
73.57
Ped-CPrompt
73.09
65.75
74.27
65.79
72.55
65.93
74.20
65.41
72.48
65.47
14.16
65.31
Powder
74.26
65.95
72.77
65.75
pFedMoAP
74.47
67 28
75.06
66.29
69 96
61.72
70.42
61.46
55.47
48.45
57.87
52.42
MoAFCI.
74.16
65.30
74.17
64.24
70.27
62.17
71.39
61.52
62.51
59.47
66.84
59.33
Fed-Duet (Ours)
82.60
77.44
83.52
75.85
81.32
75.21
81.75
74.81
80.48
74.03
81.56
73.37
4.2 OVERALL PERFORMANCE
Performance Comparison. As shown in Table 1 Fed-Duet demonstrates clear superiority across three key dimensions. 1) Absolute Accuracy: achieves the highest final accuracy, outperforming the strongest baseline (FedKNOW Luopan et al. (2023) by significant 6.67% in the challenging CIFAR-100 scenario (3 0.1, T=10). 2) Data Heterogeneity Stability: Fed-Duet is exceptionally robust to Non-III) data. While the performance of advanced methods like pFedMoAP (Luo et al.) 2024] degrades by 24% under severe heterogeneity, our method's accuracy drops by a mere 2%. 3) Continual Learning Stability: As visualized in Figure 2 Fed-Duet also shows superior resistance to catastrophic forgetting, maintaining a high and stable performance trajectory as new tasks are introduced. These advantages validate that our dual-expert architecture effectively balances client specialization with robust knowledge retention.
Evaluation Domain Shift. To assess robustness against se- vere domain shifts, evaluate our method on DomainNet in setting.
Table 2: Performance comparison on DomainNet. Best results are in bold. The blue values (A) indicate score lower than our method. Fed-Duet demonstrates strong domain general- ization capabilities.
domain-incremental As shown in Table 2 Fed-Duet achieves new SOTA performance, with an average accuracy of 68.47%. This marks a substantial improvement of 5.64% over the strongest baseline. This result underscores the effectiveness of our dual-expert adaptation, which preserves general knowledge while specializing in new visual domains to achieve strong generalization.
Method
Avg Ace
A
Last Acc
A
FedWeIT FedKNOW
62.20
-6,27
58,54
-7.51
56,09
-1..3
54.25
-1.88
FedCLIP
62.83
-5.64
60.04
-6.01
Fed-CPrompt
56.16
-12.31
$4.38
-11.67
Powder
56.14
-12.33
54.35
-11.70
PFcdMoAP
59.98
-8.49
56,35
-9.70
MoAFCL
60.92
-7.55
52.52
-13.53
Fed-Duet (Ours)
68.47
66.05
Evaluation on Cross-Modal Alignment. Table showws that stronger cross-min modal alignment leads to clear gains in retrieval accuracy. After the final task, FedDuet achieves the best I2T and T21 Re-

---

Published as a conference paper at ICLR 2026
PRENNETT
FdRKNww
FedCLIP
PrdChoopl
MAARL
Ponder
Fed- Shunt
CIFAR-100 tuska)
CIFAR-100 tasks)
Tlay-ImageNel (5 taska)
Tlay AmmaecNet (10 taska)
95
85
90
Accuray
80
75
70
75
70
75
70
65
55
60
60
60
20
40
50
80
100
50
50
70
100
40
80
120
160
200
20
60
100
140
200
Classes
Classes
Classes
Classes
Figure 2: Performance trajectory of our method compared to SOTA benchmarks as the number of classes increases. The plots show accuracy as a function of the number of learned classes under or 10 task scenarios. Our method, Fed-Duet, consistently outperforms all baselines, demonstrating superior resistance to catastrophic forgetting as tasks accumulate. Additional experimental results under various settings are presented in Appendix A.5 with details.
Table 3: Cross-modal retrieval performance on CIFAR-100. Metrics include image-to-text (I2T) and text-to- rext-to-image (T2I) Recall K. Best results are in bold. The blue values (A) indicate a score lower than our method. Results are averaged across 5 clients.
12T
T2T
Method
Avg.
A
R@1 1
R@5
R@5
R@
FedWelT
64.02
86.47
92.17
84.00
99.60
100.0
87.71
-4.74
NoW
64.02
86.47
92.17
84.00
99.60
100.0
87.71
-4.74
FedCLIP
63.96
86.69
91.76
86.60
99.00
99.80
87.97
-4.48
Fed-CPrompp
63.81
86.74
91.91
84.40
98.60
100.0
87.58
-4.87
Powder
63.91
86.94
91.82
85,80
99.40
100.0
87.98
-4.47
PFedMoAP
63.66
87.02
91.95
87.00
99.00
100.0
88.11
-4.34
MoAFCL
63.68
86.69
91.89
85,80
99.20
99.60
87.81
-4.64
FedDuet (Ours)
78.18
96.13
98.33
93.20
99.80
100.0
92.45
call@K, outperforming the strongest baseline by +13.16% (I2T @ 1) and +6.20% (T21 R@ I). The consistent improvements across all Recall @ K metrics indicate that FedDuet learns more discrimina- tive and semantically coherent representations, resulting in better cross-modal generalization under federated continual learning.
Table further explains these improvements. Baseline methods maintain nearly unchanged align- ment scores around 0.06 across tasks, revealing limited ability to preserve cooss-modal consistency. In contrast, FedDuet achieves an average alignment 0.2003 on CIFAR-100, which corresponds to more than 3x improvement compared with the typical baseline level. This substantial and sta- ble margin indicates that our decoupled optimization strategy effectively mitigates forgetting while preserving semantic coherence between modalities throughout the learning process.
Table 4: Cross-modal alignment scores across tasks CIFAR-100. Best results in bold. The blue values (A) indicate a score lower than our method. Results are averaged across 5 clients.
Method
Task
Task
Task 2
Task 3
Task
Avg.
A
FedWeIT
0.0623
0.0578
0.0613
0.0610
0.0608
0.0606
-0.1397
FodKNOW
0.0623
0.0578
0.0613
0.0610
0.0608
0.0606
-0.1397
FedCLIP
0.0625
0.0580
0.0615
0.0611
0.0609
0.0608
-0.1395
Fed-CPrompt
0.0625
0.0579
0.0615
0.0611
0.0608
0.0608
-0.1395
Powder
0.0625
0.0576
0.0613
0.0610
0.0610
0.0607
-0.1396
PFedMoAP
0.0626
0.0579
0.0614
0.0611
0.0609
0.0608
-0.1395
MoAFCL
0.0621
0.0580
0.0615
0.0610
0.0609
0.0607
-0.1396
FedDuet (Ours)
0.1929
0.1825
0.2140
0.2061
0.2061
0.2003
8

---

Published as a conference paper at ICLR 2026
Analysis of Privacy Compatibility and Robustness. To validate Fed-Duet under rigorous Differ- ential Privacy (DP) standards, we evaluate the trade-off between utility and defense by simulating noise injection (details in Appendix A.4.2). 1) Utility Preservation (Table 5la): Fed-Duet exhibits exceptional resilience, maintaining negligible accuracy degradation (< 0.3%) even under high-noise regimes (o 10). 2) Defense against Reconstruction (Table 5b:: Gradient-based attacks confirm that our aggregation mechanism inherently precludes reconstruction. Crucially, the reconstruction metrics for raw features (a 0) are statistically comparable to those under strict DP noise 10), with both yielding near-zero SSIM (= 0.01) and low PSNR (< 9 dB). This demonstrates that our inherent feature provides a primary and sufficient privacy guarantee.
Table 5: Quantitative results for privacy compatibility experiments. (a) Classification accuracy under varying Laplacian noise scales (or). (b) Image quality metries (PSNR and SSIM) derived from gradient-based reconstruction attacks at different optimization steps.
(a) Accuracy under Laplacian Noise (n)
(b) Reconstruction Attack Metries
CIFAR-100
Tiny-ImageNet
No DP (ar
Noicy (ar
10)
Iter.
Avg
Last
Avg
Last
PSNR
SSIM
PSNR
SSIM
86.21
79.11
82.60
77.44
7.62
0.034
7.05
0.017
0.1
85.96
78.86
82.55
76.17
600
8.14
0.012
7,08
0,008
1.0
85.94
78.51
82.53
76.11
800
8.11
0.013
8.04
0.008
10.0
85.91
78.48
82.52
76.25
1000
8.43
0.015
0.008
4.3 ABLATION STUDY
Analysis on Core Compo- nents. We conduct detailed ablation studies of different components on our frame- work, as shown in Table 6 from two central innova- tions of our framework: the dual-expert architecture and the composite loss function. 1) First, we evaluate the architecture by decoupling its pathways, which confirms our adaptation imbalance premise. The semantic-only pathway (Base-w/o PE) is unstable and suffers from high forgetting, while the parametric-only pathway (Base-w/o SE), though more stable, remains suboptimal. In contrast, our dual-expert Fed-Duet (Base) model substantially boosts accuracy, proving the synergistic benefit of combining semantic and parametric pathways. 2) Next, We analyze our auxiliary losses against advanced prompt-based FCL methods. While our Fed-Duet (Base) model is highly accurate, exhibits a higher forgetting score than prompt-based FCL methods. This trade-off is systematically resolved by our two complementary auxiliary losses: the expert stability loss primarily mitigates forgetting, while the routing consistency loss enhances accuracy by enforcing multimodal alignment in the expert representations. The final Fed-Duet (Full) model, benefiting from their synergy, achieves both the highest accuracy and the lowest forgetting score, outperforming the baseline and validating the contribution of our composite loss design.
Table 6: We analyze the performance contribution of Fed-Duet's core components separately. Best results are in bold. The A val- ues denote the performance difference from the full model (blue for lower, red for higher).
Method
Avg Ace f
Forget +
Fed-CPrompt
72.98
-7.45
7.95
+0.13
Powder
72.60
-7.83
8.51
+0.69
PFedMoAP
55.12
-25.31
8.28
+0.46
Fed-Duct (Basc-w/o PE)
64.34
-16.09
11.89
+4.07
Fed-Duct Bascewwo SE)
70.64
-9.79
8.89
+1.07
Fed-Duct (Base) Fed- -udeDuet (Base Camss.modal)
77.96
-2.47
9.22
+1,40
79.09
-1.34
8.96
+1.14
Fed- Duct (Base Lutablity)
79.46
-0.97
8.02
+0.20
Fed-Duet (Full)
80.43
7.82
Analysis of the Stability-laastiiity. Our framework effectively resolves the core -lasstity dilemma. As shown in Figure 3al Fed-Duet simultancously achieves both the highest stability and plasticity scores, a superior balance that contrasts sharply with leading baselines that are forced to compromise one for the other. To formally quantify this ora we introduce the Continual Utility score, unified metric representing the weighted aggregation of Stability and Plasticity. This robust balance is further validated by our framework's 's continual utility score, as shown in Figure (3b] Our method consistently dominates the top score, maintaining significant performance lead across the

---

Published as a conference paper at ICLR 2026
FeL.wC
BoLEWC
Mabillyy
Pariaay
Fablaf
5000
Bod-
MeARCI
No-AFCI
(frenet
Commentattio
Fed Drst
Pedk Door
Truder
HHANCE
280
(a) Stability plasticity
(b) Continual utility
(c) Comm. Cost
(d) Comp. Cost
Figure 3: Analysis of the stability-plastter trade-off off and efficiency CIFAR-100 with 5 incremen- tal tasks. (a) Stability and plasticity scores compared with different methods. Our method excels in both aspects. (b) The continual utility trade-off' for our method across varying values of the hy- perparameter Ç.A larger Ç indicates more emphasis on stability. Darker colors indicate better utility. (c) Total communication cost (MB, log scale) required by each method. (d) Peak GPU memory footprint (MiB) during local client training. Lower values correspond to higher efficiency.
entire range of the trade-off hyperparameter Ç. This result shows that Fed-Duet not only establishes superior balance but also demonstrates remarkable robustness to hyperparameter choices.
Efficiency Analysis. Our framework offers significant advantages in both communication and com- putational efficiency. As shown in Figure 3cl Fed-Duct is not only orders of magnitude more efficient than full- pallmparrettrr tuning methods like Fed-EWC and FedLwF but also reduces overhead by ap- proximately another order of magnitude relative to the PEFT-based MoAFCL. A rigorous baseline selection is crucial, as existing methods are largely incompatible with our research context. For instance, PFedMMAP (Luo et 2024) not designed for continual learning, while Powder (Piao it al. 2024) utilizes a standard ViT backbone. We therefore selected MoAFCL (Zhang & Liu 2025) as our primary baseline since the only framework specifically engineered for the federated con- tinual learning of VLMs like CLIP. This efficiency extends to computation, measured by GPU mem- ory burdens in Figure 3d where Fed-Duet again achieves the lowest footprint. This demonstrates that our dual-expert architecture provides highly resource-efficient solution for FCL,, enhancing continual learning while significantly reducing both communication and computation burdens.
Analysis of Large-Scale Scalability. To rig- orously assess our framework's scalability, we extended the experimental setup to more de- manding 20-client environment. The perfor- mance trajectories, visualized in Figure 4 re- veal two critical insights into Fed-Duet's ' ro- bustness. Firstly, Fed-Duet occupies the high- est accuracy plane across all 10 tasks, consis- tently surpassing all baseline methods. Sec- ondly, its performance slope is notably flat- ter, demonstrating superior resilience against the performance degradation that hinders the baselines. This visual evidence substantiates that Fed-Duet's core mechanism for mitigating catastrophic forgetting remains potent at scale, ensuring sustained high performance in larger and more complex federated systems.
CHEAR-I dleem)
They- (28 dluat)
FoWae
FedCLIP
M:AFCL
FalMaAP
Ped-CProppl
Figure 4: Performance trajectory with large-scale (20-client) setting. Accuracy (z-axis) is plotted across 10 sequential tasks on CIFAR-100 and Iiny-ImageNet. Fed-Duet remains robust under large-scale scenario.
5 CONCLUSION
We present Fed-Duet, a framework addressing Federated Continual Learning for Vision-Language Models. By synergizing server-side Knowledge Orchestrator with client-side Dual-Expert Path- ways, our approach decouples semantic guidance from parametric specialization, effectively resoly- ing the plasticity-stability dilemma. Extensive experiments confirm that Fed-Duet achieves SOTA performance in accuracy and cross-modal alignment. Furthermore, its proven compatibility with rigorous differential privacy standards validates scalable, secure solution for deploying foun- dation models in dynamic edge environments.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT.
This paper adheres to the ICLR Code of Ethics. The research presented in this paper does not involve human participants, animals, or any other ethical considerations. All data used in this study were obtained from publicly available sources. Our method does not involve discrimination, bias, or fairness concemn.
REPRODUCIBILITY STATEMENT.
To ensure reproducibility, we provide our source code, running scripts, and hyperparameter con- figurations. We also include detailed algorithm pseudocode, dataset information, and additional experimental results in Appendix A.5 to facilitate the verification our findings.
ACKNOWLEDGMENTS.
This work has been partially supported by National Natural Science Foundation of China under Grant No. U23B2026 and 6236227,35 Guangdong Basic and Applied Basic Research Foundation under Grant No. 2024B1515040011 Shenzhen Science and Technology Program under Grant No. KIZDD0230923 14809000, and Research Team Cultivation Program of Shenzhen University, Grant No.2023QNT015.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Techni- cal Report. arXiv preprint (tt.:23333.087774,, 2023.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11254- 11263, 2019.
Gaurav Bagwe, Xiaoyong Yuan, Miao Pan, and Lan Zhang. Fed-cprompt: Contrastive prompt for rehearsal-free federated continual learning. arXiv preprint arXiv:2307. 04869, 2023.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 583-592, 2019.
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class-incremental learning. In Proceedings of the IEEF/CVF conference computer vision and paltern recognition, pp. 10164-10173, 2022.
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202. 10936, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Chun-Mei Feng, Bangjun Li, Xinxing Xu. Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8064-8073, 2023.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp- 1364-1374, 2023a.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated par- ticipants cooperatively learn prompts instead of modell-feddeated learning in age of foundation model, IEEE Transactions Mobile Computing, 2355)5519-5 2023b.

---

Published as a conference paper at ICLR 2026
Tao Guo, Song Guo, and Junxiao Wang. Explore and cure: Unveiling sample effectiveness with context-aware federated prompt tuning. IEEE Transactions on Mobile Computing, 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference muchine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- Laiiunla representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, and et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learn- ing, 14(1-2):1-2) 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskkkaaaai al. Overcom- ing catastrophic forgetting in neural networks. Pruceedings of the national academy of sciences, 114(13):3521-3526, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009. Technical Report.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. Stanford University CS231N Course Project Report, 2015.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak -ak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information pro- cessing systems, 30, 2017.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint urXiv:2104.0869) 2021.
Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12151-12161, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imagge pre- training for unified vision- languaee understanding and generation. In International conference on machine learning, pp. 2888-12900. PMLR, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.:22000.00990. 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 40(12))2935-2944 2017.
David Lopez-Paz. and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, volume 30, 2017.
Wang Lu, Xixu Hu, Jindong Wang. and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. arXiv preprint a3X1v:2302.13485, 2023.
Jun Luo, Chen Chen, and Shandong Wu. Mixture of experts made personalized: Federated prompt learning for vision-language models. arXiv preprint arXiv:2410. /0/ 14, 2024.
12

---

Published as a conference paper at ICLR 2026
Yaxin Luopan, Rui Han, Qinglong Zhang, Chi Harold Liu, Guoren Wang, and Lydia Y Chen. Fed- know: Federated continual learning with signature task knowledge integration at edge. In 2023 IEEE 39th International Conference Data Engineering (ICDE), pp. 341-354. IEEE, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcienn learning of deep networks from decentralized data. In Artificiaa intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei. Federated continual learning via prompt- based dual knowledge transfer. In Forty-first International Conference on Machine Learning, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H LammPer. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Bo Zhao, Liping Yi, Alysa Ziying Tan, Yulan Gao, Anran Li, Xiaoxiao Li, et al. Advances and open challenges in federated foundation models. IEEE Communications Surveys & Tutorials, 2025.
Jiangming Shi, Shanshan Zheng, Xiangbo Yin, Yang Lu, Yuan Xie, and Yanyun Qu Clip-guided federated learning on heterogeneity and long-tailed data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14955-14963, 2024.
Jinhyun So, Kevin Hsich, Behnaz Arzani, Shadi Noghabi, Salman Avestimehr, and Ranveer Chan- dra. Fedspace: An efficient federated learning framework at satellites and ground stations. arXiv preprint arXiv:2202.0/267 2022.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-5383, 2024.
Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated contin- ual learning with weighted inter-client transfer. In International conference on machine learning, pp. 22073-12086. PMLR, 2021.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pp. 23219-232301 2024.
Dian Zhang and Bingyan Liu. Moafel: Feature-aware mixture-of-adappee for federated continual learning. In Proceedings of the 2025 International Conference on Multimedia Retrieval, pp.
1804-1813, 1813, 2025.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and ShuTao Xia. Maintaining discrimination and fairness in class incremental learning. In I Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13208-13217, 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806.0000
2018.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16816 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13019):2337-2348, 2022b.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.I ALGORITHM PSEUDOCODE
We provide the detailed pseudocode for the Fed-Duet framework. The overall process is divided into the server-side orchestration (Algorithm I and the client-side dual-expert training (Algorithm 2 This describes the federated learning process for single continual learning task.
Algorithm Fed-Duet: Server-Side Procedure
Initialize: Global Prompt Pool P, Adaptive Gating Net Iø-
2: Initialize: Global Shared Expert Eshared.
3: for each communication round r = 1,2, .... R do
4: Select subset of clients S,..
5: Initialize GateNet buffer + 2.
6: for each client E S, in parallel do
7: Receive feature summary Se from client c.
8: if fe is available then
9:
Select prompt indices idx, + (go
else
11:
[Cold Start]
Select random prompt indices idx..
13:
end if
Send selected prompts (P[i) veidx, and Exhared to client e. Receive updated shared expert Esharre and final loss Le- Add tuple (fe, idxe, Le) to buffer
15:
16:
17:
end for
18:
Aggregate Shared Experts:
19:
Estured <- Enes. Esharod {we is client weight)
20:
Train Gating Net:
21:
Compute loss Lgate - -E (f. Lix..x..)))........ E._c BCE(gg(fo)). lidx.):
Update gø by descending the gradient of Lgate.
23:
end for
A.2 DATASET AND TASK SETTINGS
We evaluate our framework under three distinct continual learning paradigms, each designed to test different facets of model performance. The statistics for all datasets are summarized in Table
Table 7: Statistics of the datasets used in our experiments.
Dataset
Classes
Train
Test
CIFAR-100
100
50,000
10,000
NmageNNt
200
100,000
10,000
DomainNet
345
586,576 (Total)
Flowers 102
102
2,040
6,149
Oxford Pets
37
3,680
3,669
Food101
101
75,750
25,250
Caltech-101
101
3,000
6,000
DTD
47
3,760
1,880
Masse-nncreanttll Learning CIFAR-100 and Imag-Im For this setting, the model must learn new classes over time without forgetting old ones.
Task Construction: For CIFAR- 100 (100 classes), we create scenarios with sequential
tasks of 20 classes each, and 10 tasks of 10 classes each. For Imag--ma (200 classes),
14

---

Published as a conference paper at ICLR 2026
Algorithm 2 Fed-Duet: Client-Side Procedure
1: Input: Shared prompts P., shared expert Esture, current round
2: Initialize: Local prompt plncal, local experts Elocal
3: Compute
feature summary Se from local data D
4: if R/2 then
5: {Stage 1: Adapt Parametric Experts}
6: for local epoch e = .... E do
7:
for batch (a: y) from D do
8:
Compute features using MoE-Adapter (combining Eshured and Elocal).
9:
Compute loss C Lcr aLmoe + LLmss + 7
10:
Update parametric experts parameters.
11:
end for
12: end for
else
14: (Stage 2: Refine Semantic Guidance)
Freeze MoE-Adapter parameters.
16:
for local epoch = ..... E do
17:
for batch (I, 3) from De do
18:
Compute logits using Dual-Stream Cross-Attention.
Compute loss L LeE. Update semantic-experts-relee parameters. 21: end for
19:
20:
22:
end for
23:
end if'
Let Liiall be the average loss from the last local epoch.
25: Send updated Eshared, Limal and fa to server.
create scenarios with 5 tasks (40 classes/task) and 10 tasks (20 classes/task). The class order is fixed for reproducibility.
Data Heterogeneity: To simulate federated environments, we partition data among clients using a Dirichlet distribution over class labels, controlled by B. We test under moderate (B 0.5), severe (B 0.1), and IID settings.
Domain-Increment Incremental Learning on DomainNet. This challenging scenario evaluates the model's ability to adapt to severe distribution shifts, where each task is a new visual domain.
Task Construction: use all six domains from the dataset (Clipart, Infograph, Painting. Quickdraw, Real, and Sketch) sequence of six tasks. The model learns these domains in alphabetical order.
This scenario uses specific protocol to measure cumulative knowl- edge. After training on each new domain, model is evaluated on dynamically ex- panding test set comprising the test splits of all previously learned domains plus the current one (e.g., after training on Sketch, the model is tested on the combined test sets of Real and Sketch). The evaluation uses zero-shot classification head spanning all 345 classes of DomainNet.
Evaluation Protocol:
Multi-Domain Task- lakkeennremetall Learning on Fine-grained Datasets. To further validate the generalizability of our framework, we construct a task-incremental scenario where the model must leam sequence of entirely different datasets, assuming the task identity is known at inference time.
Task Construction: We use a sequence of five distinct, fine lne-grained visual classification datasets, learned in the following order: Flowers OxfordPets Food101 -> Cal- tech101 + DTD. Each dataset constitutes a single, separate task, testing the model's ability acquire and retain diverse knowledge without interference.
15

---

Published as a conference paper at ICLR 2026
A.3 FORMAL DEFINITIONS METRICS AND LOSSES
A.3.1 EVALUATION METRICS
Here, we provide the formal definitions for the standard continual learning metries used to evaluate framework. Let I' be the total number of tasks, and let ai,j be the accuracy of the model on task j after has been trained on task i.
Average Accuracy (Avg Acc) This metric measures the overall performance across all tasks after training is complete.
T
Avg Acc
T
TT
Forgetting Measure (Forget)
This metric quantifies how much the model forgets about past tasks.
Forget
max 0x. -0 -TT
T
KCT
Stability This measures the ability retain knowledge of past tasks. After training on task i, is the average accuracy on all prior tasks:
Stability,
aij
(8)
j=1
Plasticity This measures the ability to learn the current new task. After training on task i, it the accuracy that task:
Plasticity; = 0j
Continual Utility This metric provides
evaluate the trade-off between stability and
plasticity, defined as their weighted linear combination. The hyperparameter Ç E [0, 1] controls the balance, assigning a weight of Ç stability and (1 () to plasticity. A larger ( places more emphasis on stability.
Utility; = Ç Stability, + (1 () Plasticity;
(10)
Retrieval @e This measures the percentage of queries where the correct match appears in the top-K retrieved results. We evaluate both Image-to-Texx (I2T) and Text-to- Imoge (T2I) retrieval at K E {1,5, 10}. For query set of size M:
Hi: rank(i) s K}
Recall @K K
(11)
M
where rank(i) denotes the rank of the correct match for query retrieval capability.
Higher recall indicates better
Alignment Score This quantifies the separation between matched and mismatched image-text pairs in the embedding space:
N s(U, 4))
N
Alignment Score
s(ci, bj)
(12)
N(N
1=1
1=1 jutt
where and t; denote the visual and textual embeddings of the i-th matched pair, s(.,) represents cosine similarity, and N is the number of samples. A higher score indicates better cross-modal alignment.
16

---

Published as a conference paper at ICLR 2026
Peak Signal-to-Noise Ratio (PSNR) This metric is used quantify the reconstruction quality in privacy attacks. It is defined via the Mean Squared Error (MSE) between the original image / and the reconstructed image /:
MAXi MSE
PSNR 10 1og10
where MSE
E 11. 1.)²
(13)
Where MAX, is the maximum possible pixel value of the image, and N is the total number of pixels. lower PSNR indicates higher distortion, signifying better privacy protection.
Structural Similarity Index (SSIM) This metric measures the perceptual structural similarity between two images, considering luminance, contrast, and structure. For an original image and reconstructed image y:
(2 + 22) + C2) SSIM v)
(14)
(n) 2 + (i) + + C2)
where Mz My represent the pixel sample means, a og are the variances, dzy is the covariance, and C1 O2 are small constants for numerical stability. An SSIM value close to 0 implies no structural similarity, indicating successful defense against reconstruction.
A.3.2 DETAILS PRIMARY AND REGULARIZATIO LOSSES
Cross-Entropy Loss (LCE) The primary objective for the main classification task is the stan- dard cross-entropy loss. designed to minimize the dissimilarity between the model's predicted probability distribution and the ground-truth label distribution. The formula is:
LCE = Σ yi log(ūs);
(15)
where C' is the total number of classes, y is the one-hot encoded ground-truth label vector, and ğ the predicted probability distribution produced by applying the Softmax function to the model's final logits. Minimizing this loss guides the model to assign the highest possible probability to the correct class.
MoE Load Balancing Loss (Lmme) To prevent the MoE router from consistently favoring a small subset of experts (a phenomenon known expert collapse). employ auxiliary load-balancing loss. This loss encourages the router to distribute the computational load as evenly as possible across all available experts, ensuring that all experts are sufficiently trained. Following common practice, we define this loss as:
N
Lmoe a N Σ Pe.
(16)
1=1
where N is the number experts and a is a scaling hyperparameter. For a batch of tokens, fi is the fraction of tokens dispatched to expert and P. is the average router probability (the gating value) assigned to expert i over the batch. This objective incentivizes the router to assign higher probabilities (P,) experts that utilized less frequently (low fi), thus promoting balanced expert utilization throughout training.
A.4 IMPLEMENTATION AND ARCHITECTURAL DETAILS
A.4.1 DUAL-STREAM
MECHANISM
To create a dynamic, content-aware fusion knowledge that is native to the CLIP architecture, introduce Dual-Stream Cross-Attention mechanism. This design moves beyond static feature combination by using the visual feature of an input image, Fing. as a dynamic Query to actively interrogate the available textual knowledge. This process unfolds in parallel streams.
17

---

Published as a conference paper at ICLR 2026
Local Knowledge Stream. First, capture client-specific nuances, each client learns private, local prompt pa This prompt is prepended to the class token embeddings and processed by the text encoder to produce the local text feature set, = The local knowledge stream then attends to this private local prompt feature, which serves as both Key and Value. This produces an image-aware local representation and its corresponding expert logits:
logits Rocal Cross-Attention/
Re.
local local)
(17)
Shared Knowledge Stream. Concurrently, evaluate the relevance of the generalized knowledge from the federation, the shared knowledge stream attends the same image query to the set of k received shared prompt features, Ze This computes the shared logits:
logits shared = Pross-AttentionnF... Z, Zs)
(18)
Final Fusion. Finally, to create prediction that flexibly balances local expertise with global con- sensus, the logits from both streams are dynamically fused via a weighted sum, controlled by a hyperparameter A: Logits = À logit + (1 - X) logits
(19)
This mechanism allows the model dynamically weigh the importance of local versus global mantic guidance based on the task and data.
A.4.2 EXPERIMENTAL DETAILS FOR PRIVACY ANALYSIS
In this section, we detail the implementation of the noise injection mechanism used for robustness evaluation and the setup for the gradient-based reconstruction attacks presented in the main text (Sec. 4.2).
Noise Injection Mechanism. To evaluate the system's resilience under striet Differential Privacy (DP) constraints, we adopt the noise injection protocol established in MoAFCI. Specifically, Lapla- cian noise is injected into the aggregated client feature summary fc E R-52 before it serves as input to the server-side gating mechanism. The perturbed feature vector, fc is computed as follows:
j.h fc + where n, ~ Lap(0, o) for j 1,...,,12,
(20)
where is the noise multiplier controlling the scale of the perturbation. This setup allows to simulate varying degrees privacy budgets and assess the trade-off between privacy protection and the utility of the gating mechanism.
Reconstruction Attack Setup. To empirically verify the defense capability against visual leakage, conduct gradient-based feature inversion attacks. The objective is reconstruct the original input image æ solely from its compressed feature representation. We treat this as optimization problem where a dummy image & is iteratively updated to match the target features. Specifically, we employ the Adam optimizer for 1, 000 steps to ensure convergence. The optimization minimizes a composite objective function Latteck consisting of three terms: Cosine Similarity loss to align the feature direction, Total Variation (TV) regularization to encourage spatial smoothness in the reconstructed image, and L2 regularization to penalize pixel outliers.
A.5 ADDITIONAL RESULTS AND VISUALIZATIONS
To further substantiate the claims made in the main paper, this section provides additional visualiza- tions and analyses of our framework's performance.
A.5.1 DETAILED PERFORMANCE TRAJECTORY ANALYSIS
Figure B presents comprehensive visualization of the accuracy trajectory for Fed-Duet against baseline methods across Non-IID scenarios. These plots complement the summary results in Table I of the main paper by showing the performance degradation as more tasks are learned. Across all settings, method consistently establishes the highest accuracy. More importantly, its perfor- mance curve exhibits a significantly flatter slope compared to all basclines, visually confirming its superior resistance catastrophic forgetting, especially in the Non-IID scenarios.
18

---

Published as a conference paper at ICLR 2026
FodkNoWw
TedEEIP
TddCTong
MAATEE
Ponder
Fed DuvejDung
CIFAR-100 tanks)
CIFAR-I00 (10 taske)
Tiny-ImugeNtt tanke)
Tiny-Imagrha tasle)
95
90
85
AIDD 8.5 Accuracy
75
75
70
13
70
56
63
120
60
100
140
CIFAR-I00
CIFAR-I0I
Tiny-EnngeeNa
Tiny-Inageeaa
100
90
85
NID Accursit
80
80
75
75
75
70
65
65
55
Figure 5: Detailed performance trajectories on CIFAR-100 and Tiny-ImageNet. Rows: The top row displays results under the IID setting, while the bottom row shows the more challenging Non-IID (3 0.1) setting. Columns: From left right, the columns correspond to CIFAR- 100 (5 tasks), CIFAR- 100 (10 tasks), Iiny-ImageNet (5 tasks), and Imay-Ima (10 tasks).
Table 8: Multi-domain Task-incremental learning (MTIL) performance five benchmark datasets. The final columns show the average accuracy and the A (blue for lower) compared to our method.
Method
Flowers102
OxfordPets
Food101
Caltech101
DTD
Average
FedWeIT
62.03
53.97
66.35
69.50
71.68
64.71
-8.61
FedKNOW
63.69
53.98
64.81
70.02
72.31
64.96
-8.36
FedCLIP
68.97
64.67
71.83
76.31
79.84
72.32
-1.00
Fel-CPrompt
63.96
54.10
64.95
70.21
72.46
65.14
-8.18
Powder
65.48
58.37
67.87
72.32
76.47
68.10
-5.22
PFedMoAP
66.56
59.58
69.82
73.22
77.77
69.39
-3,93
MoAFCL
63.18
54.09
66.32
71.45
74.85
65.98
-7.34
Fed-Duct (Ours)
70.35
64.97
73.33
77.77
80.17
73.32
A.5.2
MULTI-DOMAIN TASK-INCREMENTA LEARNING
In the main body of our paper, we extensively evaluated our framework under class-incremental and domain-increment learning scenarios. To further demonstrate the robustness and transferability of our proposed Fed-Duet framework, we present an auxiliary experiment this section under the Multi-Domaih Task-Incremen Learning setting.
Analysis of Multi-Domain Task-Incrementall The performance of all methods on this Multi- domain Task-incremental benchmark is presented in Table The results unequivocally show that our Fed-Duet framework achieves the highest accuracy across all five datasets in the sequence. This consistent state-of-the-an performance third, distinct continual learning paradigm provides strong evidence for the generalizability of our approach. It confirms that the superiority of Fed-Duet confined specific problem formulation but is instead robust attribute stemming from its core design, further validating its advanced capabilities in diverse continual learning environments.
A.5.3 AUXILIARY EXPERIMENT: PERFORMANCE WITH COMPRESSED BACKBONE
In our main experiments, we utilized the full CLIP VIT-B/16 model to ensure a fair comparison with prior works. However, practical federated learning scenarios often involve clients with limited
19

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison on CIFAR-100 and Tiny-ImageNet under compression strategy.
(a) CIFAR-100
(b) Lny-lmmgeNet
T=5
T=10
T=5
T=10
Method
Method
Avg
Last t
Avg
Last
Avg
Last
Avg 1
Last f
FedWeIT
38.71
25.21
39.81
24.52
FedWeIT
23.67
16.90
25.64
15.58
FedKNOW
42.08
34.60
46.69
35.86
FedKNOW
28.60
22.86
32.01
24.02
FedCLIP
49.39
34.27
49.39
32.87
FedCLIP
35.15
22.82
35.25
21.18
Fed-CPrompt
34.75
24.10
37.69
23.88
Ped-CPrompt
23.80
15.98
25.36
15.86
Powder
34.65
23.71
37.51
23.55
Powder
24.03
15.86
25.46
15.82
PFedMoAP
37.99
25.28
39.73
25.15
PFedMoAP
27.75
18.26
28.63
17.29
MoAFCL
40.69
27.47
42.29
27.17
MoAFCL
24.81
16,91
25.81
15.68
Fed-Duet (Ours)
61.31
49.04
63.29
46.45
Fed-Duct (Ours)
46.62
34.96
47.58
33.29
computational resources, necessitating more lightweight models. To evaluate the robustness and transferability of our framework under such constraints, conduct auxiliary experiment where all methods are built upon compressed CLIP backbone.
Pruning the CLIP Backbone. We adopt simple yet effective layer-dropping strategy com- press the CLIP backbone. Specifically, we prune the model by removing the first two and last two transformer layers from both the vision and text encoders. While more sophisticated pruning techniques could be explored in future work, this method serves to effectively reduce the model's computational footprint for this analysis.
Sustained Performance with a Compressed Model. As shown in Table 9 Fed-Duet maintains its state-of-the-arr performance even with the compressed backbone, consistently outperforming all baselines. This result demonstrates that the superiority of our framework is intrinsic to its architec- tural design, merely a byproduct of large-scale model. Retaining this performance advantage validates the robustness and practical applicability of Fed-Duet in resource-constrained federated environments.
Table 10: Hyperparameter analysis Fed-Duet on CIFAR-100.
(a) Impact of Experts (E) and Top-K (k).
(b) Impact of K-means clusters.
Top-K (k)
Clusters
Accuracy (%)
Experts (E)
k=2
k=4
k=8
8
86.23
86.17
84.97
32
86.10
85.77
64
86.21
86,21
86.42
128
86.34
16
87.12
86.98
86.75
Table 11: Sensitivity and Ablation Analysis of FedDuet.
(a) Loss Function Weights (a, n, ?
(b) Fusion Coefficient À
Value
?
À
CIFAR-100 (%)
Tiny-ImageNet (%)
0.01
86.25
86.16
86.20
0.00
31.05
25.68
0.1
86.23
86.14
86. 19
0.25
85.71
82.06
1.0
86.19
86.15
86.17
0.50
86.21
82,60
10.0
85,99
86.19
86.18
0.75
85.85
82,50
1.00
85.71
82.46
A.5.4
HYPERPARAMETER SENSITIVITY AND ABLATION
We perform a comprehensive study on FedDuet's 's hyperparameters, including the number of Experts, Top-K selection, K-means clusters, loss function weights (a, 1), 7). and the fusion coefficient X.
20

---

Published as a conference paper at ICLR 2026
As shown in Table 10 the model exhibits strong robustness to variations in Experts, Top-K, and K-means clusters. Table further demonstrates that FedDuet maintains stable performance across wide range of loss weight values, while moderate fusion (A 0.50) achieves optimal results. Overall, these analyses highlight the method's low sensitivity to hyperparameters, making it practical for deployment without extensive tuning.
A.5.5 ABLATION STUDIES
Table 12:
Ablation study on our Progressive Decoupled Optimization strategy. Best results are in
bold. The blue values (A) indicate score lower than our method.
Training Strategy
CIFAR-100
Tiny-ImageNet
Average
A
Semantic Experts First
85.53
81.28
83.41
-0.98
Joint Training
86.02
81.88
83.95
-0.44
Parametric Experts First (Ours)
86.21
82.56
84.39
Optimization Strategy. We evaluate different training strategies to validate the effectiveness of
our Progressive Decoupled Optimization. Prioritizing parametric experts consistently yields the best performance, confirming that separating parametric and semantic training stages facilitates more ef- fective knowledge acquisition and representation learning compared to joint or semantic- first train- ing.
Table 13: Impact of Stage Transition Timing on Performance. R denotes the total communication rounds. Best results are bold. The blue values (A) indicate a score lower than our method.
CIFAR-100
Image
Transition Timing
Average
A
IID
Non-IID (3=0.1)
IID
Non-IID (3=0.1)
R/4 3R/4
84.86
82.86
81.29
78.73
81.94
-1.53
86.12
84.29
82.88
80.41
83.43
-0.04
R/2 (Ours)
86.21
84.58
82.60
80.48
83.47
Stage Transition Timing, We investigate the impact of stage transition timing within Progressive Decoupled Optimization. Transitioning at the midpoint of training strikes the best balance, providing parametric experts sufficient time to develop foundational representations before semantic experts refine them. Moving the transition earlier compromises learning stability, while delaying it offers little additional benefit, highlighting the importance of timely stage scheduling.
Table 14: Comparison of Prompt Pool Initialization Strategies on CIFAR-100. Best results are in bold. The blue values (A) indicate score lower than K-means initialization.
Initialization Strategy
IID
Non-lID (3=0.5)
NID-III (3=0.1)
Average
A
Random
85.93
83.88
84.02
84.61
-0.58
Uniform
85.96
84.23
84.09
84.76
-0.43
K-means (Ours)
86.21
84.77
84.58
85.19
Prompt Pool Initialization. We analyze different strategies for initializing the prompt pool. means clustering consistently produces more robust and generalizable prompts, particularly under heterogeneous data distributions. This demonstrates that a well-structured initial prompt set en- hances the model's ability to handle diverse client data.
Prompt Selection Strategies. We compare dynamic prompt selection via our Gating Network against random assignment. The Gating Network consistently improves performance across all set- tings, especially in heterogeneous federated environments. This highlights the advantage of adaptive selection in leveraging client-specific features and maximizing the utility of the prompt pool.
21

---

Published as a conference paper at ICLR 2026
Table 15: Ablation study on prompt selection strategies. We compare our Gating Network with random selection of top-K prompts across different datasets and data distributions. The A indicates the performance improvement relative random selection, highlighted in blue.
CIFAR-100
Imag-em
Selection Strategies
Average
A
IID
Non-IIl (8=0.1)
IID
Non-IID (3=0.1)
Random Selection
85.96
83.27
82.07
79.58
82.22
-1,25
Gating Network (Ours)
86.21
84.58
82.60
80.48
83.47
Table 16: Robustness Analysis on CIFAR-100 and Tiny-ImageNet. We extend the main results by reporting mean accuracy + standard deviation across multiple runs demonstrate the robustness and reproducibility of our approach.
CIFAR-100
Tiny-ImageNe
Method
IID
Non-IID (8=0.1)
IID
Non-IID (3=0.1)
FedWeIT
72.52 = 0.62
70.43 0.64
72.06 + 0.67
71.06 + 0,64
FedKNOW
78.47 = 0.52
77.25 0,43
76.24 + 0.73
74.61 0.84
FedCLIP
78,21 = 0.37
74.17 0.57
76.14 0.51
71.94
Fed-CPrompt
73.41 + 0.42
73.00 0.33
73.09 + 0.45
72.48 + 0.32
Powder
73.32 = 0.34
72.71 0.51
72.98 0.41
72.71 + 0.62
PFedMoAP
76.61 = 0.23
$2.63 1.02
74.47 0.20
55.47 + 1.23
MoAFCL
77.93 -
65.96 1.23
74.16 1 0.59
62.51
1.17
Fed-Duet (Ours)
86.21 1 0.11
84.58 0.19
82.60 1 0.13
80.48
0.21
A.5.6 ROBUSTNESS ANALYSIS WITH STANDARD DEVIATION
Table 16 extends the main results by reporting mean accuracy with standard deviation across mul- tiple runs. FedDuet achieves both the highest accuracy and lowest standard deviation across all settings. For example, on CIFAR-100 Non-IID (8=0.1), FedDuet obtains 84.58 + 0.19 compared to the second-best FedKNOW at 77.25 + 0.43, demonstrating superior stability and reproducibility.
A.5.7 CROSS-MODAL ALIGNMENT AND RETRIEVAL PERFORMANCE
We conduct comprehensive evaluations to verify that FedDuet effectively preserves and enhances cross-modal alignment in federated continual learning scenarios. Our analysis covers three per- spectives: alignment scores across tasks, retrieval performance after cumulative training, and align- ment evolution during two-stage training.
Table 17: Cross-Modal Retrieval Performance Tiny-ImageNe We evaluate all methods after completing the final task (Task 4) on the cumulative test set. Metrics include Image-to- Text (I2T) and Text-to-Image (T2I) Recall@ K. Results averaged across 5 clients. Best results are in bold. The blue values (A) indicate a score lower than our method. Results are averaged across 5 clients.
Image-to- Text (I2T)
Text-to- Image (T2I)
Method
Average
A
R@D
R@5 f
@ 10
R@1
R@5
R @ 10
FedCLIP
64.38
85.50
90.61
88.10
98.30
99_50
87.73
-4.19
FedWelT
63.65
85.89
90.99
87,90
98.40
99.50
87.72
-4.20
FedCPrompt
63.83
85.89
90.61
88.00
98.30
99.80
87.74
-4.18
Powder
63.85
85,48
90.66
88.90
98.80
99.60
87.88
-4.04
PFedMoAP
63.52
85.71
90.93
87.60
98.40
99.40
87.59
-4.33
FedKNOW
63.65
85.89
90.99
87.90
98.40
99.50
87.72
-4.20
MoAFCI.
63.15
85.79
90.55
86.90
98.30
99.30
87.33
-4,99
FedDuet (Ours)
74.67
92.46
95.91
92.90
99.60
100.0
91.92
22

---

Published as a conference paper at ICLR 2026
Cross-Modal Retrieval Performance. Tables 17 (and the corresponding CIFAR- -EFAR-100 Table 3 port Image-to- Image-to-Text Text (I2T) and tomatee-mmme (T21) retrieval metrics on cumulative test sets. FedDuet outperforms all baselines across both datasets, with the largest gains observed in top-1 retrieval. This improvement highlights that preserving cross-modal alignment directly translates into more accurate retrieval, demonstrating that approach maintains the fundamental vision-language derstanding of CLIP models beyond mere classification accuracy.
Table 18: Cross-Modal Aligmment Score Across Tasks on Iny-ImageNet. We track the alignment score after each task to demonstrate the ability to maintain cross-modal alignment during continual learning. Results are averaged across 5 clients. Best results are in bold. The blue values (A) indicate a score lower than our method. Results are averaged across 5 clients.
Method
Task
Task 1
Task
Task
Task
Average
A
FedCLIP
0.0821
0.0829
0.0837
0.0835
0.0833
0.0831
-0.1229
FedWelT
0.0823
0.0830
0.0838
0.0835
0.0834
0.0832
-0.1228
FedCPrompt
0.0824
0.0831
0.0837
0.0834
0.0833
0.0832
-0.1228
Powder
0.0823
0.0830
0.0836
0.0835
0.0832
0.0831
-0.1229
PFedMoAP
0.0820
0.0829
0.0839
0.0834
0.0832
0.0831
-0.1229
FedKNOW
0,0823
0.0830
0.0838
0.0835
0.0834
0.0832
-0.1228
MoAFCL
0.0824
0.0828
0.0835
0.0833
0.0831
0.0830
-0.1230
FedDuet (Ours)
0.1900
0.1962
0.2065
0.1980
0.2061
0.2060
Alignment Across Tasks. Tables 18 (and the corresponding CIFAR-100 100 Table 4 track cross- modal alignment scores after each task. FedDuet consistently achieves the highest alignment scores, substantially surpassing all baseline methods. While conventional approaches maintain rela- tively low and stable alignment, FedDuet demonstrates significantly stronger alignment that is stable across sequential tasks, indicating its ability to maintain robust image-text associations throughout continual learning.
Table 19: Cross-Modal Alignment Score Evolution Across Training Rounds. Stage (Rounds 1- 5) corresponds to parametric expert training, and Stage (Rounds 6-10) corresponds to semantic prompt training.
Round
Stage
Alignment Score
Positive Sim
Negative Sim
Stage
0.1496
0.2851
0.1355
2
Stage
0.1664
0.2932
0.1267
3
Stage
0.1775
0.2942
0.1166
4
Stage
0.1873
0.2945
0.1072
Stage
0.1911
0.2965
0.1054
Stage Change
+0.0415
+0.0114
-0.0301
6
Stage 2
0.1955
0.2981
0.1027
Stage
0.1953
0.2979
0.1026
8
Stage 2
0.1952
0.2979
0,1027
9
Stage 2
0.1952
0.29799
0.1027
10
Stage 2
0.1954
0.2980
0.1027
Stage 2 Change
-0.0001
-0.0001
+0.0000
Overall Change (R1-R10)
+0.0458
+0.0130
-0.0328
Alignment Dynamics Across Two-Stage Training. FedDuet designed to balance continual task adaptation with stable cross-modal grounding. To assess how two-stage training supports this goal, we track alignment scores and cross-modal retrieval performance on Task throughout training (Tables and 20).
Stage (parametric expert training) progressively strengthens cross-modal grounding, reflected by consistent improvements in alignment scores and 12T retrieval quality.
23

---

Published as a conference paper at ICLR 2026
Table 20: Cross-Modal Retrieval Performance Evolution Across Training Rounds. We report Image- to- texText (I2T) and Text-to-lmage (T2I) Recall @ K metrics. Stage corresponds to parametric expert training (Rounds 1-5), Stage 2 corresponds semantic prompt training (Rounds 6-10).
12T Recall (%)
T21 Recall (%)
Round
Stage
R@1
R@s
R@ 10
R@D 1
R@s
@@ 10
Stage 1
94.66
99.75
99.95
100.00
100.00
100.00
2
Stage 1
95.54
99.82
99.94
100.00
100.00
100.00
Stage 1
96.00
99.86
99.95
100.00
100.00
100.00
Stage
96.33
99.82
99.96
100.00
100.00
100.00
Stage 1
96.38
99.88
99.96
99.00
100.00
100.00
Stage Change
+1.72
+0.13
+0.01
-1.00
0.00
0.00
Stage 2
96.68
99.85
99.96
100.00
100.00
100.00
Stage 2
96.44
99.86
99.94
100.00
100.00
100.00
Stage 2
96.68
99.88
99.97
99.00
100.00
100.00
Stage 2
96.69
99.88
99.96
100.00
100.00
100.00
10
Stage 2
96.51
99.86
99.98
100.00
100.00
100.00
Stage 2 Change
-0.17
+0.01
+0,02
+1.00
0.00
0.00
Overall Change (R1-RIIH
+1.85
+0.11
+0.03
0.00
0.00
0.00
Stage 2 (semantic prompt training) preserves this grounding: with experts frozen, semantic prompts adapt while alignment scores and retrieval metrics remain stable. This indicates that the learned cross-modal structure acts as reliable anchor during downstream prompt updates.
Overall, the results show that FedDuet's decoupled training cleanly separates representation ground- ing from semantic adaptation, enabling continual refinement without compromising cross-modal consistency.
A.5.8 FEATURE SPACE VISUALIZATION VIA T-SNE
To qualitatively verify the efficacy of our framework in mitigating catastrophic forgetting, em- ploy t-SNE to visualize the learned feature space after the model has trained on all sequential tasks. As depicted in Figure alization reveals that Fed-Duet maintains highly structured and separable feature space, evidenced by the dense and distinct clusters formed by each task's 's samples. This clear spatial separation serves as strong qualitative evidence that our framework preserves the integrity of feature representations for earlier tasks, thereby effectively mitigating catastrophie forgetting at the feature level.
project the feature embeddings of test samples into 2D space. The visu-
Task ID
Task
Task
Task
Task
CIFAR-100
Tiny-ImageNet
Figure 6: t-SNE visualization of feature representations for test samples from all tasks on CIFAR- 100 (Left) and -iay-Im (Right) after the final task. Each color corresponds different continual learning task.
24

---

Published as a conference paper at ICLR 2026
A.5.9 ANALYSIS OF EXPERT UTILIZATION
The expert activation patterns, visualized in Figure reveal both effective load balancing and high degree of specialization. The heatmaps show a distributed activation strategy, confirming that our model avoids expert collapse by engaging diverse of experts. Furthermore, we observe a strong tendency for specific experts to be responsible for distinct task categories, particularly in the deeper layers. This dual observation of balance and specialization mutually corroborates the findings from our t-SNE visualization (Figure 60. which similarly indicates that experts form distinct, specialized clusters.
Layer
Layer
Layer
14
2
6.6
Prequency
1v 10
Layer
Layer
Layer
Taah
2.2
1ak
Tab
588
Figure 7: Visualization of expert selection frequency per task in our proposed FedDuet framework. The heatmaps illustrate which of the experts activated for each of the 10 sequential tasks across 12 transformer layers. The color intensity corresponds to the selection frequency, highlighting how expert utilization evolves for different tasks and layers.
25


Reviews:
[{"review_id": "0u5ZT9oBB5", "reviewer": "Reviewer_RknU", "summary": "This paper explores the challenging problem of continual federated vision-language learning. The authors propose a framework named Fed-Duet, which introduces a dual-channel design that integrates server-coordinated prompts and client-adapted experts to address challenges in cross-modal alignment, adaptation imbalance, and catastrophic forgetting. Extensive experiments demonstrate the framework’s effectiveness and efficiency.", "questions": "- How does it identify and activate the correct experts to mitigate forgetting under the FL constraint of not explicitly storing old task data?\n\n- The paper proposes that Fed-Duet can maintain critical cross-modal alignment performance in Vision-Language Models (VLMs). However, the experiments are only confined to vision-text classification tasks (CIFAR-100, Tiny-ImageNet, DomainNet), with classification accuracy serving as the core evaluation metric. How can the preservation of cross-modal alignment capability be directly demonstrated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The topic is emerging and important. It represents a key challenge for deploying next-generation AI on edge devices, combining FL, CL, and VLM.\n- The idea of dual-channel orchestration, unifying prompt-based semantic alignment and adapter-based parametric specialization, is conceptually clear and inspiring.\n- Fed-Duet delivers a significant performance boost in continual learning while maintaining PEFT's communication efficiency, vital for resource-constrained FL.", "weaknesses": "- The authors introduce noise for privacy preservation, but the description of this part is limited and lacks sufficient details.\n- RELATED WORK does not discuss a related work, CLIP2FL.\n- Statistical reporting is incomplete. Providing statistics such as the mean and standard deviation would better support the robustness claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rl7yxq3fm4", "reviewer": "Reviewer_kMS8", "summary": "Fed-Duet proposes a novel framework for federated continual learning with vision-language models, aiming to address two key limitations in existing approaches, i.e., adaptation imbalance and cross-modal misalignment. The method introduces a dual-channel expert-orchestrated architecture, where the semantic experts channel uses learnable prompts to achieve high-level semantic alignment and the parametric experts channel employs adapters for fine-grained task adaptation. Experiments across multiple datasets demonstrate that Fed-Duet achieves strong performance in terms of accuracy, knowledge retention and continual utility.", "questions": "1. In Stage 2, the parametric experts are frozen and only the semantic prompts are updated using cross-entropy loss. However, the image features have already been adapted by the parametric channel, while the text prompts are tuned independently. Could this decoupled update reintroduce cross-modal misalignment?\n\n2. What is the rationale behind the specific timing for switching from parametric expert training to semantic expert training at $r = R/2$? Was this threshold tuned, or is it fixed across all settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper clearly identifies a critical gap by noting that traditional federated continual learning ignores the multimodal nature of vision-language models, and existing PEFT-based federated learning methods often disrupt cross-modal alignment under continual learning.\n\n2. The semantic channel uses a cross-attention gating mechanism to fuse prompt-based experts, while the parametric channel adapts adapter-based experts. The two channels are trained in a progressive manner that enables effective synergy.\n\n3. The experimental evaluation is comprehensive, covering class-incremental, domain-incremental, and multi-domain task-incremental scenarios under various Non-IID settings and client scales. The results consistently show superior performance compared to strong baselines.", "weaknesses": "1. The method involves a large number of hyperparameters, including $\\alpha$, $\\beta$, and $\\gamma$ in the loss function, as well as $\\lambda$ and $\\tau$ in the Eq (2) and Eq. (4). The paper provides neither justification for their chosen values nor sensitivity analysis, which limits reproducibility and practical deployment.\n\n2. The choice to freeze parametric experts after round $r = R/2$ appears arbitrary. No theoretical reasoning or empirical ablation is provided to support this scheduling decision.\n\n3. Several presentation issues reduce clarity.\n\n    3.1. Notation is inconsistent. For example, $S_r$ in Eq. (1) is undefined, and the symbol $\\beta$ is reused for both the Dirichlet parameter in data partitioning and a loss weight.\n\n    3.2. Figure 1 is visually confusing, as the semantic experts are incorrectly shown as inputs to the image encoder, and the overall layout does not clearly reflect the workflow of Fed-Duet.\n\n    3.3. Baseline names in Figure 3(a)(b) use excessively small font sizes, impairing readability.\n\n4. The global prompt pool is initialized via K-means clustering on class embeddings, but the paper does not compare this strategy against random initialization or other alternatives, making it difficult to assess its actual contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0XVQoMdYl", "reviewer": "Reviewer_JyH1", "summary": "This paper introduces a dual expert-orchestrated framework for continual federated VLM. Various experiments confirms the effectiveness of proposed method.", "questions": "Refer to weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The author conducts various experiments and ablation to confirm the effectiveness of proposed method.", "weaknesses": "This paper has several weakness.\n\n\n1. The author utilizes the confused keyword expression.  As mentioned in the title, \"DUAL EXPERT\", But in the abstract, the author utilizes the dual channel. As far as i know, channel is widely utilized in feature dimension, such as Barlow Twins or other dimension-wise operation. \n\n2. Turn to your two experts, prompt and adapter, it is hard to understand why prompts refer to high-level and adapters refer to low-level. Because the prompt and adapter are both suitable to inject into each transformer layer. Authors should pay more attention to conducting the observation experiments rather than drawing a complicated framework without meaningful information.\n\n3. Suspicious experiment results: Line 335 shows identical CIFAR-100 accuracy for T=10 across different beta values, with no explanation, harming the result's credibility. （84.20 75.97）Besides, the anonymous code does not release the compared methods. I do not trust your results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zifyScmCJp", "reviewer": "Reviewer_9Jgs", "summary": "This paper introduces Fed-Duet, a framework to address continual federated learning for CLIP models. The key idea is a dual-channel expert system: The paper adopts (1) semantic experts: server-coordinated prompts for high-level multimodal alignment and (2) parametric experts, i.e., client-specific adapters. A cross-attention fusion mechanism integrates these two channels, while auxiliary objectives preserve cross-modal consistency and mitigate catastrophic forgetting. Extensive experiments on multiple benchmarks demonstrate the promising performance of the method in both accuracy and efficiency.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method achieves promising results on multiple benchmarks.\n2. The experiments are thorough and comprehensive.", "weaknesses": "1. The proposed method is complex, I would recommend refine/extend a bit on the details of the method design and different components. Also, I personally don't think \"Semantic Experts\" is a good naming for textual prompt. \n2. There are too much information included in Figure 1, maybe considering separate it into multiple figures.\n3. The authors mentioned about \"privacy-preserving feature\" of the client images will be uploaded to the server. However, there lacks details about how the strength of the noise and whether it is still possible to reproduce the images at server-side. The author should discuss this more.\n4. How would the attention gating influence the model performance? There seems to be no ablation about this specific component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)