pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
Authors: Sajjad Ghiasvand
Mahnoosh Alizadeh
Ramtin Pedarsani
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Modal Adapter
Personalized Federated Fine-Tuning
Few-Shot Learning of Vision Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Vision-Language Models (VLMs) like CLIP have demonstrated remarkable generalization in zero- and few-shot settings, but adapting them efficiently to decentralized, heterogeneous data remains a challenge. While prompt tuning has emerged as a popular parameter-efficient approach in personalized federated learning, existing methods often sacrifice generalization in favor of personalization, struggling particularly on unseen classes or domains. In this work, we propose pFedMMA, the first personalized federated learning framework that leverages multi-modal adapters for vision-language tasks. Each adapter contains modality-specific up- and down-projection layers alongside a globally shared projection that aligns cross-modal features. Our optimization strategy allows clients to locally adapt to personalized data distributions while collaboratively training the shared projection to improve global generalization. This design is also communication-efficient, as only the shared component is exchanged during communication rounds. Through extensive experiments across eleven datasets, including domain- and label-shift scenarios, we show that pFedMMA achieves state-of-the-art trade-offs between personalization and generalization, outperforming recent federated prompt tuning methods.

Published as a conference paper at ICLR 2026
PFEDMMA:
PERSONALIZED FEDERATED FINE-
TUNING WITH MULTI-MODAL ADAPTER FOR VISION-
LANGUAGE MODELS
Sajjad Ghiasvand, Mahnoosh Alizadeh, & Ramtin Pedarsani Department of Electrical and Computer Engineering UC Santa Barbara
Santa Barbara, CA 93106, USA {sajjad, al zzaeh,, ramt in Sucseeecsbbcoo...e
ABSTRACT
Vision- LanguaLa Models (VLMs) like CLIP have demonstrated remarkable gener- alization in zero- and few-shot settings, but adapting them efficiently to decentral- ized, heterogeneous data remains a challenge. While prompt tuning has emerged as popular parameter-efficien approach in personalized federated learning, exist- ing methods often sacrifice generalization in favor of personalization, struggling particularly unseen classes domains. In this work, we propose PFedMMA, a personalized federated learning framework that leverages multi-modal adapters for vision- Laaiuu- tasks. Each adapter contains mopality-specific up- and down- projection layers alongside a globally shared projection that aligns cross-moud modal features. Our optimization strategy allows clients to locally adapt to personalized data distributions while collaboratively training the shared projection to improve only the
global generalization. This design is also communication--fficienn. shared component is exchanged during communication rounds. Through extensive experiments across eleven datasets, including domain- and label-shift scenarios, we show that PFedMMA achieves state-of offte--------trtt trade-offs between personalization and generalization, outperformng recent federated prompt tuning methods. Code is available at ht /ttp:::////ttttt com sa jad ucsb / FecMMA
INTRODUCTION
Vision-Language Models (VLMs) like CLIP Radford et al. (202T) have revolutionized multi-modal learning by jointly embedding visual and textual data through massive contrastive pre-training Jia ct al. [202]]; Li et al. (2022); Yao et al.) This paradigm empowers models to generalize effectively
shot and few-shot settings Zhang et al. (2022 Zhu al. (2023); Ghiasvand al. (2025a); Aghdam Hu (002)). Among them, larger transformer-bases variants Vaswani (201)))((007) (e.g.. CLIP ViT-L/14) consistently outperform smaller counterparts such ViT-B/I6, with margins exceeding 6% on benchmarks like ImageNet Deng et al. (2009). However, the computational demands of fine- tuning such large-scale models with billions of parameters pose significant challenges, particularly for domain-specific tasks Oskouie et al. (2025). To mitigate this, Parameter- -aaameter-ffficien Fine- ToninTunnng (PEFT) techniques have emerged, especially in NLP. These methods, including adapters Chen et al. (2022 Karimi Mahabadi et al. (2021); ;; Rebuffi et al. (2017) and prompt tuning Jia et al. (2022) Li & Liang (202)). ), introduce a lightweight set of trainable parameters or tokens, allowing the backbone
model remain frozen.
While highly effective centralized settings, these techniques fall short in scenarios involving decentralized and privacy-sensitive data, such as healthcare, legal, industrial domains Manoel et al. 2023): ;; Shoham & Rappoport] (2023); Mahjourian & Nguyen (2025). Federated Learning (FL) offers promising alternative by enabling collaborative training without raw data sharing. In FL, clients update their local models and transmit only intermediate model updates such as parameters or gradients, which aggregated into global model by central server McMahan al. (001)).

---

Published as a conference paper at ICLR 2026
In real-world scenarios, client data often exhibits variations in domain discrepancies (feature shift) Li
et al.] or imbalanced class distributions (label shift) Li et al. (2021a). Simply applying standard
aggregation strategies, such FedAvg [MMMhaa et al. (2017) 7), over prompts Guo et al. (2023b)
or other fine-tuning methods, such as LoRA, often leads to suboptimal performance due to data
heterogeneity /Zhang al. (2023): Borazjani et al. (2025). As result, Personalized Federated Learning (PFL), particularly with prompt tuning. has gained increasing attention. pFedPrompt Guo ct al. 2023a) introduces personalization by coupling global text prompt with local visual attention modules to tailor predictions to each client's 's data. FedOTP Li et al. (2024) uses Optimal Transport to align local and global representations under label shift. FedPGP et al. (2024) applies prompt-wise contrastive learning to enhance inter-client generalization. Recently, pFedMoAP Luo et al. (2025) proposes a Eexture-of-Expertt framework, where prompts from other clients serve as non-local experts, and each client learns attention-based gating mechanism for selective adaptation. While these methods achieve impressive personalization performance, they often struggle to generalize to unseen classes or domains, limiting their applicability in aat-of-distribution scenarios. For example, as shown in Fig. I FedOTP achieves poor harmonic mean accuracy, even though it has been shown to have strong personalization performance.
Figure 1: Few-shot performance across datasets using the ViT-B/16 model. Each radar chart illustrates accuracy (%) for fixed shot count, with spokes representing the evaluation datasets. Curves correspond to different methods, and values increase outward (20-100%). Accuracy is reported as the harmonic mean (HM) over local, base, and novel classes for each dataset and shot.
Beyond prompt tuning, adapters offer another PEFT strategy by introducing small trainable modules into frozen pre-trained models Cai et al. (2200); Chen al. (2022) b): Gao et al. (2024); (Hu et al. 2 et al. (2022). Unlike prompts, adapters operate independently of model architecture and can be easily inserted into various backbones, such ResNets He et al. (2016). ViTs/Dosovitskiy et al. and Swin Transformers Liu et al. (2021). However, most adapter methods like AdaptFormer Chen jet al. (002) and LoRA Hu et al. (202T) are uni-modal and do not account for the cross-modal dependencies inherent in VLMs like CLIP Radford et al. (2021). Multi-modal adapters Yang et al. (2024) address this by integrating both visual and textual signals via a shared projection layer that promotes feature alignment across modalities while preserving modality-specine knowledge. Despite their demonstrated advantages over prompt-based based approaches Yang et al. (2024); Guo & Gu 2005
their integration with PFL remains largely unexplored.
In this work, we introduce a Personalized Federated Multi-Modal Adapter (PFedMMA) architecture adapter design with three components: a modality-specific down-projection, a shared projection, and a modality-speciiff up-projection. During training, all components are updated locally by each client, but only the shared projection is globally aggregated. This asymmetric training scheme enables effective personalization through client-specific projections, while promoting space. Moreover, since only the shared adapter is
that adopt
generalization via a shared modality-alignmeen communicated during rounds, the method remains coammunccation-effcccci experiments, this design achieves the strongest trade-off between personalization and generalization under both feature and label shifts. As shown in Fig. D on average, our proposed pFedMMA delivers the best harmonic mean performance compared to ofate-of-the-art federated prompt tuning methods.
As confirmed by our
Before delving into details, we summarize our contributions: (1) We observe that while most state-of-the-art prompt tuning methods achieve strong personalization performance, they often generalize poorly to unseen classes. To address this, we introduce a multi-modal adapter framework that explicitly aims to balance personalization and generalization in federated vision- vision-language learning. (2) We propose PFedMMA, an adapter- based approach for PFL of VLMs. Our architecture incorporates modality-specee up- and down- nown-projection layers and a shared cross-modal projection. All components are updated locally, but only the shared projection is aggregated globally, enabling
2

---

Published as a conference paper at ICLR 2026
effective asymmetric optimization. (3) We conduct extensive experiments on widely used benchmarks to evaluate PFedMMA's performance on base-to-novel generalization across both category- and domain-level tasks under heterogeneous data distributions. Results demonstrate the superiority of our approach in harmonizing generalization and personalization.
PRELIMINARIES
2.1 PERSONALIZED FEDERATED LEARNING
Traditional federated learning frameworks are designed around the principle of global consensus, where the goal is to collaboratively train a single model that generalizes well across a federation of clients. The canonical approach, FedAvg McMahan et al. (2017), formalizes this the minimization of weighted average of local objectives: ming F(Ø) Ei=1 P; (0)). where denotes the global scales the contribution of each client by its dataset size 74, with n = >; n2. In this setup, each client's local loss is computed as the average over its data: Σx=1 ( (Ix. Mk))). where L, is the local loss function and (Ik yik)
model, F-(.) represents the local empirical loss of client i, and p;
the k-th data point client i.
In contrast, personalized federated learning (PFL.) challenges the one-size-fits-al paradigm by allowing each client to maintain its own model 0, This formulation acknowledges data heterogeneity and aims to tailor learning to each client's unique distribution. The objective for PFL becomes:
min O.....Ov
1N) = P(P.)
offering flexible alternative that prioritizes personalized performance over strict global consensus.
2.2 VISION-LANGUAGE CLASSIFICATION WITH FEW-SHOT ADAPTATION
In vision-language classification, predictions emerge from the powerful alignment between visual and textual modalities established during pretraining. Given a label set with K classes, the model begins by crafting natural language prompts Liu et al. (2023) semantic descriptions like "a photo of a [class name]"-for each class сą. These textual cues are passed through frozen text encoder 0, producing normalized text embeddings z
0g(c,) Rd. In parallel, each input image x; is processed by visual encoder 6g, generating a corresponding normalized image embedding z; = (xx,) E Rdd Classification then hinges comparing the cosine similarity between these multimodal representations. The result is set of logits transformed into class probabilities via K
a lemperature-scaled softmax: Pı, exp cos(z)(") (1) ZR 1)/7)) /Ej= exp (cos(z)") 2 (/)/)),
where 7 is the temperature parameter controlling distribution sharpness. The predicted label for image x; corresponds the class with the highest posterior probability: & arg max; Pe,
This zero-shot classification pipeline mirrors the contrastive training strategy employed in founda- tional vasion-language models like Radford et al. (2021), enabling impressive generalization to novel tasks without requiring any target-domain fine-tuning.
To further tailor the model to downstream tasks, the few-shot setting introduces small set of labeled examples per class, typically fewer than 16. With M support samples per class and ground-truth labels encoded one-hot vectors Yik (where Vik if x; belongs class k and 0 otherwise), classification proceeds identically to the zero-shot case. However, the model is now adapted by minimizing the cross-entropy loss over the labeled support set: Lce 3 ΣM ΣK= Vik In Pi,k-
This fine-tuning step enables the model to better capture domain-specific semantics while maintaining the efficiency and generalization capabilities of the pretrained architecture. Adaptation can be achieved through various strategies. One approach is to directly optimize the input prompts following the principles of prompt tuning Chen al. (a). Alternatively, lightweight task-specific modules such as adapter layers Gao et al. (2024) or low-rank parameterizations like LoRA Zanella & Ben Ayed (204) can be fine-tuned, while keeping the backbone encoders frozen.
3

---

Published as a conference paper at ICLR 2026
2.3 FINE-TUNING VIA PARALLEL ADAPTERS
In contrast to the serial adapter architecture introduced by [Houlsyy et al. (2019), where adapter modules are inserted sequentially after each sub-layer (e.g., attention feed-forward), parallel adapters He et al.] adopt an alternative integration strategy. Rather than placing the adapter transfor- mation after the main layer, the parallel formulation processes the input through the adapter module concurrently with the frozen backbone transformation and combines their outputs additively.
Let æ R2d be the input transformation. A parallel adapter layer computes the output as: Output(z) f(x) + (x)). where is a scaling factor and the adapter module A(x) uses the same bottleneck structure in the serial configuration: A Aax) ( D (x))), where U is up-projection affine map. D is down-projection affine map, and 8 a non-linear activation function such as ReLUU If the input ar has dimensionality d, then D I and U Rd where r < d. This bottleneck structure introduces significantly fewer trainable parameters compared to the full model. As with serial adapters, only the adapter parameters are trained during fine-tuning and the base model remains frozen. Parallel adapters preserve model expressiveness while enabling efficient adaptation with minimal architectural modifications.
transformer sub-layer, and let f()) denote the frozen pre-trained
3 PROPOSED METHOD
In this section, we introduce pFedMMA, a novel framework that leverages multi-modal adapters to efficiently and effectively adapt large pre-trained VLMs under federated learning settings. Our design consists of two central components: (i) a multi-modal adapter architecture that bridges and enriches representations across visual and textual modalities, and (ii) hybrid personalization strategy that promotes both generalization and personalization by decoupling local and shared adapter components.
3.1 MULTI-MODAL ADAPTER ARCHITECTURE
We build on the adapter-based design introduced in Yang et al. (2024) to incorporate a lightweight and efficient tuning mechanism for vision-language models. This architecture has proven effective in few-shot generalization settings, where pre-trained CLIP models are fine-tuned on a limited number of base classes and tested base and novel, unseen categories.
The motivation for this design stems from two empirical findings: (i) higher layers of both image and text encoders in CLIP contain more discriminative and dataset-specifie features, while lower layers preserve general, transferable knowledge; and (ii) larger modality gaps between text and image encoders observed in the lower layers, making cross-modal alignment particularly challenging in the early stages of the network Yang et al. (2024).
Based these insights, the multi-modal adapter is inserted into the upper transformer blocks of both encoders, starting from block f, while the lower layers remain frozen. This helps preserve general representations while enabling task-specific adaptation at the top layers.
Each adapter consists of: (i) A down-projection layer that reduces the input dimension, (ii) A shared projection layer that facilitates interaction between the modalities, (iii) An up-projection layer that restores the original dimension.
This three-part structure allows the adapter to first transform features into a low-dimensional space, fuse them through a shared module, and then project them back. Formally, for the visual adapter (indexed by (T)) and the textual adapter (indexed by (T)) at the j-th block:
A4 (x)0)) w/( -(W,. 8(W(o) jd
2 (o))).
(I,T). j fe (e. L}.
(2)
where 2/" and 27(7) denote the input hidden states at the j-th transformer layer for the vision and text encoders, (1)
jd and (T!T) are the down-projection matrices, W,, is the shared projection matrix jd used across both modalities, ( and (()) are the up-projection matrices, and 5(-) denotes the non-linear activation function (e.g., GELU), applied element-wiss wise.
This shared projection structure encourages information exchange across modalities, while still maintaining modality-specifie processing through separate up/down projections.
4

---

Published as a conference paper at ICLR 2026
Climii
Sapar
Tox Dacadar
Clear
Leamable
Fronzn
Personalized Adapper
Global Adepter
Figure 2: An overview the PFedMMA framework. Each client independently updates all trainable components of the multi-modal adapters including client-specific up/down projections and the shared projection over local epochs. After local training, only the shared adapter is uploaded and aggregated by the server. This design promotes personalization through local adapters while enabling generalization via a globally shared component.
In contrast to methods that inject prompts or adapters across all layers Chen et al. (2022) Houlsby al. (2019): ; Hu (2021); Ghiasvand al. (2025b) or some lower layers Khattak al. 2023a b);
Zhou et al. (2022baa) a), this selective, top-layer insertion strategy reduces the number of trainable parameters while maintaining cross-modal adaptability.
3.2 GENERALIZATION AND PERSONALIZATION VIA PFEDMMA
To effectively balance generalization and personalization in federated vasion-language learning, we propose hybrid training strategy within the multi-modal adapter framework. Each adapter consists of three projection components: a modality-specific down-projection, modality-specifis up-projection. In our personalization scheme, clients update the down- and up- projection components locally, while the shared projection matrix is synchronized globally via server aggregation.
shared projection, and a
This selective update mechanism provides several key benefits: (i) Local personalization: By allowing clients to optimize their own up- and down-projection matrices, each client can adapt the representation space to their unique local data distribution. This is particularly effective under label and feature heterogeneity. (ii) Global generalization: The shared projection matrix is collaboratively trained across clients and is responsible for aligning the modalities in consistent global space. This facilitates transfcrability and enables the model to generalize well across diverse domains and tasks. (iii) Communication efficiency: Since the shared projection layer is now-dimensional compared to the full model or full adapter stack, transmitting only the shared component during communication rounds results in significantly reduced communication cost. Extensive communication and computational cost analysis are provided in Appendix
Specifically, for client in communication round I, all trainable parameters
W e (T) ()) W(() id.s juss
(T) jd, ju.i W je (k. L}, (E. (3)
are updated for E local epochs using gradient descent: wh = whee- VL (W.he-)) where v is the learning rate, Lee is the cross-entropy loss, and (1.....E).
After local updates, only the shared projection parameters WT.. uploaded to the server. These are aggregated across all participating clients to obtain the updated global shared adapter: W MN Pj ! ja.i where P; 2a scales the contribution of each client by its dataset size nı, with n == E; n, and N is the number of participating clients. In contrast, the up- and down-projection parameters remain local and shared or averaged.
5

---

Published as a conference paper at ICLR 2026
This asymmetric update design enables PFedMMA to effectively capture both shared and client- specific information, resulting in an improved balance between personalization and generalization, as demonstrated in experiments tasks involving domain and label shifts. The overall training and communication flow of PFedMMA is illustrated in Fig
4
EMPIRICAL RESULTS
In this section, we conduct extensive experiments to evaluate the generalization and personalization capability of PFedMMA in heterogeneous data distribution scenarios.
4.1 EXPERIMENTAL SETUP
Datasets and Data Heterogeneity. To evaluate the effectiveness of PFedMMA, conduct periments acToss eleven public benchmark datasets that cover various types of data heterogeneity, including label shift and feature shift. Following prior work such as Guo al. (2023b), we use seven visual classification datasets: SUN397 Xiao et al. 2010). OxfordPets Parkhi hi et al. (012).. Flowers 02 Nilsback & Zisserman (2008). DTD Cimpol al. (2014), Caltech101 Fei-Fet FeiiiiFFFFFFFFe
al.)
(2004), UCFT01 Soomro et al. (2012), and Food101 Bossard et al. (2014). We refer to these as the CLIP datasets. To simulate severe label heterogeneity, we apply a pathological non-III) setting in which each client is assigned distinct, non-overlapping set of classes. Clients are trained on their local classes and evaluated on their own classes, on base classes held by other clients, and on novel classes that are unseen during the training process.
To evaluate performance under feature shift, we utilize two widely adopted multi-domain datasets: DomainNet Peng et al. (2019), which consists of six distinct domains, and Office-Caltech 10 Gong et al. (012)), which includes four domains. Following prior studies, each client assigned data from single domain, ensuring that every domain is represented by group of clients in the federation. To introduce additional heterogeneity and simulate realistic federated learning scenarios, we further partition the data within each domain using a symmetric Dirichlet distribution with concentration parameter B. This setup introduces both feature shift across domains and label shift within domains. All domains participate in both training and evaluation phases, allowing us to assess cross- coos--domain generalization and personalization performance in more realistic federated conditions.
For personalization evaluation, we include CIFAR-10 Krizhevsky et al. (2010) and CIFAR- 100hKrizyyvsky et al. (2009). These datasets are partitioned among clients using Dirichlet distribu- tion, which creates varying degrees of label skew across clients. Additionally, we apply the same pathological class split as used in the CL.IP datasets to test robustness under extreme heterogeneity. Further details on the dataset configurations and partitioning strategies can be found in Appendix C.I
Baselines. We evaluate pFedMMA across all experimental settings, including generalization, person- alization, and domain generalization, using a consistent set of five baselines. Zero-shot CLIP Radford non-adanaa ado-avda reference model that uses fixed hand-crafted prompt templates
al. 1 serves such as "a photo of [class]" without any task-specific learning. PromptFL /uo et al. (2023b) represents standard federated prompt learning approach in which shared prompt is collaboratively learned across clients using FedAvg. FedPGP Cui et al.] (2024) introduces prompt- prompt-wise contrastive learning to encourage consistency between global and local prompts. FedOTP al. (2024) applies unbalanced Optimal Transport to align global knowledge with client-specific prompt representations. Finally, PFedMoAP Luo al. (2025) leverages Mixture-of-Experts design that enables each client to access both local and non-local prompt experts through an attention- based gating mechanism. In addition to these prompt- based methods, we also consider adapter and LoRA-style PEFT baselines by implementing federated CL.IP- A.dpterr Gao et al. (2024) and federated CLIP-LORA A Zanella & Ben Ayed (2024), where only the adapter or low-rank layers are updated and aggregated across clients. These baselines cover diverse range of federated adaptation strategies, providing strong benchmark for assessing the performance of PFedMMA across different types of heterogeneity.
Implimentation Details. All methods, including PFedMMA and all baselines, are implemented on top of a frozen CLIP model. We use two backbone architectures, ViT-B 16 and ViT-B32 Dosovitskiy et al. and default VIT-B16 unless otherwise specified. For the CLIP datasets, each is split into 10 chents with non-overlapping classes, using 100 percent participation, 2 local epochs, and 50 communication rounds. For the CIFAR-10 and CIFAR-100 datasets, simulate large-scale

---

We use Published as a conference paper at ICLR 2026
Table 1: Top-1 accuracy (%) of different methods across 7 datasets in the 16-shot setting.
Aecaae ES Sranco
Flaeeitic
DTDD
Mixual
Fooil
Baun
Naaee
HMM
Tocal
Bain
Nearl
PM
1mc
Hiee
HB1
Lecel
Fuee
Nevee
cuil Soste 1538
23.35
20.33
11.21
3.3
$1.43
$4.30
20.33
12.35
01/00
2.23
76.04
21.04
$4.50
$1.00
13.05
Scu4
71.00
53180
813
$1.16
72.28
13.45
0/322
21.89
94.28
08.48
3388
8.80
03.33
Iedom
09:34
3008
244
11.03
11.38
30.97
3775
01.55
08:37
11 81
96.8
70.00
20-88
35
08
05
52.06
Tea
Send
113
Bax
Nerd
Teec
63
86.11
01:33
B614
$1.01
15.00
08.15
95.26
$8.10
92.06
0333
70.05
72.08
ted Li
18.71
$3.01
12.71
14.33
96.76
17.05
$1.00
01.11
11.04
13008
11.00
33
83
86/14
07:17
8.53
13180
73
11
1000
3001
1000
66.00
$1.00
8.0
11:08
07.33
13.05
0100
13
7608
piedh Ohung
Fed01P
fed
Avcrage 24V0
SUNE97 (HM)
DTD (HH]
Caltech101 (HN)
Food101 (HN)
UCF101 (HH)
5CC
77.5
22
30
200
Average (Loca)
SUMB99 (lors))
DTD [Lora]
Catec1111 (loral)
Toos101 (ocsll
UCF10 [lorw)
57
Figure Local and harmonic mean (HM) accuracies of various methods across different shot settings.
federated environment with 100 clients, using a varying Dirichlet distribution and 10 percent client participation rate per communication round. Training runs for 50 rounds with local epoch per round. In the case of DomainNet and Effice-CaltechIl each domain of these two datasets is partitioned to 1/2 clients, resulting in N = 6/12 for DomainNet and N = 4/8 for Office-Caltechh SGD with learning rate of 0.001, and batch sizes of for training and 100 for testing. Further implementation details are provided in the Appendix, where we also report additional results using the AdamW optimizer (Table 19), which exhibit similar trends to those obtained with SGD,
4.2 PERFORMANCE EVALUATION
Base-to-Novel Class Generalization.
We evaluate the performance of PFedMMA in terms of its ability to generalize from locally trained classes to both base and novel classes. Following prior work, we report top-1 accuracy on each client's local classes, on the base classes seen by other clients, and on novel classes that are entirely unseen during training. To capture overall effectiveness, we use the harmonic mean (HM) of these three metrics, HM = (A
Acomue + Acc which penalizes methods that over-optimize one component at the expense of the others and thus better reflects the balance between personalization (local) and generalization (base and novel) than simple arithmetic mean; this type of harmonic mean score is standard in generalized zero-shot learning and base-to-novel CL.IP adaptation, and has also been adopted recent PFL work to jointly summarize local, base, and novel accuracies Vermal et al. (2020); Du et al. (2025); Cui et al. (2024). As summarized in Table for the 16-shot setting, PFedMMA consistently achieves strong performance across all evaluation categories and delivers the best overall HM averaged across seven datasets, outperforming all baselines.
Zero-shot CLIP, PromptFL, federated CLIP-Adapte and CLIP-LoRA suffer from poor local accuracy, tending favor generalization at the expense of personalization. We also report A, which denotes the relative improvement of PFedMMA compared with the strongest non- con-baseline methods (FedPGP, FedOTP, and PFedMoAP). While FedOTP sometimes achieves high local accuracy, its extremely low
7

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy comparison (%) on the Dirichlet Non-IID setting in CIFAR-IO and CIFAR-100.
Dutuset
CIEAR 100
CIFAR 10
4/3
0,1
0.3
0.5
10
0.1
0.3
0.3
10
CLI (kadond all M021
64.93
63.90
15.00
61.95
66.95
64.91
$7.98
$7.95
87.93
81.94
83.00
87.96
Prompir Conneee 20130
75.34
13.48
72.85
72.83
72.21
72.41
92.80
92.95
98.36
91. 89
93. 11
03.00
Fediii Chtrrni suna
74.73
12.89
04.85
26. 1/
74.07
73.90
$1.69
93.
93.21
93.04
FackmT TTPuii
77.51
73.83
72.31
70.99
69.40
97.33
91:
91.
pholMea (eumen 2015
80.20
75.20
74-53
73.00
72611
97.15
94.86
01:97
92.67
92.65
|FolMMA Y00ns
81.82
78.33
75.70
24.03
23.65
96.92
95.82
94.82
93.52
+3.01%
+3.07%
+1.895
+1.0%%
-0.0%
-0.5%
+0.10%
+3.00%
+0.9%
+0.205
-0.00%
Table 3: Test accuracy (%) of different methods on DomainNet and Cffice-Caltech00 with lable shift
and domain shift using Dirichlet partitioning (3 = 0.5).
DemainNet
Offer-Caltchhil
Method
Clipart
Infograph
Painning
Quickctraw
Real
Sketch
Avg
Amazon
Caltech
DSLK
Webcam
Avg
Rullonl al 2021
10.09
11.20
10.85
8.51
11:02
11.20
13.89
7.13
Fallah Si piics
24.77
31.87
23.87
22.40
22.82
24.77
30.93
26.09
20.46
24.65
15:73
WoMMMAA (O0R))
50.38
23.81
60.27
61.44
46.79
47.17
13.64
base and novel class scores indicate poor generalization. pFedMoAP performs well on local classes due its MoE- based prompt sharing, but it lags behind PFedMMA base and novel accuracy. By contrast, pFedMMA achieves the highest base and novel accuracy, surpassing FedPGP and demonstrating excellent generalization, while remaining competitive on local classes-only 0.74% lower than PFedMoAP.
Fig. 3 illustrates local and HM accuracy across varying numbers of shots 21, 4, 8, 16), showing the same performance pattern. Detailed results for all datasets are provided in Table in the Appendix.
Evaluation on Personalization. We further evaluate the personalization capability of pFedMMA on CIFAR-10 and CIFAR- 100 under challenging Dirichlet partitioning scheme, varying the concen- tration parameter 3 across 100 clients with 10% client participation per communication round. The results, summarized in Table 2 show that PFedMMA consistently achieves the highest accuracy both datasets, demonstrating its strong adaptability to highly non-Ill) data distributions.
Model Evaluation on Feature & Label Shifts. To evaluate the robustness of PFedMMA in realistic federated learning scenarios, we examine its performance under both label shift and feature shift using the DomainNet and into two clients via a Dirichlet distribution with 3 = 0.5, yielding 12 clients for DomainNet and 8 clients for Caicce-Cateeh100 10. The results in Table show that under these challenging heterogeneous conditions, traditional methods such as CLIP and PromptFL struggle to generalize effectively. In contrast, PFedMMA consistently achieves the highest average accuracy across both datasets, highlighting its strong robustness to cross-domain shifts. Additional experiments with one or two clients per domain and varying / are provided in Tables and 12 in the Appendix.
datasets. Following the standard protocol, each domain is split
4.3 ABLATION STUDY & HYPERPARAMETER SENSITIVITY
Impact of model. To further examine the performance of PFedMMA under a different backbone, we report results with ViT-B/32 on the average of six datasets across five shot settings, comparing against three advanced baselines (Table D. While PFedMMA shows slightly lower local accuracy than FedOTP and pFedMoAP. this gap narrows as the number of shots increases. Importantly, PFedMMA consistently achieves the best trade-off between personalization and generalization, demonstrating stable improvements in the harmonic mean across all settings. Detailed results for all datasets are provided in Table 10 in the Appendix.
Dimension of the Shared Layer. Table (bottom-lett reports the average accuracies over four datasets and five shot settings. As shown, using a larger 128-dimensional representation yields slightly better performance than 32 dimensions. However, to keep the number of trainable parameters low, we consistently adopt the 32-dimensional setting throughout the paper. Detailed results are provided in Tables [1 and i4 the Appendix.
Scaling Factor a. The scaling factor controls the balance between general features and task-specific features. We systematically evaluate its effect, with results summarized in Table (top-left). Our
8

---

Published as a conference paper at ICLR 2026
Table Average performance across six datasets using the ViT-B/32 backbone under different shot settings (1, 2, 4, 8, and 16).
Meloal
Locd
Buxc
Novel
HM
Local
Boc
Novel
HB
Leca.
Bos
Novel
HM
Local
Sloc:
Novel
Lood.
Bux
Novel
HM
EatPee
79:31
79.35
80.35
83.33
81.43
77.66
1.50
59:07
75.61
79.63
75
FaIITP
56.81
11:30
70.56
18.00
89.68
11 10
23
70.00
M1.11
11 RD
19
33
02.01
9-95
19 30
17.25
94 60
6.07
16:14
16 tn
poatMAAP
98.11
5183
97:31
-0.29
-1.29%
Table 5: Ablation study on PFedMMA design choices, including scaling factor, adapter dimension, starting layer, and adapter sharing strategies.
Local
Base
Novel
HM
Novel
0.0001
91.40
5,06
5.76
7.62
12
96.49
76.85
81.37
83.57
0.0005
91.45
46.34
59.16
59.15
10 -3 12
96.61
78.14
81.98
84.38
90.91
72.21
78.48
79.37
78.43
81.82
0.005
91.03
78.65
81.77
83,25
6 12
91.53
78.53
81.76
83.32
0.01
91.47
78.12
81.56
82.03
5 12
91.58
78.67
81.71
83.38
Dims
Local
Base
Novel
HIM
8
89.15
72.23
76.41
78.24
Method
DTD
Caltech
Flowers
OxfordPets
16
89.84
72.51
77.36
78.93
Baseline
61.10
96.61
73.82
91.91
90.91
72.21
78.33
79.37
Baseline
62.19
71.37
78.31
79.15
PFedMMA
76.38
99.48
86.34
97.39
128
91.78
72.55
78.23
79.68
PFedMMA achieves the best trade-off performance (HM) between local, base, and novel classes 0.005. A larger scaling factor enables faster adaptation hase classes but leads to weaker performance on novel and base classes, whereas a smaller scaling factor hinders effective tuning for downstream tasks. Detailed results are provided in Table 16 the Appendix.
Starting Layer l. We evaluate different choices of encoder layers for integrating PFedMMA in Table 5 (top-right). As shown, updating the last three layers yields the best HM performance, which attribute to the limited amount of training data in few-shot settings. Accordingly, we consistently set 10 for CLIP datasets throughout the paper. For other datasets, updating additional layers leads to better results, so we adopt 5. Detailed results are provided in Table 17 in the Appendix.
Adapting Variant Options for Personalization. We evaluate the effectiveness of different design choices of MMA in personalized federated learning. In Table [ (bottom-right), we compare two alternative baselines: treating all adapters as global (Baseline 1) and using the shared adapter as the personalized component while treating the up- and down-projection adapters global (Baseline 2). As shown, pFedMMA achieves significantly higher local accuracy than both baselines. Moreover, achieves superior base and novel performance compared to state-of- the-art prompt learning methods, as shown earlier, underscoring its ability to strike strong balance between personalization and generalization.
Adapting Variant Options for FL Aggregation. We next ablate how the shared adapter is aggre- gated across clients to localize the main information- sharing channel. In Table we compare three vari- ants that differ in which modality-peecific shared block is federated: Vision Only, where only the vision- side shared block is aggregated; Text Only, where only the text-side shared block is aggregated; and Both Vision & Text, where separate shared blocks for each modality are aggregated simultaneously. These variants achieve very similar local accuracy, with small but consistent differences in base, novel, and HM: aggregating text-only or both modalities yields slight edge in HM over aggregating vision-only. Our full pFedMMA, which uses a single mollillmddal shared projection rather than two separate sppecii ones, further improves local, base, novel, and HM over all three variants, suggesting that tying the modalities through unified
Table 6: Comparison of FL. aggregation vari- ants (vision-only, text-only, and both-sides) for the shared adapter.
Methods
Local
Base
Novel
HM
Vision Ouly
95.81
71.19
76.07
79.31
Text Only
95 99
71.19
76 76.13
79.38
Vision
PFedMMA (Ours)
96.14
71.78
76.17
79.70

---

Published as a conference paper at ICLR 2026
shared adapter provides slightly stronger and more stable information sharran mechanism without harming personalization.
4.4 LEARNING CURVES
To further analyze the convergence behavior of PFedMMA, we plot the average local accuracy over communication rounds across five different shot settings in Fig. 4 As shown, pFedMMA consistently attains high accuracy and converges faster than the baselines. Detailed results are provided in Fig. in the Appendix.
FechCC
oPechMA Dwrd
Figure 4: Accuracy learning curves of pFedMMA and baselines.
4.5 TRAINING COST ANALYSIS
Table summarizze the computational and communication costs together with accuracy. PromptFL has the smallest footprint (8, 191 trainable; 8, 8,192/8, 092/8,192 192 per round) but-most importantly- marked drop in local accuracy (88.93%), indicating weaker personalization under heterogeneity. FedPGP increases local trainables to 12,416 without extra communication but incurs the highest memory (13,374 MiB) and slower training, and its HM accuracy (79.09%) lags its strong local accuracy (95.38%). FedOTP doubles prompt capacity (16,384 trainables; same 8, 8,192/8, 192 com- munication) and attains very high local accuracy (97.55 (97.34%) but suffers extremely low HM accuracy (31.08%), suggesting poor cross-client generalization. pFedMoAP adds local gating module, raising local trainables (74,240) and the per-round download (73,728) while achieving the shortest training time (902 s) and strong local accuracy (97.89%) pFedMMA (ours) communicates only the shared adapter blocks (3,072/3,072) while keeping 248,832 parameters local, yielding the best HM accuracy (84.15%) and competitive local accuracy (97.17%), thus offering the most favorable aecurac-coomunnica. trade-off. Please refer to Appendix F for additional details.
Table 7: Comparison of computation, communication, and accuracy for five personalized federated learning methods under CLIP ViT-B/16.
Methods
Local TrainNe Paran.
Per Com.
(MiB)
Local
Ave.
Promeefi
Komezta al b002hh
199
8,191 19972811 8,19
1.645
5,116
88. 91
$3.00
3,940
13,374
Ral0T
16.364
1.388
248,822
3/072/3,072
4,634
97.17
5 CONCLUSION
In this work, we introduced PFedMMA, a novel personalized federated learning framework that leverages multi-modal adapters to adapt large-scale vision- languaee models under heterogeneous data conditions. The proposed architecture separates each adapter into modality-specific and shared projec- tion components. Clients update all components locally, but only the shared projection aggregated globally. This asymmetric optimization strategy enables client-specific adaptation while maintaining a globally aligned feature space for effective generalization. Moreover, the communication-effice nature of the framework makes it scalable to real-world federated deployments. Our extensive experiments across diverse datasets demonstrate that PFedMMA consistently outperforms existing prompt-base based PFL methods in both domain- and category- level generalization, while retaining strong personalization capabilities. This work can motivate further exploration of adapter- based architectures for personalized federated learning in multi-modal settings.
based
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation under Grant 2419982, Grant 2342253, and Grant 2236483.
REFERENCES
Amir Aghdam and Vincent Tao Hu. Actalign: Zero-shot fine-grained video classification via language-guided sequence alignment. arXiv preprint arXiv:2506.22967 2025.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv:1912.0083 2019.
Kasra Borazjani, Payam Abdisarabshali, Fardis Nadimi, Naji Khosravan, Minghui Liwang, Xianbin Wang, Yiguang Hong, and Seyyedali Hosseinalipour. Multi-modal multi-task (m3t) federated foundation models for embodied ai: Potentials and challenges for edge integration. arXiv preprint 1r1i: 2025.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-ming discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings. part VI 13, pp. 446-461 Springer, 2014.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 13:11285-11297, 2020.
Guangyi Chen, Weiran Yao, Xiangehen Song. Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. In The Eleventh International Conference on Learning Representations, a.
H. Chen, Y. Zhang, D. Krompass, Gu, and V. Tresp. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678 2022.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference on Learning Representations, b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International conference on machine learning, pp. 2089-2099, PMLR, 2021.
Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. In Proceedings of the 41st International Conference on Machine Learning, 2024. FedPGP.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ming-Wei
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Yao Du et al. Mixture of prompt learning for vision language models. Frontiers in Artificial Intelligence, 8:1580973, 2025.
Farzan Farnia, Amirhossein Reisizadeh, Ramtin Pedarsani, and Ali Jadbahaie. An optimal transport approach to personalized federated learning. IEEE Journal on Selected Areas in Information Theory, 3(2):122-77, 2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8064-8073, 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better languuge models with feature adapters. International Journal of Computer Vision, 132(2):581-595 2024.
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, and Ramtin Pedarsani. Few-shot adversarial low-rank fine-tuning of vision-language models. arXiv preprint arXiv:2505.15130.
2025a.
Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani. Communication-effiiioii and tensorized federated fine-tuning of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 24192-24207, 2025b.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp- 2066-2073. IREE, 2012.
Shuai Gong, Chaoran Cui, Chunyun Zhang, Wenna Wang, Xiushan Nie, and Lei Zhu. Federated domain generalization via prompt learning and aggregation. arXiv preprint arXivv221 2..12111111033 2024.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp. 1364-1374 1374, 2023a.
Tao Guo. Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Prompttl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions Mobile Computing, 23(5):5179-511 2023b.
Yuncheng Guo and Xiaodong Gu. Mmrl: Multi-modal representation learning for vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25015- 25025, 2025.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of effficeerrefficient transfer learning. In International Conference on Learning Representations.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
12

---

Published as a conference paper at ICLR 2026
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference machine learning, pp. 2790-2799 PMLR. 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 106.0965, 2021.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7865-7873, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- sainn-la representation learning with noisy text supervision. In International conference muchine leurning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pp. 709-727. Springer, 2022.
Haojun Jiang, Jianke Zhang, Rui Huang, Chunjiang Ge, Zanlin Ni, Jiwen Lu, Jie Zhou, Shiji Song, and Gao Huang. Cross-modal adapter for text-video retrieval. arXiv preprint SMt::221.09962. 2022.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022-1035, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19113-19122 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsual Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15190-15200, 2023b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny i2ages. 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html, 5(4):1, 2010.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 fourth world conference on smart trends in systems, security and sustainability (WorldS4), pp. 794-797. IEEE, 2020.
Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. Fedtp: Federated learning by transformer personalization. IEEE transactions on neural networks and learning systems, 2023a.
Hongxia Li, Wei Huang, Jingya Wang. and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12151-12161, 2024.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-ima image pre- training. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10965-10975. 2022.
Qinbin Li, Bingsheng He, and Dawn Song. conel-contrastive federated learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 10713-10722, 2021a.
13

---

Published as a conference paper at ICLR 2026
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357-6368. PMLR, 2021b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021.
Xiaoxiao Li, Meirui JINGG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. In International Conference Learning Representations.
Ying Li, Xingwei Wang, Rongfei Zeng, Praveen Kumar Donta, Ilir Murturi, Min Huang, and Schahram Dustdar. Federated domain generalization: survey. CoRR, abs/2306.01134 2023b.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philipee Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprini arXiv:2001.01523, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys, 55(9):1-35, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
W. Lu, X. Hu, J. Wang, and X. Xie. Fedelip: Fast generalization and personalization for clip in federated learning. CoRR, abs/2302.13485 2023.
Wang Lu, HU Xixu, Jindong Wang, and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. In ICLR 2023 Workshop Trustworthy and Reliable Large-Scale Machine Learning Models.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5206-5215. 2022.
Jun Luo, Chen Chen, and Shandong Wu. Mixture of experts made personalized: Federated prompt learning for vision-language models. In ICLR, 2025.
Nazanin Mahjourian and Vinh Nguyen. Sanitizing manufacturing dataset labels using vision-language models. arXiv preprint arXiv:2506.23465, 2025.
Andrea Manoel, Mirian del Carmen Hipolito Garcia, Tal Baumel, Shize Su, Jialei Chen, Robert Sim, Dan Miller, Danny Karmon, and Dimitrios Dimitriadis. Federated multilingual models for medical transcript analysis. In Conference on Health, Inference, and Learning, pp. 147-162. PMLR, 2023,
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effiicei learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEFF, 2008
14

---

Published as a conference paper at ICLR 2026
Jachoon Oh, SangMook Kim, and Se- Young Yun. Fedbabu: Toward enhanced representation for federated image classification. In International Conference on Learning Representations.
Haniyeh Ehsani Oskouie, Christina Chance, Claire Huang, Margaret Capetz, Elizabeth Eyeson, and Majid Sarrafzadeh. Leveraging large language models and topic modeling for toxicity classification. In 2025 International Conference on Computing, Networking and Communications (ICNC), pp. 123-127. IEEE, 2025.
Bikang Pan, Wei Huang, and Ye Shi. Federated learning from vision-language foundation models: Theoretical analysis and method. Advances in Neural Information Processing Systems, 37:30590- 30623, 2024.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp.
IEEE, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yr Yi Lin. Text-driven prompt generation for vision-languan models in federated learning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurlPS 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-877 8763. PmLR, 2021.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model-
agnostic distributed multitask optimization under privacy constraints. IEEE transactions neural networks and learning systems, 3273)3370-3322,, 2020.
Ofir Ben Shoham and Nadav Rappoport. Federated learning of medical concepts embedding using behrt. arXiv preprint ar35: 13052, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv; 1212.0004 2012.
Shangchao Su, Mingzhao Yang. Bin Li, and Xiangyang Xue. Cross-domain federated adaptive 2022.
prompt tuning for clip. arXiv preprint (XXXx:2211077866
Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309- 23320, 2021.
Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. Fedperfix: Towards partial model personalization of vision transformers in federated learning. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 4988 2023.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in neural information processing systems, 33:21394 33:21394-21405, 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE transactions on neural networks and learning systems, (111)))55779600 2022.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learing for generalized zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6062- 6069, 2020.
15

---

Published as a conference paper at ICLR 2026
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.1023 2019.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485- 3492. IEEE, 2010.
Fu-En Yang, Chien-Y Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in feder- ated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19159-19168, 2023.
Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 23837,, 2024.
Yiyuan Yang, Guodong Long, Tianyi Zhou, Qinghua Lu, Shanshan Ye, and Jing Jiang. Federated adapter on foundation models: An -iittiiiiiiiittoon approach. arXiv preprint arXiv:2505.01075. 2025.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference Learning Representations.
Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593-1603, 2024.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations.
Renrui Zhang. Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022.
Zhuo Zhang, Yuanhang Yang, Yong Dai. Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fed- petuning: When federated learning meets the parameter-efficien tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics 2023, pp. 9963 9977. Association for Computational Linguistics (ACL.), 2023.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Reduce communication costs and preserve privacy: Prompt tuning method in federated learning. arXiv preprint 22X.:2208.122666 1 (2), 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 168 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 15659-15669, 2023.
Erfan Ziad, Zhuo Yang. Yan Lu, and Feng Ju. Knowledge constrained deep clustering for melt pool anomaly detection in laser powder bed fusion. In 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), pp. 670-675. IEEE, 2024.
16

---

Published as a conference paper at ICLR 2026
A PIPELINES OF THE PROPOSED ALGORITHM
For a better understanding of the steps of the designed algorithm, we present PFedMMA in Algorithm
Algorithm PFedMMA
1: Input: Step size y, number of communication rounds T, number of local epoch E, number of clients N.
for communication round <- to do
3: Select a subset of [St] clients, St
4: Send (Wi..
Wi.) } to the selected clients
5: for clients i E S; in parallel do
6: for local update
F do
7:
for trainable parameters W ()()) With W W Jd.i 20.0 Ju,
Woad (e
W Wi (W!
9: end for
10: Client i sends Ja,i to the server
11: end for
12:
server: W DN WLE Jaii
jE (l,......
13:
end
Return: (w() jdiss Win W W Wisij ; i (l...., LL), je j (e....,))
14:
B RELATED WORK
B. PERSONALIZED FEDERATED LEARNING
Personalized Federated Learning (PFL.) has emerged pivotal research direction address the limitations of conventional federated learning McMahan et al. (2017) when faced with heterogencous client data. Unlike standard FL, which learns a single global model, PFL aims to produce tailored models for individual clients, thus better coping with statistical and systemic heterogeneity TTnn et al. (2022) Kulkarni et (2220). Several personalization strategies have been proposed, including local fine-tuni tuning Mansour et al. (2020); Tann et al. (2022); Wang et al. (2019), regularization based optimization Li et al. (2020) 202 10 Dinh et al. 2020), and parameter decomposition into shared and client-specific components Arivazhagan et al. (2019); Oh et al. Collins et al. (202T). Other methods pursue clustering of clients to exploit latent similarities Huang et al. (2021) 1); Zhang et al.)
Sattler et al. (2020); Ziad et al. (2024), leverage attention mechanisms and adaptive layers Liang ctt al. (2020): Li et al. 20; Sun et al. (2023). To further improve adaptability. techniques like FedBBN Li et al. and PartialFed [Sun al. (2021) address feature shift via local normalization or selective personalization. Hybrid global local learning approaches have also been developed Deng al. (2020): Chen & Chao 2022). FedOT Farnia et al. (2022) proposes learning optimal transport maps that align local distributions to a shared probability space, enabling global classifier to be trained more effectively; personalization achieved by composing this shared model with each client's 's transport map. While these approaches have demonstrated success, they typically center on traditional MI. architectures and do not yer fully leverage the potential of large pre-trained models, such as vasion-language or foundation models, for personalization.
B.2 FEDERATED PROMPT LEARNING FOR VLMS
Federated Prompt Learning (FPL) extends the flexibility of prompt tuming for adapting large pre- trained models such as CLIP Radford et al. (202T) the FL settings, enabling efficient and person- alized downstream task adaptation across decentralized clients. Early works like CoOp Zhou et al.) (202bb) and CoCoOp Zhou et al. (202aa) laid the foundation by introducing learnable continuous prompt vectors, which sparked interest in federated extensions. PromptFL Guo et al. (2023b) and FedPrompt Zhao et al. (2022) introduced FL-style prompt aggregation, performing FedAvg McMahan et al. (2017) over client-specift prompt updates. FedPR Feng et al. (2023) explores visual prompt learning within the null space of global prompts for MRI reconstruction, while FedAPT Su et al.
(2022) focuses domain-ade aonaineadaptive prompt tuning for cross-domain image classification. To
17

---

Published as a conference paper at ICLR 2026
hance personalization, pFedPrommt Guo et al. (2023a) introduces a non- non-parametric attention module over local few-shot memory, and pFedPG Yang et al. (2023) and FedTPG Qiu et al. design server- side prompt generators to issue personalized prompts each client. FedCLIP et al. integrates attention-based adapters to better exploit the pre-trained model's knowledge. Furthermone, Le et al. (2024) leverages Optimal Transport to align global and local prompts, and FedPGP utilizes prompt-wise contrastive losses to better capture diverse category-level traits across clients. Recently, PFedMoAP et al. (2025) rethinks prompt sharing by treating pre-aggregated prompts from other clients as non-local experts in a Mixture-of -ixture-of-Experts framework, enabling effective personalization via a lightweight, attention- gating mechanism. Theoretical analysis of [PL et al. (2024)
provides deeper understanding of its convergence properties.
B.3 EFFICIENT TRANSFER LEARNING FOR VLMS
Traditional transfer learning approaches typically fine-tune all parameters of pre-trained VLMs Devlin (et al. (2019); He et al. (2016), but this becomes increasingly impractical as model sizes scale up, especially under computational or data constraints. To mitigate this, the community has embraced parameter-effiieent transfer learning strategies that modify only a small fraction of model parameters. Among these, prompt learning techniques, briefly introduced in the previous section, optimize lightweight vectors or tokens to steer the model without altering its backbone Zhou et al. al. Lu et al. (2022): Khattak al. (2023)) Although effective, they often limited in their expressiveness or modality interaction. As a result, adapter-base based methods have emerged as a powerful alternative. CLIP-Adapter Gao et al. (2024) and Tip-Adapter ZZang al. (022) inject lightweight MLP layers after the image encoder to refine visual representations. Tip-Adapter further improves efficiency by caching training features for fast inference. However. these image-only approaches neglect the cross- modal nature of VLMs. To address this, MMA Yang et al. (2024) introduces a multi-modal adapter architecture that fuses features across the vision and language branches via a shared representation space, enabling gradient flow between modalities. Similarly, other works explore deeper adapter integration, such as inserting adapters within self-attention and MLP blocks Tiang et al. (2022), allowing more granular control over the representation learning process. These advances mark shift from single-stream to multi-stream adaptation, aligning with the unique demands of multi-modal tasks. In federated settings, where full model updates are prohibitive, adapter- based techniques offer compelling balance between personalization, generalization, and communication efficiency-makene them well-suited foundations for multi-mode federated frameworks like ours.
B.4 FEDERATED OUT-OF-DISTRIBUTION AND DOMAIN GENERALIZATION
A complementary line of work studies federated domain generalization (FedDG), where the goal is to train a single global model that generalizes to unseen target domains under client heterogeneity. FedDAT Chen al. (2024) tackles multi-modal heterogeneous FL for foundation vision-language models via a dual-adapter teacher and mutual knowledge distillation, improving global performance across diverse vision-language tasks under domain shift. PLAN Gong et al. (2024) introduces a FedDG framework for pre-trained vision-language models based on visual and textual prompt learning and attention-based prompt aggregation. explicitly using a wwwee-oon-domin--oee
protocol to adapt a global CLIP-style model to unseen domains. Other recent methods similarly design adapter- or prompt-based FedDG algorithms to enhance out-of-domain robustness of federated foundation models [i et al. (2023b): Yang et al. (2025); Lu et al. (2023). In contrast, our work formulated in the personalized federated learning (PFL) setting, where each client maintains trade-
its own model (or adapter) and we explicitly optimize the off; consequently, we treat FedDG methods as complementary rather than direct baselines, and instead compare against personalized prompt- and adapter- adapter-based methods, while showing that our approach achieves comparable personalized performance and substantially stronger cross domain generalization within this PFL protocol.
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.I DATASET SETUP
For evaluation, we consider broad set of eleven visual recognition benchmarks that span diverse tasks and levels of granularity. Table&ppooiccs a comprehensive overview, detailing the task type, number of classes, training and testing sizes, client splits, and the heterogeneity assumption used in our experiments.
The pathological partition setting is adopted for datasets such as Caltech 101, Flowers 102, OxfordPets, Food101, DTD, SUN397, and UCF101, where each client is assigned data corresponding to a limited number of classes, creating strong non-IID conditions. This simulates realistic personalization scenarios for fine-grained recognition, texture classification, scene recognition, and video action recognition.
For CIFAR-10 10 and CIFAR-100, we follow the common Dirichlet partitioning scheme with varying / values to control the label skew among 100 clients. This allows systematic evaluation under different degrees of heterogeneity.
To capture the challenges of mulli-ddmmale learning, we also include Office-Caltech0 and DomainNet. Office-Calteche contains four domains (Amazon, Caltech, DSLR, Webcam), reflecting variations across acquisition devices and environments, while DomainNet consists six domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch), which are significantly diverse and large-scale. For these benchmarks, we use 10 selected classes and evaluate both Lungle-cientt-er--ommas and multi-clien per---e---rre domain partitions.
Table 8: Statistical details of datasets used in experiments.
Dalasct
Task
PCTassss
PClicnts
Sample Rate
Truining Size
Testing Size
Domains
Heterogensity
Calealol Fri
2014
Object recogrition
100
10
100%
4,128
2,465
Pathologica
Fowers 10
2015
Fine graited fowers recognition
102
10
1007
4,053
2.463
Patholigical
Oxlond
2014
Finegrainal pets necogaition
Patholagical
Feod11
grained recogratin
Putological
monggitton
SUN
10
76,128
21,750
Patelogical
UCP100
S1corer 2013
Action nuoegition (vilen)
101
10
1005
9.517
8765
Pathological
UIFAR 10 Knzhemse at
PUTO
Inage constrcatinn
10
160
1056
500000
T0
Dvr(.)
CIFAR: Ich
2009
Image classliaatkn
100
100
10%
30,000
10,000
Dke(d)
DamuitNe Pengusal
Image socngriiiin
48
Dixtc)
Insge resogrition
611
C.2 EXPERIMENTAL SETUP
All models are trained using the SGD optimizer with learning rate of 0.001. Each experiment repeated three times with different random seeds, and we report the average performance. The final results are obtained by averaging performance across all clients. All experiments are implemented in PyTorch and run on NVIDIA A6000 GPUs.
Base-to-Novel Class Generalization. To evaluate generalization, we divide each dataset evenly into base and novel classes. Base classes are distributed across clients without overlap, such that each client only observes subset during training. Clients train their local models on their own classes, and evaluation is performed on three levels: (i) local classes (the client's 's own training classes), (ii) base classes (classes seen by other clients but unseen locally), and (iii) novel classes (completely unseen during training). Accuracy is averaged across 10 clients.
Feature & Label Shifts. To evaluate robustness under realistic federated learning conditions, conduct experiments with both label shift and feature shift using the DomainNet and Office-Caltech10 datasets. Each domain partitioned into one or clients using a Dirichlet distribution with varying 3, resulting in 6 or 12 clients for DomainNer and 4 or 8 clients for Office-Caltechh This setup generates heterogeneous client distributions, effectively simulating domain shifts.
Personalization. For personalization analysis, CIFAR-IO and CIFAR-100 are partitioned among 100 clients using symmetric Dirichlet distribution. In addition, for the CLIP datasets, we follow the pathological partitioning strategy from the base-to-novel generalization setting, where classes are non-overlapping across 10 clients.
19

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS RESULTS
D. BASE-TO-NOVEL CLASS GENERALIZATION
Table 9: Top-1 accuracy (%) of different methods across 7 datasets using ViT-B/16 the backbone.
Imwage on Eataiee
SUNT47
BTT
Shon
Madted
Lacal
Sara
Neeol
133
Laccl
Blass
Navil
106
Loca
Sais
Noxd
184
Lecal
Bast
Nevel
IBM
76.30
26.51
51.21
T3.10
60:41
66:28
13.52
71.32
09.23
26,55
71.12
56.30
54.30
20:13
56.02
8633
6553
75.56
T1:73
77:21
07:00
03.83
99.35
80:23
75.33
50:94
97:38
19.00
30.00
5118
11:31
1.85
18.21
16.02
91:81
3697
18.05
20.15
11.63
V0.17
77:48
11:20
84.15
94.66
70.09
76.37
79:34
03.58
71.54
76.00
70.70
97:45
$5.44
01.58
67:35
OltvedPass
Pled101
Mated
Lacal
Suz
Naed
E24
L.53)
Blcc
1963
164
Lecl
Baac
Kevel
39.33
91:22
16.31
91.77
18.14
17122
14.11
10.34
15:40
91:22
18..0
351.3
00.91
12.18
41.23
Proc Lar
06.11
08:18
01:37
96.02
05.17
05.15
06:31
00-42
01.94
96.38
66:63
58.26
109.00
11:91
51.33
16-1
06.35
12.50
11.00
10
100.00
$5.30
94.78
100
18.03
04.23
18.00
07.43
91.13
94.77
10.32
05.85
65.03
74.38
76.35
Amatgge
Earants
SUN793
B7T
Mettd
Local
Saie
Need
134
Loca
Ban
194
Lock
Sais
Novd
184
1ecal
Bac
Novel
114
CLi Budjanist Jalnl
76,36
76.51
51.21
T3.10
00.11
60.38
13.32
71.32
03.29
01:23
70.55
71.12
$6.30
54.35
20:13
36.02
$5.34
$583
72.37
$3.76
73.01
12.00
95.33
00.23
$6.00
76.27
46.08
03.91
16.03
91
3.50
06.41
11.50
8.30
1956
64.13
30.0
05:33
17:00
24.11
19.00
19.29
01.01
08.25
M0:71
10.00
18.25
18.54
55.33
72.31
76.45
09125
95.02
90:30
01.34
02.03
Odteedaa
Predi0
UC2001
Mated
Lecil
Seit
Nard
120
1953
361
Lecl
Novcl
151
29.35
99:22
16:51
91.77
16.14
51:23
$4.31
10.34
15:40
91.22
$91.13
39:31
05.91
15.15
11.23
Pres Lere
07:35
9009
90.1
50
08.13
08.33
03.47
96.57
00.51
91.38
91.54
63.48
71.04
18.08
1.35
73.24
41 13
7211
9193
73.58
06.36
36.33
9903
91333
90.10
99.23
19.00
34.33
16.50
17115
91.24
98.75
10.23
96.99
65.54
12.20
75.93
1F1
Meted
Local
Suz
Need
124
Loc3
Bac
194
Lock
Saiz
Novd
184
1oc1
Bac
Novel
1151
CLi Pudianist Jaiel
76.36
76.51
51.21
13.10
00.41
60.33
13.32
71.32
67.29
01125
76.55
71.12
36.35
34.35
35:13
36.02
$7112
75306
55.15
83.75
30.72
71.51
03.76
74.66
04.48
12.43
72.53
$5.35
$2.99
03.35
$1.80
05:00
9131
05.00
01:81
17.96
76mt
19:0
19:01
1.13
41.33
09.0
16000
72'97
01.00
04.20
91.35
18.29
76.12
09.51
55.35
71.29
76.35
29/00
95.75
38.45
61.91
$7:37
Prodi0
UC2001
Maled
Levil
Need
11a
196
1
1111
Lichl
Novel
1951
20.05
09-22
16.51
91.71
16.16
51.03
$1.21
95.34
58.40
09.02
181.1
30:00
00.91
10
10.39
96.55
05.54
07.57
07.84
07.00
12.55
05.65
06.25
98.34
91.15
00.55
82:64
71.72
79.79
20.38
96.75
$5.09
96.08
98.08
65.00
03.53
05.40
$1.57
27:55
73.43
06.67
17.33
9900
9188
16600
10 21
190.0
18.00
34.40
00.01
18.00
90.34
90..3
00.03
05.97
15.00
19.30
29.25
Aveniggian
Mathal
Tamel
Rala
Novel
Tacal
Rlao
Nivo
196
1.0.8
Fale
Nevil
145
1200l
11.0
Nevel
1FM
CLif Budtiaittitt 1a1el
78.36
16.51
51.21
12.30
00.21
00.33
13.32
11.32
61.29
09125
16.55
71.12
36.05
34.25
20:10
26.02
71:30
20.34
53.85
$2.36
78.42
14.97
12.34
78.36
1058
99.35
05.35
$1.16
75.00
02.35
29.32
03.85
96.10
62.00
$7.45
28.00
36.95
13.57
23.49
01.71
41.34
26:15
27.86
$1.78
06.33
78.13
73.90
09-75
21:08
0.885
22.56
04.03
77.67
75.84
09.16
6.
01.90
06.07
Olived
Predi0l
UC2701
Meleed
Level
Seie
Neec
PPM
Lovel
1963
1
1111
Licsl
Nevel
1151
20/05
$9.02
16-31
01.71
14
01.00
$1.71
99.33
58.40
30102
911 11
30-00
00.01
15
16
30.75
04.05
06.55
91.37
00
01.10
92.99
15.66
58.11
09.95
191.0 00
09.90
01.33
15.11
13.05
77.09
00
79.38
03.16
22.49
88.87
17:17
85.30
00077
01.48
03
01.36
05.22
$3.37
12
10.12
75.02
19964
89-90
96.00
02.10
00.00
97.07
04.34
06.00
05.56
$0.40
00.36
91.78
03.90
18.34
78.12
Avenggaan
Marhal
Tamal
Rate
Nevel
FMM
Taral
Rain
Navol
19K1
Fara
Nivil
185
120sl
Nrvi
HMM
CLI Paaleaiinal S151
76.36
36.11
$1.31
75.00
(6-41
68.36
75.53
71.53
03.55
09.33
26.35
71.12
5-36
$4.82
56.15
$,.37
$2.05
$2.96
78386
80.33
14.36
14.07
18.55
72.00
$2.99
57.35
17:25
01.35
05.00
38.90
$7/83
4761
35.07
76.36
38.40
27.31
22.31
22.83
43.91
25.09
75.50
55.00
56.73
0h:00
72.99
30 00
08:43
08.90
60.13
20.37
36.55
09.75
06.07
01.00
65.0
Matal
Lacal
Sala
Novd
134
Taacal
Bamms
Novoi
184
T,c4
3ara
Navil
Loci
11mm
Novel
115
(TI Badfaaiei N101
20/05
$9.13
96-31
01.77
16.14
00
01.11
05.34
58.10
30-17
11.30
20:30
00.01
15
25.15
30.74
06.00
96.25
16.00
05.02 43
16:30
09.43
07.46
05.53
58/0
29113
11.91
39.65
77
13443
14.35
75.86
05.15
60:79
05.81
06.50
88.53
96.22
00.07
12.39
05.37
$3.00
$7.32
$7.09
05.03
04.40
71.02
06.03
39.4
07/00
04.00
00.36
07:08
$4.35
00.08
19-44
01.33
91.31
08.00
12:47
77.54
20

---

Published as a conference paper at ICLR 2026
Table 10: Top-1 accuracy (%) of different methods across 7 datasets using VIT-B/32 as the backbone.
OxfoedPes
SUNS97
DTD
Methol
Niveil
10M
Bonc
Nived
Novcl
1IM
CLI
89.57
54,47
53.27
Proce bo www.oooo p00
98.78
53.33
N4.53
94.70
73.35
15.35
68.04
72.75
77.22
76.35
57.00
68.96
Faat
2014
95.01
91:11
94:51
64.10
05:33
16.76
47:21
53.20
88-48
61.99
41.74
55.18
21:64
9.01
5.74
97.20
51.19
(PUMMA
$7.92
70.65
59.19
2,3999
-6.31.5
16
Fvas1101
U/2100
Novel
HM
Locel
Novel
Locol
Novel
HM
CLIP euall 1003
$2.53
0L.00
14.00
03.54
54.33
54.27
$5.43
84.67
65.16
65.36
71.25
67.15
Procegert
Fastm Sio0
69.55
23.12
21.36
92.21
9.61
14.00
16.13
58.36
9.84
6.15
10.45
2005
00.52
21
00.33
07.23
05.43
31.56
55.51
09.39
05:23
24.08
31.24
35.7b
(FelMMM roursy
98.56
64.74
05.70
$4.59
85.30 50
88.24
87 30
66.91
71.55
74.41
+0.37%
-2.27%
+1.05
DichedPos
SUN199
I7TD
Medul
Local
Bose
Novel
HM
Locol
Base
Novel
HM
Local
Basn
Novel
HM
72.99
Promiee
04.30
75.13
40.66
63.45
Fou
94.57
11.00
02.94
91.15
93.15
55.75
57.34
63.03
79.66
72.76
45.61
62.01
Rearra merern senny
99.84
11
6l
23.65
87.3
8.74
1.64
8.80
95.94
18:11
19.37
18.64
61.15
5161
94.79
76
75.41
19:78
08
PFeIMMA
92.67
73.41
Cilieca101
Food101
UCF100
Methed
HM
Novel
HM
CLIH ALIIHuallonian etal. 2031
92.53
04.00
04.00
93.54
$4.33
64.27
65.43
$4.67
65.16
65.36
71.25
67.15
Promece Co0EN
56.89
92.50
02.25
05.63
85.15
56.18
$7.83
96.65
50.58
$0.35
51.95
73.14
96.66
94.89
96
83:07
09.77
60:74
60.58
86.15
00
68.90
08.21
40.5%
17.06
48.37
(FeIMM 100sy
97.57
94
98.64
95:43
95.17
54.73
85.72
88.30
53.05
66.59
71,07
72.94
0.21%
-1,21%
3.95%
Methel
Local
Bose
Nondl
HMM
Locil
Buse
Novel
HM
Local
Buse
Novel
HM
CLIP CLIPR Rodoonne 2091
$6.40
56.37
90.19
89:57
09.31
(0.70
72.99
70.93
52.27
53.15
14,47
53.27
Promettr 2o00e
94.34
71.16
19.38
Fistor 200g
59.50
10:21
21.50
8.38
7.93
22.17
20.47
PPelMinn (aaae M005
100.00
13
65 34
51.07
M1.34
76 09
37 64
47.43
94.77
16.45
6M
29:13
95.64
88
92.03
13:44
90
75.08
75.04
56.95
19.05
Caltech101
1rad
UCCF10
Merhoal
Tocal
10M
Taval
Bosa
Nowd
Tawal
Boan
Novel
IMM
Rodiont en 1001
93.54
$4.27
$5.43
65.16
$1.25
67.15
Promect Picch
$5.44
06.50
02.12
95.13
83.59
55.51
$7.55
96.24
79.00
37.82
54.20
71.75
FaiN 1e 18 2015
$5.50
96.45
00.30
04355
85.11
$6.00
$8.01
86.73
81.62
76.40
56.31
94.22
Reltm lo
33.58
99:36
9:33
17:45
83:31
8.54
1.10
11.00
(PPMMMA Itheno
97.50
30
87.71
86.53
68.00
74.11
&
+0.35%
-3,15%
-0.15%
OcfondPaas
SUN397
MaBeel
Novel
HM
Novel
Locil
Novel
HM4
CLIPMooloo aod etal FORT
$6.40
30.27
14.09
89.57
09.81
59.75
72.09
70.83
52.27
53.15
54,47
$3.27
Proveeee 200 Hos
9011
91.44
No.37
92.90
74.05
74.06
73.81
73.95
61.30
60.58
47.05
55.95
93.00
67.13
61.38
2025
100.00
50.55
54.74
57.58
34,39
(PeMMMM tonesy
93.18
05.73
M1.30
79.27
71.52
74 10
74383
02.18
55.07
48.66
57.35
-0,677
-0.70%
-7.12%
Metred
Local
Bose
Novel
HIM
Lacol
Buse
Novel
HM
Locil
Base
Novel
HM
CLIP/Ru Coutodde
Provuce
45.45
55.10
64.31
73.55
64.82
05.55
01.92
91.65
83.95
94
87.81
86.35
79.23
75.44
53.50
73.09
Featm 1n 2000
92.95
14.81
20:20
27.50
83.30 34
10. 75
18.53
19.00
82.00
0.36
5.59
13.59
(FelMar MoMMeeeneeen morooaa 2055
59.02
01
50.01
85.16
05.00
08 64
08.00
05:63
43.03
43.59
53.86
OcliedPas
SUNNW
L7TD
CLIF Ballonlln oo 2031
72.99
53.27
Prombce PoPch
91.70
02.20
07.04
03.62
72.85
72.52
73.56
73.05
56.71
$7.20
44.95
$2.31
Follke Airicai 18 2015
$6.86
88.71
$7.09
07.84
74.54
74.36
$7.66
78.35
60.76
57.90
55.85
55.10
09.10
81
34.1
7.69
6.75
30119
(FEIMMA 3
54.59
..999
1..11
2.35%
Cuilech10
Food1001
10L
Metied
Novel
HM
Locel
Novel
Locil
Novel
HM
CLIF Rutionntttll sta.
2001
92.63
94.00
04.00
03.54
$4.33
$4.22
65.43
$4.63
65.16
65.36
71,28
67.15
55.11
73.17
71.78
Farrm
m 1
97.88
40:07
76.44
81.14
65
23.84
31.94
7.86
9.14
13.05
94:14
77.99
77.24
81:35
91
16.99
N8 mM
09.05
01.53
15.69
40.74
41.17
(FaMMMA tousy
94.77
94.18
93.33
94.26
9L.77
$4.75
85.84
80.33
79.24
67.30
71.29
72.29
1.11%
21

---

Published as a conference paper at ICLR 2026
D.2 MODEL EVALUATION ON FEATURE & LABEL SHIFTS
Table 11: Average test accuracy (%) of different methods on DomainNet and Caricc- with lable shift and domain shift using Dirichlet partitioning.
Dataset
Office
DomainNet
#3
0.1
0.3
0.5
10
0.1
0.3
0.5
10
One domain for one client
CLIP Radford 2021
8.24
7.78
9.60
8.98
8.98
10.27
10.15
10.37
10.52
PromptFL Cro et 2023b
14.53
15.39
13.61
14.32
13.57
14.36
12.52
11.77
11.81
12.21
11.66
11.77
FedPGP eu 2012
14.34
PFedMoAP L el 2025
12.65
16.14
12.27
14.19
14.70
17.03
14.14
13.89
14.30
14.14
14.38
13.55
PFedMMA (Oars)
21.08
22.38
19.06
20.43
18.42
18.73
36.18
37.06
42.55
43.31
46.13
34.69
One domain for clients
CLIP Radiord et al. 2
8.83
9.10
9.11
9.67
6.61
12.51
10.59
10.29
10.11
9.81
9.24
10.00
PrompiFI Gioo et 20223b
15.99
15.29
11.24
11.27
FedPGP Cur al 2024
22.55
19.29
20.71
21.96
19.63
15.19
26.08
26.30
24.90
21.22
16.14 14
15.07
PFedMoAP Luo ol al. 2025
22.73
23.06
19.55
21.67
16.57
19.02
24.99
24.79
24.65
21.59
16.43
15.24
pFedMMA (Ours)
21.66
22.07
21.33
18.47
20.96
17.73
49.45
37.61
47.17
48.95
46.90
48.54
Table 12: Test accuracy (%) of different methods on DomainNet and Office-Caltech10 with lable shift and domain shift using Dirichlet partitioning.
Method
Cllle-Caltech10
DomainNet
DSLR Webcam
Avg.
Clipart
Infogruph
Painting
Quickdraw
Avg.
0.5
domain
elient
CLIP
10.42
5.33
12.50
10.17
9.60
8.37
10.50
12.12
10.40
10:47
10.11
PromptlL
8.85
23.73
11.11
18.75
15.61
12.16
12.33
11.79
9.40
13.25
11.91
11.81
FedPGP
11.98
9.78
28.13
6.78
14.17
14.07
19.18
14.05
10.30
13.39
13.90
14.15
pledMoAP
5,21
9.33
12.50
22.03
12.27
13.69
19.18
15.35
11.50
12.90
13.18
14.30
PFadMMA (Ours)
10.71
17.81
17.46
30:26
19.06
50.38
48.82
0.00
32.56
37.31
86.21
42.55
One
clients
CLIP
11.78
6.21
9.92
8.51
9.11
8.99
10.69
11.20
10.85
9.39
10.11
PromptFI.
10.35
15,83
32.06
7.13
16.34
11:02
1.65
11,20
8.95
13.89
20.75
11.24
FedPGP
PFedMAAP
20.01
24.45
18.02
15.73
19.55
24.77
30.93
26.09
20.46
22.59
23.10
24.65
PFAIMMA (Ours)
9.26
29.15
33,26
13.64
21.33
50.38
23.81
60.27
61.44
40.35
46.79
47.17
One domain for one client
CLIP
8.33
12.89
3.13
6.78
7.78
10.08
9.44
10.82
10.30
10.68
9.57
10.15
PromptEL
11,86
10.94
25.00
13.78
15.39
10.27
11.47
10.70
11.91
14.31
11.60
FedPGP
6.77
11.11
34.37
15.25
16.88
13.50
19.33
14,22
10.10
13.39
14.08
14.10
14.86
PFedMMA (Ours)
14.29
15.07
28.57
31.58
22.38
49.12
53.21
58.44
12.38
19.83
29.37
37.06
One domain for two clients
9.92
8,51
PromptFL
10.50
15.36
25.95
3.45
15.29
11.55
12,80
13:40
8.67
20.16
5.26
12.58
FodPGP
21.02
21.38
12.70
22.07
19.29
25.6
33.52
26.51
23.62
23.62
24.89
26.30
pFedMoAP
24.01
PFedMMA (Ours)
23.73
25.88
23.67
15.00
22.07
26.61
44.42
16.94
46.09
$4.11
37.46
37.61
0.1
ome elient
CLIP
10.94
8.44
0.00
13.56
8.24
11.22
9.28
10.18
11.00
9.12
10.83
10.27
PromspdlL.
10.42
18.75
12.00
16.95
14,53
15.53
14.70
11.22
9.00
10.60
14.08
12.52
FedPGP
14.18
14,98
12.00
pledMoAP
10.42
12.44
12,50
15.25
12.65
15.21
19.33
14,22
10.90
12.57
12.64
14.14
PFedMMA (Ours)
8.93
12.33
30.16
32.89
44.19
0.00
86.21
36.
One domain for two clients
CLIP
4.13
7.57
10.83
12.79
8,83
9.46
10.80
10.48
11.30
11.51
9.99
10.59
PromptFI.
7.13
23.75
21.34
15.99
0.50
TedPGP
18.83
21.37
25.83
24.19
22.55
24.69
32.29
28.89
19.70
24.83
26.09
26.08
PRedMoAP
19.41
20.96
26.67
23.90
22.73
25.60 15.60
29.66
24.85
19:70
22.84
27.32
24.99
PFadMMA
22

---

Published as a conference paper at ICLR 2026
D.3 LEARNING CURVES
To further examine the convergence behavior of pFedMMA, we plot the local accuracy over communi- cation rounds across six representative datasets with five different shot settings Fig. 3 All methods are evaluated under the same federated setup with 2 local epochs and 50 communication rounds. As shown, PFedMMA consistently achieves high accuracy and exhibits stable, fast convergence across datasets. Notably, even while delivering superior generalization both base and novel classes (Table 9). PFedMMA converges faster local performance than pFedMoAP throughout training. These results demonstrate that PFedMMA effectively balances personalization with generalization, ensuring both rapid and reliable convergence.
U550
1ourd
Figure 5: Accuracy learning curves of pFedMMA and baselines over 10 clients.
23

---

Published as a conference paper at ICLR 2026
D.4 ABLATION STUDY
Table 13: Ablation study on the dimension of the shared adapter for a = 0.001.
Calechlo
ICP
DefoniPets
Slors
Dinooskons
Local
Nin
1IM
Lacal
Baar
New
1IM
Locé
Blar
New
IIM
Laczl
Bluor
New
10M
Lacal
Nex
92:97
55.41
83.81
70:90
S4.13
13.93
16
02:00 mg
56.00 67
61
97
60.34
99.33
95.19
91.66
$1.11
63.00
66.00
03
05.5 55
80477
01.81
89.63
8533 17
13:99
73. 1.
738.7
61.1
98.0
56.00
91.31
07.71
81:39
63:11
69 70
$1.51
07:00
74.99
18
123
18.90
98.31
$5.00
91.98
05:31
35.67
65.35
73.00
71.31
18.19
91.83
99.38
191.00
80.67
00.73
52.51
60:53
09.43
53.59
91.06
94-73
86.11
62.40
66.67
70.35
03.4
87.15
90.33 74
00.30
87.22
74.88
76.50
10.00
70.79
09.70
94.55
91.05
05.07
67.09
70.45
54.59
85.41
90.67
73.75
78.54
79.75
68
05
53 25
01.00
61.42
55.55
92.17
55.75
$7.75
65.18
10.63
73.35
98.05
17.67
02.07
88.57
00.23
73.59
79.13
79.77
134
56
18
61.17
99.51
02.91
92.71
94.96
55.53
64.01
07.83
72.10
01.83
$2.01
02.42
90.51
09.96
73.32
75
10.00
73.04
16
68.67
91.57
90.12
61.54
95.35
89.65
16
55.00
T0
$7.51
30.90
95.57
90.55
94.63
90.99
60.86
65.39
70.33
95.81
85.57
04.05
02.25
90.29
74.03
75.00
73.74
32
75.33
36
61 an
62:67
0964
53.50
50.31
94.65
92.69
66.70
71.86
75.57
95.99
84.15
04.53
91 11.54
91.29
74.78
70.53
81.0
61.11
76.00
56.55 53
$4.99
61.28
0961
56.25
91.54
85.39
19.11
57.78
63.31
61.80
95.74
$1.24
03345
89.75
90.20
72.85
75.73
08.58
90.25
99.59
01.02
92.5%
91.56
94.25
78.99
64
65.52
55.00
57.35
68.12
99.91
95.29
92.11
95.33
91.44
$4.30
66.34
63.92
99.25
51.89
02.00
99.21
94.11
70.25
77.42
73.92
129
50.02
$4.99
5K.77
64.17
100.00
92.11
91.34
94.33
91.44
59.33
68,00
70.93
99.53
$1.99
94.96
91.47
94.40
72.22
78.22
60.30
16
63.06
06
M0.59
10000
91.23
98.39
54.99
65.14
06.85
98:75
51.88
91.88
9255
91.30
65:42
16.30
12.08
15
32
85.19
67.8
56.47
59.5
99.55
16.1
58.6
91.73
91.21
57.53
65.35
69.25
99:35
76.31
01.29
85.06
04.35
6710
16.27
72.31
01.79
10.35
91.45
68,03
99:35
03.66
01.35
11.39
78.91
Table 14: Ablation study on the dimension of the shared adapter for a 0.005.
Calechi01
OrfocdPets
Avesuee
Local
Basz
Nets
1154
Local
Hasz
New
115
Laxzl
Hasc
Nict
1154
Local
Bast
New
1BM
Lacal
New
FIM
61:31
13
01:47
Wi-1
79.41
61.00
71.02
93.37
75.00
89:78
$1.85
59:34
96.94
95.63
78.35
125
68.59
56.31
60.39
$1.46
96.90
96:35
94.10
96.61
95.01
71.20
75.76
76.91
93.76
90.00
97.22
95.57
96.96
79.61
$1.87
62.14
60.87
34.00
93655
76.63
64
60.70
60.00
97.08
91.76
104
34.1%
17.51
90.14
06.35
78.54
$1.42
82.14
176
76 01
56.17
60.13
65.78
99.16
07.03
91.64
56.18
143 93
70.85
11.19
11.11
94.90
00.76
96.53
91.85
14.26
76.13
$1.35
$2.68
$9.74
57.20
12.
15.00
79.75
05.51
92007
79.14
63.55
54
76 16
57.05
61.50
63.90
$9.81
57.08
93.54
16.58
11 31
71.59
71.57
78.35
08.33
90.51
56.77
95.06
81.38
75:17
$1.77
53.57
128
61 25
$1.53
61.34
55.15
90.48
91.15
93.70
96.30
92.05
71.31
34.39
78.46
96.99
90.15
96.39
96.46
92.44
79022
$1.63
$3.72
9:
61.76
55,30
00:35
67.06
01.32
96.82
92 21
70.30
34.35
18.25
96-42
03.15
06.19
95 12
9sm
76.31
$1.7K
K3.99
64
81
$6.87
60.09
M487
59.50
07.14
01.80
96.85
91 01
71.04
74.19
18.06
08.84
53:4
06.78
95.05
92.31
75.24
$1.59
K330
128
55.63
57.01
60:11
65.54
99.83
97.04
93.79
96.32
02.66
70.35
74.92
78.25
96.99
90.20
96.67
94.51
93.53
75.66
$1.57
53.7%
94.09
$4.18
16
64
96.44
58.25
56.75
61.31
60.56
$4.95
55.94
99.84
90.95
95.37
97.25
93.85
93.60
96.91
96.74
03 28
92.00
69.91
70.15
74.74
74.45
18.06
78.11
09.10
99.63
93005
93.35
96.34
96.57
05
95
0409
94.71
71.56
75.34
$1.32
$1.72
$4.09
53.13
61.14
96.93
93.95
99.48
90.94
95.56
78.80
$1.59
Table 15: Ablation study on adapter sharing strategies with scaling factor a 0.001, adapter dimension=32, and starting layer = 5).
Local Performance
Shots
Method
DTD
Caltech101
1ow
OxfordPets
No Local Param
57.41
96.04
72.27
91.75
shot
Local Shared Adapter
58.15
96.49
74.27
92.13
PFedMMA
64.81
98.76
80.46
95.76
No Local Param
60.74
96.67
74.22
90.86
shots
Local Shared Adapter
60.74
98.33
78.24
91.31
PFedMMA
71.25
99.41
86.74
94.60
No Local Param
60.97
96.63
73.78
91.39
4 shots
Local Shared Adapter
61.72
97.85
77.30
92,82
PFcdMMA
75.97
99.66
86.51
97.10
No Local Param
62.13
96.97
74.90
92.32
shots
Local Shared Adapter
62.73
98.12
76.20
94.53
PPedMMA
80.97
99.60
86.66
99.64
No Local Param
64.21
97.05
73.92
93.22
16 shots
Local Shared Adapter
71.11
99.89
79.17
92.95
PFedMMA
88.89
99.95
91.32
99.85
24

---

Published as a conference paper at ICLR 2026
Table 16: Ablation study on scaling factor a.
DTD
Caledaloi
UCF
OxforiPes
Necrnge on 4-Untoes
Facix
D0001
06.06
6.76
7.10
5.85
98.53
3:51
1:33
6:05
65.08
3.12
3,53
4.13
07.19
0.73
3.36
12.30
56.11
3.78
6.33
1:46
D/001
66.23
98.04
56/02
76
62.11
64.53
90.52
14.99
79.00
70.77
D605
11.00
56.03
00.95
62.07
98.04
97.60
95.17
96.34
85.35
68:58
75.00
75.71
05.06
59:78
96.87
93.99
57.01
78.30
81.05
82.00
D61
05.51
55.56
59.64
29.96
97.98
$6.97
94.06
96.31
83.55
69.10
74.84
75.37
95.66
19:65
96.90
93.97
55.66
17.52
81.37
$1.40
D0001
76.30
5:82
99.54
5.05
5:37
2,00
4.18
05.00
3.41
14.62
5.34
5.00
D6000
11.10
14.05
19.01
63.11
09:1
$1.45
M6.7
80.05
90.00
16.11
14.00
9508
26:33
76 T6
87.13
88.1 11
66.34
61:33 76
14
BC01
79
47.58
61 37
$1.35
67.09
7
70.89
61.19
K5-11
W100
19.06
15.13
T5.86
74.77
15.55
94
94.00
59.50
1943
$7.70
D0001
51:57
5.50
6.84
59.00
3.30
3.30
4:53
02:61
2.10
3.11
3.12
92.1
8.58
10.15
13.00
02.15
4.76
5.40
1.31
15.83
54.36
61.40
62.87
99.64
93.39
V8L11
06.46
12.59
56.70
71.86
15.57
16.50
14.15
96.52
91.54
91.29
14.75
501.00
51.66
10006
72.59
57.90
61.81
64.72
99.24
91:20
15.85
00
02:15
71.31
75.00
78.75
15.31
5000
16.85
91.96
91.42
10.24
81.85
K3.58
$0.09
56.53
04.51
64.51
99.72
91.09
95.95
96.97
$9.99
69.51
14.15
77.14
97.45
50.25
96.92
94.15
91.95
18.33
51.62
13.32
DI0001
19.05
5.19
7.55
11:25
99.51
4.15
3.21
4.36
02.47
2.89
2.94
4.15
92.3
5.50
6:43
9.65
00.16
5.14
4.79
3.08
DI000
16.95
13.30
02.86
46.37
99.5
61:31
16.33 10
19:15
92.00
18:55
67.35
52.19
16.00
11:32
75.00
05.62
96.55
67.39
61.71
61.55
NCD1
11.35
54.35
36.00
62:15
09.02
02.18
01.08
02:11
61.53
58.35
72.18
12:11
05.00
87:56
05.16
10.47
77.82
73.97
D61
83.55
56.31
00.65
03.57
99.30
97.17
93.96
02
96.09
69.47
14.55
74.50
05.88
$9.70
1x. 59
94.11
94.41
18.15
$1.54
83.73
00301
5.09
2.3
91:33
2.36
1.1
1.91
05.15
4.30
18
85.19
41:34
56.47
59.58
99.55
15.51
16.33
01.75
69.22
$7.67
68.55
09:55
99.5
76.31
65.33
04.55
67.08
76.27
71.51
$6.44
56.75
61.32
65.94
09:54
97:20
15.18
92.55
6999
74.14
1953
50335
03.88
16.11
13.55
81-72
8409
$0.40
60.57
99.89
97.05
05
68.93
79.31
07.30
19.58
11.99
Table 17: Ablation study on scaling factor starting layer l with scaling factor a 0.005 and adapter dimension=32.
DTD
Cilech101
UCF
OxfoedPet
Awerage
Stors
Lapess
Local
Novel
Buse
Novel
HM
Bas
Novel
Xovel
HM
Baic
90.79
54.35
69.00
10.10
65.04
96.34
91
96.12
96.80
91.B4
69:03
15.31
11.42
99.60
89.54
06.24
95.00 10
27.49
$1.58
$3.62
10
12
$6.87
56.31
61.00
65:17
90:35
94.36
98.95
91.13
69.9
13.47
71.84
95.33
89.64
91.02
94.99
03.97
75.23
52.04
$3.87
18.58
61.06
61:33
99.34
98-15
10.02
14.45
06.94
75.27
56.03
68.95
56.34
70,20
74.64
95.64
99.74
$2.15
12
44
53.40
59:35
64.16
95.63
97:05
55.84
56.78
54.37
68.34
75.36
17.59
59:55
58.74
94.51
05.94
75,89
81.23
$3.41
61.92
57.12
$6.90
55.3.
70.65
14.88
55:80
16.57
75.69
$1.90
$4.14
18.25
04.34
51.11
63300
96.21
97.08
03.54
96.92
69.94
14.23
95.02
59.95
06.
05.75
90.35
75.34
$1.54
$2.58
5- 12
13.82
10
61.71
62:8
90.05
97.24
02.62
98.59
60
70.73
14.58
71.51
95.69
90:35
06.97
04.55
89.75
75.61
$1.99
$2.91
10 12
05.75
56-46
96.55
06.91
91:40
$7.00
70.71
75.12
10.38
40400
00.65
96.11
65.31
92.54
76.35
10.105
$4.12
4-12 13
02.82
67
61 m7
61.11
(0010)
97105
97.17
$1.00
05.95
70.00
74.96
70.14
08.76
89.5
06.12
97.14
74.56
31
$7.67
12
18.29
57.43
64.77
99.85
05.15
53.11
56.58
92.20
71.09
74.56
75.25
99.70
35.03
07.00
05.38
02.51
75.90
81.86
53.63
57.90
61.51
93.94
96.56
71.51
75.10
06.77
02.13
79.23
13.55
12
94.40
$3.30
59.58
65.18
99.75
95.47
94.17
96.14
04.75
69.25
13.11
71.55
100.00
81.15
05.33
94.40
07.23
76.55
$1.27
63.55
10
12 12
15.32
$6.30
61.57
67.22
09.25
06.99
04.30
98.99
16.09
09.34
73.33
78,59
99:55
$9.35
06.64
15.10 10
07.30
71.15
81.56
$4.52
12
96.95
70.51
99.45
95.27
97.09
75.45
$4.1
74.88
04.12
95.16
79.24
81.85
8367
12
94.91
53.91
60.11
65.77
95.90
12
56.59
95.45
06:23
75.05
75.01
100
58.14
96 25
94.56
97.56
75.55
53.75
56
12
56.30
56.0
61
1081
96.14
11.11
10000
95.14
76.18
6-
58.00
99
61.30
66.12
00.50
10.00
96.00
92.00
69.42
14.5%
17.77
9585
89.59
06.96
15,33
15.34
78.33
$1.74
$4.14
12
$0.40
25
60.7
6800
09.89
97.05
V4-01
96.93
92.93
09.92
74.14
75.04
99.63
90.35
06.94
15.46
03.20
71.41
81.77
$4.11
25

---

Published as a conference paper at ICLR 2026
Table 18: Comparison of FL aggregation variants (vision-only, text-only, and both-sides) for the shared adapter.
Awerye in drzwes
SUNIW?
Faxt
IIM
Nand
Local
18M
Novel
Lecal
Nosel
Viases Ouly
Torth Vilier Taxt
80.90 66.14
71 0m
7o
Th.31
81 Rg
70:45
78.11
70
96:13
69
76.35
78:77
M7.90
35.39
61 18
07 18
98 96-41
66.17
90:83
97:63
(FaIMMAA (Ckm))
71
76.17
78.76
9106
20:95
79:07
$5.38
71.51
76400
79:79
35.41
07.30
97.45
90-77
97:31
Table 19: Top-1 accuracy (%) of different methods across 7 datasets the 16-shot setting using AdamW optimizer.
Average dorneets
SUNMY
Flowersto
DX113
Method
Local
Base
Novel
HM
Local
Base
Novel
HMM
Locoll
Base
Novel
HM
Local
Blase
Novel
HB
Prompoy
(FaIMMA
OxfoniPax
Calisch01
Foad110
UCFIDI
Medhod
Local
Base
Novel
HM
Local
Buse
Novel
HBM
Locol
Base
Novel
HM
Local
Busc
Novel
HBM
Rationl
Frumprr Sero
73.00
2005
(FalMMA
26

---

Published as a conference paper at ICLR 2026
E
TRAINING COST ANALYSIS
E.1 COMPARISON
Below, we briefly describe each method and provide parametric expressions for the number of trainable parameters and per-round communication, together with instantiations for ViT-B/16; see Table 20 for notation.
PromptFI.. Each client fine-tunes only a continuous text prompt (the backbone is frozen), and the server aggregates the prompt via FedAvg and broadcasts the updated prompt. (1) Per-client counts: The number of trainable parameters is Ld, In each round, the client uploads Lde parameters and downloads Ld, parameters. (2) ViT-B/16 example. With de 512 and L = 16, one prompt has Ld, = 16 x 512 = 8,192 parameters, so each round the client uploads and downloads 8,192 parameters.
FedOTP. Each client learns a global prompt (to be aggregated) and a local prompt (kept private); training couples them via optimal transport, and only the global prompt is communicated. (1) Per- client counts: The number of trainable parameters is 2L,, In each round, the client uploads Ld, parameters and downloads Ld, parameters. (2) ViT-B/16 example. With de 512 and L 16, the client trains 2 x 16 x 512 = 16,384 parameters in total, and in each round uploads and downloads 8,192 parameters.
FedPGP. Clients share global prompt and add a low-rank personalized adapter U.V, v, locally; only the global prompt is aggregated. (1) Per-client counts: The number of trainable parameters is Ld, + 6 (d, + L). In each round, the client uploads Ld, parameters and downloads Ld, parameters. (2) ViT-B/16 example. With de 512, L 16, and 8, the low-rank component contributes 8(512 + 16) = 4,224 parameters, giving 8, 8,192 + 4,224 = 12,416 trainable parameters overall; in each round the client uploads and downloads 8,192 parameters.
PFedMoAP. Each client learns a local prompt and downloads K non-local prompt experts (without aggregation). local multi-hean attention gating network mixes local and non-local experts; the gating network is trained on-device and not communicated. Features are pooled to width d, before the MHA. (1) Per-client counts: The number of trainable parameters is Ld, (4d2 + 4dg). ). In each round, the client uploads Ld, parameters and downloads K Ld, parameters. (2) ViT-B/16 example.
With d, 512, L 16, dg 128, and K = 9. the gating network has 4 128 4. 128 66,048 parameters, so the client trains 8, 192 + 66,048 = 74,240 parameters; each round the client uploads 8,192 parameters and downloads 73, 73,728 parameters.
PFedMMA. Lightweight multimodal adapters are inserted in both vision and text blocks; in each instrumented layer the adapter comprises down-projection (d + r), shared (aggregated globally), and an up-projection (r-+d) d). The shared projection is communicated each round, while the up/down projections trainable parameters per layer is (r(d, + de) + r2, so across m layers it is m [2r(d, + d,) + +2]. In each round, the client uploads m parameters and downloads my parameters. (2) ViT-B/16 example. With d, = 768, d, 512, r = 32, and layers 10-12 (m = 3), the per-layer trainable count 2- 32 (768 + 512) + 32 82,944, for a total of 3 x 82,944 248,832 trainable parameters: in each round the client uploads and downloads 3 x 322 3,072 parameters.
projection
updated locally. (1) Per-client counts: The number of
Table 20: Notation used in this part. Examples assume CLIP ViT-B/16.
Symbol
Description
Example (VIT-B/16)
dr
CLIP text-encoder width
512
d,
CLIP vision hidden size
768
Number prompt tokens
e.g.
b
Low-rank bottleneck (FedPGP)
e.g., 8
dg
Internal width of the PFedMoAP gating MHA
e.g., 128
K
Number of non-local prompt experts downloaded per round (PFedMoAP)
e.g. 9
Adapter inner (shared) width (pFedMMA)
e.g., 32
Number of instrumented transformer layers (pFedMMA)
c.g., 3
27

---

Published as a conference paper at ICLR 2026
Table 21 summarizes the computational and communication costs together with accuracy. PromptFL has the smallest footprint (8,192 trainable; 8, 8.192/8, 192 per round) but-most most importantly shows marked drop in local accuracy (88.93%). indicating weaker personalization under heterogeneity. FedPGP increases local trainables to 12,416 without extra communication but incurs the highest memory (13,374 MiB) and slower training, and its HM accuracy (79.09%) lags its strong local accuracy (95.38%). FedOTP doubles prompt capacity (16,384 trainables; same 8,192/8, 192/8,992228 com- munication) and attains very high local accuracy (97.34%) but suffers extremely low HM accuracy (31.08%), suggesting poor cross-client generalization. pFedMoAP adds local gating module, raising local trainables (74,240) and the per-round download (73,728) while achieving the shortest training time (902 s) and strong local accuracy (97.89%). pFedMMA (ours) communicates only the shared adapter blocks (3,072/3,072) while keeping 248,832 parameters local, yielding the best HM accuracy (84.15%) and competitive local accuracy (97.17%), thus offering the most favorable trade-off.
Table 21: Comparison of computation, communication, and accuracy for five personalized federated learning methods under CLIP ViT-B/16. Columns report the number of local trainable parameters, per-round communicated parameters (upload/down), end-to-end training time, peak GPU memory, average local accuracy, and average harmonic-mean (HM) accuracy.
Methods
Local Trainae Param.
if Per-cound Com.. Parum. (spldown)
Tran Time
CPU Mem. (Mill
Arg. Local Acc.
.Vg. HM Avc.
ProneptFL Kopo et
20238
5,192
9,192 / 8,192
1,645
5.116
88,93
$3,09
RalPGi Ca
12.416
8,192 / 8,192
3,900
13,374
95.38
79.09
belt
16,154
N.192
1.318
3004
3.106
E.2 ON THE NECESSITY OF COMMUNICATING SHARED ADAPTER
We ablate adapter sharing to test whether exchanging small parameter set is sufficient for federated coordination (Table 22). In PFedMMA, only the low-rank shared r x adapter globally synchro- nized each round, while the up/down projections remain local. This design communicates just mr parameters per round, yet it is exactly these parameters that carry the essential cross-client signal: they define a common low-dime wwwdimmnniioal subspace that aligns clients' representations, while the much larger local adapters capture client- dieentsppciffic variation. The comparison with the Local Only Param variant (which updates all adapter parameters purely on-device without FL. demonstrates that global synchronization of the shared subspace is crucial, even when the communicated set small.
Table 22: Ablation study on adapter sharing strategies with scaling factor dimension=32, and starting layer 10).
0.005, adapter
Method
DTD
Caltech101
Flowers
OxfordPets
Food
UCF
Base Performance
Local Only Param
31.37
80.83
19.69
$7.68
85.04
62.11
PFedMMA
55.44
96.53
71,54
88.50
89.15
69.61
Performance
Local Only Param
$4.94
91.72
68. 30
85.76
66.13
45.47
PFaMMMA
61.55
94,29
76,00
96.60
90.77
74.88
28


Reviews:
[{"review_id": "NILAgvFwdI", "reviewer": "Reviewer_VEYb", "summary": "The paper designs an algorithm (pFedMMA) to personalize Vision-Language Models in a federated learning setup to adapt to client-side data heterogeneity. The design is based on client-specific multi-modal parallel adapters with each adapter's cross-modal shared projection layer aggregated for generalization under federated learning. The paper presents extensive empirical evidence (under data heterogeneity, label-shift, and feature-shift) that pFedMMA improves the generalization-personalization trade-off compared to personalized federated prompt tuning methods. The adapters are restricted to higher layers of the image and text encoders of the VLM to keep communication cost of aggregating the shared projection layers in check while capturing a large proportion of the full potential of the design w.r.t. generalization and personalization.", "questions": "(Q1) Lines 18-20 - \"In this work, we propose pFedMMA, the first personalized federated learning framework that leverages multi-modal adapters for vision-language tasks.\" I feel that this is a bit too strong of a claim, since several elements of the design already appear in the related works - a) [FedDAT] incorporates adapters in FL, and b) FedPGP deals with generalization-personalization trade-offs. Is it more apt to state this work as an advancement (with explicit clarifications on what parts are advancements)? Same comment for Lines 86-88.\n\n(Q2) Would pFedMMA translate well to state-of-the-art privacy and security aware enhancements to the aggregation method? Some explicit commentary on this aspect is warranted in the paper's main body (or in the appendix and referenced in the main body).\n\n(Q3) Lines 264-265: Should this aggregation formula be changed when number of samples varies across clients? How are the empirical results generated for large differences in number of samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "(S1) The paper is very well-organized and well-written. It is delightfully easy to read. Intuitions from related work are provided in several places for proper contextualization. The problem, algorithm design, and experimental setups are all well motivated. Results and intuition are well communicated.\n\n(S2) Experiments and metrics presented provide good quality empirical evidence to support the claims in the paper. A diversity of datasets and experimental conditions covering data heterogeneity, label-shift, and feature-shift have been considered. The results communicate that a demonstrated improvement is achieved by pFedMMA in the generalization-personalization trade-off in federated learning of VLMs for several datasets.", "weaknesses": "(W1) While related work is mostly well cited, I believe that one relevant paper [FedDAT] is missing. Contributions of this manuscript should be contextualized and differentiated w.r.t. this reference. [FedDAT] Chen, H., Zhang, Y., Krompass, D., Gu, J., & Tresp, V. (2024). FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10), 11285-11293.\n\n(W2) Lines 461-462: Based on Fig. 4, one can only comment about evolution of personalization accuracy, not generalization. To support the analogous generalization claim, a plot of the base-to-novel generalization vs communication round is needed.\n\n(W3) Line 357: It would be good to have results with one more optimizer (Muon or Adam) besides SGD to ensure that the conclusions hold true on change of optimizer. Results can be in the appendix, but referenced & commented on in the main body.\n\nThings to improve the paper that did not impact the score:\n- Lines 363-365: It would be good to have a cited reference for the use of \"base-to-novel generalization\" metric.\n- Section 4.3: Among all the studies presented, only the \"Adapting Variant Options for PFL\" is an ablation study. The others are hyperparameter choice experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sox5rrPEe2", "reviewer": "Reviewer_Tmrt", "summary": "This paper proposes pFedMMA, a personalized federated learning framework for Vision-Language Models (VLMs), which for the first time incorporates multi-modal adapters into federated fine-tuning. The proposed architecture includes modality-specific up- and down-projection layers along with a globally shared projection layer. During training, all components are updated locally, while only the shared projection is aggregated globally. Extensive experiments across multiple datasets demonstrate that pFedMMA achieves a SOTA trade-off between personalization and generalization.", "questions": "1. The paper claims that the shared projection layer improves generalization to unseen classes, but it does not explain why this structure effectively generalizes to semantic categories completely absent during training.\n\n2. The motivation for using harmonic mean (HM) as the main evaluation metric, rather than arithmetic mean, is not sufficiently justified. Moreover, no references are provided to support the use of HM for evaluating the balance between personalization and generalization.\n\n3. The main contribution appears to be a direct adaptation of the centralized MMA to the federated setting, with the added strategy of aggregating only the shared projection layer globally. This raises concerns about limited novelty.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. pFedMMA effectively introduces multi-modal adapters into personalized federated learning, balancing personalization and generalization. It addresses the poor generalization of existing prompt-tuning methods on unseen classes.\n\n2. The asymmetric training mechanism, which aggregates only the shared projection layer, reduces communication costs while retaining modality-specific up- and down-projections locally to adapt to local data distributions.\n\n3. Through extensive evaluation across diverse data heterogeneity scenarios, pFedMMA is shown to surpass prior prompt-based PFL techniques in generalization across both domains and categories, without compromising its personalization strength.", "weaknesses": "1. Although communication cost is reduced, the total number of trainable parameters introduced by pFedMMA is significantly larger than mainstream prompt-tuning methods, increasing local computational and memory burdens, which may not be friendly to resource-constrained devices.\n\n2. Despite achieving the best harmonic mean (HM) performance, pFedMMA shows noticeably lower local accuracy than pFedMoAP on several datasets (e.g., Flowers102 and DTD), indicating that its personalization capability is sacrificed in certain scenarios. The overall performance is sensitive to dataset distributions and lacks stability.\n\n3. In the domain generalization experiments on DomainNet and Office-Caltech10, the experiments do not include federated baselines explicitly developed for domain or feature shift scenarios, which weakens the credibility of their claims regarding domain generalization capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uWk04pGT1z", "reviewer": "Reviewer_quYy", "summary": "The paper proposes pFedMMA, a novel personalized federated learning (PFL) framework for fine-tuning vision-language models (VLMs) like CLIP in decentralized, heterogeneous settings. It introduces multi-modal adapters with modality-specific down- and up-projection layers and a shared projection layer, inserted into upper transformer blocks of both image and text encoders. Clients locally update all adapter components to adapt to their data distributions, while only the shared projection is globally aggregated via FedAvg, balancing personalization and generalization with communication efficiency.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Comprehensive Evaluation: The study spans diverse heterogeneity scenarios (label shifts via non-overlapping classes, feature shifts via multi-domain datasets like DomainNet), using Dirichlet partitioning for realistic non-IID data. Testing extensive datasets, two backbones (ViT-B/16, ViT-B/32), and few-shot regimes, provides robust evidence of applicability and interpretability.\n\nEfficiency and Scalability: As a parameter-efficient fine-tuning (PEFT) method, it freezes the VLM backbone, training only lightweight adapters. The focus on shared projection aggregation reduces communication costs.", "weaknesses": "Unverified Cross-Modal Alignment: The core claim of achieving cross-modal consistency via the shared projection lacks rigorous validation. The parallel adapter design with a shared layer assumes modality interaction without explicit mechanisms (e.g., attention or fusion gates), and no quantitative evidence (e.g., cosine similarity, t-SNE visualizations) confirms reduced modality gaps or alignment under federated heterogeneity. \n\nInsufficient Motivation and Problem Framing: The motivation relies on prompt-based PFL methods, sacrificing generalization for personalization, but lacks deep analysis on why adapters inherently outperform prompts or why a hybrid prompt-adapter approach is not explored. Baseline selection is biased toward prompt methods, omitting adapter-based methods such as FedCLIP (Lu et al., 2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTlOEvyjcX", "reviewer": "Reviewer_742x", "summary": "This paper proposes pFedMMA for personalized federated learning (PFL) on vision–language models (VLMs). The core idea is to adopt a Multi-Modal Adapter structure, aggregating only the shared projection on the server while keeping the up/down projections as local personalized parameters. This aims to balance personalization vs. generalization under limited communication cost.", "questions": "1. Why does pFedMMA achieve only 9.26% on the Amazon domain, significantly below FedPGP’s 20.34%? Please provide an explanation/diagnosis.\n2. Could you add comparisons of vision-only / text-only / both-sides shared projection to localize the main information-sharing channel and potential side effects?\n3. Please include the HM formula and a brief rationale for choosing it in the main text.\n4. Fairer baselines beyond prompts: Have you considered other VLM fine-tuning approaches such as CLIP-Adapter + FL (and more general PEFT baselines), given that current baselines are mostly prompt-based?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The method is intuitive and easy to implement.\n* It has an advantage in terms of communication efficiency.", "weaknesses": "1. While the approach is effective for personalized federated VLMs, I have concerns about novelty: the work largely looks like replacing the backbone with a Multi-Modal Adapter in a standard FL pipeline. The substantive difference from prior PEFT/Adapter + FL lines is not sufficiently quantified.\n2. The paper emphasizes applicability in out-of-distribution (OOD) scenarios, which is closely related to Federated Domain Generalization (FedDG). However, related work is not discussed in depth and experiments do not compare against FedDG-style algorithms also targeting federated VLMs (e.g., PLAN [1]).\n\n```\nReference\n[1] Shuai Gong, Chaoran Cui, Chunyun Zhang, Wenna Wang, Xiushan Nie, and Lei Zhu. Federated domain generalization via prompt learning and aggregation. arXiv:2411.10063, 2024.\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)