Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Authors: Yihao Yang
Wenke Huang
Guancheng Wan
Bin Yang
Mang Ye
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Parameter-Efficient Fine-Tuning
Vision-Language Model
Data Heterogeneity
Primary area: social_aspects->privacy
TL;DR: In this paper, we propose FedDDA, a decoupling-based method for Personalized Federated Parameter-Efficient Fine-Tuning.

Abstract: Federated Parameter-Efficient Fine-Tuning aims to adapt Vision-Language Models for downstream tasks in distributed environments. However, data heterogeneity across participants hinders collaborative effectiveness, necessitating personalized adaptation to cover distinct data distributions. Current personalized methods suffer from two limitations. 1) Textual Property Loss: Existing methods facilitate the collaboration between decoupled prompts at the feature level, which potentially undermines the textual properties of the prompts. 2) Visual Feature Diversity: The diversity of visual features makes it challenging to leverage naive image features directly for image-text alignment in downstream tasks. In this work, we propose Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation (FedDDA) to overcome the above limitations. Specifically, we encourage decoupling prompts in a way that maximizes the efficacy of prior knowledge, which is essential for maintaining a coherent linguistic context. Furthermore, we design a visual adaption model to reshape visual space to optimally align with the textual space. Extensive experiments on various image classification tasks show the effectiveness of our work in addressing data heterogeneity. The codes are released at https://github.com/MoratalYang/FedDDA.

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Yihao Yang" I
Wenke Huang"
Guancheng Wan"
Bin Yang'
Mang Ye
Abstract
Existing Decoupled Framework
Federated Parameter-Effficient Fine-Tuning aims to adapt Vision-Language Models for downstream tasks distributed environments. However, data heterogeneity across participants hinders collab- orative effectiveness, necessitating personalized adaptation to cover distinct data distributions. Current personalized methods suffer from two limitations. 1) Textual Property Loss: Exist- ing methods facilitate the collaboration between decoupled prompts at feature level, which potentially undermines the textual properties of the prompts. 2) Visual Feature Diversity: The diversity of visual features makes it challeng- ing to leverage naive image features directly for image-text alignment in downstream tasks. In this work, we propose Federated Disentangled Tuning with Textual Prior Decoupling and Vi- sual Dynamic Adaptation (FedDDA) overcome the above limitations. Specifically,, we encour- age decoupling prompts in way that maximizes the efficacy of prior knowledge, which is essen- tial for maintaining a coherent linguistic con- text. Furthermore, we design a visual adaption model to reshape visual space to optimally align with the textual space. Extensive experiments on various image classification tasks show the effectiveness of our work in addressing data het- erogeneity. The codes are released at https: / //github.. wwwm/Mooatttooo al Yang/ Fe.DDA.
Text
Visual
CLS
Derzandaa
g
[+
++
Viuual
Inkoesies
capens
Colliweztio the tustim inval
Jasty oftert bmage aeeaatgeaee challegong
Underniie
kotad popests
(a) Textual Property Loss
Visual Feature Diversity
Figure I. Background and motivation. The top row outlines the framework for existing decoupled methods, The bottom row illus- trates our motivation hoth modalities. (a) In popular decoupled methods, the collaboration between generalized and personalized prompts relies on newly loss signals at the feature level, which undermines the textual properties of the prompts. (b) The diversity of visual features, stemming from the complex information within images, makes direct image-text alignment challenging.
2017: Yang al., 2019), allowing collaborative model train- ing across decentralized clients without exposing their lo- cal data. Recently, pre-trained Vision- Vision-Language Models (VLMs), e.g.. CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), have demonstrated exceptional versatil- ity across a wide range downstream tasks. Meanwhile, Parameter-Efficient Fine- Fine-Tuning (PEFT) (Xu al., 2023; Luo al., 2023) acts a promising solution adapt to the VLM, which updates limited number of param- eters while keeping the remaining ones frozen. Leveraging its lightweight nature, Federated Parameter-Efficienn Fine- Tuning (FedPEFT) has emerged as privacy-friendly col- laboration paradigm, enabling distributed data sources to jointly fine-tune VLMs. Several studies (Guo et al., 2023b; Lu al., 2023; Qiu et al., 2024; Feng al., 2023) follow the FedAvg paradigm (McMahan et al., 2017) to aggregate selected candidate parameters.
1. Introduction
Federated learning is distributed paradigm (McMahan et al., 2017; Konečný et al., 201 6a;b; Mohassel & Zhang,
'Equal contribution 'National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China. Correspondence Mang Ye <ye- mang @ whh..edu.n>>.
However, similar to conventional FL, FedPEFT faces chal- lenges posed by data heterogeneity (Zhao et al., 2018; Kairouz et al., 2021; Li et al., 202 2020a), which is inher- ent in distributed environments. Specifically, data from mul-
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
tiple clients typically presents Non-IID (non-independent and identically distributed) distributions. Therefore, solely learning the shared parameters brings the restricted repre- sentations to cover distinct distributions. Personalized Fed- PEFT (Su et al., 2024; Guo et al., 2023a) acts as a crucial role to benefit the multiple collaboration under data het- erogeneity. For personalized adaptation, decoupling is key to achieving fine-grained representations. Recent methods (Bai et al., 2024; Cui et al., 2024; Li et al., 2024) follow the decoupling strategy by constructing independent prompts with specific properties, e.g., w.g..generliizatie generalization and personaliza- tion, and designing external signals to guide optimization.
by guidance words, with each responsible for extracting consensus and style knowledge, respectively. In this way, the semantic fusion of consensus and style knowledge draws support from prior knowledge without additional signals. In response to II,, we propose Visual Dynamic Adaptation (VDA) to reshape the visual space in a dynamic manner. Specifically, we introduce a dual adapter architecture to decouple consensus and style features from naive visual features and subsequently construct adaptive visual rep- resentation. Given the federated setting, one adapter is shared while the other is private. The shared adapter extracts client-invariant visual information, i.e., consensus knowl- edge, while the private adapter captures client-specific visual information, i.e. style knowledge. Inspired by Mixture-of- Experts (MoE) (Shazeer et al., 2017; Masoudnia & Ebrahim- pour, 2014; Cai et al., 2024; Chen et al., 2023), we further implement gating mechanism to dynamically harmonize visual representations. Based on the consensus and style knowledge, this visual adaptation module can adaptively learn the optimal visual space that aligns with textual space.
However, as illustrated in Fig. I, these decoupled researches are baffled by two certain limitations stemming from their paradigm. 1) Textual Property Loss: With respect to textual modality, these methods rely newly designed loss signals to facilitate collaboration among independent prompts at the feature level. However, this collaborative mode does not explicitly decouple prompts at the seman- tic level, potentially undermining the textual properties of the prompts. Specifically, the textual properties are shaped by prior knowledge acquired through extensive image-text alignments, which plays vital role in constructing co- herent semantic. Yet, independent prompts derive minimal benefits from prior knowledge, as they are fed into models separately, while prior knowledge is typically built upon a single input. Thus, our goal is to achieve better textual decoupling in a manner that maximizes the efficacy of prior knowledge. II) Visual Feature Diversity: Towards the vi- sual modality, crucial issue is the diversity of vision, which is overlooked by previous personalized methods. Unlike uniform labels, images within the same category often ex- hibit varied styles, leading to a diverse visual feature space. Additionally, the mode of image-text alignment depends on the distribution of both visual and textual latent spaces (Cho et al., 2023a). While the textual space can be optimized by textual prompts, the diverse visual space makes chal- lenging to leverage fixed features directly for image-text alignments. Hence, it's crucial to optimize visual space that is optimally aligned with the textual space.
In this paper, our work combines Textual Prior Decoupling with Visual Dynamic Adaptation to achieve decoupling from both textual and visual modalities. Textual Prior Decoupling learns fine- grained representations for robust language su- pervision, while Visual Dynamic Adaptation (VDA) adjusts visual features to optimally align with the textual space. In a nutshell, the main contributions are as follows:
We focus on Personalized Federated Fine-Tuning, highlighting two limitations of existing works from both textual and visual modality.
Serameter-Efficien
We propose a simple yet effective method FedDDA. Through Textual Prior Decoupling and Visual Dynamic Adaptation, FedDDA obtains effective personalized vision-language models.
We conduct extensive experiments on four datasets: Of- fice31, PACS, OfficeHome, and DomainNet. Through set of ablation studies, we validate the efficacy of FedDDA and the indispensability of each module.
2. Related Work
To address these issues, we propose a simple yet effective al- gorithm, Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation (FedDDA). For I), we introduce Textual Prior Decoupling (TPD) to decou- ple generalized and personalized prompts. Specifically, we rethink the utilization of hand-crafted prompts. As high- lighted in (Cho et al., 2023b; Gal et al., 2023), hand-crafted prompts with domain-specific labels provide greater dis- crimination across multi multi-domain scenarios by leveraging abundant prior knowledge. This motivates us to integrate generalized representations with personalized ones within robust linguistic context. Specifically, we explicitly decou- ple the prompts into global and local components connected
2.1. Parameter-Efftiient Fine-Tuning
As model parameter counts increase, fine-tuning the en- tire model becomes prohibitively expensive and impractical. Parameter-Efficien Fine-Tuning (PEFT) (Xu et al., 2023; Luo et al., 2023; Huang et al., 2025b;a) has been introduced to address this challenge by redueing the number of ac- tive parameters while maintaining comparable capability to the full fine-tuning. Specifically, PEFT involves updating only a limited number of additional parameters or selec- tively optimizing a subset of the original parameters for effi- cient adaptation. Instead of selecting a subset, adding new parameters adopts plug-and and- approach, keeping the
2

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
original model parameters frozen, such as reparameterized fine-tuning (Valipour et al., 2023; Xu et al., 2024) and addi- tive fine-tuning (Zhou et al., 2022b; Gao et al., 2024). For instance, Low-Rank Adaptation (Hu et al., 2021) (LoRA) is the foundational work of reparameterized fine-tuning, where trainable low-rank decomposition matrices are injected into the network. Additive fine-tuning encompasses two primary techniques: prompt-basee tuning and adapter-bace based tuning. Prompt-based tuning (Zhou al., 2022a; Sun et al., 2022; Khattak et al., 2023; Yao et al., 2023; Zhang et al., 2024b; Fang et al., 2025) employs a set of contextual vectors that are inserted either into the input embeddings alone or across all intermediate layers, to stimulate task-specific knowledge. Adapter (Gao et al., 2024; Zhang et al., 2024a; 2022), actu- ally lightweight adaptation model the end of the encoder, reshapes generated features into a new space, which can also be inserted into each layer. In this paper, we focus PEFT based on VLMSs and integrate textual prompts with visual adapters to establish a complementary relationship between language and vision.
2.3. Personalized Federated Parameter-Efficien
Fine-Tuning
Fine-Tuning integrates PEFT techniques into federated learning reduce communica- tion and computational burdens. Some efforts (Guo et al., 2023b; Lu et al., 2023; Qiu et al., 2024; Feng et al., 2023) focus on shared parameters across different PEFT architec- tures to achieve generalized capability, similar to FedAvg (MeMahan et al., 2017). For instance, PromptFL (Guo et al., 2023b) extends prompt tuning (Zhou al., 2022b) to the federated setting by utilizing set of shared learn- able context vectors to activate prior knowledge. Similarly, FedCLIP (Lu et al., 2023) adopts a shared attention bbased adapter within the visual branch to identify task-specific features. However, direct aggregation in these methods encounters the same impediments as conventional FL due to data heterogeneity. To address this issue, Personalized Federated Parameter-Efficien Fine-Tuning has been pro- posed to accommodate diverse distributions. Some methods introduce private parameters tailored to individual clients while maintaining shared prompts to foster global knowl- edge, such as pFedPrompt (Guo et al., 2023a) and FedAPT (Su et al., 2024). Moreover, approaches like FedOTP (Li et al., 2024), FedPGP (Cui et al., 2024), and DiPrompT (Bai et al., 2024) decouple prompts into independent components with distinct properties. These properties are typically de- rived from global or local federated setups and are further enhanced through external loss signals or multi-stage train- ing. In this paper, we analyze the limitations of existing decoupled works from both modalities and further explore the decoupling strategy.
Federated
2.2. Data Heterogeneous Federated Learning
With growing privacy concerns, Federated Learning (FL) (McMahan et al., 2017; Konečný et al., 20:6;;b; Mohas- sel & Zhang, 2017; Yang et al., 2019; Ye et al., 2025) has been proposed as framework for collaboratively training models across decentralized clients without leaking sensi- tive data. As milestone, FedAvg (McMahan et al., 2017) updates global model by aggregating local parameters from multiple clients and subsequently sends the updated model back for further training. However, FedAvg strug- gles with challenges posed by Non-IID data distributions (Zhao et al., 2018: Kairouz et al., 2021; Li et al., 202 la; 2020a; Ma et al., 2025) (i.e. data heterogeneity), which can negatively impair model performance. A common form of heterogeneity is label shifts, arising from imbalanced sam- pling. Several methods (Li et al., 2020b; Acar et al., 2021; Li et al., 202 1b; Tan et al., 2022; Hu et al., 2024; Karim- ireddy et al., 2020) leverage global penalty term to address this challenge. However, these methods focus on single domain performance and overlook domain shift, where data from diverse sources inevitably falls into distinet feature distributions. This discrepancy leads to optimization in- consistencies between clients and the central server. Some efforts (Li et al., 2021c; Liu et al., 2021; Wu & Gong, 2021; Ye et al., 2023; Huang et al., 2023; 2022) attempt to de- velop personalized models for these multi-domain scenarios. However, the above methods mainly target conventional net- works. In this paper, we explore novel federated paradigm based on the rision-language model, e.g., CLIP, which lever- ages contrastive learning to construct image-text alignment. This paradigm does not require training from scratch on closed datasets, enabling rapid adaptation.
3. Methodology
In this section, we present the details of our proposed Fed- DDA, as depicted in Fig. 2. We leverage Textual Prior Decoupling (Sec. 3.2) to learn fine-grained representations that effectively capture both consensus and style knowledge, and employ Visual Dynamic Adaptation (Sec. 3.3) to re- shape visual features in dynamic manner. After these improvements, we follow the standard CLIP loss to achieve language-vision collaboration (Sec. 3.4).
3.1. Preliminary
CLIP and Parameter-Eflicient Fine-Tunine By align- ing 400 million image-text pairs, CLIP obtains remarkable zero-shot capabilities, enabling generalized performance across diverse tasks. However, this generalized performance falls short for specific tasks. To enhance adaptability to downstream tasks, researchers have introduced Parameter- Efficient Fine- Fine-Tuning (PEFT), which fine-tunes the decision boundaries of VLMs at minimal cost by updating only lim- ited set of parameters. In this paper, we employ two typical
3

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
General Adapter
Specifie
P
[class] style
Text
Adapter
Pg
Global Prompt
P
Local Prompt
Gate
L Lce
nnl
Eq. (9)
Trainable
Fixed
Weights
Visual
Eq.(10)
Client
with
Pa
Server
L Lce
speeific
Eq.19) (9)
general
Shared
Pg
Visual
weighted
Eq. (10)
Client
Client
Pa
Private
Client
Visual
Pa
Ps
Pn
Adaptation
Local update
(b) Global aggregation
Figure 2. Architecture Illustration of the FedDDA. In Textual Prior Decoupling (Sec. 3.2), explicitly decouple the prompts into global local components connected by guidance words, with each responsible extracting consensus and style knowledge, respectively. Visual Dynamic Adaptation (Sec. 3.3), dual adapters with gate deconstruct visual features and subsequently construct adaptive visual representation in a dynamic manner.
prompt and adapter, which incorporate additional parameters into the models. For clarity, we use prompt tuning to illustrate the workflow of PEFT in this section, while details regarding the adapter can be viewed in Sec. 3.3.
PEFT techniques:
Personalized Federated PEFT. Considering a federated scenario with M clients and central server, each client is equipped with a CLIP model and holds local dataset D; { }))) with m, scale. Denote D [D1, D2, DM] } as the entire collection of datasets from diverse domain and C. as the set of active clients. During local training, the optimization objective is to minimize the cross- cross-entropy loss in Eq. (2):
Prompt tuning learns set of vectors denoted as {PI,..., PL.. Specifically, the learnable vectors are com- bined with class token embedding c. The input tokens of Let I(-) denote the image encoder and T'(-) denote the text en- coder. Given an image sample with label the prediction probabilities be calculated as:
k-th class Pk can be formulated as (pı,..., PL, PL, (+).
arg min (=,)) Lce (x, y): 0,
To minimize the prejudice of aggregated parameters per- sonalization, private parameters are introduced. Specifically, the updated parameters 0, consist of global and local com- ponents, denoted as (0g.s: 01, During the collaborative phase, each client transmits the generalized parameters "0 to the server, where they are aggregated with weights pro- portional to the scale of the client's 's data. The updated global parameters for communication round t is:
explsim( (2), Σ* (5s(ssmm(((),T((P)))))
<(k|>)-
(I)
where sim((..) denotes cosine similarity. is the temper- ature of Softmax and K is the number of class. After cal- culating the probability, additional parameters 0 (i.e. in prompt tuning) are optimized through contrastive learning while freezing the original ones:
m4 0²11 Σ
9
jec, my g.
SEC,
Subsequently, the server distributes the aggregated parame- ters back the clients for further training.
Loe(x, = Σ log a(k|z))
3.2. Textual Prior Decoupling
where y is a one-hot label vector and Lee represents the cross-entropy loss.
Motivation. Driven by the decoupling strategy, we tend to separate the prompts for fine- ine-gnained representation. Specif-

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
ically, we seek to integrate consensus and style without com- promising semantic integrity. This fusion can be effectively achieved by prior knowledge, which facilitates the nuanced understanding of the relationships between input tokens. A hand-crafted prompt that combines consensus and style, e.g. "a dog with a style of cartoon", is the most straightfor- ward way to establish the cooperation between consensus and style representations, with the aid of prior knowledge. Consequently, we aim to decouple generalized and personal- ized prompts as mrisensus-and-styy texts, leveraging prior knowledge to create a robust linguistic context.
(+), like RELU. Thus, an adapter can be formulated as:
A(z, W) = <(Wap up
xx),
(6)
where W is the parameter collection {Wup. Wdown J. Con- cretely, we propose dual adapters: the general adapter Ag and the specific adapter A,, to reshape visual features in co- operation. Similar to decoupled prompts, the parameters of the general adapter are shared across clients to aggregate the consensus information and those of the specific adapter are updated locally to handle diverse style knowledge. Given the naive visual feature Zn, the general and specific features are generated as:
Decoupled Textual Prompts. Specifically, we explicitly decouple the prompt into a global generalized prompt Pa and local personalized prompt Pt, both of which consist of set of continuous vectors {Pi, ...: PL.. To preserve textual semantics, these two prompts prompt, denoted as P.. With respect to the text modality, the text encoder has the ability to embed the textual style guidance, due to its exposure to a broad range of styles when pre-trainn wre-training. Building on this insight, we introduce guidance words, "with a style of", between two prompts to evoke stylistic semantics. Hereby, we define P. as:
Zg = Ap(Zn: Wg),
Z, = A.(n: W.).
(8)
combined into single
For precise adaptation, advocate for a dynamic combina- tion of visual components. Driven by the Mixture of Experts (MoE) (Shazeer et al., 2017; Masoudnia & Ebrahimpour, 2014; Cai et al., 2024; Chen et al., 2023), we design a gating mechanism G()) to generate weights for the feature sums, This generator implemented by a linear layer W gate that maps naive visual features into weighted vectors p E R.3 The generated weights are calculated follows:
Pg [class] with a style of Pt
9x = G( W Wgate)s
(x(x) = exp(yx:)
where Pg is sent to the server for aggregation, capturing consensus knowledge from other clients, while P, remains in the local update to extract individual characteristics. In this way, we can obtain the text feature fused with consen- sus knowledge and style knowledge through the attention mechanism in the encoder.
ΣE exp (9x,)
where (x( represents the softmax- normalized weight of the i-th dimension of 9z- Specifically, pa(x)) (i E (1,2,3) respectively correspond to the weights for the naive, gen- eral, and specific features, denoted as Pn. Py and ps. After calculating the weight of each sample, the final weighted visual feature Z is given by:
3.3. Visual Dynamic Adaptation
Motivation. Existing researches on decoupling primarily focus on different prompt settings while neglecting the di- versity of vision. Unlike uniform labels, images within the same category often exhibit varied styles, leading to a diverse visual feature space. Furthermore, the alignment between image and text is influenced by the distribution of both visual and textual latent spaces. While textual prompts can optimize the textual space, the varied visual space presents difficulties in directly utilizing fixed visual features for image-text alignment. Therefore, it is essential to optimize the visual space. A straightforward way is to introduce the adapter architecture after the image encoder to reshape visual features into new visual space.
Z Pn Z, Pg Zg Ps Za-
(10)
3.4. Language-vision Collaboration
During image-text alignment, the text encoder generates features of decoupled prompts, while dual adapters with gate reshape naive visual features produced by the image encoder. The local update follows the standard CLIP loss as Eq. (2), without any additional loss signals. The final prediction probabilities in the loss can be formulated as:
exp(sim( T(P..))/T) ((k(x) = ΣK exp(sim Zw. (PP.)))TT
(1I)
Dual Visual Adapters with Gate. The adapter we employ is a compact and scalable network composed of two linear layers. Denote W Wdown E kddr as the down projection and
After local training, the server shares the global parameters, consisting of g and W in our method:
Wep E x as the up projection, where d' is the dimension of encoder output and is the dimension of the hidden layer. Each linear layer is followed by a nonlinear activation layer
pt Σ 9
m4
pt-
(12)
Ejec, my
VEC.
5

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Algorithm FedDDA
sual adaptation module that dynamically adjusts the visual space for each client. The visual components inevitably introduce more training parameters, leading to increased computational and communication burden. It should be noted that this limitation is not unique to our approach but is an inherent aspect of PEFT shared across many methods. Through fine-grained design, we maximize the utility of these parameters achieve better performances.
Input:
Communication rounds T, participant set M, i-th client's private dataset Dj and parameter collections 0 = (Po.s, Pu,s, aa.i, Ws,,, Wo learning rate and to- kenized embedding of guidance words GM.
for = 1, 2.2..... T do Participant Side
for = 1,2, ..,, M in parallel do P g.a. W Local Update(P2) Pit
end
Server Side
4. Experiment
Pl-- P Exec, Ejec, my W Eiec. Eyeo, my
W
4.1. Experimental Setup
end
Datasets. We extensively evaluate our method on the fol- lowing four multi-domain classification tasks:
Local Update(P,,.W) Pg.
Po
Office31 (Saenko et al., 2010) contains 31 classes of com- mon objects office scenarios across 3 domains: Amazon
W
W. W
for (z. v) Dj do
(A), Webcam (W), and DSLR (D).
/* Textual Prior Decoupling */
P. Pooe, CLASS, GM, Pt,) Pt,a]
PACS (Li et al., 2017) includes 4 domains: Art-painting (A), Cartoon (C), Photo (P), and Sketch (S), with 7 classes.
/a Visual Dynamic Adaptation //
Z, = I(x)
OfficeHome (Venkateswara et al., 2017) consists of 4 domains: Art (A), Clipart (C), Product (P), and Real world (R), each with 65 categories.
Zg Ag,,n, W. Z. = A. Wo.)
Po (Zn, W s) in Eq. (9)
Z = Pn. Zn + Pp Zg + P.- Zo
DomainNet (Peng et al., 2019) includes 6 domains: Cli- part (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), and Sketch (S), each with 345 categories. To enhance evaluation efficiency, we choose the first 100 categories of every domain as the overall dataset.
/+ Language wiauuag--iiiioe
Collaboration +/
Lee = - Σx=1 ylog v exp(aimi Zu 7 (3)r))
explasm Zu TIP, (l)+r
0,-0 0, - 1/V
end
Retum Py. and Wg,s Server
Data Heterogeneity. To simulate domain shifts, we evenly assign each client a distinct domain from the dataset. We consider two client configurations: @ the client size equals that of domains; @ the client size is twice that of domains. To further evaluate performance under data heterogeneity, we take label shifts into consideration. Specifically, we partition the data within each domain based on a Dirichlet distribution (Lin et al., 2020; Kotz et al., 2004) under @ client setting, and adjust the 33 parameter to model varying degrees of unbalanced label sampling.
W = Σ
my
W
(13)
Ejec, my
iEC,
Through these fine-gree grained designs, our method implements system of functional separation within visual-language space. In the text branch, decoupled prompts concentrate solely on textual semantics. In the visual branch, dual adapters serve as feature extractors to capture complex vi- sual information from the naive visual features, with the proportion of generated features determined by the gate. Through this separation of functions, the components pay attention to their own roles, thereby enhancing the overall ability. The overall federated learning algorithm is shown in Algorithm 1.
Counterparts. We compare our FedDDA against two gen- eralized approaches (Guo et al., 2023b; Li et al., 2020b) as well two popular solutions for personalized performance (Li et al., 2024; Cui et al., 2024):
PromptFL [TMC'23] (Guo et al., 2023b): The first to integrate prompt tuning into federated setting by learning a unified textual prompt.
Discussion and Limitation. Existing methods predom- inantly achieve decoupling through collaboration among independent prompts. However, these approaches not only compromise the semantic knowledge of the textual prompt but also overlook visual diversity. In our FedDDA, we construct prompts as Eq. (5) to fully leverage prior knowl- edge for effective decoupling. Additionally, unlike previous personalized methods, which restrict additional parameters exclusively to the text branch, our method introduces vi-
PrompdFL+Prox [MLSys'20] 20] (Li et al., 2020b): Restrict updated prompts by proximal term instead of aggregation.
FedOTP [CVPR 24] (Li et al., 2024): Employ unbalanced optimal transport to align local visual features with global and local prompts.
FedPGP [ICML'24] (Cui et al., 2024): Incorporate low- rank adaptation term with an additional contrastive loss to balance generalization and personalization.
6

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Table Comparison with the state-of-the-art solutions on Officee1,, PACS, OfficeHome, and DomainNet tasks with domain shifts under 1 and @ client settings. AVG denotes average accuracy calculated on all domains and best in bold. Please see details in Sec. 4.3.
Othec31
PAC'S
Officellome
DomainNet
Methods
AVC
AVG
AVG
One domaun for one client
Zeru-sbot
81.14
72:45
74.05
75.88
97.22
99.06
99.86
88.16
96.08
84.30
66
89
06
89.66
82.33
71.93
53.30
65.73
13.57
83.49
66.46
59.08
ProngeF1.
88.90
87.55
94.30
14.30 90.25
114.90
98.72
99.23
99.88
94.11
98.16
ro.0s
75.76
94.3
32.32
93.59
87.65
86.55
70.29
79.89
34.31
91.54
79.97
73.76 114.65
PromplFF+Prox
89.22
93.04 90.68
114.50
98.76
99.19
99.88
98.02
11.94
76.28
94.25
93.59
15.24
87.47
71.25
82.15
32.63
91.79
81.20
74.41 115.33
FedOTT
85.23
94.69
94.94
91.79
115.01
98.74
99.43
99.77
95.55
98.37 19.23
79.71
76.24
18
87.10
83.81
13:46
86.73
69.80
82.05
50.37
90.75
82.69
77.06
117.96
FedPGP
89.04 95.10
96.99
93.70
99.16
99.59
99.87
95.65
98.57
88.55
77.20
95.06
93.77
88,65
19:30
89.79
77.68
87,91
53,08
93.92
86,50
81,08
FedDNAA
89.32 97.14 98.23
94.90....00
99.34
99.74
99.88
96.65
98.90-0.0
87.07
78.67
96.32
93.52
99.74
77.34
88.46
63.67
94,14
88,06
83.57
One domaon for DH0 clients
Zerosbox
81.14
72.66 73.99
75.93
97.22
99.06
99.86
88.16
96.08
84.33
66 28
89:07
89.69
82.34
71.94
53,31
65.73
13.57
83.49
66.46
59.08
PrompeH1.
89.89 84.89 92. $0.09
113.00
98.69
99.22
99.88
94.86
98.17 19.09
86.65
75.38
94.51
93.49
87.51
15.11
86.57
71.29
81.90
35.34
92.40
80.3
74.63
115.50
89.25 86.85 91.80
89.30
113.37
98.47
99.16
99.83
91.55
98.00 11:92
86.13
74.60
91.10
93.44
87.15
14.51
86.48
72.63
81.72
34.43
92.31
80.85
74.73 115.00
PedOTP
84.73
95.94
89.94
114.00
98.43
99.39
99.75
94.91
98.12 10.04
73.96
91.20
57.10
82.56
10.02
85.73
69.50
81.45
48.49
90.38
82.11
117.00
FedPGP
95. 92.03 116.10
98.47
12:09
77.01
94.96
19
88.89
51.14
93.70
86.07
80.71 120.45
FedDIAA
88.39 95.06 98.10 93,85
417.00
99.13
99.77
99,89
95.96
98.69 +0.03
86.69
77.98
95.17
92.96
88.22
88.65
76.07
86.67
61.76
93.47
86.80
82.24
Table Comparison with the state-of-the-art solutions on Office31, PACS, Officelome, and DomainNet with domain shifts and label
shifts under @ client setting. Refer to Sec. 4.3.
Office31 At 0.3
PACS
Oflicefoone (1 0.3
DomminNet (3-0.3 0.3
Methods
A D.1
A 0.55
/ 0.1
( 0.3
0-005
/7 0.1
/ 0.5
3 0.1
e 2- 0.5
Zeno-shr
75.36
75.01
75,49
95.97
96.01
95.98
82.27
82,99
82.30
59.07
59.34
59.11
PrompiFL.
89.01
90.44
88.40
97.57 13.00
98.02 pa:rs
97.98
87:35 15.06
87.23 10.04
87.01
73.14
114.00
73,88
114.64
74.23
PrompthL.
Prox
89.27 11a.91
89.66 114.00
88.11 110.03
97.94 12.00
98.01
12.00
98.05
-2.07
87.46 p5.10
87.32
14.03
57.36
-1.00
73.41
114.44
73.66
111.42
74.07
111.00
FedOTP
90.78 125.42
90.54
135.35
133.28
98.87 12.00
98.62
12:03
98.49
-2.51
84.81 13.54
85.66
15:47
84.28
03.04
70.52
115.45
77.70
115.46
76.94
117.55
FedPGP
91.78
90.88
91.68
99. 10 p3.03
98.82
12.01
98.72
89,49
17:30
89.63
17:24
88.78 78
80.72
82,46
123.20
82,12
120.01
FodDDA
94.68
94.34
94.73
99.39
99.06
99.10
89.85
17:56
90.25
89.24
82.98
123.01
83.24
124.00
82,82
(21/73
Officel
PACS
OfficHHnne
Dot
Peelop
Coumunaatiio Epaci
spach
Communkation Epech
Conmoniionion Epech
Figure 3. Comparison of convergence of average accuracy with the SOTA methods on Office31, PACS, OfficeHome, and DomainNet tasks with domain shifts under ① client setting. Refer to Sec. 4.3.
Implementation Details. For a fair comparison, follow- ing (Guo al., 2023b; Li et al., 2024; Zhang et al., 2023: Huang et al., 2023), we use the same settings across all experiments. We use the publicly available CLIP (Radford et al., 2021) model with the ViT-B/16 as the backhone model. The prompt length is set to 16 and the prompts are randomly initialized with the normal distribution. We utilize SGD optimizer (Robbins& & Monro, 1951) to optimize selected candidate parameters for 50 communication rounds with 1 local epoch. The learning rate Ir is 0.001 and the train batch size of images is 32. We fixed the random seed at 1 to ensure reproduction.
Efficacy of Textual Prior Decoupling. To further explore the impact of different prompt settings, we conduct experi- ments without visual adaptation module, as shown in Fig. 4. Notably, the case with single global prompt has 32 to- kens to eliminate the difference that comes from the number of updated parameters. The results indicate that global prompt is insufficient to match diverse distributions across clients, resulting in a lower performance compared to decou- pled prompts consisting of global and local parts. Further- more, with the help of guidance words, textual decoupling achieves higher accuracy, manifesting that leveraging prior knowledge can channel more stylized information. Notably, we observe that the benefits of guidance words are smaller than those gained from the decoupled prompts. The reason is that visual diversity hinders prompts from focusing on semantic decoupling so that guidance words only capture weak semantic knowledge.
4.2. Ablation Study
For thorough assessment, we perform a set of ablation stud- ies to investigate the efficacy of each component under multi- domain scenarios. To improve the efficiency of comparison, the experiments are set up on two datasets, Office31 and PACS, under @ client setting.
Efficacy of Visual Dynamic Adaptation. Through detailed experiments, we investigate the capabilities of three visual components: the general adapter, the specific adapter, and
7

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Office5
PACS
Table 3. Ablation study of Textual Prior Decoupling (TPD
28.55
Sec. 3.2) and Visual Dynamic Adaptation (VDA in Sec. 3.3) in
Office31 and PACS tasks. Please see details Sec. 4.2.
Office31
PACS
TPD
VDA
AVG
4
AVG
A
89.53
97.97
1.2
GPYEP
GPICOWTLL
CP GP1LP
GP:CWWLF
93.24
+3,71
98.62
+0.65
Figure 4. Comparison of different prompt designs in Office3 and PACS tasks. We present the change of accuracy from only global prompts (GP) to the decoupled prompts (GP+LP), and finally to the decoupled prompts connected by guidance words (GP+GW+LPP as shown in Eq. (5). See details See. 4.2.
/
92.03
+2.50
98.51
+0.54
93.85
+4.32
98.69
+0.72
Office31
PACS
(a) Zero-shot
(b) FedDDA
(c) Zero-shot
(d) FedDDA
Figure 6. t-SNE Visualization of Visual Space participants from Office31 (a) (b) and OfficeHome (c) (d). Refer to Sec. 4.3.
X0.9
4.3. Comparison to State-of-the-Arts
The results in Tab. I plot the final accuracy against state-of- the-art (SOTA) methods on Office31, PACS, OfficeHome, and DomainNet with domain shifts under two types of client settings. Among most domains, our work achieves the highest accuracy, confirming that FedDDA can effectively boost performance across different domains. As shown in Fig. 3, the accuracy curve reveals method's high ac- curacy and fast convergence across the four datasets. We visualize the t-SNE visualization analysis of visual space in Fig. 6. We further conduct the experiments on datasets with domain shifts and label shifts in Tab. 2. Specifically, the division of domains among clients remains unchanged as @, while clients within the same domain are assigned data partitioned by Dirichlet strategy. We set the parame- ter 3 in {0.1, 0.3, 0.5} to simulate varying levels of label shifts. FedDDA achieves the highest accuracy across all set- tings, ensuring the robustness and compatibility of our work. These comprehensive experiments illustrate that our method significantly outperforms its counterparts, highlighting the effectiveness of our design under data heterogeneity.
Raseiirr
D 0
DAA 04,+A, Ag4
Figure 5. Analysis of visual components in Office31 and PACS tasks, including general adapter (A, in Eq. (7)), specific adapter (A, in Eq. (8)) and gate (G in Eq. (9)). See details in Sec. 4.2.
the gate mechanism. To ensure fair comparison, we set the prompt as shared vectors. We consider five cases for the final visual feature: (1) naive visual feature (Zn)); (2) the average of naive and general features ( (Z,+Zy); )) (3) the average of naive and specific features (Lnt2e); (4) the av- erage of naive, general and specific features (2ntZ,t2s) (5) the weighted sum of naive, general and specific features (Eq. (10)). As seen in Fig. 5, the incorporation of general or specific adapters achieves better performance than naive learning, validating the efficacy of the adapter architecture. This also evidences that both consensus and style knowl- edge contribute to the adaptation of visual space. Moreover, explore the combination of general and specific adapters, which obtains greater discernment than using either adapter alone. Furthermore, the inclusion of the gate mechanism sig- nificantly increases performance, supporting our motivation for dynamic adaptability on multiple domains.
3
5. Conclusion
In this paper, we explore the data heterogeneity problem in FedPEFT. Our work introduces a simple yet effective algorithm. FedDDA. We leverage Textual Prior Decoupling to learn fine-grained representations for robust language supervision and introduce Visual Dynamic Adaptation to reshape visual feature space dynamically. The effectiveness of FedDDA has been thoroughly validated with popular counterparts over various classification tasks. We wish this work to pave the way for future research on FedPEFT.
Efficacy of Collaboration. Note that the textual decou- pled prompts learn fine-gra grainer representations, and that dual visual adapters with gate reshape visual features based on client specifics. We present the ablation study of two modules in Tab. 3. The final results illustrate that both Textual Prior Decoupling and Visual Dynamic Adaptation contribute significantly to the performance of the model. The combination of two modules yields the best results, underscoring the effectiveness of our work.
Acknovledgement. This work is supported by Na- tional Natural Science Foundation of China under
8

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Grant (62361 166629, 62176188, 623B2080), the Na- tional Key Research and Development Program of China (0244FC33088000) Postdoctoral Fellowship Program of China Postdoctoral Science Foundation (GZC20241268, 2024M762479), and the Wuhan University Undergraduate Innovation Research Fund Project. The supercomputing system at the Supercomputing Center of Wuhan University supported the numerical calculations in this paper.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y.,
Li, H., and Qiao, Y. Clip-adapter: Better vision-languan models with feature adapters. IJC,, 132(2):581-595, 2024.
Impact Statement
Guo, T., Guo, S., and Wang. Pfedprompt: Learning per- sonalized prompt for vision-language models in federated learning. In WWW, pp. 1364-1374, 2023a.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must specifically highlighted here.
Guo, T., Guo, S., Wang, J., Tang, X., and Xu, W. Promptfi: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. JEEE TMC, 2023b.
References
Acar, D. E., Zhao, Y., Matas, R., Mattina, M., What- mough, P., and Saligrama, V. Federated learning based dynamic regularization. In ICLR, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 111655000.09685. 2021.
Bai, S., Zhang, J., Li, S., Guo, S., Guo, J., Hou, J., Han, T., and Lu, X. Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning. In CVPR, 2024.
Hu, M., Cao, Li, A., Li, Z., Liu, C., Li, T., Chen, M., and Liu, Y. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 12528 12537, 2024.
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
Huang, W., Ye, M., and Du, B. Learn from others and be yourself in heterogeneous federated learning. In CVPR, 2022.
Chen, T., Chen, X., Du, X., Rashwan, A., Yang. F., Chen, H., Wang, 7., and Li, Y. Adamy-moe: Adaptive multi-task vision mixture-of-experts. In ICCV, pp. 17346-17357, 2023.
Huang, W., Ye, M., Shi, 7.,, Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In CVPR, pp. 16312-16322, 2023.
Cho, E., Kim, J., and Kim, J. Distribution-aware prompt tuning for vision- vision-language models. In ICCV, pp. 22004- 22013, 2023a.
Huang, W., Liang, J., Guo, X., Fang, Y., Wan, G., Rong, X., Wen, C., Shi, Z., Li, Q. Zhu, D., et al. Keeping yourself is important in downstream tuning multimodal large language model. arXiv preprint XXXv:25503.04543 2025a.
Cho. J., Nam, G., Kim, S., Yang. H., and Kwak, S. Prompt- styler: Prompt-driven style generation for source-free do- main generalization. In ICCV, pp. 15702-15712, 2023b.
Huang, W., Liang, J., Shi, Z., Zhu, D., Wan, G., Li, H., Du, B., Tao, D., and Ye, M. Learn from downstream and be yourself multimodal large language model fine-tuning. ICML, 2025b.
Cui, T., Li, H., Wang, J., and Shi, Y. Harmonizing general- ization and personalization in federated prompt learning. In ICML, 2024.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision- laigual representation learning with noisy text supervision. In ICML, pp. 1904-4166 PMLR, 2021.
Fang, X., Easwaran, A., and Genest, B. Adaptive multi-promp contrastive network for few-shot out-of- distribution detection. In ICMI., 2025.
Feng, C.-M., Li, B., Xu, X., Liu, Y., Fu, H., and Zuo, W. Learning federated visual prompt in null space for mri reconstruction. In CVPR, 2023.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji. A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
in federated learning. Found. Trends Mach. Learn., pp. 1-210, 2021.
Liu, Q. Chen, C., Qin, J., Dou, Q.. and Heng, P-A. Feddg: Federated domain generalization medical image seg- mentation via episodic learning in continuous frequency space. In CVPR, pp. 1013-1023, 2021.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. Scaffold: Stochastic controlled averaging for on-device federated learning. In ICML, 2020.
Lu, W. HU, X.. Wang, J., and Xie, X. FEDCLIP. FAST GENERALIZATION AND PERSONALIZATION FOR CLIP IN FEDERATED LEARNING. In ICLR Workshop, 2023.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Founda- tional model adaptation without forgetting. ICCV. pp. 15190-15200, October 2023.
Luo, G., Huang, M., Zhou, Y., Sun, X., Jiang, G., Wang, Z., and Ji, R. Towards efficient visual adaption via structural re- ee-paameterization. arXiv preprint arXiv:2302.08/00 000000000......... 2023.
Konečný, J., McMahan, H. B., Ramage, D., and Richtárik, P. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint 1r1iv:1610.02527, 2016a.
Ma, Y., Dai, W., Huang, W., and Chen, J. Geometric knowledge-guice localized global distribution alignment for federated learning. In CVPR. 2025.
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. Federated learning: Strate- gies for improving communication efficiency. arXiv preprint arXiv:1610.05499 2016b.
Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey. Artificial Intelligence Review, 42: 275-293, 2014.
Kotz, S., Balakrishnan, N., and Johnson, N. L. Continu- ous multivariate distributions, Volume 1: Models and applications, volume 1. John Wiley & Sons, 2004.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. cummunication-efficiee learning deep networks from decentralized data. In AISTATS, pp. 1273- 1282. PMLR, 2017.
Li, D., Yang, Y., Song, Y.ZZ. and Hospedales, T. M. Deeper, broader and artier domain generalization. In ICCV, pp. 5542-5550, 2017.
Mohassel, and Zhang, Secureml: system for scalable privacy-preservve machine learning. In IEEE SSP, pp. 19-38, 2017.
Li, H., Huang, W., Wang. J., and Shi, Y. Global and local prompts cooperation via optimal transport for federated learning. In CVPR, 2024.
Peng, X., Bai, Q., Xia, X., Huang. Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In ICCV, pp. 1015-145,, 2019.
Li, Q., Diao, Y., Chen, Q., and He, B. Federated learning non-iid data silos: An experimental study. arXiv preprint arXiv:2102.02077 2021a.
Qiu, C.,, Li, X., Mummadi, C. K., Ganesh, M. R., Li, Z., Peng, L., and Lin, W.-Y Federated text-driven prompt generation for vision-language models. In ICLR, 2024.
Li, Q., He, B., and Song, D. Model-contrasee federated learning. In CVPR, pp. 10713-10722, 2021b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763 2021.
Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Feder- ated learning: Challenges, methods, and future directions. IEEE SPM. pp. 50-60, 2020a.
Li, Sahu, A. K., Zaheer, M.. Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. In MLSys, 2020b.
Robbins, H. and Monro, S. A stochastic approximation method. AoMS, pp. 400-407, 1951.
Saenko, K., Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to new domains. In ECCV, pp. 213-226, 2010.
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fed(bn]: Federated learning non-(iid) features via local batch normalization. In ICLR, 2021c.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture- ofxture-of-experrs layer. ICLR, 2017.
Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion federated learning. NeurlPS, 33:2351-2363, 2020.
10

---

Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation
Su, S., Yang, M., Li, B., and Xue, X. Federated adaptive prompt tuning for multi-domain collaborative learning. In AAAI, pp. 15117-15125, 2024.
Zhang, Wu, S., Gao, L.., Shen, H., and Song, J. Dept: Decoupled prompt tuning. In CVPR, 2024b.
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J.. Qiao, Y., and Li, Tip-adapter: Training-fres clip-adapter for better vision-language modeling. In ECCV, 2022.
Sun, X., Hu, P., and Saenko, K. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurIPS, pp. 00569-30582,, 2022.
Zhang, R., Xu, Q. Yao, J., Zhang, Y., Tian, Q. and Wang, Federated domain generalization with generalization adjustment. CVPR, pp- 3954-3963. June 2023.
Tan, Y. Long, G., Liu, L., Zhou, T., Lu, Q.. Jiang, J., and Zhang. C. Fedproto: Federated prototype learning across heterogeneous clients. AAAI, 2022.
Zhao, Y., Li, M., Lai, I.., Suda, N., Civin, D., and Chandra, V. Federated learning with non-iid data. arXiv preprint arXiv: 1800.000822 2018.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Gh- odsi, A. Dylora: Parameter-efficien tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3274-3287, 2023.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In CVPR, 2022a.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning prompt for vision vision-language models. ICV, 130(9):2337- 2348, 2022b.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In CVPR, pp. 5018-5027, 2017.
Wu, G. and Gong, S. Collaborative optimization and ag- gregation for decentralized domain generalization and adaptation. In ICCV, pp. 6484 6493, 2021.
Xu, L., Xie, H., Qin, S.-Z. J., Tao, X., and Wang, F. L. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint ar arXiv:2312.121444 2023.
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., ZHANG, X., and Tian, Q. QA-loRA: Quantization-awee aware low-rank adaptation of large lan- guage models. In ICLR, 2024.
Yang, Q., Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. ACM TIST, pp. 1-19, 2019.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guider context optimization. In CVPR, pp. 6757-6767, 2023.
Ye. M., Fang. X., Du, B., Yuen, C., and Tao, D. Hetero- geneous federated learning: State-of-the-ar and research challenges. CSUR, 2023.
Ye, M., Shen, W., Du, B., Snezhko, E., Kovalev, V., and Yuen, P. C. Vertical federated learning for effectiveness, security, applicability: survey. CSUR, 5339):1-32, 2025.
Zhang, J., Ma. X., Guo, S., Li, P., Xu, W., Tang. X., and Hong. Z. Amend to alignment: Decoupled prompt tuning for mitigating spurious correlation in language models. In /CML., 2024a.


Reviews:
[{"review_id": "e99dzmiIC1", "reviewer": "Reviewer_Zexp", "summary": "This paper considers the collaborative effectiveness caused by data heterogeneity during Federated Parameter-Efficient Fine-Tuning process. A simple yet effective algorithm is proposed to address the above challenges via Textual Prior Decoupling and Visual Dynamic Adaptation. In this case, both modalities can be collaborative in an unbiased manner. Experiments on various datasets demonstrate the efficiency of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Zexp:\n\nThanks for raising thoughtful questions and concerns. We sincerely hope this point-to-point response allows the reviewer to update the score. We will add a more detailed comparison or analysis of the existing decoupled works in the final version.\n\n**Weakness**\n\n**W1: Analysis of motivation. (Why decoupling prompts into independent components is insufficient to achieve better collaboration?  Why define the decoupled in a combined manner instead of an independent component?)** \n\n**A1:** We apologize for the confusion. We define the problem with current decoupling methods as textual property loss. Specifically, after extensive training, the text encoder is capable of recognizing textual relationships between tokens—what we refer to as textual property. For instance, it tends to perform better with fixed texts like \"[class] with a style of [domain]\", demonstrating its ability to capture concepts such as style. However, methods that decouple prompts into independent components overlook this capability, as they process prompts in isolation while the encoder can only handle tokens that come in together. Additionally, this paradigm depends on newly designed losses to guide the optimization, which is irrelevant to textual properties. Therefore, such methods fail to learn the knowledge of textual relationships between independent prompts. We argue that textual relationship is crucial for coordinating decoupled prompts effectively. Unfortunately, current methods weaken these relationships, making them insufficient for effective collaboration.\n\nThus, we propose a combined approach, where multiple prompts are integrated into a single prompt. Decoupled prompts are fed into the textual encoder together. It can maximize the use of the encoder's knowledge to achieve better textual collaboration. As we demonstrate below, the method that simply combines global and local prompts outperforms existing methods, validating its effectiveness.\n\n*Table: Comparison of simple combination with several methods on Office31 and PACS with $\\beta=0.0$. Simple combination is denoted as $[P_g, class, P_l]$, where $P_g$ is the global prompt and $P_l$ is the local prompt.*\n\n| Method                    | Office31  | PACS      |\n| ------------------------- | :-------- | --------- |\n| PromptFL                  | 88.99     | 98.17     |\n| PromptFL+Prox ($\\mu=0.5$) | 89.30     | 98.00     |\n| FedOTP                    | 89.94     | 98.12     |\n| FedPGP                    | 92.03     | 98.47     |\n| Simple Combination        | **92.66** | **98.56** |\n\n**W2: The differences between FedDDA and DiPrompT.**\n\n**A2:** Thank you for your valuable feedback. The primary difference between FedDDA and DiPrompT is whether the decoupled prompts are independent or not. This also marks the key difference between FedDDA and other previous decoupled methods. Previous decoupled methods, including DiPrompT, decouple prompts into independent components, each designed with distinct properties. These properties are typically derived from global or local federated setups and are further enhanced through external loss signals or multi-stage training. Instead, our method combines decoupled prompts into a single prompt, which maximizes the use of prior knowledge to achieve better textual collaboration. \n\n**W3: The reason why DiPrompT is not included in the performance comparison.**\n\n**A3:**  We fully understand your concern about DiPrompT as a baseline. However, DiPrompT and our FedDDA address different problems with distinct experimental setups. DiPrompT is designed for domain generalization, while our work focuses on personalized adaptation to optimize performance within each client's own domain. Unfortunately, the lack of open-source code ultimately prevents us from including this approach in our evaluation. Once it becomes available, we will make every effort to conduct a thorough comparison. Thank you for your insightful comment.\n\n**W4: Lack of experiments on analyzing the performance impact of the number of total clients.**\n\n**A4:**   We conduct the experiments on the performance impact of the number of total clients as shown in the following Table. Our method constantly achieves better results than other methods.\n\n*Table: The performance impact of the number of total clients on PACS with $\\beta=0.0$.*\n\n| Client Number             | 4         | 8         | 20        | 40        | 50        |\n| ------------------------- | :-------- | --------- | --------- | --------- | --------- |\n| PromptFL                  | 98.16     | 98.17     | 98.08     | 97.98     | 97.96     |\n| PromptFL+Prox ($\\mu=0.5$) | 98.02     | 98.00     | 98.03     | 97.98     | 97.96     |\n| FedOTP                    | 98.37     | 98.12     | 97.71     | 97.38     | 97.25     |\n| FedPGP                    | 98.57     | 98.47     | 98.14     | 97.61     | 97.41     |\n| FedDDA (ours)             | **98.90** | **98.69** | **98.35** | **98.08** | **98.05** |", "claims_and_evidence": "Most of the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N.A. There is no proof for theoretical claims.", "experimental_designs_or_analyses": "The experimental results are comprehensive, with the author conducting various experiments under different settings to ensure thorough evaluation.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n\n1. The paper is well-written and easy to understand. The motivation and proposed solution are reasonable and have a certain degree of novelty.\n2. The authors provide a clear and well-structured discussion for the experimental part.\n\nWeaknesses\n\n1. It is suggested to add a more detailed comparison or analysis for the existing decoupled works in Related Work, e.g., why decoupling prompts into independent components is insufficient to achieve better collaboration? What are the differences between FedDDA and DiPrompT?\n2. The intuition behind the decoupled textual prompt is not clear, why define it in a combined manner instead of an independent component?\n3. It seems that DiPrompT can be also a baseline in the comparison experiments, please clarify why this method is not included in the performance comparison.\n4. There lack of experiments on analysing the performance impact of the number of total clients, i.e., 50 clients or more.", "other_comments_or_suggestions": "See the above weaknesses.", "questions_for_authors": "See the above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jN6zI6u4aR", "reviewer": "Reviewer_oran", "summary": "This paper addresses the challenge of data heterogeneity in Federated Parameter-Efficient Fine-Tuning. To tackle this issue, the authors propose FedDDA, which is designed from two modalities. In the text modality, the proposed approach decouples prompts into global and local components, which are connected through guidance words. For the vision side, the method utilizes dual adapters and a gating mechanism to blend naive, consensus, and style visual features dynamically. Extensive experiments across various scenarios demonstrate the superiority of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oran:\n\nThank you very much for your recognition of our work and for raising thoughtful questions and concerns about our work. We have carefully considered each comment and provided responses.\n\n**Weakness**\n\n**W1: Analysis of t-SNE visualization.**\n\n**A1:**  Thank you for your insightful question. Unlike traditional unimodal architectures, CLIP requires alignment between visual and textual space. In our work, each participant's visual space is optimized to better align with the textual space, which is also being refined simultaneously. This bidirectional optimization leads to improved overall performance, even though the visual space itself undergoes relatively minor changes. \n\n**W2: The results in Tab.1 and Tab.2 report improvement against a weak baseline rather than relatively SOTA methods.**\n\n**A2:**  We will revise Tab. 1 and Tab. 2 to provide a more detailed comparison in the final version, not simply an improvement over the weak baseline. Thanks for your advice.\n\n**W3: The rationale behind selecting each specific technique is not clearly explained.**\n\n**A3:** For the textual modality, prompt tuning has emerged as an efficient way to learn specific semantic knowledge on downstream tasks. Therefore, recent research focuses on how to better use prompt tuning, as we do. For the visual modality, our goal is to go for dynamic adaptation of the visual space to address visual feature diversity, thus better aligning with textual space. The available methods are adapters or visual prompts. Similar to the textual prompt, we explore the combination of global and local components, which is easy to implement in adapter tuning. However, managing the trade-off between global and local components in visual prompts remains challenging, particularly in determining their placement and properties. To handle this trade-off in adapter tuning, we introduce a gating mechanism, which dynamically learns the summation weights to coordinate multiple generated features. Thank you for your constructive comments.", "claims_and_evidence": "- The paper introduces VDA to optimize the visual space and employs t-SNE visualization to substantiate this claim. However, the changes in the visual space, as shown in Fig. 6, appear to be relatively minor. It is unclear whether such minor variations can yield significant contributions. The authors should provide further clarification on this point.", "methods_and_evaluation_criteria": "The proposed method addresses the targeted problem, and the evaluation criteria are reasonable.", "theoretical_claims": "The theoretical formula is clear and straightforward, providing an easy-to-follow structure.", "experimental_designs_or_analyses": "Strengths：\n- The experiments are comprehensive, covering a wide range of scenarios to thoroughly evaluate the proposed method. Ablation studies are thoroughly conducted, providing clear insights into the contributions of each component. \nWeaknesses:\n- The results in Tab.1 and Tab.2 report improvement against a weak baseline rather than relatively SOTA methods.", "relation_to_broader_scientific_literature": "Recent studies on the personalized adaptation of vision-language models in federated settings primarily focus on prompt decoupling, leveraging loss signals, or multi-stage training. In the text modality, this work refines previous decoupling strategies by integrating initially independent prompts into a single prompt, thereby preserving textual attributes. As for the visual modality, it takes visual diversity into consideration, a factor that is often overlooked in existing research.", "essential_references_not_discussed": "No essential references have been left out.", "other_strengths_and_weaknesses": "Strengths：\n- The paper fully considers the properties of both federated learning and vision-language models. The proposed method employs global and local components in both text and image branches. \n- The paper is well-motivated. The authors propose two key limitations based on existing studies and further provide a targeted solution to address these limitations.\n- The manuscript is well-structured, clearly delineating the proposed methodology, experimental setup, and results.\nWeaknesses:\n- This work employs several existing techniques, including prompts, adapters, and MoE. While this is not an issue, the rationale behind selecting each specific technique is not clearly explained. For instance, in response to Visual Feature Diversity, the authors aim to dynamically adjust the visual feature space, yet visual prompts could achieve a similar effect.", "other_comments_or_suggestions": "NO", "questions_for_authors": "Please refer to the above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrynpOjcZZ", "reviewer": "Reviewer_287K", "summary": "The paper proposes a novel FedDDA approach to improve federated learning for fine-tuning Vision-Language Models (VLMs) under data heterogeneity. The focus is on addressing two primary challenges in both language and vision modality. On the textual side, the method integrates two prompts (global and local) in a single part to overcome the textual property loss existing in recent works. On the visual side, it employs a dual-adapter architecture with a gate network to mitigate visual feature diversity. The paper demonstrates the effectiveness of FedDDA through extensive experiments on various datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 287K:\n\nWe sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.\n\n**Weakness**\n\n**W1: Strange decoupling.** \n\n**A1:** A key challenge in decoupling prompts lies in achieving efficient collaboration between prompts with different attributes. One approach is to decouple and collaborate at the feature level, like FedOTP [1] and FedPGP [2]. Instead, we aim to decouple at the semantic level, i.e., in the prompt construction phase, and facilitate collaboration through the pre-trained encoder. Our goal is to coordinate decoupled prompts by leveraging prior knowledge from the encoder to generate a feature that incorporates both consensus and style knowledge.\n\n**W2: Unclear rationale for specific technologies.**\n\n**A2:** With respect to the textual modality, replacing the textual inputs of the CLIP model with prompts is a common approach to learning task-specific knowledge. Our work continues this textual paradigm, and we introduce a visual limitation building on this paradigm: visual feature diversity. We aim to address this challenge by dynamically adapting the high-dimensional visual feature space. A straightforward strategy is to leverage adapters, which are flexible and efficient. We will explicitly state the reasoning behind selecting each PEFT method for different modalities in Section 2.1.\n\n**W3: Writing issues.**  \n\n**A3:** Thank you for your tips. We have corrected the error you noted in Line 81 and other potential issues.\n\n[1] Global and local prompts cooperation via optimal transport for federated learning. Li et al. CVPR 2024.\n\n[2] Harmonizing generalization and personalization in federated prompt learning. Cui et al. ICML 2024.", "claims_and_evidence": "[weakness] Strange decoupling: The authors claim to decouple the prompt, but in practice, the two decoupled parts are merged into a single feature. This undermines the effect of the decoupling. Could the authors explain how their decoupling strategy makes sense?", "methods_and_evaluation_criteria": "The proposed methods (introduction of personalized components) and evaluation criteria (average accuracy) align well with the problem of personalized federated learning for VLMs.", "theoretical_claims": "The theoretical part focuses on the definition of the proposed architecture, providing clear insights into the framework.", "experimental_designs_or_analyses": "[strength] Extensive experimental validation: This work provides a comprehensive performance evaluation through experiments conducted on an extensive set of scenarios, including a convincing ablation study.", "relation_to_broader_scientific_literature": "The authors categorize state-of-the-art methods such as FedOTP [1], FedPGP [2] as decoupled methods, where the prompt is divided into global and local parts. This paper builds upon these approaches by addressing their limitations, particularly in relation to the vision modality. Its main contribution is to fill in the neglect of vision modality in previous works.\n\nReferences:\\\n[1] Global and local prompts cooperation via optimal transport for federated learning. Li et al. CVPR 2024.\\\n[2] Harmonizing generalization and personalization in federated prompt learning. Cui et al. ICML 2024.", "essential_references_not_discussed": "The relevant references are comprehensive.", "other_strengths_and_weaknesses": "[strength] Crucial challenge: This paper addresses a crucial challenge of personalized adaptation in federated applications of VLMs to downstream tasks, which require fine-tuning with diverse data sources. \n\n[strength] Clear presentations: The figures are well-presented, making it easy to follow the proposed method. Additionally, the charts are clear and effective in illustrating the results.\n\n[weakness] Unclear rationale for specific technologies: While the paper discusses various parameter-efficient fine-tuning technologies such as prompts, adapters, and LoRa, it lacks a clear explanation of why a specific technology is chosen for a particular modality. The rationale behind these choices should be made clearer, particularly in Section 2.1.\n\n[weakness] Writing issues: Some minor writing issues need attention. For example, in Line 81, a space is missing in “… prior knowledge.II) Visual Feature Diversity …”.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check the weakness mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQpgrvd0Wg", "reviewer": "Reviewer_UBUx", "summary": "This paper proposes FedDDA, a federated framework designed to enhance the adaptation of Vision-Language Models under data heterogeneity. Toward existing works, the authors highlight two key limitations in both textual and visual modality: Textual Property Loss and Visual Feature Diversity. To address these challenges, this work proposes Textual Prior Decoupling (TPD), which learns fine-grained representations. Moreover, it proposes Visual Dynamic Adaptation (VDA) to optimize the visual space to align with the textual space. The method is evaluated across four classification tasks, demonstrating robustness and generalizability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UBUx:\n\nThank you for your positive response! We value the constructive feedback and have carefully considered the concerns raised. Below we provide detailed responses to address the reviewer's concerns:\n\n**Weakness & Question**\n\n**W1 & Q1: The rationale for employing dual adapters should be clarified.**\n\n**A1:**  We apologize for the confusion. In the vision branch, we tend to introduce the adapter architecture to reshape image features. We observe that both global and local adapters contribute to optimizing the visual feature space, thereby enhancing vision-language alignment. Building on this insight, we explore their combination (dual adapters) to enable mutual reinforcement between consensus and style knowledge. This design takes full account of the nature of federated learning, helping to preserve the generalization capability while improving personalized representation.\n\n**W2 & Q2:  Why does the performance on DomainNet appear more promising compared to the other three datasets?**\n\n**A2:**  Thank you for your detailed observation. Compared to the other three datasets, DomainNet is significantly larger and exhibits greater domain diversity. This characteristic poses a greater challenge of visual feature diversity, which is the primary focus of our study. With targeted visual components (dual adapters with gate), our approach is more compatible with handling such complex scenarios compared to other methods. This is clearly demonstrated by its superior performance on DomainNet. \n\n**W3 & Q3:  The empirical record on the associated costs.**\n\n**A3:** Thank you for this valuable suggestion. We measure the communication cost by the size of uploaded data per round per client, and computation cost by the number of floating point operations (FLOPs) in the same batch. Below we present a comparison of both communication and computation costs.\n\nTable 2. Comparison of communication cost (KB) and computation cost (G).\n\n| Method                    | Communication Cost  (KB) | Computation Cost (G) |\n| ------------------------- | :----------------------- | -------------------- |\n| PromptFL                  | 16.00                    | 1160.33              |\n| PromptFL+Prox ($\\mu=0.5$) | 16.00                    | 1160.33              |\n| FedOTP                    | 16.00                    | 1247.02              |\n| FedPGP                    | 16.00                    | 1323.26              |\n| FedDDA (ours)             | 272.00                   | 1160.38              |\n\nThe introduction of the general adapter in our method adds more global parameters, inevitably leading to an increased communication burden. Meanwhile, the local updates in our method follow the standard CLIP loss and involve only one combined prompt that needs to be encoded. As a result, the overall computational cost of our approach remains comparable to that of PromptFL.", "claims_and_evidence": "W1. The proposal of dual adapters is confusing. I can understand the need to reshape visual features for visual diversity. However, the rationale for employing both global and local adapters is unclear.", "methods_and_evaluation_criteria": "This paper introduces personalized components from both modalities, which make sense for adapting VLM to downstream tasks in a federated setting. Additionally, the average accuracy effectively demonstrates the effectiveness of the proposed method.", "theoretical_claims": "It’s easy to follow the methods with precise theoretical definitions and clear pipelines.", "experimental_designs_or_analyses": "S1. The paper provides extensive empirical validation through experiments on multiple scenarios.\n\nW2. The results on DomainNet, especially in the Quickdraw domain, are notably promising compared to the other three datasets. The authors should clarify the reasons behind these results.\n\nW3. Reducing the computational and communication demands is the foundational goal of FedPEFT. In Section 3.4, the authors only discuss the increased burden from a conceptual standpoint. An analysis backed by empirical results is expected.", "relation_to_broader_scientific_literature": "The core idea of existing personalized works [1][2][3] is to decouple prompts into independent components with specific properties (e.g., global and local). This paper figures out two key limitations of these works and designs targeted modules to tackle.\n\n[1] Global and local prompts cooperation via optimal transport for federated learning, CVPR2024 \n\n[2] Harmonizing generalization and personalization in federated prompt learning, ICML2024\n\n[3] Diprompt: Disentangled prompt tuning for multiple latent domain generalization in federated learning, CVPR2024", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "S2. This paper addresses the challenge of data heterogeneity in the downstream adaptation of VLM under federated settings, which are crucial for applications in the real world. It facilitates personalized adaptation using distributed data while preserving privacy.\n\nS3. The paper is well-structured and mostly clearly written. The authors divide the problem into two parts based on both modalities, presenting a clear flow that enhances comprehension.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I am willing to consider raising the score based on your rebuttal to the following questions:\n\nQ1. The rationale for employing dual adapters should be clarified.\n\nQ2. Why does the performance on DomainNet appear more promising compared to the other three datasets?\n\nQ3. Reducing the computational and communication demands is the foundational goal of FedPEFT. Can the authors provide more empirical details on the associated costs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received four accept ratings and the reviews are mostly positive without serious flaws. Reviewer Zexp found the method novel and the results comprehensive, but questioned the intuition behind some design choices. The rebuttal addressed most of these questions and the reviewer raised the score from weak reject to weak accept. Reviewer oran also appreciated the method and results but pointed out that the baselines compared in tables 1 and 2 are not the latest. The AC has checked the tables and found that the paper has included two latest works in the comparison, which are FedOTP (CVPR'24) and FedPGP (ICML'24). Reviewers 287K and UBUx were satisfied with the results and writing, but raised concerns about the rationale behind the design of using learnable prompt for the text modality while adapter for the vision modality. The rebuttal had addressed these concerns. Overall, the reviewers found this work interesting and the results sufficient for justifying the effectiveness of the method. The AC recommends that the paper be accepted.
Decision: Accept (poster)