Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
Authors: Linh Tran
Wei Sun
Stacy Patterson
Ana Milanova
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Federated Prompt Learning
Personalization
Differential Privacy
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a differentially private approach for federated prompt learning using multimodal large language model by leveraging low-rank factorization to balance personalization, generalization and privacy.

Abstract: Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

Published as a conference paper at ICLR 2025
PRIVACY-PRESERVING PERSONALIZED FEDERATED PROMPT LEARNING FOR MULTIMODAL LARGE LANGUAGE MODELS
Linh Tran
Wei Sun
Stacy Patterson"
Ana Milanova
Rensselaer Polytechnic Institute
IBM Research
ABSTRACT
Multimodal Large Language Models (LLMs) are pivotal revolutionizing cus- tomer support and operations by integrating multiple modalities such text, im- ages, and audio. Federated Prompt Learning (FPL) is a recently proposed ap- proach that combines pre-trained multimodal L.LMs such vision-language mod- els with federated learning to create personalized, privacy-preserving Al systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalizatte can lead to overfit- ting, reducing generalizability, while stringent privacy measures, such differen- tial privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for introduce novel method where ap-
personalization. To ensure privacy. ply local differential privacy to the two low-rank components of the local prompt, and global differential privacy the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.
INTRODUCTION
In recent years, there has been rapid advancement in multimodal large language models (LLMs) that integrate multiple modality information, including text, images, audio, and video, enhance the comprehension and generation capabilities. Vision-Language Models (VLMs) such as CLIP (Radford al., (02T) are a variant of multimodal LLMs that learn transferable image and text representations, making them highly effective in applications such image captioning and visual search. One proposed setting for deploying VLMs a Federated Learning framework that allows multiple organizations clients collaboratively train global model without directly sharing their local training data. However, fine-tuning pre-trained VLMs in a FL. system time-consuming and resource-intensiv given the massive number of parameters each VLM has. This gives rise to Federated Prompt Learning (FPL) which only fine-tunes the soft prompt embedding while freezing the rest of the VLM model parameters in the FL system (Guo al. 2023b; Cui al., 2024). In FPL, each client fine-tunes their customized prompt using their local data and shares the prompt with a central server for generalization purposes. The clients can distribute their fine-tuned prompts as prompt providers to public users who wish to perform downstream inference tasks, also known as the prompt as a service (PaaS) paradigm (Wu et al. 2024 Yao et
2024 Huang al.. 2023).
One significant challenge in such distributed systems is the presence of data heterogeneity. i.e., orga- nizations often have non-identical and non-independent (non-lID) data distributions, which can vary widely due to factors such as demographics, usage patterns, or device capabilities. To address this, personalized FL has emerged to tailor models to the unique data characteristics of each client rather than solely improving global model. Personalized FL focuses on learning customized models for each client, reflecting the heterogeneity of their data (He et al. 2020; Dinh et al.] 2020). In the context of personalized FPI., the goal is for each client to learn and utilize personalized prompts that

---

Published as a conference paper at ICLR 2025
better align with their specific data and application needs. Nevertheless, over-personalization can lead to local data overfitting, preventing the model to generalize well on non non-training data. Clients in an FPL framework may opt to distribute their customized prompts post training to public users for downstream tasks. However, these users may have different types of inputs that the client's prompt is not well generalized to, resulting in suboptimal performance. Consequently, it is crucial to achieve a nuanced balance between personalization and generalization in heterogeneous FPL system.
In addition to balancing the tradeoff between personalization and generalization, privacy poses an- other critical concern in FPL, especially in sensitive domains such as finance, law, and healthcare. In the PaaS framework, the distributed trained prompts are shown to be susceptible Membership In- ference Attack (MIA), potentially exposing details about individual clients' training data (Wu et al.) 2024). To address this issue, one may consider Differential Privacy (DP) (Dwork et al.. 2014) which ensures an adversary cannot reliably detect the presence or absence of a data sample based on the output information. However, balancing personalization and privacy under data heterogeneity is a challenging task. The non-IID nature of the data allows clients to better learn their personalized prompt, but it amplifies the performance degradation caused by DP due to the high data sensitivity, impairing both personalization and generalization capabilities. Thus, the key question we aim to address is: How can we effectively balance personalization, generalization, and privacy in a data heterogeneous FPL system?
To tackle the above question, we proposed a Differentially Private Federated Prompt Learning (DP- FPL) approach that leverages low-rank factorization and DP as part of the prompt learning process. In our framework, each client simultancously learns global prompt and a local prompt. The global prompt is shared in FL manner for generalized knowledge transfer, while the local prompt is retained at each client site for personalization. Our contributions are threefold.
We propose a srivacy-preserving personalized federated prompt learning approach with Differential Privacy for multimodal LLMs. We factorize the local prompt into two lower rank components with an additional residual term. The factorized low-rank components allow the model to capture broader patterns that are beneficial across different data distri- butions, aiding the generalization capability of each client. The residual term is crucial for retaining the expressiveness lost during the factorization process, thereby preserving the client-specific learning and improving personalization.
We preserve privacy by utilizing both Global Differential Privacy (GDP) and Local Differ- ential Privacy (LDP). Unlike conventional methods that apply noise uniformly to the entire prompt, we judiciously apply LDP to the two low-rank components of the local prompt, and GDP to the global prompt. Our privacy mechanism mitigates the effect of DP noise on model performance while preserving the privacy guarantee post training.
We conduct extensive experiments on widely adopted datasets to evaluate our proposed method against other benchmarks. The experimental results demonstrate superior per- formance of our proposed method in balancing personalization and generalization while mitigating the model degradation caused by DP noise.
RELATED WORK
Personalized Federated Learning. There are several existing approaches that aim learn per-
sonalized models for clients in FL settings, including clustering (Ghosh et al.] 2020 Berlo et al.,
2020 Shahid et al. 2021), regularization (Shoham et al. [2019] Dinh al. 22200] Li et al.) 2020 and knowledge Iaston (Li Wang. 2019; et al. 2020 Fang & Ye, 2022). Personalized FL is most commonly approached as multi-task learning problem that simultaneously learns two models for each client: a global model for generalized knowledge and a local model for personalized data. Existing methods accomplish this by decoupling the model parameters or layers into global and lo- cal learning components ( (Arivazhagan et al.] 2019 Deng et al., 2020 Zhang et al., 2020 Collins cet al., 2021 Jeong & Hwang 2022 Zhang et al. 2023). In the existing literature on personalized FL, private multi-task learning approaches aim to protect training data by retaining personalized parameters and sharing differentially private generalized parameters. Examples include Jain et al. and
(2021), Hu et al. 2202)) Bietti et al. (2022) Yang et al. (2023b) and Xu et al. (2024). However, these methods are designed for full model training and cannot be directly applied to prompt tuning due to the difference in the parameter space. [Sun et al. (2024) incorporates Low-Rank Adaptation with DP
2

---

Published as a conference paper at ICLR 2025
in standard FL setting, but they do not consider personalization and prompt learning, making their method not applicable to our setting.
Federated Prompt Learning. Recent advances in personalized FPL have garnered significant at-
tention Guo et al. 2023a Li et al. 2023 Yang et al. 2023a Sun et al. 2023 Deng et al. [024] Li et al. 2024: Cui et al. 2024). See Table for comparisons. With the exception Zhao et al. (023) which introduces a privacy-preservinn
FPL method that leverages DP to protect the under- lynn private data, none of the prior literature considers the privacy issue. Many of these works require modification to the backbone model, which is not relevant to our approach as we want to protect the personalized prompt, not the model. ZZaoo et al. (2023) does not account for the crucial aspects of personalization. Similar to our work, CCu et al. (2024) also factorizes the local prompt into two learnable low-rank components for balancing personalization and generalization. We stead have the learnable full-rank local prompt, and only keep the low-rank terms non-permanent for generalization with an additional residual retain the expressiveness for personalization.
Table 1: Recent Federated Prompt Learning algorithms
FPL Algorithm
Consider personalization
No model
Adopt low-rank factorization
Provide privacy guarantee
modification
PFedPrompt (Guo et al.
2023a
X
PromptFL Guo et al. 20236
X
PFedPT [L
2023
X
PFedPG Yang et 2023a
Fedperfix (un ct al
20233
X
SGPT [Deng et al.
2024
X
X
X
FedOTP (1.
2024
X
X
FedPGP Cui
2(14
X
Fedprompt (Zhao et al.
(0033)
DP-FPL (ours)
PROPOSED METHOD
We introduce our proposed method, Differentially Private Federated Prompt Learning (DP-FPL), shown in Figure I Our approach leverages low-rank factorization with additional residual term to balance personalization and generalization a differentially private FPL system.
3.1 PRELIMINARIES ON PERSONALIZED FEDERATED PROMPT LEARNING
Our system follows a standard FPL setting that consists of a set of N clients and a central server. Let the global dataset be
[D1, D2...., Dw}, each client i holds a local subset D, of ni samples. Each client local model involves frozen pre-trained VLM such learner, and their goal is to learn the representation between the visual and prompt information to improve multimodal classification tasks. Specifically, the frozen CLIP model involves text encoder f(.) and an image encoder g(-) that respectively transform the prompt and an image æ into text and image features. The prompt learner trains soft prompt p; for client that is optimized align with the visual features. Using cos[:, to denote the cosine similarity used by CLIP model, the classification prediction probability for each client computed as:
CLIP model and prompt
(ep(cos(f(p.cc).g
p = j|æ)
EC 1 fxp(cos(f((p...))) (x)
where ò denotes the predicted label, ey denotes label j out C number of classes, and denotes the temperature parameter of CLIP. The client personalized prompt pı is optimized with cross-entropy loss:
L
Σ
Σ y log p j(æ).
Di
(x.0)ED.
3

---

Published as a conference paper at ICLR 2025
PG
((x)
CUP
Promot Learner
Client
PG
Server
Prompt Learner
8(x)
Client
Pr
May
P2
FN
PN
Prompt Learner
Client
Differential
Local
privacy
Pa.
Pa.
Paly
prompts
Learnable
Frozen
Figure 1: Architecture of DP-FPI. with frozen CLIP models. Each client i trains global prompt PG, and local prompt pr,i. The local prompt is factorized at each training iteration as PL,; = uy24 + Ti-
In a data heterogeneity setting, each client's local dataset D; is drawn from a distinct data distri- butions. The difference among clients' data can lead to the drift problem, where the local model converges toward local solutions optimal for their specific data but fails align with the global model objective. To address this challenge, various personalized FPL solutions such as clustering, local fine-tuning and knowledge distillation have been proposed. Our work focuses the multi-task learning approach that aims to learn two models for each client: one for generalization and one for personalization. In particular, we separate each client's prompt p; into global prompt pc,, and the local prompt Pr.j- The global prompt Pc.; is shared and aggregated to improve the global learning, while the local prompt PL, retained the client level for personalized learning.
A simple example of the personalized prompt is illustrated in Figure [ in which each client has collection of images captured from different angles, reflecting the heterogeneity in their local data. Consequently, client might use the prompt "an upside-down photo of [class]", while client i could have the prompt "a upright photo of [class]", and client might use rotated photo of [class]". In this instance, the template "a photo of [class]" is the generalized global prompt shared across clients, and the terms "upside-down". "upright" and "rotated" represent the personalized characteristics of each client's 's prompt. These variations in prompts allow the client model to better adapt to the specific characteristics of their data, improving personalization.
The training process of FPL over T iterations is structured as follow. For each global training round f, each client i initializes the global prompt Pt. + PG 4-1 and the local prompt Př.,; + -t- Client then trains their personalized prompt pt using their local private data and obtains the cross-entropy loss L. At the end of the local training round, client i updates their local prompt PÝ., + Pi.. - ML L.L and sends the gradient w.r.t the global prompt VG,L to the server for aggregation. The server computes the average gradient VG + ΣN VG, and updates the new global prompt to be pla + 1-1 VG. The learning objective function of FPL system is:
N
min
LD. ビー= nj
(3)
PG
where LD. is the loss computed on dataset D, of client i.
4

---

Published as a conference paper at ICLR 2025
3.2 BALANCING PERSONALIZATION AND GENERALIZATION
Prior research has demonstrated that fine tuning pre-trained LLMs with lower dimension reparam- eterization promotes generalization capability across various tasks (Aghajanyan et al. 2020 Cui et al. 2024). Additionally, lower intrinsic dimension improves the model utility under the effect of DP noise (Yu et al., 2021 Xu al., 2024). Therefore, we utilize low-rank factorization as part of our FPL. framework to balance the personalization and generalization learning under the influence of DP noise. However, low-rank training such as Low-Rank Adaptation has difficulty matching the performance of full-rank training in many difficult tasks (Liu et al., 2024; Biderman et al. 2024] Ivison et al.] 2023 Zhuo et al. 2024). This is because low-rank factorization methods restrict the parameter space, removing some of the information of the full-rank space (Konečný) (016). Recent work Cui et al. (2024) also used the low-rank approximation in non non-private FPL systems, however, they factorize the local prompt only once at the beginning and train iteratively with the low-rank components. This approach can reduce the overall expressive power over the training phase. As we will show in Section 4 the loss of expressiveness can amplify the adverse effects of the DP noise, further diminishing the model's performance.
To overcome these issues of low-rank training, we perform the factorization process in every training round rather than only at the beginning like Cui et al. (2024), and we incorporate a residual term to compensate for the lost expressiveness. We analyze in detail the benefit of the residual term in Section 3.3 The overall personalized prompt of client i can then be expressed as = PG, + u,2; + rs where u, and 0 are the factorized low-rank components and r; the additional residual term.
We utilize a version of the Reparametrized Gradient Perturbation (RGP) method introduced in Yu al. (2021) as our low-rank factorization scheme. We perform the factorization process for each client local prompt pr. in every training iteration to get the temporary low-rank prompt components U, and U, We also compute a residual term r; which the remainder of the factorization process, i.e. Pr.i u,yy As shown by Yu et al. (202 1) each client can reconstruct the gradient of pr.,
using the gradient of the low-rank terms in back propagation as follows:
VL,C = (V (VaC)e + us (V.C) inu? (V VuCye
(4)
We note that the residual r; is used as part of the forward process, but it is not involved in the full-rank local prompt recomputation.
3.3 PRESERVING PRIVACY
In the personalized FPL framework, each client a prompt provider may distribute the trained customized prompt to public users for downstream inference tasks. These fully trained prompts, not properly protected with privacy mechanism, are vulnerable to MIA which aims to infer if an image sample was used for training or not (Wu et al., 2024). Moreover, the shared global prompt may leak information about the private data during the training process as contains the gradient information of the loss function.
We assume that clients and server are honest. We consider adversary to be potential user with access to a trained customized prompt obtained from FPL client. The adversary also has access to the publicly available pre-trained CLIP model for downstream inference purposes. The goal of the adversary is to infer whether an image data was part of the target FPL client's 's training data utilizing the MIA. In this setting, the target client's 's trained prompt p; shared with a potential adversarial user and is susceptible to privacy breach, so it is necessary to protect ps with the DP mechanism.
One may straightforwardly apply privacy noise the client's trained Pa before distributing it to public users. However, this requires a large DP noise to effectively prevent MIA (Wu et al.) 2024]. On the other hand, injecting noise gradually over the training phase allows for more control over the influence of noise on the model, leading to better utility (Abadi et al. (016). In our setting, perform gradient updates on the global prompt PG. and local prompt PL, of each client, so we add DP noise to the gradients w.r.t these two terms in each training step.
As part of the FPL procedure, the global prompt of each client PG, is aggregated and averaged by the server, while the client's local prompt PL stays locally. The final trained prompt P; is composed of the synchronized global prompt Pc: and the local prompt Pr,,i- To provide privacy guarantee, we use Global Differential Privacy (GDP) for the global prompt and Local Differential Privacy (LDP)
5

---

Published as a conference paper at ICLR 2025
for the local prompt. This is unconventional way of introducing DP noise to selective parts of the prompt, unlike a vanilla method that directly adds noise to the whole prompt before publishing it. We define these two DP notions follow.
Definition 3.1. Global Differential Privacy (GDP) A randomized mechanism M D + R with domain D and range satisfies (c, 5 for any two adjacent datasets D, D' D (i.e., datasets that differ in exactly one sample) and for any subset of outputs S E R it holds that
Pr(M(D) S) & PP((M(D)) +)+
+
Definition 3.2. Local Differential Privacy (LDP) A randomized mechanism . D - R with domain D and range R satisfies (c, 6) 3-LDP if for any two adjacent samples x, x' E D where D E DD and for any subset of outputs E R it holds that
Pr(M(a)) E S] S e <P(()((a)) E E +
We apply GDP the global prompt PG, at the server because the impact of the GDP noise on the model utility is much smaller compared to LDP (Arachchige et al. 2019). To provide privacy guar- antee for the local prompt PL, which is not shared with the server for aggregation, each client needs to obfuscate PL, locally with LDP noise. However, directly applying LDP noise to the full-rank local prompt PL.: can heavily impair the model performance (Yu al. 2021; Xu et al. 2024). In addition, the local prompt information is predominantly captured within the low-rank components as a result the factorization process. Therefore, we inject LDP noise only to the low-rank compo- nent un and v; to mitigate the negative effect of privacy noise, while still effectively protecting the local prompt during the recomputation process. We note that we do not add noise to the residual component TV because it is not used for the local prompt recomputation.
We achieve GDP and LDP using the Gaussian noise mechanism. Given function f: D - R, the Gaussian noise mechanism M is defined
M(d) A f(d) + 00)
(5)
where (00 22) is the normal distribution with mean 0 and variance o². The standard deviation is typically chosen based on the function f's sensitivity Sj to satisfy an (e, oDD guarantee.
The two main building blocks in our approach, i.e., low-rank factorization and DP, naturally intro- duce error to the training process. We conjecture that this error acts as a regularization term that prevents clients from overfitting local data, reducing personalization and improving generaliza- tion. However, under strictly private conditions (lower rank and higher DP noise), the accumulated error may become too large and potentially destroy the personalization capability. In this case, the added residual term compensates for the regularization-lik error and helps improve local learning, balancing personalization and generalization. We demonstrate the benefit of the residual term in the ablation study in Section 4.3 supporting our hypothesis. Further theoretical analysis of the residual term can be a potential future work direction.
3.4 ALGORITIIM
We now ready to describe proposed method in detail, as shown in Algorithm !
At the initial stage, the server randomizes a starting global prompt P& and each client sets up their starting local prompt Př., (line 1). The variances ag and σr, are chosen satisfy certain (c, 5)-LP and (c, 5),D) guarantee. The algorithm runs for T iterations. In each iteration each client updates P'c, be the previously aggregated p--1 and PÝ., to be the previous pf., (line 4). A minibatch 3 is sampled from the local dataset D; for training (line 5).
Each client performs parameter factorization using the power method with rank & (line 6). Lines describe the forward pass where each client runs their local frozen CLIP model to get the text features and image feature, and uses them to calculate the loss L using Equation 2 Each client then computes and clips the gradient w.r.t the two low-rank prompt components V, and v.L with threshold Ct., and add local DP noise with standard deviation dz according to 5 (lines 9 - 10).
The noisy gradient w.r.t the local prompt Vi,iL can be reconstructed from the noisy gradients V.L and V.LL using 4 4 and then is updated accordingly (lines 11 11-12). Each client also computes

---

Published as a conference paper at ICLR 2025
Algorithm DP-FPL
Initialize: p pI. for 1...N. N, variances og and og
2: for fort.e +- 1... do
3: for 4 N parallel do
4: Initialize plc, t Pg and PL.: + Pí.;
5:
Sample minibatch B from D,.
6:
Compute low-rank components u4 U. +- Factorize(p2..:1 k).
7:
Obtain text features (p'c.,) f(u,v; + +r)) and image feature g(x) (a E B³).
8:
Calculate loss L according to Equation 2 and compute gradients V.L and V.C. Clip gradients VG,C. C and v.C with threshold Cth-
9:
10:
Add local DP noise: V.C + v.C + (O of.) and V V.C V.C+N(0,0f)
11:
Recompute noisy gradient w.r.t Vi,if using VuL and v according to equation Update local prompt PL.: + PL, n2 L2,f.
12:
Send V, the server.
14:
end for
15: Server computes average gradient Va f- 1 N Va,C.
N
16: Server adds global DP noise to VG <- VG +N0,o)) oz).
17: Server updates global prompt p'g t PG nGVG.
18:
end for
the gradient w.r.t the global prompt VG,.C C and sends it to the server for aggregation (line 13). Upon receiving the locally computed gradients, the server computes the average gradient and adds global DP noise with standard deviation og perturb the gradient (lines 15 16). The server then updates the new global prompt for the next training round (line 17).
Low-rank factorization via traditional SVD method requires significant runtime. Instead, we use the power method with one iteration, significantly reducing the computational cost. Given the full-rank matrix of size m X n (assuming m S n)), the computational cost of SVD scales with O(m2n), while the power iteration only scales with (kmm) where k is the reduced rank and k < m.
3.5 PRIVACY ANALYSIS
DP has several properties and compositions that make it easier to analyze the privacy budget in repetitive algorithms such as machine learning, where the privacy loss accumulates across multiple training iterations. When DP mechanisms are applied repeatedly to the same dataset, the overall privacy budget accumulates sequentially using the advanced composition theorem (Dwork et al. 2014]. Conversely, when DP mechanisms are applied independently to disjoint subsets of a dataset, the overall privacy loss does not accumulate. In this case, the privacy guarantee remains bounded by the maximum privacy loss of any subset using parallel composition (Dwork et al.) 2014). The privacy budget term of LDP and GDP of Algorithm D is given by the following theorem.
Theorem 3.3. There exist constants C1, C2 so that given the number of global rounds T, for any > 0, DP-FPL satisfies (c, 8DDL and (c, 6DD if we choose σ and OG as following:
SLyTTog(L/))
OG 2 C2 So ScVTlog(1//)
01. 2 C1
where St. = Cap. and SG N [3
Cca are the local and global sensitivity respectively.
Proof By definition, single application of the Gaussian noise mechanism satisfies (c, 8)DDP if we choose σ > S (122og(1.55/8) where S is the sensitivity. Under the advanced composition theorem of DP, the Gaussian noise mechanism after T training steps results in an accumulated privacy loss of 6) Thus, to achieve (e, 8D-D, one would need choose
Sy T log((/0)
(6)
for the Gaussian noise mechanism where d' is constant.

---

Published as a conference paper at ICLR 2025
Applying Equation 6 can choose Tg c1 St. SLy/Tr. log(1 log//1
where is constant to make Al- gorithm I satisfy OL,D with respect to each client. Since each client operates the Gaussian noise mechanism independently on disjoint local subset D; of the global dataset D, the release of all clients' noisy mechanism output still satisfies (c, 8D-LDP by the parallel composition of DP. Sim-
Soy log(1/6)
ilarly, by choosing oG 2 02
according to Equation DP-FPL satisfies (c, 8D-
We proved in the theorem above that Algorithm satisfies (e, LDDP and (e, 6DP when pub- lishing the customized prompt potential users. Since the GDP noise is added to the aggregated gradient, the distribution of the aggregated gradient to all clients also satisfies (c,8)-GDP by the post-pree processnn property of DP. According to Theorem 3.3 we can calculate the standard deviations rr and og to achieve certain (e, 5)-LDP and (c, (D, The pair e, d can be chosen to match desired MIA accuracy rate, preferably lower than random guess (50%) (Thudi et al. 2022).
EXPERIMENTS
4.1 SETUP
Datasets. We select four visual classification datasets to investigate the task of balancing person- alization, generalization and privacy: Caltech 101 (Fei-Fei et al. 2004), OxfordPets (Parkhi et al.] 2012), OxfordFlowers (Nilsback & Zisserman 2008) and Food101 (Bossard et al. 2014). We utilize the pathological data split among 10 clients. Each client model trained on its local classes, and evaluated on both its local classes for personalization capability and neighbor classes (classes owned by other clients) for generalization capability. We also evaluate the large-scale dataset CIFAR-100 (Krizhevsky al. 2009) with the Dirichlet data split among 25 and 50 clients. The final test accu- racy is obtained by averaging the performance across all clients. Details of our implementation and hyperparameters are provided in the appendix.
Table 2: Mean test accuracy on local classes averaged across 10 clients. The baseline FedPGP and our method DP-FPL have factorization on the local prompt with rank 8.
Dataset
Noise
PromptFL
FedOIF
FedPGP
DP-FPL
None
94.45-10.30
07.06:10.48
Caltech
0.4
82.53-00552
05.09=0.50
95.12-+0.46
95.74=0063
101
0.2
81.52:10.45
87.61
90.98:2:0.00
05.28:00.66
0.1
80.42:10.65
84.98.1.339
80.01:10.339
92.71: 0.38
0.05
1.39
83.61
77.24:1.0338 38
87.64:1..79
0.01
78.52: 1.68
08.86-20.38
77.23-20.37
0.82
None
76.85:40096
99.63=0.2
94.66+20.31
96.1=
Oxford Pets
0.4
74.36:1:..6
0..03:00.33
86.56:1-0.69
95.13: 0.52
0.2
13.16.10.16
65.97 0.889
67.11.10.44
93.09 100911000 0.43
0.1
72.77:00 16
0.558
63.21:10062
85, 05.25
0.05
52.39-+0.53
1.097
57.98-00.97
81.26= 1.10
0.01
43.68:10.67
14.08
1.33
73.71
Nome
84.04:10.32
97.84 1.16
79.11:10.45
1..75:00.2
Oxford Flowers
0.4
60.31:1
19,89
77.13-1..52
80.09 1.41
0.2
40.33-10.83
65.961 0.96
70.77400.1
76.75 1.05
38.25+11.33
42.31=0.771
1.54
72.11= 1.37
0.05
37.18
18.69-00.66
39.52:10.77
69.80 1.34
0.01
36.11
35.23.10.64
$1.55
None
86.50 26
0..65
84.40-10..99
86 06.08:20.12
Food
0.4
78.70:10.39
79.45=0.23
1.55
81.45=00211
71.44:0991
77.36:10.41
17.72-+1..5
0.12
0.1
69.00-10.40
70.48 1.339
75.18:10.22
80.57 0.44
0.05
68.36-11.23
1.29
13.72-41.04
78.23--
0.01
1..47=1.20
54,70=1.12
71. 1.
77.45=00.40
Baselines. To demonstrate the effectiveness of proposed method in balancing personalization, generalization and privacy, we compare with three baseline cases: (1) PromptFL (Guo et al. 2023b),
(2) FedOTP (Li et al.] 2024) and (3) FedPGP (Cui et al. 2024). More details of the baselines are listed the appendix.
Privacy levels. We consider different noise levels for LDP and GDP: {0.01, 0.05, 0.1, 0.2, 0.4). We pick 5 = 10 and the clipping threshold Cta = 10. We provide details of how the noise is added to each baseline in the appendix.
8

---

Published as a conference paper at ICLR 2025
4.2 PERFORMANCE RESULTS
Improving personalization in private setting, Table 2 shows the average test accuracy over the last 10 epochs on local classes. In the non-private scenario (i.e., Noise None), FedOTP shows the highest urility, however, as we add more noise, DP-FPI. has the highest local classes accuracy. Under strict privacy levels (e = 0.01), we see a noticeable decrease in test accuracy. Nevertheless, DP-FPL still consistently outperforms other baselines across all datasets, showing the robustness of our method even under strictly private conditions. Table shows the overall accuracy utiliz- ing Dirichlet data split and ResNet50 the backbone model. As shown in Table [ DP-FPL demonstrates superior performance compared to baseline methods across all datasets. This shows the effectiveness and applicability of our approach in various complex settings, including different data distributions, models, and number of clients.
Privacy improves generalization. T shows the average accuracy over the last 10 epochs on neighbor classes. Similar to local classes, DP-FPL exhibits the highest utility in neighbor classes under the presence DP noise. As expected, the accuracy degrade for higher noise levels, however, we see an improvement in Caltech 101 utility when we increase privacy level from 0.4 to 0.1. This is because privacy noise act as a form of regularization that prevents overfitting, and hence improving generalization. Nevertheless, when the noise level is large enough (e {0.01, 0.05)), the overall utility is degraded and the neighbor accuracy no longer improves.
Additional results. We include additional experiment on the performance of MIA against DP-FPI. in the appendix (see Section (A.3). The results show that the attack success rate is relatively low when 0.1 for all datasets. In addition, 0.1 causes less than 10% reduction in the target model accuracy for both local and neighbor classes as shown in Tables and
Table 3: Mean test accuracy neighbor classes averaged across 10 clients. The baseline FedPGP and our method DP-FPL have factorization on the local prompt with rank 8.
Dataset
Noise e
PromptF1.
FedOTP
FedPGP
DP-PPL
None
92.88
74.91=0.30
93.44:00.5
01.3420.155
Caltech
0.1
82.66:11.22
84.26=0.84
88.24440.78
88.58:10.38
101
0.2
1.49
84.08.0.98
86.05-100799
89.72::098
0.1
80.83:20.72
78.91
79.31+:0.22
90.02:+1.47
79.96:10.40
1.03
75.62:10.82
82.76:::.12
0.01
78.89.10.33
73.54 1.00
75.60.10.82
80060 0.57
None
76.34:10.37
65.6320.14
89.71:0.49
0..19=00.85
Oxford Pets
0.4
7..43:10.87
60.95110881
72.54:10.54
81.82 0.42
0.2
73.74:10.82
59.7420.84
61.68:20.46
00.67:
73.12:10.39
08.8310.93
09.79:10.79
0..2
0.05
52.44
$4.08
11.63-11001
74.13 0.48
0.01
38.27400.6
53.63+0.91
40.20:+:0.42
Nome
69.44.1061
38.29 1.09
75.79.1 0.87
69.51 0.45
Oxford
0.4
48.03-1...69
56.65
65.9 63.93311..8 86
67.67=10.45
Flowers
0.2
18-19770.97
5..44=0..0
63.49:2:0.5
0..51=0.58
0.1
37.76-1 1.23
37.53-1
46.24-111880
66.44
0.05
38.78:41.15 1.18
35.48=11000
18.27
56.75 1.85
0.01
34.81=+1.19 19
31.26:#0.96
34.72-+1..0
43.21 1.72
None
86. 16.19:0.11
84.03220.33
86.23-00006
86.08
Food 101
76.88:10.23
80.11=00.47
78.94=11007
01.00::0.25
0.2
70.99:10.83
76.44:10.2
77.21:10.90
0.27
0.1
67.80=11.59
13.12+1..6
76.92.40.83
78.14=0.53
0.05
66.76:11.27
71.82+0.79
13.61=11.35
77.18:0.50
0.01
61.2.11.99
67.08
72.99
76.87 0.62
4.3 ABLATION STUDY
In this subsection, investigate the efficacy of the key parameters that directly affect the tradeoff between personalization, generalization, and privacy: residual term, noise level and rank value. The results for 101 are presented in Figure 2 we include other dataset results in the appendix.
Effect of residual term. We investigate the effectiveness of the residual term by separately test- ing the model without the residual component in Figure 2 We note that this setting is different from FedPGP because the factorization process is performed every training round instead of at the beginning. In Figure? 2 we observe better performance in both local and neighbor classes when

---

Published as a conference paper at ICLR 2025
Table Mean accuracy averaged across all clients under Dirichlet data distribution. The base- line FedPGP and our method DP-FPL have factorization on the local prompt with rank 8.
Noise
CIFAR-IOO 100
25
CIFAR-I00 with clients
PromptFL
FedO1P
FddPGP
DP-PPL
PromptLL
FedOTP
FedPGP
DP-FPL
None
71 2.200:..0
68.1320.33
71.54:0.17
09.8420.19
71.30:10.10
68.57=0.14
70.8240.12
69.48:0.11 1I
0.1
53.69:10.53
47.22=0.97
58.8740.335
66.23-0.233
55.44:20.31
58.23=0.42
58.43=0.51
66.40:20.18
33.02:1:0.99
47.02:41.13
56.75:10.11
62.92::0.30
54.65:1:0229
57.09:0.40
56.34:10.4
64.39:1:.29
0.1
50.86.10.68
1.095.1..06
55.76.10.33
59.53 0.25
53.17.1. 1.06
57.05
53.1910.488
601.49 0.226
0.05
50.20:1:0.0
1.16
53.80+-0.30
57.97-0..16
55.00-40.37
52.38-10.49
0.01
50.65+00.41
44,27=1.17
51.75+0034
53,31=0,35
11,92t1155
54,53=00229
52.32=0044
0..09
incorporating the residual term. The difference in accuracy more prominent under strict privacy conditions (rank and 0.01), confirming conjecture about the benefit of the residual de- scribed in Section 3.3 Lower rank and higher noise significantly reduce the overall utility due to the large accumulated error, and the residual term plays crucial role in improving local learning. balancing personalization and generalization.
(a) Local Acc, w/o Residual
(b) Local Acc. w/ Residual
(c) Neighbbrr Acc. wdo Residual (d) Neighbor Acc. wl Residual
120
35
20
0.4
Text
0.1
0.03
rast
rask 3
rasa
rank
fall
rast
rasi 2
rane A
rank
fall
rasa
rack
rask
rack B
fut rank
rast
raskk
mank
rasic
fat rank
Figure 2: Test accuracy of ablation study on noise level, rank and residual term for Caltech101
Effect of privacy noise We study how different privacy level affects the model performance both local and neighbor classes in Figure (b) and (c). As expected, the local classes accuracy gradually decreases when noise level increases. However, we some unexpected improvement in neighbor utility when we increase DP parameter from 0.4 to 0.1. As explained in Section 4.2 certain privacy noise range like regularization error that prevents overfitting, resulting in better generalization for higher noise level.
Effect of factorization rank. rank values with full-rank setting in Figure[((b and (c). Intuitively, higher rank values lead to better performance, however, this is not always the case. For local classes, rank generally performs better than full-rank under higher noise levels (e {0.01, 0.1)). Lower-rank has fewer entries, hence the amount noise added is less than full-rank for the same level of privacy guarantee, which leads to better accuracy. For neighbor classes, lower rank value tends to have higher utility because lower rank introduces more regularization eиror that prevents overfitting and improves generalization.
We explore the impact the factorization rank by comparing different
CONCLUSION
In this paper, we presented novel approach address the critical challenges of personalization, generalization, and privacy in FPL for multimodal LLMs. Our proposed framework leverages low- rank factorization balance the tradeoffs between these competing objectives. By factorizing the local prompts into low-rank components iteratively while incorporating residual term, our method effectively preserves both generalization and personalization. Moreover, we introduced a privacy- preserving mechanism that applies both global and local DP to safeguard sensitive client data. Un- like conventional methods, we selectively applied DP noise to the low-rank components, allowing maintain privacy without significantly degrading model performance. The critical role of the residual term in mitigating the effects of DP noise was also demonstrated. highlighting its impor- tance for maintaining model expressiveness and personalization.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the IBM through the IBM-Rensselaer Future of Computing Research Collaboration, and by the NSF grant CNS-2232061.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '1, pp. 308-318. Association for Computing Machinery, 2016.
Armen Aghajanyan, Luke Zettlemyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning, arXiv preprint wrXiv22012.13255, 2020.
Pathum Chamikara Mahawaga Arachchige, Peter Bertok, Ibrahim Khalil, Dongxi Liu, Seyit Camtepe, and Mohammed Atiquzzaman. Local differential privacy for deep learning. JEEE Internet of Things Journal, 7(7):5827-5842, 2019. 2019.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Fed- erated learning with personalization layers. arXiv preprint 1rXXiv1912.00811 2019.
Bram Berlo, Aaqib Saced, and Tanir Ozcelebi. Towards federated unsupervised representation learn- ing. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking, EdgeSys '20, pp. 31-36, New York, NY, USA, 2020. Association for Computing Machinery.
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673 2024.
Alberto Bietti, Chen Yu Wei, Miroslav Dudik, John Langford, and Steven Wu. Personalization im- proves privacy-accuracy tradeoffs in federated learning. In International Conference on Machine Learming, pp. 1945-1962 PMLR, 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- mining diseriminative com- ponents with random forests. In European Conference on Computer Vision, 2014.
Liam Collins, Hamed Hassani. Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Ma- chine Learning Research, pp. 2089-2099, PMLR, 18-24 Jul 2021.
Tianyu Cui, Hongxia Li, Jingya Wang. and Ye Shi. Harmonizing generalization and personalization in federated prompt learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning PMLR, 21-27 Jul 2024.
Research, pp.
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6087-6097, 2024.
Yuyang Deng, Mohammad Mahdi Kamani, and Mchrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. /3461, 2020.
Canh Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21394-21405, Curran Associates, Inc., 2020.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /929, 2020.
11

---

Published as a conference paper at ICLR 2025
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ Theoretical Computer Science, 4):211)1177,
Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10072- 10081, 2022.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training exam- ples: An incremental bayesian approach tested 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19586-19597. Curran Associates, Inc., 2020.
Tao Guo, Song Guo, and Junxiao Wang. Pfedprompt: language models in federated learning. In Proceedings of the ACM Web Conference 2023, pp- 1364-1374, 2023a.
Learning personalized prompt for vision-
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of mnodels-federatee learning in age of foundation model. JEEE Transactions on Mobile Computing, 2023b.
Chaoyang He, Murali Annavaram, and Salman Avestimehr: Group knowledge transfer: Federated learning of large cnns the edge. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14068-14080. Curran Associates, Inc., 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778. 2016.
Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Private multi-task learning: Formulation and applications to federated learning. arXiv preprint arXiv:2108. 12978, 2021.
Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Prompt backdoors in visual prompt learning. arXiv preprint drXiv:2310.07632, 2023.
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradcep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing cli- mate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10700 2023.
Prateek Jain, John Rush, Adam Smith, Shuang Song, and Abhradeep Guha Thakurta. Differentially private model personalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp- 29723-29735. Curran Associates, Inc., 2021.
Wonyong Jeong and Sung Ju Hwang. Factorized-fi fl: Personalized federated learning with parameter factorization &amp; similarity matching. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 35684 356955. Curran Associates, Inc., 2022.
Jakub Konečný. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:19/0.03581, 2019.
Guanghao Li, Wansen Wu, Yan Sun, Li Shen, Baoyuan Wu, and Dacheng Tao. Visual prompt based personalized federated learning. arXiv preprint arXiv:2303.0867 03868888. 2023.
12

---

Published as a conference paper at ICLR 2025
Hongxia Li, Wei Huang, Jingya Wang. and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning, arXiv preprint (xXiv:24033000411, 2024.
Tian Li, Anit Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In I. Dhillon, D. Papaillipoouls, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 429-450, 2020.
Shih- Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.0993: 2024.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 722-729, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Osama Shahid, Seyedamin Pouriyeh, Reza M Parizi, Quan Z Sheng, Gautam Srivastava, and Liang Zhao. Communication efficiency in federated learning: Achievements and challenges. arXiv preprint arXiv:2 2021.
Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07796. 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. Fedperfix: Towards partial model personalization of vision transformers in federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4988-4995 2023.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in learning. arXiv preprint 2r31v:2403.12311, 2024.
federated
Anvith Thudi, Ilia Shumailov, Franziska Boenisch, and Nicolas Papernot. Bounding membership inference. arXiv preprint urXiv:2202. 12232, 2022.
Yixin Wu, Rui Wen, Michael Backes, Pascal Berrang, Mathias Humbert, Yun Shen, and Yang Zhang- Quantifying Privacy Risks of Prompts in Visual Prompt Learning. In USENIX Security Sympo- sium (USENIX Security). USENIX, 2024.
Jie Xu, Karthikeyan Saravanan, Rogier van Dalen, Haaris Mehmood, David Tuckey, and Mete Ozay. Dp-dylora: Fine-tuning wansformer-basee models on-device under differentially private federated learning using dynamic low-rank adaptation. arXiv preprint 240.::2200500666. 2024.
Fu-En Yang, Chien-Ye Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in feder- ated learning via client-specifft prompt generation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19159-19168, 2023a.
Xiyuan Yang, Wenke Huang, and Mang Ye. Dynamic personalized federated learning with adap- tive differential privacy. Advances Neural Information Processing Systems, 36:72181-72192, 2023b.
Hongwei Yao. Jian Lou, Zhan Qin, and Kui Ren. Promptcare: Prompt copyright protection by watermark injection and verification. In 2024 IEEE Symposium on Security and Privacy (SP), pp- 845-861, 2024.
13

---

Published as a conference paper at ICLR 2025
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low- rank reparametrization. In Interattional Conference on Machine Learning, pp. 12208-12211 PMLR, 2021.
Jianqing Zhang, Yang Hua, Hao Wang. Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedep: Separating feature information for personalized federated learning via conditional policy. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23, pp. 2249-3261, New York, NY, USA, 2023. Association for Computing Machinery.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized feder- ated learning with first order model optimization. arXiv preprint arXiv:2012.08556 2020.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication- prompt tuning in federated learning. In ICASSP 2023-2023 IEEE
efficient and International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Teryy Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu,, and Niklas Muennighoff. Astraios: Parameter-efficient instruction tuning code large language models. arXiv preprint arXiv:2401.00788, 2024.
A EXPERIMENTAL DETAILS
A.I DATASETS
Caltech101 (Fei-Fei al. 2004) is image dataset that contains images from 101 object cate- gories (e.g., "helicopter", "elephant" and "chair" etc.). There are about 40 to 800 images for each object category, but most categories have about 50 images. The dataset is available for down- load on http: www. vision caltech. edu/ Image DbjectCategories tar gz) The dataset contains 6, 593 samples, including 128 training samples and 2, 465 testing samples.
Oxford Pets (Parkhi et 2012) is an image dataset with 37 object classes that can downloaded at futhtps:: www. r robots ox..xxoooo acx.acc.. uk vvgg dat a ptt.//ttsssseeeeee /aataaaaaaaaa/eaaaae /attaa/mmmaass.a tar. gz The dataset consists 6, 613 pet images with roughly 200 images for each class. The dataset is divided into training set of 2, 944 images and test set of 3, 669 images.
Oxford Flowers Nilsback Zisserman 2008) is an image classification dataset consisting 102 flower categories, each class has between 40 and 258 images. The dataset is can be retrieved at https: //wp://wwwww/./.m/ rowots...ooooo..oo...o...... ox. ac uk vvgg data/ fww.wrswwwr/////oo
tgz
There are 6, 556 total images, including 093 training images and 2, 463 testing images.
Food101 (Bossard et al. 2014) is a large-scale dataset containing images of 101 different types of available for download at https:: / data. vi iin. ee ethz ch./ cvi/datasets. ext ra/ foo/doooddo11 There are 80, 800 total images, and we split the dataset into train set of size 50, 500 and test of size 300.
food. The dataset
CIFAR-100 (Krizheskky et al. 2009) is a another large-scale dataset containing im- ages of 100 different object classes. The dataset is are available for download via torchvision dat asets. CIFAR10 The dataset consists of 60, 000 32x32 images, with 000 images per class. We divide the dataset into training of 50, 000 images and test set of 10, 000 images.
A.2 IMPLEMENTATION DETAILS
For the first four datasets Caltech101, OxfordPets, OxfordFlowers and Food101, we use the Vi- sion Transformer ViT-B 16 (Dosovitskiy) 2020) as the backbone for the frozen CLIP model for Cal- tech101, OxfordPets, OxfordFlowers and Food101. For each dataset, we run experiments with N 10 clients for T 100 global training rounds. We use batch size [B] = 32 for training
and [ = 100 for testing. We set the global learning rate na = 0.0001 and local learning rate Mr. 0.0001 with SGD optimizer.
14

---

Published as a conference paper at ICLR 2025
We adopt the Pathological setting for data heterogeneity among clients implemented in https: //github. com/ Kai httpaaaggngnog//////aaaaaaaa wwwloo/mmainnnii DwW.AATAATTTTSSSS mal The class labels are splitted randomly among 10 clients without overlapping, and each client owns disjoint set of local classes. For each client, we train the local model on data associated with their assigned local classes. We then evaluate each client's local model and neighbor class test set for generalization. The local class test set involve all the test image associated with the client's local class labels. The neighbor class test is the set of all test images whose labels are owned by other clients.
test sets: local class test set for personalization
For CIFAR-100, we adopt ResNet50 (He et al. 2016) the backbone model and run experiments with N = 25 and N 50 clients for T' = 200 global training rounds. We use batch size [B] = 32 for training and [3] 100 for testing- We set the global learning rate ` 0.0001 and local learning rate 1) 0.0001 with SGD optimizer. We use Dirichlet data distribution to simulate the real-world non-III setting with parameter 0.3. The test accuracy is averaged across all clients.
We provide summary of the experiment set up in Table beloow.
Table 5: Details of datasets and experimental set up
Dataset
Data distribution
Model
Number of clients
Number training rounds
Training batch size
Testing batch size
Caltech101
Pathological
ViT-B16
100
32
100
OxfordPets
Pathological
ViT-B16
10
100
32
100
OxfordFlowers
Pathological
ViT-B16
10
100
32
100
Food101
Pathological
VIT-B16
10
100
32
100
CIFAR-100
Dirichlet
ResNet50
25,
200
32
100
For the prompt learner, the length of prompt vectors is 6 = 16 with dimension of d = 512, and the token position is "end" with "random" initialization. For the factorization process, we experiment with four different factorization rank 2, 4, 8. We consider three different DP noises with privacy level from low to high: e E f0.4, 0.2, 0.13. The clipping threshold is chosen to be C. 10 for both GDP and LDP applications.
For the baseline methods, we consider three settings: PromptFI. (Guo al. 2023b), FedOTP (Li et al. (2024) and FedPGP (Cui et al.] (024) with the main difference lies in the prompt learner structure, We describe each baseline in detail below.
1. PromptFL: PromptFL follows the traditional federated learning framework where each client has single prompt ps and the aggregated prompt is the average of all clients' pi- In this setting, the privacy noise added directly to each client's 's before sharing with the server for aggregation.
2. FedOTP: In FedOTP, each client's customized prompt p: involves two full-rank global prompt Pc.; and local prompt PL,ú. To incorporate privacy, the GDP noise added to the averaged pG by the server and the LDP noise is added to each client's local prompt Pt,5-
3. FedPGP: In FedPGP, each client's customized prompt p: includes full-rank global prompt PG.; and two low-rank local components U, Us. Each client then train three pa- rameters PG,ú. U, and v, across the training process. In this baseline, the GDP noise is added to the averaged PG by the server and the LDP noise added to the two low-rank terms Ui Uú of each client.
We run our experiment on a computer cluster, each node has 6x NVIDIA Tesla V100 GPUs with 32 GiB of memory and 512 GiB RAM and 2x IBM Power 9 processors. The final result is the mean accuracy across all clients, averaged over 5 runs with different seeds.
A.3 MEMBERSHIP INFERENCE ATTACK
In this section, evaluate the -peeeccpreeeevingg performance of our method DP-FPL against Membership Inference Attack (MIA). We implement MIA as described Sn al. (2017). where the goal of the attack is to infer the appearance of data samples in one client's training dataset. We
15

---

Published as a conference paper at ICLR 2025
first train set of 50 shadow models with the same model architecture as the target client's model. The shadow models generate synthetic training data that is used to train the attack model. The attack model is two-layer MLP and classification head that predicts if given sample is part of the target client's training data or not. We run the attack on Caltech101 Oxford Pets and Oxford Flowers with different privacy levels. For each dataset, we perform separate attack on each class and compute the average success rate, i.c. percentage of correct guesses.
120
100
180
30
10
800
Accrrcy
50
Accurscy
60
60
40
baoeine
ortee__ptss
20
oeford_petts
20
enford_petss
oxford ficeers
coford foowrrs
rafort flowers
(a) Target local accuracy
(b) Target neighhor accuracy
(c) MTA accuracy
Figure 3: Target model performance (a, b) and MIA performance (c) with rank The baseline of MIA accuracy is set to 50% (random guessing).
Bigune[ddmmonsrraat the performance of the target model (local and neighbor elasses) and the MIA. We set the MIA baseline accuracy to be 50%, representing the expected success rate of random guessing. Looking at Figure 3cl the MIA accuracy is low (< 50%) when 0.2 for Oxford Pets and e = 0.1 for Caltech101 1 and Oxford Flowers. In addition, = 0.1 causes minimal loss in the target model accuracy for both local and neighbor classes (Figures 3al and (3b). Therefore, one can balance the utility-privacy tradeoff by setting 0.1 for any dataset. This shows that our approach effectively protects the training data from MIA while still maintaining good model performance.
A.4 ADDITIONAL. EXPERIMENTAL RESULTS
In this section, we include additional results from the experiments introduced in Section to further investigate how the DP parameter e, factorization rank and the residual term affect the tradeoff between personalization, generalization and privacy guarantee. Figures 5 and 6 continue the ablation study on the effect of the key parameters that directly affect the tradeoff: residual term, noise level and rank. We summarize the results for each dataset below.
Figure 4 shows the test accuracy with different parameter settings for Oxford Pets. In general, there an increase in both local and neighbor classes when incorporating the residual term, highlighting the benefit of the residual in model utility in different datasets. The difference in accuracy is more consistent across different noise and rank values compared Figure growth in accuracy when we increase the rank value. Therefore, it is beneficial to set any rank value for this particular dataset.
2 In addition, there is minimal
1.0 (a) Local Acc. w/o Residual
(b) Local Acc. w/ Residual
(c) Neighbor Acc. wño Residual (d) Neighbor Acc. Residual
task
rank
2
tank
tark
Rask
tonk
rasi
Raah
rask
rark
Figure 4: Test accuracy of ablation study on noise level, rank and residual term for Oxford Pets
Figure 5 shows the test accuracy with different parameter settings for Oxford Flowers. The addition of the residual term still improves the overall utility, though the benefit is modest for this dataset. Similar to Oxford Pets, the rank value does not significantly affect the accuracy for both local and neighbor classes. We also observe more drastic drop in accuracy under strong privacy level (e
16

---

Published as a conference paper at ICLR 2025
0.01). Overall, it is more difficult to balance the tradeoff for Oxford Flowers. One needs to sacrifice strong data protection and set é 2 0.1 to achieve good personalization and generalization utility.
(a) Local Acc. wio Residual
(b) Local Acc. w/ Residuall
(c) Neighbor Acc. wio Residual (d) Neighbor Acc. we Residual
50
Thit
0.03
rbal 3
rasd
fall
rask
rack
radk
rack
rack
fut
rask
rack
rasi
luii
Figure 5: Test accuracy of ablation study noise level, rank and residual term for Oxford Flowers
Figure shows the test accuracy with different parameter settings for F0od101. We see an overall increase in local and neighbor accuracy with the introduction of the residual term, and the difference is more significant under higher noise level. In addition, there is minimal reduction in model utility when we increase privacy noise. This behavior consistent across all rank values, indicating the robustness of our method under strict privacy constraints.
(a) Local Acc. w/o Residual
(b) Local Acc. Residual
(c) Neighbor Acc. wéo Residual (d) Neighbor Acc. Residual
1.00
ronk
raok
reck 1
rank
rask
rack
rack
rank
rank
ranki
renk
rask
rank
Figure Test accuracy of ablation study on noise level, rank and residual term for Food101
Overall, the affect of the key parameters (noise level, rank and residual term) the tradeoff between personalization, generalization and privacy varies widely among different datasets. Nonetheless, the results across all datasets show consistent trends, indicating the effectiveness and applicability of our method.
For completeness, include Tables 8 and which detail the accuracy with standard deviation of all the ablation experiments demonstrated in Figures 2 and 6
Table 6: Mean test accuracy of our method averaged across 10 clients under non-private setting (without any DP noise).
Dataset
Neighbor classes
without residual
with residual
without resicual
with residual
89.70.10.81
93.40.10.28
Caltech101
0.83
94.51:10.17
87.95-10.42
92.79-10072
39.57-10.47
95.69:10.57
86.86:10072
92.32:10071
92.12-10.47
96.06-10.39
87.58-10.11
91.54:10.73
89.78:10.71
93.46:10633
89.54:10.71
92.17:10.27
Oxford
88.24.10.73
87.37.10.33
31.28-1-0.66 64
96.36:10.28
38.70-1-0883
87.61-10.53
02.42:1-0.8
96.91-10.58
90.71=t:1009
80.19:100.64
67.41:10119
75.25:10.20
67. 67.22:10.63
71.49:10.71
Oxford Flowers
66. 66.188:00.89 89
78.56:10..0
66. 10::-::0000 10
69.76.t:1.11
67.04:10.61
81.53:10.49
07.0221-0.9
68.33-10.32
68.35.10.61
15.75
68..5.1.0.47
69.51 0.61
74.66.1 1.
86.12.10.38
74.58110.51
85,97 1..72
Food101
74.91
86.06:10.27
74.96-10.49
85.89-10.52
74.95=t0.91
86.18:100.63
74.84:1:0.99
86.16:::1.06
76.95:11.02
86.08:10.69
17

---

Published as a conference paper at ICLR 2025
Table 7: Mean test accuracy of DP-FPL averaged across 10 clients. The DP noise is set to e = 0.4.
Dataset
Rank
Local classes
Neighbor classes
without residial
with resiciaal
without resiciua
with residual
87.66:0.799
92.93-1.188
90.19:11111
92.94:1:668
Caltech101
38.98=11002
93,03=00.90
86.99+11.44
88.82-10.99
88.01 :8.011::.211
93.58
85.20:1:100
89.15:10.90
9..13:11.19
95.74
85.46.11.08
$8.58-10.36
74.87:1:0.6
90.43:10.79
72.41:10.49
87.83:10.95
Oxford Pets
74.66:2:0.88
91.67:10.22
72.69-40033
86.26:2:0.42
74.24:10.74
93.22:10.40
71.83+40.39
83,53-1-0.0
77.22.10.82
95.13:1 0.552
73.86.10.42
81.82 0.42
62.488:::.83
75.45:00555
64.41100.58
71.49:10.55
Oxford Flowers
65.67 1.096
77.39 0.66
64.53.100.75
70.97 0.58
66.23-1:0799
63.97410.77
68.10:10.67
68.03-11.41
80.09
63.22:1:0.88
67.67-10.45
$0.55=11005
69.76:10:43
50.29 14
68.11:20.94
Food101
7..65-10.66
57.10-11.13
71.15-10.66
61.02.1 1.09
77.76.10.98
61.25.10.58
76.48 00.57
69.53-10.20
1..4
67.52:40.40
$1.00:10.90
Table 8: Mean test accuracy of DP-FPL averaged across 10 clients. The DP noise is set to 0.2.
Dataset
Rank
Local classes
Neighbor classes with residual
without residual
with residual
without residual
87.79:10.811
92.16:10.38
90.43:100:1
91.61::1.02
Caltech101
88.99 1.063
93.38 0.82
87.22.10.47
90.55.10.49
89.25:1:061
94.22:10.54
85.31110.28
90.73:10.10
91.75.10.19
95.28 10.93
86.56.10.71
89.72.10.83
70.06:::0733
09.79:10.99
73.27::0.51
87.69:21.008
Oxford Pets
72.71:10.44 :10.42
90.12:10.49 0.49
72.04-10.39
86.74:10.18
72.27:10084
9..45.10.65
70.16:10.77
83.62.10.94
73.73-10.94
93.09-10.30
71.04-10.55
$0.67:10.69
55. 18.18.10.92
72.19.10.17
64.53-10.81
70.35-10.97
Oxford Flowers
59.52:1:0.53
73.58:10.57
61.41:10.11
70.50:10.29
63.89
75.18 0.57
61.50.10.81
68.65 0.46
33.86.100.68
76.75-10.97
62. 62.65:10..65
67.51:10.73
58.1110.337
Food101
59.22.400.1 16
70.52:10.75
58.1740..57
67.64:10.38
12:10.025
75.11:10.46
64.74410.44
70.21:10.80
70.16:10.32
81.25.1.511
70.21
80.79.10.17
Table 9: Mean test accuracy of DP-FPL averaged across 10 clients. The DP noise is set to e 0.1.
Dataset
Rank
Local classss
Neighbur classes
without
84.73:10.32
91.39:10.58
89.35:10.85
90.81:1-0.99
Caltech101
85.32-10.38
86.32-1...28
89.18:10.67
87.82.10.81
92.26.11.28
84.73-10.97
87.92-10.80
$8.82-10.88
92.71:10.11
85.32:40.86
90.02-1-0.88
67.59.10.19
$0.41 0.38
71.47.1..2
79.24.1061
Oxford Pets
69:22:1:0.5
81.65
67.38:1:0277
79.65:10.59
69.35-10.89
82.73-10.49
66.11+40.66
78.73-10.87
74.48-10091
85.25-10.30
66. 10:1:049
77.12-10.63
54.18:10.65
1.02
59.51
66.93.10.49
Oxford Flowers
57.16-10.33
66.21:10.99
65.35-10.64
65.21-10.91
59.71
67.32:10.27
67.714:0.41
67.26:2:0.32
59.81:10.97
72.11::1.05
66.98-10.38
66.44:10.17
62.75:10.37
68.65:10.81
58.05:10.42
68.60-100.8
Food10]
62.43.10..6
7184.10.95
38.86.10.22
69.66.10.93
64.66:1-0.75
75.64:10.41
22:1-0288
72.61-10.30
67.20:1:0.8
80.57-20.83
67.06:+:0.99
78.14:20.37
18

---

Published as a conference paper at ICLR 2025
Table 10: Mean test accuracy of DP-FPL averaged across 10 clients. The DP noise is set to 0.05.
Dataset
Rank
Local classes
Neighbor classes
without residual
with residual
without residual
with residual
78.62.1:.22
$1.59.11.55
78.75.10.23
79.07:11.59
Caltech101
78.63 18
85.42:10.33
77.67::0.3
77.42:1092
78.57=00488
85.05-+11.6
77.51+0048
82.28:+1.07
78.72:100.49
87.64:10.98
76.83:10044
82.76:11.69
63.28:10.57
70.60:11.30
61.75::0.58
72.18:11.32
Oxford Pets
63.32:4:0.68
73.66:10.94
61.76:10.70
72.62:+1..9
64.26:20.91
1..2
61.71##0.91
68.73:+1.31
05.22:400.89
61.69:1:0.22
74.13-10.99
32.98=1.115 13
59.62:10.99
29.91=:0.61
46.80=10.91
Oxford Flowers
32.84-+1.80
59.74:11.05
30.02:+:1144
49.19-1:1.47
32.45:1:1.15
65.72:11.17
30.40:1:0.75
50.32:11.18
33.09
69.80-10.94
30.21111.28
56.75
58.72:10036
56.37:1::25
67.09:10.46
Food101
57.01=00.41
67.71=1442
35.59700.77
67.27=0057
59.03-1:0:45
68.13::1.06
55,45:10.19
69.44:10.55
64.88-10.37
78.23
63.15.00.55
77.18
Table 11: Mean test accuracy of DP-FPL averaged across 10 clients. The DP noise is set 0.01.
Dataset
Rank
Local classes
Neighhor classes
withour residual
with rosidual
without residual
with rosidual
73.19:10.25
$0.89 1.662
82.59.10.22
87.98 1.8.111.5 1..39
Caltech101
75.18:10.50
83.08 1.16
02.60.10..85
85.70.10.79
78. 18.199:00448
84.35-11.07
80.61+10448
$2.81:11.07
83. 13.16:00060
85.21:22.27
77.59:10.57
80.60:::.12
53.05-10.59
68.73:11.29
18.54:11.115
65.35-11.3
Oxford Poss
53.04=11337
71662=1.10
49.54=11.40
67.58-+1.10
53.06:10091
73.14
50.52:11.35
70.39 1.36
14.29=+1.01
1.191
1..48
71.89:::.44
32.44-11.07
39.54-1.997
30.40-10.90
31.24-1 L08
Oxford Flowers
33.11=11.84
42.53-11.78
30.94-11.33
38.32:1-1.34
33.08=11.13
45.27=11.23
30.91+11.03
41.24:::1.16
33.15
$1.55
30.95.11.880
42.31 1.85
101
57. 57,40-10.43
1..28-::.40
56.70=21.02
68.48-10..0
58.98:10.55
69.13:11105
59.39:1-0.00
69.88:1-0.77
59.63-10.54
77.45
59.58-10.73
76.87:10.55
19


Reviews:
[{"review_id": "4DD7jV2fk7", "reviewer": "Reviewer_BrD2", "summary": "This paper proposes a Differentially Private Federated Prompt Learning (DP-FPL) system for multimodal large language models to address the critical need of balancing the competing goals of personalization, generalization, and privacy-preserving that are involved in serving those models. By decoupling the globally learnable prompt from the personal learnable prompt, decomposing the local prompt with LoRA with an additional residual term, and employing differential privacy techniques in both local prompt learning and global prompt aggregation, DP-FPL achieves accuracy gains while meeting specific privacy budget in a distributed and heterogenous data setting.", "questions": "Are there more results on more clients, large datasets, and other models beside ViT-B16? Any theoretical justification of why the residual term preserves the expressiveness of the local prompt?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation of the paper is well stated, and writing is easily understood. The problem the paper tries to address is certainly important and practical in multimodal LLM deployment. The experiment setting is clear, and results support the claim the authors make.", "weaknesses": "While the problems the paper tries to address are certainly important, the paper does not provide significant scientific insight nor rigorous analysis that can help solve them in broad situations. Instead, a specific set of existing techniques are shown to be effective in a narrow setting (i.e., small-scale datasets and one model architecture). Though the introduction of the residual term in the local prompt decomposition is empirically demonstrated to be effective in the experimental settings, the paper does not give theoretical justification of why such design is beneficial and how applicable it is when applying to more complex situations (e.g., larger data heterogeneity). The privacy analysis also lacks rigorous proof of why the global gradient aggregation satisfies the DP bound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VoLRXH2TmW", "reviewer": "Reviewer_Xnij", "summary": "This article focuses on the issues of balancing personalization, generalization, and privacy in Federated Prompt Learning, and proposes a Differentially Private Federated Prompt Learning method to solve them, and finally proves the effectiveness of the proposed method in three datasets.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is very important and meaningful to consider privacy in Federated Prompt Learning.\n2. The proposed method is simple and easy to deploy.\n3. Existing methods are discussed and compared in detail.\n3. The privacy guarantee of the proposed method is given.", "weaknesses": "1. Although I appreciate the simplicity of the proposed method, compared with existing methods, it seems that only the residuals and differential privacy are added, which are commonly used strategies, making the novelty insufficient.\n2. Inadequate experiments. First, the baselines are limited, and it is unclear why a range of baseline methods from related work was not used. Secondly, it is unclear about the privacy-preserving performance of the proposed method, such as its performance in the face of membership inference attacks. Furthermore, no significance tests were performed. Finally, the proposed method has a large number of components, such as global prompt, local prompt, GDP, LDP, etc., all of which require ablation experiments.\n3. There are a large number of assertions that have not been confirmed and are all the author's personal conjectures. For example, \"simply applying LoRA can result in loss of expressiveness during training, ...\", \" this approach does not protect the privacy of the prompt data..., Therefore, privacy noise must be incorporated into the gradient updates during each training step for privacy guarantee\", \"the impact of the GDP noise on the model utility is much smaller compared to LDP\".\n4. Lack of complexity discussion.\n5. Minor: What is $\\epsilon$ in Table 4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGNHgNNFY6", "reviewer": "Reviewer_U7Cw", "summary": "The paper presents a novel approach, Differentially Private Federated Prompt Learning (DP-FPL), which aims to balance privacy, personalization, and generalization in multimodal large language models (LLMs) within federated learning frameworks. DP-FPL utilizes a low-rank adaptation (LoRA) technique with differential privacy (DP) to enable prompt tuning at both global and local levels, allowing clients to personalize prompts without directly sharing sensitive data. This method integrates both local and global differential privacy mechanisms, selectively adding noise to low-rank components to preserve model performance while ensuring data privacy.", "questions": "- The model shows a performance trade-off between privacy, personalization, and generalization, yet this is not adequately explored in real-world applications. While the authors apply privacy noise at various levels, the impact on performance for real-world, heterogeneous data remains unclear. An experiment demonstrating this trade-off would strengthen the paper.\n- The paper experiments with moderate privacy levels (ϵ=0.1,0.2,0.4), but it does not examine stricter levels (ϵ<0.1ϵ < 0.1ϵ<0.1) that may be necessary for sensitive applications like healthcare or finance. Including results for ϵ=0.05 and ϵ=0.01 would provide a fuller understanding of the model’s robustness under high privacy demands.\n- The dataset heterogeneity is simulated by randomly assigning class labels to clients, which does not accurately capture real-world, non-IID data distributions. Employing a Dirichlet distribution to split the dataset would more effectively simulate realistic data diversity, enhancing the model's evaluation under practical federated learning conditions.\n- Although the paper notes that DP-FPL degrades gradually as DP noise increases, a clearer breakdown of this effect on both personalization and generalization—specifically for local and neighbor classes—across different datasets would add clarity to the trade-offs involved.\n- Privacy noise appears to improve generalization by reducing overfitting; however, Table 2 shows mixed effects across datasets and noise levels. For instance, generalization to neighbor classes improves with increased privacy noise on Caltech101 but is inconsistent on Oxford Flowers. Analyzing this inconsistency could provide insights into the regularization benefits of DP noise.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper combines local and global differential privacy with federated prompt learning, creating a robust privacy-preserving framework suitable for multimodal LLMs.\n- By using low-rank adaptation with a residual component, DP-FPL achieves a balance between generalization for broader applicability and personalization for client-specific data.\n- The experimental design demonstrates DP-FPL’s effectiveness across multiple datasets and privacy noise levels, providing evidence of the method’s performance under varying conditions.", "weaknesses": "- The evaluation lacks statistical tests to confirm the significance of performance differences, which limits the interpretation of the reported improvements.\n- The exploration of privacy-performance trade-offs is insufficient under practical, real-world conditions. While the paper discusses differential privacy noise’s impact, it does not fully assess performance under stricter privacy constraints.\n- The simulation of data heterogeneity is limited; the paper relies on randomly splitting class labels, which does not capture the complexity of real-world non-IID data distributions effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oOUvHtM5Y2", "reviewer": "Reviewer_BtXp", "summary": "This paper introduces a Differentially Private Federated Prompt Learning (DP-FPL) approach to achieve a performance balance between personalization and generalization in FL setting. Specifically, compared to previous work on this subject, DP-FPL adds a residual term to the low-rank decomposition that achieves better balance between privacy and effectiveness.", "questions": "If I understand correctly, FedPGP in Table 2 in the experiment section applies the original FedPGP with DP, therefore is the same as \"without residual\" in Table 3. Correct? In other words, are the result difference between FedPGP and DP-FPL in Table 2 due to the residual term only ?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is in general well-written and easy to follow.\n2. The effectiveness of proposed framework is testified by comparing with existing works, and improvements are significant.\n3. DP theoretical analysis is performed.", "weaknesses": "1. Since the introduction of DP part is quite standard for privacy preservation, the novelty mostly lies in the introduction of the residual term compared to FedPGP. \n\n2. Although the standard theoretical analysis on DP is provided, it didn't address the impact of the introduction of the residual term. The paper will be enhanced significantly if it can provide some theoretical insights on the impact of the residual on the utility-tradeoff, which will support its experimental results.\n\n3. The experimental baselines are a bit limited. Only two methods are compared, while many federated prompt learning methods have been proposed in the past，such as PromptFL,pFedPrompt,FedOTP etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)