Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Authors: Bikang Pan
Wei Huang
Ye Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Vision-Language Foundation Models
Prompt Learning
Theoretical Analysis
Primary area: learning_theory
TL;DR: This paper constructs a theoretical analysis framework for prompt-based federated learning via feature learning theory, and introduces a prompt portfolio mechanism to address severe data heterogeneity and balance generalization and personalization.

Abstract: Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Bikang Pan ShanghaiTTch University panke202@@ehaggaallccchccco edu.cn
Wei Huang
RIKEN Center for Advanced Intelligence Project wei. huang. vrōriken, jp
Ye Shi
ShanghaiTech University
ahiyy@shanghaitech.. edu. cn
Abstract
Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of Dision-language models employs prompt learning to reduce communication and computational costs, i.c., prompt- based federated learning. However, there is limited theoretical analysis to un- derstand the performance of prompt-based federated learning. In this work, construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance be assessed by the ratio of task-relevant to task-irrelevan coef- ficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance ad- vantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments. Our code is available at: https:://// httes:///tttubttt com Pww.oo///nnaaaagg/aaaaa Ptwppa/w///oooopoooooo.oooooooo. git
Introduction
Federated learning (FL) 33 stands out powerful framework that enables machine learning over decentralized data while maintaining data privacy and reducing reliance on centralized data repositories. Despite its advantages, the intensive computational and communication demands during the training phase often constrain the scalability of the models utilized. A transformative ad adnnemment in this field is the integration of prompt-based learning within the federated framework 42
Prompt learning adapts models such as Contrastive Language- Image Pretraining (CLIP) 38 through
minimal modifications, typically in the form of prompts cues, guiding the model's predictions. As an emerging idea in machine learning, it has shown significant promise in various applications by allowing models to perform specialized tasks without undergoing complete retraining. Prompts effectively adjust pre-trained models for new tasks or datasets, which is particularly valuable in federated environments where data privacy and bandwidth constraints often limit conventional
"Corresponing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

training methods. A prominent example is CoOp 42 with its streamlined federated version known as PromptFL [T7].
Despite the empirical success of prompt-based federated learning 28 17 16 theoretical analysis in this area remains limited. In this paper, use feature learning theory propose analytical framework for prompt-based federated learning with vision-language foundation models. Feature learning theory divides data into task-relevant and task-irrelevant features, allowing learnable weights to be expressed as linear combination of these features. By introducing feature alignment across bimodal pretrained models, we demonstrate the relationship between learnable prompts and the features in latent space. To understand the process of signal learning and noise memorization, we use a two-stage analysis to examine the dynamics of coefficients for relk-relevant and mask-irrelevan, features. Leveraging these coefficients, we demonstrate that prompt learning can be evaluated by comparing the ratio of task-relevant to task-irrelevant coefficients. In this way, we establish a theoretical foundation for prompt-based federated learning.
Additionally, we treat task-relevant coefficients and mask-rrrelvaan coefficients income and risk in investment portfolios (2]. Inspired by investment portfolios, where combining two independent assets can reduce risk, introduce two learnable prompts: a global prompt and local prompt, and simply mix them to form a prompt portfolio to balance generalization and personalization. By mixing them to create prompt portfolio. we balance the generalization and personalization under severe data heterogeneity. Leveraging the analysis framework we proposed, we prove the performance advantage of this mixed algorithm and derive the optimal coefficient. Besides, we use comprehensive experiments to support our theoretical results.
In this paper, our primary contributions are threefold:
We present an analytical framework for prompt-based federated learning using vision- language foundation models. This framework aligns text and image features into shared latent feature space and utilizes a two-stage analysis to understand the dynamics of prompt learning. By tracking the progression of signal learning and noise memorization, we show that the effectiveness of prompt-base based federated learning can be measured by the ratio of task-relevant to task-irrelevant coefficients.
Additionally, we introduce a prompt portfolio mechanism to address severe data hetero- geneity and balance generalization and personalization. Within our proposed analytical framework, we draw an intuitive analogy between task-relevant features and income in portfolio optimization, as well as task-irrelevant features and risk. Consequently, we demon- strate that the combination of prompts performs better than using a single prompt, and we provide the optimal mixing ratio.
The theoretical result has been empirically validated through rigorous experiments. Our results not only align with theoretical predictions but also consistently demonstrate the practical superiority of our approach in real-world scenarios.
2
Related Work
In this section, we examine prior research that serves as the basis for our study. Our investigation is divided into two primary areas: prompt-based federated learning and feature learning theory. Together, these fields create a complete context for our contributions.
2.1 Prompt-based federated learning
Prompt learning, initially developed in the field of natural language processing, has expanded its reach to vision language models. Examples include the CLIP model [38], which originally utilized manually engineered templates. However, more recent advancements have shifted towards developing prompts within a continuous embedding space. Innovations like CoOp 42 have refined the CLIP model by introducing continuous prompt vectors, sparking surge of studies aimed at enhancing the efficiency of prompt learning and providing a solid foundation for further investigation [2]
To enhance the integration of global data and address challenges in scenarios with limited user data, have effectively integrated the concept of prompt learning with To tackle the statistical heterogeneity often found in client data,
FedPrompt [1 and PromptFL. 17
federated learning [ 27 37
23

---

pFedPrompt [16] introduces a non- parametrric approach, providing each client with personalized attention module. This module is designed to refine spatial visual features to better align with the unique characteristics of local data. Concurrently, pFedPG 44) introduces a novel prompt generator located at the server, which customizes prompts for each client, thus enhancing the personalization
of the federated learning process. Additionally, a recent study, FedOTP [28), leverages optimal
transport theory to improve the balance between achieving global consensus and allowing for local customization through a strategic combination of global and local prompts.
However, there is limited theoretical analysis of federated prompt learning. For instance, the theoretical examination of the CLIP model by [8] enhances prompt learning theory by exploring how CLIP learns transferable representations across various modalities and improves zero-shot transfer performance with recently developed regularization technique. However, to the best of our knowledge, no research analyzes prompt learning within a federated setting that elucidates the cooperation between prompts. In this paper, we analyze how different prompts interact with feature learning theory and demonstrate the provable benefits of cooperation between global and local prompts.
2.2 Feature learning theory
Feature learning theory [1 has revolutionized our understanding of how machine learning models learn and represent information. Unlike other theories, feature learning accommodates substantial weight updates during gradient descent, enabling the network to uncover complex patterns in data. Feature learning theory has been successfully applied to various neural network architectures, in- cluding convolutional neural networks 66 (24, graph neural networks [18), and vision transformers
122 26 Moreover, feature learning theory has been used to analyze different training algorithms,, such as Adam
43 momentum (26], ofitt--distiiuutin learning (7), and Mixup 44 10, Notably,
19 have analyzed the convergence and generalization in general federated learning. Despite progress
in feature learning theory, the study of federated prompt learning is sparse. Our work uniquely ad- dresses this gap by analyzing feature learning under prompt-based federated learning, providing crucial insights for their effective adaptation and optimization in such contexts.
3 Preliminary
Notation In our notation, vectors represented by lowercase bold letters, matrices by uppercase bold letters, and scalars by regular, non-bold letters. The fz-norm of a vector v is indicated as \v/2- The spectral norm of a matrix A is denoted by A||, and its Frobenius norm by |A To compare the growth or decline of two sequences, we use standard asymptotic symbols like O(-). o(-), I(-), and (-)), which describe their behavior as they approach infinity. We introduce notations O(-). (2)),, and (-)) to obscure logarithmic factors within these expressions. Notably, 1()) represents the
indicator function. Lastly, we represent sequences of integers as [n] = 21, .., n) and sequences of elements such as vectors can also be represented similarly V(n) {vV. V....., v.).
3.1 Prompt-based federated learning
In this part, we demonstrate how fine-tune a learnable text prompt under vision language pre- trained model. Here, we consider the classification task, where we assume that we have an image x. The objective is to correctly classify the image into class y E [C]. where the total number of classes is C. From the vision language pre-trained model, we expect that the latent spaces of the text encoder and image encoder are aligned. Thus, when we input differrent prompts, the text feature generated by the correct prompt will have the highest similarity with the image feature. We input a learnable prompt p and a fixed class prompt P. {PI.-- .cc. which correspond the classes, into the text encoder h. This process generates the text feature for class c: h h)(P,p). On the other hand, the image feature g is generated by the image encoder g: g = ((x,,)) E RD We define the similarity function between the image feature and the text feature h as p == [pdl = sim(g, he). The training process mirrors traditional classification tasks, where the objective loss E(p, e,y) the distance between the similarity vector and the actual label y. Here, l represents the loss function that measures the distance between two vectors, and e, the one-hot vector derived from the ground truth label y.
3

---

To illustrate the prompt- based federated learning framework, consider a federated system with central server and K clients. Assume client k has ng training samples: {xg,i 1=- The learnable prompt in client k is denoted as pa and the learnable prompt is aggregated at each communication round.
Analysis Framework for Prompt-based Federated Learning
In this subsection, present the analysis framework for prompt-based federated learning from vision-language pre-trained models. The central concept of this framework is the alignment of features between text and image encoders the vision-language pre-trained model. To achieve this, we link the text encoder and image encoder through a shared latent feature space, described as follows.
Feature representation and client distribution Inspired by 6 24 19], we expect that the latent feature space will contain task-relevar and task-irrelevant features. In federated learning settings, the task-relevant features can be categorized into global task-relevant features J4G and local task- relevant features v Us, where S is the length of local task-relevant features. Additionally, the task-irrelevant features can be listed as f..... ÉL, where I, is the length of task-irrelevant features. Here, assume that the dimension of the latent space is m and these features (())) P()) and E(-) are elements of IRmh For simplicity, we assume that these features are orthogonal to each other. In our theoretical examination, we address a binary classification scenario with y/. {+1, We consider a scenario with K clients, each linked to distribution Dr. vk [K]. Initially, we choose the signal vector μt for client k by sampling from P(v1, ./...... vs), where represents a discrete distribution that assigns probabilities to each local telk-relevant feature vector := v/, E [S].
Text encoder Here, suggest coupling the learnable prompt and the class prompt and propose the structure of the text encoder. By adopting a similar setting as (39,, we suppose the generation of the text feature can be written as follows:
b& ; hPp, Pm...) o (Wps WP o Wp: + WPo..:).
(1)
where W RmXmp is the weight matrix, and Pyn.: RRT is the prompt linked ground truth class. In this definition, the introduction of WPB.. introduces nonlinearity between the trainable prompt and the class prompt while keeping the overall function nonlinear. Note that for binary classification problem, the vector Pys, takes P1 when yk, and p-1 when yk. = -1 To reveal the properties of the text encoder in the pre-trained model, we assume that the weight matrix W is composed of the following rows: W [AG, V1,". ,,, ,US Et.
Ms.
The assumption of the weight matrix is inspired by (0), and the evidence of this assumption is listed in the Appendix [ In our analysis framework, we adapt FedAvg [33] as prompt aggregation algorithm, which is named PrompFL [7]. The aggregation formula given by:
K
PG
(++1)
n2
Pc.
(3)
n
1=1
where n Ex nh denotes the total number of data samples across all clients.
Image encoder Let consider the image network, represented as gk. ( E Rm. We assume that the image encoder also aligns the feature space of the text encoder within the pre-trained model. As a result, we assume the image feature generated by data X in client can be expressed as follows:
ge, = g(xxs)) = [yE,,5 O, Out Whis
1) zeros
(S-s) zerus
where Thii N(,,0)), VI E [L] represents the coefficient of task-irrelevant terms in the data, and o3 is the variance. This assumption implies that task-relevant features vary based on whether the label positive or negative, whereas task-irrelevant features persist as arbitrary and unrelated to the label's polarity. The similarity score between an image Xx, and class yko is expressed as

---

sim(gk, ha,)) = (g ha.)). Moreover, the objective of the training loss is designed to enhance the resemblance between the image feature gXx,,;) and the text feature created by the ground truth prompt Py..s"
Signal-noise decomposition Based on the above model, we introduce the signal-noise decomposition of the learnable prompt here. Note that the proofs of the following lemmas and theorems are listed in the appendix.
(1) Lemma 4.1 (Feature Representation), At the t-th iteration, the learnable prompt Pkc for client k and the aggregated prompt (()) can be rewritten linear combination of features and prompt initialization:
11 NG (t) Pk
(8)
(0)
10 k
Pk
k'=1
K
Dd 330 11 PG
(T Pk + TE 2 pus) +
10 (0)
|1
(5)
where a:.: are the coefficients of initialization, T. is the coefficient of global task-relevant features. (f) Y-. is the coefficient of local task-relevwn feutures, d. are the coefficients of task-irrelevant features, and the overlined coefficients are the averaged versions of the original coefficients.
Here, since the learnable prompts can be written linear combination of the features, we can analyze the dynamics of these coefficients to understand the learning progress of the prompts. The normalized factor such as ||uG ||2 2 is used to make the coefficient similar the inner product the prompts and the features, o(( = (P
(t) pG).
Coeflicient dynamics Inspired by previous studies 6 24 19 we employ two-phase analysis to track the dynamics coefficients prompt-based federated learning from vision-language foundation models. By analyzing the dynamics of the coefficients, procedure during training. This two-stage analysis allows us to establish the order coefficients and explore how they are affected by the mixing parameter 0, For the theorem and proof of this analysis, please refer to Appendix F
can obtain the feature learning
Theorem 4.2 (Training Dynamics). There exists a total number of local updates Ti = R,E O(n K Knozoï) such that
(T) O(TKSNRR))
T T(P @(FxASNR)) 0i (Th)) - Vk [K]] (L].
(6)
Here, = En Tu/K K the average number of data in each client, and SNRG - ||MG|/// (ap mm) SNR, (/lpp//m denote the signal-to-noise ratio between the task-relevant feature and featreel We define X = - (Mus, (olss,uw///laa /11
Test performance evaluation with coefficients Here, suppose the classification output of the i-th data client is the class corresponding to the highest similarity between the text feature and image feature, denoted as it. To assess the algorithm's 's performance, we evaluate the error rate in the test procedure as our test loss Lp:
n
(p)
(fie = yk.)).
n
The following theorem demonstrates that the test loss be considered the probability that Gaussian random variable falls below zero, with the mean and variance influenced by the task-relevant and task-irrelevant coefficients.
Theorem 4.3 (Test Loss). The expectation of test loss Lp of an algorithm can be treated as the probability
B[Lo)==P(z <),, 2~ N o²),
(8)
where y and a are functions of task-relevant and (ask-irrelevant coefficients, as defined Appendix
5

---

Drawing from this theorem and the properties of Gaussian distributions, an algorithm's 's performance can be evaluated by the ratio p/o. This ratio highlights the influence task-relevant and task- irreevant features on test loss. Specifically, higher task-relevant coefficient coupled with lower task-irrelevant coefficient typically leads to better performance.
Connection with portfolio optimization The Markowitz mean-variance model is famous frame- work for assembling a portfolio of assets such that the expected return is maximized for a given level of risk 31 [2]. This model characterizes assets by their expected returns and risks. It claims that the proportionally weighted combination of the assets' returns, and the risk of the whole portfolio is a function of the correlations of the component assets. According to the properties of task relevatt and task-irre -irelirreeevant coefficients, the task-relevant coefficient can be directly added, and the task-ire irrelevann feature follows the additive property of Gaussian random variables. Thus, we connect the task-relevant coefficient to the return and the task-irrelevant feature the risk. This connection provides insight that the combination of prompts, i.e., prompt portfolio, will lead higher ratio of task-relevant features to task-irrelevant features.
return of the whole portfolio
5 PromptFolio: Global-Local Prompt Portfolio for Federated Learning
Building on the sigmificant connection between the feature learning process and portfolio optimization, we treat the prompt trained by CoOp and the prompt trained by PromptFL as the two prompt assets and propose simple yet powerful mixing algorithm, PromptFolio For simplicity, we refer to the prompt trained by CoOp as the local prompt pr. and the prompt trained by PromptFI. as the global prompt PG-
Algorithm 1 (PromptFolio) Global-Local Prompt Portfolio
Initialize Pe and Pr.ş for all clients
2: 0
Initialization of the iteration counter
3: while not converged do
4: for each client & in parallel do
5: Send _ to client k. PG. + Pg
(t)
6:
for each sample (Xx,s, yk.s) in client k's data do
7:
Compute gk f (Xx,s) for e t 1 to do Compute bg.s, (1-0) h Pa) + 0- h(PL. P.) Compute similarity Pk.i. (im hu.i,)) end for Update PG PL. by minimizing train loss ((p ey..) end for
8:
9:
11:
13:
Send Pa.k
(++1)
server
end for
16:
Update PG
(++1) + EK na (++1)
Pa.
> FedAvg to aggregate global prompt
+ t
18:
end while
return Po. PA,E for all k
5.1 PromptFolio Method
The local learning process generates the local feature by including specific local prompt, whereas the global learning process adopts a similar strategy but also uses FedAvg to compile learnable prompts from various clients. We enhance cooperation between the local and global learning processes by merging both local and global features create the final text feature. The text feature is produced as follows:
hg.i.c = (1 () - P.) + 8- 1-hh(L. Pc).
(9)
PPromptFolio is pronounced /prempt foulioo/.

---

Signal Decomposition: p
8ppp y 76p
loss
Server
Logits
label y
Task relevant Tast inrelvvnt
Image Feature
Text Feature
Better
Worse
Global Aggregation
Global-Local
local
Pz
Couperation
(1 0³) Pc 0'pr
PG
PG
Pd
Image Encoddr
Text Eneoder
Optimal PrompdFolio
PG
Pg
PG
PX,1
Pr.A
PL.K
global
PLLK
Client
Client
Client
Iinage
Text Prompts
Trainable
Global
Local
The image demonstrates the framework of the PromptFolio algorithm. The algorithm updates the global prompt and local prompt while keeping the weights of the fixed vision-language pretrained model unchanged. Additionally, aggregates the global prompts from each client. The right side of the image intuitively demonstrates the advantages of global-local cooperation for performance when global and local are treated as two assets.
Figure 1:
where 8 E [0. 1] serves as a coefficient balance the mix of the two features, which addresses the balancing between personalization and generalization. The variation in the parameter influences the outcomes of the inference. Specifically, when 0 0, the algorithm reverts to PrompFL [T7]. whereas at 0 = 1, it shifts to CoOp [42). Our approach consists of combining these features and using the resulting mixed feature to determine their similarity. This feature subsequently utilized to evaluate the similarity between text and image features. Note that this algorithm differs from typical
personalized algorithms (30) as it focuses on integrating text features instead of adjusting training weights. The framework of PromptFolio is described in Algorithm I
5.2 Analysis for PromptFolio
In this part, we offer theoretical demonstration of the performance advantage PromptFolio and the selection of the optimal mixing coefficient 6, According to Theorem 4.3 each algorithm can be regarded task-relevant features to task- features. This ratio enables us to analyze the test results of various learning algorithms.
Gaussian random variable. The test performance correlates with the ratio of
Suppose that the coefficients of the prompt via local training at step i are 3k 7h and Φx. and the coefficients of the global prompt at step are Bg Ta and Øx- We define the mean and variance of the Gaussian variable corresponding to the local prompt as μ4 and dg, and the mean and variance of the Gaussian variable corresponding to the global prompt as μ4 and dg- Let p = (11/k) E [0, 1] be the correlation between the Gaussian variables of the local prompt and the global prompt. Here,
define Ht O (T O KSNR. KSNR, and = Ze KSNR& SENRG+xxSNKK, X&RN
O de O(K) the ratio of
Fx result, we have the following theorem, and the proof can be referred Appendix E.
7.+TA
different coefficients. Here, the order of and 6 depends on the coefficient derived in Lemma F3 As
Theorem 5.1 (PromptFolic Advantage). The mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL:
Lp((1 - 0)pa + 0pL) I (1 (1-0)Lo(pc) #LD(PL)
(10)
Ca = (b --) + +
Cs Cr.
A 0
0. proj [0.1]
where
C (a + b)(b3 - 1) 4b(pb
2C
C. = (b l) 1)2 Sat2 1)2

---

The results discussed demonstrate how combining global and local text features enhances performance and illustrate the optimal way balance personalization with generalization. Here, a and are the ratio of coefficients and reveal how the global feature and local feature interact. Drawing on principles from portfolio optimization, which involves blending two assets that are not perfectly correlated, we can construct portfolio that maximizes returns while minimizing risk. Given the characteristics of Gaussian random variables, we intuitively correlate the coefficient of task-relevant features with returns and the coefficient of task-irrelevant features with risk. Thus, the first part of Theorem provides a rationale that a well-balanced portfolio of prompt features can significantly improve performance.
Similar to the portfolio optimization problem, we can also derive the optimal mixing coefficient Theorem 5.2 (Optimal Mixing Coeflicient). The optimal mixing coefficient 0* follows
a pb
0" = pproj [0.1]
(11)
(2+ pb(a
Theoretically, if we further simplify the mixing coefficient with the order of b and p. then we get that
0r 0-= (1=
(K -K-XX - )SNR 1K2-1)(KS
(12)
+ S*
In this theorem, we note that lower Xx. indicates greater data heterogeneity, which lead to a higher 0* This observation aligns with the intuition that, due to the non-i.i.d. distribution of data, the model should incorporate more local information, thereby making the optimal 0 closer to 1.
Experiments
In this section, we conduct experiments with the CLIP model to empirically demonstrate the per- formance advantages of PromptFolio. Specifically, the image network g and the text network h are components of a pre-trained CLIP model. By evaluating results obtained using various mixing coefficients across different datasets, data distribution, and client number, we align theory with practice. We use the Dirichlet distribution manage data heterogeneity and employ FedAvg as the aggregation strategy. The experiment is conducted on the CIFAR-100 dataset by default, with the model trained for epochs locally and the results evaluated over 100 communication rounds.
6.1 Performance evaluation on various datasets
In this section, we observe that the combination of global and local algorithms outperforms both the prompt-based federated learning with FedAvg and individual prompt learning approaches. Under the CLIP model setting, the global and local prompt learning algorithm degenerates to PromptFL and CoOp, respectively. To explore why this global-local collaboration is more effective than either approach alone, we evaluate the accuracy of various mixing coefficients across different datasets. We use CIFAR- 100 (25), DomainNet (36), Office-Caltech10 [15. OxfordPets [35]. and DTD (T]. adopting 0 = 0.2 as the general mixing coefficient for our algorithm. The quantitative results are shown in Table [ From this table, it evident that blending global and local prompts consistently leads to enhanced accuracy, with the accuracy curve also showing peak. Further performance evaluations can be found in Appendix A
Table 1: Accuracy of CoOp, PromptFL, PromptFolio on different datasets.
Cifar100
DomainNet
Office-Call0
OxfordPets
DTD
CoOp
76.88 + 0.07
91.83 + 0.13
97.10 + 0.20
87.85 + 0.32
56.39 + 0..8
PromptFL PromptFolio
78.16 + 0.16
92.72 + 0.16
95.51 = 2.62
88.91 + 0.72
70.99 + 0.32
80.17 0.05
93.04 0.09
97.24 0.11
92.17 = 0.32
71.32 + 0.49
6.2 Performance evaluation under various data heterogeneity
We then conduct the experiment over different data distributions. By varying the parameters of the Dirichlet distribution exponentially from 0.01 to 10, controlled the heterogeneity of the data. A

---

larger a indicates that the data is closer to an i..d. distribution. Using 10 users, we performed our experiments, and the results are shown in Figure aa).
32.
Accuracy
Accuracy
02
72
1:.0
Global
Local
Global
Local
(a)
(b)
The x-axis represents the mixing coefficients, which range from 0 to and the y-axis shows the accuracy of the test set after training. The left figure depicts the result under different data distributions, and the right figure reveals the result under different users.
Figure 2:
From Figure aa,, we observe that hybrid approaches outperform solitary methods, consistent with our theoretical analysis. Additionally, higher values, indicating more uniform distribution, generally result in a superior global model compared to local models tailored for specific users. This finding supports our conclusion that a more IID distribution leads to X being higher and closer to K, resulting in a higher task-relevant to task-irreleaan coefficient ratio, and thus better performance. Conversely, in scenarios where data is highly non-IlI,, there is a preference for using more local models to maintain high accuracy, which aligns with our analysis of the optimal mixing coefficient 0*
6.3 Performance evaluation with diffferent client number
Furthermore, we conducted experiments with different numbers of users on the CIFAR- 100 dataset, keeping the Dirichlet distribution parameter fixed at a 0.01, which represents pathological non-i.i.d. distribution. The number of users varies from 5 100, and the results are shown in figure[[b) From these results, we observe that the trend of the mixing strategy outperforming the independent global and local algorithms remains consistent, regardless number of users increases, the optimal 0 shifts closer to zero, indicating that with more users, each client's s information becomes less sigmificant, necessitating more global information. Additionally, the accuracy first increases and then decreases with the number of users, suggesting that there is an optimal number of users, consistent with theoretical results.
number of users. As
7 Conclusion
This work presents a thorough theoretical and empirical exploration of prompt-based federated learning, integrating langonelanguage foundation models such as CLIP. By developing an analytical framework based on feature learning theory, we have examined the dynamics of signal learning and noise memorization specific federated settings, providing a robust mechanism to evaluate the effectiveness of prompt-based learning strategies. Notably, our introduction of PromptFolio, which combines global and local prompts into prompt portfolio, offers an approach balancing generalization with personalization. drawing innovative parallel with portfolio optimization in

---

finance. This approach balances generalization with personalization, supported by an optimal mixing coefficient from our theoretical framework to tailor adaptability in various federated settings. Empirical tests confirm our method's 's superiority, aligning with theoretical insights and outperforming traditional federated learning approaches. Limitations include a simplified text model with a single activation function, suggesting future work with more complex models to better capture deep network behaviors in federated environments.
Acknowledgement
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (22YF1428800,, Shanghai Local College Capacity Building Program (23010503100), ShanghaiTech AI4S Initiative SHTAI4S202404, Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAD, MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (Shanghai Tech University) and Shanghai Engineering Research Center of Intelligent Vision and Imaging. Wei Huang was supported by JSPS KAKENHI Grant Number 24K20848. Additionally, we would like to thank Leqi Zhou for her contributions to proofreading this paper.
References
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and dif-ddstillattion in deep learning. In The Eleventh International Conference on Learning Representations, 2022.
[2] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Federated Learning with Personalization Layers, December 2019. arXiv: FeEXiv1191...m 1912.008 18 [cs, stat].
[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-109 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
[4] Zhongyi Cai, Ye Shi, Wei Huang, and Jingya Wang. Fed-CO2 :
Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning. Advances in Neural Information Processing Systems, 36, 2024.
[5] Tong Cao, Ye Shi, Baosheng Yu, Jingya Wang, and Dacheng Tao. Knowledge-aware federated active learning with non-iid data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 00279222289, 2023.
[6] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237- 25250, 2022.
[7] Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Under- standing and improving feature learning for -uttof-distribution generalization. Advances in Neural Information Processing Systems, 36, 2024.
[8] Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding Transferable Repre- sentation Learning and Zero-shot Transfer in CLIP. In The Twelfth International Conference on Learning Representationss 2024.
[9] Gary Cheng. Karan Chadha, and John Duchi. Fine-tuning is fine in federated learning, arXiv preprint arXiv:2/08.0731 3, 2021.
[10] Muthu Chidambaram, Xiang Wang, Chenwei Wu, and Rong Ge. Provably learning diverse features in multi-view data with midpoint mixup. In International Conference on Machine Learning, pages 5563-5599. PMLR, 2023.
[11] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[12] Tianyu Cui, Hongxia Li. Jingya Wang, and Ye Shi. Harmonizing Generalization and Person- alization in Federated Prompt Learning, In Forty-First International Conference on Machine Learning, 2024.
[13] Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompt- tuning in bridging generalized and personalized federated learning- In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6087-6097, 2024.
10

---

[14] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEFF, 2004.
[15] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pages 2066-2073. IEEE, 2012.
[16] Tao Guo, Song Guo, and Junxiao Wang. pFedPrompt: Learning personalized prompt for vision-languagg models in federated learning. In Proceedings of the ACM Web Conference 2023, pages 1364-1374, 2023.
[17] Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. PromptFI.: Let federated participants cooperatively learn prompts instead of dation model. IEEE Transactions on Mobile Computing, 2023. ISBN: 1536-1233 Publisher:
learning in age of foun-
IEEE.
[18] Wei Huang, Yuan Cao, Haonan Wang, Xin Cao, and Taiji Suzuki. Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective, August 2023, arXiv:2306. 13926 [cs].
[19] Wei Huang. Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding Convergence and General- ization in Federated Learning through Feature Learning Theory. In The Twelfth International Conference on Learning Representations, 2023.
[20] Yu Huang. Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 10944-1095, 2021.
[21] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning non-iid data. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7865-7873, 2021, Issue: 9.
[22] Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 1-37722-377832 2022.
[23] Meirui Jiang, Anjie Le, Xiaoxiao Li, and Qi Dou. Heterogeneous personalized federated learning by local-global updates mixing via convergence rate. In The Twelfth International Conference on Leurning Representations, 2024.
[24] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer ReLU convolutional neural networks. In International Conference on Machine Learning, pages 17615-17659. PMLR, 2023.
[25] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009. Publisher: Toronto, ON, Canada.
[26] Hongkang Li, Meng Wang. Sijia Liu, and Pin- Yu Chen. A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity. In The
Eleventh International Conference on Learning Representations, 2023.
[27] Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang. Weiping Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated learning by transformer personalization. IEEE transactions on neural networks and learning systems. 2023. ISBN: 2 2162 62-X Publisher: IEEE.
[28] Hongxia LI, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12151-12161, 2024.
[29] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-442 45099440002000 2020.
[30] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- chiis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint wwXiv:2001.015233 2020.
[31] Harry M Markowitz. Portfolio selection. The Journal of Finance, 7(1):77, 1952.
[32] Harry M. Markowitz and G. Peter Todd. Mean-variance analysis in portfolio choice and capital markets, volume 66. John Wiley& & Sons, 2000.

---

[33] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication cenmumication-cffcceent efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[34] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[35] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 2012.
IEEE,
[36] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1406-1415, 2019.
[37] Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi. Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Yi Lin. Federated text-driven prompt generation for vision-language models.
In The Twelfth International Conference on Learning Representations, 2024.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[39] Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pages 1 11112-11122.
PMLR, 2021.
[40] Fu-En Yang, Chien- Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19159-19168, 2023.
[41] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. FedPrompt: Communication- efficient and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
1-5. IEEE, 2023.
[42] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009))2337-2348 2022. ISBN: 0920-5691 Publisher: Springer.
[43] Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization. In The Eleventh International Conference on Learning Representations, 2021.
[44] Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. In International Conference on Machine Learning, pages 43423 43479. PMLR, 2023.
12

---

Supplementary organization:
Performance evaluation
14
Further ablation study
14
Introduction to feature learning
15
D Evidence of assumption
15
Analysis framework
16
E. Gradient update analysis
16
Signal-noise decomposition
17
Generalization analysis
18
Theoretical analysis for PromptFolio
21
Coefficient dynamics: stage one
21
Coefficient dynamics: stage two
26
13

---

A Performance evaluation
In this paper, we adopt a similar setting to the experimental result in [28] The experiments conducted on cluster with Intel Xeon 5218R, 512GB memory, and NVIDIA Tesla A40 GPUs 48GB. Here, we use Food101 3. DTD M. Caltech101 14] Flowers 102
34 OxfordPets [35]
to evaluate our algorithm. The baseline algorithms are CoOp 42). PromptFL [17], the variants of PromptFL [9 29 21] and FedTPG [37]. We Dirichlet distribution with parameter
0.3 to split the dataset. Consider we use 10 users with 100 communication rounds to assess our algorithm. 10 epochs conducted each communication round. There are 8 prompts that are randomly initialized during prompt learning. The accuracy is shown Table 2
Food101
DTD
Caltech101
Flowers 102
OxtordPets
CoOp
83.0910.58
56,40
90.62:0.72
69.03+1.04
90.1810.74
PromptFL
85.15:0.22
51.99+1.41
93.47=0.30
74.47+1.40
91.08++0.53
PromptFL+FT
80.85+0.27
50.72+1.117
89.0410.59
69.05+1.94
87.12+1.15
PromptFL=FedProx
85.48+0.19
12.38+1.95
93.57=0.46
74.27
91.07=:0.41
FromptFL+Fedder
82.20+t1.03
18.14
91.70:10.58
71.71+00.53
90.65-10.69
PrompdFL=FedAMP
83.24-10.52
56.40+0.53
91.38+00.2
70.80+00.42
90.56--0.33
FedTPG
8.059
52.01+0.80
93.38+0.12 93.5910.39 61.0441..69
72.91+1.223
90.98+00244
PromptFolio (Ours)
1..50+1.34
74.61
92.08110.14
Table 2: Comparison of different methods on various datasets.
B Further ablation study
This paper further conduct experiments different shot numbers and different backbones. The results in shown as Figure
Vii-8/18 ResSet-30
Accuracy (VIT-8/16)
Accuracy [Ros (00
Accuracy
70-
70
50
Num Shots: 16
Num Shots:
15
Num Shots:
Num Shots:
74
6.2
1
0.0
1.4
0.6
Global
Local
Global
Local
Figure 3: The accuracy curve among different shot numbers (left) and different backbones (right).
Different shots As shown in Figure 3 (), we tested the accuracy using different numbers of shots, ranging from 16-shot to 1-shot. It can be observed that the optimal coefficient remains stable and close to 0.2, demonstrating the robustness our algorithm across various shot numbers..
Different backbones Figure b)) illustrates the accuracy using different vision backbones of CLIP, namely ViT-B/16 and ResNet-50. In this experiment, a similar trend observed, where accuracy first increases and then decreases with the mixing coefficient, regardless of the backbone used. This outcome indicates that our theoretical framework is consistent across different backbones, further validating its applicability.
14

---

C Introduction to feature learning
Feature learning theory is theoretical framework proposed recently. The seminal work proposed feature learning theory to understand ensemble, knowledge distillation, and self-distillation deep learning. They show that when data has structure containing features, the network's dynam- ics can be tracked during gradient training, which can further be used to characterize generalization. Later, this theory was further systematized and standardized by [7]. forming a fundamental theoretical framework to explain benign overfitting. Since then, feature learning theory has been widely used understand algorithms theoretical system.
22 and techniques [00 deep learning, forming a comprehensive
To provide more details the general feature learning process:
Defining the feature space: As outlined in Section 4 features are categorized as task- relevant and task-inreevvant. For example, in image of cat, features representing the cat itself are task-relevant, while background features are task-irrelevant.
2. Parameter representation: As described in Lemma learnable parameters can be decomposed into the feature space. In our framework, the learnable prompts are expressed linear combination of task-relevant and aus-irrelevant features.
3. Learning dy namiii::: Theorem F.3 examines the learning dynamics of coefficients of feature learning, where coefficients are defined by the weight decomposition into the feature space, providing valuable insights into the learning process. When task-relevant features dominate, the network learns the target effectively and demonstrates good performance.
4. Generalization bound:
By leveraging the learning dynamics of the coefficients. we can demonstrate the generalization bound or test performance post-training. In our work, we connect the performance of prompt fine-tuning with the ratio between task-relevant coefficients and task-irrelevant coefficients.
D Evidence of assumption
Here's the evidence supporting the second assumption that the features of the pretrained model orthogonal. We focus on the final projection layer of the text encoder in CLIP (38) and calculate its cosine similarity between each row. The statistical distribution of the cosine similarity shown in Figure
0 020
Frequency
0 010
0 005
0 000
0.4
Casine Similarity
Figure 4: The distribution of cosine similarity between different rows of the final projection layer in CLIP 38
Here, we find that nearly all of the cosine similarity are smaller than 0.1, which is close orthogonal. This evidence support our theoretical assumption. Except the evidence of cosine similarity, the
15

---

previous research [20] also make assumption of orthonormal rows of weight matrix to the multi- modality latent space.
E
Analysis framework
In this paper, our analysis will be made under the following assumptions:
Assumption E.1. Suppose that:
1. The dimension of latent space m is sufficiently large m = [(()).
2. The number of training samples and clients follows v ((polylog(d)) K @(1). The federated network shares the same initialization.
3. The learning rate - Ō K min | op²m==)) and the standard deviation of network weight initialization do s ((m)) min((11m11) o }}
For the first assumption, we assume the dimension of latent space is large enough to make our analysis ensure over- wer-parameterized setting. The second assumption provides statistical properties to the training data and network weight initialization. The third assumption of small learning rate and small initialization guarantees that gradient descent effectively minimizes the training loss.
Note that this analysis framework can be directly applied our algorithm PromptFolio. To achieve basic federated version of prompt learning, i.c., PromptFL, we apply the learnable prompt p as the global prompt Pg PromptFolio and apply the mixing coefficient 0 result of PromptFL. Similarly, we can apply the learnable prompt as the local prompt pz and apply the mixing coefficient&
obtain the theoretical
we obtain the theoretical result of CoOp.
E.1 Gradient update analysis
This section discusses the computational approach used analyze the performance of the algorithm. focusing on the gradient calculations essential for optimizing the model parameters. To calculate the gradient, we first need to find the partial derivatives of sim(gs., hgg)) with respect to pa and pr.- Let de.r.... and L..,,.. are the r-th diagonal elements in o' o'w,, Pa. + wx Pv..)) t' +''('w''' (-w" PG, + w ) and o'ww," PL,k + Pyr.)
(-W,.PL.K + P respectively:
Ösim (g..s hr.s) = (W (o' (WPG. + WP + ( + WP g(xk,))
Öpa, Ōsim (g.. hv.)) Spr,
O(WT + WP...) + (o'(-WLLA Wp...))) g(x..). (13)
Using the logistic loss LT for client k; the goal is to minimize the subsequent loss function to achieve supervised fine-tuning of the text prompts:
LI (pG...PIN) =
log(1 + exp(sim(ge.i, ha.)))
(14)
Now, we can use these partial derivatives compute the gradients with respect to pa and PL:
1
exp(sim gk. hg.))
Ōsim @sim(gr, he.))
V Po I. (pa.e)
T1 1=1
exp(sim gk.i, ha.i)) Öpa.
(15)
exp(sim gr.i, ha.))
@sim(ga, hg.))
PL L7 (pr.a)
7kk
+ exp(sim he.)))
0pi.k
16

---

To simplify the update process, we can rewrite it using summation notation. Let f,4 denote exp(sim(gs,::b. 1+exp(sim((A) .h..)))
V L (PG,x) = 1-0
( + WP (Pte))+o' + WPy..))) ((x),))
T&k
1
(16)
Thk
i=
0
V Pz. LE (pr.k)
2 (..' (o (o'(''''WW + WPm..) +o + Wpm..))) ((x.,))
=
0
nh
m
(17)
In the above equations, &r.k. represents the r-th row of g(Xx,)). Note that w? represents the r-th row of the weight matrix W.
E.2
Signal-noise decomposition
In this subsection, provide the proof of signal-noise decomposition mentioned in Lemma Note that the local update of global prompt coefficients can be generalized to local prompts by changing the subseript G to L. The coefficient Pa/r, denotes the coefficient of the global prompt at client k. The coefficient L,/L,, denotes the coefficient of the local prompt k' at client k. The coefficient W denotes the coefficient of client k''s s i-th task-irrelevant feature at client k.
4.1
Here, suppose that the weight of the pretrained model consists of two kinds of weights, task- relevant weight and task-irrelevant weight &. Without loss of generality, we can rewrite the gradient descent update of the prompt as follows:
K
PG -GG.llMolE AG + Σ
G.G. + G. 1lve mb) + Σ G.ki 1111ll1
Ei.
#=1
(18)
-Ba ||l| pG (t) PG
G Tc.
(0) (0)
1GAllmkll 3 (4))
(8)
o 1
(19)
PŁ, = ((0 Hel2 MG + LL L. + 7 Ml
(4)
(0)
(4)
112222222m Σ
ps +
o
2
(20)
According to the update formula of the gradient descent, we have the update formula of the global prompt and the local prompt:
PG PG.kk V Pa.
(t+1)
LT (PC))
PG. + (1
(21)
1=1 1=1
(1+1)
(t)
V PL.K LT (9
PL = PL.k
m
PL. +
1
M
lg Tr Lonkk,
(22)
As result, for the row corresponding to the global feature line, we have the update formula of the coefficients:
(t-1)) a(t)
GG G.
+ 2
0)
(() G.r.k.a
(23)
7g
1=1
(t-11)
L,k
G.
a(t)
+
7 A
(t)
ki
10) G.rkt ||
(24)
7
i=1
17

---

For the row corresponding to the local feature line, we have the update formula of the coefficients:
RG = NG.kWW'
(1+1)
7
0)
W
12 = k),
(25)
Tep
BK
(H+1) (e)
+
Gork
1 k).
(26)
Tik
For the row corresponding to the task-irrelevant feature line, we have the update formula of the coefficients:
Ak
((+1) (1)
1 21 0)
H() G.nka' yka TR.a 112112
9,
G.
Ri
(27)
Tik
((+1)
(t)
0
((() k,s
(() G.nk''
yk Tka, 112/12.
(28)
L.k.l L,K,
Tkk
(t) (it)) To analyze the increase of the coefficient, we decompose the coefficient &G G.K. and (t)
PG.k.I"
G.kk
(e+1) (1)
G.
" (1
0)
(t)
G.rkk
(())
(Uk.s =1) 2R.i. |11/1
(29)
T1k
t=1
PGG.. G.G.
(t+1) (2)
T (1 - 0)
Σ
(tt
k,i
Grrkk, 1 1yKk, = --1) tkiil 1
(30)
iml
na
((+1)
((0)
Grrki 1yle, =1) FR,s,1 1|E/112
(t)
(31)
L,K.
T4k
i=1
ag
PL,kI LI,,
(t+1) (1)
1 0
(1) G.nk, Myrei -1) 0R... 1
(32)
Here, suppose that Ék
EI, thus have the following lemma.
Lemma F.2 (6)) have
Suppose that and T, > (og(4n/6), then with probability at least we
1
3
2 202 - Ex.
2 20
(33)
KE Cw.i) S o
1 1og(4n2 (an22442ooo 10)02
(34)
where denote of = |11:12 the summation of the norm among all task-ireelevant features.
E.3
Generalization analysis
For simplicity, we first introduce the definitions of F4 and F_. Here F, means the train loss corresponding to the positive class, while F_ means the train loss corresponding to the negative class.
(p) = o (Wp Wp+) - WW-Wp Wp+), F- (p) a (Wp + Wp-) - Wp-),
(35)
(36)
F(p) F+ - (p).
(37)
To illustrate the decoupling the label from the definitions of F+ and F_, we have the following lemma.
Lemma E.3. Suppose that y is the ground truth label, we have
Fy(p) - F-y(p) (pF+(p) - -F())
(38)
18

---

Proof.
We consider two situations:
+1 and -1. y +1,
Fo(P) (-y(p)= F+(p) Fp)
(39)
If y = -1,
(((P) - Fp)(p) =F -F()).
FP(P)
(40)
In conclusion, have
P2(P) F-y(p) = (p)+ - F-()))
(41)
Lemma E.4. Under the modeling of prompt-based federated learning, the expectation of test loss Lo of algorithm can be treated as the probability
E [ P( < 0), (H)
(42)
where μ and are controlled by the coefficients of feature learning. Thus, the performance of an algorithm can be evaluated by the ratio p/o.
Proof. Recall that the test error is equivalent to
(D(PL) ((((F(()), (F- 0).
(43)
According to Lemma
We have that
LD(PL) ((y(F+(p) - F- (p))).x) 0).
(44)
Note that
Bll
m /4a l2
(W.p)
7K px 12 Φ1
(45)
1112
0r.1/r.112
Wp, and Wp_ can be treated as two constant terms. We consider each line of F(p) and x, then the problem is equivalent to
v(ufc(Bllpalle) vfi(llllalk)) + Σ (x 0,
(46)
where
so(0) l2)
Atllllll
F(p) =
fx(*x|laxl (
(47)
Note that the above equation is equivalent
fo fa(llpoolla) + fi f1lllllll2)) + E (u=,fx++(0))ls)la
(48)
Note that when we finish training, the coefficients are fixed and thus can be treated as constants. Ty are zero mean Gaussian variables, are {+1, -1} Bernoulli variables and z, and are independent. Thus the error can be defined by two values
E Lpi = P(x "
(49)
19

---

where
1 fo(0) + fimllllll))
(50)
E (Ros(Osleel
(51)
The problem can be treated
Lo = P(= < <0),=~ N(,,
(52)
Theorem E.5. Suppose that the coefficients of CoOp at step k are Bk "Kk and Φk, the coefficients of
PromptFL step k are 3&r 7 and 6x. Here, we define a O Butze) and b = O as the ratio
2x)
9x+%
of different coefficients and the correlation between prompts is defined as p E 00 1]. We have that the mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL.
(pp((10)pp - -0)P + 0PL) S (1 (0--))))pp))+0ppplp 0T.((P))
(53)
Ca = ())(a)(b2)))) + 2pb
C. 36-Co Ce
)
AO E 0, proj [0.1]
where
C. (a + b)( - 4b(pb 1)
2Ca
Cc = (b () + 1)2 Sab2 (p + 1)2
Proof. Then we suppose that the end of training, the random variable corresponding to LD(PG) is za ~ N(ua og), the random variable corresponding L.D (Pr) zr. (Hr,,77) Consequently, we have the random variable corresponding to PromptFolio as ZGLe = (1 - 0)zc + Oz1, then we have
zGLo = (1 f) + 42g ~ N((1 () + 8pL, (1 - (1-8)' + 2p(1 - 0) + +PoG)) (54)
where pspsli
1 is the correlation coefficient. Assume that a = Hr. Mc
= E1.
we have
Ec.
(1 - 0)HG + 1ul
HG 8 1L
((1 0) + 2p(1 0)0000) 2 (1 0)
(55)
&G EL
when a 2 b2-b As result, we have that
b-p-
Lp((1 - 0)pG + 0PL) s (1 - LD((G) + (LD(PL).
(56)
Here, we find where this inequality becomes an equality. After further simplification, we observe that this equation is quartic equation, and we' ve discarded 0, 1, and another unreasonable solution. The remaining solution can be expressed as
C - C.
(57)
2Ca
where
C. = (b a)(b² 2pb + 1)
C. = (a + () - 1) 46(pb - 1)
(58)
C. = (b 1) 1) (a b)2(bb + 1)2 8ab2 (p 1)2
Here, for any 6, we take the
Cs Ce
0 0, proj [0.1]
(59)
2Ca
due to the derivative of this objective function being less than when 0. As a result, when take 0 in (59), we will obtain the inequality.
20

---

Additionally, we provide the optimal coefficient of the algorithm and establish the following theorem.
Theorem E.6. Suppose that the coefficients of CoOp step k are BK "Ik and Φk the coefficients of
PromptFL step k are B&r Tk and Φu. Here, we define a O( SLt and b = O 2x) as the ratio +x45
of different coefficients and the correlation between two prompts is defined as p E [0, 1). Then have the optimal mixing coefficient 8 as
pb
0- = proj [0.1]
pb(a + 1)
(60)
(a + b2)
Proof. Here, we suppose that the local prompt corresponding to random variables and a, the global prompt corresponding to the random variables f and a. To achieve the highest accuracy, want to solve the following maximization problem
(0μ + (1- - 0)1)²
max 0
0² + 2p 0)0oo + ()-0
(61)
which is equivalent to
2(0 + (1-8))p) - (02
+ 2p0 - 0)00 + (1-0)30")
=00 + (1)))))2000 + 2p(1 - 20) 2(11 - -01=)
(62)
( (0= + 2p0(1 0)00 + -1-0)200
(a)
(1-) p(1 20)020-000000 0(1 o)0²).
Note that due to μ and z are all positive numbers, so we can make a simplification (a). Here, we take n. b = q. and divide ab? on both sides, we have
(a - 1) + (1 - 0)b + (1 (1- (1)
= (Oa + (1 0))(0b² + p(1 - 20)b - (1 (1)
aab + 2p0(1 0)ab (1 (1- 0262 22p0(1 0)6 (1 (1-06 0)²
ab² + p(1 9 (1 000a (1 -0)0b² p(1 - - 0)b (1 (1-0)² 0)2.
(63)
Thus, we take further simplification and we get
pdab 0 + a = 0b + pb p0b.
(64)
Note that this linear equation, so we obtain the unique optimal 6+
pb
0* proj 0011]
(65)
(a+ 12) pb(a - 1)
F Theoretical analysis for PromptFolio
In this section, we provide the whole feature coefficient dynamics through two-stage analysis. The sections are arranged as follows: We first provide some preliminary lemmas. Then we provide the order of coefficients at the beginning of the training, which is stage one. Finally, we provide the coefficients at the convergence of the loss, which is stage two. Remark: For simplicity of the analysis, we assume that all the clients have the same number of samples and thus n1
пk n/K.
F.1 Coefficient dynamics: stage one
Lemma F.1. Suppose that max ( ((0) ( max max (...) = ((1) with E [n]] and k E [K], for E (0, Ti]. we have
-(1), max m (8)
TL.k.) O(1) and
C1 < k < 1,
(66)
where C1 is positive constant.
21

---

Proof. In the first stage, we consider that the gradient of the loss l' is bounded in constant level.
We assume that the coefficients , ol O(1) for 0 s t < Ti. Here we have that the output similarity satisfies:
(0) sim si g bR,)) - max { ( (e)) o (8) En.))), - Pk.
(67)
0
When E [0, we have
+ exp(sim hx.i))
(68)
1+0
Theorem F.2. Under Assumption
E.
there exists total number of local updates Ti
R₁ E
O o such that
f @((1 8
B @(HNSNRR))
Ak E [K]
(Ti) @((1 SNRR)
(T)) @(OSSNR)),
4h: [K]
(69)
kG.kk
L.K
G.k. = O(1 0)
(T))
(Ti)
O(0), vk KJ,, E [7.]
L.k.I
Proof. Part Analysis of B. and (T)
7a.k
We first consider the growth of local task-relevant coefficient kG and G. on the corresponding client k. Consider the iteration equation for the coefficient signal learning under local gradient descent in the first round, we have:
(1+1) (1) &, -G
21
(1 0)
10 G.nka me
(70)
ba
Tig
According Lemma F. we have an upper bound for signal learning at the first round of local updates.
(t+1) G. S A. + 71 0) 1
(1)
(71)
Taking a telescoping sum over t 0, 1, .... E, we can obtain the upper bound for signal learning before the first step of weight averaging.
G. (1 - 1
(72)
After taking the average among the coefficients, we have
K
(px, (u)) 17(1 - F) 14x
(E)
(73)
K
|| W=1
Note that we denote X In the following E gradient descent steps on each clients, we have
A
EE=1
(pL)-))) |alll
the total similarity between signal vectors among clients.
K (pAKs m46))
(2E) k.
17(1
0)
|
K
||mx 12
K
1)n(1 - 0) pxx 12.
(74)
In the second round of update, we apply average among the clients
7 (2E) < (xx
1 Xk + +1) + (K- - Xx) X )n(1 - 0)
2
Xan(1 - 80E
(75)
K K
K
K
22

---

We repeat the computation process and obtain the following results for the third rounds of local updates and the FedAvg process.
G.Ack I X 1 1 & Xk + 1)m(1 1
(3R)
2
(76)
K
1 S (xa(
(3E))
1
2 K X +1 (K -xa) xa)mBlpuR
2
3 x
(77)
K
K
K
Thus, after Ri - Ti/E rounds of communication, the noise memorization has an upper bound:
R1 11
K.kkk
(RLE)
S
K Xx + 1)m(1 0) Fl 1
(78)
G
(R, E)
R1
X
- -)
(79)
K
We can similarly obtain the lower bound of this proof. Consider the first round of gradient update, have
(++1) (1)
(1
(() G.rkk, 1
7G GG,kk
a
n
G, + 711
(80)
So before taking the first average, take telescope sum over 0 ..... E yielding:
(E) JG
2 n7( 0
(81)
After the weight average operation, we have
K
(E)
1
Σ M=1
(Ek (e) 1/(1 O)C,Ellmul
T...k
(82)
K
Recall that X In the next E' gradient descent steps on each client, we have
K k==
(pe) counts the total similarity between the signal vectors among clients. ||
(2F)
> 11(1 - 0)C, + O PA
K
Xk
1)n((1 - ()
(83)
K
After gradient descent, we apply the second weight averaging operation on the server and obtain the following result
(2E) T...K
1
+1+ + (K xx) xx)
XK ) Ellmu
[x
K
K
2
X 0) 1
(84)
K
Similarly, repeat the computation procedure and obtain the following results for the Ry-th round of local updates plus weight average operation:
R1
FR
2
X + 1)m(1 0)C
(85)
K
7G.k
(R.
>
R
1/(1 - Fillyst
(86)
K
As result, we have
(RIE)
R
K - (1=0))xxx)(iill Ex
Cn =
noza x 0) ||x 0mm(110)xxsNRR)
TG.k
(87)
Here, the global task-relevant feature follows that Xx K, so we have
(R Rí (1 8) 0x
nood Xun(1 -) =0 (0K-KR3)))))) (88)
On
23

---

Part 2: Analysis of ((2)) and (25)
k..
Similar to the global coefficient (T1) and (T1) equation for the coefficient
K..kk* we first analyze (T1) KL,kk'
Consider the iteration
((+1) (t)
" 0
10) Lrrkk 1l
(89)
R.K.K
Ra
T&
i=1
According to Lemma
F.1
we have upper bound for (t+1) 7i.kk.k
((+1) (t) kkk 3 L,,,k + r78 pe 12
(90)
Then we taking a telescoping sum over L 0, 1, ..... R, have
S n p LLk
ER.E)
(91)
Similarly, we repeat the computation procedure and have the lower bound of the (t-1)
L.Kk
LLKkk L,K.
(++1) (4)
0.
Ra i=1
10)
(92)
According to Lemma
we have
(++1) (1) L.kk 2 71,kkk + nTCC1 1
(93)
Then we taking a telescoping sum over i 0, 1, ...., E, we have
ER TL,R,
2
(94)
As result, we have
(R
= R.OENCCilMM Tom
Cn 0)Cillun113 = - O(TSSNR))
(95)
L.R.k
The global feature and local feature have the same weight and take the same order.
On mora 1|C (
R1OFxAn BLKK
(96)
Part 3: Analysis of &. and (Ti)
LLk
We first establish the lower bound for G.K.I We show that
(Ti)
PPG Ex,z) (Po Ex,i) Σ
(0)
(0)
11 EG...
(97)
log(4n2/0) /8)
(0) ( Ev.a) + + G -2
2/
Σ(1 + 110
(98)
of
log 4nn 18)
(p Ex.o) + G.G. - 4C2rg/
(0)
(99)
o7.
where C2 is positive constant that satisfies C2 2 max(vil) (b) is by definition of Cz- Here, we suppose that t (t) it is easy to check that:
(1)
& (1)
F..... (a) is by Lemma R.2 and
G. max { (P... En.i) + VGaa 4C2n
(0)
Tog(An=/6) /8) o7
f. At initialiizaoon
$ (0)
1
000p0r 4C2m
log(4n2200 (a)
(100)
G.K.
1
of
24

---

log(an2/0] where (a) is by the following condition:
σo Czn v /log442200)
(101)
Jp07
We compute the growth of (t)
G.k.1 as follows:
$
(1(+1) ((+1)
" G.k.d ( (1- 0 100
((0)
G.rks 1
n
G. + (1 of
(1)
(102)
2n
Before the first step of weight average, we take the telescoping sum:
nEC, (1 00
(E))
(103)
G.k.
2n
Then, we perform weight average operation
F,C (1 - 0)
(104)
GJ
K
2n
The next E gradient descent steps yield:
G
(2E)
K + 1 EE (1 0)0
(105)
Here, we take the average among the clients and we get:
C.
22
K
1 K
1
K K-11 11
K
K
F.C1 - 00
2n
K 2n
2 F1C
(1 (1- f)0
(106)
We the same technique to obtain the lower bound of noise memorization
(RiE)
R4
nEC
(1 0)0307.
(107)
2n K
Finally, we confirm that
(i(()
nT (1 0) -PP Ekii) 4C2mg
log (4/22//
(108)
G.A.
2nK
of
Ive IVc
nTT11 (1 - 0)0201
-PPG. Enr)) + C3
(109)
2nK
Ca.
(110)
of which holds when of. 2 Css (og(4n²/0)rz and the inequality (b) is by taking the value Ti. (Ti) Then we provide the upper bound of the &G,k1-' Here, we suppose that & (0)
where the inequality (a) is by definition of constant C3 < 402m
max
G.K.I PG,u... Similarly, define the coefficient after weight average G.K.
(1) G. Po..j.
Clearly, we have that v V.A. 0 for all [L] and k [K] by defi-
max
nition. Then we have
((+1) G.
= G.k. + 2(1 ((1-0)
(0
0)000 (e) G.rkk E
(t)
" -1-0000
(111)
G.K.1
where (a) follows 10 and 0' in Lemma F.1 and Lemma E.2 Similar above proof, have that
I (E)
G.k.l < E (1 00
(112)
2n
25

---

After t = E steps, we perform a weight average operation and we have
n (1 0)
V.G.k.
(113)
K 2n
Similar to the above proof, we have
G.E. S
E)
Rí n
K 2n (1 - 0'
(114)
Thus, is confirmed that vo (2)
G,6.1 O(1 0). Similar to the convergence of (4)
0(1 0), we
G,K.
have that v (Ti) O(0).
Lemma F.3. There exists total number of local updates Ti R. O(n (((n-1am ( such that
Ti) =((KSNRR)), Th SNRR)
(Ta) T ) O(1) Ak [K] E [L].
(115)
Here, πt = Ex ma/K K is the average number of data in each client and SNRG = /| (opell///p///mm/m// denote the signal-noie noise ratio between the task-relevant feature and task-
SNR, = irrelevant feature. Here, we define Xk
>> k'= ( pkx ) p
Proof. The proof of the result follows the proof of global coefficient in Lemma F3 obtain the final result.
We take O = 0
F.2 Coefficient dynamics: stage two
In this stage, we focus on the proof of scaling behaviour for the coefficients of features 9c, 7G. BL. NL. We first provide the following lemma.
Lemma E44. Under Assumption
E.1
we have
(0) (... ..G (DG. Muc) + +9
(0)
(t)
(DG. ((1)))44)) = -PG m14k) + Ja.k
(0)
(t)
AA E [K]
(116)
(p% Pi.ayP1) = T. vp2) + PG
(0)
(p) vp4) (p% m14)) + (1)
Ak E [K]
T.
for all k E [K].
Proof. According to the update formula of PG and Pt. we have
K
( Pg 00)
mG) - -TG
(6)
2 (p ma) +
.G. 2
=ya (e)
(117)
where the second equation by the orthogonal assumption between the feature vector and noise vector. Here, these equations follow the same proof and we thus complete the proof.
Lemma F.5. Under Assumption
E. for 0
t
S
T.
where T
polly[[uul2 or
2 pp % n, m). we prove that
0 300 < () S (1 - ONRKSNRR (TT
(118)
0 7GA < T.k. $ (1 8)TX log(()
(0)
(119)
0. (0 < PLK < SARKSNR& (T"),
(t)
(120)
0 7LA & L.k. S &OKKSNR log log(
(0) (1)
(121)
for all k E [K].
26

---

Proof. Here, we only prove that 0 S JG. S GGk S (1- SNR log(T"). The proof of other coefficients can be generalized from this proof. Considering the update formula, we have
7(0)
(1)
((++) =(()) G G +
0) nn
((+T)
oG.ndi ps
(122)
Let T3 R&E to be the last time $ T that G. 0.5log((* ) We have the lower bound (f)
and upper bound of (P ux):
1.25xx log((") & ( ma) S S05Rxx log(
(123)
Due to that
(P (4x)
(0) (PG 11)) + TG.
(t)
(124)
0.5(1 - 8)a + 0.5(1 SNR, log(T")
(125)
0.25(1 0) SNR log(T");
(126)
where the first equality (a) is by the update formula of y the second inequality (b) is by Ta Z
0.5(1 SNR] log(T*) and (Po,k: p&) 2 -0.5(1 f) due the definition of T. and a. The
(0)
last inequality (c) is by v SNR log(T"). Similarly, we can also derive the upper bound as follows:
(PG ma)
( 11x) + 7
(0)
(4)
(127)
0.5(1 0)a + 0.3(1 SNR lTg
(128)
0.5(1 - SNR3
(129)
According to the above formula, we have
(Ta) (Tb)
TIK
(T++r)
o' d'PGGT (..s
(Te+T)
7G.k
-GG
nk
ki
Σ
TXR
(RE+T) - ((PG.EET (...iH))
(RE+T)
(130)
nK
kii
R
(a)
Tb)
1
(T++r) ki
0 ('(PG, (k..
(T.+T)
G.
k
r=1
mml
Σ
TXk
exp(1 ((P
(RE+T),
K
RoRRER
(131)
(0)
2 Jc. + 0.25 - SNR? log(T")
0.257 exp(- log(T" ( log (T"
(ogg(o")))(("N SX
(132)
2 (1 - f) SNR? (T*),
(133)
where the first inequality (a) is by the definition of E, the second inequality (b) is by the above formula and the third inequality (c) due to that SNR 2 1 and the induction hypothesis (Th)
JG.k < 0.5(1 SNR log(T").
27

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated the abstract and introduction, this paper provides the first the- oretical framework analyze the dynamics of prompt learning federated learning from pretrained vision-language foundation models. These claims are reflected throughout the paper, with theoretical analysis and experimental validation provided in support.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation analysis is provided in the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification: The paper lays out a detailed theoretical framework with clear assumptions and provides mathematical proofs for the theorems introduced. Assumptions and their implications are explicitly stated in Section and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
This paper details the experimental settings thoroughly, including data distribu- tions, models used (e.g., CLIP), and the specifics of the federated learning setup Section 6 It provides enough detail for replicating the main experimental results, which align with the theoretical claims.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
29

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is available alongside the submission, including ZIP file with all relevant scripts. We will make the code open access once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co co/
mublic//giides/Codesttbmi sow.sooooisioooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctsps.cc/public//uuia httpa///guuieee/tttuuubiiissss soioiiiicsii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides comprehensive details about the experimental settings, including descriptions of datasets used, hyperparameters, data splits, and other necessary details Section 6 to understand the experiments fully.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results section includes error bars and other statistical measures to indicate the significance and reliability of the experimental outcomes. This is particularly visible in the detailed tables and figures that compare different methods and settings.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper now provides detailed information the computational resources used for the experiments. including the type of GPUs, memory requirements, and the execution time for each setup in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https: htte:://nneuiippee www.eeerrrrss.pppclueeeeeeeceeee httt:////c...e/iitcc.ccceeeeleee
Answer: [Yes]
Justification: The work conformed with the NeurlPS Code of Ethics. The paper does not contain any content or methods that would violate ethical guidelines, and it maintains anonymity and integrity in its experiments and data handling.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
31

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper primarily focuses on theoretical research in machine learning and does not address societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risk that have high risk for misuse, so this question is not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper adequately credits original sources for assets such datasets and models used in the experiments. mentions using standard datasets and references the original works where these models and datasets were introduced.
32

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pitaasetsspapessiiiii paperswi thcode. com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets that would require additional documentation or licensing information.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing direct research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
33

---

Answer: [NA]
Justification: Since the paper does not involve human subjects, there is no need for IRB approvals equivalent ethical reviews.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
34


Reviews:
[{"review_id": "FCVWxk7IZ3", "reviewer": "Reviewer_BdA2", "summary": "This paper tackles the interesting problem of analyzing federated learning (FL) from vision-language foundation models like CLIP. The authors develop a theoretical framework based on feature learning theory to understand how prompt-based FL works. They introduce a new algorithm called PromptFolio that mixes global and local prompts, drawing an analogy to portfolio optimization in finance. The theoretical analysis shows how this approach can balance generalization and personalization in FL. They back up their claims with experiments on several datasets.", "questions": "see weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The feature learning approach they develop seems pretty solid. \n- The connection between prompt mixing and financial portfolio theory is novel.\n- The ablation studies on data heterogeneity and client numbers are particularly informative.", "weaknesses": "- This reviewer is confused by the feature learning theory (I am not familiar with it), which is introduced in a relatively short section. Is it (looks like) a common sense in deep learning or a kind of new theory proposed recently? It would be helpful to have a more thorough introduction to the theory.\n- Using a single activation function for the text encoder is a pretty big simplification from how CLIP and other large language models actually work.\n- All the experiments focus on classification tasks. Any thoughts on how PromptFolio might perform on other vision-language tasks like image captioning or visual QA in a federated setting?\n- Why is the proposed method on OxfordPets dataset not as good as coop? While it seems to be dominating on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comments and questions. We appreciate that you found our feature learning approach \"solid,\" the connection between prompt mixing and portfolio optimization \"novel,\" and our ablation studies \"informative.\"\n\n> Weakness 1: This reviewer is confused by the feature learning theory (I am not familiar with it), which is introduced in a relatively short section. Is it (looks like) a common sense in deep learning or a kind of new theory proposed recently? It would be helpful to have a more thorough introduction to the theory.\n\nFeature Learning theory is a **new theoretical framework** proposed recently. The seminal work [1] (here we reuse the citations in the paper due to word limit) proposed feature learning theory to understand ensemble, knowledge distillation, and self-distillation in deep learning. They show that **when data has a structure containing features, the network's dynamics can be tracked during gradient training, which can further be used to characterize generalization**. Later, this theory was further systematized and standardized by [6], forming a fundamental theoretical framework to explain benign overfitting. Since then, feature learning theory has been widely used to understand algorithms [7, 18] and techniques [10, 17] in deep learning, forming a comprehensive theoretical system. \n\nTo provide more details on the general feature learning process:\n1. **Defining the feature space**: As outlined in Section 4, lines 145-155, features are categorized as task-relevant and task-irrelevant. For example, in an image of a cat, features representing the cat itself are task-relevant, while background features are task-irrelevant.\n2. **Parameter representation**: As described in Lemma 4.1, learnable parameters can be decomposed into the feature space. In our framework, the learnable prompts are expressed as a linear combination of task-relevant and task-irrelevant features.\n3. **Learning dynamics**: Theorem 4.2 examines the learning dynamics of coefficients of feature learning, where coefficients are defined by the weight decomposition into the feature space, providing valuable insights into the learning process. When task-relevant features dominate, the network learns the target effectively and demonstrates good performance.\n4. **Generalization bound**: By leveraging the learning dynamics of the coefficients, we can demonstrate the generalization bound or test performance post-training. In our work, we connect the performance of prompt fine-tuning with the ratio between task-relevant coefficients and task-irrelevant coefficients.\n\n> Weakness 2: Using a single activation function for the text encoder is a pretty big simplification from how CLIP and other large language models actually work.\n\nActually, we also admit this simplification as a limitation in the Conclusion section. It is worth mentioning that the theoretical analysis of CLIP is sparse and very challenging. Very recently, an initial theoretical work [8] analyzing CLIP published at ICLR 2024 even assumes a **linear layer** as the text encoder. As a result, our approach incorporating nonlinear activation functions is more **suitable for CLIP analysis** and is **non-trivial**. \n\nAlso, in feature learning theory [1,6,18], the simplification of the network is necessary for theoretical tractability. Therefore, we also adopt a network simplification here and only use it as a theoretical assumption.\n\nDespite this, our **experimental results align well with our theoretical predictions**, suggesting the practicality of our framework. Future work will aim to extend these theoretical tools to more complex and practical network architectures. \n\n> Weakness 3: All the experiments focus on classification tasks. Any thoughts on how PromptFolio might perform on other vision-language tasks like image captioning or visual QA in a federated setting?\n\nThank you for your suggestion. Firstly, it is worth mentioning that our paper mainly focuses on proposing the **first theoretical framework for federated prompt learning with feature learning theory**. We use the classification task as an example due to the following reasons:\n\n- Most **prompt fine-tuning under VLM**, as demonstrated in CoOp and PromptFL [40, 16], only conduct experiments on classification tasks. \n- Also, in the **feature learning community**, the analysis mainly focused on classification tasks [1, 6, 7, 18]. Thus, the analysis of image captioning and VQA is open and can be categorized as new work.\n- Most **federated learning on VLM** research [15, 37, 27] only conduct experiments on classification tasks. Exploring federated learning for tasks such as image captioning and VQA can also be addressed in separate work.\n\nFrom these aspects, we **use a classification task to demonstrate the application of our theoretical framework** in this paper.\n\n**Any thoughts:** We found that the idea of PromptFolio could potentially be applied to federated image captioning and VQA tasks. By combining global and local parameters, this approach is expected to achieve a **good balance between personalization and generalization**.\n\n> Weakness 4: Why is the proposed method on OxfordPets dataset not as good as coop? While it seems to be dominating on other datasets.\n\nThe discrepancy in performance on the OxfordPets dataset arises mainly due to:\n1. In our experiment, **we use a uniform mixing coefficient** $\\theta = 0.2$ for all datasets. However, the optimal coefficient may vary across datasets.\n2. The inherent high variance observed in CoOp results. The **high variance of the CoOp** results (1.30) may lead to statistical errors during comparison.\n\nThus, we fine-tune the coefficient of PromptFolio and rerun the experiment of CoOp, as detailed in **Table R1** in the PDF. As a result, our PromptFolio currently outperforms CoOp and demonstrates consistency across datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q7i21Uf3V8", "reviewer": "Reviewer_bKME", "summary": "The proposed methodology offers a novel take on analyzing federated learning using prompt learning (for foundation models) via feature learning theory. At its core, the idea is to identify and monitor task-relevant and task-irrelevant features, and leveraging inspiration from portfolio optimization which says to combine independent assets to maintain income while decreasing risk of investment, construct a 2 part prompt. One part which is global and another which is local, to help with personalization. By mixing these prompts optimally, the proposed PromptFolio method highlights an improvement in overall performance of the FL system.", "questions": "The questions have been included in the weaknesses section above.", "limitations": "yes, the authors have addressed limitations within their choices of the experiment setting. However, I would recommend drawing more broad comparisons as to potential missing pieces in comparison to more recent work as well as the finer points from the feature learning theory that don't fully match up to the settings in the FL system.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- A key strength of the proposed method is its novel take on explaining prompt-based federated learning, using feature learning theory.\n- Clarity in idea and its link to existing work.", "weaknesses": "Abstract/Conceptual Questions:\n- Could the authors comment about the few-shot learning aspect of foundation models (+prompt learning) and how the optimal mixing coefficient would be affected in that scenario?\n- An adjacent sub-field that contains foundation models and federated learning is the study of their impact in learning new classes using prior knowledge (base vs. new classes performance). Could you comment on how this sub-field could relate to the proposed method?\n\nClarifications:\n- Could the authors clarify the exact dimensionality of the prompt used in each of the baselines as well as PromptFolio?\n- Could the authors mention if there are previous works that have proposed a 2-level prompt learning idea? (since Section 2.2 L 107 mentions that an exploration of such a idea and the cooperation between prompts is sparse)\n- In Section 4. L. 224, the authors mention that \"task-relevant coefficients can be directly added\". Could you clarify this statement?\n- Could the authors highlight the data setting used to generate Table 1?\n- Could the authors explicitly state the train and test conditions (settings) used within the experiments?\n\nSuggestions:\n- Since the proposal of PromptFL, there are have been further advances to the idea of prompt learning within the federated setting. Could the authors comment and draw comparisons against the more recent methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment. We will provide a detailed, item-by-item response to these concerns.\n> Abstract/ Conceptual Question 1: Could the authors comment about the few-shot learning aspect of foundation models (+prompt learning) and how the optimal mixing coefficient would be affected in that scenario?\n\nTo address your concern, we conducted additional experiments with 16, 8, 4, 2, and 1-shot learning settings. The results are shown in **Figure R1** in the PDF. These results demonstrate that the **optimal coefficients are consistent across different number of shots**, indicating the robustness of our method in few-shot learning scenarios. \n\n> Abstract/ Conceptual Question 2: An adjacent sub-field that contains foundation models and federated learning is the study of their impact in learning new classes using prior knowledge (base vs. new classes performance). Could you comment on how this sub-field could relate to the proposed method?\n\nThank you for your suggestion. Our method can be applied to this subfield and address the high adaptation ability. In our setting, the global prompt retains global information (common prior knowledge), while the local prompt incorporates local information (novel class information). The PromptFolio mechanism allows the **aggregation of global prompts to demonstrate good generalization ability** and helps the **local prompt quickly adapt to novel classes or unique data**.\n\n> Clarification 1: Could the authors clarify the exact dimensionality of the prompt used in each of the baselines as well as PromptFolio?\n\nEach prompt in our method has 8k parameters, consisting of 64 tokens, each with 128 dimensions. We use two prompts, totaling 16k parameters. In comparison, other baselines, including CoOp, PromptFL, and the variants of PromptFL, use prompts with 8k parameters.\n\n> Clarification 2: Could the authors mention if there are previous works that have proposed a 2-level prompt learning idea? (since Section 2.2 L 107 mentions that an exploration of such a idea and the cooperation between prompts is sparse)\n\nWe are afraid the sentence in Section 2.2 L 107 may lead to some confusion. In fact, what we want to express is that **\"the theoretical analysis of federated prompt learning is sparse.\"** The absence of such an analytical framework prevents us from illustrating how two prompts cooperate.\n\nRegarding previous works, there is a similar work, FedOTP, which used two prompts combined with optimal transport for federated learning. But this approach also lacks theoretical support. Our paper aims to fill this gap by proposing the **first analytical framework for federated prompt learning**, specifically showing how constructing PromptFolio can enhance performance. \n\n> Clarification 3:  In Section 4. L. 224, the authors mention that \"task-relevant coefficients can be directly added\". Could you clarify this statement? \n\nIn our theoretical framework, \"task-relevant coefficients can be directly added\" means that **task-relevant coefficients are proportional to the calculation of learnable parameters**. This statement comes from Theorem 4.3, where we demonstrate that the test performance conforms to a Gaussian distribution. Here, the mean and variance of the Gaussian distribution are task-relevant coefficients and task-irrelevant coefficients, respectively. Since the Gaussian distribution can be additive, so can the task-relevant coefficients.\n\nFor example, if we have a learnable prompt $p = \\theta p_1 + (1-\\theta)p_2$, and $\\alpha_1$ and $\\alpha_2$ are corresponding task-relevant coefficients, then the task-relevant coefficient of $p$ is  $\\alpha = \\theta \\alpha_1 + (1-\\theta) \\alpha_2$. \n\n> Clarification 4 & 5: Could the authors highlight the data setting used to generate Table 1? Could the authors explicitly state the train and test conditions (settings) used within the experiments?\n\nThe experimental setting for Table 1 is as follows: we used the Dirichlet distribution with a parameter of $\\alpha = 1$ and 16-shot training over 100 communication rounds. The training batch size was 32, and the testing batch size was 100. The learning rate was 0.001, and we used ViT-B/16 as our vision backbone. \n\nThe detailed experimental settings can be referred to in our code and will be added in the final version.\n\n> Suggestions 1: Since the proposal of PromptFL, there have been further advances to the idea of prompt learning within the federated setting. Could the authors comment and draw comparisons against the more recent methods?\n\nWe did analyze several recent prompt-based federated learning methods in Section 2.1, Lines 80-89, including FedPrompt, PromptFL, pFedPrompt, and pFedPG. Additionally, we conducted experiments comparing our approach with FedTPG, PromptFL, and its variants.\n\nIt is worth mentioning that compared to these methods, our paper focuses **on proposing a theoretical framework for federated prompt learning** and **uses a simple but efficient algorithm** with the idea of portfolio optimization **as an example** to demonstrate the usage of our theoretical framework.\n\n> Limitations: However, I would recommend drawing more broad comparisons as to potential missing pieces in comparison to more recent work as well as the finer points from the feature learning theory that don't fully match up to the settings in the FL system.\n\nThank you for your suggestion. We've conducted experiments with more users, different shot numbers, and different backbones, and compared our results with FedTPG. The experimental results can be referred to in the PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cb9x5V9ivn", "reviewer": "Reviewer_XM5m", "summary": "This paper studies theoretical properties of prompt based federated learning methods for visual language models. Following the proposed theoretical results, a new algorithm named Global-local Prompt Portfolio for Federated Learning (PromptFolio) was proposed. The proposed algorithm was examined in image classification tasks using a CLIP model.", "questions": "In the algorithm, each client sends prompts to a server. How do assure privacy protection in this case? \n\nIn the analyses, CLIP was utilized as a foundation model. However, the main claim of the paper is providing a generic framework for a more general category of visual language foundation models. Therefore, the proposed method should be examined using the other visual language models as well.\n\nMoreover, CLIP was used for image classification tasks, with limited prompts. In order to show the generalization of the proposed method for prompts with different structures, additional results should be given for the other tasks, such as image captioning and scene description, as well.\n\nIn Fig. 2.b, the best accuracy is achieved for 20 users, and the lowest accuracy is achieved for 5 users. Could you please elaborate this result? Could you please provide additional analyses for a larger number of users as well?", "limitations": "Limitations were partially addressed.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Analyzing prompt-based federated learning methods with feature representations and exploiting their relation to portfolio optimization is a nice idea. The proposed analyses also led to a new algorithm for prompt based federated learning.", "weaknesses": "Experimental analyses are limited and should be improved as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your feedback and inquiries. We're thankful that you consider our examination of the connection between federated prompt learning and portfolio optimization to be \"a nice idea\".\n\n> Question 1: In the algorithm, each client sends prompts to a server. How do assure privacy protection in this case?\n\nThank you for pointing out this concern. Actually, prompt learning can protect privacy during federated learning. It is worth mentioning that the prompt here is a **sequence of dense vectors** sent to the text encoder with only 8K parameters. This small number of parameters make it **difficult to recover the original data information**. In our proposed method, the prompts that encapsulate information in a **highly abstract form** are sent to the server while the sensitive data are kept on the local client. Thus, there are no privacy concerns in prompt learning.\n\n> Question 2: In the analyses, CLIP was utilized as a foundation model. However, the main claim of the paper is providing a generic framework for a more general category of visual language foundation models. Therefore, the proposed method should be examined using the other visual language models as well.\n\nWe appreciate the reviewer's feedback. Since we aim to propose a theoretical analysis framework for prompt tuning, we **used CLIP as an example**. To demonstrate that our model is not limited to a specific structure, we have **replaced the ViT backbone with ResNet** in our experiments. The results are shown in **Figure R2** in the PDF. It's observed that the results with ResNet **also confirm our theoretical analysis**. Furthermore, we plan to include experiments with additional visual language models such as ALIGN in our future work.\n\n> Question 3: Moreover, CLIP was used for image classification tasks, with limited prompts. In order to show the generalization of the proposed method for prompts with different structures, additional results should be given for the other tasks, such as image captioning and scene description, as well. \n\nThank you for your suggestion. Firstly, it is worth mentioning that our paper mainly focuses on proposing the **first theoretical framework for federated prompt learning with feature learning theory**. We use the classification task as an example due to the following reasons:\n\n- Most prompt fine-tuning under VLM, as demonstrated in CoOp and PromptFL [40, 16], **only conduct experiments on classification tasks**.\n- Also, in the feature learning community, the analysis mainly **focused on classification tasks** [1, 6, 7, 18]. Thus, the **analysis of image captioning and scene description** is open and can be categorized as **new work**.\n- Most federated learning on VLM research [15, 37, 27] **only conduct experiments on classification tasks**. Exploring federated learning for tasks such as image captioning and scene description can also be addressed in separate work.\n\nFrom these aspects, we **use a classification task to demonstrate the application of our theoretical framework** in this paper.\n\n> Question 4:  In Fig. 2.b, the best accuracy is achieved for 20 users, and the lowest accuracy is achieved for 5 users. Could you please elaborate this result? Could you please provide additional analyses for a larger number of users as well?\n\nThe differences in accuracy, where 20 users achieve the highest accuracy and 5 users the lowest, are influenced by our **no-repeat data partitioning** strategy under a **few-shot learning scenario**.\n- Fewer users: Due to few-shot learning setting, each user receives a fixed number of samples. So **more users lead to more data**, facilitating better model training and higher accuracy.\n- More users: Due to the no-repeat partition strategy, the fixed total dataset size results in **less data per user**, decrease the learning efficiency and accuracy.\n\n\nFurther experiments with 100 users is shown in **Figure R3** in the PDF. From this result, we observe that the performance of 100 users is lower than 50 users which also confirmed this phenomenon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGwWs0tLPY", "reviewer": "Reviewer_kpWj", "summary": "This paper discusses the integration of pretrained vision-language foundation models, such as CLIP, into federated learning. The idea is to use prompt-based federated learning to minimize the communication and computational costs. The authors go into the theoretical analysis to understand the performance of this approach", "questions": "See above", "limitations": "See above", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors provide a theoretical understanding on prompt Federated Learning for models like CLIP.\n- The results demonstrate that the combination of global and local prompts can support increased accuracies. \n- The overall investigation of this space is interesting and useful for the community", "weaknesses": "- While the overall approach is quite interesting, the authors restrict their investigation in a much simpler classification task. It is unclear why a larger foundation model is required for simpler classification tasks, where traditional FL techniques on vanilla models (e.g., reset, mobileNet etc) might work better towards learning new unseen distributions that a foundation model might not have seen (e.g,. x-ray hospital images etc). \n- Following the previous comment, it is unclear how this method would work on unseen distributions that most FL applications are mostly useful for. Most FL models are quite good at generalising to data that can be found on the public internet, but it is unknown how well they would work on out-of-distribution private data. \n- Given the fact that there are a few related prompt-based FL methods (e.g., promtFL, FedPrompt, etc) limit a bit the novelty of this work to mostly providing a solid theoretical analysis. \n- There might be some privacy implications from this work, as sensitive data are shared. \n- The evaluation was done with mostly 10 (up to 50) users, whereas in typical FL scenarios we want to learn from million of users. It is unclear how these results would generalise. Overall, the evaluation is a bit simplistic, showing only accuracy results on a few tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our approach and consider that our paper is \"useful for the community\". Below, we address each of your points in detail.\n\n> Q1(1): While the overall approach is quite interesting, the authors restrict their investigation in a much simpler classification task. \n\nFirstly, it is worth mentioning that our paper mainly focuses on proposing the **first theoretical framework for federated prompt learning with feature learning theory**. We use the classification task as an example due to the following reasons:\n\n- Most prompt fine-tuning under VLM, as demonstrated in CoOp and PromptFL [40, 16], **only conduct experiments on classification tasks**. \n- Also, in the feature learning community, the analysis mainly **focused on classification tasks** [1, 6, 7, 18]. \n- Most federated learning on VLM research [15, 37, 27] **only conduct experiments on classification tasks**.\n\nFrom these aspects, we use a classification task to demonstrate the application of our theoretical framework in this paper.\n\n\n> Q1(2): It is unclear why a larger foundation model is required for simpler classification tasks, where traditional FL techniques on vanilla models (e.g., reset, mobileNet etc) might work better towards learning new unseen distributions that a foundation model might not have seen (e.g,. x-ray hospital images etc).\n\nTo address the first question, we further clarify why we perfer foundation models with prompt learning rather than vanilla models in federated learning.\n- Foundation models with prompt learning **address the concern of communication cost** in federated learning. In a general prompt learning setting, only **8k** learnable parameters are used, which is a **very small scale**, while the performance of foundation models can be maintained and observed at a high level. Here, large foundation models will not increase the boundary of communication but will actually decrease it.\n- Prompt learning can **address the privacy concern** during federated learning. It is worth mentioning that the prompt here is a sequence of dense vectors sent to the text encoder with only 8k parameters. This small number of parameters make it **difficult to recover the original data information**. In our proposed method, the prompts that encapsulate information in a highly abstract form are sent to the server while **the sensitive data are kept on the local client**. \n\n\n> Q2: Following the previous comment, it is unclear how this method would work on unseen distributions that most FL applications are mostly useful for. Most FL models are quite good at generalising to data that can be found on the public internet, but it is unknown how well they would work on out-of-distribution private data.\n\nThank you for these comments. **Our federated prompt learning method has good generalization ability inherently because of the integration of the large vision-language foundation models.** This has been verified in some recent works, such as **FedTPG** [a] and **FedGPG** [12] that are published at ICLR 2024 and ICML 2024, respectively. These two works both show that federated prompt learning benefits from the **generalizable representation of the foundation models** and has very **robust performance** in various out-of-distribution (OOD) scenarios, such as local, base, and novel settings.\n\nSince our paper aims to develop the first theoretical framework for federated prompt learning, the experiments of OOD scenarios are out of scope.\n\n[a] Qiu C, Li X, Mummadi C K, et al. Text-driven Prompt Generation for Vision-Language Models in Federated Learning, ICLR 2024.\n\n> Q3: Given the fact that there are a few related prompt-based FL methods (e.g., promtFL, FedPrompt, etc) limit a bit the novelty of this work to mostly providing a solid theoretical analysis.\n\nThank you for pointing out this aspect. We'd like to mention that our work mainly **focuses on proposing the first theoretical framework**. This theoretical framework introduces feature learning theory, decomposing the learnable prompts into task-relevant and task-irrelevant features. By a thorough analysis of the dynamics of feature coefficients, we demonstrate the **convergence and generalization of federated prompt learning in Theorem C.5 and Theorem D.5** by evaluating the ratio between the coefficients of task-relevant and task-irrelevant features. Besides, we leverage the **idea of portfolio optimization to derive our PromptFolio algorithm**. This simple but efficient algorithm exemplifies the practicality of our analytical framework, which can be extended to further prompt-based FL methods.\n\n> Q4: There might be some privacy implications from this work, as sensitive data are shared.\n\nPrompt learning doesn't have privacy issue. Please refer to more analysis in the answer of Q1(2). \n\n> Q5: The evaluation was done with mostly 10 (up to 50) users, whereas in typical FL scenarios we want to learn from million of users. It is unclear how these results would generalise. Overall, the evaluation is a bit simplistic, showing only accuracy results on a few tasks.\n\nThank you for your valuable comment. We need to mention that we have not seen any federated learning papers that conduct experiments with millions of users. Instead, most research papers use fewer than 100 users due to dataset limitations. During the rebuttal, we have added a new experiment with 100 users and presented the result in **Figure R3 in the global response PDF**. The experimental results with 100 users are consistent with the previous results.\n\nBesides, our paper mainly focuses on providing a theoretical feature learning framework for federated prompt learning. **As a theoretical paper, 100 users are large enough to demonstrate the practicality of our theoretical framework**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lW5sMOYQGk", "reviewer": "Reviewer_YEHg", "summary": "In this papaer, the authors proposed a method to finetune vision-language model (CLIP) via prompt learning. Specifically, the author optimize a global textual prompt which will be optimized via FedAvg, while each client additionally optimize a local textual prompt. The proposed method is a combination of the existing methods CoOp and PromptFL and outperforms both methods.", "questions": "Please see the questions listed in the weakneses section.", "limitations": "The authors do not discuss the limitation of the method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The theoretical analysis of the proposed method is thorough and clear.", "weaknesses": "1. The proposed method is basically a combination of both CoOp and PromptFL, or to be more specific, optimizing both client-specific and client-agnostic textual prompt. The novelty of the proposed needs to be further highlighted in the main paper.\n2. The combination strategy seems to be a bit straight forward, i.e., using a single scalar value $\\theta$, the author should discuss other possibilities.\n3. The authors should compare with other prompt-based tuning methods for FL, e.g., [1][2]. Also, adding more ablation studies could be beneficial.\n\n[1] Guo T, Guo S, Wang J. Pfedprompt: Learning personalized prompt for vision-language models in federated learning[C]//Proceedings of the ACM Web Conference 2023. 2023: 1364-1374.\n\n[2] Qiu C, Li X, Mummadi C K, et al. Text-driven Prompt Generation for Vision-Language Models in Federated Learning[J]. arXiv preprint arXiv:2310.06123, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for mentioning that our theoretical analysis is \"thorough and clear.\"\n\n> Weakness 1: The proposed method is basically a combination of both CoOp and PromptFL, or to be more specific, optimizing both client-specific and client-agnostic textual prompt. The novelty of the proposed needs to be further highlighted in the main paper.\n\nWe wish to emphasize that our paper mainly focuses on developing **a theoretical framework for federated prompt learning**. Our theoretical framework introduces feature learning theory, decomposing the learnable prompt into task-relevant and task-irrelevant features. By a thorough analysis of the dynamics of feature coefficients, we demonstrate the **convergence and generalization of federated prompt learning in Theorem C.5 and Theorem D.5** by evaluating the ratio between the coefficients of task-relevant and task-irrelevant features.\n\nUnder this theoretical framework, we further construct the portfolio optimization method for federated prompt learning, i.e., PromptFolio, and **derive an analytical solution based on this theoretical framework**. PromptFolio seeks to balance the global consensus (PromptFL) and local personalization (CoOp) of federated prompt learning. This simple but efficient method can even outperform many SOTA federated prompt learning methods, such as PromptFL, PromptFL+FedProx, PromptFL+FedAMP, FedTPG, etc. The novelty will be highlighted in the final version.\n\n> Weakness 2: The combination strategy seems to be a bit straight forward, i.e., using a single scalar value $\\theta$, the author should discuss other possibilities.\n\n**The combination strategy:** We would like to clarify the portfolio optimization is not a simple combination. Instead, this is an **analytical solution under the feature learning theoretical framework**. Our feature learning framework evaluates the performance of prompt fine-tuning by considering the ratio of task-relevant to task-irrelevant factors. We draw an analogy with portfolio optimization, where the precise fine-tuning of the combination ratios of various assets can maintain returns while mitigating risk. By employing this simple yet effective algorithm, we demonstrate the practical application of our theoretical framework. It's important to note that in portfolio optimization, **adjusting the scalar value of each asset significantly impacts the performance**, making the fine-tuning of mixing ratios and the determination of optimal coefficients a non-trivial task.\n\n**Other possibilities:** Also, our theoretical framework is flexible and can be **extended to incorporate more complex combination strategies**. For example, some optimal transport-based algorithms can be a possible way to construct the combination of global and local prompts. \n\n> Weakness 3: The authors should compare with other prompt-based tuning methods for FL, e.g., [1][2]. Also, adding more ablation studies could be beneficial.\n\n**Further comparisons**: We've added the comparison results between our PromptFolio algorithm and FedTPG [2]. The results are listed in **Table R1 in the PDF** in the global response. Since the code for pFedPrompt [1] is not available online, we were unable to include a comparison with it due to time limitations. It is worth mentioning that our algorithm demonstrates good performance.\n\n**More ablation studies**: Additionally, we have included ablation studies on more users, different shot numbers, and different backbones. The further experimental results can also be referred to in the PDF in the global response.\n\n> Limitations: The authors do not discuss the limitation of the method.\n\nClarification: Actually, **the limitation was discussed at the end of the Conclusion**. We mentioned that the limitation of this work lies in the use of a simplified text encoder with a single activation function. We hope that the development of theoretical tools will provide future work for more practical and complex models to capture the behaviors of foundation models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and effort in evaluating our work and providing constructive comments. We appreciate that the reviewers consider our theoretical framework for federated prompt learning to be "pretty solid" (BdA2), "thorough and clear" (YEHg), and "novel" (bKME). The relation between prompt mixing and portfolio optimization is viewed as "novel" (BdA2) and a "nice idea" (Xm5m). The ablation study towards data heterogeneity and client numbers is particularly informative (BdA2). We are further glad that the reviewers agree our work is interesting and useful for the community (kpWj).

In the following, we will try to address some common concerns/questions of the reviewers and provide a detailed item-by-item response to these concerns.

> Q1: The novelty of this paper.

Our paper mainly focuses on developing **a theoretical framework for federated prompt learning**. This theoretical framework introduces feature learning theory, decomposing the learnable prompts into task-relevant and task-irrelevant features. By a thorough analysis of the dynamics of feature coefficients, we demonstrate the **convergence and generalization of federated prompt learning in Theorem C.5 and Theorem D.5** by evaluating the ratio between the coefficients of task-relevant and task-irrelevant features.

Under this theoretical framework, we further construct the portfolio optimization method for federated prompt learning, i.e., PromptFolio, and **derive an analytical solution within this theoretical framework**. PromptFolio seeks to balance the global consensus and local personalization of federated prompt learning. This simple but efficient method can even outperform many SOTA federated prompt learning methods, such as PromptFL, FedTPG, PromptFL+FedProx, PromptFL+FedAMP, etc. 

> Q2: How do prompt learning assure privacy protection?

Actually, prompt learning can protect privacy in federated learning. It is worth mentioning that the prompt here is a sequence of dense vectors sent to the text encoder with only 8K parameters. This small number of parameters make it **difficult to recover the original data information**. In our proposed method, the prompts that encapsulate information in a highly abstract form are sent to the server while the **sensitive data are kept on the local client**. Thus, there are no privacy concerns in prompt learning.

> Q3: Why you use classification task as an example?

Classification, as the most popular tasks, is widely used for a machine learning theoretical paper. In addition, most existing works in the area we consider only test on classification tasks.

- Most **prompt fine-tuning under VLM**, as demonstrated in CoOp and PromptFL [40, 16], only conduct experiments on classification tasks. 
- Also, in the **feature learning community**, the analysis mainly focused on classification tasks [1, 6, 7, 18]. 
- Most **federated learning on VLM** research [15, 37, 27] only conduct experiments on classification tasks.
--- 
**Illustration of the attached PDF file:**

The attached PDF file includes one table and three figures.

Table R1 compares our PromptFolio with other state-of-the-art algorithms, including a comparison with FedTPG as requested by Reviewer YEHg. This table shows that PromptFolio performs well across various datasets. The experiments were conducted using a Dirichlet distribution with $\alpha = 0.3$ over 100 communication rounds. We used the ViT-B/16 backbone, with a training batch size of 32 and a test batch size of 100.

Figure R1 presents experimental results for different numbers of shots, following Reviewer bKME's suggestion. We conducted experiments with 16, 8, 4, 2, and 1 shots, demonstrating that the optimal coefficients are consistent across different shot numbers, indicating the robustness of our method in few-shot learning scenarios.

Figure R2 shows experimental results for different backbones, as per Reviewer XM5m's recommendation. We tested different backbones, including ViT-B/16 and ResNet-50. The results confirm our theoretical analysis, validating the application of our theoretical framework.

Figure R3 includes experiments on different users. We've added results for 100 users according to Reviewer XM5m and kpWj. The experiment demonstrates the consistency of our theoretical framework when applied to a larger number of users.

Meta-review: The paper proposes a theoretic framework for understanding prompt-based FL based on feature learning theory [1].
They then draw an analogy to portfolio optimization from the finance field and prove that a mixture of global and local prompts can balance between generalization and personalization.

The reviewers generally agreed that the connection to feature learning theory and the resulting theoretic method was novel and insightful. One of the key challenges was appropriately explaining feature learning theory for new readers and properly framing the main contributions. In the revision, please include better introduction, background, and intuition for the feature learning theory to aid new readers.

There were some additional minor concerns that this was only evaluated on the classification task but this seems reasonable given that the paper aims towards theoretic contributions primarily. Given the confusion, I would recommend revision of the paper to more carefully emphasize the theoretic focus of the paper and deemphasize the algorithmic novelty.

Overall, the paper provides theoretic insight into the federated prompt learning. This theoretic insight is used to develop a simple but effective strategy for combining local and global prompts. Because this paper adds some theoretic grounding for federated prompt learning, the paper provides a valuable contribution to the NeurIPS community. 

[1] Allen-Zhu, Z., & Li, Y. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. In The Eleventh International Conference on Learning Representations.
Decision: Accept (poster)