Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

Authors: bikang  pan
wei  huang
ye  shi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:13:17

Abstract: Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Main paper: 
 introduction: 
 Federated learning (FL) [33] stands out as a powerful framework that enables machine learning over decentralized data while maintaining data privacy and reducing reliance on centralized data repositories. Despite its advantages, the intensive computational and communication demands during the training phase often constrain the scalability of the models utilized. A transformative advancement in this field is the integration of prompt-based learning within the federated framework [42, 17, 13] .Prompt learning adapts models such as Contrastive Language-Image Pretraining (CLIP) [38] through minimal modifications, typically in the form of prompts or cues, guiding the model's predictions. As an emerging idea in machine learning, it has shown significant promise in various applications by allowing models to perform specialized tasks without undergoing complete retraining. Prompts effectively adjust pre-trained models for new tasks or datasets, which is particularly valuable in federated environments where data privacy and bandwidth constraints often limit conventional training methods. A prominent example is CoOp [42] , with its streamlined federated version known as PromptFL [17] .Despite the empirical success of prompt-based federated learning [28, 17, 16] , theoretical analysis in this area remains limited. In this paper, we use feature learning theory [1] to propose an analytical framework for prompt-based federated learning with vision-language foundation models. Feature learning theory divides data into task-relevant and task-irrelevant features, allowing learnable weights to be expressed as a linear combination of these features. By introducing feature alignment across bimodal pretrained models, we demonstrate the relationship between learnable prompts and the features in latent space. To understand the process of signal learning and noise memorization, we use a two-stage analysis to examine the dynamics of coefficients for task-relevant and task-irrelevant features. Leveraging these coefficients, we demonstrate that prompt learning can be evaluated by comparing the ratio of task-relevant to task-irrelevant coefficients. In this way, we establish a theoretical foundation for prompt-based federated learning.Additionally, we treat task-relevant coefficients and task-irrelevant coefficients as income and risk in investment portfolios [31, 32] . Inspired by investment portfolios, where combining two independent assets can reduce risk, we introduce two learnable prompts: a global prompt and a local prompt, and simply mix them to form a prompt portfolio to balance generalization and personalization. By mixing them to create a prompt portfolio, we balance the generalization and personalization under severe data heterogeneity. Leveraging the analysis framework we proposed, we prove the performance advantage of this mixed algorithm and derive the optimal coefficient. Besides, we use comprehensive experiments to support our theoretical results.In this paper, our primary contributions are threefold:• We present an analytical framework for prompt-based federated learning using visionlanguage foundation models. This framework aligns text and image features into a shared latent feature space and utilizes a two-stage analysis to understand the dynamics of prompt learning. By tracking the progression of signal learning and noise memorization, we show that the effectiveness of prompt-based federated learning can be measured by the ratio of task-relevant to task-irrelevant coefficients.• Additionally, we introduce a prompt portfolio mechanism to address severe data heterogeneity and balance generalization and personalization. Within our proposed analytical framework, we draw an intuitive analogy between task-relevant features and income in portfolio optimization, as well as task-irrelevant features and risk. Consequently, we demonstrate that the combination of prompts performs better than using a single prompt, and we provide the optimal mixing ratio.• The theoretical result has been empirically validated through rigorous experiments. Our results not only align with theoretical predictions but also consistently demonstrate the practical superiority of our approach in real-world scenarios.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: As stated in the abstract and introduction, this paper provides the first theoretical framework to analyze the dynamics of prompt learning federated learning from pretrained vision-language foundation models. These claims are reflected throughout the paper, with theoretical analysis and experimental validation provided in support. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In this section, we examine prior research that serves as the basis for our study. Our investigation is divided into two primary areas: prompt-based federated learning and feature learning theory. Together, these fields create a complete context for our contributions.Prompt learning, initially developed in the field of natural language processing, has expanded its reach to vision language models. Examples include the CLIP model [38] , which originally utilized manually engineered templates. However, more recent advancements have shifted towards developing prompts within a continuous embedding space. Innovations like CoOp [42] have refined the CLIP model by introducing continuous prompt vectors, sparking a surge of studies aimed at enhancing the efficiency of prompt learning and providing a solid foundation for further investigation [12] .To enhance the integration of global data and address challenges in scenarios with limited user data, FedPrompt [41] and PromptFL [17] have effectively integrated the concept of prompt learning with federated learning [4, 5, 27, 37, 23] . To tackle the statistical heterogeneity often found in client data, pFedPrompt [16] introduces a non-parametric approach, providing each client with a personalized attention module. This module is designed to refine spatial visual features to better align with the unique characteristics of local data. Concurrently, pFedPG [40] introduces a novel prompt generator located at the server, which customizes prompts for each client, thus enhancing the personalization of the federated learning process. Additionally, a recent study, FedOTP [28] , leverages optimal transport theory to improve the balance between achieving global consensus and allowing for local customization through a strategic combination of global and local prompts.However, there is limited theoretical analysis of federated prompt learning. For instance, the theoretical examination of the CLIP model by [8] enhances prompt learning theory by exploring how CLIP learns transferable representations across various modalities and improves zero-shot transfer performance with a recently developed regularization technique. However, to the best of our knowledge, no research analyzes prompt learning within a federated setting that elucidates the cooperation between prompts. In this paper, we analyze how different prompts interact with feature learning theory and demonstrate the provable benefits of cooperation between global and local prompts.Feature learning theory [1] has revolutionized our understanding of how machine learning models learn and represent information. Unlike other theories, feature learning accommodates substantial weight updates during gradient descent, enabling the network to uncover complex patterns in data.Feature learning theory has been successfully applied to various neural network architectures, including convolutional neural networks [6, 24] , graph neural networks [18] , and vision transformers [22, 26] . Moreover, feature learning theory has been used to analyze different training algorithms, such as Adam [43] , momentum [26] , out-of-distribution learning [7] , and Mixup [44, 10] . Notably, [19] have analyzed the convergence and generalization in general federated learning. Despite progress in feature learning theory, the study of federated prompt learning is sparse. Our work uniquely addresses this gap by analyzing feature learning under prompt-based federated learning, providing crucial insights for their effective adaptation and optimization in such contexts.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitation analysis is provided in the Conclusion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Notation In our notation, vectors are represented by lowercase bold letters, matrices by uppercase bold letters, and scalars by regular, non-bold letters. The ℓ 2 -norm of a vector v is indicated as |v| 2 .The spectral norm of a matrix A is denoted by |A| 2 , and its Frobenius norm by |A| F . To compare the growth or decline of two sequences, we use standard asymptotic symbols like O(•), o(•), Ω(•), and Θ(•), which describe their behavior as they approach infinity. We introduce notations Õ(•), Ω(•), and Θ(•) to obscure logarithmic factors within these expressions. Notably, 1(•) represents the indicator function. Lastly, we represent sequences of integers as [n] = {1, 2, . . . , n} and sequences of elements such as vectors can also be represented similarly v[n] = {v 1 , v 2 , . . . , v n }.In this part, we demonstrate how to fine-tune a learnable text prompt under a vision language pretrained model. Here, we consider the classification task, where we assume that we have an image x.The objective is to correctly classify the image into class y ∈ [C], where the total number of classes is C. From the vision language pre-trained model, we expect that the latent spaces of the text encoder and image encoder are aligned. Thus, when we input different prompts, the text feature generated by the correct prompt will have the highest similarity with the image feature. We input a learnable prompt p and a fixed class prompt p c ∈ {p 1 , • • • , p C }, which correspond to the classes, into the text encoder h. This process generates the text feature for class c: h c = h(p, p c ). On the other hand, the image feature g is generated by the image encoder g: g k,i = g(x k,i ) ∈ R m . We define the similarity function between the image feature g and the text feature h as ρ := [ρ c ] = sim(g, h c ).The training process mirrors traditional classification tasks, where the objective loss ℓ(ρ, e y ) is the distance between the similarity vector and the actual label y. Here, ℓ represents the loss function that measures the distance between two vectors, and e y is the one-hot vector derived from the ground truth label y.To illustrate the prompt-based federated learning framework, consider a federated system with a central server and K clients. Assume client k has n k training samples: {x k,i , y k,i } n k i=1 . The learnable prompt in client k is denoted as p k and the learnable prompt is aggregated at each communication round.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
analysis framework for prompt-based federated learning: 
 In this subsection, we present the analysis framework for prompt-based federated learning from vision-language pre-trained models. The central concept of this framework is the alignment of features between text and image encoders in the vision-language pre-trained model. To achieve this, we link the text encoder and image encoder through a shared latent feature space, described as follows.Feature representation and client distribution Inspired by [6, 24, 19] , we expect that the latent feature space will contain task-relevant and task-irrelevant features. In federated learning settings, the task-relevant features can be categorized into global task-relevant features µ G and local taskrelevant features ν 1 , • • • , ν S , where S is the length of local task-relevant features. Additionally, the task-irrelevant features can be listed as ξ 1 , • • • , ξ L , where L is the length of task-irrelevant features.Here, we assume that the dimension of the latent space is m and these features µ (•) , ν (•) , and ξ (•) are elements of R m . For simplicity, we assume that these features are orthogonal to each other. In our theoretical examination, we address a binary classification scenario with y k,i ∈ {+1, -1}. We consider a scenario with K clients, each linked to a distribution D k , ∀k ∈ [K]. Initially, we choose the signal vector µ k for client k by sampling from P (ν 1 , ν 2 , . . . , ν S ), where P represents a discrete distribution that assigns probabilities to each local task-relevant feature vectorµ k := ν s , s ∈ [S].Text encoder Here, we suggest coupling the learnable prompt and the class prompt and propose the structure of the text encoder. By adopting a similar setting as [39] , we suppose the generation of the text feature can be written as follows:EQUATIONwhere W ∈ R m×mp is the weight matrix, and p y k,i ∈ R mp is the prompt linked to a ground truth class. In this definition, the introduction of Wp y k,i introduces nonlinearity between the trainable prompt and the class prompt while keeping the overall function nonlinear. Note that for a binary classification problem, the vector p y k,i takes p 1 when y k,i = 1 and p -1 when y k,i = -1. To reveal the properties of the text encoder in the pre-trained model, we assume that the weight matrix W is composed of the following rows:EQUATIONThe assumption of the weight matrix is inspired by [20] , and the evidence of this assumption is listed in the Appendix D. In our analysis framework, we adapt FedAvg [33] as our prompt aggregation algorithm, which is named PrompFL [17] . The aggregation formula is given by:EQUATIONwhere n := k n k denotes the total number of data samples across all clients.Image encoder Let us consider the image network, represented as g k,i = g(x k,i ) ∈ R m . We assume that the image encoder also aligns the feature space of the text encoder within the pre-trained model. As a result, we assume the image feature generated by data x k,i in client i can be expressed as follows:EQUATIONwhere x k,i,l ∼ N (0, σ 2 p ), ∀l ∈ [L] represents the coefficient of task-irrelevant terms in the data, and σ 2 p is the variance. This assumption implies that task-relevant features vary based on whether the label is positive or negative, whereas task-irrelevant features persist as arbitrary and unrelated to the label's polarity. The similarity score between an image x k,i and class y k,i is expressed as sim(g k,i , h k,i ) = ⟨g k,i , h k,i ⟩. Moreover, the objective of the training loss is designed to enhance the resemblance between the image feature g(x k,i ) and the text feature created by the ground truth prompt p y k,i .Signal-noise decomposition Based on the above model, we introduce the signal-noise decomposition of the learnable prompt here. Note that the proofs of the following lemmas and theorems are listed in the appendix. Lemma 4.1 (Feature Representation). At the t-th iteration, the learnable prompt p (t) k for client k and the aggregated prompt p (t) can be rewritten as a linear combination of features and prompt initialization:EQUATION)where α (t)•,• are the coefficients of initialization, β• is the coefficient of global task-relevant features, γ (t)•,• is the coefficient of local task-relevant features, ϕ (t)•,• are the coefficients of task-irrelevant features, and the overlined coefficients are the averaged versions of the original coefficients.Here, since the learnable prompts can be written as a linear combination of the features, we can analyze the dynamics of these coefficients to understand the learning progress of the prompts. The normalized factor such as ||µ G || -22 is used to make the coefficient similar to the inner product of the prompts and the features,β (t) k ≈ ⟨p (t) k , µ G ⟩.Coefficient dynamics Inspired by previous studies [6, 24, 19] , we employ a two-phase analysis to track the dynamics of coefficients in prompt-based federated learning from vision-language foundation models. By analyzing the dynamics of the coefficients, we can obtain the feature learning procedure during training. This two-stage analysis allows us to establish the order of coefficients and explore how they are affected by the mixing parameter θ. For the theorem and proof of this analysis, please refer to Appendix F. Theorem 4.2 (Training Dynamics). There exists a total number of local updatesEQUATIONHere, n = k n k /K is the average number of data in each client, andSNR G = ||µ G ||/(σ p √ m), SNR k = ||µ k ||/(σ p √ m)denote the signal-to-noise ratio between the task-relevant feature and task-irrelevant feature. We defineχ k = K k ′ =1 ⟨µ k , µ k ′ ⟩/||µ k || 2 2 .Test performance evaluation with coefficients Here, we suppose the classification output of the i-th data in client k is the class corresponding to the highest similarity between the text feature and image feature, denoted as ŷk,i . To assess the algorithm's performance, we evaluate the error rate in the test procedure as our test loss L D :EQUATIONThe following theorem demonstrates that the test loss can be considered as the probability that a Gaussian random variable falls below zero, with the mean and variance influenced by the task-relevant and task-irrelevant coefficients. Theorem 4.3 (Test Loss). The expectation of test loss L D of an algorithm can be treated as the probabilityEQUATIONwhere µ and σ are functions of task-relevant and task-irrelevant coefficients, as defined in Appendix E.Drawing from this theorem and the properties of Gaussian distributions, an algorithm's performance can be evaluated by the ratio µ/σ. This ratio highlights the influence of task-relevant and taskirrelevant features on test loss. Specifically, a higher task-relevant coefficient coupled with a lower task-irrelevant coefficient typically leads to better performance.The Markowitz mean-variance model is a famous framework for assembling a portfolio of assets such that the expected return is maximized for a given level of risk [31, 32] . This model characterizes assets by their expected returns and risks. It claims that the return of the whole portfolio is a proportionally weighted combination of the assets' returns, and the risk of the whole portfolio is a function of the correlations of the component assets. According to the properties of task-relevant and task-irrelevant coefficients, the task-relevant coefficient can be directly added, and the task-irrelevant feature follows the additive property of Gaussian random variables. Thus, we connect the task-relevant coefficient to the return and the task-irrelevant feature to the risk. This connection provides insight that the combination of prompts, i.e., a prompt portfolio, will lead to a higher ratio of task-relevant features to task-irrelevant features. for each client k in parallel do for each sample (x k,i , y k,i ) in client k's data do Update p7: Compute g k,i ← g(x k,i ) 8: for c ← 1 to C do 9: Compute h k,i,c ← (1 -θ) • h(p (t) G,k , p c ) + θ • h(p (t) L,k , p c ) 10: Compute similarity ρ k,i,c ← sim(g k,i , h k,i,c(t+1) G ← K k=1 n k n p (t+1) G,k ▷ FedAvg to aggregate global prompt 17: t ← t + 1 18: end while 19: return p G , p L,k for all k 
promptfolio method: 
 The local learning process generates the local feature by including a specific local prompt, whereas the global learning process adopts a similar strategy but also uses FedAvg to compile learnable prompts from various clients. We enhance cooperation between the local and global learning processes by merging both local and global features to create the final text feature. The text feature is produced as follows: where θ ∈ [0, 1] serves as a coefficient to balance the mix of the two features, which addresses the balancing between personalization and generalization. The variation in the parameter θ influences the outcomes of the inference. Specifically, when θ = 0, the algorithm reverts to PrompFL [17] , whereas at θ = 1, it shifts to CoOp [42] . Our approach consists of combining these features and using the resulting mixed feature to determine their similarity. This feature is subsequently utilized to evaluate the similarity between text and image features. Note that this algorithm differs from typical personalized algorithms [30] as it focuses on integrating text features instead of adjusting training weights. The framework of PromptFolio is described in Algorithm 1.EQUATIONIn this part, we offer a theoretical demonstration of the performance advantage of PromptFolio and the selection of the optimal mixing coefficient θ. According to Theorem 4.3, each algorithm can be regarded as a Gaussian random variable. The test performance correlates with the ratio of task-relevant features to task-irrelevant features. This ratio enables us to analyze the test results of various learning algorithms. := µ k µ k = Θ( β k +γ k β k +γ k ) = Θ( KSNR G +KSNR k KSNR G +χ k SNR k ) and b := σ k σ k = O( ϕ k ϕ k ) = O(K)as the ratio of different coefficients. Here, the order of a and b depends on the coefficient derived in Lemma F.3. As a result, we have the following theorem, and the proof can be referred to in Appendix E.3. Theorem 5.1 (PromptFolio Advantage). The mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL:EQUATIONEQUATION     C a = (b -a)(b 2 + 2ρb + 1) C b = (a + b)(b 2 -1) -4b(ρb -1) C c = (b -1) (a + b) 2 (b + 1) 2 -8ab 2 (ρ + 1) 2 .The results discussed demonstrate how combining global and local text features enhances performance and illustrate the optimal way to balance personalization with generalization. Here, a and b are the ratio of coefficients and reveal how the global feature and local feature interact. Drawing on principles from portfolio optimization, which involves blending two assets that are not perfectly correlated, we can construct a portfolio that maximizes returns while minimizing risk. Given the characteristics of Gaussian random variables, we intuitively correlate the coefficient of task-relevant features with returns and the coefficient of task-irrelevant features with risk. Thus, the first part of Theorem 5.1 provides a rationale that a well-balanced portfolio of prompt features can significantly improve performance.Similar to the portfolio optimization problem, we can also derive the optimal mixing coefficient θ. Theorem 5.2 (Optimal Mixing Coefficient). The optimal mixing coefficient θ ⋆ followsEQUATIONTheoretically, if we further simplify the mixing coefficient with the order of a, b and ρ, then we get thatEQUATION)In this theorem, we note that a lower χ k indicates greater data heterogeneity, which lead to a higher θ ⋆ . This observation aligns with the intuition that, due to the non-i.i.d. distribution of data, the model should incorporate more local information, thereby making the optimal θ closer to 1.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: The code is available alongside the submission, including a ZIP file with all relevant scripts. We will make the code open access once the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we conduct experiments with the CLIP model to empirically demonstrate the performance advantages of PromptFolio. Specifically, the image network g and the text network h are components of a pre-trained CLIP model. By evaluating results obtained using various mixing coefficients across different datasets, data distribution, and client number, we align theory with practice. We use the Dirichlet distribution to manage data heterogeneity and employ FedAvg as the aggregation strategy. The experiment is conducted on the CIFAR-100 dataset by default, with the model trained for 10 epochs locally and the results evaluated over 100 communication rounds.In this section, we observe that the combination of global and local algorithms outperforms both the prompt-based federated learning with FedAvg and individual prompt learning approaches. Under the CLIP model setting, the global and local prompt learning algorithm degenerates to PromptFL and CoOp, respectively. To explore why this global-local collaboration is more effective than either approach alone, we evaluate the accuracy of various mixing coefficients θ across different datasets. We use CIFAR-100 [25] , DomainNet [36] , Office-Caltech10 [15] , OxfordPets [35] , and DTD [11] , adopting θ = 0.2 as the general mixing coefficient for our algorithm. The quantitative results are shown in Table 1 . From this table, it is evident that blending global and local prompts consistently leads to enhanced accuracy, with the accuracy curve also showing a peak. Further performance evaluations can be found in Appendix A. We then conduct the experiment over different data distributions. By varying the parameters of the Dirichlet distribution exponentially from 0.01 to 10, we controlled the heterogeneity of the data. A larger α indicates that the data is closer to an i.i.d. distribution. Using 10 users, we performed our experiments, and the results are shown in Figure 2(a) . From Figure 2 (a), we observe that hybrid approaches outperform solitary methods, consistent with our theoretical analysis. Additionally, higher α values, indicating a more uniform distribution, generally result in a superior global model compared to local models tailored for specific users. This finding supports our conclusion that a more IID distribution leads to χ k being higher and closer to K, resulting in a higher task-relevant to task-irrelevant coefficient ratio, and thus better performance.Global Local Global Local (a) (b)Conversely, in scenarios where data is highly non-IID, there is a preference for using more local models to maintain high accuracy, which aligns with our analysis of the optimal mixing coefficient θ ⋆ .Furthermore, we conducted experiments with different numbers of users on the CIFAR-100 dataset, keeping the Dirichlet distribution parameter fixed at α = 0.01, which represents a pathological non-i.i.d. distribution. The number of users varies from 5 to 100, and the results are shown in Figure 2 (b). From these results, we observe that the trend of the mixing strategy outperforming the independent global and local algorithms remains consistent, regardless of the number of users. As the number of users increases, the optimal θ shifts closer to zero, indicating that with more users, each client's information becomes less significant, necessitating more global information. Additionally, the accuracy first increases and then decreases with the number of users, suggesting that there is an optimal number of users, consistent with theoretical results.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper provides comprehensive details about the experimental settings, including descriptions of datasets used, hyperparameters, data splits, and other necessary details in Section 6 to understand the experiments fully.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 This work presents a thorough theoretical and empirical exploration of prompt-based federated learning, integrating vision-language foundation models such as CLIP. By developing an analytical framework based on feature learning theory, we have examined the dynamics of signal learning and noise memorization specific to federated settings, providing a robust mechanism to evaluate the effectiveness of prompt-based learning strategies. Notably, our introduction of PromptFolio, which combines global and local prompts into a prompt portfolio, offers an approach to balancing generalization with personalization, drawing an innovative parallel with portfolio optimization in finance. This approach balances generalization with personalization, supported by an optimal mixing coefficient from our theoretical framework to tailor adaptability in various federated settings. Empirical tests confirm our method's superiority, aligning with theoretical insights and outperforming traditional federated learning approaches. Limitations include a simplified text model with a single activation function, suggesting future work with more complex models to better capture deep network behaviors in federated environments.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The results section includes error bars and other statistical measures to indicate the significance and reliability of the experimental outcomes. This is particularly visible in the detailed tables and figures that compare different methods and settings.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:The paper now provides detailed information on the computational resources used for the experiments, including the type of GPUs, memory requirements, and the execution time for each setup in Section 6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Our paper primarily focuses on theoretical research in machine learning and does not address societal impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risk that have a high risk for misuse, so this question is not applicable. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The paper adequately credits original sources for assets such as datasets and models used in the experiments. It mentions using standard datasets and references the original works where these models and datasets were introduced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not introduce any new assets that would require additional documentation or licensing information. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The paper does not involve crowdsourcing or direct research with human subjects, so this question is not applicable. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: A In this paper, we adopt a similar setting to the experimental result in [28] . The experiments are conducted on a cluster with 2 Intel Xeon 5218R, 512GB memory, and 8 NVIDIA Tesla A40 GPUs 48GB. Here, we use Food101 [3] , DTD [11] , Caltech101 [14] , Flowers102 [34] , OxfordPets [35] to evaluate our algorithm. The baseline algorithms are CoOp [42] , PromptFL [17] , the variants of PromptFL [9, 29, 2, 21] and FedTPG [37] . We use Dirichlet distribution with parameter α = 0. This paper further conduct experiments on different shot numbers and different backbones. The results in shown as Figure 3 . Different shots As shown in Figure 3 (a), we tested the accuracy using different numbers of shots, ranging from 16-shot to 1-shot. It can be observed that the optimal coefficient remains stable and close to 0.2, demonstrating the robustness of our algorithm across various shot numbers.. Feature learning theory is a new theoretical framework proposed recently. The seminal work [1] proposed feature learning theory to understand ensemble, knowledge distillation, and self-distillation in deep learning. They show that when data has a structure containing features, the network's dynamics can be tracked during gradient training, which can further be used to characterize generalization. Later, this theory was further systematized and standardized by [7] , forming a fundamental theoretical framework to explain benign overfitting. Since then, feature learning theory has been widely used to understand algorithms [8, 22] and techniques [20] in deep learning, forming a comprehensive theoretical system.To provide more details on the general feature learning process:1. Defining the feature space: As outlined in Section 4, features are categorized as taskrelevant and task-irrelevant. For example, in an image of a cat, features representing the cat itself are task-relevant, while background features are task-irrelevant. 2. Parameter representation: As described in Lemma 4.1, learnable parameters can be decomposed into the feature space. In our framework, the learnable prompts are expressed as a linear combination of task-relevant and task-irrelevant features. 3. Learning dynamics: Theorem F.3 examines the learning dynamics of coefficients of feature learning, where coefficients are defined by the weight decomposition into the feature space, providing valuable insights into the learning process. When task-relevant features dominate, the network learns the target effectively and demonstrates good performance. 4. Generalization bound: By leveraging the learning dynamics of the coefficients, we can demonstrate the generalization bound or test performance post-training. In our work, we connect the performance of prompt fine-tuning with the ratio between task-relevant coefficients and task-irrelevant coefficients.Here's the evidence supporting the second assumption that the features of the pretrained model are orthogonal. We focus on the final projection layer of the text encoder in CLIP [38] and calculate its cosine similarity between each row. The statistical distribution of the cosine similarity is shown in Figure D .Figure 4 : The distribution of cosine similarity between different rows of the final projection layer in CLIP [38] .Here, we find that nearly all of the cosine similarity are smaller than 0.1, which is close to orthogonal. This evidence support our theoretical assumption. Except the evidence of cosine similarity, the previous research [20] also make assumption of orthonormal rows of weight matrix to the multimodality latent space.In this paper, our analysis will be made under the following assumptions:Assumption E.1. Suppose that:1. The dimension of latent space m is sufficiently large m = Ω(n).2. The number of training samples and clients follows n = Ω(polylog(d)), K = Θ(1). The federated network shares the same initialization.3. The learning rate η ≤ Õ( K E min{||µ|| 2 2 , σ -2 p m -1 }) and the standard deviation of network weight initializationσ 0 ≤ Õ(mn) • min{(||µ|| 2 2 , σ p √ d -1)}For the first assumption, we assume the dimension of latent space is large enough to make our analysis ensure an over-parameterized setting. The second assumption provides statistical properties to the training data and network weight initialization. The third assumption of a small learning rate and small initialization guarantees that gradient descent effectively minimizes the training loss.Note that this analysis framework can be directly applied to our algorithm PromptFolio. To achieve a basic federated version of prompt learning, i.e., PromptFL, we apply the learnable prompt p as the global prompt p G in PromptFolio and apply the mixing coefficient θ = 0, we obtain the theoretical result of PromptFL. Similarly, we can apply the learnable prompt as the local prompt p L and apply the mixing coefficient θ = 1, we obtain the theoretical result of CoOp.This section discusses the computational approach used to analyze the performance of the algorithm, focusing on the gradient calculations essential for optimizing the model parameters. To calculate the gradient, we first need to find the partial derivatives of sim(g k,i , h k,i ) with respect to p G and p L . Let σ ′ G,r,k,i and σ ′ L,r,k,i are the r-th diagonal elements inEQUATION)Using the logistic loss L T for client k, the goal is to minimize the subsequent loss function to achieve supervised fine-tuning of the text prompts:EQUATIONNow, we can use these partial derivatives to compute the gradients with respect to p G and p L :EQUATIONTo simplify the update process, we can rewrite it using summation notation. Let ℓ ′ k,i denoteEQUATIONEQUATIONIn the above equations, x r,k,i represents the r-th row of g(x k,i ). Note that w T r represents the r-th row of the weight matrix W.In this subsection, we provide the proof of signal-noise decomposition as mentioned in Lemma 4.1. Note that the local update of global prompt coefficients can be generalized to local prompts by changing the subscript G to L. The coefficient β G/L,k denotes the coefficient of the global promptat client k. The coefficient γ G/L,k,k ′ denotes the coefficient of the local prompt k ′ at client k. The coefficient ϕ G/L,k,k ′ ,i denotes the coefficient of client k ′ 's i-th task-irrelevant feature at client k.Here, we suppose that the weight of the pretrained model consists of two kinds of weights, taskrelevant weight µ and task-irrelevant weight ξ. Without loss of generality, we can rewrite the gradient descent update of the prompt as follows:EQUATIONEQUATIONEQUATIONAccording to the update formula of the gradient descent, we have the update formula of the global prompt and the local prompt:EQUATIONEQUATION)As a result, for the row corresponding to the global feature line, we have the update formula of the coefficients:EQUATIONEQUATION)For the row corresponding to the local feature line, we have the update formula of the coefficients:EQUATIONEQUATIONFor the row corresponding to the task-irrelevant feature line, we have the update formula of the coefficients:EQUATIONEQUATION)To analyze the increase of the coefficient, we decompose the coefficient ϕ(t) G,k,l to ψ (t)G,k,l and φEQUATIONEQUATIONEQUATIONEQUATIONHere, we suppose that ξ k,i = L l=1x k,i,l ξ l , thus we have the following lemma.). Suppose that δ ≥ 0 and L ≥ log(4n/δ), then with probability at least 1 -δ, we haveEQUATIONEQUATION)where we denoteσ 2 L = L l=1||ξ l || 2 2 as the summation of the norm among all task-irrelevant features.For simplicity, we first introduce the definitions of F + and F -. Here F + means the train loss corresponding to the positive class, while F -means the train loss corresponding to the negative class.EQUATIONEQUATIONEQUATIONTo illustrate the decoupling of the label from the definitions of F + and F -, we have the following lemma. Lemma E.3. Suppose that y is the ground truth label, we haveEQUATIONProof. We consider two situations: y = +1 and y = -1. If y = +1,EQUATIONIf y = -1,EQUATIONIn conclusion, we haveEQUATIONLemma E.4. Under the modeling of prompt-based federated learning, the expectation of test loss L D of an algorithm can be treated as the probabilityEQUATION)where µ and σ are controlled by the coefficients of feature learning. Thus, the performance of an algorithm can be evaluated by the ratio µ/σ.Proof. Recall that the test error is equivalent toEQUATIONAccording to Lemma E.3, we have thatEQUATIONNote thatEQUATION)Wp + and Wp -can be treated as two constant terms. We consider each line of F (p) and x, then the problem is equivalent toEQUATIONwhereEQUATION)Note that the above equation is equivalent toEQUATION)Note that when we finish training, the coefficients are fixed and thus can be treated as constants. x j are zero mean Gaussian variables, y are {+1, -1} Bernoulli variables and x j and y are independent. Thus the test error can be defined by two valuesEQUATIONwhereEQUATIONEQUATIONThe problem can be treated as ) as the ratio of different coefficients and the correlation between two prompts is defined as ρ ∈ [0, 1]. We have that the mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL.EQUATIONEQUATIONEQUATION     C a = (b -a)(b 2 + 2ρb + 1) C b = (a + b)(b 2 -1) -4b(ρb -1) C c = (b -1) (a + b) 2 (b + 1) 2 -8ab 2 (ρ + 1) 2 .Proof. Then we suppose that at the end of training, the random variable corresponding toL D (p G ) is z G ∼ N (µ G , σ 2 G ), the random variable corresponding to L D (p L ) is z L ∼ N (µ L , σ 2 L ). Consequently, we have the random variable corresponding to PromptFolio as z GLo = (1 -θ)z G + θz L , then we haveEQUATIONwhereEQUATIONHere, we find where this inequality becomes an equality. After further simplification, we observe that this equation is a quartic equation, and we've discarded 0, 1, and another unreasonable solution. The remaining solution can be expressed asEQUATIONwhereEQUATION)Here, for any a, b, we take theθ ∈ 0, proj [0,1] C b -C c 2C a (59)due to the derivative of this objective function being less than zero when θ = 0. As a result, when we take θ in (59), we will obtain the inequality.Proof. In the first stage, we consider that the gradient of the loss ℓ ′ is bounded in a constant level. We assume that the coefficients γ (t) , β (t) , ϕ (t) = O(1) for 0 ≤ t ≤ T 1 . Here we have that the output similarity satisfies:EQUATION) When t ∈ [0, T ], we have ℓ ′(t) k,i = 1 1 + exp(sim(g k,i , h k,i )) ≥ 1 1 + O(1).(EQUATIONTheorem F.2. Under Assumption E.1, there exists total number of local updatesT 1 = R 1 E = O(η -1 Knσ 2 p σ 2 L ) such that β (T1) G,k = Θ((1 -θ)nKSNR 2 G ) β (T1) L,k = Θ(θnSNR 2 G ), ∀k ∈ [K] γ (T1) G,k,k = Θ((1 -θ)nχ k SNR 2 k ) γ (T1) L,k,k = Θ(θnSNR 2 k ), ∀k ∈ [K] ϕ (T1) G,k,l = O(1 -θ) ϕ (T1) L,k,l = O(θ), ∀k ∈ [K], l ∈ [L] (69) Proof. Part 1: Analysis of β (T1) G,k and γ (T1) G,kWe first consider the growth of local task-relevant coefficient γ(t) G,k,k and γ (t)G,k on the corresponding client k. Consider the iteration equation for the coefficient of signal learning under local gradient descent in the first round, we have:EQUATION)According to Lemma F.1, we have an upper bound for signal learning at the first round of local updates.EQUATION)Taking a telescoping sum over t = 0, 1, . . . , E, we can obtain the upper bound for signal learning before the first step of weight averaging.EQUATION)After taking the average among the coefficients, we haveEQUATION)Note that we denote χ k∆ = K k ′ =1 ⟨µ ′ k ,µ k ⟩ ||µ k ||as the total similarity between signal vectors among clients. In the following E gradient descent steps on each clients, we haveEQUATION)In the second round of update, we apply average among the clientsEQUATION)We repeat the computation process and obtain the following results for the third rounds of local updates and the FedAvg process.γ (3E) G,k,k ≤ 2 K χ k ηE||µ k || 2 2 + ηE||µ k || 2 2 = ( 2 K χ k + 1)η(1 -θ)E||µ k || 2 2 (76) γ (3E) G ≤ 1 K (χ k ( 2 K χ k + 1) + (K -χ k ) 2 K χ k )ηE||µ k || 2 2 = 3 K χ k η(1 -θ)E||µ k || 2 2 .(77) Thus, after R 1 = T 1 /E rounds of communication, the noise memorization has an upper bound:EQUATION)We can similarly obtain the lower bound of this proof. Consider the first round of gradient update, we haveEQUATION)So before taking the first average, we take a telescope sum over t = 0, . . . , E -1 yielding:EQUATION)After the weight average operation, we haveEQUATION) Recall that χ k ∆ = K k ′ =1 ⟨µ k ,µ k ′ ⟩ ||µ k || 2 2counts the total similarity between the signal vectors among clients. In the next E gradient descent steps on each client, we haveEQUATION)After gradient descent, we apply the second weight averaging operation on the server and obtain the following resultEQUATIONSimilarly, we repeat the computation procedure and obtain the following results for the R 1 -th round of local updates plus weight average operation:EQUATION)As a result, we haveEQUATIONHere, the global task-relevant feature follows that χ K = K, so we have EQUATIONEQUATIONAccording to Lemma F.1, we have upper bound for γEQUATIONThen we taking a telescoping sum over t = 0, 1, . . . , R 1 E, we haveEQUATIONSimilarly, we repeat the computation procedure and have the lower bound of the γEQUATIONAccording to Lemma F.1, we haveEQUATIONThen we taking a telescoping sum over t = 0, 1, . . . , R 1 E, we haveEQUATIONAs a result, we haveEQUATIONThe global feature and local feature have the same weight and take the same order. We first establish the lower bound for ϕEQUATIONEQUATION≥ ⟨pEQUATION≥ ⟨pEQUATIONwhereC 2 is a positive constant that satisfies C 2 ≥ max{ψ (t) G,k,l , φ (t) G,k,l }, (a) is by Lemma E.2 and (b) is by definition of C 2 .Here, we suppose thatΦ (t) G,k,l = max ⟨p (0) G,k , ξ k,i ⟩ + ψ (t) G,k,l -4C 2 n log(4n 2 /δ) σ 2 L. At initialization, it is easy to check that:EQUATIONwhere (a) is by the following condition:EQUATIONWe can compute the growth of ΦG,k,l as follows:EQUATIONBefore the first step of weight average, we take the telescoping sum:EQUATIONThen, we perform weight average operationEQUATIONThe next E gradient descent steps yield:EQUATIONHere, we take the average among the clients and we get:EQUATIONWe use the same technique to obtain the lower bound of noise memorizationEQUATIONFinally, we confirm thatEQUATIONEQUATIONEQUATIONwhere the inequality (a) is by definition of constantC 3 ≤ 4C 2 n log(4n 2 /δ) σ 2 Lwhich holds when σ 2 L ≥ C 5 log(4n 2 /δ)n 2 and the inequality (b) is by taking the value of T 1 . Then we provide the upper bound of the ϕ(T1) G,k,l .Here, we suppose thatEQUATIONG,k,l . Similarly, we define the coefficient after weight averageEQUATIONwhere (a) follows |ℓ′(t)k,i | ≤ 1 and σ ′ ≤ 1 in Lemma F.1 and Lemma E.2. Similar to above proof, we have thatEQUATIONAfter t = E steps, we perform a weight average operation and we haveEQUATIONSimilar to the above proof, we haveEQUATIONThus, it is confirmed thatΨ (T1) G,k,l = O(1 -θ). Similar to the convergence of Ψ (T1) G,k,l = O(1 -θ), we have that Ψ (T1) L,k,l = O(θ).EQUATIONHere, n = k n k /K is the average number of data in each client andSNR G = ||µ G ||/(σ p √ m), SNR k = ||µ k ||/(σ p √ m)denote the signal-noise ratio between the task-relevant feature and taskirrelevant feature. Here, we defineχ k = K k ′ =1 ⟨µ k , µ k ′ ⟩/||µ k || 2 2 .Proof. The proof of the result follows the proof of global coefficient in Lemma F.3. We take θ = 0 to obtain the final result.In this stage, we focus on the proof of scaling behaviour for the coefficients of features β G , γ G , β L , γ L . We first provide the following lemma.Lemma F.4. Under Assumption E.1, we haveEQUATIONEQUATIONfor all k ∈ [K].Proof. According to the update formula of p G and p L , we haveEQUATIONwhere the second equation is by the orthogonal assumption between the feature vector and noise vector. Here, these equations follow the same proof and we thus complete the proof.Lemma F.5. Under Assumption E.1, for 0 ≤ t ≤ T * , whereEQUATIONEQUATIONEQUATIONEQUATIONfor all k ∈ [K].Proof. Here, we only prove that 0 ≤ γ(t) G,k ≤ γ (t) G,k,k ≤ (1 -θ)nχ k SNR 2 k log(T * ).The proof of other coefficients can be generalized from this proof. Considering the update formula, we haveEQUATION) Let T b = R b E to be the last time t ≤ T * that γ (t)G,k ≤ 0.5 log(T * )nSNR 2 k . We have the lower bound and upper bound of ⟨pEQUATIONDue to thatEQUATIONEQUATIONEQUATIONwhere the first equality k log(T * ). Similarly, we can also derive the upper bound as follows:EQUATIONEQUATIONEQUATIONAccording to the above formula, we haveEQUATION≤ γEQUATIONEQUATIONEQUATIONwhere the first inequality (a) is by the definition of ℓ ′ , the second inequality (b) is by the above formula and the third inequality (c) is due to that nχ k SNR 2 k ≥ 1 and the induction hypothesis γ(T b ) G,k ≤ 0.5(1 -θ)nχ k SNR 2 k log(T * ).NeurIPS Paper ChecklistJustification: The work conformed with the NeurIPS Code of Ethics. The paper does not contain any content or methods that would violate ethical guidelines, and it maintains anonymity and integrity in its experiments and data handling.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).PromptFolio is pronounced as /pr5mpt"foUlioU/. 


Decision: The paper proposes a theoretic framework for understanding prompt-based FL based on feature learning theory [1].
They then draw an analogy to portfolio optimization from the finance field and prove that a mixture of global and local prompts can balance between generalization and personalization.

The reviewers generally agreed that the connection to feature learning theory and the resulting theoretic method was novel and insightful. One of the key challenges was appropriately explaining feature learning theory for new readers and properly framing the main contributions. In the revision, please include better introduction, background, and intuition for the feature learning theory to aid new readers.

There were some additional minor concerns that this was only evaluated on the classification task but this seems reasonable given that the paper aims towards theoretic contributions primarily. Given the confusion, I would recommend revision of the paper to more carefully emphasize the theoretic focus of the paper and deemphasize the algorithmic novelty.

Overall, the paper provides theoretic insight into the federated prompt learning. This theoretic insight is used to develop a simple but effective strategy for combining local and global prompts. Because this paper adds some theoretic grounding for federated prompt learning, the paper provides a valuable contribution to the NeurIPS community. 

[1] Allen-Zhu, Z., & Li, Y. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. In The Eleventh International Conference on Learning Representations.

Reviews: review: paper_summary: paper_summary: This paper tackles the interesting problem of analyzing federated learning (FL) from vision-language foundation models like CLIP. The authors develop a theoretical framework based on feature learning theory to understand how prompt-based FL works. They introduce a new algorithm called PromptFolio that mixes global and local prompts, drawing an analogy to portfolio optimization in finance. The theoretical analysis shows how this approach can balance generalization and personalization in FL. They back up their claims with experiments on several datasets.
strength_weakness: strength_weakness: - This reviewer is confused by the feature learning theory (I am not familiar with it), which is introduced in a relatively short section. Is it (looks like) a common sense in deep learning or a kind of new theory proposed recently? It would be helpful to have a more thorough introduction to the theory.
- Using a single activation function for the text encoder is a pretty big simplification from how CLIP and other large language models actually work.
- All the experiments focus on classification tasks. Any thoughts on how PromptFolio might perform on other vision-language tasks like image captioning or visual QA in a federated setting?
- Why is the proposed method on OxfordPets dataset not as good as coop? While it seems to be dominating on other datasets.
questions: questions: see weaknesses.
limitations: limitations: yes
paper_summary: paper_summary: The proposed methodology offers a novel take on analyzing federated learning using prompt learning (for foundation models) via feature learning theory. At its core, the idea is to identify and monitor task-relevant and task-irrelevant features, and leveraging inspiration from portfolio optimization which says to combine independent assets to maintain income while decreasing risk of investment, construct a 2 part prompt. One part which is global and another which is local, to help with personalization. By mixing these prompts optimally, the proposed PromptFolio method highlights an improvement in overall performance of the FL system.
strength_weakness: strength_weakness: Abstract/Conceptual Questions:
- Could the authors comment about the few-shot learning aspect of foundation models (+prompt learning) and how the optimal mixing coefficient would be affected in that scenario?
- An adjacent sub-field that contains foundation models and federated learning is the study of their impact in learning new classes using prior knowledge (base vs. new classes performance). Could you comment on how this sub-field could relate to the proposed method?

Clarifications:
- Could the authors clarify the exact dimensionality of the prompt used in each of the baselines as well as PromptFolio?
- Could the authors mention if there are previous works that have proposed a 2-level prompt learning idea? (since Section 2.2 L 107 mentions that an exploration of such a idea and the cooperation between prompts is sparse)
- In Section 4. L. 224, the authors mention that "task-relevant coefficients can be directly added". Could you clarify this statement?
- Could the authors highlight the data setting used to generate Table 1?
- Could the authors explicitly state the train and test conditions (settings) used within the experiments?

Suggestions:
- Since the proposal of PromptFL, there are have been further advances to the idea of prompt learning within the federated setting. Could the authors comment and draw comparisons against the more recent methods?
questions: questions: The questions have been included in the weaknesses section above.
limitations: limitations: yes, the authors have addressed limitations within their choices of the experiment setting. However, I would recommend drawing more broad comparisons as to potential missing pieces in comparison to more recent work as well as the finer points from the feature learning theory that don't fully match up to the settings in the FL system.
paper_summary: paper_summary: This paper studies theoretical properties of prompt based federated learning methods for visual language models. Following the proposed theoretical results, a new algorithm named Global-local Prompt Portfolio for Federated Learning (PromptFolio) was proposed. The proposed algorithm was examined in image classification tasks using a CLIP model.
strength_weakness: strength_weakness: Experimental analyses are limited and should be improved as well.
questions: questions: In the algorithm, each client sends prompts to a server. How do assure privacy protection in this case? 

In the analyses, CLIP was utilized as a foundation model. However, the main claim of the paper is providing a generic framework for a more general category of visual language foundation models. Therefore, the proposed method should be examined using the other visual language models as well.

Moreover, CLIP was used for image classification tasks, with limited prompts. In order to show the generalization of the proposed method for prompts with different structures, additional results should be given for the other tasks, such as image captioning and scene description, as well.

In Fig. 2.b, the best accuracy is achieved for 20 users, and the lowest accuracy is achieved for 5 users. Could you please elaborate this result? Could you please provide additional analyses for a larger number of users as well?
limitations: limitations: Limitations were partially addressed.
paper_summary: paper_summary: This paper discusses the integration of pretrained vision-language foundation models, such as CLIP, into federated learning. The idea is to use prompt-based federated learning to minimize the communication and computational costs. The authors go into the theoretical analysis to understand the performance of this approach
strength_weakness: strength_weakness: - While the overall approach is quite interesting, the authors restrict their investigation in a much simpler classification task. It is unclear why a larger foundation model is required for simpler classification tasks, where traditional FL techniques on vanilla models (e.g., reset, mobileNet etc) might work better towards learning new unseen distributions that a foundation model might not have seen (e.g,. x-ray hospital images etc). 
- Following the previous comment, it is unclear how this method would work on unseen distributions that most FL applications are mostly useful for. Most FL models are quite good at generalising to data that can be found on the public internet, but it is unknown how well they would work on out-of-distribution private data. 
- Given the fact that there are a few related prompt-based FL methods (e.g., promtFL, FedPrompt, etc) limit a bit the novelty of this work to mostly providing a solid theoretical analysis. 
- There might be some privacy implications from this work, as sensitive data are shared. 
- The evaluation was done with mostly 10 (up to 50) users, whereas in typical FL scenarios we want to learn from million of users. It is unclear how these results would generalise. Overall, the evaluation is a bit simplistic, showing only accuracy results on a few tasks.
questions: questions: See above
limitations: limitations: See above
paper_summary: paper_summary: In this papaer, the authors proposed a method to finetune vision-language model (CLIP) via prompt learning. Specifically, the author optimize a global textual prompt which will be optimized via FedAvg, while each client additionally optimize a local textual prompt. The proposed method is a combination of the existing methods CoOp and PromptFL and outperforms both methods.
strength_weakness: strength_weakness: 1. The proposed method is basically a combination of both CoOp and PromptFL, or to be more specific, optimizing both client-specific and client-agnostic textual prompt. The novelty of the proposed needs to be further highlighted in the main paper.
2. The combination strategy seems to be a bit straight forward, i.e., using a single scalar value $\theta$, the author should discuss other possibilities.
3. The authors should compare with other prompt-based tuning methods for FL, e.g., [1][2]. Also, adding more ablation studies could be beneficial.

[1] Guo T, Guo S, Wang J. Pfedprompt: Learning personalized prompt for vision-language models in federated learning[C]//Proceedings of the ACM Web Conference 2023. 2023: 1364-1374.

[2] Qiu C, Li X, Mummadi C K, et al. Text-driven Prompt Generation for Vision-Language Models in Federated Learning[J]. arXiv preprint arXiv:2310.06123, 2023.
questions: questions: Please see the questions listed in the weakneses section.
limitations: limitations: The authors do not discuss the limitation of the method.

Comments: title: Reviewer Feedback Request on Rebuttal
comment: Dear Reviewer XM5m and AC, 

Hope this message finds you well. As the discussion deadline approaches, we sincerely hope to receive a response to our rebuttal. We believe we have properly addressed the concerns by Reviewer XM5m as our reply to these similar concerns have been recoginized by other reviewers. 

We observed that the concerns by Reviewer XM5m regarding privacy issues in federated prompt learning and experiments involving a larger number of users are similar to those raised by Reviewer kpWj. They have acknowledged our response and maintained their score at 6. In our response, we explained that prompts are small vectors (16k parameters) without explcit semantic meaning passed through text encoder, which **actually helps protect privacy** compared to standard federated learning setting. Besides, we illustrated the phenomenon mentioned in Question 4, where "20 users achieve the highest accuracy," by discussing **the partition strategy used for few-shot learning**. We also included experiments with a larger number of users, as shown in **Figure R3** of the PDF.

The other concern by Reviewer XM5m about generalization to different tasks was also addressed for Reviewer BdA2, who **appreciated our response and increased their score from 4 to 5**. We highlighted that the classification tasks exemplify our theoretical framework and that previous work considers only classification tasks as well. Therefore, **extending to different tasks is beyond the scope of this paper**. Additionally, we demonstrated generalization to different structures through further experiments, as shown in **Figure R2** of the PDF.

Thank you again for comments. We sincerely hope you can reconsider the score for our work. 

Best regards, 

Submission 6698 Authors
Dear Reviewer bKME,

We thanks for your comments again. As the discussion process is coming to a close, please let us know if you have any further questions.

Best regards, 

Submission 6698 Authors
Thank you for reconsidering our work and raising your score. We greatly appreciate your constructive feedback and support.
Thanks for the rebuttal.

Some of my concerns have been addressed.

I would raise my score.
Thank you for your response. We'd like to address your concerns. 

> Q1: As a natural follow-up to the responses to Q1 under clarifications, have the authors observed the change in performance of baselines when switching to prompts of 16k params? 

1) First, we would like to emphasize that in our algorithm, although 16k parameters (8k global + 8k local) are used, only the 8k parameters corresponding to the global prompt are shared across clients in federated learning. Thus, our comunication cost is exact the same as baselines with 8k parameters. Using baselines with 16k parameters will double the communication cost, resulting in an unfair comparison. 

2) To further alleivate your concerns, we have added experiments by increasing the parameters of baselines from 8k to 16k. From the following table, we can obsesrve that: 1) increasing the parameters will not always increase the performance; 2) PromptFolio still has the advantage even under this unfair comparsion benefit from the prompt portfolio mechanism.

|                      | Cifar100             | DomainNet            | Caltech10            | OxfordPets           | DTD                  |
| -------------------- | -------------------- | -------------------- | -------------------- | -------------------- | -------------------- |
| CoOp (8k param)      | 76.88 $\pm$ 0.07     | 91.83 $\pm$ 0.13     | 97.10 $\pm$ 0.20     | 87.85 $\pm$ 0.32     | 56.39 $\pm$ 0.48     |
| CoOp (16k param)     | 77.74 $\pm$ 0.21     | 91.92 $\pm$ 0.18     | 96.41 $\pm$ 0.43     | 87.68 $\pm$ 0.44     | 58.33 $\pm$ 0.41     |
| PromptFL (8k param)  | 78.16 $\pm$ 0.16     | 92.72 $\pm$ 0.16     | 95.51 $\pm$ 2.62     | 88.91 $\pm$ 0.72     | 70.99 $\pm$ 0.32     |
| PromptFL (16k param) | 78.53 $\pm$ 0.33     | 92.88 $\pm$ 0.29     | 96.23 $\pm$ 1.41     | 91.81 $\pm$ 0.64     | **71.73 $\pm$ 0.31** |
| PromptFolio (8k + 8k param) | **80.17 $\pm$ 0.05** | **93.04 $\pm$ 0.09** | **97.24 $\pm$ 0.11** | **92.17 $\pm$ 0.32** | 71.32 $\pm$ 0.49     | 


> Q2: Could the authors provide more context for the selection of $\alpha = 1$ and discuss the impact of lower $\alpha$ on the algorithm? (if possible also provide results) I am specifically interested in the lower $\alpha$ setting since it would allow for more in-depth and interesting analyses given the theoretical framework outlines in the manuscript. 


We conducted experiments with multiple different $\alpha$ values instead of using only $\alpha = 1$. For instance, in Table 2, we used $\alpha = 0.3$ to compare our result with baseline methods. In Figure 2(a), we conducted parameter analysis for $\alpha$ by implementing PromptFolio with different $\alpha$ values (0.01, 0.1, 1, 10). Here, we observe that a lower $\alpha$ indicates a more non-i.i.d. data distribution, requiring a larger mixing coefficient $\theta$ to retain more local information. 

Our theoretical analysis also supports this result. In our framework, $\Chi_k$ measures the heterogeneity across clients. A lower $\alpha$ leads to more data heterogeneity, meaning $\Chi_k$ will be lower as well. As described in Theorem 5.2 on Line 279, a lower $\Chi_k$ (lower $\alpha$) corresponds to a higher $\theta^\star$, also aligning with the experimental results in Figure 2. Further analysis on how different levels of data heterogeneity affect learning dynamics can be found in Theorem D.2.
Thanks for your positive feedbacks and consistent supports for our work.
Thanks for reconsidering our work and raising your score. We appreciate your constructive feedback and supports.
I thank the authors for their in-depth responses to my comments. As a follow up to their responses,

- As a natural follow-up to the responses to Q1 under clarifications, have the authors observed the change in performance of baselines when switching to prompts of 16k params?
- Could the authors provide more context for the selection of $\alpha=1$ and discuss the impact of lower $\alpha$ on the algorithm? (if possible also provide results) I am specifically interested in the lower $\alpha$ setting since it would allow for more in-depth and interesting analyses given the theoretical framework outlines in the manuscript.
I would like to thank the authors for clarifying multiple points and overall providing a detailed rebuttal to all authors. After reading all reviews and comments, I think I will stay with my weak accept recommendation.
Thanks for the rebuttal. I have raised my score to Borderline Accept.
Dear Reviewers and AC,

Thank you for your valuable comments. We have made every effort to address the concerns raised by the reviewers.

We notice that none of the reviewers have provided follow-up comments on our responses. As the discussion stage deadline approaches, please let us know if you have any further questions.


Thank you and best regards,
Submission 6698 Authors