Title: Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated Learning

Vision-Language Foundation Models

Prompt Learning

Theoretical Analysis
Abstract: Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

Abstract

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., promptbased federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments. Our code is available at: https://github.com/PanBikang/PromptFolio.git .

1 Introduction

Federated learning (FL) [33] stands out as a powerful framework that enables machine learning over decentralized data while maintaining data privacy and reducing reliance on centralized data repositories. Despite its advantages, the intensive computational and communication demands during the training phase often constrain the scalability of the models utilized. A transformative advancement in this field is the integration of prompt-based learning within the federated framework [42, 17, 13].

Prompt learning adapts models such as Contrastive Language-Image Pretraining (CLIP) [38] through minimal modifications, typically in the form of prompts or cues, guiding the model's predictions. As an emerging idea in machine learning, it has shown significant promise in various applications by allowing models to perform specialized tasks without undergoing complete retraining. Prompts effectively adjust pre-trained models for new tasks or datasets, which is particularly valuable in federated environments where data privacy and bandwidth constraints often limit conventional

training methods. A prominent example is CoOp [42], with its streamlined federated version known as PromptFL [17].

Despite the empirical success of prompt-based federated learning [28, 17, 16], theoretical analysis in this area remains limited. In this paper, we use feature learning theory [1] to propose an analytical framework for prompt-based federated learning with vision-language foundation models. Feature learning theory divides data into task-relevant and task-irrelevant features, allowing learnable weights to be expressed as a linear combination of these features. By introducing feature alignment across bimodal pretrained models, we demonstrate the relationship between learnable prompts and the features in latent space. To understand the process of signal learning and noise memorization, we use a two-stage analysis to examine the dynamics of coefficients for task-relevant and task-irrelevant features. Leveraging these coefficients, we demonstrate that prompt learning can be evaluated by comparing the ratio of task-relevant to task-irrelevant coefficients. In this way, we establish a theoretical foundation for prompt-based federated learning.

Additionally, we treat task-relevant coefficients and task-irrelevant coefficients as income and risk in investment portfolios [31, 32]. Inspired by investment portfolios, where combining two independent assets can reduce risk, we introduce two learnable prompts: a global prompt and a local prompt, and simply mix them to form a prompt portfolio to balance generalization and personalization. By mixing them to create a prompt portfolio, we balance the generalization and personalization under severe data heterogeneity. Leveraging the analysis framework we proposed, we prove the performance advantage of this mixed algorithm and derive the optimal coefficient. Besides, we use comprehensive experiments to support our theoretical results.

In this paper, our primary contributions are threefold:

We present an analytical framework for prompt-based federated learning using visionlanguage foundation models. This framework aligns text and image features into a shared latent feature space and utilizes a two-stage analysis to understand the dynamics of prompt learning. By tracking the progression of signal learning and noise memorization, we show that the effectiveness of prompt-based federated learning can be measured by the ratio of task-relevant to task-irrelevant coefficients.

Additionally, we introduce a prompt portfolio mechanism to address severe data heterogeneity and balance generalization and personalization. Within our proposed analytical framework, we draw an intuitive analogy between task-relevant features and income in portfolio optimization, as well as task-irrelevant features and risk. Consequently, we demonstrate that the combination of prompts performs better than using a single prompt, and we provide the optimal mixing ratio.

The theoretical result has been empirically validated through rigorous experiments. Our results not only align with theoretical predictions but also consistently demonstrate the practical superiority of our approach in real-world scenarios.

2 Related Work

In this section, we examine prior research that serves as the basis for our study. Our investigation is divided into two primary areas: prompt-based federated learning and feature learning theory. Together, these fields create a complete context for our contributions.

2.1 Prompt-based federated learning

Prompt learning, initially developed in the field of natural language processing, has expanded its reach to vision language models. Examples include the CLIP model [38], which originally utilized manually engineered templates. However, more recent advancements have shifted towards developing prompts within a continuous embedding space. Innovations like CoOp [42] have refined the CLIP model by introducing continuous prompt vectors, sparking a surge of studies aimed at enhancing the efficiency of prompt learning and providing a solid foundation for further investigation [12].

To enhance the integration of global data and address challenges in scenarios with limited user data, FedPrompt [41] and PromptFL [17] have effectively integrated the concept of prompt learning with federated learning [4, 5, 27, 37, 23]. To tackle the statistical heterogeneity often found in client data,

pFedPrompt [16] introduces a non-parametric approach, providing each client with a personalized attention module. This module is designed to refine spatial visual features to better align with the unique characteristics of local data. Concurrently, pFedPG [40] introduces a novel prompt generator located at the server, which customizes prompts for each client, thus enhancing the personalization of the federated learning process. Additionally, a recent study, FedOTP [28], leverages optimal transport theory to improve the balance between achieving global consensus and allowing for local customization through a strategic combination of global and local prompts.

However, there is limited theoretical analysis of federated prompt learning. For instance, the theoretical examination of the CLIP model by [8] enhances prompt learning theory by exploring how CLIP learns transferable representations across various modalities and improves zero-shot transfer performance with a recently developed regularization technique. However, to the best of our knowledge, no research analyzes prompt learning within a federated setting that elucidates the cooperation between prompts. In this paper, we analyze how different prompts interact with feature learning theory and demonstrate the provable benefits of cooperation between global and local prompts.

2.2 Feature learning theory

Feature learning theory [1] has revolutionized our understanding of how machine learning models learn and represent information. Unlike other theories, feature learning accommodates substantial weight updates during gradient descent, enabling the network to uncover complex patterns in data. Feature learning theory has been successfully applied to various neural network architectures, including convolutional neural networks [6, 24], graph neural networks [18], and vision transformers [22, 26]. Moreover, feature learning theory has been used to analyze different training algorithms, such as Adam [43], momentum [26], out-of-distribution learning [7], and Mixup [44, 10]. Notably, [19] have analyzed the convergence and generalization in general federated learning. Despite progress in feature learning theory, the study of federated prompt learning is sparse. Our work uniquely addresses this gap by analyzing feature learning under prompt-based federated learning, providing crucial insights for their effective adaptation and optimization in such contexts.

3 Preliminary

Notation In our notation, vectors are represented by lowercase bold letters, matrices by uppercase bold letters, and scalars by regular, non-bold letters. The ℓ 2 -norm of a vector v is indicated as | v | 2 . The spectral norm of a matrix A is denoted by | A | 2 , and its Frobenius norm by | A | F . To compare the growth or decline of two sequences, we use standard asymptotic symbols like O ( · ) , o ( · ) , Ω( · ) , and Θ( · ) , which describe their behavior as they approach infinity. We introduce notations ˜ O ( · ) , ˜ Ω( · ) , and ˜ Θ( · ) to obscure logarithmic factors within these expressions. Notably, 1 ( · ) represents the indicator function. Lastly, we represent sequences of integers as [ n ] = { 1 , 2 , . . . , n } and sequences of elements such as vectors can also be represented similarly v [ n ] = { v 1 , v 2 , . . . , v n } .

3.1 Prompt-based federated learning

In this part, we demonstrate how to fine-tune a learnable text prompt under a vision language pretrained model. Here, we consider the classification task, where we assume that we have an image x . The objective is to correctly classify the image into class y ∈ [ C ] , where the total number of classes is C . From the vision language pre-trained model, we expect that the latent spaces of the text encoder and image encoder are aligned. Thus, when we input different prompts, the text feature generated by the correct prompt will have the highest similarity with the image feature. We input a learnable prompt p and a fixed class prompt p c ∈ { p 1 , · · · , p C } , which correspond to the classes, into the text encoder h . This process generates the text feature for class c : h c = h ( p , p c ) . On the other hand, the image feature g is generated by the image encoder g : g k,i = g ( x k,i ) ∈ R m . We define the similarity function between the image feature g and the text feature h as ρ := [ ρ c ] = sim ( g , h c ) . The training process mirrors traditional classification tasks, where the objective loss ℓ ( ρ , e y ) is the distance between the similarity vector and the actual label y . Here, ℓ represents the loss function that measures the distance between two vectors, and e y is the one-hot vector derived from the ground truth label y .

To illustrate the prompt-based federated learning framework, consider a federated system with a central server and K clients. Assume client k has n k training samples: { x k,i , y k,i } n k i =1 . The learnable prompt in client k is denoted as p k and the learnable prompt is aggregated at each communication round.

4 Analysis Framework for Prompt-based Federated Learning

In this subsection, we present the analysis framework for prompt-based federated learning from vision-language pre-trained models. The central concept of this framework is the alignment of features between text and image encoders in the vision-language pre-trained model. To achieve this, we link the text encoder and image encoder through a shared latent feature space, described as follows.

Feature representation and client distribution Inspired by [6, 24, 19], we expect that the latent feature space will contain task-relevant and task-irrelevant features. In federated learning settings, the task-relevant features can be categorized into global task-relevant features µ G and local taskrelevant features ν 1 , · · · , ν S , where S is the length of local task-relevant features. Additionally, the task-irrelevant features can be listed as ξ 1 , · · · , ξ L , where L is the length of task-irrelevant features. Here, we assume that the dimension of the latent space is m and these features µ ( · ) , ν ( · ) , and ξ ( · ) are elements of R m . For simplicity, we assume that these features are orthogonal to each other. In our theoretical examination, we address a binary classification scenario with y k,i ∈ { +1 , -1 } . We consider a scenario with K clients, each linked to a distribution D k , ∀ k ∈ [ K ] . Initially, we choose the signal vector µ k for client k by sampling from P ( ν 1 , ν 2 , . . . , ν S ) , where P represents a discrete distribution that assigns probabilities to each local task-relevant feature vector µ k := ν s , s ∈ [ S ] .

Text encoder Here, we suggest coupling the learnable prompt and the class prompt and propose the structure of the text encoder. By adopting a similar setting as [39], we suppose the generation of the text feature can be written as follows:

where W ∈ R m × m p is the weight matrix, and p y k,i ∈ R m p is the prompt linked to a ground truth class. In this definition, the introduction of Wp y k,i introduces nonlinearity between the trainable prompt and the class prompt while keeping the overall function nonlinear. Note that for a binary classification problem, the vector p y k,i takes p 1 when y k,i = 1 and p -1 when y k,i = -1 . To reveal the properties of the text encoder in the pre-trained model, we assume that the weight matrix W is composed of the following rows:

The assumption of the weight matrix is inspired by [20], and the evidence of this assumption is listed in the Appendix D. In our analysis framework, we adapt FedAvg [33] as our prompt aggregation algorithm, which is named PrompFL [17]. The aggregation formula is given by:

where n := ∑ k n k denotes the total number of data samples across all clients.

Image encoder Let us consider the image network, represented as g k,i = g ( x k,i ) ∈ R m . We assume that the image encoder also aligns the feature space of the text encoder within the pre-trained model. As a result, we assume the image feature generated by data x k,i in client i can be expressed as follows:

where x k,i,l ∼ N (0 , σ 2 p ) , ∀ l ∈ [ L ] represents the coefficient of task-irrelevant terms in the data, and σ 2 p is the variance. This assumption implies that task-relevant features vary based on whether the label is positive or negative, whereas task-irrelevant features persist as arbitrary and unrelated to the label's polarity. The similarity score between an image x k,i and class y k,i is expressed as

sim ( g k,i , h k,i ) = ⟨ g k,i , h k,i ⟩ . Moreover, the objective of the training loss is designed to enhance the resemblance between the image feature g ( x k,i ) and the text feature created by the ground truth prompt p y k,i .

Signal-noise decomposition Based on the above model, we introduce the signal-noise decomposition of the learnable prompt here. Note that the proofs of the following lemmas and theorems are listed in the appendix.

Lemma 4.1 ( Feature Representation ) . At the t -th iteration, the learnable prompt p ( t ) k for client k and the aggregated prompt p ( t ) can be rewritten as a linear combination of features and prompt initialization:

where α ( t ) · , · are the coefficients of initialization, β ( t ) · is the coefficient of global task-relevant features, γ ( t ) · , · is the coefficient of local task-relevant features, ϕ ( t ) · , · are the coefficients of task-irrelevant features, and the overlined coefficients are the averaged versions of the original coefficients.

Here, since the learnable prompts can be written as a linear combination of the features, we can analyze the dynamics of these coefficients to understand the learning progress of the prompts. The normalized factor such as || µ G || -2 2 is used to make the coefficient similar to the inner product of the prompts and the features, β ( t ) k ≈ ⟨ p ( t ) k , µ G ⟩ .

Coefficient dynamics Inspired by previous studies [6, 24, 19], we employ a two-phase analysis to track the dynamics of coefficients in prompt-based federated learning from vision-language foundation models. By analyzing the dynamics of the coefficients, we can obtain the feature learning procedure during training. This two-stage analysis allows us to establish the order of coefficients and explore how they are affected by the mixing parameter θ . For the theorem and proof of this analysis, please refer to Appendix F.

Theorem 4.2 ( Training Dynamics ) . There exists a total number of local updates T 1 = R 1 E = O ( η -1 Knσ 2 p σ 2 L ) such that

Here, n = ∑ k n k /K is the average number of data in each client, and SNR G = || µ G || / ( σ p √ m ) , SNR k = || µ k || / ( σ p √ m ) denote the signal-to-noise ratio between the task-relevant feature and task-irrelevant feature. We define χ k = ∑ K k ′ =1 ⟨ µ k , µ k ′ ⟩ / || µ k || 2 2 .

Test performance evaluation with coefficients Here, we suppose the classification output of the i -th data in client k is the class corresponding to the highest similarity between the text feature and image feature, denoted as ˆ y k,i . To assess the algorithm's performance, we evaluate the error rate in the test procedure as our test loss L D :

The following theorem demonstrates that the test loss can be considered as the probability that a Gaussian random variable falls below zero, with the mean and variance influenced by the task-relevant and task-irrelevant coefficients.

Theorem 4.3 ( Test Loss ) . The expectation of test loss L D of an algorithm can be treated as the probability

where µ and σ are functions of task-relevant and task-irrelevant coefficients, as defined in Appendix E.

Drawing from this theorem and the properties of Gaussian distributions, an algorithm's performance can be evaluated by the ratio µ/σ . This ratio highlights the influence of task-relevant and taskirrelevant features on test loss. Specifically, a higher task-relevant coefficient coupled with a lower task-irrelevant coefficient typically leads to better performance.

Connection with portfolio optimization The Markowitz mean-variance model is a famous framework for assembling a portfolio of assets such that the expected return is maximized for a given level of risk [31, 32]. This model characterizes assets by their expected returns and risks. It claims that the return of the whole portfolio is a proportionally weighted combination of the assets' returns, and the risk of the whole portfolio is a function of the correlations of the component assets. According to the properties of task-relevant and task-irrelevant coefficients, the task-relevant coefficient can be directly added, and the task-irrelevant feature follows the additive property of Gaussian random variables. Thus, we connect the task-relevant coefficient to the return and the task-irrelevant feature to the risk. This connection provides insight that the combination of prompts, i.e., a prompt portfolio, will lead to a higher ratio of task-relevant features to task-irrelevant features.

5 PromptFolio: Global-Local Prompt Portfolio for Federated Learning

Building on the significant connection between the feature learning process and portfolio optimization, we treat the prompt trained by CoOp and the prompt trained by PromptFL as the two prompt assets and propose a simple yet powerful mixing algorithm, PromptFolio 2 . For simplicity, we refer to the prompt trained by CoOp as the local prompt p L and the prompt trained by PromptFL as the global prompt p G .

Algorithm 1 (PromptFolio) Global-Local Prompt Portfolio

5.1 PromptFolio Method

The local learning process generates the local feature by including a specific local prompt, whereas the global learning process adopts a similar strategy but also uses FedAvg to compile learnable prompts from various clients. We enhance cooperation between the local and global learning processes by merging both local and global features to create the final text feature. The text feature is produced as follows:

Text

Encoder

Image

Encoder

Logits

Image

Feature

Text

Feature

Text Prompts

Image

Global-Local

Cooperation

Server

Frozen

Global Aggregation

Trainable

Global

Local

Client

Client

Client

...

...

Signal Decomposition:

Task-relevant Task-irrelevant

global

Better

Optimal PromptFolio

local

Worse

label

loss

where θ ∈ [0 , 1] serves as a coefficient to balance the mix of the two features, which addresses the balancing between personalization and generalization. The variation in the parameter θ influences the outcomes of the inference. Specifically, when θ = 0 , the algorithm reverts to PrompFL [17], whereas at θ = 1 , it shifts to CoOp [42]. Our approach consists of combining these features and using the resulting mixed feature to determine their similarity. This feature is subsequently utilized to evaluate the similarity between text and image features. Note that this algorithm differs from typical personalized algorithms [30] as it focuses on integrating text features instead of adjusting training weights. The framework of PromptFolio is described in Algorithm 1.

5.2 Analysis for PromptFolio

In this part, we offer a theoretical demonstration of the performance advantage of PromptFolio and the selection of the optimal mixing coefficient θ . According to Theorem 4.3, each algorithm can be regarded as a Gaussian random variable. The test performance correlates with the ratio of task-relevant features to task-irrelevant features. This ratio enables us to analyze the test results of various learning algorithms.

Suppose that the coefficients of the prompt via local training at step k are β k , γ k and ϕ k , and the coefficients of the global prompt at step k are β k , γ k and ϕ k . We define the mean and variance of the Gaussian variable corresponding to the local prompt as µ k and σ k , and the mean and variance of the Gaussian variable corresponding to the global prompt as µ k and σ k . Let ρ = Θ(1 /k ) ∈ [0 , 1] be the correlation between the Gaussian variables of the local prompt and the global prompt. Here, we define a := µ k µ k = Θ( β k + γ k β k + γ k ) = Θ( K SNR G + K SNR k K SNR G + χ k SNR k ) and b := σ k σ k = O ( ϕ k ϕ k ) = O ( K ) as the ratio of different coefficients. Here, the order of a and b depends on the coefficient derived in Lemma F.3. As a result, we have the following theorem, and the proof can be referred to in Appendix E.3.

Theorem 5.1 ( PromptFolio Advantage ) . The mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL:

The results discussed demonstrate how combining global and local text features enhances performance and illustrate the optimal way to balance personalization with generalization. Here, a and b are the ratio of coefficients and reveal how the global feature and local feature interact. Drawing on principles from portfolio optimization, which involves blending two assets that are not perfectly correlated, we can construct a portfolio that maximizes returns while minimizing risk. Given the characteristics of Gaussian random variables, we intuitively correlate the coefficient of task-relevant features with returns and the coefficient of task-irrelevant features with risk. Thus, the first part of Theorem 5.1 provides a rationale that a well-balanced portfolio of prompt features can significantly improve performance.

Similar to the portfolio optimization problem, we can also derive the optimal mixing coefficient θ .

Theorem 5.2 ( Optimal Mixing Coefficient ) . The optimal mixing coefficient θ ⋆ follows

Theoretically, if we further simplify the mixing coefficient with the order of a , b and ρ , then we get that

In this theorem, we note that a lower χ k indicates greater data heterogeneity, which lead to a higher θ ⋆ . This observation aligns with the intuition that, due to the non-i.i.d. distribution of data, the model should incorporate more local information, thereby making the optimal θ closer to 1.

6 Experiments

In this section, we conduct experiments with the CLIP model to empirically demonstrate the performance advantages of PromptFolio. Specifically, the image network g and the text network h are components of a pre-trained CLIP model. By evaluating results obtained using various mixing coefficients across different datasets, data distribution, and client number, we align theory with practice. We use the Dirichlet distribution to manage data heterogeneity and employ FedAvg as the aggregation strategy. The experiment is conducted on the CIFAR-100 dataset by default, with the model trained for 10 epochs locally and the results evaluated over 100 communication rounds.

6.1 Performance evaluation on various datasets

In this section, we observe that the combination of global and local algorithms outperforms both the prompt-based federated learning with FedAvg and individual prompt learning approaches. Under the CLIP model setting, the global and local prompt learning algorithm degenerates to PromptFL and CoOp, respectively. To explore why this global-local collaboration is more effective than either approach alone, we evaluate the accuracy of various mixing coefficients θ across different datasets. We use CIFAR-100 [25], DomainNet [36], Office-Caltech10 [15], OxfordPets [35], and DTD [11], adopting θ = 0 . 2 as the general mixing coefficient for our algorithm. The quantitative results are shown in Table 1. From this table, it is evident that blending global and local prompts consistently leads to enhanced accuracy, with the accuracy curve also showing a peak. Further performance evaluations can be found in Appendix A.

6.2 Performance evaluation under various data heterogeneity

We then conduct the experiment over different data distributions. By varying the parameters of the Dirichlet distribution exponentially from 0.01 to 10, we controlled the heterogeneity of the data. A

larger α indicates that the data is closer to an i.i.d. distribution. Using 10 users, we performed our experiments, and the results are shown in Figure 2(a).

Global

Local

Global

Local

(a)

(b)

From Figure 2(a), we observe that hybrid approaches outperform solitary methods, consistent with our theoretical analysis. Additionally, higher α values, indicating a more uniform distribution, generally result in a superior global model compared to local models tailored for specific users. This finding supports our conclusion that a more IID distribution leads to χ k being higher and closer to K , resulting in a higher task-relevant to task-irrelevant coefficient ratio, and thus better performance. Conversely, in scenarios where data is highly non-IID, there is a preference for using more local models to maintain high accuracy, which aligns with our analysis of the optimal mixing coefficient θ ⋆ .

6.3 Performance evaluation with different client number

Furthermore, we conducted experiments with different numbers of users on the CIFAR-100 dataset, keeping the Dirichlet distribution parameter fixed at α = 0 . 01 , which represents a pathological non-i.i.d. distribution. The number of users varies from 5 to 100, and the results are shown in Figure 2(b). From these results, we observe that the trend of the mixing strategy outperforming the independent global and local algorithms remains consistent, regardless of the number of users. As the number of users increases, the optimal θ shifts closer to zero, indicating that with more users, each client's information becomes less significant, necessitating more global information. Additionally, the accuracy first increases and then decreases with the number of users, suggesting that there is an optimal number of users, consistent with theoretical results.

7 Conclusion

This work presents a thorough theoretical and empirical exploration of prompt-based federated learning, integrating vision-language foundation models such as CLIP. By developing an analytical framework based on feature learning theory, we have examined the dynamics of signal learning and noise memorization specific to federated settings, providing a robust mechanism to evaluate the effectiveness of prompt-based learning strategies. Notably, our introduction of PromptFolio, which combines global and local prompts into a prompt portfolio, offers an approach to balancing generalization with personalization, drawing an innovative parallel with portfolio optimization in

finance. This approach balances generalization with personalization, supported by an optimal mixing coefficient from our theoretical framework to tailor adaptability in various federated settings. Empirical tests confirm our method's superiority, aligning with theoretical insights and outperforming traditional federated learning approaches. Limitations include a simplified text model with a single activation function, suggesting future work with more complex models to better capture deep network behaviors in federated environments.

References

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations , 2022.

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated Learning with Personalization Layers, December 2019. arXiv:1912.00818 [cs, stat].

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Zhongyi Cai, Ye Shi, Wei Huang, and Jingya Wang. Fed-CO2 : Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning. Advances in Neural Information Processing Systems , 36, 2024.

Yu-Tong Cao, Ye Shi, Baosheng Yu, Jingya Wang, and Dacheng Tao. Knowledge-aware federated active learning with non-iid data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22279-22289, 2023.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:2523725250, 2022.

Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Understanding and improving feature learning for out-of-distribution generalization. Advances in Neural Information Processing Systems , 36, 2024.

Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP. In The Twelfth International Conference on Learning Representations , 2024.

Gary Cheng, Karan Chadha, and John Duchi. Fine-tuning is fine in federated learning. arXiv preprint arXiv:2108.07313 , 3, 2021.

Muthu Chidambaram, Xiang Wang, Chenwei Wu, and Rong Ge. Provably learning diverse features in multi-view data with midpoint mixup. In International Conference on Machine Learning , pages 5563-5599. PMLR, 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. Harmonizing Generalization and Personalization in Federated Prompt Learning. In Forty-First International Conference on Machine Learning , 2024.

Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li. Unlocking the potential of prompttuning in bridging generalized and personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6087-6097, 2024.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition , pages 2066-2073. IEEE, 2012.

Tao Guo, Song Guo, and Junxiao Wang. pFedPrompt: Learning personalized prompt for vision-language models in federated learning. In Proceedings of the ACM Web Conference 2023 , pages 1364-1374, 2023.

Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. PromptFL: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing , 2023. ISBN: 1536-1233 Publisher: IEEE.

Wei Huang, Yuan Cao, Haonan Wang, Xin Cao, and Taiji Suzuki. Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective, August 2023. arXiv:2306.13926 [cs].

Wei Huang, Ye Shi, Zhongyi Cai, and Taiji Suzuki. Understanding Convergence and Generalization in Federated Learning through Feature Learning Theory. In The Twelfth International Conference on Learning Representations , 2023.

Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems , 34:10944-10956, 2021.

Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 7865-7873, 2021. Issue: 9.

Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems , 35:37822-37836, 2022.

Meirui Jiang, Anjie Le, Xiaoxiao Li, and Qi Dou. Heterogeneous personalized federated learning by local-global updates mixing via convergence rate. In The Twelfth International Conference on Learning Representations , 2024.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer ReLU convolutional neural networks. In International Conference on Machine Learning , pages 17615-17659. PMLR, 2023.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009. Publisher: Toronto, ON, Canada.

Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen. A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity. In The Eleventh International Conference on Learning Representations , 2023.

Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated learning by transformer personalization. IEEE transactions on neural networks and learning systems , 2023. ISBN: 2162-237X Publisher: IEEE.

Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi. Global and local prompts cooperation via optimal transport for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12151-12161, 2024.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 , 2020.

Harry M Markowitz. Portfolio selection. The Journal of Finance , 7(1):77, 1952.

Harry M. Markowitz and G. Peter Todd. Mean-variance analysis in portfolio choice and capital markets , volume 66. John Wiley & Sons, 2000.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision , pages 1406-1415, 2019.

Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Federated text-driven prompt generation for vision-language models. In The Twelfth International Conference on Learning Representations , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning , pages 11112-11122. PMLR, 2021.

Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19159-19168, 2023.

Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. FedPrompt: Communicationefficient and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022. ISBN: 0920-5691 Publisher: Springer.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization. In The Eleventh International Conference on Learning Representations , 2021.

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. In International Conference on Machine Learning , pages 43423-43479. PMLR, 2023.

Supplementary organization:

A Performance evaluation

In this paper, we adopt a similar setting to the experimental result in [28]. The experiments are conducted on a cluster with 2 Intel Xeon 5218R, 512GB memory, and 8 NVIDIA Tesla A40 GPUs 48GB. Here, we use Food101 [3], DTD [11], Caltech101 [14], Flowers102 [34], OxfordPets [35] to evaluate our algorithm. The baseline algorithms are CoOp [42], PromptFL [17], the variants of PromptFL [9, 29, 2, 21] and FedTPG [37]. We use Dirichlet distribution with parameter α = 0 . 3 to split the dataset. Consider we use 10 users with 100 communication rounds to assess our algorithm. 10 epochs are conducted each communication round. There are 8 prompts that are randomly initialized during prompt learning. The accuracy is shown in Table 2.

B Further ablation study

This paper further conduct experiments on different shot numbers and different backbones. The results in shown as Figure 3.

Global

Local

Global

Local

Different shots As shown in Figure 3(a), we tested the accuracy using different numbers of shots, ranging from 16-shot to 1-shot. It can be observed that the optimal coefficient remains stable and close to 0.2, demonstrating the robustness of our algorithm across various shot numbers..

Different backbones Figure 3(b) illustrates the accuracy using different vision backbones of CLIP, namely ViT-B/16 and ResNet-50. In this experiment, a similar trend is observed, where accuracy first increases and then decreases with the mixing coefficient, regardless of the backbone used. This outcome indicates that our theoretical framework is consistent across different backbones, further validating its applicability.

C Introduction to feature learning

Feature learning theory is a new theoretical framework proposed recently. The seminal work [1] proposed feature learning theory to understand ensemble, knowledge distillation, and self-distillation in deep learning. They show that when data has a structure containing features, the network's dynamics can be tracked during gradient training, which can further be used to characterize generalization. Later, this theory was further systematized and standardized by [7], forming a fundamental theoretical framework to explain benign overfitting. Since then, feature learning theory has been widely used to understand algorithms [8, 22] and techniques [20] in deep learning, forming a comprehensive theoretical system.

To provide more details on the general feature learning process:

Defining the feature space : As outlined in Section 4, features are categorized as taskrelevant and task-irrelevant. For example, in an image of a cat, features representing the cat itself are task-relevant, while background features are task-irrelevant.

Parameter representation : As described in Lemma 4.1, learnable parameters can be decomposed into the feature space. In our framework, the learnable prompts are expressed as a linear combination of task-relevant and task-irrelevant features.

Learning dynamics : Theorem F.3 examines the learning dynamics of coefficients of feature learning, where coefficients are defined by the weight decomposition into the feature space, providing valuable insights into the learning process. When task-relevant features dominate, the network learns the target effectively and demonstrates good performance.

Generalization bound : By leveraging the learning dynamics of the coefficients, we can demonstrate the generalization bound or test performance post-training. In our work, we connect the performance of prompt fine-tuning with the ratio between task-relevant coefficients and task-irrelevant coefficients.

D Evidence of assumption

Here's the evidence supporting the second assumption that the features of the pretrained model are orthogonal. We focus on the final projection layer of the text encoder in CLIP [38] and calculate its cosine similarity between each row. The statistical distribution of the cosine similarity is shown in Figure D.

Here, we find that nearly all of the cosine similarity are smaller than 0 . 1 , which is close to orthogonal. This evidence support our theoretical assumption. Except the evidence of cosine similarity, the

previous research [20] also make assumption of orthonormal rows of weight matrix to the multimodality latent space.

E Analysis framework

In this paper, our analysis will be made under the following assumptions:

The dimension of latent space m is sufficiently large m = ˜ Ω( n ) .

The number of training samples and clients follows n = Ω( polylog(d) ) , K = Θ(1) . The federated network shares the same initialization.

The learning rate η ≤ ˜ O ( K E min {|| µ || 2 2 , σ -2 p m -1 } ) and the standard deviation of network weight initialization σ 0 ≤ ˜ O ( mn ) · min { ( || µ || 2 2 , σ p √ d -1 ) }

For the first assumption, we assume the dimension of latent space is large enough to make our analysis ensure an over-parameterized setting. The second assumption provides statistical properties to the training data and network weight initialization. The third assumption of a small learning rate and small initialization guarantees that gradient descent effectively minimizes the training loss.

Note that this analysis framework can be directly applied to our algorithm PromptFolio. To achieve a basic federated version of prompt learning, i.e., PromptFL, we apply the learnable prompt p as the global prompt p G in PromptFolio and apply the mixing coefficient θ = 0 , we obtain the theoretical result of PromptFL. Similarly, we can apply the learnable prompt as the local prompt p L and apply the mixing coefficient θ = 1 , we obtain the theoretical result of CoOp.

E.1 Gradient update analysis

This section discusses the computational approach used to analyze the performance of the algorithm, focusing on the gradient calculations essential for optimizing the model parameters. To calculate the gradient, we first need to find the partial derivatives of sim ( g k,i , h k,i ) with respect to p G and p L . Let σ ′ G,r,k,i and σ ′ L,r,k,i are the r -th diagonal elements in σ ′ ( w T r p G,k + w T r p y k,i ) + σ ′ ( -w T r p G,k + w T r p y k,i ) and σ ′ ( w T r p L,k + w T r p y k,i ) + σ ′ ( -w T r p L,k + w T r p y k,i ) respectively:

Using the logistic loss L T for client k , the goal is to minimize the subsequent loss function to achieve supervised fine-tuning of the text prompts:

Now, we can use these partial derivatives to compute the gradients with respect to p G and p L :

To simplify the update process, we can rewrite it using summation notation. Let ℓ ′ k,i denote exp( sim ( g k,i , h k,i )) 1+exp( sim ( g k,i , h k,i )) :

In the above equations, x r,k,i represents the r -th row of g ( x k,i ) . Note that w T r represents the r -th row of the weight matrix W .

E.2 Signal-noise decomposition

In this subsection, we provide the proof of signal-noise decomposition as mentioned in Lemma 4.1. Note that the local update of global prompt coefficients can be generalized to local prompts by changing the subscript G to L . The coefficient β G/L,k denotes the coefficient of the global prompt at client k . The coefficient γ G/L,k,k ′ denotes the coefficient of the local prompt k ′ at client k . The coefficient ϕ G/L,k,k ′ ,i denotes the coefficient of client k ′ 's i -th task-irrelevant feature at client k .

Here, we suppose that the weight of the pretrained model consists of two kinds of weights, taskrelevant weight µ and task-irrelevant weight ξ . Without loss of generality, we can rewrite the gradient descent update of the prompt as follows:

According to the update formula of the gradient descent, we have the update formula of the global prompt and the local prompt:

As a result, for the row corresponding to the global feature line, we have the update formula of the coefficients:

For the row corresponding to the local feature line, we have the update formula of the coefficients:

For the row corresponding to the task-irrelevant feature line, we have the update formula of the coefficients:

To analyze the increase of the coefficient, we decompose the coefficient ϕ ( t ) G,k,l to ψ ( t ) G,k,l and φ ( t ) G,k,l .

Here, we suppose that ξ k,i = L ∑ l =1 x k,i,l ξ l , thus we have the following lemma.

Lemma E.2 ([6]) . Suppose that δ ≥ 0 and L ≥ log(4 n/δ ) , then with probability at least 1 -δ , we have

where we denote σ 2 L = L ∑ l =1 || ξ l || 2 2 as the summation of the norm among all task-irrelevant features.

E.3 Generalization analysis

For simplicity, we first introduce the definitions of F + and F -. Here F + means the train loss corresponding to the positive class, while F -means the train loss corresponding to the negative class.

To illustrate the decoupling of the label from the definitions of F + and F -, we have the following lemma.

Lemma E.3. Suppose that y is the ground truth label, we have

Proof. We consider two situations: y = +1 and y = -1 . If y = +1 ,

Lemma E.4. Under the modeling of prompt-based federated learning, the expectation of test loss L D of an algorithm can be treated as the probability

where µ and σ are controlled by the coefficients of feature learning. Thus, the performance of an algorithm can be evaluated by the ratio µ/σ .

Proof. Recall that the test error is equivalent to

According to Lemma E.3, we have that

Wp + and Wp -can be treated as two constant terms. We consider each line of F ( p ) and x , then the problem is equivalent to

where

Note that the above equation is equivalent to

Note that when we finish training, the coefficients are fixed and thus can be treated as constants. x j are zero mean Gaussian variables, y are { +1 , -1 } Bernoulli variables and x j and y are independent. Thus the test error can be defined by two values

If y = -1 ,

In conclusion, we have

Note that

where

The problem can be treated as where

Theorem E.5. Suppose that the coefficients of CoOp at step k are β k , γ k and ϕ k , the coefficients of PromptFL at step k are β k , γ k and ϕ k . Here, we define a := O ( β k + γ k β k + γ k ) and b := O ( ϕ k ϕ k ) as the ratio of different coefficients and the correlation between two prompts is defined as ρ ∈ [0 , 1] . We have that the mixed PromptFolio algorithm has a lower test loss than the mixing test loss of CoOp and PromptFL.

Proof. Then we suppose that at the end of training, the random variable corresponding to L D ( p G ) is z G ∼ N ( µ G , σ 2 G ) , the random variable corresponding to L D ( p L ) is z L ∼ N ( µ L , σ 2 L ) . Consequently, we have the random variable corresponding to PromptFolio as z GLo = (1 -θ ) z G + θz L , then we have

where 0 ≤ ρ ≤ 1 is the correlation coefficient. Assume that a = µ L µ G , b = ξ L ξ G , we have

when a ≥ b 2 -b b -ρ . As a result, we have that

Here, we find where this inequality becomes an equality. After further simplification, we observe that this equation is a quartic equation, and we've discarded 0, 1, and another unreasonable solution. The remaining solution can be expressed as

Here, for any a, b , we take the

due to the derivative of this objective function being less than zero when θ = 0 . As a result, when we take θ in (59), we will obtain the inequality.

Additionally, we provide the optimal coefficient of the algorithm and establish the following theorem. Theorem E.6. Suppose that the coefficients of CoOp at step k are β k , γ k and ϕ k , the coefficients of PromptFL at step k are β k , γ k and ϕ k . Here, we define a := O ( β k + γ k β k + γ k ) and b := O ( ϕ k ϕ k ) as the ratio of different coefficients and the correlation between two prompts is defined as ρ ∈ [0 , 1] . Then we have the optimal mixing coefficient θ ⋆ as

Proof. Here, we suppose that the local prompt corresponding to random variables µ and σ , the global prompt corresponding to the random variables µ and σ . To achieve the highest accuracy, we want to solve the following maximization problem

which is equivalent to

Note that due to µ and µ are all positive numbers, so we can make a simplification as ( a ) . Here, we take a = µ µ , b = σ σ , and divide ab 2 on both sides, we have

Thus, we take further simplification and we get

Note that this is a linear equation, so we obtain the unique optimal θ ⋆

F Theoretical analysis for PromptFolio

In this section, we provide the whole feature coefficient dynamics through a two-stage analysis. The sections are arranged as follows: We first provide some preliminary lemmas. Then we provide the order of coefficients at the beginning of the training, which is stage one. Finally, we provide the coefficients at the convergence of the loss, which is stage two.

Remark: For simplicity of the analysis, we assume that all the clients have the same number of samples and thus n 1 = · · · = n K = n/K .

F.1 Coefficient dynamics: stage one

where C 1 is a positive constant.

Proof. In the first stage, we consider that the gradient of the loss ℓ ′ is bounded in a constant level. We assume that the coefficients γ ( t ) , β ( t ) , ϕ ( t ) = O (1) for 0 ≤ t ≤ T 1 . Here we have that the output similarity satisfies:

When t ∈ [0 , T ] , we have

Theorem F.2. Under Assumption E.1, there exists total number of local updates T 1 = R 1 E = O ( η -1 Knσ 2 p σ 2 L ) such that

Proof. Part 1: Analysis of β ( T 1 ) G,k and γ ( T 1 ) G,k

We first consider the growth of local task-relevant coefficient γ ( t ) G,k,k and γ ( t ) G,k on the corresponding client k . Consider the iteration equation for the coefficient of signal learning under local gradient descent in the first round, we have:

According to Lemma F.1, we have an upper bound for signal learning at the first round of local updates.

Taking a telescoping sum over t = 0 , 1 , . . . , E , we can obtain the upper bound for signal learning before the first step of weight averaging.

After taking the average among the coefficients, we have

Note that we denote χ k ∆ = ∑ K k ′ =1 ⟨ µ ′ k , µ k ⟩ || µ k || as the total similarity between signal vectors among clients. In the following E gradient descent steps on each clients, we have

In the second round of update, we apply average among the clients

We repeat the computation process and obtain the following results for the third rounds of local updates and the FedAvg process.

Thus, after R 1 = T 1 /E rounds of communication, the noise memorization has an upper bound:

We can similarly obtain the lower bound of this proof. Consider the first round of gradient update, we have

So before taking the first average, we take a telescope sum over t = 0 , . . . , E -1 yielding:

After the weight average operation, we have

Recall that χ k ∆ = ∑ K k ′ =1 ⟨ µ k , µ k ′ ⟩ || µ k || 2 2 counts the total similarity between the signal vectors among clients. In the next E gradient descent steps on each client, we have

After gradient descent, we apply the second weight averaging operation on the server and obtain the following result

Similarly, we repeat the computation procedure and obtain the following results for the R 1 -th round of local updates plus weight average operation:

As a result, we have

Here, the global task-relevant feature follows that χ K = K , so we have

Part 2: Analysis of β ( T 1 ) L,k and γ ( T 1 ) L,k,k

Similar to the global coefficient β ( T 1 ) G,k and γ ( T 1 ) G,k,k , we first analyze γ ( T 1 ) L,k,k . Consider the iteration equation for the coefficient

According to Lemma F.1, we have upper bound for γ ( t +1) L,k,k .

Then we taking a telescoping sum over t = 0 , 1 , . . . , R 1 E , we have

Similarly, we repeat the computation procedure and have the lower bound of the γ ( t +1) L,k,k

According to Lemma F.1, we have

Then we taking a telescoping sum over t = 0 , 1 , . . . , R 1 E , we have

As a result, we have

The global feature and local feature have the same weight and take the same order.

Part 3: Analysis of ϕ ( T 1 ) G,k,l and ϕ ( T 1 ) L,k,l

We first establish the lower bound for ϕ ( T 1 ) G,k,l . We show that

̸

where C 2 is a positive constant that satisfies C 2 ≥ max { ψ ( t ) G,k,l , φ ( t ) G,k,l } , ( a ) is by Lemma E.2 and ( b ) is by definition of C 2 .

Here, we suppose that Φ ( t ) G,k,l = max { ⟨ p (0) G,k , ξ k,i ⟩ + ψ ( t ) G,k,l -4 C 2 n √ log(4 n 2 /δ ) σ 2 L } . At initialization, it is easy to check that:

where ( a ) is by the following condition:

We can compute the growth of Φ ( t ) G,k,l as follows:

Before the first step of weight average, we take the telescoping sum:

Then, we perform weight average operation

The next E gradient descent steps yield:

Here, we take the average among the clients and we get:

We use the same technique to obtain the lower bound of noise memorization

Finally, we confirm that

where the inequality ( a ) is by definition of constant C 3 ≤ 4 C 2 n √ log(4 n 2 /δ ) σ 2 L which holds when σ 2 L ≥ C 5 log(4 n 2 /δ ) n 2 and the inequality ( b ) is by taking the value of T 1 .

Then we provide the upper bound of the ϕ ( T 1 ) G,k,l . Here, we suppose that Ψ ( t ) G,k,l = max { ψ ( t ) G,k,l , -φ ( t ) G,k,l } . Similarly, we define the coefficient after weight average Ψ ( t ) G,k,l = max { ψ ( t ) G,k,l , -φ ( t ) G,k,l } . Clearly, we have that Ψ (0) G,k,l = 0 for all l ∈ [ L ] and k ∈ [ K ] by definition. Then we have

where ( a ) follows | ℓ ′ ( t ) k,i | ≤ 1 and σ ′ ≤ 1 in Lemma F.1 and Lemma E.2. Similar to above proof, we have that

After t = E steps, we perform a weight average operation and we have

Similar to the above proof, we have

Thus, it is confirmed that Ψ ( T 1 ) G,k,l = O (1 -θ ) . Similar to the convergence of Ψ ( T 1 ) G,k,l = O (1 -θ ) , we have that Ψ ( T 1 ) L,k,l = O ( θ ) .

Lemma F.3. There exists total number of local updates T 1 = R 1 E = O ( η -1 Knσ 2 p σ 2 L ) such that

Here, n = ∑ k n k /K is the average number of data in each client and SNR G = || µ G || / ( σ p √ m ) , SNR k = || µ k || / ( σ p √ m ) denote the signal-noise ratio between the task-relevant feature and taskirrelevant feature. Here, we define χ k = ∑ K k ′ =1 ⟨ µ k , µ k ′ ⟩ / || µ k || 2 2 .

Proof. The proof of the result follows the proof of global coefficient in Lemma F.3. We take θ = 0 to obtain the final result.

F.2 Coefficient dynamics: stage two

In this stage, we focus on the proof of scaling behaviour for the coefficients of features β G , γ G , β L , γ L . We first provide the following lemma.

⟨ ⟨ p ( t ) L,k , y µ L ⟩ = ⟨ p (0) L,k , y µ L ⟩ + β ( t ) G , ⟨ p ( t ) L,k , y µ k ⟩ = ⟨ p (0) L,k , y µ k ⟩ + γ ( t ) L,k , ∀ k ∈ [ K ] (116) for all k ∈ [ K ] .

Proof. According to the update formula of p G and p L , we have

where the second equation is by the orthogonal assumption between the feature vector and noise vector. Here, these equations follow the same proof and we thus complete the proof.

Lemma F.5. Under Assumption E.1, for 0 ≤ t ≤ T ∗ , where T ∗ = η -1 poly ( || µ || -1 2 , σ -2 L σ -2 p , σ -1 0 , n, m ) , we prove that

for all k ∈ [ K ] .

Proof. Here, we only prove that 0 ≤ γ ( t ) G,k ≤ γ ( t ) G,k,k ≤ (1 -θ ) nχ k SNR 2 k log( T ∗ ) . The proof of other coefficients can be generalized from this proof. Considering the update formula, we have

Let T b = R b E to be the last time t ≤ T ∗ that γ ( t ) G,k ≤ 0 . 5 log( T ∗ ) n SNR 2 k . We have the lower bound and upper bound of ⟨ p ( t ) G,k , µ k ⟩ :

Due to that

where the first equality ( a ) is by the update formula of γ , the second inequality ( b ) is by γ G,k ≥ 0 . 5(1 -θ ) nχ k SNR 2 k log( T ∗ ) and ⟨ p (0) G,k , µ k ⟩ ≥ -0 . 5(1 -θ ) α due to the definition of T c and α . The last inequality ( c ) is by α ≤ 0 . 5 nχ k SNR 2 k log( T ∗ ) . Similarly, we can also derive the upper bound as follows:

According to the above formula, we have

where the first inequality ( a ) is by the definition of ℓ ′ , the second inequality ( b ) is by the above formula and the third inequality ( c ) is due to that nχ k SNR 2 k ≥ 1 and the induction hypothesis γ ( T b ) G,k ≤ 0 . 5(1 -θ ) nχ k SNR 2 k log( T ∗ ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As stated in the abstract and introduction, this paper provides the first theoretical framework to analyze the dynamics of prompt learning federated learning from pretrained vision-language foundation models. These claims are reflected throughout the paper, with theoretical analysis and experimental validation provided in support.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation analysis is provided in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper lays out a detailed theoretical framework with clear assumptions and provides mathematical proofs for the theorems introduced. Assumptions and their implications are explicitly stated in Section 4 and Appendix E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper details the experimental settings thoroughly, including data distributions, models used (e.g., CLIP), and the specifics of the federated learning setup in Section 6. It provides enough detail for replicating the main experimental results, which align with the theoretical claims.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available alongside the submission, including a ZIP file with all relevant scripts. We will make the code open access once the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides comprehensive details about the experimental settings, including descriptions of datasets used, hyperparameters, data splits, and other necessary details in Section 6 to understand the experiments fully.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results section includes error bars and other statistical measures to indicate the significance and reliability of the experimental outcomes. This is particularly visible in the detailed tables and figures that compare different methods and settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper now provides detailed information on the computational resources used for the experiments, including the type of GPUs, memory requirements, and the execution time for each setup in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The work conformed with the NeurIPS Code of Ethics. The paper does not contain any content or methods that would violate ethical guidelines, and it maintains anonymity and integrity in its experiments and data handling.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper primarily focuses on theoretical research in machine learning and does not address societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risk that have a high risk for misuse, so this question is not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper adequately credits original sources for assets such as datasets and models used in the experiments. It mentions using standard datasets and references the original works where these models and datasets were introduced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce any new assets that would require additional documentation or licensing information.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or direct research with human subjects, so this question is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Since the paper does not involve human subjects, there is no need for IRB approvals or equivalent ethical reviews.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this papaer, the authors proposed a method to finetune vision-language model (CLIP) via prompt learning. Specifically, the author optimize a global textual prompt which will be optimized via FedAvg, while each client additionally optimize a local textual prompt. The proposed method is a combination of the existing methods CoOp and PromptFL and outperforms both methods.

STRENGTHS:
1. The paper is well-written.
2. The theoretical analysis of the proposed method is thorough and clear.

WEAKNESSES:
1. The proposed method is basically a combination of both CoOp and PromptFL, or to be more specific, optimizing both client-specific and client-agnostic textual prompt. The novelty of the proposed needs to be further highlighted in the main paper.
2. The combination strategy seems to be a bit straight forward, i.e., using a single scalar value $\theta$, the author should discuss other possibilities.
3. The authors should compare with other prompt-based tuning methods for FL, e.g., [1][2]. Also, adding more ablation studies could be beneficial.

[1] Guo T, Guo S, Wang J. Pfedprompt: Learning personalized prompt for vision-language models in federated learning[C]//Proceedings of the ACM Web Conference 2023. 2023: 1364-1374.

[2] Qiu C, Li X, Mummadi C K, et al. Text-driven Prompt Generation for Vision-Language Models in Federated Learning[J]. arXiv preprint arXiv:2310.06123, 2023.

QUESTIONS:
Please see the questions listed in the weakneses section.

LIMITATIONS:
The authors do not discuss the limitation of the method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the integration of pretrained vision-language foundation models, such as CLIP, into federated learning. The idea is to use prompt-based federated learning to minimize the communication and computational costs. The authors go into the theoretical analysis to understand the performance of this approach\n\nSTRENGTHS:\n- The authors provide a theoretical understanding on prompt Federated Learning for models like CLIP.\n- The results demonstrate that the combination of global and local prompts can support increased accuracies. \n- The overall investigation of this space is interesting and useful for the community\n\nWEAKNESSES:\n- While the overall approach is quite interesting, the authors restrict their investigation in a much simpler classification task. It is unclear why a larger foundation model is required for simpler classification tasks, where traditional FL techniques on vanilla models (e.g., reset, mobileNet etc) might work better towards learning new unseen distributions that a foundation model might not have seen (e.g,. x-ray hospital images etc). \n- Following the previous comment, it is unclear how this method would work on unseen distributions that most FL applications are mostly useful for. Most FL models are quite good at generalising to data that can be found on the public internet, but it is unknown how well they would work on out-of-distribution private data. \n- Given the fact that there are a few related prompt-based FL methods (e.g., promtFL, FedPrompt, etc) limit a bit the novelty of this work to mostly providing a solid theoretical analysis. \n- There might be some privacy implications from this work, as sensitive data are shared. \n- The evaluation was done with mostly 10 (up to 50) users, whereas in typical FL scenarios we want to learn from million of users. It is unclear how these results would generalise. Overall, the evaluation is a bit simplistic, showing only accuracy results on a few tasks.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies theoretical properties of prompt based federated learning methods for visual language models. Following the proposed theoretical results, a new algorithm named Global-local Prompt Portfolio for Federated Learning (PromptFolio) was proposed. The proposed algorithm was examined in image classification tasks using a CLIP model.

STRENGTHS:
Analyzing prompt-based federated learning methods with feature representations and exploiting their relation to portfolio optimization is a nice idea. The proposed analyses also led to a new algorithm for prompt based federated learning.

WEAKNESSES:
Experimental analyses are limited and should be improved as well.

QUESTIONS:
In the algorithm, each client sends prompts to a server. How do assure privacy protection in this case? 

In the analyses, CLIP was utilized as a foundation model. However, the main claim of the paper is providing a generic framework for a more general category of visual language foundation models. Therefore, the proposed method should be examined using the other visual language models as well.

Moreover, CLIP was used for image classification tasks, with limited prompts. In order to show the generalization of the proposed method for prompts with different structures, additional results should be given for the other tasks, such as image captioning and scene description, as well.

In Fig. 2.b, the best accuracy is achieved for 20 users, and the lowest accuracy is achieved for 5 users. Could you please elaborate this result? Could you please provide additional analyses for a larger number of users as well?

LIMITATIONS:
Limitations were partially addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed methodology offers a novel take on analyzing federated learning using prompt learning (for foundation models) via feature learning theory. At its core, the idea is to identify and monitor task-relevant and task-irrelevant features, and leveraging inspiration from portfolio optimization which says to combine independent assets to maintain income while decreasing risk of investment, construct a 2 part prompt. One part which is global and another which is local, to help with personalization. By mixing these prompts optimally, the proposed PromptFolio method highlights an improvement in overall performance of the FL system.\n\nSTRENGTHS:\n- A key strength of the proposed method is its novel take on explaining prompt-based federated learning, using feature learning theory.\n- Clarity in idea and its link to existing work.\n\nWEAKNESSES:\nAbstract/Conceptual Questions:\n- Could the authors comment about the few-shot learning aspect of foundation models (+prompt learning) and how the optimal mixing coefficient would be affected in that scenario?\n- An adjacent sub-field that contains foundation models and federated learning is the study of their impact in learning new classes using prior knowledge (base vs. new classes performance). Could you comment on how this sub-field could relate to the proposed method?\n\nClarifications:\n- Could the authors clarify the exact dimensionality of the prompt used in each of the baselines as well as PromptFolio?\n- Could the authors mention if there are previous works that have proposed a 2-level prompt learning idea? (since Section 2.2 L 107 mentions that an exploration of such a idea and the cooperation between prompts is sparse)\n- In Section 4. L. 224, the authors mention that \"task-relevant coefficients can be directly added\". Could you clarify this statement?\n- Could the authors highlight the data setting used to generate Table 1?\n- Could the authors explicitly state the train and test conditions (settings) used within the experiments?\n\nSuggestions:\n- Since the proposal of PromptFL, there are have been further advances to the idea of prompt learning within the federated setting. Could the authors comment and draw comparisons against the more recent methods?\n\nQUESTIONS:\nThe questions have been included in the weaknesses section above.\n\nLIMITATIONS:\nyes, the authors have addressed limitations within their choices of the experiment setting. However, I would recommend drawing more broad comparisons as to potential missing pieces in comparison to more recent work as well as the finer points from the feature learning theory that don't fully match up to the settings in the FL system.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper tackles the interesting problem of analyzing federated learning (FL) from vision-language foundation models like CLIP. The authors develop a theoretical framework based on feature learning theory to understand how prompt-based FL works. They introduce a new algorithm called PromptFolio that mixes global and local prompts, drawing an analogy to portfolio optimization in finance. The theoretical analysis shows how this approach can balance generalization and personalization in FL. They back up their claims with experiments on several datasets.

STRENGTHS:
- The feature learning approach they develop seems pretty solid. 
- The connection between prompt mixing and financial portfolio theory is novel.
- The ablation studies on data heterogeneity and client numbers are particularly informative.

WEAKNESSES:
- This reviewer is confused by the feature learning theory (I am not familiar with it), which is introduced in a relatively short section. Is it (looks like) a common sense in deep learning or a kind of new theory proposed recently? It would be helpful to have a more thorough introduction to the theory.
- Using a single activation function for the text encoder is a pretty big simplification from how CLIP and other large language models actually work.
- All the experiments focus on classification tasks. Any thoughts on how PromptFolio might perform on other vision-language tasks like image captioning or visual QA in a federated setting?
- Why is the proposed method on OxfordPets dataset not as good as coop? While it seems to be dominating on other datasets.

QUESTIONS:
see weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18