Personalized Collaborative Fine-Tuning for On-Device Large Language Models

Authors: nicolas  wagner
dongyang  fan
martin  jaggi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 17:27:59

Abstract: We explore on-device collaborative fine-tuning of large language models under limited local data availability. We introduce three distinct dynamic collaborator selection schemes, allowing trust-weighted personalized update aggregation: model-similarity-based, prediction-similarity-based and validation-performance-based. To minimize communication overhead, we integrate Low-Rank Adaptation (LoRA) and only exchange LoRA model updates. Our protocols, driven by prediction and performance metrics, surpass both FedAvg and local fine-tuning methods, which is particularly evident in realistic distributed scenarios with more diverse local data distributions. The results underscore the effectiveness of our approach in addressing heterogeneity and scarcity of the local datasets.

Main paper: 
 introduction: 
 Recently, there has been an unprecedented surge in the popularity of Large Language Models (LLMs), driven by their versatile capability to solve a wide range of tasks and serve as general-purpose models (Touvron et al., 2023; OpenAI et al., 2024; Jiang et al., 2024) . Their downstream performances can be further enhanced by fine-tuning, which is typically conducted on fewer data and independently by different users, rather than in a centralized way. The scarcity of local data often renders local fine-tuning ineffective, necessitating collaboration. However, in many cases, end users usually have privacy concerns over their local data, such as patient records for hospitals (Li et al., 2023) and typing history for mobile phone users (Hard et al., 2018) . Naturally, one might wonder how we could enable users to collaborate to still obtain better models in the presence of small and privacy-sensitive local data.The possibility of on-device fine-tuning of LLMs has been facilitated by parameter-efficient techniques such as prompt tuning (Lester et al., 2021) , adapters (Houlsby et al., 2019) and Low-Rank Adaptation (LoRA, Hu et al. (2022) ). LoRA has gained widespread popularity for fine-tuning large language models since its introduction, which approximates model updates through the multiplication of low-rank matrices. LoRA modules align with our collaboration initiative, as they allow for a significant reduction in communication overhead. A naive approach to leveraging local data from other users would be to average LoRA weights, which is the popular FedAvg algorithm proposed by McMahan et al. (2017) . However, the resulting one global model solution might not fit all users' data distributions, especially when there is more heterogeneity present. As a result, we aim to develop personalized collaboration strategies for each user, which should surpass the performance of local fine-tuning and naive averaging approaches.In pursuit of this objective, we introduce several collaborator selection protocols tailored for optimal LoRA weight averaging across diverse user profiles. While similar concepts have been extensively investigated within collaborative learning communities (Zhang et al., 2021; Sui et al., 2022; Fan et al., 2024) , these approaches are only designed for supervised classification tasks and thus are not directly applicable to Large Language Models (LLMs). Nevertheless, we investigate how we can adapt and apply analogous principles to determine optimal aggregation weights for LoRA matrices on a per-user basis.Following some computational social science literatures (Urena et al., 2019; Zhang et al., 2022) , we term the aggregation weight matrix "trust", as if we view clients as members of a social group, their exchanged information can thus be regarded as opinions. Similarity measures can be useful for assessing trust between users, as users are more likely to consider and trust others with similar opinions. Such similarity measures are core to our proposed protocols, which we will discuss in Section 3.1.Our contributions can be summarized in the following aspects:• We are the first to tackle realistic data heterogeneity among users in personalized collaborative language modeling, encompassing diverse topic distributions and varying language usage.• We demonstrate that collaboration can be leveraged to improve personalization performance in language modeling and show that predictions are more effective than model parameters for identifying collaborators in the language domain.• Our approach is well-suited for on-device fine-tuning, effectively mitigating challenges related to data scarcity and resource limitations. 
related work: 
 We focus on a peer-to-peer decentralized learning setting, where the existence of a central server is not assumed. Instead, the end users conduct peer-to-peer communication using decentralized schemes such as gossip averaging to aggregate local information across agents. To enable a personalized weighted aggregation, several recent works have proposed data-dependent communication protocols based on task similarities and node qualities. Notably, Zhang et al. (2021) derives a first-order approximation for optimal aggregation weights w ⋆ ij for collaboration, which happens to be proportional to how well client j's model generalizes on client i's data. Li et al. (2022) directly optimizes the mixing weights by minimizing the local validation loss per node. Sui et al. (2022) uses the E-step of the EM algorithm to gauge the significance of other agents for a specific agent i, achieved through assessing the accuracy of these agents' models on the local data of agent i. Fan et al. (2024) adopt a different approach by comparing prediction similarity on a publicly available dataset, which largely reduces communication overhead and allows for model heterogeneity. These approaches have been showing promising performances in supervised deep learning experiments. For language models, the data heterogeneity across users is not well studied, and the effectiveness of collaborator selection methods has not been explored for LLMs, where self-supervised training is performed.Recently, groups of researchers have been investigating the intersection of Federated Learning and Large Language Model training. Due to the substantial number of model parameters, these efforts primarily focus on post-training stages, often incorporating parameter-efficient techniques (Fan et al., 2023; Ye et al., 2024; Flower.ai, 2024; Flare, 2024) . Federated pretraining was studied in (Sani et al., 2024) . Zhang et al. (2023) employ FedAvg for collaborative instruction tuning, where they demonstrate superior performance in task generalization compared to local instruction tuning. Che et al. (2023) design an adaptive optimization method for collaborative prompt tuning, where a scoring method is applied to measure the importance of each layer and only prompt parameters of more important layers are exchanged during the tuning process, ensuring communication efficiency. Cho et al. (2024) tackles the challenge brought up by different LoRA ranks on heterogeneous devices. By performing rank self-pruning locally and sparsity-weighted aggregation at the server, improved convergence speed and final performance can be achieved. In contrast to these methods, we aim at providing personalized models to each user, instead of a global solution. 
method: 
 To start with, we first introduce how LoRA works. For a pre-trained model parameterized with Φ ∈ R m×n , the model updates during fine-tuning stage can be approximated by [Θ❄ 🔥 , θ (0) 4 ] [Θ❄ 🔥 , θ (0) 3 ] [Θ❄ 🔥 , θ (0) 2 ] [Θ❄ 🔥 , θ (0) 1 ] [ Δ θ 1 , s 1 ] [Δθ 3 , s 3 ] [ Δ θ 4 , s 4 ] θ i ← θ i - ∑ j w ij Δθ j W = g(We work in a setting where all users have the same LoRA rank r, and aim to arrive at their personalized ∆Φ i . During the fine-tuning phase, only LoRA parameters will be updated, and the original weights of the pre-trained model will remain frozen. Therefore, it suffices to communicate LoRA models, instead of the big chunk of whole parameters.We stick to the standard approach of sharing and aggregating model updates. Each user, through their unique data distribution, contributes to the collective learning process by sharing local model updates. Central to our approach is the so-called trust matrix, which we define as the gossip aggregation matrix to guide each user's aggregation of model updates.The calculation of trust matrix W will be detailed in Section 3.2. We use W i• to denote the ith row of W. A simplified diagram of our protocol is shown in Figure 1 .Our proposed protocol is described in Algorithm 1. To summarize, the protocol is constituted with three phases: 1) each user i ∈ [N] computes their LoRA model update ∆θ i with respect to their local data (and if required, predictions f θ i (X S ) with respect to the shared dataset X S ); 2) each user communicates [θ i , ∆θ i ] or [ f θ i (X S ), ∆θ i ] to all other users, depending on the trust calculation strategy; 3) each user i ∈ [N] calculates their unique trust weights W i• locally in parallel and updates their LoRA parameters using trust-weighted received model updates. It is noteworthy that different means of trust weight calculation require different information to be shared across the users, thus inducing distinct computation complexities and communication overheads. We will discuss those aspects in Section 3.3.In this section, we explore different ways of building trust -gossip aggregation graph across users. Essentially, the weight of edge (i, j) should denote to what extent can user j's model help to facilitate user i's learning progress.We employ pairwise LoRA parameter similarity to compute trust scores among users. The underlying idea is that users whose model parameters exhibit closer alignment likely possess data distributions that are more similar, thereby rendering their contributions more pertinent and advantageous to one another. This methodology shares a resemblance with (Li et ϑ (t) i = ∑ j∈[N] w ij ∆θ (t-1) jUpdate personal LoRA parameters, with learning rate ηθ (t) i = θ (t-1) i -ηϑ (t) i end for 2022), where the mixing weight is derived from the inner product between representations of different users' models via an encoder model. However, our approach involves directly measuring the similarity between LoRA models without needing an encoding module.It appears that in addition to LoRA model updates, LoRA models also need to be shared to compute the trust matrix. However, this additional communication can be efficiently circumvented, as the updates to LoRA models are already exchanged, and the models themselves can be readily derived by aggregating these updates in each iteration. The collaboration trust matrix W is computed as follows:wt ij = ⟨θ (t-1) i , θ (t-1) j ⟩ ∥θ (t-1) i ∥∥θ (t-1) j ∥ , W t = SoftMax( W t , dim = 1) (2)Drawing inspiration from Zhang et al. (2021) , we evaluate the pairwise trust w ij based on how well user j's model performs on the validation sets of user i. This method requires extra validation sets within each user, on which the performance assessment directly reflects the relevance and potential benefit of one user's model to another. The aim is to guide the aggregation process by favoring models that demonstrate compatible performance levels, thereby enhancing the overall effectiveness of collaborative learning. The calculation is denoted as:wt ij = L( f θ t-1 j (X val i ), X val i ), W t = SoftMax(-W t , dim = 1) (3)wheref θ t-1 j (X val i )denotes a forward pass of user j's model on user i's local validation set, and L is the cross entropy loss for next token prediction.In the prediction-similarity-based approach, following the trust weighing strategy from Fan et al. (2024) , we derive aggregation weights by comparing the model's predictions on a shared dataset, in this case, logits across the vocabulary. A public dataset X S will be added to the input of Alg. 1, and additionally communicating those predictions between users is necessary, increasing communication cost. This method assesses how closely the predictions of different models align, prioritizing contributions from users whose models exhibit similar predictive behaviors. The trust is calculated based on l 1 distance between logits:wt ij = | f θ t-1 i (X S ) -f θ t-1 j (X S )|, W t = SoftMax(-W t , dim = 1) (4)Table 1 provides an overview of the communication and computation costs associated with the three proposed strategies.In terms of communication, both strategies 1 and 2 need to share LoRA models (θ i ) on top of LoRA model updates (∆θ i ). Strategy 3 requires sharing logits ( f θ i (X S )) across the public dataset, rather than LoRA models. Although the logits encompass the entire vocabulary, it is possible to heavily reduce its memory cost by artificially transforming it into a sparse matrix. The computation of trust through prediction similarity relies on the l 1 distance, and most entries of logits tend to be very close to zero, making it feasible to only keep the top-k logits values. This approach greatly cuts down on communication costs when sharing logits while still maintaining desired performance levels. For a detailed comparison of the measured communication costs, please refer to Appendix A.1. Nonetheless, as the communication of ∆θ i s remains unavoidable and constitutes the primary communication cost, there is not much variation in communication costs across all three strategies.On extra computation costs. Regarding additional computation costs to obtain W, Strategy 1 involves only calculating cosine similarities between LoRA models in addition to existing processes, the cost of which is in the order of the number of LoRA parameters. However, Strategies 2 and 3 necessitate multiple forward passes to generate predictions either on local validation sets or the publicly shared dataset. A forward pass is significantly more resource-intensive compared to cosine similarity calculations due to the model's size.In Strategy 2, evaluating all models' performance on each local validation set requires N 2 model inferences. In Strategy 3, each user needs extra forward passes to obtain predictions on X S , resulting in N times the model inference. Since X S and X val i are similar in size, the number of forward passes per inference doesn't vary much. Thus, Strategy 2 requires roughly N times more extra computation cost than Strategy 3. 1: Model-sim {θ i , ∆θ i } N i=1 O(Ldk) ⋆ 2: Validation {θ i , ∆θ i } N i=1 O(Ln 2 d + Ld 2 n) • N 2 ⋆ ⋆ ⋆ 3: Predictions { f θ i (X S ), ∆θ i } N i=1 O(Ln 2 d + Ld 2 n) • N ⋆⋆The first category is Java code from GitHub (HuggingFace) and the second category is English Wikipedia text (Wikimedia, 2022).The corresponding distributed datasets are presented in Table 2 . Note that AG News and Codes-Wikipedia simulate different topic distributions across users, while Multilingual Wikipedia simulates different language usage across users. It is worth noting that even though the topics/languages are different, they can nevertheless share the same tokens in their vocabularies. The portion of overlapped tokens is indeed not small, as measured by the Jaccard Index in Table 3 . Jaccard Index for multi-sets, is calculated asJ(D i , D j ) = |D i ∩D j | |D i |+|D j | ∈ [0, 0.5].A high Jaccard Index indicates that clients' data share a substantial number of the same tokens. Although this measure doesn't account for the context information, it provides insight into the likelihood of finding collaborators. In all our experiments, the shared public dataset X S is sampled equally from each local distribution to ensure that the logit predictions are meaningful. In practice, such X S can be chosen as any publicly available dataset from the same domain, or private data with data sanitization procedures enforced on top to secure user privacy. 
experimental details: 
 Our experimental setup is structured as follows: Each user is configured with a uniform model architecture, to ensure model updates can be shared and aggregated. We chose relatively small LLMs constrained by the limited computing resources in academia. For each user, we equip them with a GPT21 base model of 124 million parameters. It has 12 layers, 12 attention heads, embedding size 768 and vocabulary size 50257.Figure 2 : Ablation study of whether to add LoRA modules compared to full fine-tuning. Left bars correspond to Ratio, which denotes the decrease in test perplexity compared to the pre-trained model per trainable parameter (higher is better); right bars correspond to perplexity, which denotes test performance after fine-tuning (lower is better).LoRA modules were applied on top of both Causal Self-Attention (SA) and MLP layers, constituting 0.47% of full parameters. The reason why we chose the two blocks is a trade-off between the model performance and communication overhead, as shown in Figure 2 In vision tasks, realistic data heterogeneity arises from diverse domain distributions across clients and variations in image quality resulting from the use of different devices for data acquisition (Ogier du Terrail et al., 2022) . However, within the domain of written language, realistic data heterogeneity frequently arises from variations between sources. Owners of such sources can differ in writing styles, vocabulary choices, grammar structures, and topic distribution. Our experiments are designed to examine diverse topic distributions and language usages exhibited by different users.We focus on on-device fine-tuning scenarios, where the amount of available local training data may be limited. The diverse local data distribution originates from user-specific characteristics. For instance, users often possess unique topic preferences concerning news reading, and mobile phone users often type in different languages, prompting smart keyboards to enhance next-word prediction across different language blends. Following this, we investigate two levels of data heterogeneity: 1) Low heterogeneity, where each user is assigned two data categories from the entire set, and a (3/4, 1/4) mixture of data of these categories is allocated to each user. 2) High heterogeneity, where each user is exclusively assigned to one data category.The descriptions of the three dataset partitionings utilized in our experiments are as follows:1. AG News: News articles in four categories: "World", "Sports", "Business" and "Sci/Tech". (Zhang et Our chosen baseline methods are: 1) Local Fine-Tuning, where there is no communication between users, 2) FedAvg (McMahan et al., 2017) , where model updates are aggregated to obtain a global model for each user, and 3) FedAvg + Local Fine-Tuning (Jiang et al., 2023) , where we spare 10% of training data for local fine-tuning after FedAvg training. We offer an extended table where we vary the amount of training data used for fine-tuning for completeness in Table 9 in Appendix. We further incorporate a strong baseline -oracle. The determination of oracle collaboration weights relies on the similarity of underlying data source distributions, which is typically unknown in practice. We give an example of how the oracle weights are determined: if user 1 has 1/4 German texts and 3/4 French texts, and user 2 has 1/4 German texts and 3/4 Italian texts, the oracle weights would be determined by the dot product of [1/4, 3/4, 0] and [1/4, 0, 3/4]. After obtaining the pairwise dot product matrix, row normalization is performed.Low High Low High Our main findings are summarized in Table 5 . Compared to baseline approaches-local training, FedAvg, and FedAvg + FT-our test performance and prediction-based aggregation methods consistently outperform them across all datasets and scenarios. We emphasize that in language modeling, achieving a balance between personalization and collaboration is not straightforward. Despite significant differences between categories, there are notable similarities in sentence structure and word choices, alongside domain-specific vocabulary. Thus, the preferences for collaboration or personalization vary across different datasets. For example, in the case of AG News, local training yields better results than FedAvg, whereas this is not true for Codes-Wiki. The collaboration-then-personalization approach (FedAvg + Local Fine-Tuning) fails to achieve this balance. In contrast, our methods successfully accomplish it.Among our protocols, the predictions-based method exhibits the most superior test performance. Notably, the model-similarity-based aggregation method yields performance akin to FedAvg, where the trust weight is uniformly set to 1/N across all clients. Our methods can sometimes outperform oracle aggregation, suggesting that a dynamic collaborator selection protocol might be favored in different fine-tuning stages.Ablation studies on the effect of LoRA ranks and local training set size can be found in Table 8 and Table 7 in Appendix A.3. Our prediction-based strategy consistently delivers superior performance.A note on communication topology: While the paper demonstrates a fully connected communication topology, our protocols can support sparser communication patterns, as long as the graph represented by W is strongly connected (i.e., every vertex is reachable from every other vertex), though convergence will be slower. We provide experimental results on the training time required for a ring topology to achieve the same perplexity level as a fully connected topology in Table 4 . In high heterogeneity scenarios where each client is allocated a specific category, a ring topology results in cases where adjacent clients do not share any categories, leading to an oracle trust weight of 0 and hence the NA entries. Despite this, our algorithm can still learn, converging to a perplexity approximately one point higher than reported for the fully connected case.We compare our learned trust matrices with the oracle trust matrix in Figures 4 and 5 suggesting that LoRA models are not informative in collaborator selection when cosine similarity is used as the distance metric.The success of Strategies 2 and 3 underscores the advantage of utilizing predictions to identify collaborators. It is interesting how the distinctions between different users' models become more pronounced after a forward pass. Such a phenomenon was not previously observed in the vision domain. We further noticed that trust matrices in Strategy 3 are more stable across communication rounds compared to Strategy 2, with fewer abrupt changes in values. As shown in Figure 3 , the l 1 norm of the differences between Ws at consecutive steps fluctuates significantly for Strategy 2. It is speculated that the more stable trust matrix in Strategy 3 contributes to stabilizing the learning process, leading to better results.  
conclusions: 
 Drawing from insights in conventional peer-to-peer collaborative learning approaches, we propose 3 collaboration protocols tailored for jointly on-device LLM fine-tuning. To our best knowledge, this is the first work demonstrating that collaboration can be leveraged to improve personalization performance in language modeling. Our methods can efficiently address challenges encountered in on-device personalized LLMs, such as limited availability of local data and data heterogeneity stemming from user characteristics.We show that uniform collaboration as in federated averaging is not sufficient, but that dynamic collaborator selection strategies enable better and personalized models. Remarkably, for collaborator selection, predictions are more informative than model weights. Our Strategy 3, the prediction-similarity-based protocol, demonstrates promising practical applications for on-device deployment. It achieves superior performance while managing to maintain reasonable communication and computation costs. 
discussions and future work: 
 In this study, we examined resource-constrained scenarios of collaborative LLM fine-tuning. Our collaborator selection scheme successfully addresses realistic data heterogeneity across users. However, heterogeneities can arise in different aspects, such as model and resource diversities. When users possess different capacities, enabling efficient collaboration remains an important challenge to address. Cho et al. (2024) have developed collaboration protocols tailored for circumstances wherein users exhibit varying levels of support for LoRA ranks.When users are equipped with distinct numbers of LoRA modules corresponding to their capacities, can we still harness the advantages of collaborative training?Furthermore, within the scope of this research, particular emphasis is placed on unsupervised fine-tuning. The prospect of supervised fine-tuning, i.e. instruction tuning, is worth further investigation. An intriguing question arises: Can collaboration among users enhance their ability to generalize and improve performance on downstream tasks?Our collaborator selection protocols are centered around trust. While the efficacy of trust in identifying collaborators has been thoroughly explored, there is potential for further investigation into its role in identifying adversarial users, which is one of the major challenges in federated LLMs (Chen et al., 2023) . We finally note that our scheme can be extended to the case of partial participation of users and asynchronous updates, as the trust matrix could be updated less frequently than in every iteration, and decentralized averaging algorithms can tolerate asynchronous updates (Even et al., 2024) .  
 
 Appendix: To understand better the relationship between the performance of FedAvg + Local Fine-Tuning and the relative size of the local fine-tuning dataset, we spare different percentages we use one global round to represent every 25 iterations, without any actual communication between users. Figure 8 shows the learned trust matrices at convergence using the three different strategies.Even without communication, the learned trust matrix using Strategy 1 becomes uniform. This indicates that the users learned almost identical LoRA weights. We dive into this bizarre observation. In Figure 9 , we plot out the exact same trust matrices as in Figure 7 , but present with a much smaller scale. This visualization indicates that Strategy 1 indeed can identify users who are more helpful to collaborate with, but the distinction between more and less helpful collaborators is not pronounced. https://github.com/karpathy/nanoGPT 


Decision: The paper discusses methods for collaborative / distributed fine tuning, and introduces several such "trust based" methods. Experiments compare their methods to federated learning and other known methods for aggregating ML models. 
The problem is important and I find the paper to be well written. 
Reviewers ranged from lukewarm to positive, and I recommend acceptance. Please address the comments by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper discusses weighted aggregation methods for the LoRA module in decentralized learning. When freezing base networks, the LoRA modules are weighted by “trust” calculation before aggregation. The “trust” weights are computed by weights similarity, validation set, and prediction similarity.



========== after rebuttal and discussion ===============
I thank the authors for the response and raised the score to 6.

Why not lower:
I appreciate the clarification and extra work during rebuttal, and acknowledge the contribution of discussing collaboration and personalization of LoRA training of LLM in decentralized training. This is timely and relevant to COLM community. 

Why not higher:
As also mentioned by other reviewers, the strategies discussed are not entirely new, and somewhat ad-hoc to me. The motivation of "realistic" heterogeneity makes sense, but I am still a little skeptical about the setting and baseline choices in the draft. 

Finally, I would strongly encourage the authors to take the discussion into consideration, and more carefully adjust the technical terms and relationship with the literature.
strength_weakness: strength_weakness: I may have missed something, but in my humble opinion, this paper does not meet the bar of top conference. 

1) The motivation is not quite clear to me. The method seems to aim for cross-device federated learning, but it seems to also assume full participation of clients every round instead of a small subset of clients. Moreover, it seems to assume a fully connected network topology among all clients, i.e., every communication is a full reduce step. It is challenging for me to understand this setting compared to the common setting, e.g., [Advances and Open Problems in Federated Learning](https://arxiv.org/abs/1912.04977) Section 1. 

2) I somewhat feel strongly and disagree with users' usage of "trust". The weights for aggregation seems to merely depend on some kind of similarity. "Trust" is often a terminology for privacy and security, and it is hard for me to understand what the connection with "trust" is. On the other hand, the authors do not seem to properly discuss the privacy implication of the settings and the algorithms. 

3) The proposed weighted aggregation methods are somewhat ad-hoc and lacking strong supports. The proposed method did not compare with any baseline method. There are many personalization methods in FL, and they can be applied to the LoRA training setting, e.g., [Motley: Benchmarking Heterogeneity and Personalization in Federated Learning](https://arxiv.org/abs/2206.09262) and methods therein.
questions: questions: Could the authors explain how the "theoretical" matrix is computed in experiments?
paper_summary: paper_summary: This paper introduces a method for on-device collaborative fine-tuning of large language models using Low-Rank Adaptation (LoRA). The approach is distinct for its minimal communication overhead and integration of three innovative trust-weighted gradient aggregation schemes: weight similarity-based, prediction similarity-based, and validation performance-based. Through their experiments, the author shows that the aggregation trust matrix guided by prediction information (strategy 2 and 3) yields better performance than the one based on similarities.
strength_weakness: strength_weakness: - While the paper demonstrates that strategy 3 consistently outperforms strategy 2, there is no detailed analysis explaining why this is the case. Since both strategies use prediction-guided trust matrices, additional insights into their differences would strengthen the conclusions.
- The methods may not scale well with a large number of clients, which is a significant limitation given the growing scale of data and model deployments.
- The primary contribution seems to be the application of existing techniques (from Zhang 2021 and Fan 2023) to LoRA, which makes the contribution of the paper a bit incremental. 
- To enrich the comparative analysis and foster a deeper discussion, it is suggested that the manuscript incorporate several relevant on-device federated studies such as [1,2,3,4,5].


[1] "Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes" (arXiv'23, ICML'24)

[2] "FwdLLM: Efficient FedLLM using Forward Gradient" (arXiv'23)

[3] "Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources" (arXiv'24) 

[4] "Can Public Large Language Models Help Private Cross-device Federated Learning?" (arXiv'23, NAACL'24)

[5] "FS-Real: Towards Real-World Cross-Device Federated Learning" (KDD'23)
paper_summary: paper_summary: This paper investigates a personalized collaborative fine-tuning of LLM on-device, proposing three trust-weighted gradient aggregation strategies to deal with local data heterogeneity and scarcity problems. The significance of the work lies in the current landscape where data privacy, device-centric ML and federated learning are growing in importance. By pushing forward with collaborative learning directly on devices through parameter efficient fine tuning, this approach has the potential to open up new avenues for more personalized, secured and efficient applications. The paper succinctly explains three strategies, conducting limited experiments in unsupervised fine-tuning tasks, comparing them with one SOTA federated learning method. Additionally, it provides supplementary experiments aimed at interpreting the underwhelming performance of the first strategy. However, the lack of theoretical explanation or analysis for this fact and the other two strategies is notable. The paper is clear and easy to understand, although some ambiguity may arise from certain figures and letter symbols. Nevertheless, the three strategies lack significant novelty compared to the referenced work and do not introduce substantial modifications or improvements.
strength_weakness: strength_weakness: 1. The paper lacks sufficient experiments with multitask and larger datasets, and controlled experiments to demonstrate the methods’ generalization performance claimed by the authors. The three strategies were only tested in unsupervised fine-tuning task (focus on diverse topic, no considering other variation like writing styles, vocabulary choices, grammar structures, etc.), with perplexity as the sole evaluation metric. The dataset size is relatively small (approximately 1600 samples based on the provided token count and context length). Additionally, the comparison with the sole federated learning algorithm, FedAvg, and the fixed rank of the LoRA adapter (set to a small value of 4) limit the persuasiveness of the paper.
2.  Some figure and letter symbols may cause ambiguity. From the Algorithm, each user should broadcast their LoRA weight update to all others agents, while in Figure 1 just user 1, 3, 4 broadcast their updates to user 2. What does the symbol si message refer to? (LoRA weight for user i?)
3. Lack of novelty. Three strategies do not introduce substantial modification compared to the corresponding three referenced works. According the the experiment results, these methods exhibit limited improvement compared to the baselines, even perform worse than the simple but strong baseline theoretical but consume much more computing power.
4. Lack of theoretically explanation to the underwhelming performance of Strategy 1 and that of Strategy 2 and 3 compared to the strong baseline. Some facile explanations or assumptions (like "This could be attributed to the high-dimensional, sparse nature, and small magnitudes of LoRA weights"), but lack of further experiment to verify them.
5. Lack of limitation discussion.
paper_summary: paper_summary: This paper proposes an approach to fine-tune personalized language models in a collaborative setting. Specifically, the idea is to derive a set of LoRA weights for each user by aggregation across other users who are similar to this user. The paper presents 3 strategies for aggregation based on a) similarity of weights, b) similarity of predictions based on user specific validation sets, i.e. how well LORA weight of user j performs on validation set of user i, c) similarity of predictions based on a shared dataset. The paper finds prediction based similarity works best and furthermore using a shared dataset works somewhat better.

Strengths:
* Presents an effective strategy for developing personalized language models while still benefiting from other users in a collaborative setting
* Demonstrates that a weight aggregation strategy based on predictions outperforms one based on model-similarity.

Weaknesses:
* In a realistic scenario, users may be wary of sharing information with other users. Therefore, we may want to add some level of noise to a user's prediction or LoRA weights before using those weights in an aggregation setting. The paper does not discuss this privacy angle.
* The paper does not report an ablation wrt the amount of fine-tuning data and how that can affect the performance of the 3 strategies.
strength_weakness: strength_weakness: * In realistic scenarios, users are concerned about privacy. Thus, it would be important to add noise to the user's weights or predictions before incorporating them in an aggregation setting. The paper does not discuss this aspect.
* In a collaborative setting, users exhibit heterogeneity in terms of usage I.e. in terms of amount of fine-tuning data available. The paper does not discuss this dimension and how it may impact performance of the proposed approach.
questions: questions: * Why is strategy 3 not reported for AG News?
* In Figure 1, should  \delta \theta_2, s_2 be \delta \theta_3, s3?
* In Sec 3.2, "Essentially, the weight of edge ( i , j ) should denote to what extent can user j ’s gradients help to facilitate user j ’s learning progress" should this be  "user i's gradients help to facilitate user j's ..." ?

Comments: title: Request for Participation in Discussion
More clarifications
Clarification of our contributions
comment: Thanks for providing more details. I would like to my rating the same.
Dear Reviewers,

We kindly request that you review our response and join the discussion if you have not done so. As the discussion period is nearing its conclusion, we hope our efforts have adequately addressed your concerns. Thank you very much!

Best,
Submission 837 authors
Thanks for your response and score adjustment. We will include your helpful comments in future iterations.

We address your concerns as follows:

- We opted for a fully connected topology because we are focusing on a cross-silo scenario with a small number of clients. This is the case for the standard federated learning setup, where the server can easily relay meta-data between all clients (see Table 2.1 in [4]). In the more general decentralized case where a different  connected topology is preferred, one can replace the current trust matrix W with a sparse version, where some entries are zero. Our setting thus includes both federated learning as well as arbitrary decentralized training. As long as the graph represented by W is strongly connected (i.e., every vertex is reachable from every other vertex), our algorithm should still function, though convergence will be slower. 
      
We provide experimental results on the training time (X times the needed training iterations as in the fully connected case) required for a ring topology to achieve the same perplexity level as a fully connected topology in the table below. Note that NA indicates we did not reach the same perplexity after ten times the training iterations. In high heterogeneity scenarios with a very sparse ring topology, each client is allocated a specific category. This can result in cases where adjacent clients in the ring do not share any categories, leading to an oracle trust weight of 0 and hence the NA entries. Despite this, our algorithm can still learn, converging to a perplexity approximately one point higher than what is reported for the fully connected case.

Method | Multi-Wiki Low | Multi-Wiki High | Codes-Wiki Low | Codes-Wiki High |
| ----------- | ----------- | ----------- | ----------- | ----------- |  
Local | 1 | 1 | 1| 1 |
FedAvg | 1.6 | 4.2 | 2.0| 1.7 |
S1 | 5.9 | 12.0 |2.0 | 2.1 |
S2 | 1.1 | 2.6 | 1.5 | NA |
S3 | 2.9 | NA |1.2 | 3.8 |

2. We summarize the insights as follows:
- From strategy 1 underperforming the other two strategies, we conclude that predictions are more informative than model weights in identifying collaborators in language modeling.
- Both as predictions-based collaborator selection methods, strategy 3 outperforms strategy 2. We further investigated the evolution of learned trust matrices. We found the trust matrices in strategy 3 are more stable across communication rounds compared to strategy 2, with fewer abrupt changes in values, and the trust matrices from strategy 3 have higher diagonal values than those from strategy 2. We cannot definitively explain why strategy 2 underperforms, but we speculate that the more stable trust matrix and higher self-trust in strategy 3 contribute to stabilizing the learning process, leading to better results.

3. As we mentioned in an earlier response, we are not proposing a SOTA method, instead we aim to understand how to balance personalization and collaboration. Moreover, as we focus on a server-less setup, the standard way is to use gossip averaging algorithm (which does collaborator selection), without getting a global model state involved [1,2,3]. That is also a reason why we did not compare personalization algorithms where a server is assumed.

Thanks a lot and we welcome further discussions if things are unclear. 

[1] Sui et al. Find your friends: personalized federated learning with the right collaborators

[2] Zhang et al. Personalized Federated Learning with First Order Model Optimization

[3] Fan et al. Collaborative Learning via Prediction Consensus

[4] Kairouz et al. Advances and Open Problems in Federated Learning
I thank the authors for the efforts, and raised the score to 5.

I am not fully convinced on my previous concern 1 about the contributions of this paper and position it in the literature, particularly a) justify the motivation of fully connected topology of decentralized learning b) justify the insights behind the three strategies studied in this draft and c) justify FedAvg/LocalTraining as the main baseline. I acknowledge c) has been partially addressed by the new results of FedAvg + FT.  

I acknowledge the contribution on heterogenous data and the motivation of combining collaboration and personalization. I would encourage the authors to improve the writing, and happy to further engage before discussion ends.
Thanks for your quick response.

1. Regarding topology and partial participation, we understand that they should naturally be compatible with our current scheme. For instance, consider a specific client i. In rounds where client i is selected, we reweight all other selected clients connected via edges in that round. If no clients have a similar data distribution, our scheme assigns high self-trust and low trust towards others. Conversely, if there are clients with similar data distributions, our scheme assigns higher trust to these clients, allowing client i to make a better update in that round. A more sparse topology and low participation ratio will require more communication rounds to converge. 


 We agree FedAvg + Local FineTuning would be a valid baseline to compare to, but in our experimental setup, we only have a training set (where we train models) and validation set (where we report the test performance, it is more like a test set).  To our understanding, local fine-tuning requires some additional personalized data before evaluation, which we do not have. Therefore, we allocated a certain ratio (% in the parentheses) of the training data for fine-tuning and kept the rest for FedAvg training. The results are shown in the table below. With fine-tuning, the perplexity even went higher for some scenarios. We speculate that this may be due to the reduced amount of data available for FedAvg training and overfitting on personalized data. 


Method | AGNews Low | AGNews High | Multi-Wiki Low | Multi-Wiki High | Codes-Wiki Low | Codes-Wiki High |
| ----------- | ----------- | ----------- | ----------- | ----------- |  ----------- |----------- |
Local| 30.17 (0.17)| 28.67 (0.13) |40.00 (0.33) |40.24 (0.25) | 19.57 (0.23)|17.56(0.08)
FedAvg | 31.66 (0.20)| 32.08 (0.13) | 52.75 (0.57) |53.23 (0.51) | 17.53 (0.19)| 16.68 (0.06)
FedAvg + FT (10%) | 32.25 (0.20) | 33.74 (0.17)| 48.02 (0.25) | 48.07 (0.18)| 20.17 (0.33)| 18.43 (0.18)|
FedAvg + FT (25%) | 31.29 (0.15)| 32.11 (0.16) |43.74 (0.24)|43.68 (0.13)| 20.10 (0.25) |18.15 (0.11)
FedAvg + FT (50%) | 30.03 (0.23) | 29.84 (0.09) | 39.79 (0.21) | 40.14 (0.19) | 19.59 (0.26) | 17.88 (0.16)
S1 |31.43 (0.35) |31.93 (0.86)|45.59 (0.66)|49.34 (2.46)|17.57 (0.21) |16.84 (0.05)
S2 |29.75 (0.23)|28.29 (0.06)|36.93 (0.17)|37.20 (0.20) |17.61 (0.40)|16.22 (0.17)
S3 | - |-|**36.70 (0.23)** |**36.92 (0.16)** |**17.35 (0.18)**|**16.23 (0.12)**|

Please note that we did not claim to propose new decentralized personalization algorithms. In our opinion, the **main contribution** of this paper lies in two aspects: 1) we explore _realistic_ data heterogeneity among users in decentralized language modeling; and 2) we demonstrate that collaboration can be leveraged to improve personalization performance in language modeling. By personalization, we mean that each client has a personalized state. We hope the FedAvg + FT experiment illustrates that balancing personalization and collaboration is not straightforward in language modeling. Despite the drastic differences between categories, there are notable similarities in sentence structure and word choices, alongside domain-specific word choices, unlike typical classification tasks. Additionally, the preferences for collaboration or personalization vary across different datasets. For example, in the case of AG News, local training yields better results than FedAvg, whereas this is not true for Codes-Wiki. Our strategies can intelligently adapt to these different scenarios.

2. Yes, here are some references: 
- "Similarity measures could be useful to assess trust between users. For example, in opinion dynamics models, it has been proved that users tend to consider and trusting more other users with similar opinions to them” [1]
- “The trust degree between individuals derived from opinion similarity at time t.[2]
   
If we view clients as members of a social group, their model state or predictions on a shared dataset can thus be regarded as opinions.

3. We adopt the experimental setup as we focus on the same cross-silo scenario in the language domain. Please note that there are not many existing works that studied decentralized learning in language modeling. We think such an experimental setup demonstrates our purposes (see our clarification on contributions in point 1)

4. Yes that’s a good suggestion, we will adopt this term in future iterations.


[1] Ureña et al. A review on trust propagation and opinion dynamics in social networks and group decision making frameworks

[2] Zhang et al. Consensus reaching with trust evolution in social network group decision making
Thanks for the clarifications.
Thanks for the response. 

1. Could the authors clarify again about the main contribution of the paper? A few things I hope to understand more a) the fully connected topology seems to be a strong assumption; consider the empirical nature of this study, I am not convinced that extending to other topology and/or partial participation is easy for the reweighting strategies studied in this paper. b) related to a) there are many decentralized algorithms studied before, maybe I missed it, but I am not sure if they are discussed at all. For example [𝐷2: Decentralized Training over Decentralized Data] c) if the main focus is the reweighting strategies, it is also a problem that has been extensively studied in the literature, see section 3.2.4 in [A Field Guide to Federated Optimization] for papers before 2021, and more recent papers like [Revisiting Weighted Aggregation in Federated Learning with Neural Networks]. d) if the main focus in personalization, I am not convinced that a simple baseline of FedAvg + local finetuning is out of scope for comparison. Note that I did not ask for an extensive table like in [pFL-Bench: A Comprehensive Benchmark for
Personalized Federated Learning] which benchmarks 10+ methods, but IMHO, ignoring the entire literature seems a little disappointing. e) if this paper considers their contribution is to study a problem that has been largely overlooked before, please clarify the motivation related to question a) about topology and assumption. And carefully clarify their relationship with the literature. 

2 I can accept the "trust" term if the authors can provide a few concrete reference that uses this term in this manner. They can be from the social science literature. Otherwise I would strongly encourage a replacement like "similarity" or something. 

3 Following settings in existing papers is a great practice. But I also want to highlight that experiments should be designed to support the contributions of the paper. The focus of  [Sun et al. Improving LoRA in Privacy-preserving Federated Learning] is quite different from this paper, and hence I may need the authors to clarify why this setting is supporting their contributions. See 1 before. 

4 I thank the author and acknowledge the explanation of "theoretical" in experiments. I would probably use a term like "oracle" if I were writing the paper. 

My main question is 1 about understanding the contribution of this paper in the literature.