Personalized Collaborative Fine-Tuning for On-Device Large Language Models
Authors: Nicolas Wagner
Dongyang Fan
Martin Jaggi
Venue: COLM (2024)
Keywords: Collaborative fine-tuning, On-device LLMs, Personalized learning
TL;DR: We explore on-device collaborative fine-tuning of large language models under heterogeneous and scarce local datasets.

Abstract: We explore on-device collaborative fine-tuning of large language models under limited local data availability. We introduce three distinct dynamic collaborator selection schemes, allowing trust-weighted personalized update aggregation: model-similarity-based, prediction-similarity-based and validation-performance-based. To minimize communication overhead, we integrate Low-Rank Adaptation (LoRA) and only exchange LoRA model updates. Our protocols, driven by prediction and performance metrics, surpass both FedAvg and local fine-tuning methods, which is particularly evident in realistic distributed scenarios with more diverse local data distributions. The results underscore the effectiveness of our approach in addressing heterogeneity and scarcity of the local datasets.

Published as a conference paper at COLM 2024
Personalized Collaborative Fine-Tuning for On-Device Large Language Models
Nicolas Wagner, Dongyang Fan, Martin Jaggi
EPFL, Switzerland meepf120@gail.com com, {dongyang. fan, martin. jaggi @epfl.ch
Abstract
We explore on-device collaborative fine-tuning of large language models under limited local data availability. We introduce three distinct dynamic collaborator selection schemes, allowing trust-weighted personalized up- date aggregation: model-similarrty-basee cetraatioon-prffrmmnnceoo
citiiiio-siiiilitt---bareee and To minimize communication overhead, we integrate Low-Rank Adaptation (LoRA) and only exchange LoRA model updates. Our protocols, driven by prediction and performance metrics, surpass both FedAvg and local fine-tuning methods, which is particularly evident in realistic distributed scenarios with more diverse local data dis- tributions. The results underscore the effectiveness of our approach in addressing heterogeneity and scarcity of the local datasets.
1 Introduction
Recently, there has been an unprecedented surge in the popularity of Large Language Models (LLMs), driven by their versatile capability to solve wide range of tasks and serve as general-purpose models (Touvron et al., 2023; OpenAl et al., 2024; Jiang et al., 2024). Their downstream performances can be further enhanced by fine-tuning, which is typically conducted on fewer data and independently by different users, rather than in a centralized way. The scarcity of local data often renders local fine-tuning ineffective, necessitating collaboration. However, in many cases, end users usually have privacy concerns over their local data, such as patient records for hospitals (Li et al., 2023) and typing history for mobile phone users (Hard et al., 2018). Naturally, one might wonder how we could enable users to collaborate to still obtain better models the presence of small and prioacy-sensitioe local data.
The possibility of on-device fine-tuning of LLMs has been facilitated by parameter-efficient techniques such as prompt tuning (Lester et al., 2021), adapters (Houlsby et al., 2019) and Low-Rank Adaptation (LoRA, Hu et al. (2022)). LoRA has gained widespread popularity for fine-tuning large language models since its introduction, which approximates model updates through the multiplication of low-rank matrices. LoRA modules align with our collaboration initiative, as they allow for a significant reduction in communication overhead. A naive approach to leveraging local data from other users would be to average LoRA weights, which is the popular FedAvg algorithm proposed by McMahan et al. (2017). However, the resulting one global model solution might not fit all users' data distributions, especially when there is more heterogeneity present. As result, we aim to develop personalized collaboration strategies for each user, which should surpass the performance of local fine-tuning and naive averaging approaches.
In pursuit of this objective, we introduce several collaborator selection protocols tailored for optimal LoRA weight averaging across diverse user profiles. While similar concepts have been extensively investigated within collaborative learning communities (Zhang et al., 2021; Sui et al., 2022; Fan et al., 2024), these approaches are only designed for supervised classification tasks and thus are not directly applicable to Large Language Models (LLMs). Nevertheless, investigate how we adapt and apply analogous principles to determine optimal aggregation weights for LoRA matrices on a per-user basis.
Our code is available https:: //gg thub. http.////pppmmppppssaaaaaiieeeea col Laboratiiee

---

Published as a conference paper at COLM 2024
Following some computational social science literatures (Urena et al., 2019; Zhang et al., 2022), we term the aggregation weight matrix "trust", as if we view clients as members of social group, their exchanged information can thus be regarded as opinions. Similarity measures can be useful for assessing trust between users, as users are more likely to consider and trust others with similar opinions. Such similarity measures core to our proposed protocols, which we will discuss in Section 3.1.
Our contributions can be summarized in the following aspects:
We are the first to tackle realistic data heterogeneity among users in personalized collaborative language modeling, encompassing diverse topic distributions and varying language usage.
We demonstrate that collaboration can be leveraged to improve personalization performance in language modeling and show that predictions are more effective than model parameters for identifying collaborators in the language domain.
Our approach is well-suited for on-device fine-tuning, effectively mitigating chal- lenges related to data scarcity and resource limitations.
Related Work
We focus on a peer-to-peer decentralized learning setting, where the existence of a central server is not assumed. Instead, the end users conduct peer-to-peer communication using decentralized schemes such as gossip averaging to aggregate local information across agents. To enable a personalized weighted aggregation, several recent works have proposed data-dependent communication protocols based on task similarities and node qualities. Notably, Zhang al. (2021) derives a first-order approximation for optimal aggregation weights wij for collaboration, which happens to be proportional to how well client j's model generalizes on client i's data. et al. (2022) directly optimizes the mixing weights by minimizing the local validation loss per node. Sui et al. (2022) uses the E-step of the EM algorithm to gauge the significance of other agents for a specific agent i, achieved through assessing the accuracy of these agents' models on the local data of agent i. Fan et al. (2024) adopt a different approach by comparing prediction similarity on a publicly available dataset, which largely reduces communication overhead and allows for model heterogeneity. These approaches have been showing promising performances in supervised deep learning experiments. For language models, the data heterogeneity across users is not well studied, and the effectiveness of collaborator selection methods has not been explored for LLMs, where self-supervised training is performed.
Recently, groups of researchers have been investigating the intersection of Federated Learn- ing and Large Language Model training. Due to the substantial number of model parameters, these efforts primarily focus on post-training stages, often incorporating parameter-efficient techniques (Fan et al., 2023; Ye et al., 2024; Flower.ai, 2024; Flare, 2024). Federated pre- training was studied in (Sani et al., 2024). Zhang et al. (2023) employ FedAvg for collabora- tive instruction tuning, where they demonstrate superior performance in task generalization compared to local instruction tuning. Che et al. (2023) design an adaptive optimization method for collaborative prompt tuning, where scoring method is applied to measure the importance of each layer and only prompt parameters of more important layers are exchanged during the tuning process, ensuring communication efficiency. Cho et al. (2024) tackles the challenge brought up by different LoRA ranks on heterogeneous devices. By performing rank self-pruning locally and sparsity-weighted aggregation at the server, im- proved convergence speed and final performance can be achieved. In contrast to these methods, we aim at providing personalized models to each user, instead of a global solution.
3 Method
To start with, we first introduce how LoRA works. For a pre-trained model parameterized with & Rx the model updates during fine-tuning stage can be approximated by
2

---

Published as a conference paper at COLM 2024
[
3
0
[0 0 (
140
Upciate Strategy: W g(5
0 Σ MgAn
+
(00)
Figure Diagram of our protocol. 0; and 20, represent LoRA parameters and LoRA model updates respectively. s; denotes messages to send beside Δ0, which represents either 0, or fe, (Xs) depending on the protocol (see Table 1). g(-) is our proposed trust calculation approach as detailed in Section 3.2.
multiplication of two low-rank matrices A R B E RR
AΦ ~ AB,
where
< min(m,,
(1)
We work in a setting where all users have the same LoRA rank r, and aim to arrive at their personalized AΦ; During the fine-tuning phase, only LoRA parameters will be updated, and the original weights the pre-trained model will remain frozen. Therefore, it suffices to communicate LoRA models, instead of the big chunk of whole parameters.
3.1 Our Protocol
We stick to the standard approach of sharing and aggregating model updates. Each user, through their unique data distribution, contributes to the collective learning process by sharing local model updates. Central to our approach is the so-called trust matrix, which we define as the gossip aggregation matrix to guide each user's s aggregation of model updates. The calculation of trust matrix W will be detailed Section 3.2. We use W, to denote the ith row of W. A simplified diagram of our protocol is shown in Figure 1.
Our proposed protocol is described in Algorithm 1. To summarize, the protocol con- stituted with three phases: 1) each user E [N] computes their LoRA model update A0; with respect to their local data (and if required, predictions fe, (Xs) with respect the shared dataset Xs); 2) each user communicates [0, or [fe, (Xs), A0;] all other users, depending on the trust calculation strategy; 3) each user E [N] calculates their unique trust weights W; locally in parallel and updates their LoRA parameters using trust-weighted received model updates. It is noteworthy that different means of trust weight calculation re- quire different information to be shared across the users, thus inducing distinct computation complexities and communication overheads. We will discuss those aspects in Section 3.3.
3.2 Trust-based Collaborator Selection
In this section, we explore different ways of building trust gossip aggregation graph across users. Essentially, the weight of edge (i, should denote to what extent can user j's model help to facilitate user i's learning progress.
3.2.1 Model-simillritt-bassdd
We employ pairwise LoRA parameter similarity to compute trust scores among users. The underlying idea is that users whose model parameters exhibit closer alignment likely possess data distributions that are more similar, thereby rendering their contributions more pertinent and advantageous to one another. This methodology shares resemblance with (Li et al.,
3

---

Published as a conference paper at COLM 2024
Algorithm 1 Our proposed protocol
Require: Number of communication rounds T, pre-trained model θ, initialized LoRA
models , local dataset X,, shared dataset Xş (required for strategy 3.2.3).
for ,,,, T do in parallel for each user i E [N] do
1. Compute personal update A0!-1) on 0/--1) with respect to local data X
2. Broadcast each Ao. 10) to all others agents
3. Compute collaboration weight matrix W, based on trust-based strategy 3.2.1, 3.2.2 or 3.2.3 Compute trust-weighed personalized model update:
810
E
wi (111)
JE[
Update personal LoRA parameters, with learning rate
0 0 (111) no! (1)
end for
2022), where the mixing weight is derived from the inner product between representations of different users' models via an encoder model. However, our approach involves directly measuring the similarity between LoRA models without needing an encoding module.
appears that in addition LoRA model updates, LoRA models also need to be shared to compute the trust matrix. However, this additional communication can be efficiently circumvented, as the updates to LoRA models are already exchanged, and the models themselves can be readily derived by aggregating these updates in each iteration. The collaboration trust matrix W is computed as follows:
(0 1)
W SoftMax (W*, dim = 1)
(2)
|0
3.2.2 Validation-perfommane-bbel
Drawing inspiration from Zhang et al. (2021), we evaluate the pairwise trust wij based on how well user j's model performs on the validation sets user i. This method requires extra validation sets within each user, on which the performance assessment directly reflects the relevance and potential benefit of one user's 's model to another. The aim is to guide the aggregation process by favoring models that demonstrate compatible performance levels, thereby enhancing the overall effectiveness of collaborative learning. The calculation is denoted as:
क 1 X,a)),,
W = SoftMaxx W dim =1)
(3)
where ( denotes a forward pass of user j's model on user i's local validation set, and L is the cross entropy loss for next token prediction.
3.2.3 Prediction-smiilartt--asss
In the
approach, following the trust weighing strategy from Fan et al. (2024), we derive aggregation weights by comparing the model's predictions shared dataset, in this case, logits across the vocabulary. public dataset Xş will be added to the input of Alg. 1, and additionally communicating those predictions between users is
4

---

Published as a conference paper at COLM 2024
necessary, increasing communication cost. This method assesses how closely the predictions of different models align, prioritizing contributions from users whose models exhibit similar predictive behaviors. The trust is calculated based on I distance between logits:
t = 1 - f
W = SoftMax(- W dim =
(4)
3.3 Remarks
Table provides an overview of the communication and computation costs associated with the three proposed strategies.
On communication costs. In terms of communication, both strategies and 2 need to share LoRA models (0:) on top of LoRA model updates (A0;). Strategy requires sharing logits (fe, (Xs)) across the public dataset, rather than LoRA models. Although the logits encompass the entire vocabulary, ficially transforming it into a sparse matrix. The computation of trust through prediction similarity relies on the distance, and most entries of logits tend be very close zero, making it feasible to only keep the top-k logits values. This approach greatly cuts down on communication costs when sharing logits while still maintaining desired performance levels. For detailed comparison of the measured communication costs, please refer to Appendix A.1. Nonetheless, as the communication of 40,s remains unavoidable and consti- tutes the primary communication cost, there is not much variation in communication costs across all three strategies.
possible to heavily reduce its memory cost by arti-
On extra computation costs. Regarding additional computation costs to obtain W, Strat- egy involves only calculating cosine similarities between LoRA models in addition to existing processes, the cost which is in the order of the number of LoRA parameters. However, Strategies 2 and 3 necessitate multiple forward passes to generate predictions either on local validation sets or the publicly shared dataset. A forward pass is significantly more resource-intensive compared to cosine similarity calculations due to the model's ss size. In Strategy 2, evaluating all models' performance on each local validation set requires N2 model inferences. In Strategy 3, each user needs extra forward passes to obtain predictions on Xs, resulting in N times the model inference. Since Xş and X are similar in size, the number forward passes per inference doesn't vary much. Thus, Strategy 2 requires roughly N times more extra computation cost than Strategy 3.
Strategy
Communicated Elements
Extra Computation Costs
1: Model-sim
(0, A0; N N !
O O(Ln'd Ld N2 O(Lnd + Ld2n) N
2: Validation
(0, 40, i=1
***
3: Predictions
{fe, (Xs), 00
Table 1: A comparison of communication and computation complexity across the strategies. denotes the number of layers, d denotes the embedding size, n denotes context length and k denotes LoRA rank. N denotes the number of users in the system. It is clear Strategy adds the most extra computational cost, followed by Strategy 3. Strategy 1 requires the least additional amount of computation.
4 Experiments
4.1 Setup
4.1.1
Experimental details
Our experimental setup is structured as follows: Each user is configured with a uniform model architecture, to ensure model updates can be shared and aggregated. We chose

---

Published as a conference paper at COLM 2024
relatively small LLMs constrained by the limited computing resources in academia. For each user, we equip them with GPT2! base model of 124 million parameters. It has 12 layers, 12 attention heads, embedding size 768 and vocabulary size 50257.
LoRA modules were applied top of both Causal Self-Attention (SA) and MLP layers, con- stituting 0.47% of full parameters. The reason why we chose the two blocks is a trade-off be- tween the model performance and commuuicaa- tion overhead, as shown in Figure 2. Although incorporating LoRA modules onto Self Atten- tion layers results in the most significant reduc- tion test perplexity per trainable parameter, integrating LoRA modules into MLP layers can contribute further decreasing test perplexity while maintaining reasonable number of train- able parameters.
27.5
10
25.0
Perplexity
Ratio
Ratio
22.5
10-5
Perplexity
20.0
10-6
17.5
Full
SA
MLP SA+
Figure 2: Ablation study of whether to add LoRA modules compared to full fine-tuning. Left bars correspond to Ra- tio, which denotes the decrease in test perplexity compared to the pre-trained model per trainable parameter (higher is better); right bars correspond to perplex- ity, which denotes test performance after fine-tuning (lower is better).
The experiments were conducted, for each client, with a learning rate of 0.002, batch size of 50 with 4 accumulation steps, a context length of 512, and a total of 500 iterations. Every user performs local training for 100 iterations before communication, so that the local models can start to diverge due to the variation of local data distributions. A Afterward, communication and aggregation are done every 25 iterations. As we target limited data regime, the number of iterations ensures that over 50 epochs of training are conducted in most datasets. Throughout all experiments, we use the same LoRA configurations: rank of 4, value of 32.0, and dropout rate of 0.1. The selection of hyperparameters is determined through a sweep of various values while considering the limitations posed by our computing resources, as presented in Appendix A.2.
4.1.2 Datasets
In vision tasks, realistic data heterogeneity arises from diverse domain distributions across clients and variations in image quality resulting from the use of different devices for data acquisition (Ogier du Terrail et al., 2022). However, within the domain of written language, realistic data heterogeneity frequently arises from variations between sources. Owners of such sources can differ in writing styles, vocabulary choices, grammar structures, and topic distribution. Our experiments are designed to examine diverse topic distributions and language usages exhibited by different users.
We focus on on-device fine-tuning scenarios, where the amount of available local training data may be limited. The diverse local data distribution originates from user-specific characteristics. For instance, users often possess unique topic preferences concerning news reading, and mobile phone users often type in different languages, prompting smart keyboards to enhance next-word prediction across different language blends. Following this, we investigate two levels of data heterogeneity: 1) Low heterogeneity, where each user is assigned two data categories from the entire set, and (3/4, 15/4,1// 1/4) mixture of data of these categories is allocated to each user. 2) High heterogeneity, where each user is exclusively assigned to one data category.
The descriptions of the three dataset partitionings utilized in our experiments are as follows:
1. AG News: News articles in four categories: "World", "Sports", "Business" and "Sci/'Tech". (Zhang et al., 2015)
2. Multilingual Wikipedia: Wikipedia texts in three languages (categories): French, Italian, and German. (Wikimedia, 2022)

---

Published as a conference paper at COLM 2024
Datasets
# users
# Categories
Training tokens
Test tokens
AG News 4/8
4
1'500'000/~ 750'000
75'000/~ 50'000 160'000
Multi-Wiki
9
3
840'000
Codes-Wiki
8
2
840'000
160'000
Table 2: Dataset configurations used in the experiments. Reference data Xs is slightly larger in size than test tokens and 100'000 tokens are sampled for logits computation. There is insufficient data from AG News to establish a meaningful public reference dataset, so we resort to ChatGPT generate synthetic one.
Datasets
Heterogeneity
with the most task overlap
with the least task overlap
Low
0.39-0.41 0.42-0.44
0.31-0.32 0.28-0.33
AG News
High
Low
0.45-0.46
0.19-0.24
Multi-Wiki
High
0.45-0.47
0.19-0.23
Low
0.40-0.41
0.39-0.42
Codes-Wiki
High
0.41-0.42
0.12-0.13
Table 3: Jaccard Index values for various datasets, indicating the range of token overlap among clients in different configurations.
3. Codes- Wikipeda (Eng): The first category is Java code from GitHub (HuggingFace) and the second category is English Wikipedia text (Wikimedia, 2022).
The corresponding distributed datasets are presented in Table 2. Note that AG News and Codes-Wikipees simulate different topic distributions across users, while Multilingual Wikipedia simulates different language usage across users. It is worth noting that even though the sopics/languaa their vocabularies. The portion of overlapped tokens is indeed not small, as measured by the [D,nD,] Jaccard Index in Table 3. Jaccard Index for multi-sets, is calculated as [(;;, Dj) [D, + PD] E
are different, they can nevertheless share the same tokens in
[0,0.5). high Jaccard Index indicates that clients' data share a substantial number of the same tokens. Although this measure doesn't account for the context information, it provides insight into the likelihood finding collaborators. In all our experiments, the shared public dataset Xs is sampled equally from each local distribution to ensure that the logit predictions are meaningful. In practice, such Xş can be chosen as any publicly available dataset from the same domain, private data with data sanitization procedures enforced on top to secure user privacy.
4.1.3 Baselines
Our chosen baseline methods are: 1) Local Fine- Tuning, where there is communication between users, 2) Fed (McMahan et al., 2017), where model updates are aggregated to obtain global model for each user, and 3) FedAvg + Local Fine-Tuning (Jiang et al., 2023), where we spare 10% of training data for local fine-tuning after FedAvg training. We offer an extended table where we vary the amount of training data used for fine-tuning for completeness in Table 9 in Appendix. We further incorporate a strong baseline oracle. The determination of oracle collaboration weights relies on the similarity
underlying data source distributions, which is typically unknown in practice. We give example of how the oracle weights are determined: if user has 1/4 German texts and 3/4 French texts, and user 2 has 1/4 German texts and 3/4 Italian texts, the oracle weights would be determined by the dot product of [1/4, 3/ 344,0 0) and [1/4, 4, 0, 3/44 After obtaining the pairwise dot product matrix, row normalization is performed.
7

---

Published as a conference paper at COLM 2024
8
Method
Multi-Wiki
Codes-Wo Wiki High
L1 Norm Difference
Strategy 2
Low
High
Low
6
Strategy 3
Local
1
1
1
FedAvg
1.6
4.2
2.0
1.7
4
Strategy
5.9
12.0
2.0
2.1
Strategy
1.1
2.6
1.5
NA
2
Strategy 2.9
NA
1.2
3.8
0
5
10
15
Table 4: Training time (X times the needed training iterations as in the fully connected case) required for ring topology to achieve the same perplex- ity level as a fully connected topology. NA indi- cates we did not reach the same perplexity after ten times the training iterations.
Time Step
Figure L1 norm of differences be- tween W at two consecutive steps for Strategy 2 and Strategy
4.2 Results
Our main findings are summarized in Table 5. Compared to baseline approaches- local training, Fvd and FedAvg + our test performance and prediction-based aggregation methods consistently outperform them across all datasets and scenarios. We emphasize that in language modeling, achieving a balance between personalization and collaboration is not straightforward. Despite sigmificant differences between categories, there are notable similarities in sentence structure and word choices, alongside domain-specific vocabulary. Thus, the preferences for collaboration or personalization vary across different datasets. For example, in the case of AG News, local training yields better results than Fed Avg, whereas this is not true for Codes-Wiki. The collaboration-then seetiionthe---ersseee then-personalizationn approach (FedAvg + Local Fine- Tuning) fails to achieve this balance. In contrast, our methods successfully accomplish it.
Among our protocols, the predictions-based method exhibits the most superior test perfor- mance. Notably, the somiia-timllarity-- -based aggregation method yields performance akin to FedAvg, where the trust weight is uniformly set to 1/N across all clients. Our methods can sometimes outperform oracle aggregation, suggesting that a dynamic collaborator selection protocol might be favored in different fine-tuning stages.
Ablation studies on the effect of LoRA ranks and local training set size can be found in Table 8 and Table 7 in Appendix A.3. Our prediction-based strategy consistently delivers superior performance.
A note on communication topology: While the paper demonstrates a fully connected communication topology, our protocols can support sparser communication patterns, as long as the graph represented by W is strongly connected (i.e., every vertex reachable from every other vertex), though convergence will be slower. We provide experimental results on the training time required for a ring topology to achieve the same perplexity level as a fully connected topology Table 4. In high heterogeneity scenarios where each client is allocated specific category, ring topology results in cases where adjacent clients do not share any categories, leading an oracle trust weight of and hence the NA entries. Despite this, our algorithm can still learn, converging to perplexity approximately one point higher than reported for the fully connected case.
4.3 Trust matrix
We compare our learned trust matrices with the oracle trust matrix in Figures 4 and 5. is evident that Strategies 2 and 3 uncover similar collaboration patterns as suggested by the oracle matrix, whereas Strategy falls short in this regard. Specifically, Strategy assigns nearly identical trust weights to all other users, indicating nearly identical LoRA weights learned at convergence. This strange behavior is further investigated in Appendix A.4: even without communication, the LoRA weights across users are almost equally different,
8

---

Published as a conference paper at COLM 2024
Datasets
Heterogeneity
Method
AG News
Multi-Wiki
Codes-Wiki
Local
30.17(0.17)
40.00(0.33)
19.57(0
FedAvg
31.66(0.20)
52.75(0.57)
17.53
FedAvg +FT
32.25(0.20)
48.02(0.25)
20.17
Low
Strategy 1
31.43(0.35)
45.59(0.66)
17.57(0.21)
Strategy 2
29.75(0.23)
36.93(0.17)
17.61 (0.40)
Strategy 3
29.81(
36.70(0.23)
17.35(0.18)
Oracle
29.56(0.21)
39.55(0.19)
17.42(0.19)
Local
28.67
40.24(0.25)
17.56(0.08)
Fed
32.08(0.13)
53.23(0.51)
16.68(0.06)
High
FedAvg +FT
33.74(0.17)
48.07(0.18)
18.43(0.18)
Strategy 1
31.93(0.86)
49.34(2.4)
16.84(0.05)
Strategy 2
28.29(0.06)
37.20(0.20)
16.22(0.17)
Strategy 3
28.72(0.27)*
36.92(0.16)
16.23(0.12)
Oracle
28.08(0.11) 35.96(0.24)
16.20(0.05)
Table 5: Test perplexities (standard deviation) of our proposed strategies and baseline methods (lower is better). Strategy 1: modellsimiilariy-bbased Strategy 2: validation- performance-based, Strategy 3: prediction-simillrity-bbee dataset a synthetic dataset we constructed using ChatGPT.
denotes that the used public
0.3
0.3
0.2
0.2
676547
6
0.1
0.1
8
0.0
0.0
0.6
0.6
0.4
0.4
1
2
4 6
1
3 4 6
8
(a) Oracle
(b) Strategy
Figure 4: Oracle trust matrix versus learned trust matrix using Strategy 1 when users allocated with Multilingual Wikipedia datasets. The diagonal entries are masked out and the trust is measured when the training ends.
suggesting that LoRA models not informative in collaborator selection when cosine similarity is used as the distance metric.
The success of Strategies and 3 underscores the advantage of utilizing predictions to identify collaborators. It is interesting how the distinctions between different users' models become more pronounced after forward pass. Such a phenomenon was not previously observed in the vision domain. We further noticed that trust matrices in Strategy 3 are more stable across communication rounds compared to Strategy 2, with fewer abrupt changes in values. As shown in Figure 3, the norm of the differences between Ws at consecutive steps fluctuates significantly for Strategy 2. It is speculated that the more stable trust matrix in Strategy contributes to stabilizing the learning process, leading to better results.

---

Published as a conference paper at COLM 2024
D
0.3
0.3
2
0.2
4
7 6
0.1
0.1
-0.0
-0.0
0.6
0.6
0.4
0.4
4
4
6
(a) Strategy
(b) Strategy
Figure 5: Learned trust matrix using Strategies 2 and 3 when users are allocated with Multilingual Wikipedia datasets. The diagonal entries are masked out and the trust is measured when the training ends.
Conclusions
Drawing from insights in conventional peer-to-peer collaborative learning approaches, we propose 3 collaboration protocols tailored for jointly on-device LLM fine-tuning. To our best knowledge, this the first work demonstrating that collaboration can be leveraged to improve personalization performance in language modeling. Our methods can efficiently address challenges encountered in on-device personalized LLMs, such as limited availability of local data and data heterogeneity stemming from user characteristics.
We show that uniform collaboration as in federated averaging is not sufficient, but that dy- namic collaborator selection strategies enable better and personalized models. Remarkably, for collaborator selection, predictions are more informative than model weights. Our Strategy the penfiction-immllaitty--eee
protocol, demonstrates promising practical applications for on-device deployment. It achieves superior performance while managing to maintain reasonable communication and computation costs.
Discussions and Future Work
In this study, we examined resource-constrained scenarios of collaborative LLM fine-tuning. Our collaborator selection scheme successfully addresses realistic data heterogeneity across users. However, heterogeneities can arise in different aspects, such as model and resource diversities. When an important challenge to address. Cho et al. (2024) have developed collaboration protocols tailored for circumstances wherein users exhibit varying levels of support for LoRA ranks. When users are equipped with distinct numbers of LoRA modules corresponding to their capacities, can still harness the advantages of collaborative training?
possess different capacities, enabling efficient collaboration remains
Furthermore, within the scope of this research, particular emphasis is placed on unsuper- vised fine-tuning. The prospect of supervised fine-tuning, i.e. instruction tuning, is worth further investigation. An intriguing question arises: Can collaboration among users enhance their ability to generalize and improve performance on downstream tasks?
Our collaborator selection protocols are centered around trust. While the efficacy of trust in identifying collaborators has been thoroughly explored, there is potential for further investi- gation into its role in identifying ad adversarial users, which is one of the major challenges in federated LIMss (Chen al., 2023). We finally note that our scheme can be extended to the case of partial participation of users and asynchronous updates, the trust matrix could be updated less frequently than in every iteration, and decentralized averaging algorithms can tolerate asynchronous updates (Even et al., 2024).
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank Soumajji Majumder and Bettina Messmer for their helpful review and all the anonymous reviewers for their constructive feedback. We acknowledge funding from Huawei Cloud Intelligent Cloud Technologies Initiative, Google Research Collaborations, and from the Swiss National Science Foundation (SNSF) grant number 200020 200342.
References
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7871-7888, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /tt////112222mmm/// coinnlp-minn.eee m8in URL https: https:///cccaaaoooaaaaao ge2g/ 202..emnl...mm...mmmmm emnlp-main. 488.
Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Yin, and Xiaolin Zheng, Federated large language model: A position paper, 2023.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models. ar preprint arXiv:2401.06432 2024.
Mathieu Even, Anastasia Koloskova, and Laurent Massoulie. Asynchronous SGD on graphs: a unified framework for asynchronous decentralized and federated optimization. In ATStats Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238, pp. 64-72. PMLR, 02-04 May 2024. URL https: https::/procceedings mlr. press/v238/er /2ee////2//eeeeaaaaaea html.
Dongyang Fan, Celestine Dennder-Düinner, and Martin Jaggi. Collaborative learning via prediction consensus. Advances in Neural Information Processing Systems, 36, 2024.
Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang, FATE-LLM: A industrial grade federated learning framework for large language models, 2023.
Nvidía Flare. Scalable Federated Learning with NVIDIA FLARE for En- hanced LLM Performance, 2024. URL iivie nvidia. com/blog/ -oddia- 11m -er--erereeanccee//////
Flower.i. FlowerTune LLM example, 2024. URL wtw.eaapppee/mmmlloleeerrrrereee
Andrew Hard, Chloé M Kiddon, Daniel Ramage, Francoise Beaufays, Hubert Eichner, Kanishka Rao, Rajiv Mathews, and Sean Augenstein. Federated learning for mobile keyboard prediction, 2018. URL
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019,
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https: /ttp://////eeeeee...nneteeee net/ forum?iid=neeWeeFFyr
HuggingFace. URL http:///wggggggeeooo////ttttt/ts
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud,
11

---

Published as a conference paper at COLM 2024
Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning, 2023. URL https://arxiv. prggabs/1909. 12488.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.
Rumeng Li, Xun Wang, and Hong Yu. Two directions for clinical data generation with large language models: Data-to-label and label-to-data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7129-7143, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/ v / findingiinnggs-emmllp 474. URL Findings-emmlp.477 474.
hwwpr/hooooooooogg///ee 202/.
Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to collaborate in decentralized learning of personalized models. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9756-9765, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, al. Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. Advances in Neural Information Processing Systems, 35:5315-5334, 2022.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapire Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongiik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan,
12

---

Published as a conference paper at COLM 2024
Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Aili
Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Lorenzo Sani, Alex lacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William F. Shen, Preslav Aleksandrov, Xinchi Qiu, and Nicholas D. Lane. The future of large language model pre-training is federated, 2024. URL 10853.
orgg/bs/24855
Yi Sui, Junfeng Wen, Yenson Lau, Brendan Leigh Ross, and Jesse C. Cresswell. Find your friends: Personalized federated learning with the right collaborators, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Raquel Urena, Gang Kou, Yucheng Dong, Francisco Chiclana, and Enrique Herrera-Viedma A review on trust propagation and opinion dynamics in social networks and group decision making frameworks. Information Sciences, 478:461-475, 2019.
Wikimedia. Wikimedia downloads, 2022. URL httt::: wi wikimedia.org
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning, 2024. URL https: //axxiv wrg. /rgg/bb//402.... 06954.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning. Xiv preprint ar Xie:2305.00644 2023.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representation, 2021.
13

---

Published as a conference paper at COLM 2024
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Yangjingjng Zhang, Xia Chen, Lei Gao, Yucheng Dong, and Witold Pedrycze. Consensus reaching with trust evolution in social network group decision making. Expert Systems with Applications, 188:116022, 2022.
14

---

Published as a conference paper at COLM 2024
A
Appendix
A.1
Communication cost experiments
Tables a more detailed analysis of the communication costs within each communication round. In this specific case, communicating logits is the most costly. However, the cost can be largely reduced (to 0.014% the original size) via sparse encoding techniques.
Note that different communicated units scale with different quantities. When the model size is larger, for example in billions, models and model updates can be larger than logits. In this case, logits can even emerge preferable communication unit over weights or gradients without compression.
Communicated Elements
Communication Costs
Measured Costs (MB)
LoRA weights (0;)
(
2.359
LoRA model updates (A0;)
(Lakk
2.359
Logits (fo, (Xs))
(VV)
40.206
Sparse Top-K Logits (fe, (Xs)coo)
O(K)
0.560
Table 6: A comparison of communication complexity across the different shared units. denotes the number of layers, d denotes the embedding size, k is the LoRA rank, V denotes the vocabulary size and K denotes the number of logits kept for communication. COO denotes the sparse encoding used, Coordinate list.
A.2 flyper-parameter details
The choice of our hyper-parameters is influenced by resource constraints, primarily due to memory limitations (training conducted a single NVIDIA A100 GPU with 40GB memory) and time constraints, as we needed to fine-tune up to 9 large language models (LLMs). Through empirical observation, steps and context length of 512 performed well within memory constraints. Additionally, while increasing the LoRA rank slightly improved test perplexity, the associated increase in training time rendered it uninteresting for our experiments. Therefore, we opted for LoRA rank 4. Three hyper-parameters remain to be tuned: learning rate, LoRA alpha, and LoRA dropout. In Figure 6, the best perplexity was achieved with a learning rate of 0.002, LoRA dropout of 0.1, and LoRA alpha of 32.
found that batch size of 50 with 4 accumulation
A.3 Additional Ablations
A.3.1 Impact of local dataset size
We increase the number of training tokens by 100 times, resulting in 84 million training tokens per client in the multilingual setup. The ranking of all methods remained consistent with what we report in Table
Method
Local
FedAvg
Strategy
Strategy
Strategy
Oracle
val ppl
37.80(0.155) 56.69(0.35)
48.47(2.55)
36.17(0.30)
35.63(0.15)
35.59 (0.16)
Table 7: Comparison of methods on Multi-Wiki dataset
A.3.2 Impact of LoRA ranks
In the main body, we restrict the LoRA rank to to maintain a low computational budget. We conduct an ablation study the LoRA ranks and present the results in Table 8. The rankings of the methods are consistent with our previous findings, with Strategy 3 being the best method. Additionally, as expected, higher LoRA rank results in lower test perplexity.
15

---

Published as a conference paper at COLM 2024
200
Dropout=0, LoRA Alpha=16
Dropout=0, LoRA Alpha=32
Dropout=0, LoRA Alpha=64
160
Dropout=0.1, LoRA Alpha=16
Perplexity
140
Dropout=0.1, LoRA Alpha=32
120
Dropout=0.1, LoRA Alpha=64
100
Dropout=0.2, LoRA Alpha=16
Dropout=0.2, LoRA Alpha=32
80
Dropout=0.2, LoRA Alpha 64
60
10
0.000
0.001
0.002
0.003
0.004
0.005
Learning Rate
(a) With respect to learning rate
Alpha=16
43
LoRA Alpha=32
LoRa Alpha=64
Perplexity
41
39
0.000
0.025
0.050
0.075
0.125 Dropout
0.100
0.150
0.175 0.200
(b) With respect to LoRA dropout
Dropout
Dropout=0.1
Dropout=0.2
Perplexity
41
40
39
20
30
40
50
60
LoRA Alpha
(c) With respect LoRA alpha
Figure 6: Hyperparameter tuning learning rate, dropout, and LoRA alpha. LoRA dropout and LoRA alpha have a very small impact on the perplexity, the most important hyperparameter the learning rate.
A.3.3 FedAvg with Local Fine-Tuning
To understand better the relationship between the performance of FedAvg + Local Fine- Tuning and the relative size of the local fine-tuning dataset, spare different percentages
16

---

Published as a conference paper at COLM 2024
LoRA Ranks
8
16
32
Local
40.24(0.25))
33.15(0.15)
29.14(0.11)
27.41(0.11)
Fed
53.23(0.511)
48.84(0.34)
42.68(0.06)
40.12(0..5)
Strategy
49.34(2.46)
48.37(1.26)
40.50(0.11)
34.65(5.73)
Strategy 2
37.20(0.20)
30.41(0.13)
26.36(0.12)
24.37(0..0)
Strategy 3
36.92(0.12)
30.25(0.19)
26.30(0.11)
24.39(0.02)
Oracle
35.96(0.24)
30.06(0.16)
26.16(0.07)
24.43(0.10)
Table 8: Comparison of different methods with various LoRA ranks
of local data for fine-tuning purposes, and the results are presented in Table 9. With more local fine-tuning data, the performance gets closer to Local training without collaboration.
Datasets
Heterogeneity
Method
AG News
Multi-Wiki
Codes-Wikk
Local
30.17(0.17)
40.00(0.33)
19.57(0.23)
FedAvg
31.66(0.20)
52.75(0.57)
17.53(0.19)
FedAvg +FT (10%)
32.25(0.20)
48.02(0.25)
20.17(0.33)
Low
FedAvg +FT (25%)
31 31.29(0.15)
43.74(0.24)
20.10(0.25)
FedAvg +FT (50%)
30.03(0.23)
39.79(0.21)
19.59(0.26)
Strategy
31.43(0.35)
45.59(0.66)
17.57(0.21)
Strategy 2
29.75(0.23)
36.93(0.17)
17.61 (0.40)
Strategy 3
29.81
36.70(0.23)
17.35(0.18)
Oracle
29.56(0.21)
39.55(0.19)
17.42(0.19)
Local
28.67(0.13)
40.24(0.25)
17.56(0.08)
FedAvg
32.08(0.13)
53.23(0.51)
16.68(0.06)
High
FedAvg +FT (10%)
33.74(0.17)
48.07(0.18)
18.43
FedAvg +FT (25%)
32.11
43.68(0.13)
18.15(0.11)
FedAvg +FT (50%)
29.84(0.09)
40.14(0.19)
17.88(0.16)
Strategy
31.93(0.86)
49.34(2.46)
16.84(0.05)
Strategy
28.29(0.06)
37.20(0.20)
16.22(0.17)
Strategy 3
28.72(0.27)*
36.92(0.16)
16.23(0.12)
Oracle
28.08(0.11)
35.96(0.24)
16.20(0.05)
Table 9: (Extended) Test perplexities (standard deviation) of our proposed strategies and baseline methods (lower is better). Strategy 1: moodel-simillrity-basee Strategy validation- performance-based Strategy 3: prediction-similrrity-bbb dataset a synthetic dataset we constructed using ChatGPT.
denotes that the used public
A.4 Further analysis of modelssimilariy--bsed trust
We offer comprehensive analysis elucidate why Strategy underperforms the other two strategies. The convergent model-simillritt-basee trust matrix resembling uniform matrix noteworthy. This occurrence is peculiar, as we would anticipate heterogeneity due to the varying distributions of users' data.
A.4.1 Strategy trust matrix converges to uniform even without communication
We let each user perform local fine-tuning for 500 iterations (~ 50 epochs). Figure illustrates learned trust matrices using Strategy in different global rounds. Note that here,
17

---

Published as a conference paper at COLM 2024
87654321
0.3
876543210
0.3
876543210
0.3
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
rd:3
0
10:3
0 012345678
012345678
012345678
(a) Round 1
(b) Round 2
(c) Last Round
Figure Learned trust matrix using Strategy at different global rounds, without commu- nication, when users are allocated with Multilingual Wikipedia datasets.
we use one global round to represent every 25 iterations, without any actual communication between users. Figure 8 shows the learned trust matrices at convergence using the three different strategies.
Even without communication, the learned trust matrix using Strategy becomes uniform.
This indicates that the users learned almost identical LoRA weights.
876543210
0.3
076543210
0.3
01654377
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.7
0.7
0.7
0.4
0.4
0.4
0 012345678
012345678
012345678
(a) Strategy
(b) Strategy
(c) Strategy 3
Figure 8: Learned trust matrix using Strategies 2, and 3 when users are allocated with Multilingual Wikipedia datasets. No communication and aggregation are performed.
A.4.2 Strategy learns the collaboration pattern, but a tiny scale
We dive into this bizarre observation. In Figure 9, we plot out the exact same trust matrices as in Figure 7, but present with much smaller scale. This visualization indicates that Strategy indeed can identify users who are more helpful to collaborate with, but the distinction between more and less helpful collaborators is not pronounced.
0.140
0.113
0.111
076543211
0.132
076543211
0.112
0.111
0.123
0.112
16549
0.111
0.115
0.111
0.111
67
0.107
0.110
0.111
dg
012345678
012345678
012345678
(a) Round 1
(b) Round 2
(c) Last Round
Figure Learned trust matrix using Strategy 1 at different global rounds, without commu- nication, when users are allocated with Multilingual Wikipedia datasets. Displayed with smaller scale.
18


Reviews:
[{"review_id": "okAVeYQ7qd", "reviewer": "Reviewer_pToR", "summary": "This paper discusses weighted aggregation methods for the LoRA module in decentralized learning. When freezing base networks, the LoRA modules are weighted by “trust” calculation before aggregation. The “trust” weights are computed by weights similarity, validation set, and prediction similarity.\n\n\n\n========== after rebuttal and discussion ===============\nI thank the authors for the response and raised the score to 6.\n\nWhy not lower:\nI appreciate the clarification and extra work during rebuttal, and acknowledge the contribution of discussing collaboration and personalization of LoRA training of LLM in decentralized training. This is timely and relevant to COLM community. \n\nWhy not higher:\nAs also mentioned by other reviewers, the strategies discussed are not entirely new, and somewhat ad-hoc to me. The motivation of \"realistic\" heterogeneity makes sense, but I am still a little skeptical about the setting and baseline choices in the draft. \n\nFinally, I would strongly encourage the authors to take the discussion into consideration, and more carefully adjust the technical terms and relationship with the literature.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and feedback.\n\nRegarding the motivation, we are not targeting cross-device federated learning; our goal is to achieve a **personalized state** for each client, resembling more of a cross-silo setup, which requires stateful clients. This paper focuses on understanding personalization and collaboration within the context of privacy-preserving language models. Our experimental setup is similar to [1]. Although we considered a fully connected setup here, partial participation per communication round is orthogonal to our scheme and can be integrated. However, investigating the effects of partial participation in detail is beyond the scope of this paper.\n\nRegarding the term \"trust,\" you raise a valid point. In this project, our use of the term is motivated by social science literature: clients can be seen as members of social groups who make decisions (personalized models) through interaction with other members. This interaction, performed via collaborator selection, depends on the trust between group members, with similarity serving merely as a metric. We will clarify this in future versions.\n\nWe want to clarify that we are not providing state-of-the-art personalized federated learning algorithms. While many methods are applicable, this is the first work showing that collaboration improves stateful personalization in language modeling. Moreover, we have baseline methods including: 1) FedAvg and local training, demonstrating the need for _both_ personalization and collaboration for better validation perplexity; 2) a theoretical baseline using a collaboration graph based on underlying data mixture similarities.\n\nThe theoretical matrix assumes _known_ data mixtures for each client. For instance, if client 1 has 1/4 German and 3/4 French texts, and client 2 has 1/4 German and 3/4 Italian texts, the theoretical weights are the dot product of [1/4, 3/4, 0] and [1/4, 0, 3/4], followed by row normalization. While not necessarily the optimal trust matrix, it uses underlying mixture information (_unrealistic_ in practice), hence \"theoretical\". It is worth noting that our strategy can even outperform this baseline, emphasizing the importance of a dynamic collaboration graph.\n\nHope those clarifications clear your doubts. We welcome further discussions and would appreciate a score adjustment if we manage to address your raised concerns. Thank you.\n\n[1] Sun et al. Improving LoRA in Privacy-preserving Federated Learning. ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is relevant to the COLM conference.", "reasons_to_reject": "I may have missed something, but in my humble opinion, this paper does not meet the bar of top conference. \n\n1) The motivation is not quite clear to me. The method seems to aim for cross-device federated learning, but it seems to also assume full participation of clients every round instead of a small subset of clients. Moreover, it seems to assume a fully connected network topology among all clients, i.e., every communication is a full reduce step. It is challenging for me to understand this setting compared to the common setting, e.g., [Advances and Open Problems in Federated Learning](https://arxiv.org/abs/1912.04977) Section 1. \n\n2) I somewhat feel strongly and disagree with users' usage of \"trust\". The weights for aggregation seems to merely depend on some kind of similarity. \"Trust\" is often a terminology for privacy and security, and it is hard for me to understand what the connection with \"trust\" is. On the other hand, the authors do not seem to properly discuss the privacy implication of the settings and the algorithms. \n\n3) The proposed weighted aggregation methods are somewhat ad-hoc and lacking strong supports. The proposed method did not compare with any baseline method. There are many personalization methods in FL, and they can be applied to the LoRA training setting, e.g., [Motley: Benchmarking Heterogeneity and Personalization in Federated Learning](https://arxiv.org/abs/2206.09262) and methods therein.", "questions_to_authors": "Could the authors explain how the \"theoretical\" matrix is computed in experiments?", "extra_scores": {}}, {"review_id": "vQZlB6nXEX", "reviewer": "Reviewer_E7HE", "summary": "This paper introduces a method for on-device collaborative fine-tuning of large language models using Low-Rank Adaptation (LoRA). The approach is distinct for its minimal communication overhead and integration of three innovative trust-weighted gradient aggregation schemes: weight similarity-based, prediction similarity-based, and validation performance-based. Through their experiments, the author shows that the aggregation trust matrix guided by prediction information (strategy 2 and 3) yields better performance than the one based on similarities.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful feedback.\n\nRegarding the differences between strategies 2 and 3, we identified three major points:\n1). Strategy 2 learns a sparser pattern than strategy 3, with many entries in the trust matrix being close to 0, unlike in strategy 3.  See Figure 4 of the paper. 2). We observed that the trust matrix in strategy 3 is more stable across communication rounds compared to strategy 2, with fewer abrupt changes in values. 3). The trust matrices from strategy 3 have higher diagonal values than those from strategy 2.\n\nWe cannot definitively explain why strategy 2 underperforms, but we speculate that the more stable trust matrix and higher self-trust in strategy 3 contribute to stabilizing the learning process, leading to better results. We plan to include more visual evidence on the evolution of learned trust matrices in future iterations.\n\nYes, we acknowledge that the method cannot scale up very well to a large number of clients. Being difficult to scale up is a general problem in all collaborator-selection type personalized methods, including ours. However, we emphasize that in this project, we target stateful clients (each client has a personalized model state) instead of stateless clients as in cross-device setup[1]. \n\nRegarding the incremental contribution comment, methodologically, we acknowledge that we are not making novel contributions. However, to the best of our knowledge, this is the first work demonstrating that collaboration can be leveraged to improve personalization performance in language modeling.  By personalization, we mean that each client has a personalized state. We consider this a significant contribution. In language modeling, the balance between personalization and collaboration is not straightforward. Despite the drastic differences between categories, there are notable similarities in sentence structure and word choices, alongside domain-specific word choices. We would also like to draw your attention to Table 4, which shows that in different scenarios, the performances of FedAvg and Local methods vary significantly, and our collaborator-selection-based method can give consistently good performances. \n\nThanks for the related work suggestions. We will make sure to include them in future iterations.\n\n[1] Kairouz et al. Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning Vol 4 Issue 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The three strategies mentioned in this paper are easy to implement and do not require heavy hyperparameter search. \n-  The experiments are robust, covering different data heterogeneities and showing clear benefits of the proposed methods over traditional FedAvg and local fine-tuning.\n- The paper provides the analysis on the superiority of prediction-based aggregation methods (strategy 2 and 3), which helps in understanding why these methods might be more effective.", "reasons_to_reject": "- While the paper demonstrates that strategy 3 consistently outperforms strategy 2, there is no detailed analysis explaining why this is the case. Since both strategies use prediction-guided trust matrices, additional insights into their differences would strengthen the conclusions.\n- The methods may not scale well with a large number of clients, which is a significant limitation given the growing scale of data and model deployments.\n- The primary contribution seems to be the application of existing techniques (from Zhang 2021 and Fan 2023) to LoRA, which makes the contribution of the paper a bit incremental. \n- To enrich the comparative analysis and foster a deeper discussion, it is suggested that the manuscript incorporate several relevant on-device federated studies such as [1,2,3,4,5].\n\n\n[1] \"Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes\" (arXiv'23, ICML'24)\n\n[2] \"FwdLLM: Efficient FedLLM using Forward Gradient\" (arXiv'23)\n\n[3] \"Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources\" (arXiv'24) \n\n[4] \"Can Public Large Language Models Help Private Cross-device Federated Learning?\" (arXiv'23, NAACL'24)\n\n[5] \"FS-Real: Towards Real-World Cross-Device Federated Learning\" (KDD'23)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SQPphgmQgD", "reviewer": "Reviewer_jbCH", "summary": "This paper investigates a personalized collaborative fine-tuning of LLM on-device, proposing three trust-weighted gradient aggregation strategies to deal with local data heterogeneity and scarcity problems. The significance of the work lies in the current landscape where data privacy, device-centric ML and federated learning are growing in importance. By pushing forward with collaborative learning directly on devices through parameter efficient fine tuning, this approach has the potential to open up new avenues for more personalized, secured and efficient applications. The paper succinctly explains three strategies, conducting limited experiments in unsupervised fine-tuning tasks, comparing them with one SOTA federated learning method. Additionally, it provides supplementary experiments aimed at interpreting the underwhelming performance of the first strategy. However, the lack of theoretical explanation or analysis for this fact and the other two strategies is notable. The paper is clear and easy to understand, although some ambiguity may arise from certain figures and letter symbols. Nevertheless, the three strategies lack significant novelty compared to the referenced work and do not introduce substantial modifications or improvements.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We acknowledge our test scenarios are limited due to computing resources. Regarding datasets, we argue our constructed scenarios reflect realistic language heterogeneity, encompassing variations in language usage, topic-specific wording, and data-source-specific wording. \n\nAdditionally, as suggested, we tested our methods on a larger dataset (84 million training tokens per client, 100 times larger than in the paper, 1st Table below). We also compared our strategies against baseline methods across different LoRA ranks (2nd Table). Both show consistent trends as in the paper, confirming the validity of our methods. (S denotes strategy in tables)\n\n| Local | FedAvg |S1 | S2 | S3 | Theoretical |\n| ----------- | ----------- |----------- |----------- |----------- |----------- |\n| 37.80 (0.15) | 56.69 (0.35)|48.47 (2.55) |36.17 (0.30)|35.63 (0.15)|35.59 (0.16)|\n\n| LoRA Ranks      | 4 |8 | 16 | 32 |\n| ----------- | ----------- |----------- |----------- |----------- |\n| Local | 40.24 (0.25) |33.15 (0.15)| 29.14 (0.11)| 27.41 (0.11)|\n| FedAvg   | 53.23 (0.51) | 48.84 (0.34) |42.68 (0.06) |40.12 (0.25)|\n| S1 | 49.34 (2.46)| 48.37 (1.26) | 40.50 (0.11) |34.65 (5.73)|\n| S2 | 37.20 (0.20) |30.41 (0.13) |26.36 (0.12) |24.37 (0.20)|\n| S3  | 36.92 (0.12) |30.25 (0.19) |26.3 (0.11) |24.39 (0.02)|\n| Theoretical | 35.96 (0.24) |30.06 (0.16)| 26.16 (0.07) |24.43 (0.10)|\n\nClarifications on symbol ambiguity: The algorithm assumes an all-to-all communication graph. While Figure 1 illustrates one client for readability, communication occurs in parallel for each client. Symbol $s_i$ denotes the message client i sends apart from $\\Delta \\theta_i$, which varies by strategy. Their meanings and their extra computation costs can be found in Table 1 in the paper.\n\n2. Methodologically, we acknowledge limited novel contributions. However, this is the first work demonstrating collaboration leveraged to improve personalization performance (each client has a personalized state) in language modeling, which is a significant contribution.\nNote that the  _theoretical_ baseline is not available in practice, as it requires information on underlying data mixtures on the device. The theoretical baseline helps to demonstrate how well our method can perform even without knowing data distribution a priori.\n\n3. Regarding why strategy 1 underperforms, please refer to Figure 8 in the appendix: In strategy 1, the trust matrix converges to a uniform matrix during training, resembling FedAvg's performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Developing device-centric collaborative parameter efficient fine-tuning methods to address data privacy concerns and heterogeneity challenge holds significant promise.", "reasons_to_reject": "1. The paper lacks sufficient experiments with multitask and larger datasets, and controlled experiments to demonstrate the methods’ generalization performance claimed by the authors. The three strategies were only tested in unsupervised fine-tuning task (focus on diverse topic, no considering other variation like writing styles, vocabulary choices, grammar structures, etc.), with perplexity as the sole evaluation metric. The dataset size is relatively small (approximately 1600 samples based on the provided token count and context length). Additionally, the comparison with the sole federated learning algorithm, FedAvg, and the fixed rank of the LoRA adapter (set to a small value of 4) limit the persuasiveness of the paper.\n2.  Some figure and letter symbols may cause ambiguity. From the Algorithm, each user should broadcast their LoRA weight update to all others agents, while in Figure 1 just user 1, 3, 4 broadcast their updates to user 2. What does the symbol si message refer to? (LoRA weight for user i?)\n3. Lack of novelty. Three strategies do not introduce substantial modification compared to the corresponding three referenced works. According the the experiment results, these methods exhibit limited improvement compared to the baselines, even perform worse than the simple but strong baseline theoretical but consume much more computing power.\n4. Lack of theoretically explanation to the underwhelming performance of Strategy 1 and that of Strategy 2 and 3 compared to the strong baseline. Some facile explanations or assumptions (like \"This could be attributed to the high-dimensional, sparse nature, and small magnitudes of LoRA weights\"), but lack of further experiment to verify them.\n5. Lack of limitation discussion.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PHrO9ZT1A0", "reviewer": "Reviewer_nijS", "summary": "This paper proposes an approach to fine-tune personalized language models in a collaborative setting. Specifically, the idea is to derive a set of LoRA weights for each user by aggregation across other users who are similar to this user. The paper presents 3 strategies for aggregation based on a) similarity of weights, b) similarity of predictions based on user specific validation sets, i.e. how well LORA weight of user j performs on validation set of user i, c) similarity of predictions based on a shared dataset. The paper finds prediction based similarity works best and furthermore using a shared dataset works somewhat better.\n\nStrengths:\n* Presents an effective strategy for developing personalized language models while still benefiting from other users in a collaborative setting\n* Demonstrates that a weight aggregation strategy based on predictions outperforms one based on model-similarity.\n\nWeaknesses:\n* In a realistic scenario, users may be wary of sharing information with other users. Therefore, we may want to add some level of noise to a user's prediction or LoRA weights before using those weights in an aggregation setting. The paper does not discuss this privacy angle.\n* The paper does not report an ablation wrt the amount of fine-tuning data and how that can affect the performance of the 3 strategies.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback.\n\nRegarding privacy concerns, while the paper didn't extensively address them, we ensured a degree of privacy by keeping data on local devices. However, we agree that discussing privacy implications further would enhance practical value. Our primary focus is demonstrating that collaboration can enhance personalization performance in language modeling, which is a significant contribution. We also conducted experiments in realistic language heterogeneity scenarios to empirically validate our strategies.\n\nRegarding the ablation on the size of fine-tuning data, we conducted supplementary experiments, increasing the training tokens by 100 times, resulting in 84 million training tokens per client in the multilingual setup. The ranking of all methods remained consistent with our paper's reported results.\n\n\n| Local | FedAvg |Strategy 1 | Strategy 2 | Strategy 3 | Theoretical |\n| ----------- | ----------- |----------- |----------- |----------- |----------- |\n| 37.80 (0.15) | 56.69 (0.35)|48.47 (2.55) |36.17 (0.30)|35.63 (0.15)|35.59 (0.16)|\n\n\nWith regard to your raised questions, we answer them as follows:\n\n- The AG News dataset is very small, and we do not have enough data to construct a shared dataset across the clients, which is why we left those entries empty. However, we explored other possibilities for obtaining a publicly shared dataset and generated some synthetic news articles using ChatGPT. With this synthetic shared data, we obtained the following results: our strategy 3 performs on par with strategy 2, and both outperform all other baseline methods by a large margin. This further illustrates the practical potential of strategy 3, as one can easily create such synthetic datasets.\n\n\n| Method | Local | FedAvg | Strategy 1 | Strategy 2 | Strategy 3 | Theoretical |\n| ----------- | ----------- | ----------- | ----------- |----------- |----------- |----------- |\nLow heterogeneity | 30.17 (0.17) | 31.66 (0.20)|31.43 (0.35) |29.75 (0.23)|29.81 (0.13) |29.56 (0.21)\nHigh heterogeneity | 28.67 (0.13) | 32.08 (0.13)|31.93 (0.86) |28.29 (0.06)|28.72 (0.27) |28.08 (0.11)\n\n\n- Yes you are right about the mistake in Figure 1. We will make sure this is corrected in future iterations.\n\n- Thanks for pointing this typo out, it should be “ the weight of edge (i, j) should denote to what extent can user j ’s gradients help to facilitate user i’s learning progress.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents an effective strategy to train personalized language models in a collaborative setting. Such an approach seems worthwhile given that lot of meaningful data is in user specific private repositories and unavailable for LLM training\n* The paper compares multiple strategies for aggregating user weights and demonstrates that a prediction based strategy works better than one based on model-similarity, a good finding", "reasons_to_reject": "* In realistic scenarios, users are concerned about privacy. Thus, it would be important to add noise to the user's weights or predictions before incorporating them in an aggregation setting. The paper does not discuss this aspect.\n* In a collaborative setting, users exhibit heterogeneity in terms of usage I.e. in terms of amount of fine-tuning data available. The paper does not discuss this dimension and how it may impact performance of the proposed approach.", "questions_to_authors": "* Why is strategy 3 not reported for AG News?\n* In Figure 1, should  \\delta \\theta_2, s_2 be \\delta \\theta_3, s3?\n* In Sec 3.2, \"Essentially, the weight of edge ( i , j ) should denote to what extent can user j ’s gradients help to facilitate user j ’s learning progress\" should this be  \"user i's gradients help to facilitate user j's ...\" ?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper discusses methods for collaborative / distributed fine tuning, and introduces several such "trust based" methods. Experiments compare their methods to federated learning and other known methods for aggregating ML models. 
The problem is important and I find the paper to be well written. 
Reviewers ranged from lukewarm to positive, and I recommend acceptance. Please address the comments by the reviewers.
Decision: Accept