Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
Authors: Jiamu Bai
Daoyuan Chen
Bingchen Qian
Liuyi Yao
Yaliang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Large Language Models
Parameter Efficient Fine-tuning
Primary area: other
TL;DR: We propose FlexLoRA, an aggregation scheme for federated learning of LLMs that dynamically adjusts LoRA ranks to harness the full potential of diverse client resources, enhancing generalization, and validated on thousands of heterogeneous clients.

Abstract: Federated Learning (FL) has recently been applied to the parameter-efficient fine-tuning of Large Language Models (LLMs). While promising, it raises significant challenges due to the heterogeneous resources and data distributions of clients.This study introduces FlexLoRA, a simple yet effective  aggregation scheme for LLM fine-tuning, which mitigates the "buckets effect" in traditional FL that restricts the potential of clients with ample resources by tying them to the capabilities of the least-resourced participants. FlexLoRA allows for dynamic adjustment of local LoRA ranks, fostering the development of a global model imbued with broader, less task-specific knowledge. By synthesizing a full-size LoRA weight from individual client contributions and employing Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Involving thousands of clients performing heterogeneous NLP tasks and client resources, our experiments validate the efficacy of FlexLoRA, with the federated global model achieving consistently better improvement over SOTA FL methods in downstream NLP task performance across various heterogeneous distributions. FlexLoRA's practicality is further underscored by our theoretical analysis and its seamless integration with existing LoRA-based FL methods, offering a path toward cross-device, privacy-preserving federated tuning for LLMs.

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
Jiau Bai"
Daoyuan Chen'
Pennsylvania State University jvb6867@psu. edu
Alibaba Group
daoyuanchen. cdy@al wwwllliiaabannnn.c. com
Bingchen Qian Alibaba Group
Liuyi Yao
Alibaba Group
qianbingchen. wwwpeeennnneelaabbb-ibiieen com
>lyy27738@llibbbaincc. com
Yaliang Li
Alibaba Group
yaliang. li@al wweeealbaba--ie ibaba-inc.com
Abstract
Federated Learning (FL) has recently been applied to the pnrameter-efficien fine- tuning of Large Language Models (LLMs). While promising, raises significant challenges due to the heterogeneous resources and data distributions of clients. This study introduces Foe, simple yet effective aggregation scheme for LLM fine-tuning, which mitigates the "bucket effect" in traditional FL. that restricts the potential of clients with ample resources by tying them to the capabilities of the least-resourced participants. FlexLoRA allows for dynamic adjustment of local LoRA ranks, fostering the development a global model imbued with broader, less task-specific knowledge. By synthesizing full-size LoRA weight from individual client contributions and employing Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Involving thousands of clients performing heterogeneous NLP tasks and client resources, experiments validate the efficacy of FlexLoRA. with the federated global model achieving consistently better improvement over SOTA FL methods downstream NL.P task performance across various heterogeneous distributions. FlexLoRA' A's practicality is further underscored by our theoretical analysis and its seamless integration with existing LoRA-based FL methods, offering path toward cross-device, privacy-preserving federated tuning for LLMs.
1 Introduction
Large Language Models (LLMs) have propelled advancements in natural language processing (NLP), offering breakthroughs in various tasks [45). Finetuning LLMs specific datasets enhances their yet collecting such datasets raises concerns regarding cost and privacy 29]
applicability 12 33
Researchers have turned to Federated Learning (FL.) as means to fine-tune tune L.MMs using more data across distributed clients without compromising data privacy [28] 43 32). In these settings, parameter-efficciet fine-tuning techniques [34), particularly Low-Rank Adaptation (LoRA) [16), become attractive for reducing computational and communicational burdens
"Equal contribution. Work done during Jiamu Bai's internship at Alibaba Group.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Test Loss of FediT under Different LoRA Ranks
Aggregation heicregenrons ranks
Glibal Nade Gemmer side
Aggregatin
rank-
Aval
W.
-...
Decarspood distriitt looal en local
nank, 200
FleciorfA
222
Getting LaR
Gloaal Noad
A
1.8
Local Training
1.6
G
L +
Luke
NS
Trassiation
15 Communizaiinn Round
30
B
Figure 1: Test loss of Flexl L and FedIT [43 across communication rounds under LoRA ranks of 1, 8, and 200. FlexLoRA demonstrates adaptability in an "extreme heavy tail" scenario and increasingly aligns with the performance of FedIT at the highest LoRA rank as rounds progress. Implementa- tion details are Appendix
Generiliation
Figure 2: Illustration of FlexLoRA.The server ini- tially constructs full-size LoRA weight, which is then averaged across client-contributed weights with different ranks. The aggregated global weights decoupled via SVD and sent back to clients.
Despite its efficiency, LoRA's use in FL is challenged by the heterogeneity of downstream tasks and available resources among clients, especially in cross-device scenarios [39] 5. Traditional FL methods often suffer from "bucket effect", converging to the use of the smallest viable LoRA rank for all clients, even though many clients typically have more resources that remain underutilized. small LoRA rank, optimizing weights in a task sask-specific manner [16]. can be sensitive to heterogeneous data distributions and compromised generalization when applied to all clients, as evidenced in Figure [ Ideally, we hope all clients can fully leverage their advantages by sizing their local LoRA ranks with their resources to contribute models with less aasc-specific more generalized knowledge.
To address these challenges, we propose FleXLoRA, a simple yet effective FL aggregation scheme that enables the mixture of diverse LoRA weights across individual clients. It accounts for local resource and task differences and aims for a well-generalized global model. With the heterogeneous aggregation and redistribution of weights through Singular Value Decomposition (SVD), LoR ensures all clients contribute effectively, regardless of resource capacity. Thanks to the simplicity, FlexLoRA can be pluggable into a series of LoRA-based FL methods, unlocking their potential to leverage available yet muder-utilized, resources to contribute more generalized knowledge via larger LoRA ranks, which also supported by our theoretical analysis.
Our empirical study, simulating a cross-device federate fine-tuning scenario with thousands of clients various of NLP tasks [39] and resource distributions, underscores the ity of FlexLoRA Notably, FlexLoRA can be readily applied to several SOTA FL baselines in plug-and and-paa manner and achieves significant performance enhancements, including 3.1% and 4% improvements in zero-shot Rouge-L scores and language understanding tasks such as overlap extrac- tion and textual entailment, demonstrating robust generalization capability. We further conduct an extensive study on the aggregation scheme and scalability of FlexLoRA. furnishing a more nuanced understanding of the underlying mechanisms that facilitate its effectiveness
applicabil-
Our contributions can be summarized as follows:
We propose simple-yet-effeen scalable method fully leverage local client resources for enhancing the global model's generalization ability, supported by both theoretical analysis and extensive empirical evidence.
To our knowledge, this is the first work to demonstrate the feasibility of federated tuning of billion- sized LLMs across thousands of NLP tasks in large-scale, resource-heterogeneeos scenarios.
We explore the interplay between LoRA ranks, client numbers, specific heterogeneous language tasks, and resource distributions, offering practical insights. Our code is made available at inviting further research and appli-
www.a//iiioaaaeeeeeeeeeeeeeeeeee cation in roal-world cross device FL for LLMs.

---

2 Related Work
Seannette-Efficien Fine-tuning of LLMs. The computation and storage demands of traditional fine- tuning processes have spurred the development of parameter-efficient fine-tuning (PEFT) techniques such as adapter and prefix tuning [15]2]] Among existing PEFT techniques, we choose to employ LoRA due to its simplicity and outstanding performance 224 [8] Despite this, our aggregation scheme can be easily extended into other PEFT methods by replacing the LoRA weights with their
alternative weights to be tuned.
PEFT in Federated Learning. PEFT techniques have been integrated into FL to minimize communi- cation costs and maximize efficiency. Several works employ LoRA for local model updates within an FL framework 43 41 19] 8 26 30 38 42]. For instance, 43] combines LoRA- R-aaedd local updates with FedAvg for model aggregation, while 2 intersperses sparse finetuning with LoRA fine- tuning for improved initialization for LoRA in FedAvg. [36] proposes a technique to improve LoRA performance in FL scheme, 42 exploits performing SVD on pretrained model weights to resolve data heterogeneity, and [32] reduces communication cost through zeroth-order optimization. Distinct from these methods, our work, by introducing simple yet effective aggregation scheme, leverages heterogeneous client resources to enhance the generalization and natural language understanding of the global FL model, addressing limitations seen in current FI. paradigms.
Data and Resource Heterogeneity in FL. Data and resource heterogeneity remain significant challenges in FL, impacting both training and performance
228 20]. Fruitful solutions have been explored to tackle the data heterogeneity 10 25 42 or resource heterogeneity [ while not in LLM context. A concurrent work, HETLORA 8 proposes allowing heterogeneous LoRA ranks by zero- pero-padding local LoRA weights for aggregation and truncating global weights to match the local rank for distribution, all while employing sparsity regularization. However, approach distinguishes itself through a focus on zero-shot task generalization and large-scale experiments inclusive of thousands of NL.P tasks and clients, aiming to synthesize well-generalized global L.M.. Moreover, our method is simple and easy to use without any syper-parameters for the aggregation, thereby circumventing the need for case-by-case tuning of newly introduced variables such as the decay and regularization factors of HETLORA.
3 Methodology of FlexLoRA
3.1 Intrinsic Dimension and Generalization
Fine-tuning LLMs to enhance task task-specific performance inevitably encounters cost of reduced generalization ability: trade-off supported by the "no-free-lunch" theorem and empirical evidence usually called "alignment tax" of LLM [40 31. The generalization capability of LLMs is influenced by complexity of applied tasks and their solution spaces, which can be characterized by the concept of an intrinsic dimension - typically far smaller than the total number of model parameters [
The insight of intrinsic dimension informs the design of LoRA to fine-tune LLMs' pre-trained weights in a parameter-efficient manner, utilizing compact and low-rank matrices. Specifically, matrices A E Rx'x, B E dx are introduced, where denotes the rank that encapsulates intrinsic dimension. These matrices form a low-rank approximation for tuning original weights W, as h = Wa + sBAz, where is the input of the parameter to be tuned, h is the output, and s is a scaling constant. Previous studies show that different ranks produce weights with attributes particularly tailored to specific downstream tasks [16 18] Consequently, the rank value plays critical role in not only task-specific solution subspaces but also in determining a model's ability to generalize to various tasks.
In scenarios where clients have highly heterogeneous task and resource distributions, uniform LoRA rank usually does not suffice for model performance, especially in its zero-shot generalization ability for unseen clients and tasks. Employing small LoRA rank potentially leads to under-fittin fitting in global context by capturing only a subset of task-specific features, while a large rank is usually infeasible due the "bucket effect" of existing FL solutions constrained by least-resourced clients.
Loe emerges as a solution to this dilemma by dynamically adjusting the rank response to the variability in local client resources. By increasing the LoRA rank for clients with greater resources to contribute more global knowledge, FleX enhances the model's 's ability to generalize across diverse data distributions without sacrificing local performance accuracy. This strategy allows for
3

---

federated lin-ttunng of LLMs to navigate between the extremes of task-specific optimization and generalization to unseen clients and tasks.
3.2 Aggregation with Heterogeneous Ranks
Traditional FL methods like FedAvg aggregate local LoRA weights by computing a weighted average of the decomposed matrices A and B Bg (2 n' Af) n'), where Bg Ag are the global LoRA decomposed matrices, and B,, Aj are the local LoRA decomposed matrices of i-th client, n² is the size of the i-th client's local training dataset, m is the number of Fl. clients. However, this scheme is restricted by the lowest LoRA rank among participating clients for aggregation compatibility, which makes it hard to capture the full diversity of client contributions and fully utilize ample client resources.
Ag
FlexLoRA takes a different yet simple approach to enable decomposed matrix with different LoRA ranks be mixed together. Specifically, first forms a low-rank approximation of the LoRA matrix for each client, W7, before computing the weighted average: W = ( (E''W))/E n²) =
(E n s s)JAB/)))))) ( n²).
After the weighted average with heterogeneous LoRA ranks, the resulting global LoRA weight Wg is decomposed using SVD. Then the SVD components U, Σ. V are redistributed to clients in a low-rank approximation that preserves as much information of W, as possible meanwhile based on clients' local resources characterized by r":
SVD(W,) - UEV,,
W - r r ~ W
where U, E, and V are the SVD components of Wg. the r¹ within [] indicates the indexing operator of each client to select their singular vectors corresponding to top r' singular values. As a result, client i receives the aggregated knowledge W from server and incorporates W into its local LoRA weight W BB A with B U[: r]][[r+ r//s,, and A V(: :
The local training then proceeds as similar to those in standard FL approaches, using Wi as the local weights to be tuned. This aforementioned process is repeated until convergence is achieved or predetermined number of rounds completed.
3.3 Maximizing Local Rank with Local Resources
To fully utilize the local resource of a local client, we adhere to the principle of allocating the highest feasible rank given a client's resource budget, which is motivated by our empirical finding that larger ranks generally yield better generalization. Figure I demonstrates that models trained with uniformly higher ranks outperform those with lower ranks under conventional parameter-average aggregation schemes. For single-client performances, the zero-shot performance is also boosted in the majority of the cases. The Table 13 and Figure 9 from Appendix [ show that performance improves with higher LoRA ranks uniformly regardless of the tasks assigned to each client. While there might be an ideal LoRA rank that maximizes single client's
poteniial as high as 200 ppacticca resource limitations may necessitate settling for a lower rank, such as 8. Therefore, we adopt the principle of setting the rank to be as large as possible to completely utilize the resources FlexLoRA. which is easy to implement and under low risk of overfitting as Occam's 's razor suggests.
In Appendix B the overall procedure of Foe and its core function of server update summarized in Algorithm Algorithm Aggorithm@respecctii respecttivll Foe optimally leverages the inherent characteristics of LoRA, boosting model generalization effectively by increasing local ranks, while without sacrificing overall training efficiency. Compared to FedAvg and homogeneous bask- FL methods, FlexLoRA incorporates a lightweight SVD procedure, but the overhead from SVD is negligible compared to the local LLM training procedure. Moreover, the SVD is performed only once per round and is independent of client numbers. Notably, FlexLoRA enables heterogeneous ranks without the need for any additional hyperparameter tuning. As we empirically demonstrate in Table 4 the improved convergence rate, thanks to larger ranks, more than compensates for the extra overhead more parameters per round, resulting in a net gain in overall efficiency
introduced by training and a reduced overall time to completion. These features enhance its efficiency and scalability in cross-device FL settings where thousands or millions of devices are involved.

---

3.4 Generalization Analysis
We analyze the generalization ability of FlexLoRA by extending Baxter's is model of learning 33. Here, (-) represents the hypothesis generated by the model with LoRA weights W, and f (W(xx,)) denotes the loss function for single data point (z. y). The expected loss is denoted as ((,,) -
B(aa)) f (: (z. (z.))) The two key assumptions underpinning our analysis are as follows:
Assumption 1. The following Lipschitz conditions hold: [f(W; f x.,)) S W II and [|((W;;) x) - h(W': :z)]1 S L.llW W'l-
Following current analysis in SOTA methodologies in LLM research, we assume Lipschitz conditions in our analysis for f and h [27 17 [3]. This assumption indicates that f and h are Lipschitz continuous with respect to the LoRA weights W, ensuring the stability of the loss landscape. For simplicity, we denote SVD(W,, g. r²) as using the top r' singular values and the corresponding singular vectors to approximate W.. We denote the parameter solution space of W, g to be k. Next, due the federated average and indexing operation based on the largest singular values, dissimilarity between the global model and its rank- consttandd approximation can be bounded:
assume that the
Assumption 2.
The LoRA weights can be bounded in ball with radius R, and the error induced by the SVD approximation for each client is bounded by a constant o' as ISVD(W,,, r²) - Wa I| S Φ². 7 and 2 with probability at least 1 1-6 8. there exists a sample size log such that for all Wg the bound (| - L(W e Tep2
Theorem 1. Under Assumptions
Ñ ( [og(=-20
holds when the number of local data samples for each client exceeds Ñ.
Detailed proof is in Appendix C This theorem suggests that the generalization ability of the global model is influenced by the LoRA rank r¹ chosen by each local client. Specifically, d' is the error bound of SVD approximation, increasing rank r' makes the approximation more accurate, thus reducing ø'. Consequently, this reduction in eтror bound decreases the requisite number of samples, denoted as Ñ, required for effective generalization. Moreover, increase in the number of clients |C| also contributes positively to the generalization of the federated model in the order of ((2))) a
stronger impact than o' whose impact in a logarithmic fashion (log( + )). Collectively, FlexLoRA effective in cross-device FL settings, where the generalization capability of the global model can be significantly enhanced by the participation of massive clients (larger (C') with heterogeneous resources (larger r²), Note that Assumption [ is standard in FL literature [22], and our proof do not rely on simplified assumptions that often do not hold in cross-device cases, such as identically distributed data. We provide empirical support for distribution-relate generalization ability, the effect
of SVD (Assumption 2). and the effect of client numbers in Sections 4.3 4.5 and 4.6 respectively.
4 Experiments
4.1 Setup for Cross-Device FL Environments
Resource Heterogeneity. We make FL clients resource-heterogenenn by crafting four distinct LoRA configurations as listed in Table ! Type 1, 2, and 4 assign the same LoRA on all tunable layers, while Type 3 assigns small ranks on attention layers and large ranks on FFN layers, following the
design of the MAM adapter [4]. Clients are randomly assigned a configuration type, simulating four types of heterogeneous resource environments similar to 5 As shown in Figure 3] we consider uniform resource distribution where each LoRA configuration type is equally likely to be assigned to each client, heavy tail resource distribution where either Type I or Type 4 is dominant, and normal distribution where the LoRA configuration types are normal distribution and Type 2 and Type 3 are dominant. A comparison of the active memory cost with different LoRA configurations is shown in Appendix D Besides, we add LoRA on top of all the linear layers of LLMS based on the empirical results in Appendix E.3
Task Heterogeneity. We further make FL clients sess-heterogeeeuus by utilizing the natural instruc- tion dataset (39]. The dataset consists of over 1600 distinct natural language tasks that come from 76 NI.P task types and split based on its meta-info of the belonging NI.P tasks, such that each client holds a unique task to mirror task heterogeneous environment. Notably, while the FL setup
5

---

includes over 1600 clients, the distribution of 76 task types across these clients means that some will inherently share similar local data distributions, thereby mirroring the natural variability and overlapping task characteristics often encountered in real-world settings. Unless stated otherwise, we conduct all our FI. experiments on this dataset and adopt DataJucier (1.3B) as our foundation models [6], chosen for its suitability for edge devices with constrained resources. More details about data preparation are included in Appendix E.2
Heterogeneous Resource Distribution Composition
05
oongguration
J.
Table 1: The LoRA configurations that compose heterogeneous resource distribu- tions, detailed Figure
Type
Typw?
Type
LoRA Config
Params
Type 1 = on all layers
0.12 %
A.1
Type
30 all layers
2.46 %
Unfoom
Heavy heavytall-lgh
Heavy ta strong
Norma
T == 30 on atten layer,
Resource Distrbutions
Type 3
8.22
200 FFN layer
Figure 3: Heterogeneous resource distributions contain-
Type 4 r r= 200 on all layers
12.22 % ing different ratios of various LoRA configuration types.
4.2 Setup for FL Baselines
Baselines with Homogeneous Rank, We adopt FedAvg
28 FedIT
43 and SLoRA 2 as
baselines utilizing unvarying LoRA ranks. FedIT aggregates the LoRA module weight of each client by averaging which limits the local LoRA rank to meet the lowest resource constraint. Comparing with FedAvg, FedIT adopts Adam optimizer for local training instead of SGD optimizer. SLoRA first trains local client models with sparse fine-tuning for several epochs then switches to LoRA for PEFT and uses the updates from sparse fine-tuning as initialization.
Baselines with Heterogeneous Ranks. Besides, we compare with HETLORA [8. a concurrent work exploring the effective utilization of diverse LoRA ranks in FL. It first employs rero-padding all the LoRA matrices based on the largest rank, then conducts element-wist averages like FedAvg, and finally truncates the aggregated model to fit the local client LoRA rank.
We note that both Fle and HETLORA are able to be plugged into the above-mentioned FL methods with homogeneous LoRA ranks. In our experiment, for each baseline with homogeneous rank, we also examine their performance after integration by either FlexLoRA or HETLORA.
4.3 Unseen Client Generalization
Evaluation Setup. Our initial examination focuses on the generalization capabilities of global models to unseen clients by deploying the models to clients with unseen data distributions. The unseen clients are newly sampled clients from the next communication round. This assessment allows us to measure zero-shot performance, key indicator of a model's ability to generalize beyond the data available during the training phase. This approach is simulate the real FL setting, where well-trained global weights will be deployed to new clients rather than clients participating in the previous round. Specifically, we investigate the performance of global models trained with baseline methods both with and without the integration of FlexLoRA and HETLORA under four distinct resource heterogeneity scenarios.
Overview Performance Comparison. Table displays the zero-shot performance under Rouge-L scores on the test set from unseen clients, facilitating comparison of the generalization capabilities across different federated global models. It is observed from the table that in most of the cases, methods with heterogeneous LoRA ranks have better performance than that of homogeneous ranks, indicating that heterogeneous LoRA ranks enhance the clients with larger ranks to fully exploit their capability. Furthermore, among the heterogeneous LoRA rank methods, our proposed AoA consistently outperforms HETLORA across all resource distribution settings. This shows that A is able to take advantage of heterogeneous resource distribution, and more capable of leveraging the general information from heterogencous LoRA configurations.

---

Table 2: The weighted average Rouge-L scores of unseen clients provide insights into the global model's generalization ability. Results from baseline methods with homogeneous ranks (Line 3, denoted as Homo Rank) are compared with those incorporating FlexLoRA and HETLORA across various resource distributions (Line 4-7)) The significant test are presented in Appendix
FedAvg
FedIT
SLoRA
FlexLoRA
HETLORA
FlexLoRA
HETLORA
FlcxLoRA
HETLORA
Homo Rank
56.53
61.29
60.01 + 0.74
Uniform
58.07 +:
56.85
+ 0.18
61.34
60.74
60.75
60. 70.74.0
Heavy-Tail-Lighh
57.39 1: 0.54
56,24
1: 0.30
61.88
1: 0.89
61.53
0.93
60.40
0.40
59.97 0.62
Normal
57.78
1: 0.33
56,50
1 0.05
62.01
: 0.91
61.03
= 0.54
61.67
1: 1.07
61.14
1: 0.71
Heavy-Tail--SSe Tail-Strong
57.73 1: 0,08
55.74
1: 0.93
62.20 1.12
61.06 0.95
61.86
1.24
61.29 0.95
Answerability Classification Coreference Resolution
Question Rewriting
Keyword Tagging
Unifom
Category
Heewy-thl-lght
Normal
96
Heavy
0.0
0.5
1.5
Improvament
Improvament
Improvement
Improve montt
Overlap Extraction
Data to Text
Word Analogy
Title Generation
Uniform
Catagory
teeayy-tllllight
$5
Normal
04
Heavy tail--tll strong
10
53
0.5
1.0
0.0
0.5
Improvement %.
Imprr ovorovem %
Improvament %
Improvement %
Textual Entailment
Cause Effect Classification Dialogue Act Recognition Grammar Error Correction
Uniform
51
84
88
Category
Heavy tall light
Nornal
55
64
Hamoy-tal-sttorg
0.0
0.5
1.0
1.5
-0.5
0.0
0.5
1.0
Improvement
Imppr Inprovement..
Improvement 9%
Improvement
Figure 4: Task-specific improvements achieved by FlexLoRA in comparison with the homogeneous rank implementation of FedAvg, across different resource distribution settings.
Effect of Specific Resource Distributions. To gain further insight into the effect of FlexLoRA and the effect of heterogeneity of LoRA ranks, in Table i0 Appendix we list the percentage improvement for each FL methods when incorporating FlcxLoRA in comparison with the respective standard homogeneous rank implementations. Notably, after integrating LRR,, the average performance gains are 2.14% for FedAvg, 0.86% for FedIT, and 1.94% for SLoRA. These enhancements lend empirical support to our theoretical generalization analysis that clients utilizing higher LoRA ranks tend to exhibit improved generalization abilities. The most substantial performance improvements are observed the heavy-tail-strom resource distribution, followed by the normal distribution. This is consistent with our expectations since the heavy-tail-strong distribution predominantly comprises clients with Type 4 LoRA configurations (rank 200). The limited presence of Type clients (rank 8) in the normal distribution minimizes the risk of the global model being excessively influenced by musk-specifie LoRA weights. Therefore, FlexLoRA is able to leverage the heterogencous resource distributions to boost the zero-shot generalization, and the gain from integrating FlcA is directly related to the ratio of clients with heavy resources.
4.4 Cross-Tase Generalization
Evaluation Setup. To assess the natural language understanding capabilities of the FlexLoRA- enhanced global model, we evaluate its performance on range of downstream NLP tasks. Specif- ically, the model tested on the English Track of the evaluation tasks from 33), featuring 12 categories and 119 tasks. For each task, a random sample of 100 data points is chosen for testing. We
7

---

Client Number vs Performance
Reicenia Debill cork
Table 3: Average percentage improvement of FlexLoRA over baseline methods (FedAvg, FedIT, SLoRA) across different resource distributions, calculated over 12 NLP task categories. More de-
Reation
Aste (A
logie Anl)
Haroo Saee Lose Thes
26
Liifuiry Loss Than
Homoo Torn Low Miled
Hraey Lona
tailed comparison is presented Figure
MMRL.
90739943
06/55344
FedAvg
FedIT
SLoRA
Avg
37335443
50,33343
Uniform
1.99%
0.97%
0.74%
1.23%
200
Heavy-Tail (l.)
1.24%
0.63%
-0.47%
0.47%
Number Glents (CD
Normal
1.34%
0.75%
0.96%
1.02%
Figure 5: Client number (IC) v.s. generaliza- tion heterogeneous distributions. Increasing
Heavy- Tail (S)
1.66%
0.95%
1.12%
1.24%
Avg
1.56%
0.83%
0.59%
1.00%
[C] improves both convergence loss and rouge-L (scores marked as (R-L)).
summarize the average percentage improvement achieved by integrating FlexLoRA across different resource distributions in T
Effect of Specific Resource Distributions. In the majority of cases, the global models augmented with FlexLoRA demonstrate marked improvements over the vanilla implementations of FedAvg, FedIT, and SLoRA by up to 1.99%, suggesting that the FlexLoRA also improves the natural language analysis capabilities. An exception is noted in the SLoRA on the heavy-tail-light distribution, potentially due to the predominance of the Type LoRA configuration (rank 8), which may limit the overall language processing capabilities when such clients are disproportionately represented. This configuration's minimal rank assignment across all linear layers suggests that the local weight aggregation the server side might not fully leverage the capabilities of clients with larger resources, potentially detracting from the model's performance on certain tasks.
Effect of Specific Language Tasks. We further illustrate the task specifcc improvements of in- tegrating FlexLoRA in comparison with the standard FedAvg configuration for various resource distributions in Figure The wase improvement figures for other FL methods are included in Appendix
G We observe that the global models trained using the FlexLoRA aggregation scheme generally outperform others on tasks requiring the parsing of logical relationships between sentences. Particularly, gains improvements most 4% in the overlap extraction task, and around 2.5% the textual entailment, cause-effect classification, and dialogue act recognition task, verifying again the effectiveness of Flc
4.5
Aggregation Scheme Study
Effect SVD
Q_proj FlexLoRA heavy-tail-stronn distribution
Aggreggtion
block 1 singular value
2.6
FediT rank
FedIT rank
block Seaii--ringggeeeooe
singular
rank
block 1 wrrr ralio
singular value
block Same calin
block rado
rallo
2.2
0.4
2.0
0.16
1.6
2.5
5.0
10.0
12.5
15.0
17.5
20.0
Commurication
Order
(a) Losses FedIT and FlexLoRA-innegratee FedIT.
(b) The Aproy singular values and estimate error.
Figure 6: The sub-figure 6(a) shows that FedIT with LoRA rank 8 has comparable test loss curves for standard and FlexLoRRA integration. At rank 200, though, standard FedIT differs from other versions. debicts singular value distributions and approximation errors, where the red cross indicates the average error for rank 30 groo weights in specific blocks. Further details are in Appendix H

---

SVD on Convergence. FlexLoRA's aggregation scheme constructs full-size LoRA weights before averaging, unlike the conventional FedAvg method's parameter- -eiie averaging. To understand the effect of this difference on model performance, we assess the performance of Foe in con- trolled environment using homogeneous LoRA ranks and compare to the standard FL aggregation scheme. Figure 6(a) presents the test loss trajectories for FedIT with and without the Foe enhancement, both utilizing a homogeneous rank of 8. The loss curves for the standard FedIT and FedIT with Fle closely align, suggesting comparable performance. In nutshell, we empiri- cally demonstrate that under homogeneous conditions, FlexLoRA's aggregation does not negatively impact model performance compared to traditional methods. Besides, it's worth noting that the test loss for FedIT with a homogeneous rank of 200 is significantly lower, underscoring the benefits of higher rank configurations and evidencing our Theorem !1
SVD V.S. LoRA Rank. To gain further insight into the effect of SVD, we calculate and sort the singular values of the global LoRA weights from largest to smallest. We focus on specific layers where LoRA is applied within the transformer blocks 1, 8, and 14 (each block includes one attention layer and one FFN layer). Figure displays the scale of singular values and the error ratio between the global LoRA weights approximated by the top i singular values and the full-rank global LoRA weights in the FedIT setting with heavy-tail-strone is quantified as the norm of the difference between the approximated and full-rank weights. The error curves for Ooroj layers across all transformer blocks nearly overlap. With the weight approximated from the top 30 ranks, the error ratio is as low as 0.16, suggesting the approximated weights are in close proximity to the actual full-rank weights, lending empirical support to our Assumption 2
resource distribution. The approximation error
4.6 Scalability Study
Larger Model Size and Lower Degree of Task Heterogeneity. While the aforementioned exper- iments with 1.3B LLM and meta-task dataset split effectively showcases FlexLoRA's capabilities against baselines in highly heterogeneous cross-device environment, real-world settings may be relaxed involving fewer clients with a mixture of tasks and larger LLM. To better evaluate FlcxLoRA's performance in such scenarios, we expanded our study to include settings where each client manages not just one specific "meta" task but variety of different tasks. We use Dolly-15K dataset (9), which supports instruction tuning and includes 8 tasks in total. We distributed this dataset among 200 clients using Dirichlet distribution with 0.5 to simulate the non-IlD data distributions. Moreover, we also incorporate the most cutting-edge LLaMA-3 (37 with 8B parameter size as our foundation model to assess FleXLoRA's effectiveness with advanced, larger-scale models.
Table 12 in Appendix | illustrates FlexLoRA's efficacy under both smaller and larger foundation mod- els within the mixture of task settings. Compared with LLaMA-33 (8B), finetuning on (ataJucier(1.3B) demonstrates more generalization improvement for unseen clients. This enhanced performance when using the smaller DataJuicer model suggests that FlcxLoRA is particularly effective for foundation models that are scalable to edge devices. Such ability is instrumental in maximizing the utility and efficiency of smaller models in resource-constrainee environments.
System Costs. We empirically demonstrate that in- creasing the rank improves overall efficiency. The experiment is conducted on the Dolly 15K dataset and the DataJuicer 1.3B model, with the same FL setting as Table [2] We summarize the results in Ta- ble 4 where the "T" indicates the number of rounds to reach loss of approximately 75% progress to convergence. The "C'ostR" stands for per-round FL cost in terms of the average model parameters com- pared with the foundation model parameters, as both the local training and communication cost positively correlated to the rank value in FlexLoRA (as analyzed in Sec. B.3). The "Costan" indicates total cost calculated as multiply of R and CostR. and by setting the result of "Homo Rank" as the baseline. From the results, we can see that A achieves faster convergence with slightly increased parameter percentages under heterogeneous resource distributions. For the overall efficiency, and "Uniform" has 66.9%, indicating good scalability of Fle
Table 4: Convergence round and FL. cost per round for different LoRA ranks.
R
Costn
Costaul
Homo Rank
48
1.001x
100%
Heavy-Tail (L)
24
1.014x
= 50.66%
Uniform
15
1.061x
33.1%
has a total reduction of 49.4%,

---

Effect of Larger Client Number. We designed an experiment to empirically validate Theorem
T and demonstrate how FlexLoRA performs with varied client numbers. We use subsets of 10,
50, and 100 clients from pool of 200 clients on Dolly-15K dataset and DataJuicer (1.3B), with FedAvg as baseline FL. method and each FI. round always sampling 10 participants. From Figure the number of participating clients increases. Moreover, from Theorem 1 we can derive the functional relationship between client number |C] and log(= Aa)). where 21,2.3 are constants absorbing the other factors impacted by specific resource distribution in this experiment. We thus fit these coefficients using the derived form and empirical observations, and find that the dotted curves gain good fitness for all the tested distributions. There results affirm the preciseness of our Theorem ! again, indicating the suitableness of FlexLoRA for cross-device FL scenarios where leveraging broad client pool can boost the generalization across diverse data distributions.
5 there is marked improvement in generalization
the generalization loss ICI- A1/
5 Conclusion
In this work, we propose a simple yet effective method named FleXLoRA to address the challenges posed by resource and data heterogeneity among clients during the federated fine-tuning of LLMs. By leveraging larger local LoRA ranks, FlexLoRA not only improves the generalization ability of the global model but also ensures that all clients, irrespective of their resource capabilities, can contribute meaningfully. Theoretical analysis and extensive experiments verify the effectiveness and scalability of FlexLoRA. Due to resource limitations, we have not tested the LLaMA-3 model on thousands-client scenarios, which we leave as future work. We hope this study can enlighten more future research and development data-efficient and
enhancement of L.I.Ms.
References
[1] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 7319-7328, August 2021.
[2] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El- El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint 2xXiv:2308.06522 2023.
[3] Jonathan Baxter. model of inductive bias learning. Journal of Artificial Intelligence Research, 2000.
[4] Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding, pFL-bench: A com- prehensive benchmark for personalized federated learning. In Advances in neural information processing systems (NeurIPS), 2022.
[5] Daoyuan Chen, Dawei Gao, Yuexiang Xie, Xuchen Pan, Zitao Li, Yaliang Li, Bolin Ding, and Jingren Zhou. Fs-real: Towards real-world cross-device federated learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
[6] Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexi- ang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. In International Conference Management of Data, 2024.
[7] Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, and Yaliang Li. Efficient personalized federated learning via sparse model-adaptation. In International Conference on Machine Learning (ICML), 2023.
[8] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low- rank approximation for federated fine-tuning of on-device foundation models. arXiv preprint 2rXiv:2401.06432, 2024.
10

---

[9] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 11m, 2023.
[10] Luca Corinzia, Ami Beuret, and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprint arXiv: 1906. 06268, 2019.
[11] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations (ICLR), 2020.
[12] Ning Ding, Yujia Qin, Guang Yang. Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023.
[13] Zihao Fu, Anthony Man-Cho So, and Nigel Collier. A stability analysis of fine-tuning pre-trained model. arXiv preprint arXiv:2301.09822 2023.
[14] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kerg-Kiikpatrick, and Graham Neubig. To- wards unified view of parameter-efficien transfer learning, In The International Conference on Learning Representations (ICLR), 2022.
[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for nlp. In International Conference Machine Learning (ICMI.), 2019.
[16] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations (ICLR), 2022.
[17] Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhong Xu, and Jiebo Luo. Improving pretrained language model fine-tuning with noise stability regularization. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[18] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13200 2023.
[19] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Tedd:attdssoop-LLLM:: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309. 003633, 2023.
[20] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE signal processing magazine, 2020.
[21] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2020.
[22] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In The International Conference on Learning Representations (ICLR), 2019.
[23] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 2021.
[24] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReloRA: High-rank training through low-rank updates. In Workshop Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023), 2023.
[25] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems (NeurIPS), 2020.
11

---

[26] Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the convergence of zeroth-order federated tuning in large language models. arXiv preprint arXiv:2402.05926. 2024.
[27] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36, 2024.
[28] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. coamumication--fffcceee learning of deep networks from decentralized data. In Artificial intelligence and statistics, 2017.
[29] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ip- polito, Christopher A Choquette- Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023.
[30] Duy Phuong Nguyen, Pablo Munoz, and Ali Jannesari. Flora: Enhancing vision-language models with pfffiettr-efffcient federated learning. arXiv preprint arXiv:2404. 15182, 2024.
[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in neural information processing systems (NeurIPS), 2022.
[32] Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Federated full-parameter tuning of fillion-sized language models with communication cost under 18 kilobytes. In International Conference on Machine Learning (ICML), 2024.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[34] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems (NeurIPS), 2017.
[35] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. In International Conference Machine Learning (ICML), 2021.
[36] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in pnivacy-preserving federated learning. The Twelfth International Conference on Learning Representations, 2024.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. /3971, 2023.
[38] Boxin Wang, Yibo Jacky Zhang, Yuan Cao, Bo Li, H Brendan McMahan, Sewoong Oh, Zheng Xu, and Manzil Zaheer. Can public large language models help private cross-device federated learning? arXiv preprint arXiv:2305. 12.305.11133. 12132, 2023.
[39] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstretiins Generalization via declarative instructions on 1600+ nip tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
[40] David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1997.
[41] Mingbin Xu, Congzheng Song, Ye Tian, Neha Agrawal, Filip Granqvist, Rogier van Dalen, Xiao Zhang, Arturo Argueta, Shiyi Han, Yaqiao Deng, et al. Training large-vocabulary neural language models by private federated learning for resource-constrained devices. In ICASSP 2023- 2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
12

---

[42] Yuxuan Yan, Shunpu Tang, Zhiguo Shi, and Qianqian Yang. Federa: Efficient fine-tuning of language models in federated learning leveraging weight decomposition. arXiv preprint ar04v:2404. 18848, 2024.
[43] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning. arXiv preprint 23X5v:2305.05644, 2023.
[44] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre- trained language models. In Annual Meeting of the Association of Computational Linguistics
2023, 2023.
[45] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
13

---

Appendix
We provide an outline of how we organize the appendix in below:
Implementation details of our experiments:
Appendix A Implementation details for the experiments shown in TigureTrelatee the
zero-shot test loss of naive FedIT with LoRA rank of 1, 8, and 200, including an "extreme heavy tail" scenario.
Appendix E General experimental details, including hyperparameter settings, cross-task splitter details, and the choice of layers for applying LoRA.
Appendix L Provides details about the computational infrastructure and cost associated with the experiments mentioned in the document.
Algorithm, proof and analysis of FlexLoRA:
Appendix B The pseudocode for Fle'LoRA's algorithm in a federated learning context,
detailing the process from initialization, client sampling, local updates, server updates.
Appendix Proof of Theorem 1, providing mathematical derivation of the bounds C
generalization error under specific assumptions.
Appendix DJ Analysis of efficiency improvements from LoRA under different ranks, de- tailing the trainable parameters, memory costs, and efficiency gains compared with full finetuning.
More experimental results including:
Appendix F. Statistics of the Table 2 including 1. Percentage of improvement w/ FlexLoRA vs w/o FlexLoRAA 2. significant test results comparing FlexLoRA with its baselines across different resource distribution types.
Appendix G More results on task-specific performance improvements of global models trained with FleXLoRA, particularly natural language tasks.
Appendix Additional results on the effect of SVD, including the distribution of singular values and the approximation error ratio.
Appendix Results on the efficacy of Fle result under a mixture of task setting.
Appendix Discusses the single client's 's performance impact of different LoRA ranks, showcasing how performance improves with higher LoRA ranks across various NLP tasks.
A Implementation of Figure
For the experiments in Figure 0 we plotted the zero-shot test loss of naive FedIT with LoRA rank equal to 1, 8, and 200. For the experiment of FedIT with FlexLoRA, we adopt an "extreme heavy tail" scenario where all the clients have LoRA rank of 200 except one with LoRA rank of 8. All the hyperparameters and FL experiment settings are the same as the experiments in Table 2
B
The Algorithm of FoA
We summarize the Pseudocode of Foe in Algorithms and 2
C Proof of Theorem 1
Let TH denote the function space with its elements parameterized by Wg and the distance metric is defined as:
A(W
w
(1)
Bn,yon n
Er
Σ
(r(w a.
Σ
14

---

Algorithm FlexLoRA for Federated Learning
Input: T. g' Ag. Hec. 8, Egg bec where is total communication rounds and C the set of all FL clients
for t 1,... , do
Server samples clients C sampled from C, sends By and Ay to e C
for client E C' in parallel do
Update local LoRA module weight:
(Bf, Ke)e LOCALUPDATE ((0) Athee)
Server updates global model:
/* Implement FlexLoRA Here */
( Aghec SERVERUPDAATE (i, hec)
return 0, BBz AyEEE
Algorithm 2 FlexLoRA Server Update
Input: {Bi, Ai, r", ," hec, 8, Tg. fr' hec, where 7 average constant for client i depending on size of its local dataset
6 Aggregation with Heterogeneous LoRA Ranks:
Initialize global weight W, 00
for E C% do
Compose local client LoRA weight: W = sBi
Weighted Average client weight:
Wa = Wg W
Decompose global LoRA weight:
/* Only do once */
U,E.V = SVD SVD (Wg) Distribute Global Weight: for i C do
Compute each client LoRA weight based on their resource limitation:
Ul: r [[[ r r/)//
9
Ag
V|: r',"""
return
BBy A AiEc
With the inequalities introduced by Assumptions [
anndd
we have:
A(W; -W:')
[ C((W (W (.,
Σ s(w ;x, 2)
< <L Lyllww - Ww
L, L.II//VV r²) SVD(W r'))l
=LyZN||VV(WW, 8, - W - -SVD(W' g.
+ +W' WW, - W'll
< ILINN|/VD(W, S, ri) -Wall
-LLLI|/VDD g: r') WWIll
L,L,llW, W'll
LL (20* + +IW W
Let the parameter solution space of W,, denote as We can get an e-covering in metric A(W,. g.. w' 9. if we select a covering in the parameter space with I|W, - W'll equal to I,L, -20 Therefore, covering number of [H||), denoted as B(e, IHll) is: log (B(e, H(el)) = (k (klog( ( {L
15

---

According to previous studies
1
35
7
there exists N and Ñ O log
Ble.Bll)) 8
ne?
O ce log( RL
log
Ly
c
D Efficiency Improvement from LoRA under Different Ranks
We calculate the trainable parameters and active memory cost for LoRA under different ranks compared with full finetuning in experiment setting, summarized in Table Although the size of LoRA weights is small compared with the pretrained model because LoRA only tunes a small proportion of the trainable parameters, LoRA tuning is still much more efficient than full parameter finetuning, which highlights the necessity to scale LoRA rank size on clients with diverse computation resources. Table 6 illustrates the empirical time consumption for tuning client with LoRA and full finetuning.
Table 5: Trainable parameters and memory cost for different LoRA configurations, and the corre- sponding efficiency improvement compared with full finetuning.
LoRA Config
% Trainable Parameters
Memory Cost
Improvement
Type
r 8 on all layers
0.12
34.71GB 38.70GB
833.3x 1.83x
Type
r 30 on all layers
2.46 %
40.7x 1.64x
r 30 on atten layer,
Type
8.22 %
42.57GB
12.2x 1.49x
r 200 on FFN layer
Type
r = 200 on layers
12.22 %
46.54GB
8.2x 1.37x
Full finetuning
N/A
100 %
63.62GB
1.0x 1.0x
Table 6: Speedup of LoRA tuning for each communication round.
LoRA Rank (homo)
Average time per client
Speedup
r=8
min 17 sec
2.31x
r=200
min 41 sec
1.76x
Full finetuning
min 57 see
1.0x
Table 7: Illustration of the original data structure for tasks in natural instruction dataset
39
Task Type Task
Cause Effect Classification
lask828_copp htc_pcpccpseeeeeeeeeeeee ltt_e_aueeeeeee__lleeeeee clasettcaassiiicatioon
Definition
In this task your given two statements. You must judge whether the second sentence the cause or effect of first one. Label the instances "cause" or "effect" based on your judgment. The sentences are separated by newline character.
Positive Example
Input: The women met coffee. They wanted cutch up with each other. Output: cause
Explanation: women met for coffee because they wanted catch up with each other.
Negative
Input: My hody cast a shadow over the grass. The sun was rising.
Example
Output: effect
Explanation: rising the sun isn't an effect of casting shadow over grass.
Instance
Input: The woman tolerated her friend's difficult behavior. The woman knew her friend was going through hard time.
Valid Output: ["cause")
E Implementation Details
E.1
Hyperparameter Setting
All experiments are conducted with a client participation rate 0.05 each round, and with an early stopping mechanism that terminates training if the validation loss does not improve over consecutive FL. rounds. These values are adopted to better reflect the operational challenges inherent in world cross-device scenarios, which often involve limited device responsiveness and restricted
16

---

Table 8: The example of prompt template for training, adapting from Table
Instruction
In this task your given two statements. You must judge whether the second sentence the cause or effect of the first one. Label the instances as "cause" or "effect" based on your judgment. Thee sentences are separuted by newline character.
Input
The woman tolerated her friend's 's difficult behavior. The woman knew her friend was going through hard time.
Output
"cause"
Category
Cause Effect Classification
training duration.
Besides, the batch size is set as 4 via searching from a range of (2,4,16). The maximum token length is 512. All the experiments are repeated with random seeds and we report the standard deviations. More details about the computational infrastructure and cost for our experiments are in Appendix [
For the sparse fine-tuning stage of SLoRA, we set its sparsity corresponding with the resource cost of the client's 's LoRA configuration shown in Table For example, if the client is assigned with Type LoRA configuration, we will generate mask with a sparsity of 0.12%. For HETLORA, following the original paper, we adopt 0.99 as the decay factor for rank pruning and search the regularization factor from a range of (5e-2, 5e-3, 5e-4). For both the experiments integrated with and without Lo,,, we grid search their learning rates from range of (5e-2, 5e-3, 5e-4) for FedAvg, and (5e-4, le-4, 5e-5, le-5) for FedIT and SLoRA, both accompanied with linear scheduler which decays from the initial learning rate to O.
E.2 Cross-Task Splitter Details
Echoing findings from 39 on model performance saturation with few data instances, we randomly sample 10% of each task's 's data to scale experiments. For all the datasets we used, data for each client is partitioned into training, validation, and testing sets in ratio of 8:1:1. In the initial setup of the natural instruction dataset, each task is accompanied by its definition, along with a positive example and negative example. To adapt this dataset for model fine-tuning through instruction tuning, we transform structure so that the task definition serves as the direct instruction for each data instance. This restructuring is illustrated by comparing the original and modified data formats in the natural instruction dataset, as shown in Table Jnd now explicitly aligned with its instructional context, facilitating more straightforward and effective fine-tuning process.
and Table g This adjustment ensures that each instance is
E.3 Choice of Layers for applying LoRA
We explore the influence of insertion layers of LoRA on the performance of the fine-tuning. While [16] and [ add LoRA on all the attention layers, 43 adds LoRA on all the linear layers, i.e., the attention layers and Feed-Forward Network layers. We experimented with both approaches to adding LoRA. From Table we demonstrate that adding LoRA to both the attention layers and Feed-Forward Network layers boosts generalization performance and adopt this setting our experiments.
Table 9: Impact of choosing different layers apply LoRA module. The results zero-shot Rouge-L score of the global model. For the experiment that uses FOA for aggregation, the client resource distribution is uniform.
Experiment
Atten Layers
All layers
FedIT
0.5701 0.5751
0.6195 0.6211
w/ FlexLoRA
F Relative Improvement for Table 2 and Significant Test
Table T0 presents the percentage improvement for Table providing more straightforward overview of the enhancement of Fle
17

---

Table 10: Percentage of improvement of FedAvg, FedIT, and SLoRA incorporating with FlexLoRA compared with their respective configurations without FlexLoRA, as shown in Table
FedAvg
FedIT
SLoRA
Avg
Uniform
2.72%
0.08%
1.22%
1.33%
Heavy-Taie Tail-LigiiLh
1.52%
0.97%
0.65%
1.05%
Normal 2.20%
1.17%
2.78%
2.05%
Heavy-Tail- Tail-Strong
2.12%
1.24%
3.09%
2.15%
Avg
2.14%
0.86%
1.94%
1.65%
The statistical test results presented in Table I1 provide quantitative comparison between the performance FlexLoRA across different FL basclines to be integrated, namely FedAvg, FedIT, and SLoRA. The p-values obtained from the statistical tests are used to determine whether the observed differences in performance are statistically significant.
Table 11: Significant test results (in p-values) between FlcxLoRA and its FL baselines be integrated.
FedAvg
FedIT
SLoRA
Uniform
0.064
0.358
0.044
Heavy- Saily--ail--aae Tigh-Light
0.168
0.018
0.175
Heavy-Tail Tail-Strong
0.016
<0.001
0.060
Normal
0.029
0.006
0.044
As shown in the table, Fle consistently outperforms its homogeneous baselines under various distribution types, indicated by p-values smaller than the conventional significance level of 0.05 in several cases. Notably, under the 'Heavy Tail Light' distribution, FlexLoRA achieves statistically significant improvements in FedIT (p=0.018) scenarios. Highly significant improvements are also observed the "Heavy Tail Strong" " and "normal" distributions, especially in the FedIT setting, where the p-values smaller than 0.001 and 0.006 respectively, strongly suggesting that FlexLoRA's performance enhancement not due to random chance.
Overall, the results indicate that FlexLoRA is robust approach that can yield performance im- provements federated tuning environments for LLMs, especially scenarios characterized by resource-heterogeneous distribution of client resources.
G Natural Language Task Performance
Figure illustrates the performance of global models on natural language tasks when trained with Flc in the FedIT setting. In line with results from the FedAvg setting, global models that leverage heterogencous ranks in the FedIT setting also perform better on tasks involving logical rela- tionships between sentences, such overlap extraction, textual entailment, cause-effect classification, and dialogue act recognition. However, the global models underperform in word-level analysis tasks, such word analogy and keyword tagging. We hypothesize that this underperformance in the FedIT setting is due to the local clients being trained with the Adam optimizer rather than the conventional SGD optimizer. The top decoding block of the LLaMA transformer, which processes word-level information, likely overfits specific information because of the momentum component inherent to the Adam optimizer.
H
More Results for Effect of SVD
Figure also verifies the empirical information loss from SVD by illustrating the singular value distribution and error ratio of kproj weights. kpray weights shares similar trends with Sproj weights regarding the singular value distribution and error ratio.
Similar to Figure 6(b)
18

---

Answerability Classification Coreference Resolution
Question Rewriting
Keyword Tagging
Calegory
24
Narmal
51
00
24
4
4.77
033
1.D
0.0
-0.5
Improvement %
Implovements %
Improvensett %
Improvament
Overiap Extraction
Text
Word Analogy
Title Generation
Uniiomm
-1.50
Category
High
8ks
os
Peary-tll 2r0ng
0.25
Improvement
Improvement
Textual Entallment
Cause Effect Classification Dialogue Act Recognition Grammar Error Correction
Galagory
Normal
75
F2
66
Heary-ti 5t0gg
0.0
0.5
1.5
Improvement %
Improvement
Improvement
Improvement
Figure 7: Task-specifie improvements achieved by FlexLoRA in comparison to the homogeneous rank implementation FedIT, across different resource distribution settings.
k proj under FlexLoRA w/ heavy-tall-strong distribution
10
block singular
block singular value
0.8
block 14 singular value
10"
..... block error ratio
singular value
block error ratio
0.6
block
ratio
0.4
10
0.2
10
0.0
100
300
400
500
700
800
Order
Figure 8: Distribution singular values and the approximation error ratio between the top i-th singular value approximated kproj weights and the actual full-rank keroj weights. The red cross denotes the average error for weights with rank 30 of kproj across blocks 1, 8, and 14.
I Experiment Result for Mixed Task Heterogeneity Scenario
Table 12 provides the result of standard FedAvg LexxLoRA-interran FedAvg under uniform and heavy-tail teavy-tail-light resource distributions. The foundation model includes DataJucier(1.3B) and LLaMA 3 (8B).
Table 12: Results of homogeneous LoRA configurations versus FlexLoRA under FedAvg methods. The experiments are conducted on both Datalucier (1.3B) and LLaMA-3(8B)) models on Dolly-15K.
DataJucier (1.3B)
LLaMA-3 (8B)
Homo Rank
55.34 + 0.59
64.26 + 0.80
Uniform
58.56 + 1.35
64.93 0.33
Heavy-tail-light
58.39 + 1.26
64.58 + 0.62
19

---

J Single Client's Performance under Different Rank
Table 13 below shows the single client performance under a small rank of 8 and large rank of 200. This data shows that performance improves with higher LoRA ranks uniformly regardless of the tasks assigned to each client, motivating us allocate the highest feasible rank given client's resource budget. Figure 9 illustrates all the clients' performance under rank 8 and rank 200.
Table 13: examples of single-client performance under FedIT with homo rank 8 and homo rank 200 distribution in a single round.
rank=8
rank=200
Task Type
0.8977
0.9157
Translation
0.9084
0.9272
Translation
0.549
0.5749
Program Execution
0.5031
0.5301
Program Execution
0,4414
0,4507
Sentiment Analysis
0,5173
0.5328
Fact Verification
0.6995
0.7354
Program Execution
0.4801
0.5166
Question Rewriting
Rougel, scare for single clients coffrratore
Rouged for wwg.iiiiee.oo
Figure 9: Performance FedIT with homo rank 8 vs homo rank 200 across all the clients.
K FleXLoRA with Centralized LoRA Adaptive Methodologies
Besides directly applying LoRA to each clients, also investigate the efficacy of existing dynamic LoRA methods in central learning. We adapts concept of ReLoRA A (44 into our setting and explore its utility when combining with FlexLoRA. ReLoRA trains LoRA for several epochs, uploads the LoRA weight to the pretrained weights, then initializes a new LoRA module and trains based on the new frozen weights, repeating this process for several LoRA modules. We create a baseline by incorporating the concept of ReLoRA, uploading aggregated LoRA weights to the pretrained models after several communication rounds.
However, we observed a slower convergence speed than the regular FlexLoRA. We show in Table 14 below that incorporating the step of "uploading LoRA weights" does lead to better performance. This suggests that observations and improvements seen in centralized dynamic LoRA methods may not translate directly federated learning due to the unique challenges posed by heterogeneity in FL, which underlines the importance of tailored solutions like FlexLoRA for effectively managing heterogeneity in FL environments.
Table 14: Results with/without incorporating ReLoRA into either regular FedAvg or FlexLoRA.
Homo Rank (Baseline)
Uniform
Table 2 Result
56.53
58.07
+ReLoRA Result
53.9
56.56
20

---

L Computational Resources and Infrastructure Report
In our study, we employ the LLaMA-1.3B from Data-Juicer P as the foundational architecture for our FlexLoRA framework, which consists of approximately 1.35 billion parameters. During our experiments, the DataJuicer-1.38 model itself is kept frozen, and the tunable parameter size for each client within our federated learning framework are determined by the size of the LoRA module, with specific configurations detailed Table Our experiments are conducted on a cluster equipped with 16 NVIDIA A100 GPUs, each with 40GB or 80GB of memory. The total GPU hours for running all the experiments is approximately 1200 GPU hours. the experiments are implemented using PyTorch package with version 2.0.0 and Huggingface's s Transformers package with version 4.3..0.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are supported by both theoretical and empirical results in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Conclusion section, the paper mentions limitation and future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
The paper provides full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper has disclosed all information needed to reproduce the main experi- mental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
Justification:
We have released the code and data used in our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ctttps://nips.cc//
ALic//uides/Cooeeuubee ottpie/ss/oooeuubbbbssssooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips. whips.cccpubb...... hetpllligggie//eeeeeeeoee CodeSummi asionoolicyl for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Implementation details can be founded both Section and Appendix E.1 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The error bars shown in the tables of main results, and significance analysis is in Appendix(F
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is in Appendix L
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics httpp////eeerrrsssccpppp///ce/es
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification; This paper focuses more on proposing a new techniques in Federated Learning setting and does not bring societal impacts or concerns.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not found such risks in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors properly credit the assets used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |paper swi thcode. com/ conasdatsess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We create anonymous code repository where pertinent details of code
stored.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "qavu4v7fcs", "reviewer": "Reviewer_XizQ", "summary": "This paper proposes a LoRA-based federated fine-tuning algorithm called FlexLoRA. FlexLoRA keeps full-size LoRA modules on the server, and decomposes them into heterogeneous rank LoRA modules according to the client task and capacity. The heterogeneous local LoRA modules will be transferred back to full-size and then aggregated on the server. FlexLoRA can make LoRA-based federated fine-tuning more flexible and easier to be implemented.", "questions": "please see the weaknesses", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper uses SVD to solve the heterogeneous LoRA rank problem in FL, which looks valid and straightforward. The explanation of the proposed method is clear and easy to follow.\n(2) The paper provides sufficient scalability study. With slightly increased cost per round, FlexLoRA can convergence much faster than baselines.", "weaknesses": "(1)The main concern is the privacy issue of FlexLoRA. Since the server needs to preset all the local LoRA ranks, the server needs to get access to some client information such as the amount of data, computation resources, ..., etc. Will this threaten the privacy of local data?\n(2)The experiment shows results of language model around 1B. Why larger models (e.g., llama) and benchmarks (e.g., mmlu, gsm8k) are not shown in the body part. There are some results in the appendix but may need more detailed discussion in the experiment section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and their recognition of our work's contribution to addressing heterogeneity problems in FL! In response to the raised comments, we summarize the two main questions that the reviewer pointed out in the weakness section. \n\n---\n\n## Response to W.1\n>  Since the server needs to preset all the local LoRA ranks, the server needs to get access to some client information such as the amount of data, computation resources, ..., etc. Will this threaten the privacy of local data?\n\nThank you for your comments. The information disclosed from the clients to the server is minimal: **Compared with the vanilla FL method, only the preset LoRA ranks are required to be shared from the clients.** We discuss the influence of disclosure of LoRA rank as the following:\n\n- [**Computation resource**] The LoRA rank is primarily correlated with the computation resources of each client. If the information about computation resources is sensitive to the client, the client can choose to employ encryption techniques or rely on trusted third parties for secure communication. \n- [**Amount of data**] In Federated Learning, the amount of data is usually shared with the server for calculating weight aggregation. **FlexLoRA does not spill more information than vanilla FL methods.** If the amount of data becomes sensitive information in some FL training settings, we can encrypt such information with existing approaches like secure aggregation [1]. \n\n---\n\n## Response to W.2\n> The experiment shows results of language model around 1B. Why larger models (e.g., llama) and benchmarks (e.g., mmlu, gsm8k) are not shown in the body part.\n\nWe answer the reviewer's question about the experiment setting in the following two aspects:\n\n1. **Model Size:**\n   - We selected a 1B base model for the main experiments to accommodate the real-world cross-device FL setting where thousands of devices are present and a majority of clients are edge devices with limited computation power. \n   - Following the reviewer's suggestion on accounting for FlexLoRA's performance under larger models, we present the results of llama 3(8B) on Dolly-15K dataset with 1000 clients a supplement experiment in Table 1. The client sample rate is 0.01 and we use FedIT as the base framework. We can see that FlexLoRA achieves better performance under uniform distribution and heavy tail light distribution compared with regular FedIT, which assigns homogeneous rank to each client's base model.\n\n2. **Benchmark:**\n   - For the results under Natural Instructions (Table 2 in the paper), we followed the standard of the original Natural Instruction paper [2], which involves using Rouge-L scores and task-specific evaluations for evaluating model performance. [2] has shown that **Rouge-L is positively correlated with the human evaluation result and is a valid indicator of model language processing performance**.\n\n| **Resource Dist** | **RougeL** |\n| --- | --- |\n| Homo Rank (FedIT) | 68.88 |\n| Uniform | 69.49 |\n| Heavy tail light | 69.29 |\n\nTable 1: FlexLoRA's performance on llama-3 (8B) under 1000 clients. \n\n---\n\n## Mentioned Refs\n\n[1] \"Practical secure aggregation for federated learning on user-held data.\" Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017.\n\n[2] \"Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.\" arXiv preprint arXiv:2204.07705 (2022).\n\n---\n\n## Closing Remark\n\nWe truly appreciate your valuable comments, which have contributed to the improvement of our paper. We hope that our responses demonstrate our commitment to addressing your comments and encourage you to favorably consider accepting the paper. We will ensure that the new discussion and results are incorporated into the final version. Thank you once again for your support!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHE4uCRjnN", "reviewer": "Reviewer_18q4", "summary": "This paper proposes FlexLora to tackle FL's heterogeneous resources and data distribution problem. FlexLora allows for dynamic adjustment of local Lora ranks by employing SVD for weight distribution, improving the global model’s generalization ability.", "questions": "Please refer to weakness.", "limitations": "The authors have discussed the limitations of the model and client scale. For the small client scale, claiming a cross-device setting is not so rigorous. Cross-silo setting might be more suitable.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The experiments are comprehensive. The experiments verified that the proposed method is efficient and improves the generalization.\n2.  The paper is well-written and easy to follow.", "weaknesses": "1.  This paper claims to follow a cross-device setting, but cross-device setting usually involves thousands of clients, while the client number in the experiment of this paper is much smaller. If the method can only be verified in a small-scale experiment, cross-device setting is not a very rigorous statement. Cross-silo setting might be more suitable for this paper.\n2.  One of the main contributions of this paper is allowing for dynamic adjustment of local Lora ranks in FL. Some dynamic/flexible Lora methods already exist in central learning. I wonder whether these methods can also be applied to tackle the heterogeneity of downstream tasks in FL. Related discussions or experiments are suggested here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's acknowledge of our contribution! We respond the reviewer's suggestions in the following two replies. \n\n## Response to W.1\n> This paper claims to follow a cross-device setting, but cross-device setting usually involves thousands of clients, while the client number in the experiment of this paper is much smaller. ... Cross-silo setting might be more suitable for this paper.\n\n- While it is true that our experiments do not reach the scale of millions of clients for cross-device simulation, our study involves over 1,600 clients, which is comparable with existing cross-device FL work. For example, [1] implements cross-device FL benchmarks on the Landmarks dataset with 1,262 clients, and [2] uses more than 700 clients in their cross-device FL setting. \n- Besides, to account for FlexLoRA's scalability on FL settings with larger client pool, we present Figure 5 in our paper, which demonstrates that **the efficacy of our method grows with the client numbers**, showcasing its potential for even larger cross-device federated learning (FL) settings.\n\n---\n## Response to W.2 \n> One of the main contributions of this paper is allowing for dynamic adjustment of local Lora ranks in FL. Some dynamic/flexible Lora methods already exist in central learning. I wonder whether these methods can also be applied to tackle the heterogeneity of downstream tasks in FL. \n\n- We thank the reviewer for their insightful comment. While dynamic/flexible LoRA methods do exist in centralized learning, their direct applicability to FL settings, especially in handling the heterogeneity of downstream tasks, is not guaranteed. \n- Per your suggestion, we newly conducted preliminary experiments by adapting concepts from one of the centralized dynamic LoRA works, ReLoRA, [3] to federated settings. ReLoRA trains LoRA for several epochs, uploads the LoRA weight to the pretrained weights, then initializes a new LoRA module and trains based on the new frozen weights, repeating this process for several LoRA modules. We create a baseline by incorporating the concept of ReLoRA, uploading aggregated LoRA weights to the pretrained models after several communication rounds. \n- However, we observed a slower convergence speed than the regular FlexLoRA. We show in Table 1 below that incorporating the step of \"uploading LoRA weights\" does not lead to better performance. This suggests that **observations and improvements seen in centralized dynamic LoRA methods may not translate directly to federated learning due to the unique challenges posed by heterogeneity in FL**, which underlines the importance of tailored solutions like FlexLoRA for effectively managing heterogeneity in FL environments.\n\n\n|  | Homo Rank (Baseline) | Uniform |\n| --- | --- | --- |\n| Table 2 Result (from paper) | 56.53(from paper) | 58.07(from paper) |\n| +ReLoRA Result | 53.9 | 56.56 |\n\nTable 1: Results with/without incorporating ReLoRA into either regular FedAvg or FlexLoRA. \n\n---\n## Mentioned Refs\n\n[1] \"Motley: Benchmarking heterogeneity and personalization in federated learning.\" arXiv preprint arXiv:2206.09262 (2022).\n\n[2] \"Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] \"Relora: High-rank training through low-rank updates.\" The Twelfth International Conference on Learning Representations. 2023.\n\n--- \n## Closing Remark\n\nWe hope our responses can address your comments. Again, we would like to thank you for your  support of our work and detailed comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N1YXMdWGoa", "reviewer": "Reviewer_V4JD", "summary": "Under the framework of federated fine-tuning large model, this paper proposes a simple and effective LoRA aggregation method with different ranks, which mainly focuses on the problem of client resource heterogeneity. Specifically, the full-size LoRA is first obtained, and then after normal aggregation, it is decomposed into a matrix BA with the corresponding rank of each client by SVD. Extensive experiments demonstrate the effectiveness of the proposed approach when client resources are inconsistent.", "questions": "As shown in the weaknesses section, we won't add anything more here.", "limitations": "This paper lacks a detailed discussion of the limitations and shortcomings of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Different rank-based LoRA is directly matched to client resource inconsistency. The method of first multiplying into full-size LoRA and then decomposing into different ranks based on SVD is simple and effective, and has certain rationality.\n2. The information loss of the parameters after SVD decomposition compared with the original global parameters is given in Section 3.4, and this method is generally reasonable.", "weaknesses": "1. Does it increase the communication cost to convert matrix BA to full-size LoRA before uploading?\n2. It seems that the configurations of ranks in Table 1 are not very diverse. Can the proposed method adapt to very diverse and more flexible configurations of ranks?\n3. Different ranks seem to be specific to client resource heterogeneity, but do not seem to be particularly optimized for task heterogeneity?\n4. While there are performance comparisons with the very related HETLoRA which is used to solve the resource heterogeneity problem, should comparison results with the rest of the state of the art FL methods also be given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's positive score and insightful feedback on our paper! In the following replies, we address all the reviewer's comments point-by-point.\n\n## Response to W.1\n\n> Does it increase the communication cost to convert matrix BA to full-size LoRA before uploading?\n\n- The current design of our method **does not introduce more communication costs** due to the conversion of matrix BA to full-size LoRA. The local clients directly send their trained B and A matrices to the server, which then handles the full-size computation. \n- **This design is a tradeoff between local and global resource availability.** Typically, in cross-device FL scenarios, local clients are numerous but have limited resources, whereas the server is a device with ample computation power. Therefore, by offloading the full-size computation to the server, we avoid burdening the local clients with the communication cost of transmitting full-size BA matrices.\n---\n## Response to W.2\n\n> It seems that the configurations of ranks in Table 1 are not very diverse. Can the proposed method adapt to very diverse and more flexible configurations of ranks?\n\n- **Our method can indeed scale to more diverse and flexible configurations of LoRA ranks.** Our current design utilizes a few typical ranks to validate our method, and the assignment of rank also follows previous works: our baseline FedIT[1] uses rank 8 in their experiment and [2] experiments on both rank 30 and rank 200 in their investigation on LoRA performance.\n- Following the reviewer's suggestion, we conduct supplementary experiments with a more diverse set of rank configurations, similar to those used in HETLORA, to demonstrate FlexLoRA’s adaptability to a broader range of scenarios. We select rank from the following set: $r^i \\in \\{1,5,10,20,50,100,150,200\\} $uniformly (i.e, each rank is selected with the same probability). We conducted experiments on Dolly 15K with the same experiment setting as in section 4.6: by partitioning data into 200 clients with Dirichlet distribution $\\alpha=0.5$, and the testing the resulted DataJucier-1.3B model.  From Table 1 below, we can observe that **under a more diverse rank setting, FlexLoRA is still able to maintain its superior performance than regular FedAvg.**\n\n|  | RougeL |\n| --- | --- |\n| Homo Rank | 55.34 (from paper Table 12) |\n| Uniform | 58.56 (from paper Table 12) |\n| Uniform (more diverse rank) | 58.81 |\n\nTable 1: Results of FlexLoRA under a more diverse rank choice. \n\n---\n## Response to W.3\n> Different ranks seem to be specific to client resource heterogeneity, but do not seem to be particularly optimized for task heterogeneity?\n\n- We'd like to point out that **using task characteristics to define ranks is not always feasible**. Each client may not have a single task, and determining ranks based on the client’s data distribution could be complex, potentially requiring additional training phases. Although the data partition in our main result in Table 2 is split by task, such a data partition approach is only for mimicking an extreme case of data heterogeneity, and in the real world we anticipate more task overlaps among clients. \n\n- To find a rank suitable for a client's local distribution, one may need to consider prune rank on different weights or layers to optimize its performance [3, 4], which introduces numerous computations. In this case, designing ranks for different tasks is not as meaningful as scaling ranks based on computation resources. \n\n- Our approach, FlexLoRA, was designed for a realistic scenario with both resource and task heterogeneity. **Although ranks are directly scaled by client resources, our method aims to address both resource and task heterogeneity.**\n\n---\n## Response to W.4\n> While there are performance comparisons with the very related HETLoRA which is used to solve the resource heterogeneity problem, should comparison results with the rest of the state of the art FL methods also be given.\n\n- While we directly compared FlexLoRA with HETLORA, we also referenced SLoRA(2023) as another method addressing LoRA heterogeneity. We did not compare SLoRA with FlexLoRA as pair-wisely as HETLORA, because *SLoRA is used as one of our FL frameworks that can be incorporated into FlexLoRA*. We'd like to mention one of FlexLoRA’s strengths lies in its **ability to integrate with various existing methods rather than competing directly.**\n- Following the reviewer's suggestion on adding more relevant works, we newly conduct experiments to compare FlexLoRA with FedRA (2024), another SOTA method employs a random allocation strategy to handle resource heterogeneity. In Table 2 below, we can observe that FlexLoRA is more suitable than FedRA for fine-tuning LLMs in cross-device FL settings. \n\n|  | Table 12 Result (from paper) | FedRA |\n| --- | --- | --- |\n| Homo Rank | 55.34  | 53.29 |\n| Heavy Tail Light | 58.39  | 56.37 |\n\nTable 2: Result from FlexLoRA vs FedRA\n\n---\n## Mentioned Refs\n\n[1] \"Towards building the federatedGPT: Federated instruction tuning.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n[2] \"Towards a unified view of parameter-efficient transfer learning.\" arXiv preprint arXiv:2110.04366 (2021).\n\n[3] \"Efficient personalized federated learning via sparse model-adaptation.\" International Conference on Machine Learning. PMLR, 2023\n\n[4] \"Think locally, act globally: Federated learning with local and global representations.\" arXiv preprint arXiv:2001.01523 (2020)\n\n---\n## Closing Remark\n\nWe sincerely hope that our responses have adequately addressed your comments. If you feel they have, we would be grateful if you could consider adjusting the evaluation of our manuscript accordingly. Thanks again for your valuable suggestions! We look forward to any further guidance you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "69QlzAlBDV", "reviewer": "Reviewer_ug2H", "summary": "The use of LoRA in FL is challenged by the heterogeneity of downstream tasks and available resources among clients. Traditional FL methods often use the smallest viable LoRA rank for all clients for aggregation compatibility, which makes it hard to capture the full diversity of client contributions and fully utilize ample client resources. To fully leverage heterogeneous client resources for enhancing the global model's generalization ability, this paper proposes FlexLoRA, which enables the mixture of diverse LoRA weights across individual clients to account for local client resources and task differences by allocating the highest feasible rank given a client's resource budget to ensure all clients contribute effectively regardless of resource capability, with heterogeneous aggregation and redistribution of weights through SVD. Extensive experiments and theoretical analysis verify the effectiveness and scalability of the proposed FlexLoRA.", "questions": "1.\tThe method described in Section 3 omits too many preliminaries and algorithmic details, which is reader-unfriendly.", "limitations": "1.\tThe proposed approach essentially fails into the area of federated fine-tuning of LLMs with parameter-efficient fine-tuning (PEFT) techniques. However, this paper does not discuss any limitations that may be imposed by PEFT especially LoRA used in this paper, e.g., PEFT methods sacrifice performance due to the parameter update is limited to a much lower-dimensional subspace, especially when data across clients are non-IID.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tExploring heterogeneous rather than homogeneous LoRA ranks across clients to fully utilize their resources is an imperative research direction.\n2.\tNumerous experiments and theoretical analyses are illustrated to support the effectiveness of the proposed method.", "weaknesses": "1.\tNumerous typos remain in this paper, including but not limited to: 1) \"${Client_j}$\" should be corrected to \"${Client&nbsp;j}$\" in Figure 2; 2) \"Effect\" should be corrected to \"effect\" in line 249; 3) \"r=200 2\" should be corrected to \"r=200\" in Table 6.\n2.\tIn contrast to HETLORA [8], the proposed FlexLoRA needs to perform SVD instead of redistributing it directly before redistributing the aggregated global LoRA weight to clients. This seems to be because it provides better initialization for the low-rank matrices B and A. However, the authors do not mention this point, and this practice has already been presented in FeDeRA [42], so this may not be considered one of this paper's contributions. \n3.\tSome relevant work in this field is missing, including but not limited to: \n[1] Liping Yi, Han Yu, Gang Wang, Xiaoguang Liu, Xiaoxiao Li. FedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning. arXiv preprint arXiv:2310.13283, 2023.\n[2] Shangchao Su, Bin Li, Xiangyang Xue. FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients. arXiv preprint arXiv:2310.13283, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive score and valuable feedback from the reviewer! We'd like to respond to the reviewer's questions and comments on the following aspects:\n\n- Paper presentation comments: W1, Question, Limitation\n- SVD contribution: W2\n- Additional related work: W3\n\n---\n## Response to Paper Presentation Comments (W1, Q1 & L1)\n\nWe deeply thank the reviewer's time for reading our paper and providing suggestions on typos and paper presentation. We will fix the typo issues that the reviewer mentioned in our next version. For section 3, we present our FlexLoRA algorithm pseudo-code and theorem 1 proof in the Appendix B and C sections due to page limit. We will consider moving the algorithm pseudo-code and any relevant details from the Appendix to the main content to make the paper more reader-friendly if our paper is selected and allowed for one additional page. We'd also like to include more discussions regarding the potential performance sacrifice of PEFT methods, as shown in relevant works such as [1]. \n\n## Response to W2 (SVD contribution)\n\n> SVD seems to be because it provides better initialization for the low-rank matrices B and A. However, the authors do not mention this point, and this practice has already been presented in FeDeRA \n\n- [**Algorithm Difference**] We'd like to clarify that **our method differs from FeDeRA**. FeDeRA's approach involves the following steps:\n\n1. `SVD Decomposition`: Decomposing the model's pretrained weight, $W_0$, using SVD: $SVD(W_0)=U\\Sigma V^T$\n\n2. `LoRA Initialization`: Initializing LoRA with the top r singular values: $B=U[:r, :] \\sqrt{\\Sigma[:r]}, A=\\sqrt{\\Sigma[:r]}V^T[:, :r]$\n\n3. `Freezing Weights`: Using the difference between pretrained weights and LoRA as frozen weights:$\\text{freeze} (W_0-BA)$\n\n4. `Federated Averaging`: Apart from the initialization, the method follows standard federated averaging with LoRA.\n\nThis is very different from FlexLoRA, as **FlexLoRA uses SVD for aggregations while FeDeRA uses SVD for their LoRA weights initialization**. \n\n- [**Relationship between SVD and initialization**]: There exists research indicating that the initialization of LoRA can impact its performance (e.g., SLoRA[2]). FeDeRA's improved performance is attributed to better initialization, where the initial LoRA weights contain information from the pre-trained weights, not due to the use of SVD. In contrast, our paper does not claim *SVD as a major contribution*. Our key contribution lies in **enabling clients to scale ranks based on local resources, thereby enhancing performance, and SVD is only utilized for scaling rank to local clients and does not contribute to improving model performance.** In our setting, we have demonstrated that when all clients use the same rank, aggregation with SVD is equivalent to FedAvg, underscoring that the contribution of model performance is not from SVD but an increase in rank.\n\n---\n## Response to W3 (Additional Related Works)\n\n> Some relevant work in this field is missing, including: pFedLoRA, FedRA\n\nWe thank the reviewer for providing the relevant work.  These works will be cited and discussed in our final version to provide a more comprehensive context. Here is an overview comparison between the work that the reviewer suggests and FlexLoRA:\n\n1. FedRA: FedRA uses a random allocation strategy for federated tuning to leverage heterogeneous clients. Compared with FedRA which randomly selects several layers for LoRA tuning on each client, FlexLoRA tunes all the layers with LoRA.\n\n2.  pFedLoRA: pFedLoRA mainly addresses personalization FL with LoRA, which is a different aspect of federated learning. pFedLoRA utilizes both global LoRA and local LoRA weights to boost the local performance of each client, which provides a solution in the personalization area, while our method focuses on an orthogonal optimization perspective: scaling rank adjustment based on local resources. Both approaches have the potential to be combined together to further improve the utility in FL. \n\nBesides, following the reviewer's suggestion to add more relevant works, we newly conduct experiments and would like to provide the result of our supplement experiment on comparing FlexLoRA with FedRA (Table 1 below). From it we can observe that **FlexLoRA is more suitable than FedRA for fine-tuning LLMs in cross-device FL settings**. \n\n|  | Table 12 Result (from paper) | FedRA |\n| --- | --- | --- |\n| Homo Rank | 55.34 | 53.29 |\n| Heavy Tail Light | 58.39 | 56.37 |\n\nTable 1: Result from FlexLoRA vs FedRA\n\n---\n## Mentioned Refs\n[1] \"Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment.\" arXiv, 2023\n\n[2] \"SLoRA: Federated parameter efficient fine-tuning of language models.\" arXiv preprint arXiv:2308.06522 (2023).\n\n---\n## Closing Remark\n\nWe believe the paper has been further improved with your helpful comments, and hope that these responses will convince you to lean more toward acceptance of the paper. We will include the new discussion and results in our final version. Thank you once again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This study introduces FlexLoRA which is a federated finetune algorithm based on LoRA. It operates by maintaining a full LoRA module on a central server while decomposing it via SVD as heterogeneous rank LoRA which are tailored to specific client tasks and their capabilities. The authors have conducted thorough experiments and provided a detailed theoretical analysis to demonstrate the proposed method's efficacy and scalability.

All reviewers are positive to the paper and agree that the paper makes significant contributions including it is well-motivated and straightforward and provides comprehensive experiments.

The main concerns raised by reviewers including privacy issues of FlexLoRA, its overhead and lack of a few baselines were properly addressed during the discussion phase.

Overall, I agree with the reviewers and recommend its acceptance
Decision: Accept (poster)