FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
Authors: Ziyao Wang
Zheyu Shen
Yexiao He
Guoheng Sun
Hongyi Wang
Lingjuan Lyu
Ang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Large Language Models
LoRA
Primary area: infrastructure

Abstract: The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ computation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with constrained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. This approach led to mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, we first highlight the mathematical incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLoRA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through a novel stacking-based aggregation method. Our approach is noise-free and seamlessly supports heterogeneous LoRAs. Extensive experiments demonstrate FLoRA's superior performance in both homogeneous and heterogeneous settings, surpassing state-of-the-art methods. We envision this work as a milestone for efficient, privacy-preserving, and accurate federated fine-tuning of LLMs.

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations
Ziyao Wang', Zheyu Shen', Yexiao He', Guoheng Sun' Hongyi Wang
Lingjuan Lyu', Ang Li
1. University of Maryland, College Park
2. Rutgers University 3. GenBio.ai 4. Sony Reasearch {ziyaow, zyshen, yexiaohe, ghsun, angliece) @umd. edu, hongyi wang. 001 fri@rrrgers.... edu, Lingjuaalvamile@@malll
Abstract
The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ com- putation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with con- strained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. These approaches led mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, first highlight the mathemati- cal incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLORA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through novel stacking-basee aggregation method. Our approach is noise-fre free and seamlessly supports hetero- geneous LoRA A adapters. Extensive experiments demonstrate FL.R A superior performance in both homogeneous and heterogeneous settings, surpassing state- of-the-art methods. We envision this work as milestone for efficient, privacy- preserving, and accurate federated fine-tuning of LLMs. Our code is available at htttp://ggithie www. -TP- wttp:///ATT/110//eeeeeeeeeeeeaaa
1 Introduction
The Large Language Models (LLMs) have shown remarkable performance on various tasks, such as chatbots [1] virtual assistants 4 search engines [1], and healthcare (00 [8]. However, adapting pre- trained LLMS (e.g., Llama 2 to downstream tasks requires tremendous computation resources to fine-tune all the model parameters. To mitigate this issue, a variety of parameter-efficient fine-tuning (PEFT) methods have been proposed. One of the most widely used PEFT methods low-rank adaptation (LoRA) [10]. As shown in the top of Figure [ LoRA adds a parallel branch of trainable adapters A and B to compute the model update A W. where the ranks of A and are much smaller than the pre-trained model parameter W. When applying LoRA for fine-tuning, only A and B are updated while the entire Wi is frozen, thereby significantly reducing the GPU memory consumption.
Fine-tuning LLMs requires ample data for adaptation to specific downstream tasks B Often, this data dispersed across multitude of devices, raising privacy concerns. For instance, aggre- gating medical data from hospitals for centralized LLM fine-tuning poses significant challenges. Consequently, to facilitate fine-tuning without compromising private data, federated learning (FL) becomes essential, enabling LLM fine-tuning across distributed clients while preserving data pri- vacy [16] 27 33 24 In this work, we focus on federated fine-tuning, enabling distributed clients to collaboratively fine-tune LLMs for adaptation downstream tasks while preserving data privacy.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prior work, FedIT. proposed federated fine-tuning method [28], integrating LoRA with FedAvg [T]] In each FL round of FedIT, clients fine-tune LoRA modules using their local data and then send the fine-tuned modules to the server. The server averages all the local L modules to obtain global LoRA. Since only the weights of the LoRA modules are fine-tuned and communicated, FedIT effectively reduces both computation and communication costs.
However, FedIT faces two key issues. First, the naive averaging of local LoRA modules in FedIT introduces noise to the global model update. Specifically, FedIT averages local A and B independently, which introduces mathematical errors to the global LoRA. In short,
The cause of aggregation noise:
BA ΣA
FedIT
mathematically correct
We will elaborate on this issue in Section 2 with theoretical analysis. Such an inaccurate aggregation
will hinder convergence, leading to higher fine-tuning costs. Second, due to the heterogeneous data
distribution [2]]2] and heterogeneous hardware resources, clients need to adapt LoRA ranks [30]
according to the system and data heterogeneity. However, FedIT cannot aggregate local LoRAs with heterogeneous ranks.
In this work, we present FLORA, an eegregation-noise-ooo tuning method that supports heterogeneous LoRAs. Specifically, as shown in Figure 2 we propose to stack the local LoRA mod- ules A₁ and BN separately construct the global LoRA modules and B, where As; and B, denote the corresponding LoRA modules on the k-th client. This stack- ing method is theoretically proven to be accurate for the aggregation of local LoRA modules (Section 3.1 Additionally, it
Centralized LoRA: =
federated fine-
Pre-trained Parameter
Parameter
X
AW
Federated LoRA: =W=?
Introducing Noise
FedIT:
Averaiing
AW
Homogeneeus Only
FLoRA:
Noise-free Aggregation
Stacking
AW-
Heterogencous Aggregation
can naturally accommodate heterogeneous LoRA settings (Section 3.2), since stacking does not require the local LoRA modules to have identical ranks across clients. The noise-free aggregation of FLORA accel- erates convergence, which will in turn im- prove the overall computation and commu- nication efficiency of federated fine-tuning. Furthermore, FLORA can effectively cater to heterogeneous data and computational resources across clients, where heterogeneous ranks are applied. Our key contributions are summarized as follows:
Figure 1: The overview LoRA, FedIT, and our FLORA. The top row shows how LoRA updates the model in centralized fine-tuning. The middle and bot- rows show the global model updating strategies in FedIT and our FLORA respectively.
We propose FLORA, federated fine-tuning algorithm based on LoRA that can perform noise-free aggregation of local LoRA modules. Theoretical analysis shows that FLORA eliminates the meaningless intermediate term in the global model update, leading to faster convergence and improved performance.
The proposed stacking mechanism for aggregating LoRA modules supports heterogeneous LoRA ranks across clients, accommodating data and system heterogeneity in realistic settings. This encourages the broader participation of clients with heterogeneous data and resources in federated tinettuning, We use FLORA fine-tune LLaMA, Llama2 21 and TinyLlama 29 four benchmarks for two downstream tasks. Results show that FLORA surpasses state-of-the methods for both homogeneous and heterogeneous settings.
2 Preliminaries
Fine-tuning LLMs with LoRA. the update of the target module:
LoRA 10 uses two decomposed low-rank matrices to represent
W
W
AWW
W
BA,
(I)

---

I2
FedIT
The aggregation of local update
FlooR((ours)
Figure 2:
Module stacking in FLORA noise-free aggregation for LoRA, while the module averaging in FedIT cannot accurately aggregate the local updates.
where W E R and W E _ denote the pre-trained and fine-tuned parameters of target modules (e.g, attention modules), respectively. A and B are low-rank decomposition of A W, where A 1 ,B E Rmxr, such that AW = BA with the identical dimensions as W and W' The rank of LoRA, denoted by r, typically significantly smaller than and n. leading to dramatic parameter reduction of A AW During the fine-tuning phase, LoRA optimizes matrices A and instead of directly updating W, thus achieving substantial savings in GPU memory usage. For example, in the context of the LLaMA-7B [1], the original dimension of attention modules is W E TR 4096 1096 setting the LoRA rank
reduces the decomposed matrices A E TRL 4096 and B R-4090 16 This approach decreases the number of trainable parameters to merely 0.78% of the entire parameter of the pre-trained model, offering significant GPU memory footprint reduction.
The most widely used FL algorithm, i.e. FedAvg [16]. aggregates all the local model updates by weighted averaging to update the global model in each communication round:
FedIT: Averaging Homogeneous LoRA.
W = W + E PLAW, W + AW
A=1
where W and W denote the global model parameters before and after communication round. AWa à represents the local model update from the k-th client, with yı being the corresponding scaling factor that is typically weighted by the local data size, and AWW AW represents the global model update.
directly integrates FedAvg with LoRA enable federated fine-tuning, where each client fine-tunes LoRA modules with homogeneous rank. Specifically, the clients download the pre-trained LLM from the server, locally initialize and fine-tune the LoRA modules, and then send the updated LoRA modules to the server. The server updates the global LoRA modules A and B by independently applying weighted averaging across all local modules A and B
FedIT 28
K
A =
A
B =
km
1=0
This aggregation of FedIT is almost the same as FedAvg except that only the LoRA modules are trained and communicated. However, such a naive aggregation introduces additional issues for federated
First, each single module A or B is not the model update, and only BA represents the model update. Thus, averaging A4 and B₃ independently gradients will introduce noises to the global model update. Here we use simple example to explain how the noise is generated, and we assume that two clients are applying FedIT to perform federated fine-tuning. In communication round, the two clients train Ag. Bn and A,, B, respectively. The local model updates Wo and A WW are the product corresponding LoRA modules:
compute the aggregated
AW Ba {0, 1}.
According to Equation the expected global model update A WW can be obtained by weighted averaging A W. and A AWr
AW Pa A. + pAW, Ao + PiB
(5)
However, according to Equation FedIT aggregates A and B independently:
AW = BA Bp + p. Ap + P1A1)
(6)
1 B.An Pi + PoPI (BrA B,Aa).
The global model update in Equation 6 differs from the expected one in Equation 2 mainly due to the underlined intermediate term that is obtained by the cross-product of LoRA modules from

---

different clients. This intermediate term introduces unexpected noise in the model aggregation. As the number of clients increases, this noisy term becomes much larger than the real global updates, significantly slowing down the fine-tuning progress. In addition, FedIT applies the scaling factor pk to both A and B;, resulting in a p coefficient for the local model update A Was k. exacerbating the error in LoRA aggregation. As Figure illlstrrats,,, the averaging algorithm in FedI'T' is an inaccurate aggregation method, leading to slower convergence and higher computation costs.
The other deficiency of FedIT is that it cannot support aggregation on heterogeneous LoRA modules. The local data in FL may exhibit significant heterogeneity across clients [T 12). If client configures a higher rank than the actual one required by the local data complexity, this may result in overfitting. Conversely, the rank too small, it may lack the necessary generalization capacity to effectively learn from the local dataset (Figure 4. Moreover, the heterogencous computational resource across clients also requires heterogeneous rank deployment, e.g., clients with smaller memory can only afford to train LoRA modules with smaller ranks. AdaLoRA [30] has been proposed to adapt LoRA ranks based on available computation resources. Therefore, deploying heterogeneous ranks across clients is pressing requirement for accommodation to data and system heterogeneity. However, according to Equation 3.
FedIT is only able to aggregate LoRA modules with the homogeneous rank.
3 Proposed Method: FLoRA
3.1 Stacking-baser Noise-free Aggregation
Motivated by the aforementioned problem, we propose novel aggregation mechanism that accurately computes global model update A by aggregating local LoRA modules and effectively supports the heterogeneous LoRA. According to matrix multiplication principles and the model update rule in LoRA (i.e., Equation []. the element at position (I, v) of the model update A WW is computed as the sum of the products of corresponding elements from the x-th column of B and the y-th row A:
8ay
Bgy bai;
where kay represents the element at position (I. in AW. ays. bui are the elements positions (v. i) and (x, a) in A and B, respectively. According to Equation 3.1 the model update in LoRA can be expressed as the sum of the products of the corresponding rows of and the columns of B.
To illustrate this concept further, let us consider a simplified example where the dimensions of LoRA modules are given by
_2x3 and B 3 As described in Equation 8) A and B can be decomposed to two sub-matrices with rank = 1, and the product of A and B then are computed as the sum of the products of two respective sub-matrices:
Touo: bo1
buo
buu
BA
b1o, b11
Apo, a10: 020
b10
(avo: 010. Q20]
+
b11 b21
aoo1. a11,
a21.]
(8)
b2o: b21
001, a11. Q21
b20
To address the aggregation challenge from an alternative perspective, let consider the scenario where we have multiple pairs of LoRA modules, Ag. Bg, optimized by the clients. Each pair satisfies the dimensions A E Rr and B₃ Rmxr,, Similar to Equation 8 the sum of the products of these module pairs is the product of the stacked modules, i.e., Σk B, = BA, where B represents the stacking of all B₁ modules aligned through dimension m and A is the stacking of all A₁ aligned through dimension n. Figure visually illustrates this concept, where the orange, green, and blue rectangles symbolize Ag. Bg- and their respective products. The aggregation of three products mirrors the product of the stacked B and A from all B2 and A pairs trained by clients. This mechanism demonstrates that, in the context of federated fine-tuning, we can achieve noise-free aggregation of local updates by simply stacking the local LoRA modules. This process also avoids transmitting the full model parameters, thus reducing communication costs.
To facilitate our discussion, we introduce the stacking operation symbolized by "ø" to denote the module aggregation as depicted in Figure This operation is mathematically defined as:
A Ag A A2, B Bp ① B ① B2,
(9)
A E RR A E
R f2)
B& E I B (rx

---

Devices
Heterogeneous LoRA
Matrix B
Matrix A
Local Full Updates
Aa E RooXn
Client
Be
A Wo
Bo E Rmxro
Ai E R*
Client
Bi E mm
B1
AW1
Ag E Reaxn B₂ E gmxr
Client
B2
A
AW2
Upload Local LoRA Matrices
Stacking all along m
Stacking all along "
Get the Accurate
Aggregation
A ndxe
A
Update Local Models
Server
BE (
Bo
B
B
*
A1
A2
Figure 3: FLORA workflow. The local LoRA modules are initialized and optimized each round, and stacked by the server obtain the global LoRA modules. The global modules are then sent back to clients to update local models.
Equation "œ" indicates that for A, each subsequent module is vertically stacked below the preceding one, whereas for B, each module is horizontally stacked the right of the one before it.
We can now formalize our conclusion regarding the aggregation of LoRA modules. The sum of the products of K LoRA module pairs is equivalent to the product of their stacked matrices:
B&A: (Bo ... BR)(An e @ Ak)
(10)
0
This foundational principle will guide the design of FLORA, as it allows for the efficient and effective aggregation of local updates without the transmission entire model parameters.
3.2 FLORA: Stacking-based Federated Fine-tuning for Heterogeneous LoRA
The stacking-based aggregation facilitates not only the accurate aggregation of LoRA modules but also inherently supports the heterogeneous LoRA ranks. This approach imposes constraints the ranks of each local LoRA module as long as each client fine-tunes the same pre-trained model, i.e., they share the same dimension m and
By employing the stacking- based aggregation mechanism, we introduce FLORA, an approach designed to facilitate federated fine-tuning of LLMs with heterogeneous LoRA. Let concrete example to illustrate the key steps of applying FLORA, where K' heterogeneous clients are involved in fine-tuning an LLM, and the pre-trained parameters denoted by W.
Initialization. The server first disseminates the pre-trained model parameters W to all K clients. Then, the clients initialize their local LoRA modules based the complexity of local data and available local resources. The adaptation of LoRA ranks is beyond the scope of this paper, but existing work like AdaLoRA 30 can facilitate the rank adjustment.
Local Fine-tuning. Following initialization, the clients train their local LoRA modules with the local data for several iterations. Then, the clients send the local LoRA modules back the server. Note that the clients initialize local LoRA modules each round before local fine-tuning.
Stacking-based LoRA Aggregation. Upon receiving the heterogeneous LoRA modules from participating clients, the server proceeds to aggregate them by stacking all B₁ and A₁ according to Equation 10 resulting in the global A R R xx)xn and B e I Tx). The aggregation

---

process of FLORA can be described as follows:
- Po 2 P1 0 APKAK B B + B1 @ BK
(11)
A R B& RO
A AER ru)xn B T
where Pk represents the scaling factor for each local update, determined by the relative size of the local data to the global data:
len(Di (D))
Pk: len( Ten Dz)
(12)
Note that the scaling factor pk should be only applied one of A and B3 to avoid squaring the factor in the final model update BA. This method ensures a noise-free fase-frre aggregation mechanism as described in Equation 10
Update Local Models. After each round of noise-free aggregation, the server redistributes the updated global LoRA modules and back to the clients. The clients then proceed to update the local models using BA and continue the fine-tuning. Using the stacking approach, the dimensions of updated global LoRA modules A and B are larger than those of FedIT, potentially leading to larger communication overhead in each round. However, empirical observations indicate that federated fine-tuning typically requires only a limited number of communication rounds achieve satisfactory results, as detailed in Section 4 In addition, it is important to note that the LoRA modules A and B constitute small fraction of the overall size of the pre-trained model, which is distributed to clients during the initialization phase. Thus, the additional communication overhcad of the stacking approach is negligible and does not significantly impact the efficiency federated fine-tuning.
Experiments
The key features of FLORA A are (i) noise-free aggregation and (ii) support for heterogeneous LoRA modules. In this section, we verify these key features across various LLM fine-tuning tasks. We first study the performance of FLoRA and compare it against FedIT under homogeneous settings to demonstrate the advantages of noise-free aggregation (28). Then, we examine performance in synthetic heterogeneous setup and compare FLoRAwine vanilla zero-padding method. Finally, we conduct ablation studies on the scaling factor, the heterogeneity of LoRA ranks, and the extra communication overhead of FLoRA.
4.1 Experiment Setup
Models, Datasets and Experiment Settings. We employ three Llama-based models with different scales in our experiments: TinyLlama with 1.1 billion parameters (2)] and the billion parameter
versions of Llama [21] and Llama2 (22). evaluating FLORA across different model capacities. Following the configurations the original LoRA paper [10), the LoRA modules are applied the
self-attention layers only.
We use the Databricks-dole sitabricks-dolly-1555 15kk (28 instruction dataset, Alpaca dataset [19. and Wizard dataset [4 for the question-ant inestion-answering aawweeinnwwggg (QA) task, and Wizard and ShareGPT for the chat assistant task. We evaluate the federated fine-tuned models on MMLU [8] for the QA task and MT-bench (32) for the chat assistant task, respectively. We sample 10 clients uniformly at random following the non-IID
setting in FedIT 28 The other experimental configurations are elaborated in Appendix A
Baselines. We compare FLORA with four baselines. (1) FedIT: It is the SOTA federated fine- tuning method (8 that integrates LoRA with FedAvg. We only apply FedIT to homogeneous, LoRA
experiments as it does not support heterogeneous LoRA. (2) Zero-Padding: It is an approach that enables FedIT to support heterogeneous LoRA 3 It extends all the heterogeneous local ranks maximum rank among the clients and pads their remaining parts by 0. (3) Centralized Fine-tuning: we compare FLORA A with centralized LoRA with the same hyperparameters and configurations. (4) Standalone: the client fine-tunes the pre-trained model locally without federations.

---

Table 1: Comparison of FLORA with baselines MMLU and MT-bench. "Homo" represents the settings with homogeneous LoRA ranks, and "Heter" denotes those with heterogeneous' LoRA ranks.
Foundation
Fine-tuning algorithm
MMLU
MT-bench
Strategy
model
Dolly
Alpaca
Wizard
Wizard
ShareGPT
Centralized
LoRA
27.99
28.03
29.13
2.34
2.79
FedIT
16.35
30.02
42.51
2.92
2.55
Homo
TinyL.ama
FLORA
30.80
31.92
43.87
3.13
2.77
Zero-Padding
13.76
29.56
40.79
1.56
1.29
Heter
FLORA
18.45
29.69
41.48
3.14
2.71
Centralized
LoRA
35.91
29.18
31.68
4.38
3.99
FedIT
29.67
29,41
33.43
3.07
3.73
Homo
Llama
FLORA
30.99
29.85
34.26
4.21
3.93
Zero- Pero-Padding
26.46
7.97
26.98
3.51
3.26
Heter
FLORA
28.50
29.54
27.91
4.14
3.64
4.2 Experiment Results
Homogeneous LoRA. We first evaluate the performance of FLORA with homogeneous LoRA. Specifically, all the clients share the identical LoRA rank of 16. As Table d FL.RRA A achieves consistently better performance than FedI' across all the evaluated models and tasks. This is evident in the MT-bench scores for both TinyLlama and Llama models, where FLORA's 's performance exceeds that of FedIT by at least 0.2. A notable example is the MT-bench score for the Llama model fine-tuned with Wizard dataset, where FLORA scores 4.21, surpassing FedIT's 3.07. On the MMLU test set, FLORA outperforms FedIT in all the settings. For example, considering the TinyLlama model fine-tuned with Dolly, FLORA nearly doubles the accuracy achieved by FedIT. While FedIT occasionally matches the performance of FLORA, as observed with the Alpaca dataset on MMLU, the performance gap marginal. Interestingly. in several scenarios, the performance of FLORA not only outpaces FedIT but also exceeds the performance achieved by the centralized fine-tuning. This phenomenon, observed in the TinyLlama model fine-tuned with the Alpaca and Wizard datasets, suggests that the smaller data volume on clients for federated fine-tuning may help mitigate overfitting, thereby enhancing model generalization. The experiment results of the Llama2 model are presented in Appendix AA, which reveal the same trend as that in Tiny and Llama. The consistent observations across three models demonstrate that FLoRA consistently outperforms FedIT in the homogeneous LoRA setting.
Global Model
64
32
16
MT--bench Score
3.16
4.14
3.64
3.62
2.65
4.04
3.58
3.083.063.00
3.97
3.51
2.44
2.35
3.683.67
(n) LinyLl Wizard
(b) TinyLlumu ShareGPT
(c) Llama Wizard
(d) Llama ShureGPT
Figure 4: Standalone experiment results. The red bars represent the global model performance and the blue bars represent the local model performance with varying LoRA ranks.
Compared with FedIT, a distinctive strength of FLORA lies in its inherent capability to accommodate heterogeneous LoRA configurations. In the heterogeneous LoRA settings, we apply varied local LoRA ranks, i.e., [64, 32, 16. 16, 8. 8, 4, 4, 4, 4]. to 10 clients, simulating realistic scenario where clients have heterogeneous computational resources. As Table Table (illusttrate FLORA A not only adapts to heterogeneous ranks without performance degradation but also maintains consistency with the results observed in most homogeneous settings. This contrasts sharply with the performance of FedIT, where the application of zero- zero-padding significantly degrades its performance on MMLU and MT-bench. It reveals that zero-padding exacerbates FedIT's inherent noise issues in the aggregation process, posing significant challenges in managing fine-tuning perfor- mance. For example, by applying the zero-padding method, the MMLU accuracy of Ilama model fine-tuned with Alpaca dataset dramatically drops 7.97%. The results demonstrate that FLORA
Heterogeneous LoRA.
7

---

not only accommodates heterogeneous LoRA ranks effectively but also sustains robust training performance compared to baseline methods. It efficiently facilitates the participation of devices with varied computational capacities in heterogeneous federated fine-tuning tasks. Additionally, FLORA can be seamlessly integrated with A E0, which dynamically adjusts the LoRA rank on the clients, the results are presented in Appendix A
The Impact of Scaling Factor: The scaling factor, denoted Pk: in Equation 12 plays piv- otal role in the efficacy of FL [23]. We conduct experiments investigating how varying scaling factors influence the performance of FLORA A. Given that the default scaling factor is set 0.1 for all clients, assuming 10 clients with equal local dataset sizes as per Equation [2] we ex- plored the effects of alternative scaling factors, namely 0.01, 0.05, and 0.2. The results are sum- marized in Figure 5 The results do not reveal clear pattern or optimal scaling factor for fed- erated fine-tuning across different settings. The efficacy of specific scaling factor appears to be contingent upon the dataset, task, and model in use. For example, when fine-tuning TinyL- lama the Dolly dataset, lower scaling fac- tor of 0.01 yields the highest accuracy, signif- icantly outperforming the 0.1 and 0.2 scaling factors. Conversely, the model fine-tumed on Wizard dataset demonstrates preference for higher scaling factor of 0.2, achieving the best performance, whereas the lowest scaling factor of 0.01 was the least effective. In the case of the Llama model, larger scaling factors consistently facilitated better fine-tuning performance. Ap- plying FLORA to Dolly and Alpaca shows the optimal performance with a scaling factor of 0.2. These observations suggest that the choice of an appropriate scaling factor is highly dependent on specific datasets and model characteristics, underscoring the necessity for a tailored approach in federated fine-tuning.
Doily
Alboca
40
Wizard
20
26
0.05
0.2
TinyL MMLUU
(b) llama MMLUU
3.5
3.8
Wizard ShareGPT
3.45
2.9
3.1
2.6
2.75
0.2
(c) TimyLlama- MT-bench
(d) Llama MT-bench
Figure 5: The impact of the scaling factor on FLORA. The x-axis is the scaling factor, and the y-axis represents the MMLU accuracy for (a)-(b) and the MT-bench score for (c)-(d). The results of Llama2 are in Appendix A
The Impact of Heterogeneous LoRA Ranks. Although the above results demonstrate FLORA effectively enables the federated fine-tuning with heterogeneous LoRA, it is worth further investigating how the federated fine-tuning improves the local models with various ranks. Motivated by this, we evaluate MT-bench scores for local models with LoRA ranks of 64, 32, 16, 8, and 4, presenting the results in Figure 4 Global model scores are shown in red bars, while local models are in blue, with deeper shades indicating higher ranks. The results show that the global model outperforms all local models, except for a case with the Tinyl model fine-tuned on the Wizard dataset, where the client with rank 32 slightly exceeds the global model. This demonstrates FL.RAY 's ability to synthesize knowledge from diverse clients effectively.
Regarding the LoRA rank's impact, rank of 8 consistently yields strong performance across various models and datasets. However, performance diverges at extreme ranks; for instance, the TinyLlama model fine-tuned smaller ranks, but the Llama model with the rank of 64 excels the counterparts with smaller ranks. This also demonstrates the heterogeneous rank deployment across clients is realistic setting, These observations suggest a potential positive correlation between optimal LoRA rank and model capacity, motivating further exploration in future research.
Wizzard with the LoRA rank of 64 underperforms the ones with
5 Discussion
The Communication Overhead of FLORA. As discussed in Section 3 the server needs to send global LoRA modules to the clients in FLORA, potentially raising concerns about increased commu-
8

---

nication overhead. To quantify this, we compare the communicated parameters of full fine- tie-ttnnngg FedIT, and FLORA over three communication rounds.
As Figure 6 shows, although FLORA transmits slightly more parameters than FedIT, it still sig- nificantly reduces the overhead compared to full fine-tuning. This due to the fact that the primary
Full Fine-tuning
FedIT
FLoRA
Federated fine-tuning algorithms
communication load in federated fine-tuning, es- pecially with large models, is the initial full model parameter transmission. Subsequent rounds pri- marily involve smaller updates (e.g., LoRA ma- trices). Thus, even though FLoRA introduces additional communication for these updates, the overall impact on total communication costs re- mains marginal, making it comparable FedlT"s costs. Despite the minor communication increase compared to FedIT, FLORA enhances fine-tuning effectiveness and supports heterogeneous LoRA ranks, making it preferable solution in federated fine-tuning.
1.000
0.177
0.215
Ratio of communicated parameter # to full fine-tuning - communication rounds
Figure 6:
The ratio of communicated parameter numbers to full fine-tuning.
The Privacy Preservation of FLORA. The requirement of FLORA stack the LoRA modules uploaded by all clients introduces a potential privacy concern, as malicious clients might infer the LoRA matrices of other clients through the global LoRA modules sent from the server. To address this issue, we split all the local LoRA modules into sub-modules with rank=1 and then stack the sub-modules together in random order. This approach prevents malicious clients from recovering the local LoRA modules from other clients. In addition, FLORA is also compatible with standard privacy mechanisms such as encryption ( and differential privacy (DP) [25]. aligning it with the privacy-preserving nature of FL.
Related Work
Parameter-efficiet Fine-tuning of LLMs. Parameter-efficient fine-tuning (PEFT) aims reduce the number of trainable parameters. BitFit (26) fine-tunes only the biases while achieving similar accuracy with full fine-tuning. Other works such as 9 and [17] apply transfer learning that adds pre-trained adapter layers between transformer blocks. LoRA [10] adopts the product of two low-rank matrices to represent the gradient in full fine-tuning, which achieves memory-efficient fine-tuning, AdaLoRA (30) optimizes LoRA by adaptively allocating the parameter budget, which enhances the ficxibility of LoRA. There are also many works regarding optimizing LoRA in various aspects (5] 15).
Federated Fine-tuning of L.MMs. Federated fine-tuning aims to extract knowledge from multiple on- device datasets while preserving data privacy. FedIT (28) leverages the FL framework for fine-tuning LLMs. uses LoRA as the local fine-tuning strategy. However, concerns related the deficiency in supporting heterogeneous LoRA limit its utilization. (3] tries to solve this problem by zero- zero- pero-padding the local LoRA modules. However, this padding process causes additional computing overhead. Besides, it separately averages A and B modules, introducing noise to the global model.
Conclusion
In this work, identified the limitations in current federated fine-tuning methods (e.g., FedIT), and the challenges of applying federated fine-tuning in realistic settings, i.e., the heterogeneous LoRA ranks across clients. To overcome these practical challenges and broaden the applicability of federated fine-tuning, we introduced FLORA to enable the accurate aggregation on heterogeneous LoRA modules using the proposed stack-based LoRA aggregation mechanism. Our extensive experiments demonstrate that FL.ORA outperforms the SOTA method in both homogeneous and heterogeneous LoRA settings. Moreover, our inspiring results provide valuable insights for future research in the federated fine-tuning of large language models in lightweight and accurate manner.

---

References
[1] Desirée Bill and Theodor Eriksson. Fine-tuning a IIm using reinforcement learning from human feedback for a therapy chatbot application, 2023,
[2] Yansong Chang, Kai Zhang, Junqing Gong, and Haifeng Qian. Privacy-preserve federated learning via functional encryption, revisited. IEEE Transactions on Information Forensics and Security, 18:1885 18:1855-1869, 2023.
[3] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous lora for federated tne-tuning of on-device foundation models. In International Workshop on Federated Learning the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.
[4] Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next-generation intelligent assistants leveraging Ilm techniques. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5792-5793, 2023.
[5] Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-efficieng of adapters. arXiv preprint arXiv:2210.04288, 2022.
[6] Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Mera: Merging pretrained adapters for few-shot learning. arXiv preprint arXiv:2308. 15982, 2023.
[7] Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. Shed: Shapley-based automated dataset refinement for instruction fine-tuning. arXiv preprint arXiv:2405.007700 000055 2024.
[8] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference on Learning Representations, 2020.
[9] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[10] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 206602200..09855 2021.
[11] Dominique Kelly, Yimin Chen, Sarah Cornwell, Nicole Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L Rubin. Bing chat: The future of search engines? Proceedings of the Association for Information Science and Technology, 60(1):1007-1009, 2023.
[12] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
[13] Fei Liu, Yiming Yao, Ping Guo, Zhiyuan Yang. Xi Lin, Xialiang Tong, Mingxuan Yuan, Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. A systematic survey on large language models for algorithm design. arXiv preprint arXiv:2410. 14716. 2024.
[14] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
[15] Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. arXiv preprint arXiv:2407. 1046, 2024.
[16] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efio -fffcciinn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
10

---

[17] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint MoXiv:2005.002445 2020.
[18] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-1188 2023.
[19] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunoii B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. https://gt http://ggcuub..ccomttttttu--aaa
alacaa 2023.
[20] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting, Large language models in medicine. Nature medicine, 29(8):1930-
19011999-9440,,,,0 2023.
[21] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[22] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2tti::3307.092888, 2023.
[23] Ziyao Wang, Jianyu Wang, and Ang Li. Fedhyper: A universal and robust learning rate scheduler for federated learning with hypergradient descent. arXiv preprint arXlv:2310003156, 2023.
[24] Rui Ye. Wenhao Wang. Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du Yanfeng Wang. and Siheng Chen. Openfedlim: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6137-6147, 2024.
[25] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi. Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXhv:2110.06500 2021.
[26] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10193 2021.
[27] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. survey on federated learning. Knowledge- -aaeed Systems, 216:106775, 2021.
[28) Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning. arXiv preprint arXiv:2305.05644 2023.
[29] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
[30] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffcient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[31] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv: 00582, 2018.
[32] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging auddeee- with mt- bench and chatbot arena, 2023.
[33] Weiming Zhuang, Jian Xu, Chen Chen, Jingtao Li, and Lingjuan Lyu. Coala: A practical and vision-centric federated learning platform. arXiv preprint arXiv:2407. 16560, 2024.
11

---

A Additional Experiments and Setup Details
A.1 Environments, Datasets and Metric
Computer Resources. We use a 236GB AMD EPYC 7763 64-Core Processor on Linux 18.18.0 to run the experiments. For LoRA fine-tuning all the models, we use 4 NVIDIA RTX A6000 GPUs.
Dolly dataset. The Dolly dataset an open-source dataset with 15k text samples generated by Databricks employees. The topics include brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization [28]
Alpaca dataset. The Alpaca dataset contains 52K inslructionfoolooiingg data used for fine-tuning the Alpaca model [9] This dataset is believed be diverse enough for fine-tuning LLMs.
Wizard dataset. The Wizard dataset we use is the training data of the WizardLM model. It includes 70k pairs of instructions and outputs. The Wizard dataset generally features more complex instructions compared to the other datasets. Its fine-tuning results are typically better, which has been confirmed by experiments, especially those evaluated by the MT--ench scores.
ShareGPT dataset. The ShareGPT dataset a collection of approximately 52.000 conversations scraped via the ShareGPT API. The conversations in ShareGPT include both user prompts and re- sponses from ChatGPT, our experiments, we split the conversation dataset into question -enstion-answering pairs.
MMLU test set. The MMLU dataset is widely used question-and-answer dataset in LLM fine- tuning. It has 14,024 questions in 57 different subjects, which can evaluate the logical reasoning capabilities of LLMs. We selected 1444 samples from the dataset for quick and comprehensive evaluation.
MT-bench evaluation. MT-bench is of challenging multi-turn open-ended questions for evaluating chat assistants [32] It evaluates the performance of LLMs by using the GPT-4 API to score the LLM-generated conversations. LLMs that behave more like GPT-4 will receive higher scores.
A.2 Hyperparameter Details
In all our experiments, the learning rate of fine-tuning set to 0.0003; the batch size 128 and the micro batch size is 16. Due to the large dataset and model sizes selected, federated fine-tuning consumes significant computational resources and time. Therefore, we opted for fewer fine-tuning rounds (even just one round) to ensure that could observe enough data. Additionally, the MMLU dataset is prone to overfitting on these large datasets, resulting decrease in accuracy, Therefore, fewer training rounds ensure the effectiveness of the observed phenomena. Table 2 shows the fine-tuning rounds and local epochs we selected.
Table 2:
The communication rounds and local epochs on each experiment setting. The Rounds column represents the number communication rounds and the Epochs column represents the number of local fine-tuning epochs in each round.
Foundation
Datasets
Rounds
Epochs
Dolly
Alpaca
3
Liaya
Wizard
3
ShareGPT
Dolly
3
3
Alpaca
3
Llama
Wizard
ShareGPT
Wizard
Llama2
ShareGPT
12

---

Table 3: The performance of FLORA + AdaLoRA. AdaLoRA can reduce the rank while preserving the fine-tuning effectiveness.
Foundation model
Fine-tuning algorithm
Sum of local ranks
MT-bench
FLORA
160
3.13
TinyLlama
FLORA+AdaL.RR,
120
3.14
FLORA
160
4.21
Llama
FLORA+AdaLORA
131
4.10
FLORA
160
4.17
Llama2
FLORA+AdaLoRA
140
4.25
Table 4: Compare FL.RA A with baselines in Llama2.
Fine-tuning algorithm
Strategy
Wizard
ShareGPT
Centralized
LoRA
4.24
3.99
FedIT FL.RA
4.03
3.87
Homo
4.22
3.96
Zero-Padding FL.ORA
4.01
3.70
Heter
4.17
3.91
A.3 Supplementary Experiment Results
Integrating FLORA with AdaLoRA All the observations about the impact of rank the model performance, despite being influenced by data heterogeneity, still manage to reveal the importance of selecting an appropriate LoRA rank for specific task. Thus, some algorithms such AdaLoRA [30] are designed to adaptively adjust the LoRA rank to optimize the model performance and save computational resources. With support for heterogeneous LoRA, we flexibly utilize AdaLoRA with adaptive LoRA ranks. We conducted corresponding experiments to demonstrate that we can Adal LoRA to further improve the efficiency of federated fine-tuning. We implement Adal.ok o on each client to adjust LoRA modules during local fine-tuning. The results are shown in Table 3 The "Sum of local ranks" column means the sum of all local LoRA rank values after fine-tuning. Since our FLORA A does not adjust the rank, its value is 160, the same as the initial value. On the other hand, AdaLoRA dynamically adjusts the rank to maximize training effectiveness and minimize rank values to save resources. From Table Table 3 we can see that AdaLoRA on TinyLlama and Llama reduced the sum of local ranks to 120 and 131 from 160 respectively. We further conclude that FLOR A AdaLoRA can further reduce the trainable parameter count while ensuring comparable or even improved performance compared to simply using LoRA on the clients. Our support for such rank adaptation further demonstrates the effectiveness and applicability of the FL.RAA A approach.
Experiment results of Llama2. Due to the inherently strong performance of Llama2, the im- provement in the QA dataset is not significant. Therefore, we fine-tuned Llama2 using the Wizard and ShareGPT datasets. Overall, Llama2 exhibits similar experimental results to Tinyllama and Llama. Table shows the comparison between FLORA and our baselines. In the homogeneous and heterogeneous settings, the MT-bench scores of Wizard and ShareGPT all surpass those in FedIT and Mero-Padding As for the impact of scaling factors in Figure / Llama2 has similar trend to the Llama-7b 7b model, in which higher scaling factors exhibit better fine-tuning performance.
B Convergence Analysis
In this section, we demonstrate the convergence of FLORA following the standard convergence analysis in [2]. The FedAvg algorithm exhibits convergence the global optimum at a rate of O(1/T) for non-III (independent and identically distributed) data under full client participation. This convergence is based on four assumptions mentioned in [2]:
13

---

4.25
Wizard ShareGPT
4.1
3.95
3.8
0.01
0.05
0.1
0.2
Figure 7: The impact of scaling factor on Llama2 model.
Assumption 1. Each local objective function is L- smooth, that is, for all x and Fa(x) s Fa(v) + (x V + 2 | y12
Assumption 2. Each local objective function is jr- strongly convex that is, for all and y, Fx(2)) 2 Fx(v) + (a v)" + $ || v
Assumption The variance of stochastic gradients in each client bounded: E V <<<))
Aw(" < of for k = I,.... K, where c is the subset of training data randomly sampled from k-th client.
Assumption 4. The expected squared norm of stochastic gradients is uniformly bounded: (l .5% (1 < G2 for all k = I....,K K and I,...,,, where ((1) is the subset of training data randomly sampled from k-th client.
For the convergence analysis of FLORA, we introduce an additional assumption 5 tailored to the specific dynamics LoRA fine-tuning and its relation to traditional SGD-based full fine-tuning:
Assumption 5. (Unbiased LoRA Gradient). The updates applied to LoRA modules by each client serve unbiased estimators of the gradient that would be directly computed the base model through SGD: ((+11) A (t+1) B A & V e()). Note that we define the model
parameter in t-th round by W(t).
Theorem 1. Based on Assumptions 1-5, choose k 4,
# 7 max E}. The local learning rate Ct ( Then, we can deduce that the expectation of the fine-tuning error in FLORA A can be bounded by:
2k M T+T + (L W
8(T) S
(13)
where g(T) is the fine-tuning enoor in T-th round. 5(T) and M are defined as follows:
8(2) = F[ (T-1) + ( A A)) F*,
K
(14)
M Σ P +
77T + 8 G
k=1
where T., og, and G are defined by the assumptions 1-4. I is defined F F for quantifying the degree non-iid. This theorem posits that as the number rounds T approaches infinity, the expectation of the fine-tuning error 5(T) converges to zero. In contrast, FedIT deviates from the FedAvg model updating rule as depicted in Equation 2 introducing non-gradient noises through averaging process. Therefore, fails achieve convergence the rate of O(I/T) While this deviation does not invalidate FedIT's utility in federated fine-tuning, it significantly impairs its convergence rate and overall effectiveness.
Pa
C Limitation
Our approach has the limitation that the server sends the stacked LoRA modules to the client, thereby increasing the communication costs. We discussed this limitation both theoretically and
14

---

experimentally in Section 3 and Section 4, respectively. We believe that the increase in communication overhead is acceptable under the premise of improving fine-tuning effectiveness and accelerating convergence. In addition, due to constraints on computational resources and time, we only utilized Ilama models in the experimental section. We aim to observe experimental phenomena of different types of LLM federated fine-tuning in future research and derive more general principles.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper has a clear abstract and an introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The Limitation section presented in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
All proofs are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental settings are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: The code is on GitHub. We already have a discussion with other researchers about the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips.cc/pp wwwppbblccgguiicuceccccccooooope wetggudddeeeoeeubbinnnnn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are reproducible with low errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information on GPUs and resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////uuurrr..cccaaaall/cctat
Answer: [Yes]
Justification: Conducted.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: It is only technical paper, which has social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All the models and datasets used are open-sourced.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Models and datasets are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No human-involved evaluations.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "cZOTwObvVh", "reviewer": "Reviewer_4FRM", "summary": "This paper focuses on developing a novel aggregation strategy for training LLMs in FL with LoRA. Specifically, they identify that separately aggregating two matrices of LoRA module is not identical to aggregating model updates. Therefore, they propose a stack based aggregation strategy that merge matrices before aggregating them.", "questions": "no", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tExperiments show that the proposed method is effective.\n2.\tClear presentation.", "weaknesses": "1.\tThe novelty is limited. Although the proposed method is effective, the contribution of the stack-based aggregation strategy may be too limited.\n\n2.\tThe motivation of the stack based aggregation is not clear. Since the computational resource of the server is usually sufficient, it is totally acceptable that recovering the local update and then aggregating them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's recognition of the effectiveness of our proposed method. Regarding the weaknesses raised, we address all the concerns in detail below.\n\n>W1: The novelty is limited. Although the proposed method is effective, the contribution of the stack-based aggregation strategy may be too limited.\n\nThe primary contribution of our paper is identifying a mathematically inaccurate aggregation noise present in nearly all existing LoRA-based federated fine-tuning algorithms. We then propose a novel and efficient stacking method to address this issue. Our paper introduces significant novelty by being the first to highlight the noise generated by the simplistic averaging process commonly used. Given that almost all current LoRA federated fine-tuning algorithms and codebases rely on this averaging method, our stacking algorithm offers a mathematically accurate and provably convergent solution. We believe this contribution will provide substantial value to the community.\n\n\n>W2: The motivation of the stack based aggregation is not clear. Since the computational resource of the server is usually sufficient, it is totally acceptable that recovering the local update and then aggregating them.\n\nRecovering the local updates and then aggregating them may be acceptable from a computational cost perspective. However, this approach necessitates the server sending back the full model parameter updates to the clients, rather than just the LoRA modules. As illustrated in Figure 6 of our paper, this method results in approximately five times the communication cost compared to our FLoRA approach, which is impractical. Therefore, we argue that recovering the local updates and then aggregating them is not a feasible solution due to the prohibitive communication overhead.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjIytN2B6e", "reviewer": "Reviewer_7eRV", "summary": "Previous methods using Low-Rank Adaptation (LoRA) for efficient federated fine-tuning may have led to mathematically inaccurate aggregation noise, reducing the effectiveness of fine-tuning and being unable to handle heterogeneous LoRA. The authors analyzed the mathematical inaccuracies in LoRA aggregation in existing federated fine-tuning methods and proposed a new stack-based aggregation method that supports federated fine-tuning across client machines with heterogeneous LoRA adapters. Extensive experiments demonstrate the superior performance of FLORA in homogeneous and heterogeneous environments.", "questions": "see the weaknesses above", "limitations": "see the weaknesses above", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple, effective, and easy to implement.\n2. The stacking mechanism for aggregating LoRA modules supports heterogeneous LoRA ranks across clients, which has broader application scenario.\n3. The experiments utilized various models and benchmarks, and the results validated their effectiveness.", "weaknesses": "1. Writing needs to be calibrated. The sentences in line 64 and line 70 are repeated.\n2.The discussion on accelerating convergence is not indexed in the main text (Appendix B) and lacks experimental validation.\n3. More discussion may be needed on the advantages of FLoRA compared to the implementation of 'the aggregation of local update' in Figure 2.\n4. The foundation models chosen for the experiment are all from the Llama series. Are there other types of foundation models that can be used for validation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's recognition of our proposed method's effectiveness and broad application scenarios. Regarding the weaknesses raised, we address all the concerns in detail below.\n\n>W1: Writing needs to be calibrated. The sentences in line 64 and line 70 are repeated. 2. The discussion on accelerating convergence is not indexed in the main text (Appendix B) and lacks experimental validation.\n\nWe thank the reviewer for pointing out errors in our writing and section arrangement. We will correct the repeated part and discuss more about the convergence in the final manuscript. For the experimental validation, we provide some data points here to prove that FLoRA converges faster than our baselines. We use the Wizard dataset to fine-tune a Llama-7b and test on MT-bench. The LoRA rank here we use is 8. We display the global model performance in the first 5 rounds. FLoRA converges earlier than FedIT (in 3rd round) and has higher accuracy.\n\n=================================\n\nEpochs  $\\quad$   0   $\\quad$   1   $\\quad$  $ $ 2   $\\quad$  $ $ 3    $\\quad$  $ $ 4   $ $ $\\quad$  5\n\n—-------------------------------------------------------\n\nFLoRA  $\\ $ $ $ 2.86 $ $ 3.79 $ $ 3.88 $ $ 3.90 $ $ 3.88 $ $ 3.83\n\nFedIT  $\\ $ $ $ $ $ 2.86 $ $  2.93 $ $ 3.12 $ $ 3.47 $ $ 3.66 $ $ 3.62\n\n=================================\n\n>W2: More discussion may be needed on the advantages of FLoRA compared to the implementation of 'the aggregation of local update' in Figure 2.\n\nFigure 2 in our paper illustrates the process where the server first recovers the local updates from the LoRA modules and then aggregates them. While this method might be acceptable if the server has ample computational resources, it significantly increases communication costs. Recovering the full gradient on the server necessitates sending back the full model parameter updates to the clients rather than just the LoRA modules. As shown in Figure 6 of our paper, this results in approximately five times more communication cost compared to our FLoRA approach, which is impractical in most scenarios. Therefore, our FLoRA method offers a substantial reduction in communication overhead, making it a more efficient solution than the traditional method of recovering and aggregating local updates on the server.\n\n\n>W3: The foundation models chosen for the experiment are all from the Llama series. Are there other types of foundation models that can be used for validation?\n\nWe thank the reviewer for the advice. Due to the limited time of the rebuttal, we are only able to conduct the experiments on homogeneous LoRA fine-tuning on Gemma models. Here are the results of the MT-bench evaluation:\n\n===========================================\n\nTraining Set    $\\quad$   $\\quad$    $\\quad$     Gemma-2b  $\\quad$    Gemma-7b\n\n—------------------------------------------------------------------------\n\nMMLU   $\\quad$   $\\quad$  FLoRA    $\\quad$ $\\quad$  3.40      $\\quad$   $\\quad$  $\\quad$  5.89\n\n$\\quad$  $\\quad$ $\\quad$  $\\quad$ $\\ $ FedIT     $\\quad$  $ $ $\\quad$ 3.33  $\\quad$   $\\quad$ $\\quad$  5.72\n\nWizard   $\\quad$   $\\ $ $\\ $  FLoRA    $\\quad$  $\\quad$  3.54   $\\quad$   $\\quad$ $\\quad$  6.65\n\n$\\quad$  $\\quad$ $\\quad$  $\\quad$ $\\ $ FedIT    $\\quad$ $ $ $\\quad$  3.44    $\\quad$    $\\quad$  $\\quad$ 6.65\n\n===========================================\n\nWe can see that FLoRA still outperforms the baseline in Gemma models. We will further conduct heterogeneous experiments and experiments of MMLU and ARC evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XChyNzqi0C", "reviewer": "Reviewer_G8j8", "summary": "This paper proposes a novel aggregation algorithm FLoRA for low-rank federated fine-tuning. Compared to prior works, FLoRA stacks the LoRA matrices together rather than averaging them, thus avoid the mathematical errors during the aggregation process on the server. The paper provides experiments compared with baselines and discusses about the efficiency and privacy preservation of FLoRA. The paper provides convergence analysis of FLoRA.", "questions": "Please see the weaknesses. The author might need to explain why they are using Llama rather than stronger models like Llama-2 or Llama-3.", "limitations": "The limiations are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper identifies and addresses an issue in LoRA federated learning algorithm from a unique perspective. The mathematical analysis provided in the paper is straightforward and convincing. This work can also provide intuition to other works related to LoRA-based merging. \n(2) The paper is well-written and easy to follow.\n(3) The paper provides sufficient experiments and analysis.", "weaknesses": "(1) The paper misses some related works. For example, [1] also talks about LoRA in federated fine-tuning and optimize its efficiency and accuracy. [1] and other possible related works should be considered and added to the paper.\n(2) The experiment results presented in Figure 4 show that some single client perform better than the server, which needs further explanation in the paper. Additionally, the subtitle 'The Impact of Heterogeneous...' here does not match the title 'Standalone...' of Figure 4, which needs further polishing.\n(3) The paper should explain why it uses Llama rather than SOTA models (e.g., Llama-3).\n\n[1] Sun, Y., Li, Z., Li, Y., & Ding, B. (2024). Improving LoRA in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's recognition of the contributions of our paper. Regarding the weaknesses raised, we address all the concerns in detail below.\n\n>W1 The paper misses some related works. For example, [1] also talks about LoRA in federated fine-tuning and optimize its efficiency and accuracy. [1] and other possible related works should be considered and added to the paper. \n\n[1] is a quite relevant paper also discussing the error of simply averaging LoRA, and focuses on the privacy issues in federated learning to aggregate LoRA modules. We will add a detailed discussion regarding the methods and insights of [1] in our future version.\n\n>W2 The experiment results presented in Figure 4 show that some single client perform better than the server, which needs further explanation in the paper. Additionally, the subtitle 'The Impact of Heterogeneous...' here does not match the title 'Standalone...' of Figure 4, which needs further polishing. \n\nIn our m=10 experiments, the training dataset is randomly divided into ten parts and distributed to clients for local training, resulting in an uneven quality distribution of the local datasets. Many studies have pointed out that datasets often contain a large amount of useless or even harmful data. As a result, it is acceptable that some clients may train local models that perform better than the global model due to receiving higher-quality or more suitable data for the test set.\nWe will change the title of Figure 4 to make it clearer and more accurate.\n\n>W3 The paper should explain why it uses Llama rather than SOTA models (e.g., Llama-3).\n\nIn our experiments, we observed that state-of-the-art (SOTA) models are not very sensitive to common fine-tuning datasets. Some models show no performance improvement after fine-tuning or even experience performance declines. For instance, after 3 epochs of fine-tuning on MMLU, we tested Llama-2-7b and obtained the following results:\n\n==========================\n\n$\\quad$ $\\quad$ $\\quad$   $\\quad$   Before      $\\quad$  $\\quad$     After\n\n—-------------------------------------------\n\nMMLU  $\\ $  $\\ $  $\\quad$    45.80   $\\quad$  $\\quad$   45.19\n\nHellaSwag $\\ $ $\\ $58.69 $\\quad$   $\\quad$59.47\n\n==========================\n\nWe believe this occurs because stronger models like Llama-2-7b have already converged on some common benchmarks, making further fine-tuning less effective in increasing accuracy. The performance of the Llama-3 series is also similar to this. To better demonstrate the effectiveness of fine-tuning, and to provide a clearer comparison between FLoRA and baseline methods, we chose models like Llama-7b where the benefits of fine-tuning are more evident.\n\n\nStill, following the advice of reviewer 7eRV, we extend our experiments to other series of models, here are some results:\n\n===========================================\n\nTraining Set    $\\quad$   $\\quad$    $\\quad$     Gemma-2b  $\\quad$    Gemma-7b\n\n—------------------------------------------------------------------------\n\nMMLU   $\\quad$   $\\quad$  FLoRA    $\\quad$ $\\quad$  3.40      $\\quad$   $\\quad$  $\\quad$  5.89\n\n$\\quad$  $\\quad$ $\\quad$  $\\quad$ $\\ $ FedIT     $\\quad$  $ $ $\\quad$ 3.33  $\\quad$   $\\quad$ $\\quad$  5.72\n\nWizard   $\\quad$   $\\ $ $\\ $  FLoRA    $\\quad$  $\\quad$  3.54   $\\quad$   $\\quad$ $\\quad$  6.65\n\n$\\quad$  $\\quad$ $\\quad$  $\\quad$ $\\ $ FedIT    $\\quad$ $ $ $\\quad$  3.44    $\\quad$    $\\quad$  $\\quad$ 6.65\n\n===========================================", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a federated learning mechanism to train LLMs with LoRA, The proposal is simple, but practical, with its effectiveness demonstrated in experiments. Therefore, we recommend accepting it.
Decision: Accept (poster)