Federated Residual Low-Rank Adaptation of Large Language Models
Authors: Yunlu Yan
Chun-Mei Feng
Wangmeng Zuo
Rick Siow Mong Goh
Yong Liu
Lei Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Languagel model
Federated Learning
Parameter-Efficient Fine-Tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Low-Rank Adaptation (LoRA) presents an effective solution for federated fine-tuning of Large Language Models (LLMs), as it substantially reduces communication overhead. However, a straightforward combination of FedAvg and LoRA results in suboptimal performance, especially under data heterogeneity. We noted this stems from both intrinsic (i.e., constrained parameter space) and extrinsic (i.e., client drift) limitations, which hinder it effectively learn global knowledge. In this work, we proposed a novel Federated Residual Low-Rank Adaption method, namely FRLoRA, to tackle above two limitations. It directly sums the weight of the global model parameters with a residual low-rank matrix product (\ie, weight change) during the global update step, and synchronizes this update for all local models. By this, FRLoRA performs global updates in a higher-rank parameter space, enabling a better representation of complex knowledge structure. Furthermore, FRLoRA  reinitializes the local low-rank matrices with the principal singular values and vectors of the pre-trained weights in each round, to calibrate their inconsistent convergence, thereby mitigating client drift. Our extensive experiments demonstrate that FRLoRA consistently outperforms various state-of-the-art FL methods across nine different benchmarks in natural language understanding and generation under different FL scenarios.

Published as a conference paper at ICLR 2025
FEDERATED RESIDUAL LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Yunlu Yan
Chun-Mei Feng?*
Wangmeng Zuo 3,4 Rick Siow Mong Goh
Yong Liu? Lei Zhul.31
Thhe Hong Kong University of Science and Technology (Guangzhou), China
2 Institute of High Performance Computing (IHPC),
Agency for Science, Technology and Research (A*STAR), Singapore
3 Harbin Institute of Technology, China
4 Pengcheng Laboratory, China
5 The Hong Kong University of Science and Technology, China
ABSTRACT
Low-Rank Adaptation (LoRA) presents an effective solution for federated fine- tuning of Large Language Models (LLMs), as it substantially reduces communi- cation overhead. However, a straightforward combination of FedAvg and LoRA results in suboptimal performance, especially under data heterogeneity. We noted this stems from both intrinsic (i.e., constrained parameter space) and extrinsic (i.e., client drift) limitations, which hinder it effectively learn global knowledge. In this work, we proposed novel Federated Residual Low-Rank Adaption method, namely FRLORA, to tackle above two limitations. It directly sums the weight of the global model parameters with a residual low-rank matrix product (Le., weight change) during the global update step, and synchronizes this update for all local models. By this, FRLoRA performs global updates in a higher-rank parameter space, enabling better representation of complex knowledge struc- ture. Furthermore, FRLORA reinitializes the local low-rank matrices with the principal singular values and vectors of the pre-trained weights in each round, to calibrate their inconsistent convergence, thereby mitigating client drift. Our ex- tensive experiments demonstrate that FRLORA consistently outperforms various state-of-the-an FL. methods across nine different benchmarks in natural language understanding and generation under different FL scenarios. Codes are available at https: thu. com
INTRODUCTION
Federated Learning (FL.) (Li et al., 2020a; Yin et al., 2021; et al., 2021b) allows multiple clients to collaboratively train a globally shared model by transferring model parameters, emerging as an effective distributed solution. However, applying this approach to Large Language Models (LLMs) faces significant challenges due to their enormous parameter sizes (Liu et al., 2021; Ye et al., 2024b). For example, GPT-4 (Achiam et al., 2023) has 1.75 trillion parameters, and even the smallest version of LLaMA-2 (Touvron et al., 2023) has 7 billion parameters. Fully Fine-Tuning (FFT) such large models in federated setting leads substantial communication overhead, making it impractical for real-world applications. A promising solution is to integrate Parameter-Effficiet Fine-Tuniin (PEFT) techniques reduce the number of trainable parameters in LLMs.
Low-Rank Adaptation (LoRA) (Hu et al., 2021), state-of-the-art PEFT technique, freezes the pre- trained weights and injects low-rank matrices into specific layers of the model, offering excellent solution for federated fine-tuning of LLMs. However, we found that a straightforward combination of FedAvg (McMahan et al., 2017) and LoRA (see Figure (a)) struggle to effectively learn global knowledge (see Figure (c)), particularly when the data across clients is a non- non-Inneependen and
"Equal contribution.
Tei Zhu (1 e zhuêust. hk) is the corresponding author.

---

Published as a conference paper at ICLR 2025
degpense driineg stis
(8D)
Fachin
70.79
70.73
75.01
Qortralio ood
1.25
Fadhii (hen-BD)
Follfig (10)
Fine funaci iedihta provizus vardd
(8)
Chent
Server
1.20
Oum (pan-l)))
4AM
0.50
Truef
Acerage
0.23
Time- kowed prnvo sunds
biokilod"
Dilett
Serwer
33 100
Roand
190
202
50
Round
120
150
200
(4)
(4)
Figure 1: Illustration of (a) FedAvg with LoRA and (b) FRLORA, where superscripts t denote timestamps and - highlights their differences. Specifically, FedAvg iteratively updates local and global low-rank matrices, whereas FRLORA directly updates the parameters of the local and global models W by summing their weights with residual low-rank matrix product. (c) Average Frobenius norm of global AW and (d) Average standard deviation of local low-rank matrices ver- sus the communication rounds, revealing that FRLoRA can effectively learn global knowledge and mitigate client drift. More details can be seen in Appendix SA
Identically Distribution (non-IID)), leading to serious performance degradation. This contrary to the goal of FL, which aims to represent the learned knowledge of all clients through the global model. The above issue arises from both intrinsic and extrinsic factors. Extrinsically, the data heterogeneity can result in client drift (Karimireddy et al., 2020) (see Figure 1 (d)), and this phe- nomenon is more pronounced with LoRA compared to FFT, as the parameters in LoRA are locally quadratic (Liu al.. 2024). Intrinsically, both the local and global low-rank matrices con- strained to updates in the parameter space with a rank no greater than r. Such constrained parameter space makes it difficult to effectively capture the diverse knowledge of all clients.
While the non-IID has been widely explored in FFT-based FL (Ye et al., 2023), little is understood about how to tackle in LoRA- FL. Previous empirical studies (Ye al., 2024a;b) have indi- cated that simply integrating LoRA with existing heterogeneous FL methods like FedProx (Li et al., 2020b) and SCAFFOLD (Karimireddy et al., 2020) fails to address data heterogeneity. This primar- ily arises from the different update patterns and learning capabilities between LoRA and FFT (Liu et al., 2024). Consequently, there a need for more nuanced approach to improve the performance of LoRA-based FL..
To this end, we propose a novel Federated Residual Low-Rank Adaptation approach, namely FRLORA, to handle the data heterogeneity in LoRA-based FL. Instead of only updating local and global low-rank matrices, during the global update step, FRLORA (see Figure I (b)) directly update the weight of the global model parameters by summing its weights with a residual low-rank ma- trix product (i.e., weight change) and synchronize this update for all local models, which makes the update pattern similar to that of FFT-based FL. By accumulating residuals with increasing rounds, the global updates can be conducted in higher-rank parameter space, enabling more flexible rep- resentation of complex knowledge structures. Moreover, FRLORA reinitializes the local low-rank matrices with the principal singular values and vectors of the pre-trained weights in each round. This ensures that the local low-rank matrices are consistently updated in the principal singular space of the pre-trained weights, benefiting to alleviate client drift (see Figure I (d)). Based on the above two aspects, FRLORA effectively captures global knowledge (see Figure (c)), thereby significantly improving the performance of the global model.
Contribution. In this work, we improve the LoRA-based FL by addressing both intrinsic and extrinsic limitations, yielding a novel FL. method, FRLORA. Different from previous paradigms that only update local and global low-rank matrices, FRLORA directly update the weight of the local and global model parameters. We conducted extensive experiments on nine benchmarks across different language tasks under various FL scenarios, including IID and non-IlI, as well as partial and full participation, adequately demonstrating the superiority of FRLORA over the oate-of-the-ant FL methods.
We use "data hetcrogeneity" and "non-IllD" interchangeably.
We term the original part of the L.I without injected low-rank matrices as "model".
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 LARGE LANGUAGE MODELS
Generally, LLMs, also known as foundational language models (Myers et al., 2024) like GPT- 3/4 (Brown, 2020; Achiam al., 2023), LLaMA-2/3 (Touvron et al., 2023; Dubey et al., 2024), and PaLM (Chowdhery et al., 2023) have learned abundant knowledge by being pre-trained on mas- sive public language corpus (Raffel et al., 2020; Gao al., 2020), achieving tremendous success in various applications. Their versatility in language tasks has attracted researchers to fine-tune them across a broad spectrum of fields, such as healtheare (Singhal al., 2023; Xie al., 2024; Wang et al., 2023), to further enhance their capabilities for specialized downstream tasks. Accompanying this trend is the emergence of various language model fine-tuning methods, such as Instruction- Tuning (IT) (Mishra et al., 2022; Zhang al., 2023b) and Reinforcement Learning from Human Feedback (RLHF) (Bai et al., 2022). However, all of them train the models using centralized data. In real-world applications, data is typically distributed across different users' edge devices, such as mobile phones and laptops. Due to privacy regulations like GDPR (Voigt & Von dem Bussche, 2017), collecting this data can be expensive and even infeasible. In this work, we aim to explore how to utilize FL to fine-tune L.MSs in a distributed and effective manner. This facilitates the effective utilization of discrete data from different users.
2.2 PARAMETER CHAAMETER-EFFIIIET -UNNIGGNIGG
PEFT (Mangrulkar et al., 2022) has emerged as an important technique for reducing training param- eters in fine-tuning large pre-trained models (Xu et al., 2023). Instead of fine-tuning all parameters, PEFT introduces only a small number of trainable parameters at specific locations of networks, such as input tokens (Li & Liang, 2021; Lester et al., 2021) and layers (Houlsby et al., 2019), while keeping the rest of parameters frozen. Recently popular LoRA methods (Hu et al., 2021; Liu et al., 2024: Zhang et al., 2023a; Gao et al., 2024; Lin et al., 2024) fine-tune LLMs by injecting two train- able low-rank matrices into the layers of the base model. It not only effectively compresses training costs but also shows outstanding performance across range of language tasks. Inspired by this, several works (Wu et al., 2024; Vavekanand & Sam, 2024; Ye et al., 2024b;a) introduced LoRA into FL to decrease the communication cost of federated fine-tuning L.Mss. However, most of them ignore data heterogeneity in real-world scenarios, which can significantly degrade the performance of LoRA-based FL. This motivates us improve the performance of LoRA-based FL under data heterogeneity.
2.3 FEDERATED LEARNING
Federated Fine-tune Large Language Models. FL has become de facto distributed solu- tion. Since FL can protect user privacy, it has been widely used in privacy-sensitive areas like finance (Long et al., 2020; Chatterjee et al., 2023) and medical (Feng et al., 2022; Yan et al., 2024a; Jiang et al., 2023; Yan et al., 2024b; Feng et al., 2023a; Yan et al., 2023a). The tremendous success of FL has also attracted the attention of researchers introduce it into L.MS (Qu, 2024; Fan al., (Kuang et al., 2024)
2023; Han et al., 2024; Zhang et al., 2024). For example, FederatedScope--L.. built a comprehensive package for federated fine-tuning LLMs and explored the federated instruc- tion tuning. OpenFedLLM (Ye et al., 2024b) provided the empirical results for federated instruction tuning and federated value alignment. Besides, Ye al. (2024a) provided realistic benchmarks and empirical results for federated fine-tuning LLMs. These foundational studies have greatly advanced the development of this field. In contrast to them, this work focuses on a different challenge: how to effectively fine-tune I. LLMs, especially under data heterogeneity. Similar to us, Sun et al. (2024) proposed FFA-LORA to improve the performance LoR A-RR-based FI. under differential privacy and data heterogeneity by fixing the matrix A in LoRA. This can further diminish the learning capability of LoRA, resulting in degraded performance under realistic heterogencous FL settings.
Data Heterogeneity in Federated Learning. The standard FL process was introduced by Fe- dAvg (McMahan et al., 2017), a pioneer algorithm, which updates the global model by averaging the weights of local model parameters. Despite its success, the performance of FedAvg is inevitably compromised by practical challenges such as data heterogeneity (Zhao et al., 2018; Ye et al., 2023) or model heterogeneity (Alam al., 2022). Data heterogeneity (Ye et al., 2023; Feng et al., 2023b) is
3

---

Published as a conference paper at ICLR 2025
fundamental challenge in FL, with typical example being the non-llI across different clients (Zhao et al., 2018). This issue causes local model bias due to skewed data, which affects global aggrega- tion and leads to significant degradation in the global model's 's performance. phenomenon known as client drift. (Karimireddy et al., 2020; Li et al., 2022). To address this, several studies have proposed enhanced versions of FedAvg, either by modifying local training (Li et al., 2021a; Tan et al., 2022; Luo et al., 2021; Zhang et al., 2022: Gao et al., 2022; Yan et al., 2023b), global aggregation (Li et al., 2020c; Wang et al., 2020; Li et al., 2023; Fallah al., 2020), both (Jiang et al., 2022; Huang al., 2023). However, these enhancements primarily target data heterogeneity in FFT-based FL. Apply- ing them to LoRA-based FL may not yield favourabl results due the distinet update patterns and learning capabilities between LoRA and FFT (Liu et al., 2024). As evidence, previous empirical studies (Ye et al., 2024a;b) have shown that combining several well-known heterogeneous FL algo- rithms (e.g., FedProx (Li et al., 2020b) and SCAFFOLD (Karimireddy et al., 2020)) with LoRA can result in worse performance than FodAvg+LoRA. Therefore, how to address the data heterogeneity in LoRA-basck FL remains an unsolved question, which is the main goal of this work.
3 METHODOLOGY
3.1 PRELIMINARY
Low-Rank Adaptation. When fine-tuning LLMs on downstream tasks, LoRA (Hu et al., 2021) only updates the parameters of some layers, e.g., self-attention. Consider layer in the network, LoRA freezes the pre-trained weight W E Rd xd2 and leans the weight change A AW by injecting two trainable low-rank matrices B E Rdix and A Rrxds where rank < min(d1, d2). The
updated weight is expressed as:
W W + AW W + BA.
(1)
To ensure consistency with the pre-trained weight during the initial phase, B is initialized zero matrix, while A is initialized with Gaussian noise (0, Besides, hyper-paramette a is typically used scale AW, and the scaling factor is f.
Federated Learning with Low-Rank Adaptation. Consider a FL system with K clients and a central server, each client & E [K] has a private dataset D& for downstream tasks such as natural language understanding or generation. Each dataset D2 contains training samples {(r; v1) ) and our goal to federated fine-tune a global LLM f(W) on these discrete data. Due to the large size of LLMs, we cannot train and transmit all model parameters. A straightforward approach to directly combine FedAvg with LoRA (Ye et al., 2024b;a), where each client maintains local low-rank matrices and optimizes them by minimizing the empirical risk:
Th Fa
1
Σ FR.
and Fx
1
Σ (SR63A))
(( W B A.)
K
TAE:
where FG and F& are global and local objectives, Bi and A are the low-rank matrices at 1-th round training. and the loss function. All updated local low-rank matrices will be transferred to the server and we can get the global low-rank matrices:
K Σ A
By
=
B
A
= K
(3)
K
k=1
k=1
The updated global lowkrank matrices B and A will be returned to each client as the initialization for the next round of training, where Ba ~ 0 and A ~ -N(0, o2). The weight of final fine-tuned global model parameters after T rounds can be written as:
W = WD - = W + ATAT.
(4)
Limitation. The above method is easy to implement and can significantly reduce communication overhead when fine-tuning LLMs. However, it fails to effectively capture global knowledge, es- pecially under data heterogeneity. This issue arises from two factors: O constrained parameter space and @ client drift, where 1 is the intrinsic limitation of LoRA- based FL and @ is the extrinsic influence caused by data heterogeneity.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: FRLORA
Input:
Number of clients K, communication rounds T', learning rate 7, Pre-trained weight W Datasets Dr, Dz...., .......,xx. Dx. rank
Output: Fine-tuned weight W
Server-side Execution:
USV SV
B% < U
A v Global initialize for LoRA Consistent with pre-trained model
W < W -BB
Ag 0//// //
for round 1 1,2, T...''o do
for client k 2.., K parallelly do
Bi A - Local Training (k: B 1/''6,0
end
Ba + 1
K Ek= B k A // Parameter aggregaticn
AW' c B& B2 W' W AW" // W AWW keep consistent between server and clients, thus use same symbol
10
11 end
WT + WT + B.4%
12
return W // Pi tine-tuned weight
14 Local Training (I,, 1/ 1 ,0)
15 Save By and A
16 Bi <- BBg A <- A // Local initialize for LoRA
17 for (x,. ~ Da do
FR [[f(r;; W B A)):v)
B A T/F.FK
20 end
return B and A server
3.2 FEDERATED RESIDUAL LOW-RANK ADAPTATION
To tackle the above two limitations in LoRA-based FL, we propose new method, FRLORA. Algorithm 1 shows its whole training procedure, and describe each part of framework in detail below.
Initialization in Principal Singular Space. LoRA typically initializes B as 0 and A with Gaussian noise. Such initialization makes convergence challenging, and optimizing with the data heterogene- ity results in inconsistent convergence rates across different clients, further exacerbating client drift. This hinders the global AW from capturing task-specific information. To this end, we initialize B and A in principal singular space of the pre-trained weight inspired by Meng al. (2024), enabling faster learning of task-specific information for downstream tasks. Specifically, we first decompose the pre-trained weights using Singular Value Decomposition (SVD) as follows:
USV SV SVW("),
(5)
where U E xd and V dd are the singular vectors with orthonormal columns, and S E Xd is a diagonal matrix containing the singular values in descending order. We then use the principal singular values and vectors initialize B and A which can be written
B = U.:llsS::
A%
(6)
B and Ag will be sent to each client to initialize the local low-rank matrices. Additionally, to ensure consistency, the weight of the global model parameters will be updated accordingly:
V Wo 4%
(7)
Local Update Step. Each client performs local training on its dataset and the local objective of Eq. (2) be rewritten
F
Σ e((x: Wi-1, B B v)):
(8)
(D
5

---

Published as a conference paper at ICLR 2025
We send the optimized local low-rank matrices B and A to the server, which then uses Eq. (3) to update the global low-rank matrices.
Global Update Step. Different from the traditional LoRA-based FL methods, FRLORA directly updates the weight of the global model parameters by adding a residual the low-rank matrix product, which bridges the gap with FFT-based FL..
After getting global low-rank matrices B and A we can get a residual of the low-rank matrix product as:
AW A A
The residual matrix AW' represents the weight change learned in this round, and it will be added to the weights of the global model parameters:
W W AW
(10)
The updated weight W of the global model parameters will be frozen again, and the local low-rank matrices of FRLORA will be reinitialized to By and A preparing for the next round of training. This makes the update pattern of FRLORA similar to that of FFT-based FI., where the weight of model parameters from the previous round serves as the starting point for the next round of training-
Parameter Synchronization. Notably, the process of Eq. (7) and Eq. (10) are synchronized across clients. As a result, each local model remains consistent with the global model, and we use the same symbol W to represent them. The final fine-tuned weight of Eq. (4) can be rewritten as:
W T = WT + AG
(11)
Analysis in $3.3 reveals that FRLORA enables the global model to be updated in higher-rank space, effectively addressing Limitation Besides, reinitialization can be viewed as recalibration of local and global low-rank matrices, ensuring that the local training of each client always occurs in the same global principal singular space, which benefits mitigate client drift (Limitation @).
3.3 THEORETICAL INSIGHTS
In this section, we conduct a theoretical analysis of the update patterns of FRLoRA and LoRA-based FL. For LoRA-based FL, the final weight change of the global model parameters is represented by B AT. Applying the basic rank inequality, we obtain:
rank B A S min(rank(AA)) rank BB)),
(12)
where (A)) min(r, d2) and rank ( min(d1, r). Consequently, since LoRA-based FL only updates the low-rank matrices, the global updates are restricted to a low-rank subspace with a rank greater than r.
In contrast, the final fine-tuned weight of FRLORA can be expressed as:
W = W + WW 4 +... + AWT
(13)
residual accumulation
We also present the following additive rank inequality:
Lemma 1. (Additive Rank Inequality) For any two matrices M1 and M2. it holds that:
vank(M; + +M2 & rank (M) + rank(M2).
(14)
Given that FRLORA reinitializes in each training round and that the local and global updates for each round are independent, the rank of the residual accumulation increases with the number rounds according Lemma 1. This implics that global updates occur in higher-rank parameter space, which enables the global model to capture more complex knowledge structures.
From another perspective, AW' can be interpreted as the total gradient of the global model over a round by minimizing FG. As FRLORA learns in the principal singular space of the pre-trained model during each round, the gradient AW' of FRLORA is larger compared to that of LoRA-based FL, based on the analysis of Meng et al. (2024). This allows the global model to acquire more task-specific knowledge, illustrated in Figure (c).

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
To validate the effectiveness of our method. We conducted comprehensive experiments across both Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks. We com- pared our method against 9 state-of-the-art baselines, including FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020b). SCAFFOLD (Karimireddy al., 2020), FedAvgM (Hsu et al., 2019),
FedAdagard (Reddi et al., 2021). Fed (Reddi et al., 2021). FedAdam (Reddi et al., 2021),
FlexLoRA (Bai et al., 2024) and FFA-LORA (Sun et al., 2024). The experiments involved
NLU benchmarks: RTE (Wang et al., 2019), COLA (Wang al., 2019), 20NG (Lang, 1995)
and QNLI (Wang et al., 2019), as well as 5 NLG benchmarks: MetaMathQQ (Yu et al., 2023), Alpaca-GPT4 (Peng et al., 2023), FedAya (Ye et al., 2024a), Fed-Chatbotll (Ye et al., 2024a), and Fed- WildChat (Ye et al., 2024a). All experiments were implemented using Py PyTorch and con- ducted an NVIDIA A A100 GPU with 40 GB of memory. Due to page limitations, details and results are presented in the Appendix $B and SC
4.1 QUANTITATIVE COMPARISON ON NLU TASKS
Task Setup. Following Kuang et al. (2024), we randomly partition the training set of each bench- mark using the Dirichlet distribution sampling (Dx Dir(3)), which is a commonly employed strategy for simulating realistic data heterogeneity (Ye et al., 2023). The level of data heterogeneity controlled by 3. where a smaller means higher heterogeneity. In our experiments, /3 is set to 0.5. We simulate scenario with 5 clients, all of which participate in training during each round. To evaluate the performance of all methods, we use the validation sets of RTE, COLA, and QNLI, and the test set of 2ONG. Accuracy is used the evaluation metric for RTE, QNLI, and 20NG, while the Matthews correlation coefficient is employed for COLA.
Implementation Details. We utilize ROBERTa- basse (Liu, 2019) for the NLU tasks. For LoRA, set the parameter r to 16 and a to 32. The AdamW optimizer is used with batch size of 64, learning rate of 2e-4 and cosine annealing schedules. All methods are trained for 200 rounds. The lo- cal update step is set to 10 for RTE and 30 for QNLI, 20NG, and COLA based on the quantity of data in each dataset.
Table 1: Experimental results varios NLU benchmarks. Best results are marked in bold.
Method
RTE
COLA
20NG
QNLI
FedAvg
70.75
63.96
66.83
90.37
FedProx
69.31
63.41
66.48
90.05
SCAFFOLD
67.15
62.83
65.78
89.60
FedAvgM
61.73
43.82
67.04
89.91
Results. As shown in Table 1, directly in- tegrating heterogeneous FI. methods with LoRA does not achieve better performance on the NLU task compared to FedAvg. This is primarily due to the differing update pat- tern between FFT and LoRA, which is con- sistent with previous empirical findings (Ye al., 2024b;a). Furthermore, we observe that FFA-LoRA performs worse than Fe- dAvg in our experimental setting. This is
Fedadagrad
66.42
54.27
63.48
87.57
FedYogi
63.53
55.20
66.44
86.21
FedAdam
66.06
56.30
66.94
89.69
FFA-LoRA
68.69
58.43
66.88
89.33
FlexLoRA
70.28
62.56
65.98
90.03
FRLORA (Ours)
75.81
64,80
69.41
91.10
due to FFA-LoRA's focus on differential privacy, as well as the discrepancy between our task setup and its original configuration, which employed manually partitioned data. Such partitioning does not fully reflect the complexity of real-world scenarios, leading to a mismatch in performance under our experimental settings. The results in the real-world data heterogeneity setting suggest that up- dating only matrix B in FFA-LORA fails to effectively capture global knowledge. In contrast, our meticulous design enables F FRLORA to consistently outperform all baselines. achieves significant improvements over FedAvg, such increasing performance from 70.35% to 75.81% % on RTE and from 66.83% to 69.41% on 20NG. This demonstrates that F FRLORA can effectively address the issue of data heterogeneity in NLU tasks.
7

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results NLG tasks, where MetaMathQA and Alpaca-GPT4 used the training dataset, respectively. Results copied from (Ye al., 2024b). Avg. is the average result of corresponding metrics, and the best results are marked in bold.
MetaMathQA
Alpaca-GPT4
Method
GSM8K
Math
Avg.
Vicuna
MT-1
MT-2
MT-Avg
Avg.
FedAvg
34.95
4.48
19.71
7.925
4.650
2.025
3.346
4.486
FedProx
35.40
4.66
20.03
7.875
4.538
1.848
3.201
4.320
SCAFFOLD
35.78
5.08
20.43
7.675
4.689
2.288
3.488
4.535
FedAvgM
34.79
4.64
19.71
7.938
4.838
2.038
3.456
4.567
FedAdagrad
29.64
4.06
16.85
7.931
4.675
2.025
3.350
4.495
Yed
30.09
4.04
17.06
8.031
4.550
1.938
3.244
4.440
FedAdam
31.84
4.12
17.98
7.975
4.650
2.175
3.413
4.508
FFA-LoRA
28.05
3.78
15.91
7.862
4.712
1.950
3.331
4.463
FlexLoRA
34.09
4.31
19.20
7.884
4.561
2.012
3.286
4.435
FRLORA (Ours)
44.27
5.22
24.74
8.044
4.775
2.481
3.635
4.733
Table 3: Experimental results on NLG tasks, where Fed-Aya is used as the training dataset and evaluated on Ref-GPT4. Results are copied from (Ye et al., 2024a). Avg. is the average result of all evaluation metries, and the best results are marked bold.
Method
fr
pt
r'U
te
zh
Avg.
FedAvg
2.50
8.00
5.50
5.35
4.95
5.65
2.00
5.25
4.90
FedProx
3.20
7.10
5.90
5.65
4.85
5.20
1.60
5.80
4.92
SCAFFOLD
2.65
7.75
6.30
5.35
5.00
6.35
1.45
4.90
4.97
FedAvgM
3.00
7.80
5.35
5.00
5,30
5.65
1.90
5.00
4.86
FedAdagrad
2.50
7.85
5.15
5.25
4.45
5.75
1.55
5.50
4.75
Yed
2.00
8.45
6.15
4.55
3.85
6.30
1.65
4.93
4.73
FedAdam
2.40
8.50
5.25
4.70
4.35
5.40
1.90
5.20
4.71
FFA-LoRA
2.10
7.90
5.70
5.15
4.30
5.15
1.65
4.60
4.56
FlexLoRA
2.60
8.20
5.25
5.05
4.70
5.20
1.85
4.75
4.70
FRLORA (Ours)
4.45
7.75
6.15
6.65
4.75
6.25
1.55
6.95
5.56
4.2
QUANTITATIVE COMPARISON ON NLG TASKS
Task Setup. Our experiments for NLG tasks follow previous empirical studies (Ye al., 2024abb). For MetaMathQA and Alpaca-GPT4, we partition the datasets in an IID manner, with 10 clients for MetaMathQA and 20 clients for Alpaca-GPT4, In each round, we randomly select 2 clients to par- ticipate in training. Fed-Aya, Fed-ChatbodTT and Fed-WildChat are realistic benchmarks with data heterogeneity, consisting of 38, 237, and 100 clients, respectively. Correspondingly, we randomly select 4, 10, and 5 clients to participate in training each round.
Evaluation.
The evaluation process NLG tasks fundamentally different from NLU tasks. We use the following benchmarks for evaluation: GSM8K (Cobbe et al., 2021) Math (Yu et al., 2024), Ref-GPT4 (Ye al., 2024a), MT-Bench (Zheng et al., 2024) and Vicuna (Chiang et al., 2023). For GSM8K and Math, use accuracy the evaluation metric. For the remaining benchmarks, we employ GPT-4 (Achiam et al., 2023) to rate the generated responses on scale from I to 10. All benchmarks are open-ended evaluations. MT-Bench assesses both one-turn and two-turn versations, with MT-1 and MT-2 representing the scores for one-turn and two-turn interactions, respectively. Other benchmarks focus solely one-turn conversations.
Implementation Details. We employ LLaMA-2-7B (Touvron et al., 2023) for NLG tasks. For MetaMathQA and Alpaca-GPT4, we 32 and to 64. For the remaining three benchmarks,
8

---

Published as a conference paper at ICLR 2025
Table 4: Experimental results on NLG tasks, where Fed-ChatbotIT and Fed-WildChat used training dataset, respectively. Results are copied from (Ye al., 2024a). Avg. the average result of corresponding metrics, and the best results are marked in bold.
Fed-Chathoth
Chl-
Method
MT-1
Vicuna
Ref-GPT4
Avg.
MT-1
Vicuna
Ref-GPT4
Avg.
FedAvg
4.30
6.93
5.29
5.51
4.81
7.99
5.88
6.22
FedProx
4.25
7.21
5.00
5.49
4.86
7.93
5.74
6.17
SCAFFOLD
3.86
7.35
4.82
5.34
4.78
7.93
5.57
6.09
FedAvgM
4.34
7.17
4.76
5.42
4.52
8.07
5.85
6.14
FedAdagrad
3.94
7.50
4.99
5.48
4.76
7.76
5.93
6.14
Yed
4.13
7.20
5.00
5.44
4.78
8.04
5.48
6.10
FedAdam
3.88
7.32
5.02
5.41
4.54
8.03
5.68
6.08
FFA-LORA
3.78
6.85
5,45
5.36
4.37
7.79
5.57
5.91
FlexLoRA
4.17
7.02
5,40
5.53
4.88
7.91
5.78
6.19
FRLORA (Ours)
4.31
7.49
5.62
5.80
4.64
8.24
7.00
6.63
12
2.5
hashght
-064
1.0
2.0
FodFrow
haskin
Packdagad
Trening
0.85
Fartnoll
-0.72
-6.79
150 125
200
75
100
25
80
75
100
125
150
175
200
Round
Round
Round
(a)
(b)
(c)
Figure 2: Illustration of global training loss communication rounds (a) RTE, (b) 20NG.
and (c) MetaMathQA.
and a are set to 16 and 32, respectively. The Adam optimizer used with learning rate of 5e-4 for MetaMathQA and A/paca-GPT4, and 2e-4 for other benchmarks, following a cosine annealing schedule. We conduct training with rounds of either 100 or 200. For all benchmarks, is employed as the fine-tuning paradigm, utilizing the instruction template of Alpaca (Taori et al., 2023).
Results. Table 2 depicts results of methods on MetaMathQA and Alpaca-GPT4 benchmarks with IID distribution. In this setting, the impact of client drift is decreased. However, LoRA-based FL still suffers challenges from the intrinsic limitation. In contrast, FRLcRA can effectively address all baselines. For example, it improves
it, resulting in significant performance improvements the average accuracy from 19.71 % to 24.74% on MetaMathQA. Furthermore, Tables 3 and 4 show the performance of all methods three realistic benchmarks, i.e., Fed-Aya, FedChatbotIT and Fed- WildChat, with data heterogeneity. Although FRLORA performs lower than the best baselines on some metries, it consistently outperforms all baselines the average metrics across various bench- marks. The above results show the superiority of FRLoRA compared to all baselines, demonstrating its effectiveness in addressing data heterogeneity NLG tasks.
Moreover, the experiments are conducted in the partial participation setting, demonstrating that FRLoRA can effectively learn global knowledge even when only subset clients participates in training. This expands the range of scenarios where our method can be applied. In contrast, FFA- LoRA yields worse performance on five benchmarks compared to FedAvg, as freezing matrix A restricts its learning capability, particularly in the partial participation setting.
4.3 CONVERGENCE
In Figure 2, we show the global training loss, i.e., Fa in Eq. (2), of all FL methods versus com- munication rounds RTE, 20NG, and MetaMathQA. Since FRLORA train low-rank matrices in

---

Published as a conference paper at ICLR 2025
Table 5: Results of ablation study on RTE, 20NG, MetaMathQA, and Fed-WildChat. Best results are marked in bold.
NLU
MetaMathQA
Fed-WildCher
Method
RTE
20NG
GSM8K
Math
Avg.
MT-1
Vicuna
Ref-GPT4
Avg.
FedAvg
70.75
66.83
34.95
4,48
19.71
4.81
7.99
5.88
6.22
FRLoRA- v1
74.12
67.68
36.02
4.73
20.37
4.84
7.96
5.91
6.23
FRLoRA-VZ
69.81
65.77
36.81
4.92
20.86
4.58
7.84
6.08
6.16
FRLORA-VE 58.62
63.19
39.08
5.01
22.04
4.43
8.03
6.34
6.26
FRLORA (Ours)
75.81
69.41
44.27
5.22
24.74
4.64
8.24
7.00
6.63
the principal singular space, it exhibits faster convergence rate compared to other methods. This further presents the superiority of our method. Moreover, as illustrated in Figure 2 (a), the learning mechanism of FRLORA leads to more stable convergence.
4.4 ABLATION STUDY
Effectiveness of Update Mechanism. To access the effectiveness of update mechanism in FRLoRA, built up FRLORA-vi FedAvg initializes B and A using the principal singular values and vectors of pre-trained weight, i.e. FedAvg+PiSSA (Meng et al., 2024). The results in Table 5 show that FRLORA yields consistent improvements over FRLORA-vI on all datasets across different tasks. This strongly demonstrates the effectiveness of update mechanism in F FRLoRA, which enables the global model be updated higher-rank parameter space, allowing better capture of the diverse knowledge from all clients.
Effectiveness of Initialization Mechanism. To further explore the effectiveness of initialization mechanism in F FRLRA,, we constructed FRLORA-V2, a variant that employs standard initialization methods, i.e., zero and Gaussian noise, to reinitialize B and A at each round. Additionally, we developed FRLORA-V3, variant that the weight of local and global model parameters is updated directly using the global low-rank matrix and the Eq. (10) can be rewritten as W W +
By Following, we reinitialize B and A ++1 with the principal singular values and vectors of W similar to Eq. (5)-(7).
As shown in Table 5, the performance of FRLORA-V2 suggests that relying solely on the parameter residual mechanism is insufficient to handle data heterogeneity, standard initialization struggles to converge properly, resulting in the degradation of global knowledge A AW As an excellent complement, reinitializing the principal singular space at each round can effectively address this issue and significantly improve the performance under both IID and non-lID settings. Additionally, a comparison between FRLORA and FRLORA-V3 reveals that using SVD to decompose new model weights for initialization at each round results in worse performance compared to our approach. This attributed to the instability of the SVD operation, using frequently can significantly affect model convergence. Additionally, it also incurs a large amount of computational overhead.
CONCLUSION
In this work, we address the challenge of data heterogeneity in LoRA-based FL. We observe that LoRA struggles to effectively capture global knowledge in heterogeneous FL settings due to both intrinsic and extrinsic limitations. To overcome these limitations, we introduce a novel approach, FRLORA, which directly updates model parameters by learning a residual low-rank matrix product in the principal singular space of pre-trained weight. We validate the effectiveness of FRLORA through extensive experiments across nine benchmarks from different language tasks. Furthermore, offer deeper understanding of our method from both theoretical and empirical perspectives.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the Major Key Project of PCL under Grant PCL2024A06, the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-TC- 2021-003), Agency for Science, Technology and Research (A*STAR) through its AME Program- matic Funding Scheme Under Project A20H4b0141, A*STAR Central Research Fund "A Secure and Privacy Preserving Al Platform for Digital Health", and Agency for Science, Technology and Research (A*STAR) through its RIE2020 Health and Biomedical Sciences (HBMS) Industry Align- ment Fund Pre- Pre-Positioning (IAF-PP) (grant no. H20C6a00322, National Natural Science Founda- tion of China (Project No. 61902275), Guangdong Provincial Key Lab of Integrated Communica- tion, Sensing and Computation for Ubiquitous Internet of Things(No.22) 2023B1.22222222 121200007), and the Guangdong Science and Technology Department (No. 2024ZDZX2004).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang, Fedrolex: Model-heterogeene federated learning with rolling sub-model extraction. Advances neural information processing systems, 35:29677-29690, 2022.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous language tasks and client resources. arXiv e-prints, pp. arXiv-2402, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165,, 2020.
Pushpita Chatterjee, Debashis Das, and Danda B Rawat. Federated learning empowered recom- mendation model for financial consumer services. IEEE Transactions on Consumer Electronics, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
// ///mmyss.cc. http://rrgg//oo////e//////////// -00----333--iiiii0 -00300vicnna/.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint 10.1661 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.2178. 2024.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3563 2020.
11

---

Published as a conference paper at ICLR 2025
Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate- Ilm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310. 100499 2023.
Chun-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu. Specificity- preserving federated learning for mr image reconstruction. IEEE Transactions on Medical Imag- ing, 42(())2010-200 202002 2022.
Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8064-8073, June 2023a.
Chun-Mei Feng, Kai Yu, Nian Liu, Xinxing Xu, Salman Khan, and Wangmeng Zuo. Towards anstance-adaptive inference for federated learning. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pp. 23287-23296, October 2023b.
Ieo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gh dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002: 2020.
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhone Xu. Fedde: Feder- ated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference computer vision and pallern recognition, pp. 101 12-1121,,,, 2022.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. murameter-efficiere fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.0300 03003, 2024.
Shanshan Han, Baturalp Buyukates, Zijian Hu, Han Jin, Weizhao Jin, Lichao Sun, Xiaoyang Wang, Wenxuan Wu, Chulin Xie, Yuhang Yao, al. Fedsecurity: benchmark for attacks and defenses in federated learning and federated lims. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5070-5081 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. crrameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-27999 2799. PMLR, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.000 06335, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with do- main shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16312-16322. IEEE, 2023.
Meirui Jiang, Zirui Wang. and Qi Dou. Harmofl: Harmonizing local and global drifts in federated learning heterogeneous medical images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 1087-1095, 2022.
Meirui Jiang, Holger R Roth, Wenqi Li, Dong Yang. Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair federated medical image segmentation via client contribution estima- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16302-16311, 2023.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Weirui Kuang, Bingehen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding. and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5260-5271, 2024.
Ken Lang. Newsweeder: Learning to filter netnews. In Armand Prieditis and Stu- art Russell (eds.), Machine Learning Proceedings 1995, pp. 331-339. Morgan Kaufmann, San Francisco (CA), 1995. ISBN 978-1-55800-377-6
doi: htps://doi.org/110. 1016/ URL https: //t::///wwwwww www. www.scceeeeeeet................. wtw.e//ccddrrett.ccocccccceeeeee
articleess1//87711588000333
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
Qinbin Li, Bingsheng He, and Dawn Song, contt-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10713-10722, 2021a.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions Knowledge and Data Engineering, 35(4):3347-3366, 2021b.
Qinbin Li. Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pp. 965-978. IEEE, 2022.
Qinbin Li, Bingsheng He, and Dawn Song. Adversarial collaborative learning on non-III features. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19504-19526. PMLR, 23-29 Jul 2023. URL htwps:tpp/..........
1i23 hhtml.
Tian Li. Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-60) 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- fems, 450,, 2020b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00190, 2021.
Xiaoxiao Li, Meirui JIANG, Xiaofci Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learn- ing non-iid features via local batch normalization. In International Conference on Learning Representations, 2020c.
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei. Lora dropout as a sparsity regularizer for overfitting control. arXiv preprint arXiv:2404.09611 2024.
Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. Federated learning meets natural language processing: A survey, arXiv preprint arXiv:2107.12603, 2021.
Shih-yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Interna- tional Conference on Machine Learning, 2024.
Yinhan Liu. Roberta: robustly optimized bert pretraining approach. 1rXiv:1907.11692, 2019.
arXiv preprint
Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated learning: privacy and incentive, pp. 240-254. Springer, 2020.
13

---

Published as a conference paper at ICLR 2025
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-599 5984, 2021.
Sourab Mangrulkar, Sylvain Gugger. Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-ar parameter-efficient fine-tuning methods. https: //ghtub.. com/hagg face cacccepee peft, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcieent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02944 022444 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pp. 3470-3487. Association for Computational Linguistics (ACL), 2022.
Devon Myers, Rami Mohawesh, Venkata Ishwarya Chellaboina, Anantha Lakshmi Sathvik, Praveen Venkatesh, Yi-Hui Ho, Hanna Henshaw, Muna Alhawawreh. David Berdik, and Yaser Jararweh. Foundation and large language models: fundamentals, challenges, opportunities, and social im- pacts. Cluster Computing, 27(1):1-26, 2024.
Baolin Peng, Chunyuan Li, Pengeheng Hc, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXhv:2304.0322 2023.
Youyang Qu Federated learning driven large language models for swarm intelligence: A survey. arXiv preprint wrXiv:2406.09831, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
Chongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xi- aokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405.14733 2024.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 122-7972) 2023.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora privacy-preservie federated learning. arXiv preprint arXiv:2403.1 12313, 2024.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fed- proto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8432-8440, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnstruction-oolloiiin llama model. https: /ttgttttttttttt com/ catsu-lab/ sstaafrddaappcce 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.0822 2024.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09928 2023.
Raja Vavekanand and Kira Sam. Cellm: An efficient communication in large language models training for federated learning. arXiv preprint arXiv:2407:205557 2024.
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical
Guide, Ist Ed., Cham: Springer International Publishing, 1031156666:10-555,, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019.
Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li. Clinicalgpt: large lan- guage models finetuned with diverse medical data and comprehensive evaluation. arXiv preprint arXiv:2306.09968 2023.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623, 2020.
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp.
2024.
Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongei Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, et al. Me llama: Foundation large language models for medical applications. arXiv preprint (xXi:22402.12442 2024.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. 2023.
Yunlu Yan, Chun-Mei Feng, Yuexiang Li, Rick Siow Mong Goh, and Lei Zhu. Federated pseudo modality generation for incomplete multi-modal mri reconstruction. arXiv preprint arXiv:2308. 10910, 2023a.
Yunlu Yan, Chun-Mei Feng, Mang Ye, Wangmeng Zuo, Ping Li, Rick Siow Mong Goh, Lei Zhu, and CL Chen. Rethinking client drift in federated learning: A logit perspective. arXiv preprint arXiv:2308. 10162, 2023b.
Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yong Xu, Yuexiang Li, and Yefeng Zheng. Cross-modal vertical federated learning for mri reconstruction. IEEE Journal of Biomed- ical and Health Informatics, 2024a.
Yunlu Yan, Lei Zhu, Yuexiang Li, Xinxing Xu, Rick Siow Mong Goh, Yong Liu, Salman Khan, and Chun-Mei Feng. A new perspective boost performance fairness for medical federated learning- In International Conference Medical Image Computing and Computer-Assiste Intervention, pp. 13-23. Springer, 2024b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1667, 2024.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogencous federated learning: State-of-the-at and research challenges. ACM Computing Surveys, 56(3):1 2023.
Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. benllm-bench: Realistic benchmarks for federated learning of large language models. arXiv preprint ar arXiv:2406. Atl::2406.04845.. 2024a.
15

---

Published as a conference paper at ICLR 2025
Rui Ye, Wenhho Wang. Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via fed- erated learning. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6137-6147, 2024b.
Xuefei Yin, Yanming Zhu, and Jiankun Hu. A comprehensive survey of privacy-preserving federated learning: taxonomy, review, and future directions. ACM Computing Surveys (CSUR), 54(6); 1-36, 2021.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint (.tiv:2339.122888. 2023.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, 2024.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang. and
Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6915 6919. IEEE, 2024.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning mmlr.
Research. pp. 26311-26329 PMLR, 17-23 Jul 2022. URL hitps: press v 62/ zhang22p. html.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh ternational Conference on Learning Reprexentutions, 2023a. URL https:/ /ttp:::///pperreiee.. net http://ouunnddd1122222
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 11Xiv:1806.004 00582, 2018.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mi-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2025
Table 6: Detailed information of NLU datasets.
Datasets
RTE
COLA
QNLI
20NG
Task
Inference
Linguistic acceptability
Inference
Text classification
# Labels
2
2
2
20
# Train
2.49k
8.55k
105k
11.31k
Test
277
1.04k
5.46k
7.53k
Clients (Total)
5
Clients (Train)
5
5
5
5
Metrie
Accuracy
Matthews correlation
Accuracy
Accuracy
Table 7: Detailed information of NLG datasets.
Datasets
MetaMathQA
Alpaca-GPT4
Fed-Aya
Fed-ChatbotIT
Fed- Chtd
Task
Question answering Single-turn chat Single-turn chat Single-turn chat Multi-turn chat
Domain
Math
General
Multilingual
General
Multilingual
Train
395k
52k
25,513
6166
52,703
Clients (Total)
20
237
100
Clients (Train)
2
2
10
A EMPIRICAL ANALYSIS
In this section, conduct empirical analysis reveal the update pattern of LoRA in FL. Drawing on RTE as the running example, we first record the average Frobenius norm of global AW, which serves indicator of the task-specific knowledge that LoRA acquires for downstream tasks (Si et al., 2024). Notably, the global A W for FRLcRA is the residual accumulation term in Eq. (13). Additionally, we track the standard deviation among all local low-rank matrices across clients during the update process, which provides a measure of the level of client drift.
The results versus the communication rounds, illustrated in Figure (c) and (d), show that data het- erogeneity hinders LoRA from effectively capturing global knowledge and incurs significant client drift among local low-rank matrices, resulting in serious performance degradation from 76.89% to 70.75%. In contrast, our proposed method, FRLORA, exhibits distinct update pattern and enhanced learning capacity under data heterogeneity. As we can see, FRLORA learns more global knowledge throughout the entire communication process compared FedAvg (non-IID). Even compared to Fe- dAvg (IID), FRLoRA also learns more global knowledge in the carly stages (before 50 rounds). This indicates that the update mechanism of FRLORA facilitates the rapid capture of global knowledge. Moreover, we further found that FRLORA RA can significantly mitigate client drift due to its reinitial- ization mechanism. As result of these positive effects, FRLORA improves accuracy from 70.75% to 75.81 %.
B EXPERIMENTAL DETAILS
Our experimental setup follows previous empirical studies (Ye et al., 2024a;b). Detailed information about the datasets used for NLU tasks is provided in Table 6, while Table 7 outlines the datasets for NLG tasks. The training configuration is summarized Table
B. DATASETS OF NLU TASKS
As shown in Table 6, the NLU experiments are conducted on four datasets, involving different tasks:
RTE (Wang al., 2019): A two-class classification task based on news and Wikipedia text, where the goal is to determine whether one sentence entails another.
COLA (Wang et al., 2019): It consists of English sentences annotated with acceptability judgments, indicating whether each sentence is grammatically correct based on linguistic theory.
17

---

Published as a conference paper at ICLR 2025
Table 8: Detailed information of training configuration.
Configuration
Batch size
Rounds
Local steps
Learning rate
RTE
32
16
32
200
10
2e-4
COLA
32
16
32
200
30
2e-4
QNLI
32
16
32
200
30
2e-4
20NG
200
30
2e-4
MetaMathQA
200
10
5e-4
Alpaca-GPT4
16
32
64
200
10
5e-4
Fed-Aya
4
16
32
200
10
2e-4
Fed-ChatbouT
4
16
32
100
5
2e-4
Fed- WildChat
4
16
32
100
10
2c-4
QNLI (Wang et al., 2019): A sentence pair classification task where the goal is to determine if context sentence from a paragraph contains the answer given question.
20NG (Lang, 1995): A text classification task to categorize news documents into 20 differ- ent newsgroups.
The 20NG dataset contains categories, while the other three datasets consist of two categories. For all datasets. partition their training set into 5 clients using Dirichlet distribution sampling (Ds ~ Dir(B)), where a smaller 3 indicates higher data heterogeneity. By default, / is set to 0.5. All clients participate in the training process during each round. For evaluation, report the accuracy (%) on the validation set RTE, COLA, and QNLI, and Matthews correlation (%) on the test of 20NG. # Train and Test represent the number of training and testing samples.
B.2 DATASETS NLG TASKS
Training Datasets. Table 7 provides detailed information on five training datasets for various NLG tasks, including:
MetaMathQA (Yu al., 2023): A dataset containing 395k question answering pairs for math problem solving, augmented from the training sets of GSM8K (Cobbe et al., 2021) and MATH (Yu et al., 2024).
Alpaca-GPT4 (Peng et al., 2023): A collection 52k English instruction cow.ucctooooooooiinnggo exam- ples about general knowledge generated by GPT-4, used fine-tune large language models (L.LMs).
Fed-Aya (Ye et al., 2024a): A multilingual instruction tuning dataset with 38 clients and 25,513 samples, including high-resource and low-resource languages.
Fed-ChatbouT (Ye et al., 2024a): It is a collection of numan-annotated preference data from 237 clients with 6166 data samples that capture diversities of realistic cases in single-turn query of LLMs.
Fed- Cil--ithhhat (Ye et al., 2024a): It is a collection of multilingual conversations between humans and ChatGPT, containing 100 clients with 52,703 data samples, representing real- world multi-turn interactions between humans and chatbots.
For MetaMathQA and Alpaca-GPT4, we select 20k samples and partition them in an IID manner, with 10 and 20 clients, respectively. At each round, 2 clients are randomly selected participate in the training. Fed-Aya, Fed-ChatboulT, and Fed-WilChat are real-world benchmarks with data het- erogeneity constructed by Ye al. (2024a), where randomly select 10, and clients per round for training. Additionally, we fine-tune LLMs using instruction tuning with Alpaca's instruction template (Taori et al., 2023).
Evaluation Benchmarks. We evaluate fine-tuned LLMs on the following test benchmarks:
GSM8K (Cobbe et al., 2021) & Math (Yu et al., 2024): Both of them are question- answering datasets focused math problem solving. We use their test sets for evaluation.
18

---

Published as a conference paper at ICLR 2025
MT-Bench (Zheng et al., 2024): A benchmark designed to evaluate both one-turn (MT-1) 1) and two-turn (MT-2) conversational capabilities of language models across various tasks, including writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities.
Vicuna (Chiang et al., 2023): This benchmark evaluates the one-turn cntruuctionfoolloiint capability of language models, covering a range of topics including coding, writing, math, counterfactual reasoning. Fermi problems, common sense, and roleplay.
Ref-GPT4 (Ye et al., 2024a): It serves as an in-domain evaluation benchmark, where 50 unseen data samples are randomly selected the test set. For instance, when model is fine-tuned on the Fed-Aya dataset, the unseen samples are drawn from Fed-Aya for evalu- ation.
For GSM8K and Math, problems. For the other three benchmarks, we input the outputs of the LLMs into GPT-4 using instruction templates (Ye al., 2024a), and GPT-4 score them scale of to 10, as shown in Table 12 and 13. The prompt template used in GPT-4 Judger from Ye et al. (2024a:b).
accuracy (%) evaluate the performance of LLMs in solving math
B.3 BASELINES
To demonstrate the effectiveness of our approach, we employ several state-of-the-art heterogeneous FL methods as baselines:
FedAvg (McMahan et al., 2017): A pioneering algorithm that trains models on local datasets and updates the global model by averaging the local model parameters.
FedProx (Li et al., 2020b): guides local training by introducing proximity term between the local model and global model parameters.
SCAFFOLD (Karimireddy et al., 2020): During local training, it corrects local gradients by introducing control variates.
FedAvgM (Hsu et al., 2019): It updates the global model through the momentum update mechanism.
FedAAagrad Fed FedAdam (Reddi et al., 2021): They integrate adaptive op- the difference lying in the use of different computation strategies to obtain the global update. This global update is then applied to the global model through the momentum update mechanism.
timization methods into FL, with
FFA-LORA (Sun et al., 2024): It addresses privacy concerns and mitigates data hetero- geneity issues by fixing the A matrix for each client and only optimizing the B matrix.
Aside from FFA-LoRA, the remaining methods are centered on FFT-based FL., which we integrate with LoRA. We utilize the hyperparameters specified in Ye al. (2024a) for these methods.
B.4 MODELS
We use RoBERTa-base and I.aMM-2-7B as the base models for NLU and NLG tasks, respectively. The total number of model parameters (# Param.base base) and trainable parameters (# Param.trainable) are shown in the Table 9.
Table 9: Illustration of the number of model parameters.
Contigure
Param.base
# Param.trainable
RoBeRTa-base
r 16 & a = 32
125.82M
1.77M
LLaMA- 2-BB
16 & a = 32
6738M
2.09M
19

---

Published as a conference paper at ICLR 2025
Table 10: Experimental results under various ranks, where MetaMathQA is used the training dataset. Avg. is the average result of corresponding metrics, and the best results are marked bold.
16
Method
GSM8K
Math
Avg.
GSM8K
Math
Avg.
GSM8K
Math
Avg.
FedAvg
32.67
4.64
18.65
34.95
4.48
19.71
37.45
5.38
21.41
FedProx
32.29
4.32
18.30
35,40
4.66
20.03
36.39
4.98
20.68
SCAFFOLD
32.97
4.70
18.84
35.78
5.08
20.43
32.37
4.64
18.50
FedAvgM
32.44
4.42
18.43
34.79
4.64
19.71
35.57
4.72
20.14
FedAdagrad
28.65
4.18
16.41
29.64
4.06
16.85
31.76
4.86
18.31
Fed
30.32
4.00
17.16
30.09
4.04
17.06
33.96
4.70
19.33
FedAdam
31.23
4.14
17.68
31.84
4.12
17.98
34.26
5.18
39,44
FFA-LORA
25.17
3.60
14.38
28.05
3.78
15.91
31.00
4.50
17.75
FRLORA (Ours)
39.57
5.60
22.58
44.27
5.22
24.74
45.56
6.88
26.22
Table 11:
Experimental results under various LLMs, where Meth is used the training dataset. Avg. is the average result of corresponding metrics, and the best results are marked in bold.
Qwen2-1.5B
Gemma-2B
LLaMA-2-7B
Method
GSM8K
Math
Avg.
GSMSK
Math
Avg.
GSM8K
Math
Avg.
FedAvg
16.30
1.66
8.98
36.08
15.00
25.54
34.95
4.48
19.71
FedProx
16.98
2.20
9.59
35.63
14.30
24.96
35.40
4.66
20.03
SCAFFOLD
14.02
1.56
7.79
37.07
14.52
25.79
35.78
5.08
20.43
FedAvgM
10.31
0.76
5.53
36.69
13.80
25.24
34.79
4.64
19.71
FedAdagrad
43.82
2.92
23.37
34.64
14.14
24.39
29.64
4.06
16.85
FedYogi
0.83
0.40
0.61
34.49
13.84
24.16
30.09
4.04
17.06
FedAdam
1.28
0.48
0.88
34.57
13.46
24.01
31.84
4.12
17.98
FFA-LoRA
36.01
4.24
20.12
32.90
14.28
23.59
28.05
3.78
15.91
FRLORA (Ours)
52.08
17.34
34.71
42.30
15.24
28.77
44.27
5.22
24.74
C ADDITIONAL EXPERIMENTS
C.I VARIOUS RANKS
In this section, investigate the impact of varying ranks. Specifically, adjust the rank to {16, 32, 64) while keeping the scale factor 2 fixed at 2, with other experimental settings remaining constant. The experimental results MetaMathQA methods exhibit a consistent trend: higher ranks yield improved performance. This is attributed to the greater learning capacity of larger ranks, enabling more effective representation of knowledge across diverse clients. These findings corroborate our claim in Moreover, FRLORA consistently outperforms all other methods different ranks, indicating its scalability and robustness rank variations. Besides, we also observe that F FRLoRA (r=16) achieved even higher performance than FedAvg (r=64), demonstrating its effectiveness addressing the constrained parameter space.
shown in Table 10. As observed, most
C.2 VARIOUS LLMS
We further evaluate the effectiveness of our method across different LLMs. To this end, we selected two models with varying architectures and sizes: Qwen2-1.5B (Yang et al., 2024) and Gemma- 2B (Team et al., 2024), and conducted additional experiments on MetaMathQA. The r and a were set to 32 and 64, respectively, while keeping all other experimental settings unchanged. Specifically,
20

---

Published as a conference paper at ICLR 2025
FodYogi and FedAdam achieved poor results, with average accuracies (Avg.) of only 0.61% and 0.88%, respectively. On the other hand, FedAdagrad exhibited significant performance improve- ments compared to other baselines, which contrasts with its behaviour Gemma-2B and Llama-2- 7B. Notably, FRLORA consistently outperforms all baselines across the three LLMss, even achieving superior performance with Qwen2-1.55 1.52, smaller model compared the other two. This demon- strates the strong generalizability of our method across different LLM architectures and scales.
Table 12: Examples of judgement by GPT-4 on Vicuna.
Question: How many text messages are sent globally a minute? Try to explain your answer. Your explanation should take the reader through your reasoning step- step-by-step bt-step.
FedAvg; Let's 's' take a look at the data and see what we can uncover. According to a report by The Guardian, there were 47.9 billion text messages sent globally in 2019.
Now, let's s break down:
The report states there were 4.6 trillion text messages sent globally in 2019. This equals to 12.6 billion text messages sent per day.
According to a report by GlobalData, there were 6.18 trillion texts sent in 2019, which is roughly 17 billion texts per day.
To explain this further, we can look the data from the reports and consider that the reports are from 2019, and that it is likely that the number of text messages sent has increased since then.
In the end, there is no definitive answer as how many text messages are sent globally in minute, but we can estimate that average, around 12-17 billion messages are sent in minute. The figure could be higher or lower depending on the source and the time period considered.
GPT- Judger: The assistant's 's response is detailed and well-reasoned, but there is significant error in the calculations. The assistant correctly cites data from The Guardian and GlobaiData, but the con- version from daily minute messages is incorrect. there are 12.6 billion 17 billion texts sent per day, this does not translate the same number being sent per minute. The assistant should have divided the daily figure by the number of minutes in a day (1440) to get the correct answer. This error significantly impacts the accuracy and relevance of the response. Therefore, the rating is lower. Rat-
ing: [[3]]
FRLORA: Unfortunately, there is no definitive answer to the number of text messages sent globally in minute, as this figure would require the collection of data from all teleconmunication companies and mobile phone operators across the globe. It difficult to estimate the rumber of texts sent per second, let alone per minute. However, uccording to Statista, in 2022, the number of text messages sent in the U.S. alone was 114.7 billion per month, which equals to about 2.978 million per day. This means that a large number of text messages are being sent globally every minute, but it is difficult to determine an exact figure.
To explain further, the number of text messages sent in a minute would depend on several factors such as the population of the countries, the frequency of texting, and the number of mobile phone subscribers. For example, in countries with a large population and a high rate of mobile phone penetration, such as India and China, the number of text messages sent per minute would be significantly higher than in countries with smaller populations
lower mobile phone usage, like Sweden or Norway. Additionally, the number of messages sent would also vary depending on the time of the day, as people tend to send more texts during peak hours, such as during office hours or after work.
In conclusion, the number of text messages sent globally in minute not a fixed figure but varies depending on several factors. While we cannot give an exact number, we can say that texting is a popular form of communication and a large number of texts are being sent globally every minute.
GPT4 Judger: The assistant's 's response is highly informative and detailed. It explains why it's difficult
provide an exact number of text messages sent globally per minute, citing the need for data from all telecom- munication companies and mobile phone operators worldwide. The assistant also provides a statistic from Statista to give a sense of the scale of text messaging in the U.S., which helps to illustrate the global scale. The assistant further explains the factors that would influence the number of texts sent per minute, such as population size, mobile phone usage, and time of day. The assistant's 's response is accurate, relevant, and provides good depth of information, even though couldn't 't provide specific number. Rating: [[9]]
21

---

Published as a conference paper at ICLR 2025
Table 13: Examples of judgement by GPT-4 on Vicuna.
Question: What are the main differences between Python and JavaScript programming languages?
FedAvg: Python and JavaScript are two popular programming languages that have many similarities. but there are some key differences between them. Here are some of the main ones:
1. Python is object-oriented progranming language, while JavaScript is a functional programming language. Object-oriented programming is used to organize and structure code by grouping related data and methods into objects. Functional programming. on the other hand, focuses on functions and their interactions with other functions rather than on objects and classes. 2. Python is a high-level proggranm programming language, while JavaScript is low-level programming language. This means that Python requires less manual memory management and handles memory allocation automatically, while JavaScript requires the programmer to manage memory manually. 3. Python has a built-in collection of standard libraries, while JavaScript does not. Python's 's standard library is extensive, making it easier for programmers to write code for common tasks without having to reinvent the wheel. JavaScript, on the other hand, requires the programmer write their own code use third-party libraries to perform the same tasks. 4. Python has syntax that is mare readable and natural than JavaScript, making easier for programmers understand and write code in Python. JavaScript has a syntax that is more compact and concise, which can make it easier to write code quickly. 5. Python is an interpreted programming language, while JavaScript is compiled programming language. This means that JavaScript code is first translated into machine code before it is executed, while Python code is executed directly by the interpreter. 6. Python is a dynamic programming language, while JavaScript is a static programming language. This means that Python code can be executed at any time and the results may change, while JavaScript code must have a specific order and cannot change.
Overall, the choice between Python and JavaScript depends on the specific needs and preferences of the programmer While both languages have their strengths and weaknesses, they are both widely used and have a large developer communitty.
GPT-4 Judger: The assistant's response is detailed and provides a comprehensive comparison be- tween Python and JavaScript. However, there are some inaccuracies in the information. For instance,
JavaScript is not low-level programming language, it's also high-level like Python. Also, JavaScript not a compiled language, it's interpreted just like Python. Furthermore, JavaScript is not purely functional programming language, it supports both object-oriented and functional programming paradigms. Lastly, JavaScript is not a static language, it's dynamic like Python. These inaccuracies
significantly affect the quality of the response. Therefore, the rating is: [[3]]
FRLORA: Python and JavaScript are both popular programming languages used for web development and general-purpose scripting. However, there are several key differences between the two languages:
1. Language Typing: Python is strongly ryped language, which means that variahles must have specific data type. JavaSeript, on the other hand, a loosely typed language, meaning that variahles can have any data type. Syntax: Python has a more rigid syntax compared to JavaScript. For example, in Python, indentation is used to indicate structure, while in JavaScript, curly bruces are used to denote blocks of code. 3. Data Structures: In Python, there are several built-in data structures such as lists, dictionaries. sets, and tuples. JavaScript, on the other hand, has more limited set of built-in data structures, including arrays and objects. 4. Functions: Functions in Python are first-class citizens, which means that they can be assigned, passed as arguments, and returned from other functions. In JavaScript, functions are second- class citizens, meaning that they can only be passed as arguments and returned from other functions. 5. Programming (OOP), but Python
funcciins
Programming: Both languages support Object-Orientee has more explicit focus on with features such as classes and inheritance. JavaScript has more limited support for OOP, with only classes and prototypes. 6. Web Development: Both languages can be used for web development, but Python is typically used for back-end development. while JavaScript is used for fromt- end development. Python can be used to build web applications using frameworks like Django or Flask, while JavaScript is used to build interactive client-sie side interfaces using frameworks like React or Angular.
Overall, both Python and JavaScript are powerful and versatile languages, but they have different strengths and weaknesses, and they are used for different types of development projects.
GPT-4 Judger: The assistant's response is highly informative, accurate, and detailed. covers the main differences between Python and JavaScript, including language typing, synlax, data structures, functions, object-oriented programming. and their uses web development. The assistant also provides examples to clarify the points. The response stll-structured and easy to understond, making it very helpful for someone trying to understand the differences between these two programming languages. Therefore, 1 would rate this response highly, Rating: [[9]]
22

---

Published as a conference paper at ICLR 2025
Table 14: Communication cost per round Alpaca-GPT4 with LLaMA-2-7B, where is 32. Cost.P denotes the communication overhead from participating training clients, Cost.NP represents the overhead from pon-patiicipating clients, and Cost.T indicates the total communication cost.
Method
Cost.P (MB) Cost.NP (MB) Cost. T (MB) Vicuna MT-1 MT-2 MT-Avg Avg.
Partial Participation Setting
FFT-based FL
I3476x 2
0
26952
Out of Memory
FedAvg
8.388 X 2
0
16.77
7.925
4.650
2.025
3.346
4,486
FRLORA
8,388
4.194 18
92.26
8.044
4.775
2.481
3.635
4.733
Full Participation Setting
FFT-based FL
x 20
269520
Out of Memory
FedAvg
8.388 x 20
0
167.76
8.039
4.833
2.458
3.645
4.743
FRLORA
8.388 20
0
167.76
8.125
4.984
2.806
3.895
4.952
C.3 COMMUNICATION OVERHEAD
In the full participation setting, where all clients are involved in training every round, FRLORA in- curs no additional overhead, as local models remain consistent through parameter synchronization. However, in the partial participation setting where only subset of clients participate in training ery round, F FRLORA introduces extra communication overhead because inactive clients must receive the global low-rank matrix from the server for the parameter synchronization. Since synchronization occurs through the global low ow-kran matrices, the increased communication cost remains acceptable. As shown in Table 14, the additional overhead introduced by FRLoRA per round is 75.49 MB, which accounts for only 0.28% of the communication cost in FFT-based FL. This demonstrates that SoRA-based FL can significantly reduce the communication overhead when fine-tuning LLMs. Furthermore, FRLORA does not increase the communication cost per client. As the communication channels between each client and the server are independent and parallel, the overhead from inactive clients does not affect overall communication efficiency.
Besides, we further present the performance in the full participation setting. It can be observed that under this setting, FRLoRA does not incur any additional communication overhead while main- taining superior performance. This also indicates that the performance improvement in the partial participation setting does not stem from the increased communication overhead caused by inactive clients.
C.4 EFFICIENCY OF SVD COMPUTATION
Table 15: Time and peak memory for the SVD computation.
We conducted analysis of the time and mem- ory requirements for the SVD computation in our method. As shown in Table 15, the computa- tional cost of SVD is minimal in terms of mem- ory, with peak memory consumption remaining under MB for both models. While the time cost scales with model size. SVD is performed only once, rendering its overall impact negligible compared to the whole training phase.
Model
Time
Peak Memory
RoBERTa-bass,
1.48
0.25 MB
I.LaMA-2-7B
197.82 s
0.65 MB
C.5 ANALYSIS OF CLIENT DRIFT
In this section, we further analyze the effectiveness our method mitigating client drift. To achieve this, calculate the cosine similarity among local updates AW; across different clients. We compare the results of FedAvg and FRLORA on RTE, using the same number of training rounds to ensure fair evaluation. The results show that the cosine similarity between clients in FedAvg
23

---

Published as a conference paper at ICLR 2025
Cliints
Clients
Cilents
1.0
10
1.00
0.71
0.73
0.45
0.72
1.00
0.93
0:83
0:54
0.94
1.00
0:67
3.9
0.
Clients
0.71
1.00
3.0
Ciients
1:00
0.66
0:95
0.91
0.
0.05
0.00
0:33
0.53
0.83
0.7
0:3
1:00
0.78
0.50
0.45
0.46
1000
3.8
0:35
1.00
0.50
06
0.72
0.74
0.74
0.66
1.00
2.5
0.94
0.94
0.96
0.50
2.00
0.5
0:87
0:83
0.29
0.52
1.00
FodAvg Figure 3: Illustration of the cosine similarity distribution among local updates across different clients on RTE.
(a) FedAvg (non-llI)
FRLORA (non-lID)
1.4
FRLORAVV2
75
1.2
70
Accuracyy (%)
88
FadAddnn
Training
FadAugg
SCAFFOLD
60
Fediog
0.4
Faskiagged
55
FadYopi
0.2
75
B
Round
Figure 4: Accuracy comparison of different FL methods in terms of various 3 on RTE.
Figure 5: Illustration global training loss versus communication rounds RTE.
(IID) (see Figure (b)) is distributed significantly higher than in FedAvg (non-IID) (see Figure (a)), and FRLORA (non-IID) (see Figure 3 (c)) exhibits higher distribution compared to FedAvg (non-lID). This further demonstrates the effectiveness of RLORA in reducing client drift, promoting more consistent model convergence under data heterogeneity.
C.6 EFFECT OF DATA HETEROGENEITY
In this section, we investigate the impact of data heterogeneity on our method by varying the 3 of the Dirichlet distribution across the values 0.1, 0.3, 0.5, 0.7, 0.9 the RTE dataset, where smaller values of 3 correspond to higher data heterogeneity. As shown in Figure 4, we can observe that the accuracy of all methods increases with the increase of f3, and FRLORA significantly outperforms the other methods at different values B. Moreover, the gap is larger when / is small, indicating the effectiveness of FRLORA in addressing data heterogeneity.
C.7 ANALYSIS INITIALIZATION
To further investigate the impact of initialization on FRLORA, we compared the convergence behav- ior of FRLORA-V2 with method. As stated in $4.4, FRLORA-V2 variant that reinitializes the low-rank matrices to zero and Gaussian noise in each round. The results in Figure 5 demon- strates that FRLORA-V2 exhibits slower and less stable convergence compared to FRLORA. This demonstrates the effectiveness of FRLORA's 's initialization strategy in addressing the convergence challenges associated with standard initialization methods, enabling faster and more stable conver- gence. This facilitates AW' in effectively capturing global knowledge.
D ADDITIONAL THEORETICAL ANALYSIS
In this section, we present additional theoretical analysis the rank of the global update.
24

---

Published as a conference paper at ICLR 2025
To begin with, define the global update of FedAvg with LoRA
AW FedAng = B
Here, B is a di r matrix and A is arr x d2 matrix. Based on Eq.(12), we have,
rank(AW FedAug) ram())n S
Then, we define the global update of FRLoRA as,
AW FRLORA AW' +AW +AW
=B AG + (B BG ....4 + AB BBA
AT + B + BB A + + B.A
Obviously, WT
PRLORA can be rewritten as,
AW AWT FRLORA [-TBa B B B [Ag: Ag; AB:...;
Here, the size of [-TB]: B B : is dj x r(T + 1), and the size of [A:; A Algi...; A I is (T + +1 x d2. Thus, we have,
rank A AW FRLORA) & min(r(T + 1), d2)
and
rank ( WW 2 rank( A A FedAug)
Finally, we note that rank (A WT FRLORA) can be greater than rank (WW FedAog) only except that B Ag...... BB Ag are completely linearly independent.
25


Reviews:
[{"review_id": "VWJEfEiMWZ", "reviewer": "Reviewer_RHQ7", "summary": "This paper introduces FRLoRA, a novel Federated Learning (FL) method that addresses limitations in global knowledge learning by combining residual low-rank adaptation with periodic recalibration of local models. FRLoRA expands the effective parameter space during global updates and mitigates client drift by reinitializing local model components with principal components from the pre-trained model.  Extensive experiments across various natural language processing tasks demonstrate FRLoRA's superior performance compared to existing FL methods.", "questions": "Can the authors should further explore and demonstrate the method's effectiveness in a specific application domain and address the communication cost limitations to enhance its practical value.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly presented. The proposed method is sound. Experiment results are comprehensive and validate the quality improvement of the proposed method.", "weaknesses": "This paper presents a novel Federated Learning method, FRLoRA, which aims to improve global knowledge learning. However, the paper's practical value is limited due to two main weaknesses:\n\nLack of a compelling use case: The authors fail to clearly motivate the need for their proposed method within a real-world application scenario. While the technical contributions may be sound, the lack of a clear practical context diminishes the paper's impact and relevance.\n\nHigh communication cost: Despite claims of efficiency, the method introduces significant communication overhead, hindering its practicality in real-world deployments, where many LoRA customers care about.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PmNrbYXbAw", "reviewer": "Reviewer_pHac", "summary": "This paper introduces a method known as FRLoRA (Federated Residual Low-Rank Adaption) to address the problem of Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs) in Federated Learning (FL) scenarios. FRLoRA updates global model parameters by introducing a residual low-rank matrix product and reinitializes local low-rank matrices in each training round to mitigate client drift. The paper compares FRLoRA with existing FL baseline methods across multiple datasets in natural language understanding and generation tasks, demonstrating its consistent superiority and validating the effectiveness of the proposed FRLoRA.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. FRLoRA effectively addresses the significant impact of data heterogeneity on PEFT in FL scenarios by accumulating the product of residual low-rank matrices, enabling the global model to learn more comprehensive knowledge.\n2. By reinitializing local low-rank matrices at each training round, FRLoRA alleviates client drift issues, enhancing the convergence and performance of the model.\n3. Extensive experiments validate the performance improvements of FRLoRA over existing methods across various natural language processing tasks.", "weaknesses": "1. The updates derived from averaging the uploaded matrices A and B at the server differ from the averaged updates across clients, and this discrepancy is likely to amplify in heterogeneous data scenarios. [Improving LoRA in Privacy-preserving Federated Learning]\n2. Although the paper proposes extending the parameter space through residual updates, it does not clearly specify whether the rank of the model parameters is strictly improved after each residual update, lacking further explanation on how this mechanism operates, particularly regarding its impact on model rank and representational capacity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vj6t6lPaxA", "reviewer": "Reviewer_vtoo", "summary": "The paper presents Federated Residual Low-Rank Adaptation (FRLoRA), a federated learning method that improves LoRA for LLMs. FRLoRA overcomes challenges like constrained parameter spaces and client drift by updating in a higher-rank parameter space and reinitializing local low-rank matrices based on the principal singular values of pre-trained weights. Experiments show that FRLoRA consistently outperforms existing federated learning methods across benchmarks in NLU and NLG tasks.", "questions": "Q1: Could the authors provide further analysis on the computational impact of using SVD for reinitialization in terms of time and memory requirements?\n\nQ2: Could the authors provide further analysis on the behaviour on Table 3 and 4?\n\nQ2:How does FRLoRA handle scenarios where singular value decomposition may be unstable? Are there alternative initialization strategies?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a unique adaptation of LoRA for federated learning with residual low-rank updates and reinitialization in the principal singular space.\n\n2. Extensive experiments on multiple benchmarks confirm FRLoRA’s robust improvements across NLU and NLG tasks in various federated settings.", "weaknesses": "1. The reinitialization of local low-rank matrices and the use of SVD may introduce significant computational overhead, particularly for large-scale models. An in-depth analysis of this aspect or potential optimizations would provide added value to the paper.\n\n2. Although the method shows performance improvements overall, in Table 3, the proposed approach does not appear to dominate across all evaluations. A similar pattern is observed in Table 4, indicating that while the method is strong, it may not consistently outperform existing baselines in certain subcases.\n\n3. While the paper briefly touches on SVD stability, it lacks a detailed analysis of potential impacts and does not explore alternative methods for scenarios where SVD might lead to convergence issues. Additional insights on this aspect could reinforce the robustness and general applicability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yu4OQH7pPW", "reviewer": "Reviewer_Acqq", "summary": "This paper explores the challenges of using LoRA in Federated Learning (FL) for fine-tuning large language models (LLMs) on non-IID data. The authors identify that LoRA in FL struggles to learn global knowledge due to two key issues: extrinsic client drift and an intrinsic constrained update space. The authors propose FRLoRA (Federated Residual Low-Rank Adaptation) to overcome these challenges. FRLoRA initializes the LoRA matrices  $A$  and  $B$  using the Singular Value Decomposition (SVD) of the initial weights  $W_0$, while freezing the residual initial weight. During training, the locally updated LoRA matrices are aggregated and combined with the residual weight, creating the initial model for the next training round. This method addresses both client drift and limited update space, of which theoretical justification is presented. The experiments on nine language model benchmarks show that FRLoRA consistently outperforms various baseline methods.", "questions": "## Questions \n**Does FRLoRA Fully Address the Intrinsic Challenge?**: While FRLoRA expands the parameter space globally, it does not necessarily address the constrained optimization space at the local level. Can the expanded global space fully mitigate intrinsic issues without modifying local optimization constraints?\n\n**Catastrophic Forgetting at Low Ranks**: Direct updates to $W_0$ in a low-rank context may risk catastrophic forgetting. Can FRLoRA’s SVD initialization mitigate this risk, or could additional ablation studies on learning rates clarify this?\n\n**Comparison to Chain of LoRA’s Residual Framework**: Both FRLoRA and Chain of LoRA use residual structures for fine-tuning. How does FRLoRA’s SVD-based initialization impact its effectiveness relative to the iterative approach of Chain of LoRA? Further analysis of residual efficacy could enhance the comparative evaluation.\n\n**More Diverse Data Heterogeneity**: Is there an empirical study for NLG with severe heterogeneity?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Addressing Non-IID Challenges in FL with LoRA:** FRLoRA tackles an important challenge in federated fine-tuning of LLMs by addressing both client drift and constrained parameter space through aggregated $\\Delta W$ updates via SVD.\n\n**Theoretical Justification:** The authors provide a simple theoretical justification of FRLoRA using a rank analysis of the parameter space, although the rank analysis can be far from the practice.\n\n**Comprehensive Evaluation:** The study provides extensive experiments across nine benchmarks, covering Natural Language Understanding and Generation, validating FRLoRA’s efficacy in terms of performance and communication cost.", "weaknesses": "**Insufficient Evidence on the Problem Formulation:** The authors characterize the client drift using Figure 1 (c-d) but it is not convincing. To be specific, the norm of $\\Delta W$ and the standard deviation can be simply reduced by the other factors, e.g., learning rate. Perhaps, it would be better to (i) focus on a single round rather than the sequence of rounds and (ii) observe the distribution of cosine similarity rather than the standard deviation. It is also helpful to analyze the other FL methods.\n\nIn addition, the authors claim that the constrained parameter space is one of the major problems in the naive method (FedAVG + LoRA). The simple rank analysis supports this claim. However, it is not thoroughly studied in empirical analysis. Hypothetically, the constrained parameter space problem can be simply addressed by employing large rank $r$.\n\n\n\n**Limitations in Non-IID Testing Scenarios:** Experiments use a Dirichlet (0.5) Non-IID setting with five clients for binary classification, while NLG tests are performed on IID data. Broader Non-IID experiments would better showcase FRLoRA's robustness instead of just increasing the number of benchmarks.\n\n\n**Limited Comparision to Existing Works:** The baselines used for comparison, such as FedYogi, FedAdam, and FedProx, do not integrate LoRA, making comparisons with these methods less relevant. Only FFA-LoRA offers a directly comparable baseline. Apparently, there is another LoRA method for FL, FlexLoRA (https://arxiv.org/abs/2402.11505, Feb 2024), in which a similar idea of using SVD per round is proposed. In addition, the idea of using the residual is analog to the one in Chain of LoRA (https://arxiv.org/abs/2401.04151, Jan 2024), which needs to be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0HySPmWID", "reviewer": "Reviewer_Ddf1", "summary": "In this paper, the author proposes a novel framework named federated residual low-rank adaption to effectively fine-tune the pretrained large language model in a privacy-preserving manner. The proposed method addresses both the intrinsic problem caused by constrained parameter space and the extrinsic problem caused by the drift among clients. Extensive experiments on several datasets have shown the effectiveness of the proposed method.", "questions": "The items listed in the weaknesses. If the author can address or partially address these problems, I will be pleasure to improve my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of enlarging the parameter space when finetuning large laugunge model in federated learning is promising.\n2. The paper is well organized and the related works are thoroughly summarized.\n3. The discussion of the weakness of exsiting related works and the novelty of the proposed method are clear.\n4. The author conducts comprehensive experiments to demonstrate the effectiveness of the proposed method, which is convincing.", "weaknesses": "1. In Eqs. (6) and (7), the author initializes the local low-rank matrices by the low-rank matrices reconstructed from the top singular values, while remaining the residual part as the global model . In my opinion, this makes the low-rank matrices retain the most information in the pretrained model, while the information remained in the global model can be significanly reduces.  However, in FL, the global model should contain more generalized information from the pre-trained model, which is contrast to the above parameter decoupling operation. So I am curious that whether this operation can reduce the generalization capacity or convergence speed of FL?\n\n2. The motivation of the proposed low-rank matrice initilization lacks further discussion. Can the author provide more detailed discussion about this by such as landscape visualization?\n\n3. Why the global model are initialized by the residual matrice? It seems that directly using the pre-trained model parameters as the global model can provide a more generalized initialization during the FL training. Can the author provide a more detailed discussion about this?\n\n4. In Eqs. (13) and (14),  the author claims that by using residual accumulation, the model fine-tuned space can be expanded. However, Eq. (14) seems to only give a upper bound of the summed matrices rather than increasing its lowe bounder of the summed matrices. So it can not guarantee that the rank of the summed matrices can be increased by such a residual accumulation operation. Can the author provide a more in-depth discussion about this?\n\n5. Actually, the standard PEFT by LoRa in FL can also be formulated as Eq. (13), with $\\delta W^t$ as the global update of the LoRa parts at each global round. So what is the real differences between the standard method and the proposed FRLoRA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)