Federated Residual Low-Rank Adaptation of Large Language Models

Authors: Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, Lei Zhu
Primary area: foundation or frontier models, including LLMs
Keywords: Large Languagel model, Federated Learning, Parameter-Efficient Fine-Tuning

Abstract:
Low-Rank Adaptation (LoRA) presents an effective solution for federated fine-tuning of Large Language Models (LLMs), as it substantially reduces communication overhead. However, a straightforward combination of FedAvg and LoRA results in suboptimal performance, especially under data heterogeneity. We noted this stems from both intrinsic (i.e., constrained parameter space) and extrinsic (i.e., client drift) limitations, which hinder it effectively learn global knowledge. In this work, we proposed a novel Federated Residual Low-Rank Adaption method, namely FRLoRA, to tackle above two limitations. It directly sums the weight of the global model parameters with a residual low-rank matrix product (\ie, weight change) during the global update step, and synchronizes this update for all local models. By this, FRLoRA performs global updates in a higher-rank parameter space, enabling a better representation of complex knowledge structure. Furthermore, FRLoRA  reinitializes the local low-rank matrices with the principal singular values and vectors of the pre-trained weights in each round, to calibrate their inconsistent convergence, thereby mitigating client drift. Our extensive experiments demonstrate that FRLoRA consistently outperforms various state-of-the-art FL methods across nine different benchmarks in natural language understanding and generation under different FL scenarios.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730293204878

thread_type:
official_review

<thread_object>
creator: reviewer_ddf1

<content>
summary:
In this paper, the author proposes a novel framework named federated residual low-rank adaption to effectively fine-tune the pretrained large language model in a privacy-preserving manner. The proposed method addresses both the intrinsic problem caused by constrained parameter space and the extrinsic problem caused by the drift among clients. Extensive experiments on several datasets have shown the effectiveness of the proposed method.

strengths:
1. The idea of enlarging the parameter space when finetuning large laugunge model in federated learning is promising.
2. The paper is well organized and the related works are thoroughly summarized.
3. The discussion of the weakness of exsiting related works and the novelty of the proposed method are clear.
4. The author conducts comprehensive experiments to demonstrate the effectiveness of the proposed method, which is convincing.

weaknesses:
1. In Eqs. (6) and (7), the author initializes the local low-rank matrices by the low-rank matrices reconstructed from the top singular values, while remaining the residual part as the global model . In my opinion, this makes the low-rank matrices retain the most information in the pretrained model, while the information remained in the global model can be significanly reduces.  However, in FL, the global model should contain more generalized information from the pre-trained model, which is contrast to the above parameter decoupling operation. So I am curious that whether this operation can reduce the generalization capacity or convergence speed of FL?

2. The motivation of the proposed low-rank matrice initilization lacks further discussion. Can the author provide more detailed discussion about this by such as landscape visualization?

3. Why the global model are initialized by the residual matrice? It seems that directly using the pre-trained model parameters as the global model can provide a more generalized initialization during the FL training. Can the author provide a more detailed discussion about this?

4. In Eqs. (13) and (14),  the author claims that by using residual accumulation, the model fine-tuned space can be expanded. However, Eq. (14) seems to only give a upper bound of the summed matrices rather than increasing its lowe bounder of the summed matrices. So it can not guarantee that the rank of the summed matrices can be increased by such a residual accumulation operation. Can the author provide a more in-depth discussion about this?

5. Actually, the standard PEFT by LoRa in FL can also be formulated as Eq. (13), with $\delta W^t$ as the global update of the LoRa parts at each global round. So what is the real differences between the standard method and the proposed FRLoRA?

questions:
The items listed in the weaknesses. If the author can address or partially address these problems, I will be pleasure to improve my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534345526

thread_type:
official_review

<thread_object>
creator: reviewer_acqq

<content>
summary:
This paper explores the challenges of using LoRA in Federated Learning (FL) for fine-tuning large language models (LLMs) on non-IID data. The authors identify that LoRA in FL struggles to learn global knowledge due to two key issues: extrinsic client drift and an intrinsic constrained update space. The authors propose FRLoRA (Federated Residual Low-Rank Adaptation) to overcome these challenges. FRLoRA initializes the LoRA matrices  $A$  and  $B$  using the Singular Value Decomposition (SVD) of the initial weights  $W_0$, while freezing the residual initial weight. During training, the locally updated LoRA matrices are aggregated and combined with the residual weight, creating the initial model for the next training round. This method addresses both client drift and limited update space, of which theoretical justification is presented. The experiments on nine language model benchmarks show that FRLoRA consistently outperforms various baseline methods.

strengths:
**Addressing Non-IID Challenges in FL with LoRA:** FRLoRA tackles an important challenge in federated fine-tuning of LLMs by addressing both client drift and constrained parameter space through aggregated $\Delta W$ updates via SVD.

**Theoretical Justification:** The authors provide a simple theoretical justification of FRLoRA using a rank analysis of the parameter space, although the rank analysis can be far from the practice.

**Comprehensive Evaluation:** The study provides extensive experiments across nine benchmarks, covering Natural Language Understanding and Generation, validating FRLoRA’s efficacy in terms of performance and communication cost.

weaknesses:
**Insufficient Evidence on the Problem Formulation:** The authors characterize the client drift using Figure 1 (c-d) but it is not convincing. To be specific, the norm of $\Delta W$ and the standard deviation can be simply reduced by the other factors, e.g., learning rate. Perhaps, it would be better to (i) focus on a single round rather than the sequence of rounds and (ii) observe the distribution of cosine similarity rather than the standard deviation. It is also helpful to analyze the other FL methods.

In addition, the authors claim that the constrained parameter space is one of the major problems in the naive method (FedAVG + LoRA). The simple rank analysis supports this claim. However, it is not thoroughly studied in empirical analysis. Hypothetically, the constrained parameter space problem can be simply addressed by employing large rank $r$.



**Limitations in Non-IID Testing Scenarios:** Experiments use a Dirichlet (0.5) Non-IID setting with five clients for binary classification, while NLG tests are performed on IID data. Broader Non-IID experiments would better showcase FRLoRA's robustness instead of just increasing the number of benchmarks.


**Limited Comparision to Existing Works:** The baselines used for comparison, such as FedYogi, FedAdam, and FedProx, do not integrate LoRA, making comparisons with these methods less relevant. Only FFA-LoRA offers a directly comparable baseline. Apparently, there is another LoRA method for FL, FlexLoRA (https://arxiv.org/abs/2402.11505, Feb 2024), in which a similar idea of using SVD per round is proposed. In addition, the idea of using the residual is analog to the one in Chain of LoRA (https://arxiv.org/abs/2401.04151, Jan 2024), which needs to be discussed.

questions:
## Questions 
**Does FRLoRA Fully Address the Intrinsic Challenge?**: While FRLoRA expands the parameter space globally, it does not necessarily address the constrained optimization space at the local level. Can the expanded global space fully mitigate intrinsic issues without modifying local optimization constraints?

**Catastrophic Forgetting at Low Ranks**: Direct updates to $W_0$ in a low-rank context may risk catastrophic forgetting. Can FRLoRA’s SVD initialization mitigate this risk, or could additional ablation studies on learning rates clarify this?

**Comparison to Chain of LoRA’s Residual Framework**: Both FRLoRA and Chain of LoRA use residual structures for fine-tuning. How does FRLoRA’s SVD-based initialization impact its effectiveness relative to the iterative approach of Chain of LoRA? Further analysis of residual efficacy could enhance the comparative evaluation.

**More Diverse Data Heterogeneity**: Is there an empirical study for NLG with severe heterogeneity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720037392

thread_type:
official_review

<thread_object>
creator: reviewer_vtoo

<content>
summary:
The paper presents Federated Residual Low-Rank Adaptation (FRLoRA), a federated learning method that improves LoRA for LLMs. FRLoRA overcomes challenges like constrained parameter spaces and client drift by updating in a higher-rank parameter space and reinitializing local low-rank matrices based on the principal singular values of pre-trained weights. Experiments show that FRLoRA consistently outperforms existing federated learning methods across benchmarks in NLU and NLG tasks.

strengths:
1. The paper introduces a unique adaptation of LoRA for federated learning with residual low-rank updates and reinitialization in the principal singular space.

2. Extensive experiments on multiple benchmarks confirm FRLoRA’s robust improvements across NLU and NLG tasks in various federated settings.

weaknesses:
1. The reinitialization of local low-rank matrices and the use of SVD may introduce significant computational overhead, particularly for large-scale models. An in-depth analysis of this aspect or potential optimizations would provide added value to the paper.

2. Although the method shows performance improvements overall, in Table 3, the proposed approach does not appear to dominate across all evaluations. A similar pattern is observed in Table 4, indicating that while the method is strong, it may not consistently outperform existing baselines in certain subcases.

3. While the paper briefly touches on SVD stability, it lacks a detailed analysis of potential impacts and does not explore alternative methods for scenarios where SVD might lead to convergence issues. Additional insights on this aspect could reinforce the robustness and general applicability of the approach.

questions:
Q1: Could the authors provide further analysis on the computational impact of using SVD for reinitialization in terms of time and memory requirements?

Q2: Could the authors provide further analysis on the behaviour on Table 3 and 4?

Q2:How does FRLoRA handle scenarios where singular value decomposition may be unstable? Are there alternative initialization strategies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730819999408

thread_type:
official_review

<thread_object>
creator: reviewer_phac

<content>
summary:
This paper introduces a method known as FRLoRA (Federated Residual Low-Rank Adaption) to address the problem of Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs) in Federated Learning (FL) scenarios. FRLoRA updates global model parameters by introducing a residual low-rank matrix product and reinitializes local low-rank matrices in each training round to mitigate client drift. The paper compares FRLoRA with existing FL baseline methods across multiple datasets in natural language understanding and generation tasks, demonstrating its consistent superiority and validating the effectiveness of the proposed FRLoRA.

strengths:
1. FRLoRA effectively addresses the significant impact of data heterogeneity on PEFT in FL scenarios by accumulating the product of residual low-rank matrices, enabling the global model to learn more comprehensive knowledge.
2. By reinitializing local low-rank matrices at each training round, FRLoRA alleviates client drift issues, enhancing the convergence and performance of the model.
3. Extensive experiments validate the performance improvements of FRLoRA over existing methods across various natural language processing tasks.

weaknesses:
1. The updates derived from averaging the uploaded matrices A and B at the server differ from the averaged updates across clients, and this discrepancy is likely to amplify in heterogeneous data scenarios. [Improving LoRA in Privacy-preserving Federated Learning]
2. Although the paper proposes extending the parameter space through residual updates, it does not clearly specify whether the rank of the model parameters is strictly improved after each residual update, lacking further explanation on how this mechanism operates, particularly regarding its impact on model rank and representational capacity.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731204588500

thread_type:
official_review

<thread_object>
creator: reviewer_rhq7

<content>
summary:
This paper introduces FRLoRA, a novel Federated Learning (FL) method that addresses limitations in global knowledge learning by combining residual low-rank adaptation with periodic recalibration of local models. FRLoRA expands the effective parameter space during global updates and mitigates client drift by reinitializing local model components with principal components from the pre-trained model.  Extensive experiments across various natural language processing tasks demonstrate FRLoRA's superior performance compared to existing FL methods.

strengths:
The paper is clearly presented. The proposed method is sound. Experiment results are comprehensive and validate the quality improvement of the proposed method.

weaknesses:
This paper presents a novel Federated Learning method, FRLoRA, which aims to improve global knowledge learning. However, the paper's practical value is limited due to two main weaknesses:

Lack of a compelling use case: The authors fail to clearly motivate the need for their proposed method within a real-world application scenario. While the technical contributions may be sound, the lack of a clear practical context diminishes the paper's impact and relevance.

High communication cost: Despite claims of efficiency, the method introduces significant communication overhead, hindering its practicality in real-world deployments, where many LoRA customers care about.

questions:
Can the authors should further explore and demonstrate the method's effectiveness in a specific application domain and address the communication cost limitations to enhance its practical value.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732337238308

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear reviewers and meta-reviewers,

We appreciate all reviewers for their valuable comments and suggestions. We've revised our manuscript based on reviewers' comments as follows:

1. **For Reviewer RHQ7**, we have revised the **analysis of communication costs**, which can now be found in **Appendix C.3, Table 14** due to page limitations.
   
2. **For Reviewer vtoo**, we have included an analysis of **SVD computation** in **Appendix C.4, Table 15**.

3. **For Reviewer Acqq**, we have added a **clarification** in **Lines 1073-1074**.

4. **For Reviewer Acqq**, we have conducted an analysis of **client drift by cosine similarity** in **Appendix C.5, Figure 3**.

5. **For Reviewer Acqq**, we have explored the **impact of data heterogeneity** on our method for NLU tasks in **Appendix C.6, Figure 4**.

6. **For Reviewer Acqq**, we have added the results of **FlexLoRA** in **Tables 1, 2, 3, and 4**.

7. **For Reviewers Acqq and Ddf1**, we have added an analysis of the **impact of FRLoRA's initialization** in **Appendix C.7, Figure 5**.

The changes have been highlighted in **blue** in the revised paper. Please see below for our responses to each reviewer. If you have any further questions or suggestions, please feel free to share them on OpenReview.
</content>
</thread_object>

</thread>
<thread>
creation date:
1754011468994

thread_type:
public_comment

<thread_object>
creator: ~kevin_kuo1

<content>
title:
Cites a plagiarized work

comment:
I would like to bring to the authors' attention that this work cites an arXiv preprint (https://arxiv.org/abs/2407.20557) which has been removed due to plagiarism. The original work can be found on Google Scholar: https://scholar.google.com/scholar?cluster=577696854221514582&hl=en&as_sdt=0,47. Although ICLR 2025 has already passed, I am writing in the hope that the authors can request an update to the camera-ready and that the program committee allows for a revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces FRLoRA, a novel approach designed to overcome the challenges of applying LoRA in a Federated Learning (FL) environment for acquiring global knowledge. Specifically, FRLoRA tackles the issues of "extrinsic client drift" and "intrinsic constrained update space."  Extensive experiments across nine benchmarks demonstrate FRLoRA's effectiveness in achieving strong performance while minimizing communication costs. To further enhance the paper, please include the reviewers' suggestions especially detailed analysis of the proposed method in the revised version.