Title: Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Venue: NeurIPS 2025
Accepted: yes
Keywords: LoRA

Federated Learning
Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs.  We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.


Peer reviews:
comments: SUMMARY:
In this paper, the authors present the robust federated finetuning of LLMs using alternating optimization of LoRAA–RoLoRA. This is a novel federated learning framework for addressing the limitations of previous work for training LoRA adapters in distributed settings.

STRENGTHS AND WEAKNESSES:
The strengths of this paper are summarized as follows:

1. This paper is well-motivated. It gives a very rigorous mathematical analysis and proofs for showing both linear and non-convex cases. It shows why using alternating optimization is necessary. Also, the algorithm is simple and effective. It maintains computational efficiency and exact model aggregation.

2. Also, this paper gives a thorough experimental result on multiple models, tasks, and federated settings. 

The weaknesses of this paper are summarized as follows:

Alternating optimization has been extensively studied in many other areas, such as weighted low rank approximation and low rank matrix completion.  The key contribution of this paper seems to be applying this alternating optimization to study federated LoRA. However, this paper does not make any fundamental changes to this algorithm. It would be better if the authors could compare this paper with alternating optimization in other fields and highlight the novelty of this paper.

I am not an expert in this field, so I hope the AC will put more weight on the other reviewers' comments.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a novel method for federated learning with LoRA that improves on prior art performance.  It belongs to a class of algorithms that address the shortcoming of directly applying standard federated optimization methods such as FedAvg on LoRA modules.  LoRA's efficiency in module storage opens up impactful opportunities in federated learning applications where communication efficiency is crucial.  However, brute force combining of the two methods often leads to interference and poor performance as theoretically and empirically shown in this paper.  Hence a prior body of research has been dedicated to resolving the performance gap.  This paper builds on prior arts of novel strategies based on treatment of the adaptation matrices A and B during the federated optimization process.  The authors observe that a lot of the performance gap can be recovered by taking turn between optimizing A and B in alternating communication rounds, versus methods from prior arts that freeze A or B, or that optimize A and B in every round.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Quality and clarity: well written and comprehensive, well structured with detailed analysis and easy to follow\n2. Originality: novel method that wins in its simplicity and effectiveness\n3. Significance: especially applicable today where LLMs have found mainstream and user privacy is crucial \n\nWeaknesses:\n1. Societal impact evaluation may be missing\n\nQUESTIONS:\n1. Could you comment on how the number of local steps impact the performance of RoLoRA and FFA-LoRA?  \n2. Could you comment on the intuition behind why FedAvg + LoRA performs relatively well in Table 2?  How does data heterogeneity impact performance of RoLoRA?\n\nLIMITATIONS:\nNo except with one mention of data privacy guarantee.\nSuggestion is to stress significance of privacy protection for client data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RoLoRA, an alternating optimization method for LoRA in LLMs, and provides a thorough theoretical analysis to demonstrate the importance of training the down-projection matrix in federated LoRA updates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The theoretical analysis is well-grounded. The use of both linear and non-linear models to demonstrate how the initialization of matrix $A$ significantly affects the performance of FFA-LoRA via angle distance is insightful.\n\nWeaknesses:\n- The proposed method is not inherently specific to LLMs and could be applied to any LoRA-based update, reducing its uniqueness in the LLM context.\n\n- The theoretical analysis focuses on linear models, which do not capture autoregressive characteristic of modern LLMs.\n\nQUESTIONS:\n- It seems that you used 3, 20, and 50 clients in your experiments. Could you clarify the rationale behind choosing these specific values, especially the large jump from 3 to 20?\n\n- (Discussion) In my opinion, a core issue in FFA-LoRA is that random initialization does not guarantee the subspace spanned by matrix $A$ includes the optimal subspace spanned by $A^*$. I believe that alternating optimization of $A$ and $B$ could mitigate this issue. However, it's worth noting that in standard LoRA implementations, $A$ is typically initialized with Gaussian random values while $B$ is initialized to zero. This suggests that $A$ serves as a basis while $B$ acts as a coefficient matrix. Therefore, treating $A$ and $B$ symmetrically may not be optimal. Is there a principled approach to gradually adapt matrix $A$ toward the optimal basis while ensuring matrix $B$ is updated effectively?\n\n- Is there any evaluation on how rank affects the performance gap between RoLoRA and FFA-LoRA in a centralized setting? Since angle distance may decrease as the rank increases, it would be valuable to understand whether higher ranks reduce the effectiveness gap between the two methods.\n\nLIMITATIONS:\nYes, the paper have addressed that the theoretical analysis centers around linear models.\n- Moreover, the theoretical analysis fail to capture the **autoregressive** architecture of modern decoder-only LLMs. **As such, the theoretical analysis may not be sufficient to support the claimed benefits in LLM training**. If the authors are unable to extend the theory to autoregressive or non-linear models more representative of LLMs, I recommend placing greater emphasis on empirical evaluations or supporting the claim of effectiveness on general neural architectures rather than LLMs through theoretical analysis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
This paper proposes RoLoRA, a parameter-efficient fine-tuning method for federated fine-tuning of LLMs. RoLoRA learns the two LoRA low rank terms in an alternating minimizing style, achieving SoTA fine-tuned accuracy without increasing the communication cost.

STRENGTHS AND WEAKNESSES:
As an emergency reviewer, I was assigned with this paper without bidding. Sorry for the fact that I have limited knowledge in federated learning and hope the following comments & questions are reasonable.

## Strength 
1. Clear method description: the method and derivations in this paper are well organized
2.  Sufficient experiments: the paper covers both ideal toy experiments and practical problems, with additional experiments in the appendix.
3. The improvement of the baselines are clear: the fine-tuning process of RoLoRA is more stable, and achieves much higher model performance when the number of clients is large. 

# Weakness
Limited by my knowledge background, I did not observe weakness/limitations of this paper.

QUESTIONS:
1. According to line 288, comparison against baselines are run under the same number of communication rounds. Does this mean roughly the same amount of communication cost like the amount of data transferred?
2. In Table 1, the rank is 4 and 8, which seems smaller than the common values of 32 used in PEFT papers. Could the author explain the reason behind it, or show the results with higher rank?
3. In Figure 3, could the author explain why the error band of MNLI accuracy is much wider than the other two baselines?

LIMITATIONS:
Limited by my knowledge background, I did not observe weakness/limitations of this paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30