Title: FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

LoRA

Differential Privacy
Abstract: Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). 
However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update ($BA$) intensifies this effect. Freezing one matrix (*e.g.*, $A$) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation.
To address this, we propose $\texttt{FedSVD}$, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). 
In our approach, each client optimizes only the $B$ matrix and transmits it to the server. 
The server aggregates the $B$ matrices, computes the product $BA$ using the previous $A$, and refactorizes the result via SVD. 
This yields a new adaptive $A$ composed of the orthonormal right singular vectors of $BA$, and an updated $B$ containing the remaining SVD components. 
This reparameterization avoids quadratic noise amplification, while allowing $A$ to better capture the principal directions of the aggregate updates.
Moreover, the orthonormal structure of $A$ bounds the gradient norms of $B$ and preserves more signal under DP-SGD, as confirmed by our theoretical analysis.
As a result, $\texttt{FedSVD}$ consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm, called FedSVD, which allows efficient LoRa fine-tuning in the federated setting. More precisely, a difficulty in the federated setting is that local updates cannot be easily summed up while preserving the low-rank structure. One way to mitigate this, for the update rule \\$W\\_0 + \\Delta W = W\\_0 + BA\\$, is to keep \\$A\\$ fixed over time, as was previously done by FFA-LoRA. FedSVD improves upon this existing method by proposing to update \\$A\\$ on the server side as well. After presenting the algorithm, the authors theoretically analyze the algorithm, deriving privacy properties and Hessian properties, and demonstrate empirically that FedSVD improves upon FFA-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n+ The paper is very clearly written, easy to follow, with clear explanations of the setup, the theoretical results, and the experiments.\n+ The topic is very timely, as adapting federated learning to foundation models and using techniques such as low-rank updates could interest many people in the community.\n+ The proposed method is simple and could be implemented in practice.\n\n----\n\n- The optimality of the approach is not clear; see questions.\n- The experiments and the privacy part could be more developed; see questions.\n- The paper might feel a bit incremental with respect to FFA-LoRA. However, I don't believe this to be a major issue, as the new algorithm still needed to be properly written up and has some theoretical results, but people more familiar with the field might disagree.\n\nQUESTIONS:\n- Do you think FedSVD is optimal for federated LoRA fine-tuning in terms of the utility of the final result? Is it possible to consider other update rules that would lead to better results?\n- The numerical experiments only test a very small number of clients (maximum 12). Can the method scale? Did you have a specific reason to keep it very small? In particular, would the algorithm perform well if only a small portion of the clients is updated at a given round (\\$K' \\ll K\\$)?\n- I am not sure I understand the heterogeneity you studied (l.277). Can you explain your method? Could you test more diverse heterogeneity settings (covariate shift, concept drift, unbalanced clients), or explain how your method would behave?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission address low-rank adaptation in federated learning (FL) under the use of differential private stochastic gradient descent (DP-SGD). The authors tackle a known problem in literature, which is related to the noise amplification of DP when applied to both matrices (A,B) of the low-rank reparametrization. Similarly to previous works, they propose to explicitly train locally only the B matrix, while A ramains fixed during each round. Differently from other works, the A matrix changes across rounds, as an effect of further singular value decomposition at the server side. The authors provide a motivation for the proposed approach, highlighting the orthonormalization of A as a factor that promotes better conditioned optimization.\nExperiments show promising results on 5 GLUE datasets, with/without DP.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenghts\n\n* **The paper is well written:** the paper is easy to follow, sections are clear and reasonable coverage of closely related approaches is provided. Some details appear superflous for a research paper, and risk to be not accurate, such as eq. 1, which is just the GD update rule at each local step and the def. of the cross-entropy loss. This could be a typo, since the algorithms should use SGD, and not GD locally (as per eq. 1, the loss is calculated on the entire local dataset $\\mathcal{D}_k$).\n* **The proposed approach is simple:** conceptually, the approach is of straightforward understanding, and it is trivial to implement. Under full comprehension of the induced dynamics, which is not trivial at all (see weaknesses on that), the approach could have immediate practical implementation.\n\n## Weaknesses\n1. **Weak motivation for the proposed approach:** the aim of the work is to avoid the noise amplification that follows the use of DP-SGD in FL. To do so, authors rely on what other works do, that is train locally only the B matrix, and add on top a further SVD at the server after aggregation with the purpose of not limiting the expressiveness of A. However, the motivation behind the precise choice of recomputing an SVD after training only B is not clear, it seems an heuristic similar to other intuitive ideas one can come up. For example, clients could optimize A or B depending on the round: this would similarly avoid the noise amplification and effectively train both matrices, with the same communication complexity and without the burden of the additional SVD. Without stronger motivations for the precise steps of the proposed algorithm, this work does not offer a clear advancement of our understanding on low-rank (federated) training.\n2. **Insufficient theoretical rationale:** The main point of the theoretical discuss points to the orthonormality of A as a way to have a stricter upper bound on the condition number of the problem. This analysis is insufficient in explaining the characteristics of the method for two reasons: (i) It holds on a very restrictive setting (logistic regression and one layer network) and (ii) does not reflect where the real advatage comes from experimentally. Indeed, the orhonormality of A does not appear as the main factor improving the performance (see Table 3). Better analysis should be provided to offer relevant insights on the induced dynamics of low-rank training\n3. **Missing comparison with low-rank training works:** this paper compares with closely related lora-based methods, but does not discuss the relationship with related FL papers on low-rank training. For example [1] exploits low-rankness of the subspace spanned by gradients to drastically reduce the number of parameters transmitted through the network. That algorithm should be compliant with the premises of this work (avoid DP noise amplification or restricting expressiveness), and so it should be considered as competing baseline. \n\n\n[1] Azam et. al, Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?, ICLR 2022\n\nQUESTIONS:\n* Can the authors explain why other intuitive strategies for avoiding the noise amplification (see example above) are not equally valid ad the proposed approach?\n* Can the authors better motivate why the proposed approach does not limit the expressiveness of the decomposition, even if the matrix A is effectively never directly updated by training?\n* Can the authors discuss theoretically how the techniques relates to priors works and guarantess with low-rank training?\n\nLIMITATIONS:\nYes, even if the limitations appear significant for large language models (e.g. computing SVD very often)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FedSVD, a method designed to improve the efficiency and stability of federated learning (FL) with Low-Rank Adaptation (LoRA) under differential privacy (DP). The key innovation is a global reparameterization based on singular value decomposition (SVD), which allows the matrix $A$ to adapt over time while mitigating noise amplification. In this approach, clients optimize and transmit only the $B$ matrix, while the server aggregates these updates, computes $BA$, and then refactors it via SVD to produce a new adaptive $\\hat{A}$ and updated $B$ based on the clients' data. The paper claims that FedSVD consistently improves performance across various privacy settings and benchmarks, outperforming relevant baselines in both private and non-private regimes. Empirical results on GLUE tasks demonstrate the effectiveness of FedSVD.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The introduction of SVD-based reparameterization to mitigate noise amplification appears to be an effective approach in DP settings.\n2. The paper provides deep theoretical analysis showing how orthonormal rows of $A$ can improve the condition number of the Hessian, leading to better optimization dynamics.\n3. Empirical evaluations on GLUE tasks demonstrate consistent performance improvements over baselines.\n\n### Weaknesses:\n1. A major concern is the novelty. There have been other studies on decomposition-based FL approaches in DP settings [1, 2, 3] that are not compared in the paper. The novelty here is mostly limited to applying these techniques to the new problem domain of LoRA training.\n2. Although the authors claim that FedSVD improves performance across various DP settings, the privacy budget comparison with recent methods is not provided.\n3. While the method introduces additional communication costs of broadcasting newly initialized $\\hat{A}$ matrix to clients after each SVD step, the performance improvement seems to be marginal compared to the prior method FFA-LoRA.\n4. While the paper demonstrates effectiveness on GLUE tasks, it remains unclear how well FedSVD scales to larger datasets or more complex models.\n\n[1] Privacy-Preserving Distributed SVD via Federated Power (FedPower), CoRR 2021  \n\n[2] Privacy-Preserving Federated Singular Value Decomposition, Applied Sciences 2023\n\n[3] Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models, AAAI 2025\n\nQUESTIONS:\n- How does FedSVD fundamentally differ from the decomposition-based federated learning methods in [1, 2,3 ] (or other similar studies)?\n  - Can you clarify the unique contributions of FedSVD beyond its application to LoRA?\n- How does the privacy-utility trade-off of FedSVD compare with other recent DP-FL methods? Could you provide a more comprehensive evaluation of privacy budgets across different methods?\n- Does the additional communication cost of broadcasting $\\hat{A}$ after each SVD step become a bottleneck in large-scale settings? How does this compare to existing lightweight methods like FFA-LoRA? (Could this be beneficial for federated fine-tuning a.k.a. maml or partial learning?)\n- Could you evaluated FedSVD on larger or more complex datasets beyond GLUE? (e.g., SNLI[4], HellaSwag[5], MATH[6] )\n\n[4] A large annotated corpus for learning natural language inference (2015)\n\n[5] Hellaswag: Can a machine really finish your sentence? (2019)\n\n[6] Measuring mathematical problem solving with the math dataset (2021)\n\nLIMITATIONS:\nThe authors have discussed some limitations of the work in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a new FL algorithm for LoRA training, FedSVD. FedSVD uses reparameterization of LoRA adopter matrices BA based on SVD and only requires clients to train a matrix B, reducing the amount of noise under DP while keeping the learning capability. Through empirical experiments, the authors show that their proposed algorithm outperforms the baseline LoRA algorithms.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper is clearly written 
- The idea of using SVD in LoRA training is simple but effective—the experiment results support the superiority 
- The experiment demonstrates the validity of the algorithm through multiple angles. In particular, the experiments listed in Section 3.3 support that the proposed algorithm is robust and its components, e.g., orthonormal initialization of A, are effective respectively.

Weaknesses
- The experiment uses only one model, RoBERTa-large, which is much smaller than the recent models such as Llama. Running experiments on larger models would validate that the scalability of the proposed method.

QUESTIONS:
- What are the baseline results for non-finetuned models and centralized finetuned models?
- The performance of the proposed method when K=12 in Figure 4(b) is significantly better than the baseline. Do the authors have any intuition behind the results? Do they think the gap becomes more significant when we have larger number of clients?
- Do the authors try different DP parameters from $\epsilon=6$? Why not sweeping $\epsilon$ values?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03