FedSVD: Adaptive Orthogonalization for Private Federated Learning with
  LoRA
Authors: name: Seanie Lee; name: Sangwoo Park; name: Dong Bok Lee; name: Dominik Wagner; name: Haebin Seong; name: Tobias Bocklet; name: Juho Lee; name: Sung Ju Hwang
Published: 2025-05-19 07:32:56

Abstract: Low-Rank Adaptation (LoRA), which introduces a product of two trainable
low-rank matrices into frozen pre-trained weights, is widely used for efficient
fine-tuning of language models in federated learning (FL). However, when
combined with differentially private stochastic gradient descent (DP-SGD), LoRA
faces substantial noise amplification: DP-SGD perturbs per-sample gradients,
and the matrix multiplication of the LoRA update (BA) intensifies this
effect. Freezing one matrix (e.g., A) reduces the noise but restricts model
expressiveness, often resulting in suboptimal adaptation. To address this, we
propose FedSVD, a simple yet effective method that introduces a global
reparameterization based on singular value decomposition (SVD). In our
approach, each client optimizes only the B matrix and transmits it to the
server. The server aggregates the B matrices, computes the product BA using
the previous A, and refactorizes the result via SVD. This yields a new
adaptive A composed of the orthonormal right singular vectors of BA, and an
updated B containing the remaining SVD components. This reparameterization
avoids quadratic noise amplification, while allowing A to better capture the
principal directions of the aggregate updates. Moreover, the orthonormal
structure of A bounds the gradient norms of B and preserves more signal
under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD
consistently improves stability and performance across a variety of privacy
settings and benchmarks, outperforming relevant baselines under both private
and non-private regimes.