FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
Authors: Seanie Lee
Sangwoo Park
Dong Bok Lee
Dominik Wagner
Haebin Seong
Tobias Bocklet
Juho Lee
Sung Ju Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
LoRA
Differential Privacy
Primary area: deep_learning
TL;DR: FedSVD mitigates noise amplification in differentially private federated fine-tuning with LoRA by periodically refactorizing LoRA matrices via SVD, improving stability and performance under DP-SGD.

Abstract: Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). 
However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update ($BA$) intensifies this effect. Freezing one matrix (*e.g.*, $A$) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation.
To address this, we propose $\texttt{FedSVD}$, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). 
In our approach, each client optimizes only the $B$ matrix and transmits it to the server. 
The server aggregates the $B$ matrices, computes the product $BA$ using the previous $A$, and refactorizes the result via SVD. 
This yields a new adaptive $A$ composed of the orthonormal right singular vectors of $BA$, and an updated $B$ containing the remaining SVD components. 
This reparameterization avoids quadratic noise amplification, while allowing $A$ to better capture the principal directions of the aggregate updates.
Moreover, the orthonormal structure of $A$ bounds the gradient norms of $B$ and preserves more signal under DP-SGD, as confirmed by our theoretical analysis.
As a result, $\texttt{FedSVD}$ consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.

FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
Seanie Lee
Sangwoo Park
Dong Bok Lee!"
Dominik Wagner
Haebin Seong'
Tobias Bocklet Juho Lee'
Sung Ju Hwang!
KAIIS
Technische Hochschule Nümberg Georg Simon Ohm
DeepAuto.ai
(Isnfamily02 swgger, markhi]@kaist.... ac kr
Abstract
Low-Rank Adaptation (LoRA), which introduces a product of two trainable low- rank matrices into frozen pre-trained weights, is widely used for efficient fine- tuning language models in federated learning (FL). However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces sub- stantial noise amplification: DP-SGD perturbs per-sample gradients, and the ma- trix multiplication of the LoRA update (BA) intensifies this effect. Freezing matrix (e.g., A) reduces the noise but restricts model expressiveness, often result- ing in suboptimal adaptation. To address this, propose FedSVD, simple yet ef- fective method that introduces a global reparameterization based on singular value decomposition (SVD). our approach, each client optimizes only the / matrix and transmits it to the server. The server aggregates the / matrices, computes the product B using the previous A, and refactorizes the result via SVD. This yields a new adaptive A composed of the orthonormal right singular vectors of BA, and an updated B containing the remaining SVD components. This reparameteriza- tion avoids quadratic noise amplification, while allowing A to better capture the principal directions of the aggregate updates. Moreover, the orthonormal structure of A bounds the gradient norms of B and preserves more signal under DP-SGD, as confirmed by our theoretical analysis. As result, FedSVD consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes, Our code is publicly available at https: :/gt ligggttin thub. com/ seanie eamiieee//ffae f2/fed-svdd.
Introduction
Language models have demonstrated remarkable performance across various tasks [32, 24, 8]. While these models provide strong general capabilities, adapting them specific domains or tasks typically requires fine-tuning with domain-specific datasets [4]. In real-world deployments, however, training data is frequently fragmented across various organizations or user devices, and strict privacy regulations often prohibit direct data sharing [10]. Federated Learning [FL; 21] provides viable solution by allowing clients fine-tune models locally on their private data, while central server aggregates model updates without accessing raw training data, enabling privacy-preserving collaborative training.
In FI., individual clients often lack the computational and memory capacity required for full fine- tuning of large models, making such approaches impractical. Parameter-efficient fine-tuning ad- dresses this by freezing most model parameters and updating only a small subset, enabling scalable
$ Equal Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SVD
A
A
Wo
&
Server
Glann
Arg
B.
x
D
U/
A
Dg
B
LatA Weight
Svate
(a) Reparameterization with SVD
(b) Local Update
(c) Appregation
Figure 1: (a) At communication round i, the server computes the SVD of B, B i.e. U, = B, and initializes A; V and B; = U, These reparameterized matrices are then broadcast to all clients. (b) Each elient updates only the matrix B initialized with B, using its local dataset, while keeping A, fixed. (c) The locally optimized A(*) matrices are aggregated at the server to update the global model.
model adaptation in resource-constrained settings. In particular, Low-Rank Adaptation [LoRA; 16] has been widely adopted for fine-tuning models in FI. environments due to its low local computation and communication requirements [40, 31, 12, 34].
Although FL improves privacy by exchanging model updates instead of raw data, it does not pro- vide formal guarantees against information leakage. Sophisticated attacks such membership in- ference [27] or model inversion [11], can reconstruct sensitive information from shared updates, particularly given the capacity of language models to memorize training data [6, 7]. Therefore, inte- grating differential privacy [DP; 9] is essential to provide formal privacy guarantees and enhance the trustworthiness of collaborative model training. A common approach enforcing DP in deep neural networks is DP-SGD [30, 3, IJ, which clips the norm of each per-sample gradient to a predefined threshold and adds Gaussian noise to the average of the clipped gradients.
Recent work [31] has shown that naïve integration of LoRA into DP-SGD significantly degrades model performance. Following single DP-SGD update of the LoRA adapter matrices A and B, the noise added to both matrices is amplified through their product BA, as shown Eq. 5. To mitigate this amplification, FFA-LoRA [31] fixes the A matrix and updates only the matrix during training. However, using a fixed random matrix for A limits the learning capability of LoRA, and observe that optimizing only B leads to significantly slower convergence. Ideally, we would like to adapt A over time to better capture the principal direction of aggregated updates without incurring noise amplification under DP-SGD.
randomly initialized constant
To this end, we propose FedSVD, simple yel effective method that introduces global reparameteri- zation based on singular value decomposition (SVD). In the first communication round, the server randomly initializes Ao and Bo and broadcasts them to the participating clients. Each client then optimizes only the matrix B using its local data, and the server aggregates the updated B matrices. each subsequent round, the server refactorizes the product the aggregated B and the previous A using SVD to obtain the matrices for the next iteration. As shown in Fig. la, the re- ie-iniiialized with orthonormal right singular vectors (i.e., V, of BA obtained from the SVD. The re-initialization of B uses the remaining components of the SVD, namely the left singular vectors and singular values (ie., U,E,). The newly initialized matrices A, and B, then broadcast to all clients. Each client k. initializes its local matrix ((*) with B, and subsequently optimizes it to obtain B²a)) while keeping A, fixed (Fig. 1b). The resulting ((() matrices then collected and aggregated on the server (Fig. Ic).
of A are
This SVD-based reparameterization offers several advantages. It allows A to adapt based on the aggregated B without amplifying noise, while maintaining the differential privacy guarantee, since SVD is applied only post-processing step after local DP-SGD updates. The orthonormality of A beneficially bounds the gradient norms of B, preserving stronger update signals under DP-SGD compared to random initialization. Theoretically, lower Hessian condition number than random matrix in a two-layer multilayer perceptron (MLP) with ReLU activations, implying nenter-conditioned loss landscape that can potentially lead to faster convergence. Empirically, we observe that this property translates into accelerated accuracy improvement for deep models with orthonormal rows of A (Fig. 3).
show that the orthonormal rows of A yield
We empirically evaluate FedSVD on several benchmark datasets, including SNLI [5], MNLI [35], SST2 [29], QQP [26], QNLI [33], and HellaSwag [39], both in private and non-private settings. In

---

both regimes, FedSVD consistently outperforms the relevant baselines during most communication rounds and achieves the highest final accuracy.
We summarize our findings and contributions as follows:
We propose FedSVD, simple yel effective method allowing the LoRA matrix to adapt over time based on aggregated updates of B using SVD, while eliminating noise amplification under DP-SGD.
We theoretically show that orthonormal rows of A yield a better-conditioned Hessian of the training loss with respect to B in a two-layer MLP with ReLU.
We empirically demonstrate that our FedSVD approach achieves higher accuracy and faster convergence than relevant baselines under DP-SGD in several benchmark datasets.
2
Background
This section reviews the necessary background, including federated learning with LoRA, DP-SGD, and FFA-LoRA. A detailed discussion of related work is deferred to Appendix B.
Federated learning with LoRA. Let pe X'' } be language model (e.g.. Devlin et al. [8], Liu al. [20]) parameterized by 0, which maps an input token sequence
to an output class label E V. In the FL framework, each client & E [K] = {l..... K} has access only to its local training
dataset D {(x{") yr )+)) where Dx Dr Ø for all k, k e [K] with k * k'. Furthermore,
the central server never accesses any local datasets directly. At each update round i E [R]. a random subset of client indices S, < [K] is selected such that [SS]] K'. Each selected client e S then receives copy of the current global model parameters 0; from the central server and trains its local model Pa(0) using private dataset DD as follows:
((k)
((! VoL 12 Da)..
[(O()) Da)
Σ log Poce x),
(1)
Tuk (x.U)ED
for to O,... 1, where > O is the learning rate and 0 is initialized with 0.. Since full fine-tuning of pe computationally expensive, LoRA is commonly employed to reduce overhead by injecting trainable low-rank matrices into the weight matrix of each layer l:
(k.1) w + (k.)) (k.1)
A.t
i,t
where ( is frozen pre-trained weight matrix of pa., and ( E RRxxd and B(ha1) E Rduxx
are the corresponding low-rank matrices. We denote ( = ((A(k.))
(k.1) B(kk"))), as the set of LoRA it
adapter weights for client at step of round i, where each pair (A,h..) B(kk)) represents the LoRA
matrices in layer 1. In methods such as FedAvg [21] and FedIT [40], the server updates its parameters
8, = [(A B > by aggregating the weights from the participating clients as follows:
1
Tk (k.)) Σ ke
(!0) 241
Σ
The
B
(k.l) L.Tk
(3)
1.
m,
k
where m, kE ni and T&k [Dx]. At round i + 1, the central server model pe... uses the updated weight matrix for each layer [L] as follows:
( ++1
w(O + (20) (0)
4)
i+1
where ((()) denotes the frozen pre-trained weights and 4 is the aggregated low-rank update.
Differential privacy. Language models tend to memorize training data, which can lead to the leakage of private information from local client datasets [6. Differential privacy (DP; 9] provides a formal privacy guarantee by limiting the influence of any individual data point on the model, thus mitigating such leakage risks.
Definition 2.1 ((c, 6)-DP). randomized algorithm M is dciff--iifrrenniiall private if, for all neighboring datasets D. D' that differ in exactly entry, and all subsets E of the possible outputs of M. we have (D) E E) & e Pr(M(DD)) E) 8, where e the privacy budget, and bounds the probability that the privacy loss exceeds e, i.e., the probability that the DP guarantee may fail.
3

---

In FL, privacy guarantee depends whether the central server is trusted. In the centralized DP setting, clients send raw updates without local privacy measures, and DP is applied during global aggregation [22]. In the local DP setting. which assumes an untrusted server, each client ensures that its update differentially private before communication [36, 18, 25J. Our work adopts this stronger local DP setting: we apply DP at the client level so that any shared updates (Le., model parameters) are already privatized. By the post- post-procsssin invariance property of DP [9, Proposition 2.1], the final global model also satisfies DP.
Fixed LoRA A matrix. A common approach to ensuring the differential privacy of deep neural networks is DP-SGD [30, 3. I]. DP-SGD first clips each per-sample gradient g(x,) from sam- pled mini-batch to have a bounded norm by applying g(x;)/ max(1, (xg (1)x))]22 /C), where C' is a predefined threshold. Gaussian noise Ç ~ (0, 1²(1 is then added to the average of the clipped gradients, and the resulting noisy average is used to update the model parameters. However, jointly updating and aggregating both A and B, introduces a challenge for fine-tuning models with DP- SGD. During client-side fine-tuning, Gaussian noise is added to the average of the clipped gradients of and B, which becomes amplified through their post-update matrix product after a single DP- SGD step:
(5) where ((k.1) and </ represent the Gaussian noise added by DP-SGD. To mitigate the noise am- plification caused by the LoRA matrix product, FFA-LORA [31] fixes A randomly initialized matrix and performs aggregation only on B:
(B(L.))
9 (R.)) A!! + B sh. (k.0 (R.)) (AA.)) + e((00))- ( A C A SB
(K.)
+ (
(R.)) (R.).
Σ
nkk (k.)) mi
iite
A
(6)
NES.
This removes the quadratic noise term in Eq. 5 (i.e., ((2.)) cx s(hi1)) as well as (R..)) thus stabiliz-
ing model training under DP-SGD. However, using fixed random matrix A can affect LoRA learning capacity, potentially leading to suboptimal performance.
3 Method
Although FFA-LoRA mitigates noise amplification by freezing A, this can lead to suboptimal adap- tation, as a fixed random projection may not align well with the data distribution or the dynamics of local model updates. Ideally, A should adapt time to better capture the principal directions of aggregated updates, while avoiding noise amplification under DP-SGD.
Periodic re-initialization of A via SVD. To this end, we propose FedSVD, a simple yet effective approach that avoids direct optimization of A by periodically resetting it to a new matrix with or- thonormal rows, obtained via SVD of the aggregated product B.. Specifically, before broadcasting the newly aggregated matrix B, to the participating clients, the server computes the SVD of B, where A;- is the matrix from the previous round i 1, and initializes Å, and B, as follows:
B, U. -E..rr,
Á V [:r. U.E.V. = B,Å,-1
where Bo 0, Ås is initialized with Kaiming uniform [14], M[; and M[: r::] denote the first r columns and rows of the matrix M, respectively. Note that we omit the superscript 7 for brevity. Each client k receives A; and B, and optimizes only B,, using Eq. 1 on its local dataset DE. The server then aggregates the optimized B, matrices from all participating clients. We outline our complete method in Alg.
Importantly, this reparameterization does not change the value of B, i.e., B,A, B, since rank (, follows from the low-rank structure of LoRA. Therefore, the rank-r SVD exactly recovers B, As a result, all clients receive a consistent, globally synchronized ini- tialization after SVD, while benefiting from updated, data-informed matrices instead of relying on fixed random projection. As shown in Sec. 4, this strategy empirically stabilizes training and accelerates optimization.

---

with Algorithm FedSVD
1: Input: Pre-trained language model po, client datasets {D&] total optimization rounds R, learning rate 7/, batch size b, rank r. the number of participating clients K".
2: for i= 0, R do
3: for for / 1....,...... L do
Broadcast global parameters
4:
if then
U. Ei V,T SVD(B,) A22 U.[ r r1, [.
5:
6:
else
D - 0, - Kaiming_Uniforne -d
7:
8:
end if
9:
end for
10: Sample a set of clients S; < (1....,,K
.K. K} with |S:| = K" m, + 0.
11:
for each client & S; do
Done in parallel
Initialize the client parameter (+t) ((AAta)) ( {(A,)) B
Optimize ( (B(b ) on D with SGD for 76 steps with Eq. 14: Tik - [Dx], m, <- my + flg
15: end for
16: for 1
1 ,.... do
Aggregation of parameters updated by the clients
ny ((.)) 17: (14 - EKEs,
my 1,7k
18: end for
19:
end for
Bounding the gradient norm. Moreover, the orthonormality of Å ensures that its spectral norm is exactly 1, which leads to a tighter bound on the gradient norm of B. Denoting the output as
(Wo + BA)x, compute:
at(z)
[l(z) oz
al(z) dz
@l(z))
aE((z) Oz
AB
Ax)
F =
-||Ax|12
Oz
-11A |x|1 =
.
|||x||
where {(z) is loss function with the corresponding label y, ||-I| is the Frobenius norm, ||Al|2 the spectral norm of Å, and ||x||z is the 12-norm x. Under DP-SGD, each per-example gradient is clipped to a fixed norm before noise addition. Thus, any implicit amplification introduced by Á directly increases the amount of clipping, distorting the original gradient, and weakening the update signal. Since || = 1, the gradients reach the clipping threshold with minimal norm, preserving a more genuine update signal under given privacy budget. In contrast, random initializations usually yield |||l|2 > 1, necessitating more aggressive clipping and slowing optimization.
Privacy guarantee of FedSVD. Due to the post-processing invariance property of DP [9, Propo- sition 2. FedSVD guarantees DP by design, as SVD is applied only after B has already been privatized.
Corollary 3.1 (Privacy guarantee). By Theorem and the moment accountant from Abadi al. [1], FedSVD with DP-SGD and FedAvg aggregation satisfies (c, D-DP, given sampling rate q. the total number of local updates T 7R per client, and noise multiplier > c. log(g/)) logg/oog for
some constant
Proof. This is a direct application of the post-processing invariance property of DP [9, Proposition 2.1] and Theorem 1 in Abadi et al. [1].
Theoretical analysis. We analyze how reparameterizing A and B via SVD affects the opti- mization dynamics of C-class classification. Consider a labeled dataset Dj =
one-hot labels y; E {0,1] Let
Wi xd,, Ae R'd,, B Bddor, W2 RCxss
be parameters of the classification model. With these parameters, let h; = (W1 + BA)x; Rdd,, 2; W2 ReLU(h;) E RRC and p, = softmax(z;). We define the cross-entropy loss (with
element-wise logarithm) (B;A)) = 1 2 (-y, log p.). Let Hx(B; A) be the Hessian of
Ls( B; A) with respect to B. Set A A 2 la, and, for each i, let S, diag(p.) P.P. 2 0
5

---

and D; diag(1 > 0}). where 1 denotes elementwise indicator function and X denotes the Kronecker product. Then
Ha(B; A) AMEA
Mr = -
1 BE
Σ (Ia, @ x.)
S.W.D)) (Ia, 2 x.
(10)
Thk
Proposition 3.2. Assume A has full row rank. Then the condition number of the Hessian satisfies
(d:H( & x²(A))²
(11)
Mm (R(A)))
where X.min()) (-) and ( denote the smallest and largest eigenvalues of a symmetric matrix. Ihe rows of A are orthonormal (so K2(AA) = -1,, the bound tightens to
Amax(MM)
N2l (B)(B:A)) <
(12)
Xminn (M& (MMM((())))
The proof is deferred to Appendix A. By reparameterizing using the SVD of BA, we write BA UEV and choose Á [ [: r,i] (the top r rows of VT). Then the rows of Á are or- thonormal, hence maxx (A) min (A) = 1 and (A) = 1. This removes the K22 (2)) factor that appears with a fixed random A. In contrast, for randomly initialized Aixed (e.g.. Gaussian, or uniform distribution), its condition number satisfies (sse)))
with high probability. A smaller Hessian condition number generally indicates a better-conditioned optimization landscape, leading faster and more stable gradient-based updates to B. Thus, our SVD-based reparameterization improves the stability of local client optimization steps by promoting a well-conditioned projection matrix A. To directly compute the actual condition number, we use simple logistic regression and show that enforcing the orthonormal structure of A yields lower condition number (see Table 9 in Appendix D).
4 Experiments
In this section, we empirically validate the effectiveness of FedSVD.
4.1 Experimental Setups
Datasets. Following FFA-LORA [31], five datasets, including four from the GLUE bench- mark [33]: Stanford Natural Language Inference [SNLI; 5], a sentence- parre classification task for textual entailment with three labels (entailment, neutral, contradiction), i.e., NLI task (or recognizing textual entailment); Multi-Gena Natural Language Inference [MNLI; 35], the same NLI task, eval- uated on both matched (in-domain) and mismatched (cross-domain) test sets; Stanford Sentiment Treebank v2 [SST-2; 29], a single-sentence sentiment classification task with two labels (positive, negative); Quora Question Pairs [QQP: 26], a paraphrase detection task with two labels (duplicate, not duplicate); and Question Natural Language Inference [QNLI: 33]. binary classification task with labels (entailment, not entailment) that determines whether context sentence answers given question. We use the validation split for evaluation, as test splits are unavailable for all datasets except SNLI, which is evaluated on its test split. See Table Appendix C for the dataset statistics.
Baselines. We compare our method, FedSVD, against the following baselines:
1. FedAvg [21, 40]: Both and B matrices are fine-tuned locally and averaged independently. as described in Eq. 3.
2. FFA-LORA [31]: The A matrices are initialized with Kaiming Saiming_Uniformn fniiuomn------e (ofommmdd,,00 d) [14] and remain fixed during training. Only the B matrices are fine-tuned and aggregated.
FLoRA [34]: Both A and B matrices
fine-tuned locally and aggregated by stacking the individual matrices from all clients, rather than averaging them independently. The central server computes the product BAA from the stacked matrices and adds it to the pre-trained weight matrix Wo. After aggregation, randomly re- re-initialized A, B and updated Wo are sent back to the clients.
FeREX-LoRA [28]: Both and B matrices are fine-tuned and aggregated individually as de- scribed in Eq. 3. The residual, which is defined as the difference between the aggregated BA and the product of the aggregated and fixed A, is added to the frozen pre-trained matrix W..

---

Table I: Results 6 GLUE tasks without privacy constraints. We report average accuracy and 95% confi- dence intervals over 5 runs. The best/second-best results are highlighted in bold/underline, respectively.
MNLI
Method
SNLI
SST-2
QQP
QNLI
Average
Matched
Mismatched
FedAvg
84.16 + a.2
74.79 11492
75.09 11504
85.89
112:2
61.75 11005
71.40
-1278
75.51
6.1
FFA-LoRA
82.54 +:2 2.13
82.75
1.72
83.45
1.4
94.06
c.s
78.00
: 308
86.61
1.22
84.57
: 0.99
FLoRA
62.17 1:326
50.49
134
50.81
+1527
58.99
1:247
57.91
+: 731
62.16
57.09
9.26
FedEX-LoRA
70.08 11.00
56.85
21441
57.74
11481
59.43
1:244
64.86
64.90
12:44
62.31
FedSVD (ours)
85.70 = 1.25
83.96 2.12
84.32 247
94.26
0.51
79.82
2.43
88.98
1.4
86.18 1.44
MNU (matched)
MNU (mismatched)
SST-2
QQP
QNL
90-
Accuracy (%)
50-
70-
50
50-
60
50
200
203
202
200
Communication Rounds
FedSVD (Ours)
FedAvg
FFA-LORA
Figure 2: Accuracy vs. communication rounds without privacy constraints across 5 GLUE tasks. Curves show average accuracy over 5 runs, with shaded regions indicating 95% confidence intervals.
Data distribution. Following Hsu et al. [15], we sample client data proportions from a Dirichlet distribution, with concentration parameter stated otherwise (Fig. 4b), we use six clients in total (K = 6). To better emulate realistic federated settings, only half of the clients are randomly sampled for participation in each communication round (K' = 3). See Table 8 in Appendix C for per-label distribution across six clients with a 0.5.
0.5 (except in Fig. 4a) for non-iid data. Unless
Implementation details. Following FFA-LoRA [31], we use RoBERTa-large [20] as base model and apply LoRA [16] with rank r = 8 and scaling factor a = 8 to the query and value projections, using a LoRA A dropout rate of 0.05. All non-LoRA parameters, including the classification head, are frozen. We run
100 communication rounds, with participating clients in each round updating their weights using vanilla SGD for 10 local steps. Due to the absence of separate validation splits (except for SNLI), we refrain from extensive hyperparameter tuning. Instead, we adopt values that work reasonably well for FedAvg: learning rate 7) 0.5, clipping norm C' = 2, and 6 10 The same hyperparameters are applied to all methods for a fair comparison. We consider two privacy budgets, c E (3, 6). where
the Opacus library [37] compute the noise multiplier for total T R x r training steps. We use NVIDIA RTX A6000 GPUs for all experiments.
4.2 Main Results
Effectiveness of FedSVD without privacy constraints. We first assess FedSVD on the GLUE benchmark in a non-private setting. In Table 1, FFA-LoRA outperforms FedAvg, which we at- tribute to the reduced aggregation error. In contrast, FLoRA, which transmits large number of parameters, underperforms due to the frequent random re-initialization
the A and B matrices in experimental setups. We observe similar pattern in FedEX-LoRA. The proposed FedSVD fur- ther improves the performance by periodically adapting A through SVD of the product BA rather than using fixed A. As result, FedSVD achieves the highest average accuracy, outperforming the second-best baseline (FFA-LoRA) by +1.29 percentage points (pp). Fig. 2 illustrates accuracy function of communication rounds for FedAvg. FFA-LORA, and FedSVD. FedSVD consistently outperforms the baselines across all rounds. This robustness to early stopping makes FedSVD well- suited for scenarios with limited communication budgets uncertain convergence points.
Effectiveness of FedSvD with DP-SGD. We next evaluate the performance of FedSVD under DP constraints (c (3. 6), 5 = 10-5). Table shows that the average gain of FedSVD over FFA-LORA increases substantially in the DP settings, iee, from +1.29 pp without privacy constraints +8.77 PP with 6. Even under a stricter privacy budget (c 3), where the injected noise intensifies and the signal-to-noise ratio of gradients degrades notably, our method still achieves an accuracy improvement of +9.63 pp. demonstrating its robustness to tighter DP constraints. We attribute this improvement to the SVD-based re-initialization of FedSVD which allows to capture the principal directions of the aggregated updates more reliably. Furthermore, orthonormal rows of A bound the
7

---

Table 2: Results on GLUE tasks with DP (e E (3,6),8 10-9). We report average accuracy and 95%
confidence intervals over 10 runs. The best/second-best results are highlighted in hold/underline, respectively.
DP Budget
MNLI
Method
SNLI
Matched
Mismatched
SST-2
QQP
QNLI
Average
FedAvg
61.37
65.45
67.02
89.41
58.59
60.70
67.17
FFA-LoRA
62,55 9A8
55.56
838
56.39 134
91.42 10s7
64.35 13:2
72.39 14th
68.02 LN3
6
FLoRA
39.14 + 6.3
48.01
21066
48.86
21.22
91.83
11:33
63.18
15.16
49.48 100
59.78 2457
FedEX-LoRA
54.27 1:00r
54.98
56.02
87.34
53.29
49.86
60.86
FedSVD (ours)
72.77 2:022
71.68
331
2000 73.03
91.32 20.55
72.42
#2.36
75.50
+4:30
76.79 118
FodAvg
36,70 +4.56
49.91
1:2.16
50,53
+12.18
61.87 #9:0
55.27 7.89
50.00 +0.35
50.71 #4300
FEA-LoRA
18
$7.76
7:0
59.19 100
4.51
19:0
3
FLoRA
33.42 2007
41.36
1130
41.81
90.46
57.91
49.68
52.44
FedEX LoRA
55.62 1948
40.92
7:22
41.32
7.54
74.29
1800
50.00
49.78
10.31
49.28 1281
FedSVD (ours)
70.89
2852
70.65
3.9
72.02 3.9
90:46
20.56
72.65
2.00
77.10
13.50
75.63 1199
MNU (matched)
MNLI (mismatched)
55T-2
QQP
QNLI
Accuracy (%)
70
70
20
50
50
50
63
53
Communication Rounds
FedSVD (Ours)
FedAvg
FFA-LoRA
Figure 3: Accuracy vs. communication rounds with DP (c = 6, = 10 ") across 5 GLUE tasks. Curves show average accuracy over 10 runs, with shaded regions indicating 95% confidence intervals.
gradient norm of B (cf. Eq. 8), making gradient clipping more robust under DP-SGD settings. Fig. 3 demonstrates the effectiveness of SVD re-initialization: FedSVD consistently exhibits better conver- gence behavior compared to FFA-LoRA across most training rounds. Although we observe slight accuracy drop on SST-2 after round 80, FedSVD maintains strong overall accuracy, which demon- strates its robustness to DP noise and suitability for real-world federated learning deployments.
Results on HellaSwag. To verify the scalability of FedSVD more complex tasks, compare with FedAvg and FFA-LoRA using the HellaSwag [39] dataset. We partition the training split with = 0.5 based on the seriivitt_labbee field (i.e., labels associated with each caption), since it does not contain explicit labels. We fine-tune SmolLM-360M [2] with LoRA under DP constraints = 6, 50-5), We the same experimental setups as in Sec. 4.1. The models trained with the next-token prediction objective only on the correct endings. At test, we select the endings with the highest normalized likelihooooo::
arg max [x] log (x c),
(13)
x=X(()
where x and c are the token sequences of endings and ctx, and (e) is the set of candidate endings. Table presents results averaged over 5 runs, where FedSVD outperforms all baselines (+1.34 pp), demonstrating its effectiveness on more complex commonsense reasoning task.
Table 3: Results on the HellaSwag [39] dataset.
Method
Accuracy
FedAvg
48.81 + oas
FEA-LORA
49.76 1 one
Integration with DoRA. We also show that FedSVD can be successfully integrated with DoRA [19]; see Table 11 in Appendix D for details.
FedSVD
51.10 0.16
4.3 Analysis
Initialization of A. To better understand the effect of initialization strate-
gies for matrix A, we compare three classes of configurations. First, we randomly initialize A with orthonormal rows and keep fixed during training (Afied w/ random orthonormal). Second, fol- lowing PiSSA [23], we factorize the frozen pre-trained matrix using SVD: Wo = Vo>oV and initialize A and B with / /Eof: r, r] [: r, :] and Uol:.: rly///f r.: rl, respectively. The base matrix Wo re-initialized with its residual component W/6 Col:rr :]Eolr + ;, + 1 IV [r :. Both W/l and are frozen, while only B is updated during training (Arved w/ PiSSA). Lastly, we consider an alternative SVD-based initialization where A is periodically re- initialized with / S/: r.: ro: [[:r.]] and B with UL:,: rly E|: r,: from the SVD of BA, which does not preserve the orthonormality of A's rows (FedSVD w/o orthonormal).
8

---

FedAvg
FFA
FedSvD (Ours)
MNU (matched)
Imismatched)
(matched)
MNU (mismatc hed)
80
Accuracy P%0
02-
Accuracy (8)
60
25
20-
633
(a) for Dirichlet distribution
(b) Total number of clients (K)
Figure 4: (a): Results of varying a E (0.1, 0.2, 0.3, 0.4, 0.5) for Dirichlet distribution on the MNLI dataset.
(b): Results of varying the total number of clients (K e [6, 9, 12), and K' 3) on the MNLI dataset.
Table 4 shows that introducing structural pri- ors into matrix A, i.e. Asxed w/ random orthonor- mal, or Afxed w/ PISSA, helps to stabilize training and yields better performance compared to the unstructured baseline, i.e., Anxed (FFA-LoRA) However, when A is kept fixed throughout training (methods without f), the improvements are limited, suggesting that adaptivity plays crucial role beyond the structural prior it- self. In addition, we find that removing the or- thonormal constraint from FedSVD (denoted as FedSVD w/o orthonormal) degrades performance, indicating that the orthonormal structure of A is not only beneficial for initialization but remains important throughout training. Although the effective- ness of enforcing the orthonormal constraint appears marginal on the MNLI dataset in Table 4 (e.g., +0.92/41.17 pp for Matched/Mismatched it yields much larger improvement the SNLI dataset (i.e,, +11.68 pp). as shown in Table 10 of Appendix D.
Table 4: Results the MNLI dataset with different initializations of A under DP-SGD (e 6, 10-5). indicates that A matrices are periodically updated.
Method
Matched
Mismatched
Atsed (FFA-LaRA)
55.56 + 8.58
56.39 + 8.94
Anned w/ rarddmo orthonormal
55.58 + 5.97
56,96 5.98
Arssdd w/ PISSA
66.32 + 2.87
67.57 2.79
FedSVD wio orthonormal'
70.76 + 3.75
71.86 3.79
FedSVD' (Ours)
71.68 = 3.31 73.03 2.00
Heterogeneity of the data distribution (ax). To assess the robustness of FedSVD under varying degrees of non-i.i.d. data, partition the MNLI dataset across clients using various concentration parameters E {0.1, 0.2, 0.5) for the Dirichlet distribution. For each setting, we train models under DP-SGD
6, 5 10-5) and report the mean and standard deviation over 5 independent runs. We compare FedSVD with FedAvg and FFA-LoRA across all levels of heterogeneity. As shown in Fig. 4a, our proposed FedSVD consistently outperforms the baselines across all tested levels of data heterogeneity, except at a = 0.1, where extreme heterogeneity causes all methods to fail.
Varying the number of clients (K), To evaluate the robustness of each method in more realistic federated settings, we vary the total number of clients K E [6, 9, 12] while keeping the number of participating clients per round fixed at K" = 3. Here, K = 12 is near the maximum feasible value, as some clients already have fewer training samples than the data processed per round (i.e., batch size x local steps 7). We compare the performance of FedAvg, FFA-LORA, and FedSVD under DP-SGD (E = 6, 8 = 10-5 on the MNLI dataset and report the mean and standard deviation over 5 independent runs for each configuration. Fig. 4b shows that FedSVD consistently outperforms the baselines across all values of K. Notably, the performance degradation with increasing K is significantly smaller for FedSVD, showing its robustness to the number of clients.
Overcoming the SVD bottleneck: (1) Low-rank SVD. Although FedSVD introduces additional overhead due to the SVD step, this cost can be substantially mitigated by using randomized low-rank approximation techniques, e.g., the algorithm proposed by Halko al. [13, Algorithm 5.1 p. 29]. It iteratively approximates the leading singular components with high fidelity while significantly reducing computational complexity, making FedSvD feasible for practical use in large-scale feder- ated settings. We set the number of iterations for the low-rank approximation (niter) to 2 or 10. In Table 5, the approximation (denoted as Low-rank SVD) achieves similar accuracy to Full SVD (even better with niter=2 and 10 on ANLI/MNLI-Matchee approximately 60x (niter=2) or 10x (niter=10) faster than Full SVD. This demonstrates that Low-rank SVD can serve as an efficient alternative to full SVD without sacrificing accuracy.
and QNLI, respectively). while running
Overcoming the SVD bottleneck: (2) Frequency of SVD. To further alleviate the computational burden, we explore reducing the frequency of SVD itself. Specifically, we conduct an ablation study in which S based re-initialization is applied every 1, 2, 5, or 10 communication rounds.

---

Table 5: Results of FedSVD using the low-rank approximation (Low-rank SVD) with niter=2 or 10 under
DP-SGD (c = 6, 0 10-3) ). The average run-time per SVD step (in seconds) for Full SVD is 9.12 008, and for Low-rank SVD 0.15 aos (niter=2; 60x faster) or 0.89 (niter=10; 10x faster).
SNLI
MNLI (miter=2)
SST-2
QQP
QNLI
SVD strategy
(niter=2)
Matched
Mismatched
(niter=2)
(niter=2)
(niter=2)
(niter=10)
Full SVD
72.71 +788
71.57 4
73.03 +
91.32 055
72.42 + 1.10
75.50 + 4.20
75.50 4:40
Low-rank SVD
74.92 5:006
72.76 + 182
72.74 1.00
92.34 0.00
76.66 + 1.02
68.76 + 7.89
79.84 2006
Each configuration is denoted FedSVD (n), where n E {1, 2, 5, 10) denotes the re- initialization interval in rounds. As shown in Fig. 5, all FedSVD (n) variants exhibit better convergence than FFA-LORA, confirming the benefit of SVD re-initialization and its robust- ness the choice of interval n. Given their comparable performance, variants with less fre- quent re-initialization offer a favorable trade-
FedSVD (1)
FedSVD (2)
-
FedSVD (5)
FedSVD (10)
FFA-LORA
MNLI (matched)
MNLI (mismatched)
80-
Accuracy (%)
70-
6.0
5.0
50
40
40
30
30-
off' when computational efficiency is priori- tized. Accuracy remains stable across different re-initialization schedules, demonstrating the robustness of FedSVD to hyperparameter n.
20
40
60
80
100
20
40
60
80
100
Communication Rounds
Figure 5: Results of varying the SVD frequeney using the MNLI dataset under DP-SGD (E 6, 10-
5 Conclusion
In this work, we proposed FedsVD, simple yet effective method for fine-tuning language models with DP-SGD in FL. Instead of using a fixed random matrix A for LoRA, we periodically refactor the product of two LoRA adapter matrices B A with SVD and initialize A with the right singular vectors of BA. As A remains untrained and SVD is applied post-privatization of B, our method preserves differential privacy without incurring additional noise from matrix multiplication. Empir- ically, FedSVD consistently outperforms the relevant baselines, often achieving faster convergence.
Limitations. Although our approach shows promising re- sults in both private and non-private federated learning set- tings, the computation of SVD incurs additional overhead on the server side. However, since SVD is performed on low-rank matrices, this overhead can be significantly reduced by em- ploying randomized low-rank approximation methods, such as the algorithm proposed by Halko et al. [13, Algorithm 5. as shown in Table 5. Another limitation is the additional commu- nication overhead associated with the broadcast of the newly initialized Á matrix to clients after each SVD step. However, this cost can be avoided by decentralizing the SVD computa- tion. After aggregating B,, the server computes A, via SVD on the product B: and transmits only B, to the clients. Each client then reconstructs A, locally using the same procedure and obtains the updated pair (B,, A.). Since only B; is optimized during training while A; remains fixed, it is not necessary to transmit or aggregate A, at the server. Table 6 compares the communication cost when both the server and the clients perform SVD computations, showing that our proposed FedSVD, along with FFA- -FALL, achieves the lowest communication cost, since only the LoRA B matrix is transmitted.
Table 6: Communication cost per round (iee, the number of parame- ters exchanged between the server and clients) when using RoBERTa-large and applying LoRA with rank
Comm. Cost
Method
(# parameters.)
FedAvg FFA-LORA FLORA
786,432
393,216
52,169,730
FedEX-LoRA
52,169,730
FedSVD (ours)
393,216
Future work. Since our method is compatible with any FL setup employing LoRA, extending the empirical evaluation of FedSVD to a wider range of foundation models across different modalities promising direction for future work. Furthermore, deeper theoretical analysis of FedSVD's convergence dynamics, particularly for complex non-linear models, could provide valuable insights.
Broader impact. FedSVD advances data privacy in AI development by enabling stable and effec- tive training of neural networks under differential privacy within a federated learning framework, ensuring that sensitive data remains locally available to each client. By improving the robustness of presecyg fine-tuning for foundation models, FedSVD contributes to reducing the risk of information leakage and supports the responsible deployment of AI systems in sensitive domains.
10

---

Acknowledgments
We express our sincere gratitude to the anonymous reviewers (z29n, toPF, yjb1, and w3wU) for their valuable feedback and efforts in helping us improve this paper.
Funding. This work was supported by Institute for Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (RS-2019-11190755, Artificial Intelligence Graduate School Program (KAIST), No.RS-2002-1122077. 13, Meta-learning Applicable to Real-world Problems, No. R8-20201200153, Penetration Security Testing of ML Model Vulnerabilities and Defense), National Research Foundation of Korea (NRF) grant funded by MSIT (No. RS-2023-00256259, grant of the Korea Machine Learning Ledger Orchestration for Drug Discovery Project (K-MELLODDY), the Ministry of Health & Welfare and Ministry of Science and ICT, Republic Korea (grant number: 2024-00460870) the Bavarian State Min- istry of Science and the Arts (grant number: H.2- 11-F1116.N(//1122, and Samsung Research.
References
[1] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-318, 2016.
[2] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. SmolLM blazingly fast and remarkably powerful, 2024.
[3] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Ef- ficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473, IEEE, 2014.
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. opportunities and risks of foundation models. arXiv preprint arXiv:2108.072.: 2021.
[5] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluís Marquez, Chris Callison- Burch, and Jian Su, editors, Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/1//D15-1007 URL https: //tp//aclaaaaa aelannhology. org/D15-1075.
[6] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), 2021.
[7] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. International Conference Learning Representations (ICLR), 2023.
[8] Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Do- ran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.11653//1/N19-44223 URL petts://aallnttholle
[9] Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy, vol- ume of Foundations and Trends Theoretical Computer Science. Now Publishers Inc., Hanover, MA, 2014. ISBN 978 9781601988188.

---

[10] European Parliament and Council of the European Union. Regulation (eu) 2016/679 of the eu- ropean parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repeal- ing directive 95/46/ec (general data protection regulation). https: //t://euu htt///errll// lex..ex.e. europa eu/ con.eet/nnnnn/EN/////ee 1-88.
2016: FULL, 2016. OJ L. 119, 4.5.2016,
[11] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[12] Pengxin Guo, Shuang Zeng, Yanran Wang, Hujjie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation federated learning. International Conference on Learning Representations (ICLR), 2025.
[13] Nathan Halko, Per-Gumnas Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217-288, 2011.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. International Conference on Computer Vision (ICCV), 2015.
[15] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint vrXivv:909.066 2019.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In- lernational Conference on Learning Representations (ICLR), 2022.
[17] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured compression of large language models based on low-rank and sparse approximation.
International Conference on Machine Learning (ICML), 2023.
[18] Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou, Federated matrix factorization with privacy guarantee. Proceedings of the VLDB Endowment, 15(4), 2021.
[19] Shih- Shih-Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Ting Cheng, and Min-Hung Chen. DoRA: Weight-decompose low-rank adaptation. International Conference on Machine Learning (ICML), 2024.
[20] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint 1977v:190. 2019.
[21] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication--fficcenn learning of deep networks from decentralized data. Artificial intelligence and statistics (AISTATS), 2017.
[22] H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. International Conference on Learning Representations (ICLR), 2018.
[23] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and sin- gular vectors adaptation of large language models. Neural Information Processing Systems (NeurIPS), 2024.
[24] Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am'elie H'eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Cl'ement Crepy, Daniel Cer. Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni,
12

---

Eric Noland, Geng Yan, George Tucker, George-Christian Christian Muraru, Grigory Rozhdestvenskiy, Hemryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane La- banowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoe- sund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang. Oscar Wahltinez, Paige Bai- ley, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebas- tian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg. Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Brian Warkentin, Ludovie Peran, Minh Giang, Cl'ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Dou- glas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08230 2024.
[25] Chen Qu, Weize Kong, Liu Yang. Mingyang Zhang. Michael Bendersky, and Mare Najork. Natural language understanding with privacy-preservii bert. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1488-1497, 2021.
[26] Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. Natural language understand- ing with the quora question pairs dataset. arXiv preprint 1107.0090..0044. 2019.
[27] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[28] Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. FedEx-LoRA: Exact aggre- gation for federated parameter-efficien fine-tuning of foundation models. In NeurIPS 2024 Workshop Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024.
[29] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sen- timent treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. Associa- tion for Computational Linguistics. URL attpss:/acclatthllL
org/ 2rg/D13-1170/
[30] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing, pages 245-248. IEEE, 2013.
[31] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preservin federated learning. Internationl Conference on Learning Representations (ICLR), 2024.
[32] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Na- man Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint a 2rXiv:2307.09288, 2023.
[33] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations (ICLR), 2019.
13

---

[34] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adap- tations. Neural Information Processing Systems (NeurIPS), 2024.
[35] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent, edi- tors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. / httphhttpss//aaaaaataaaaee http//aaaaanoooooorrrrr....
doi: 10.18653/v1/N18- 101. URL.
1101/.
[36] Nan Wu, Farhad Farokhi, David Smith, and Mohamed Ali Kaafar. The value of collaboration in convex machine learning with differential privacy. In 2020 IEEE Symposium on Security and Privacy (SP), pages 304-317. IEEE, 2020.
[37] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in Py Torch. arXiv preprint arXiv:2109. (20.09..2299 /2298, 2021.
[38] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, al. Differentially private fine-tuning of language models. arXiv preprint arXiv:21 2021.
[39] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- ties, pages 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19 1472. URL https: aclaathooognto org/P19-1472/.
[40] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024
[41] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRa: Adaptive budget allocation for parameter- efficient fine-tuning, International Conference on Learning Representations (ICLR), 2023.
[42] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- LoRA: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint a0Xiv:2309.02411, 2023.
Appendix
A Proof of Proposition 3.2
Proof. Let x; dd with one-hot y; R Parameters:
A E rrxd B Rdxrr, Wi Rdaxds
W2 E R Cxds.
Now we assume has full row rank, Le. rank(A) r. We define activations with forward pass as:
h (W1 BA)x Rd With elementwise logarithm, let
a, ReLU(h;)) Rd
Wza, RCC
P: = softmax(z:),
l = -y log
L
71
Standard logit-space derivatives are Pi.e - Vieer
6e,
826
St,oo',
S diag(p.) - -P.P. 2
0z 02i.d
14

---

Let D. = diag( 1(b; > 0f) E Rd so Dł = D; D.T On any open region where the sign pattern of h, fixed. D; is constant and
a, = D; (W1 + BA) x,, Set (1,,))..;;,,, (1,r) Ax; R..
2; z (W + B
Then for each (I,...,C,,
dp
dp
2i,c
>
W2,co a, aa (Wxx)aa +
Σ
W2,co Di Babti..
Hence, on a fixed mask,
dzi
0 zi
W2 Dilii
(independent B);
i Bab
Bab B
Now apply the full second-order chain rule for (B)):
826
dza
Ozi Bpg
C Oli
Σ
Stod
Bub 2Bpg
B
Ōzi,e
Bas Q 8
c=1e==1
The second sum vanishes (affine logits in B). Substituting the first derivatives gives
b2
4.0 big
E(W2.ca Da, Si.ce WepepDppp) = (t,tt Doy [D,W]] S.W.D. Jay-
Bab BBp
Gee!
Therefore the per-sample Hessian (indexed by (a, b) rows/cols) is
H, V Vee[B) = (t,t,)) @ (D:W, S,W,D,) 2 O.
Averaging,
H 2 Be C Σ (t,t. @ (D.WTS.Wa.
Let == A Idn ( (d.dd) and define
M: 1 Σ ULn xx. (D.W S.W..) (la, 8 (x,) E R ((dnda)((ddds))
Then A) AMAT Now goal is to bound the following quantity
Amax (B; A))
Amax (ALMAAAT
K2( (B; Amin (B; Amin (AMRAT)
(14)
Using
Amax( (b;(B;A) |AM | |All (m)x)
and
|All2 IIc ② Al12 ||Al|2 Tmax (A), we bound the numerator in Eq. 14 follows: Amax (B; A)) & Omax (A)x \max(Mss).
Let R(AT)= Rd.. be an image of AT. By Rayleigh quotient characterization,
w MMW
Amin (MallrAM)())))) =
min
min w TMM,
WER(A W.W
W w
WER(A
11
for every w E R(AT) we have
w M&w Amin Amin(MaRRAAT wwl
Applying this to w = AT with v Rdr and then minimizing over || gives
Amin (H{(B:A)
min v TAMRAT |
min w -MM 1
> Amin(M
min ||ATv]] vl
[vll==1
15

---

Using Rayleigh quotient characterization again,
min AA'v Amin (AA") (An oni (A). l
Thus obtain
Tmax (max(A)² (A))2max (M&)
Amax(Ma)
N2 (H)(::A)
mmin (A)2 (MElR(AT)) (2))
(15)
Amin (Mr ((AA))
If the rows of A are orthonormal (so AAT I and m (A) mmnn 1), then (Mss)
(B;A) A)) & Xmii(MMelR(AN) K2 (B;
(16)
B Related Work
Federated learning. Federated Learning (FL) enables decentralized clients to collaboratively train models without sharing raw data. FedAvg [21] averages locally updated model weights to form global model, offering simple yet effective baseline. Built upon FedAvg, recent work has explored integrating Low-Rank Adaptation [LoRA; 16] into FL to reduce communication and computation overhead during model fine-tuning. For instance, Fed-IT [40] updates the adapter matrices and B of LoRA, averages each matrices separately. To aggregate product of B and A, several meth- ods have been proposed. FedEX-LoRA [28] introduces an additional correction matrix to mitigate aggregation error. FLoRA [34] stacks adapter matrices and reinitializes them randomly at the end of each communication round. FFA-LoRA [31] proposes to use a fixed randomly initialized matrix A, while training and aggregating only B. Lastly, Fed-SA [12] proposes learning both matrices A and B, but shares only A during aggregation. Our method is based on FFA-LoRA; however, we reinitialize the adapter matrices after aggregation to promote gradient stability and learning efficacy. Instead of using a fixed random matrix for A, we periodically reinitialize A using orthonormal hases via singular value decomposition (SVD) of BAA, which empirically accelerates optimization.
Differential privacy guaranteed federated fine-tuning. (). privacy [DP; 9] pro- vides rigorous framework ensuring that models trained on neighboring datasets, differing by only one data point, produce similar outputs, thereby preserving individual privacy. DP-SGD [30, 3, i] brings this guarantee to deep learning by adding noise to stochastic gradient updates. In FL. privacy guarantees depend on whether the central server trusted. In the centralized DP setting, clients send raw updates without local privacy, and DP is applied during global aggregation [22]. In the local DP setting, which assumes an untrusted server, each client ensures its update is differentially private before communication [36, 18, 25]. Our work adopts this stronger setting: we apply DP at the client level, so any shared updates (i.e., model parameters) are already privatized. By the composition property of DP, the final global model also satisfies DP. DP-SGD is unstable with large numbers of trainable parameters due to increased gradient sensitivity and noise injection [1, 38]. J. To address this, FFA-LORA [31] fixes the adapter matrix A in LoRA to reduce trainable parameters, limiting noise amplification and avoiding quadratic noise growth.
Parameter efficient fine-tuning. To mitigate the computational cost of fine-tuning language models, LoRA [16] injects trainable low-rank adapter matrices into some model components. Subsequent works have proposed variants to improve adaptability and efficiency. For example, DeltaLoRA [42] improves LoRA's expressivity by combining original weights with adapter outputs, thereby enhancing the representational power. LoSparse [17] integrates LoRA with sparsity constraints to prevent the pruning of essential neurons. DoRA [19] separates the magnitude and direction of the update by learning scaling factor for the update AW, while keeping the direction determined by the LoRA update BA.
Unlike these approaches, which aim to learn expressive low-rank approximations of weight updates, PiSSA [23] takes a more structural approach. first decomposes the original weight matrix us- ing SVD, then fine-tunes only the low-rank components corresponding to the top-r singular values, while freezing the residual parts. Our method differs from PiSSA in two key aspects. First, rather than decomposing the pretrained weights, we perform SVD on the aggregated adapter product B. A
16

---

to reinitialize low-rank components after aggregation of optimized B on the client side. This is distinct from PiSSA's fixed decomposition of model weights. Second, we enforee the rows of A to be orthonormal by initializing them with right singular vectors of BA, which empirically sta- bilizes training and accelerates optimization compared to a non-orthonormal structure. AdaLoRA [41] dynamically learns the optimal rank by parameterizing incremental updates through an SVD to dynamically prune and reallocate rank budget across layers based on the magnitude the sin- gular values during training. Unlike AdaLoRA, product B A and enforce the rows of A to be orthonormal by initializing them with right singular vectors.
employ SVD to refactor the aggregated adapter
C Dataset Statistics
In this section, we summarize the statistics of the datasets used our experiments (Table 7), and present the per-label data distribution across clients (%) for both two-class (SST-2, QQP, QNLI) and three-class (MNLI, SNLI) datasets using a Dirichlet distribution with a 0.5 and six clients in total (Table 8).
Table An overview datasets used our experiments.
Dataset
# Classes
Train
# Val
# Test
SNLI
3
550,152
10,000
10,000
MNLI (matched)
9,815
3
392,702
MNLI (mismatched)
9,832
SST-2
67,349
QQP
2
363,846
40,430
QNLI
2
104,743
5,463
HellaSwag
N/A
39,905
10,042
Table Per-label data distribution across clients (%) for datasets with two labels (SST-2, QQP. QNLI) and
three labels (MNLI, SNL.) under the Dirichlet partition (a: = 0.5, 6 clients). For the HellaSwag dataset, which contains 137 distinct activity labels, we do not report the detailed per-label distribution here; however, the partitioning strategy be found at https http:s:///e //github. com/ seanie e 12/ffed- avd.
Labels
Label
Client
Client
Client
Client 3
Client
Client
0.196
0.469
0.187
0.018
0.130
0.001
2
0.020
0.100
0.004
0.089
0.186
0.600
0.104
0.025
0.673
0.024
0.100
0.073
3
1
0.049
0.016
0.007
0.405
0.058
0,465
2
0.333
0.168
0.113
0.000
0.064
0.322
D
Additional Experiments
In this section, present additional experiments to empirically support the effectiveness of the proposed FedSVD.
Table 9: Comparison condition numbers, Kg (TT A))
Amax [I [B; A))/Amin (H(B;
Method
lk
3k
5k
FFA-LoRA
10.18
10.15
9.78
10.09
10.13
10.23
FedSVD
1.67
1.52
1.51
1.50
1.50
1.51
Oracle
1.01
1.04
1.03
1.02
Empirical validation of Proposition 3.2, To empirically support Proposition 3.2, we consider simple logistic regression setup which allows us to directly compute the actual condition num- ber during optimization. Therefore, directly measure the condition number (B;
17

---

for FFA-LORA, FedSVD, and oracle. Specifically, using the SST- 2 dataset, we (1) extract features X E Raxd using a pretrained BERT model, (2) train a logistic regression head on top of these frozen features, and (3) compute the Hessian's condition number directly during optimization. As shown in Table 9, we observe that the condition number of FedSVD remains consistently smaller than that of FFA-LoRA throughout training (up 5,000 iterations), closely matching the oracle. This confirms the practical advantage of FedSVD for optimization.
Amax (H
Table 10: Results the SNLI [5] dataset with different initializations of A. We report average accuracy and 95% confidence intervals over 5 runs.
Method
Accuracy
FedSVD wio orthonormal
69.48 i 9:45
FedSVD
81.16 2.37
Impact of orthonormal initialization on the SNLI dataset. In Table 4, we present the impact of different initializations of A and the effeet of orthonormality. The improvement from enforcing orthonormality (row 4: FedSVD w/o orthonormal vs. row 5: FedSVD (Ours)) appears marginal on the MNLI dataset-e.g- +0.92 percentage points (pp) for Matched and +1.17 pp for Mismatched. However, we find that the influence of orthonormality can vary considerably across datasets. To further investigate this, conducted additional experiment the SNLI dataset. As shown in Table 10, maintaining the orthonormal structure yields substantial performance gain of nearly 12 pp.
Integration of FedSVD to DoRA [19]. To investigate the potential of FedSVD on different parameter-efficiet fine-tuning methods, we conduct experiments using DoRA by learning only the B E Pd matrix and freezing the A E [R"xd matrix, the magnitude vector m E Rda, and the initial weight Wo agduexd,, For given input 1R the output of the DoRA layer is defined as
diag(m) diag( W + BA] (Wo BA)x,
where ||W BAll2, dR is a row-wise norm. After computing the SVD of BA, we re-initialize
the magnitude vector
m diag(m) diag( + BAl (Wo + BA).
The matrices and B are re-initialized in FedSVD with LoRA, ie,, UÍ:; r|E[;, r] and A V[; r]].
Table 11: Results DoRA [19] on the MI.N dataset. We report average accuracy 95% confidence
intervals over runs.
Method
Matched
Mismatched
FedSVD w/ LoRA
71.57
73.03
FedSVD
72.12
73.13
In Table 11, we observe that FedSVD with DoRA shows similar performance to FedSVD with LoRA on the MNLI dataset, demonstrating the generalizability of FedSVD across different parameter- efficient fine-tuning parameterizations. We note that DoRA is known to bring benefits primarily in complex tasks (e.g,, image generation, text generation), and its improvements on text classifica- tion benchmarks are often marginal (e.g., Table 3 in [23]).
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] " generally preferable to "[No] is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
used"). In
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We summarize our contribution in the introduction and support all the claims in the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included limitations of our proposed method in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
19

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
We have provided a complete proof in the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have specified all the implementation details in section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
20

---

If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use public benchmark datasetes for our experiments and include our code in supplementary file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/niips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips wwtcccccppccccoocococccccccoollt w/wpcd/s//////ee/eo/ooo/cc Sewidee//oeeeuumitiii Cote//ooeeeeeeeeeeeee commisssion ssionpolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
21

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: (Ys]
Justification:
We have specified all the implementation details in section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We perform experiments multiple times with different random seeds and pro- vide means and confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all details Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
22

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httees///eeuiipppeeeee www.eeiw....puu..a.eeee wwtp:cccppppcccceeeeeeeec httpa////ehcccciieeleeleeeee
Answer: [Yes]
Justification: We have reviewed the NeurIPS COde of Ethics and make sure the reserach conform with
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included societal impacts in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No, we do not describe any safeguards in paper.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we properly use public benchmark datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the pack- age should be provided. For popular datasets, paperswi www.wwttthhhee..ooeeooooooooeeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [No]
Justification:
We do not perform crowdsourcing experiments or research with human sub-
jects.
24

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not perform research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only use LLMs for writing and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
25


Reviews:
[{"review_id": "qma4EolHP5", "reviewer": "Reviewer_z79n", "summary": "This paper proposes a new FL algorithm for LoRA training, FedSVD. FedSVD uses reparameterization of LoRA adopter matrices BA based on SVD and only requires clients to train a matrix B, reducing the amount of noise under DP while keeping the learning capability. Through empirical experiments, the authors show that their proposed algorithm outperforms the baseline LoRA algorithms.", "questions": "- What are the baseline results for non-finetuned models and centralized finetuned models?\n- The performance of the proposed method when K=12 in Figure 4(b) is significantly better than the baseline. Do the authors have any intuition behind the results? Do they think the gap becomes more significant when we have larger number of clients?\n- Do the authors try different DP parameters from $\\epsilon=6$? Why not sweeping $\\epsilon$ values?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is clearly written \n- The idea of using SVD in LoRA training is simple but effective—the experiment results support the superiority \n- The experiment demonstrates the validity of the algorithm through multiple angles. In particular, the experiments listed in Section 3.3 support that the proposed algorithm is robust and its components, e.g., orthonormal initialization of A, are effective respectively.\n\nWeaknesses\n- The experiment uses only one model, RoBERTa-large, which is much smaller than the recent models such as Llama. Running experiments on larger models would validate that the scalability of the proposed method.", "was_revised": true, "final_justification": "The authors addressed most of the concerns/questions that I raised in my review, except for the limited model variation. While I still believe the experiment on larger models is insightful and natural thing to do, the rest of contributions made in the paper outweigh the weakness. As such, I would keep my recommendation for accepting the paper.", "rebuttal": "We sincerely appreciate you for your time, effort, and recognizing the simple-yet-effective design and robust experimental validation, and we kindly acknowledge that other reviewers emphasize **theoretical soundness** (toPF), **clear and well-written presentation** (w3wU, yjb1), **straightforward implementation** (yjb1, w3wU), **performance consistency** (toPF), and **practical relevance and timeliness** (toPF, yjb1, w3wU). We address your concerns below and would be glad to dicuss additional points during discussion period.\n\n---\n>**[Q1]** The experiment uses only one model, RoBERTa-large, which is much smaller than the recent models such as Llama. Running experiments on larger models would validate that the scalability of the proposed method.\n- We agree that experiments with larger models would help validate the scalability of $\\texttt{FedSVD}$. However, we assume that **individual clients operate under limited computational and memory resources**.\n- This **low-resource assumption** is commonly adopted in the federated learning literature, for example, in mobile-device federated learning scenarios [1, 2, 3, 4].\n- In such realistic settings, deploying billion-scale language models like LLaMA is usually not feasible. \n- Instead, we extend our experiments to more complex data. Specifically, we use the HellaSwag benchmark [5] along with SmolLM-360M [6], a recent and efficient model. This setup demonstrates the scalability of $\\texttt{FedSVD}$ with respect to data size and task complexity.\n- As shown in the table below, FedSVD again outperforms other baselines under DP-constraints with $\\epsilon=6$ and $\\delta=10^{-5}$. The results highlight the core strengths of our method. FedAvg's performance degrades significantly due to the noise amplification from the DP-SGD. FFA-LoRA improves upon FedAvg by fixing one of LoRA adapters $A$. FedSVD provides further improvement by adaptively aligning $A$ with principal directions of aggregated updates, while also avoiding noise amplification.\n\n    **[Table R1. HellaSwag with SmolLM-360M]**\n    |Method|Acc|\n    |-|-|\n    |FedAvg|$48.81\\pm 0.28$|\n    |FFA-LoRA|$49.76\\pm0.09$|\n    |$\\texttt{FedSVD}$|$\\textbf{51.10}\\pm0.16$|\n---\n>**[Q2]** What are the baseline results for non‑finetuned models and centralized finetuned models?\n- Thank you for the question. GLUE consists of text classification tasks, so a model **without task‑specific fine‑tuning** (e.g., a randomly initialized linear head) performs around **random/majority‑class** levels: roughly **$\\approx33\\%$** for 3‑way tasks (e.g., MNLI) and **$\\approx50\\%$** for balanced binary tasks (e.g., SST‑2). For imbalanced binaries (e.g., QQP), the majority‑class baseline can be slightly higher.\n- We report the performance of a **centralized full fine‑tuned** model by training RoBERTa‑Large on GLUE without FL or DP and then evaluating as follows:\n\n    **[Table R2. Centralized full fine-tuned model.]**\n    |MNLI (matched)|MNLI (unmatched)|SST-2|QQP|QNLI|Avg|\n    |-|-|-|-|-|-|\n    |$90.2$|$90.2$|$96.4$|$92.2$|$94.7$|$92.74$|\n- We will include the above as the **upper bound** in the revision.\n---\n>**[Q3]** The performance of the proposed method when $K=12$ in Figure 4(b) is significantly better than the baseline. Do the authors have any intuition behind the results?\n- Thank you for the insightful question. Our core intuition is that the performance gap widens with a larger client pool because FedSVD's adaptive mechanism creates a shared, well-conditioned projection matrix $A$ that is derived from the collective updates of the diverse clients using SVD. This provides a stable optimization landscape for all participants, a task where static baselines increasingly underperforms as data heterogeneity grows.\n- FedSVD leverages diversity to create a data-driven **consensus projection**. With a larger client pool, the server aggregates $B$ matrices from a more heterogeneous set of local data distributions. It then performs SVD on the product of the aggregated $B$ and the previous $A$. The new shared $A$ is composed of the orthonormal right singular vectors from this operation, allowing it to **better capture the principal directions of the aggregate updates** across the entire federation.\n- This data-driven consensus projection is crucial for heterogeneous data. Our theoretical analysis shows that using an orthogonal $A$ improves the condition number of the Hessian matrix for the local optimization of B. This creates a **better-behaved optimization landscape** that promotes **faster and more stable convergence** for every client. \n- In contrast, a baseline like FFA-LoRA uses a fixed, randomly initialized matrix for A, which becomes an increasingly poor, one-size-fits-none projection as client diversity grows, thus hindering performance.\n---\n>**[Q4]** Do they think the gap becomes more significant when we have larger number of clients?\n- Yes, we hypothesize that **the performance gap will become more significant with a larger number of clients**. As the client pool and data heterogeneity grow, the limitation of a non-adaptive projection matrix becomes a more severe bottleneck. \n- Conversely, FedSVD's adaptive SVD mechanism receives richer information from the diverse updates, enabling it to construct an even more robust and generalized A matrix. This should further widen the performance gap between our method and the baselines.\n---\n>**[Q5]** Try different DP parameters, other than $\\epsilon=6$.\n- Thank you for pointing this out. We conducted an additional experiment on the GLUE benchmark using $\\epsilon=3$, $\\delta=10^{-5}$, with the same experimental setup used in `Table 2`:\n\n    **[R3. GLUE experiments with $\\epsilon=3,\\delta=10^{-5}$.]**\n    |Method|MNLI (matched)|MNLI (unmatched)|SST-2|QQP|QNLI|Avg|\n    |-|-|-|-|-|-|-|\n    |FedAvg|$50.45\\pm14.35$|$50.82\\pm14.59$|$50.64\\pm0.55$|$57.91\\pm7.31$|$50.11\\pm0.37$|$51.99\\pm5.11$|\n    |FFA|$57.39\\pm9.32$|$59.00\\pm9.42$|$\\textbf{90.69}\\pm{1.23}$|$70.88\\pm2.23$|$74.59\\pm1.70$|$70.51\\pm3.30$|\n    |$\\texttt{FedSVD}$ (ours)|$\\textbf{69.84}\\pm5.36$|$\\textbf{70.94}\\pm5.40$|$89.93\\pm0.80$|$\\textbf{73.67}\\pm1.67$|$\\textbf{77.25}\\pm2.09$|$\\textbf{76.32}\\pm2.29$| \n   \n- We observe that FedAvg collapses to near random guess under the stricter privacy constraint, demonstrating its sensitivity to reduced $\\epsilon$.\n- In contrast, $\\texttt{FedSVD}$ continues to outperform all baselines even at $\\epsilon=3$, highlighting its robustness to stronger privacy constraints.\n- We will include this result and discussion in the revision.\n---\n### References\n\n[1] Kairouz, Peter, et al. “Advances and Open Problems in Federated Learning.” Foundations and Trends in Machine Learning. 2021.\n\n[2] Bonawitz, Keith, et al. “Towards Federated Learning at Scale: System Design.” SysML. 2019.\n\n[3] Hard, Andrew, et al. “Federated Learning for Mobile Keyboard Prediction.” arXiv. 2018.\n\n[4] McMahan, H. Brendan, et al. “Communication‑Efficient Learning of Deep Networks from Decentralized Data.” AISTATS. 2017.\n\n[5] Zellers, Rowan, et al. \"HellaSwag: Can a machine really finish your sentence?.\" ACL. 2019.\n\n[6] Allal, Loubna Ben, et al. \"SmolLM2: When Smol Goes Big--Data-Centric Training of a Small Language Model.\" arXiv 2025."}, {"review_id": "0Q1ulAH1Zf", "reviewer": "Reviewer_toPF", "summary": "The paper introduces FedSVD, a method designed to improve the efficiency and stability of federated learning (FL) with Low-Rank Adaptation (LoRA) under differential privacy (DP). The key innovation is a global reparameterization based on singular value decomposition (SVD), which allows the matrix $A$ to adapt over time while mitigating noise amplification. In this approach, clients optimize and transmit only the $B$ matrix, while the server aggregates these updates, computes $BA$, and then refactors it via SVD to produce a new adaptive $\\hat{A}$ and updated $B$ based on the clients' data. The paper claims that FedSVD consistently improves performance across various privacy settings and benchmarks, outperforming relevant baselines in both private and non-private regimes. Empirical results on GLUE tasks demonstrate the effectiveness of FedSVD.", "questions": "- How does FedSVD fundamentally differ from the decomposition-based federated learning methods in [1, 2,3 ] (or other similar studies)?\n  - Can you clarify the unique contributions of FedSVD beyond its application to LoRA?\n- How does the privacy-utility trade-off of FedSVD compare with other recent DP-FL methods? Could you provide a more comprehensive evaluation of privacy budgets across different methods?\n- Does the additional communication cost of broadcasting $\\hat{A}$ after each SVD step become a bottleneck in large-scale settings? How does this compare to existing lightweight methods like FFA-LoRA? (Could this be beneficial for federated fine-tuning a.k.a. maml or partial learning?)\n- Could you evaluated FedSVD on larger or more complex datasets beyond GLUE? (e.g., SNLI[4], HellaSwag[5], MATH[6] )\n\n[4] A large annotated corpus for learning natural language inference (2015)\n\n[5] Hellaswag: Can a machine really finish your sentence? (2019)\n\n[6] Measuring mathematical problem solving with the math dataset (2021)", "limitations": "The authors have discussed some limitations of the work in the conclusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. The introduction of SVD-based reparameterization to mitigate noise amplification appears to be an effective approach in DP settings.\n2. The paper provides deep theoretical analysis showing how orthonormal rows of $A$ can improve the condition number of the Hessian, leading to better optimization dynamics.\n3. Empirical evaluations on GLUE tasks demonstrate consistent performance improvements over baselines.\n\n### Weaknesses:\n1. A major concern is the novelty. There have been other studies on decomposition-based FL approaches in DP settings [1, 2, 3] that are not compared in the paper. The novelty here is mostly limited to applying these techniques to the new problem domain of LoRA training.\n2. Although the authors claim that FedSVD improves performance across various DP settings, the privacy budget comparison with recent methods is not provided.\n3. While the method introduces additional communication costs of broadcasting newly initialized $\\hat{A}$ matrix to clients after each SVD step, the performance improvement seems to be marginal compared to the prior method FFA-LoRA.\n4. While the paper demonstrates effectiveness on GLUE tasks, it remains unclear how well FedSVD scales to larger datasets or more complex models.\n\n[1] Privacy-Preserving Distributed SVD via Federated Power (FedPower), CoRR 2021  \n\n[2] Privacy-Preserving Federated Singular Value Decomposition, Applied Sciences 2023\n\n[3] Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models, AAAI 2025", "was_revised": true, "final_justification": "Based on the author's additional explanation of the novelty and the extended experimental efforts, I have decided to raise my overall score.", "rebuttal": "We sincerely appreciate you for recognizing the theoretical soundness and performance consistency of our approach, and we kindly acknowledge that other reviewers emphasize **effective and simple design** (z79n, yjb1, w3wU), **robust experiments** (z79n, w3wU), **clear and well-written presentation** (z79n, yjb1, w3wU), and **practical relevance and timeliness** (yjb1, w3wU). We address your concerns below and would be glad to dicuss additional points during discussion period.\n\n---\n>**[Q1]** A major concern is the novelty. There have been other studies on decomposition-based FL approaches in DP settings [1, 2, 3] that are not compared in the paper. The novelty here is mostly limited to applying these techniques to the new problem domain of LoRA training.\n- Thank you for your constructive feedback and providing relevant literature. However, we **respectfully disagree with the assessment that our novelty is limited** with respect to [1, 2, 3].\n- The cited works address **fundamentally different problems** from our own. The fundamental difference between our approach and FedPower, is that FedSVD is a model adaptation algorithm, whereas FedPower is a data analysis algorithm. We believe our contribution is a novel algorithmic solution to a specific challenge in FL with DP, not merely an application of existing techniques to LoRA.\n- The core novelty of our $\\texttt{FedSVD}$ is its **use of SVD as a server side, postprocessing step to reparameterize LoRA adapters** $A$ and $B$ to solve the problem of optimization dynamics due to using fixed $A$ and noise amplicafication that arises when combining LoRA and DP-SGD.\n- Specifically, Guo et al. [1] and Liu et al. [2] focus on computing the **SVD of a large data matrix** distributed across clients with privacy guarantees, which is **not the same as optimizing neural network parameters**.\n- In contrast, $\\texttt{FedSVD}$ performs a standard, centralized SVD on the small, aggregated LoRA adapters $BA$ on the server. Rather than analyzing client data, we use SVD to **re-initialize one of the LoRA adapters**, $A$, to align with the principal directions of the aggregated updates.\n- This process is designed to **avoid noise amplification** and, as our analysis shows, simultaneously helps to **stabilize training and accelerate convergence** with formal DP guarantee. These are fundamentally different goals and technical challenges.\n- Mahla et al. [3] fine-tunes all up-projection layers with GaLore [4], which significantly increases training and communication cost due to the vastly larger number of trainable parameters (e.g., **from 393,216 in our LoRA setup to 357,199,876**). This is **infeasible in our problem setting**, because individual clients often lack the computational and memory capacity required for full fine‑tuning of large models, as noted in `L39–40`. \n- Moreover, Mahla et al. [3] did not target DP guarantees, and a **larger number of parameters can make DP‑SGD unstable** [5].\n- On the other hand, FedSVD is designed for the highly parameter-efficient LoRA framework, making it suitable for resource-constrained federated learning environments, while providing a formal privacy guarantee. \n### Summary\n- Guo et al. [1], Liu et al. [2], and Mahla et al. [3] are relevant to our work; however, they address **fundamentally different problems** than ours and are **not applicable to our problem setting**.\n- The fundamental difference between our approach and FedPower, is that FedSVD is a model adaptation algorithm, whereas FedPower is a data analysis algorithm.\n- We will clarify this difference in the revision.\n---\n>**[Q2]** Although the authors claim that FedSVD improves performance across various DP settings, the privacy budget comparison with recent methods is not provided.\n- Thank you for pointing this out. We conducted an additional experiment on the GLUE benchmark using $\\epsilon=3$, $\\delta=10^{-5}$, with the same experimental setup used in `Table 2`:\n\n    **[Table R1. Experiments with $\\epsilon=3,\\delta=10^{-5}$.]**\n    |Method|MNLI (matched)|MNLI (unmatched)|SST-2|QQP|QNLI|Avg|\n    |-|-|-|-|-|-|-|\n    |FedAvg|$50.45\\pm14.35$|$50.82\\pm14.59$|$50.64\\pm0.55$|$57.91\\pm7.31$|$50.11\\pm0.37$|$51.99\\pm5.11$|\n    |FFA|$57.39\\pm9.32$|$59.00\\pm9.42$|$\\textbf{90.69}\\pm{1.23}$|$70.88\\pm2.23$|$74.59\\pm1.70$|$70.51\\pm3.30$|\n    |$\\texttt{FedSVD}$ (ours)|$\\textbf{69.84}\\pm5.36$|$\\textbf{70.94}\\pm5.40$|$89.93\\pm0.80$|$\\textbf{73.67}\\pm1.67$|$\\textbf{77.25}\\pm2.09$|$\\textbf{76.32}\\pm2.29$|\n\n    \n- We observe that **FedAvg collapses to near random guess** under the stricter privacy constraint, demonstrating its sensitivity to reduced $\\epsilon$.\n- In contrast, $\\texttt{FedSVD}$ **continues to outperform all baselines** even at $\\epsilon=3$, highlighting its robustness to stronger privacy constraints.\n- We will include this result and discussion in the revision.\n---\n>**[Q3]** While the method introduces additional communication costs of broadcasting newly initialized matrix to clients after each SVD step, the performance improvement seems to be marginal compared to the prior method FFA-LoRA.\n- Thank you for your comments on efficiency and performance. We would like to respectfully clarify what we believe is a misunderstanding on two points: **1) communication cost** and **2) significance of the performance improvement**.\n- Our proposed method, $\\texttt{FedSVD}$, is designed to have the exact **same communication cost as the FFA-LoRA**. While we acknowledge in `L320-329` of `limitations` that broadcasting the newly initialized $A$ matrix is a possibility, we also describe the efficient mechanism used in our experiments to avoid this cost. \n- In this approach, the server **transmits only the aggregated $B$ matrix**, and each client locally reconstructs the new $A$ matrix, thereby incurring no additional communication overhead compared to FFA-LoRA.\n- While $\\texttt{FedSVD}$ introduces additional overhead due to the SVD step, as discussed in `Section 4`, **this cost can be substantially mitigated** by leveraging randomized low-rank approximation techniques, such as the method described in [6]. \n- As demonstrated in the table below, this approximation maintains performance while **significantly reducing the runtime of the SVD operation**. Moreover, as shown in `Fig. 5`, the computational cost can be further reduced by performing the SVD step less frequently.\n    \n    **[Table R2.Experiemtns with low-rank SVD.]**\n    |Method|Matched Acc|Mismatched Acc|Avg. Run-time|\n    |-|-|-|-|\n    |$\\texttt{FedSVD}$ w/ full SVD|$71.57\\pm3.18$|$73.03\\pm2.89$|$9.07\\text{s}\\pm0.07$|\n    |$\\texttt{FedSVD}$ w/ low-rank SVD|$72.76\\pm1.82$|$72.74\\pm1.64$|$0.13\\text{s}\\pm0.06$|\n- Under DP constraints ($\\epsilon=6,\\delta=10^{−5}$) in `Table 2`, $\\texttt{FedSVD}$ achieves an average accuracy of 76.79%, which is a substantial **8.77% improvement** over FFA-LoRA's 68.02%. \n- Furthermore, in additional experiments (see our answer to **[Q2]**) under an even stricter privacy budget of $\\epsilon=3$, $\\texttt{FedSVD}$ continues to significantly **outperform FFA-LoRA by 5.81%** in average accuracy (76.32% vs. 70.51%).\n---\n>**[Q4]** While the paper demonstrates effectiveness on GLUE tasks, it remains unclear how well FedSVD scales to larger datasets (SNLI[7], HellaSwag[8], MATH[9]) or more complex models.\n- Following your suggestion, we first fine-tune RoBERTa-large on SNLI dataset [7] under DP constraints $(\\epsilon=6, \\delta=10^{-5})$ and report accuracy on the table below. FedSVD  again outperforms the baselines on the SNLI dataset.\n\n    **[Table R3. SNLI with RoBERTa-large.]**\n    |Method|Acc|\n    |-|-|\n    |FedAvg|$74.00\\pm1.82$|\n    |FFA|$73.98\\pm4.23$|\n    |$\\texttt{FedSVD}$|$\\textbf{81.16}\\pm2.37$|\n- Furthermore, we fine-tune a SmolLM-360M model [10] on more complex dataset, HellaSwag [8] under DP constraints ($\\epsilon=6, \\delta=10^{-5}$).\n\n\n- FedSVD again outperforms other baselines under DP-constraints with $\\epsilon=6$ and $\\delta=10^{-5}$. The results highlight the core strengths of our method. FedAvg's performance degrades significantly due to the noise amplification from the DP-SGD. FFA-LoRA improves upon FedAvg by fixing one of LoRA adapters $A$. FedSVD provides further improvement by adaptively aligning $A$ with principal directions of aggregated updates, while also avoiding noise amplification.\n\n    **[Table R4. HellaSwag with SmolLM-360M.]**\n    |Method|Acc|\n    |-|-|\n    |FedAvg|$48.81\\pm 0.28$|\n    |FFA-LoRA|$49.76\\pm0.09$|\n    |$\\texttt{FedSVD}$|$51.10\\pm0.16$|\n---\n\n### References\n\n\n[1] Guo, Xiao, et al. \"Privacy-preserving distributed SVD via federated power.\" arXiv. 2021.\n\n[2] Liu, Bowen, Balázs Pejó, and Qiang Tang. \"Privacy-preserving federated singular value decomposition.\" Applied Sciences. 2023.\n\n[3] Mahla, Navyansh, Kshitij Sharad Jadhav, and Ganesh Ramakrishnan. \"Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models.\" AAAI. 2025.\n\n[4] Zhao, Jiawei, et al. \"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.\" ICML. 2024.\n\n[5] Hölzl, Florian A., Daniel Rueckert, and Georgios Kaissis. \"Equivariant differentially private deep learning: Why DP-SGD needs sparser models.\" ACM Workshop. 2023.\n\n[6] Halko, Nathan, Per-Gunnar Martinsson, and Joel A. Tropp. \"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.\" SIAM. 2011.\n\n[7] Bowman, Samuel R., et al. \"A large annotated corpus for learning natural language inference.\" ACL. 2015.\n\n[8] Zellers, Rowan, et al. \"Hellaswag: Can a machine really finish your sentence?.\" ACL. 2019.\n\n[9] Hendrycks, Dan, et al. \"Measuring mathematical problem solving with the math dataset.\" NeurIPS. 2021.\n\n[10] Allal, Loubna Ben, et al. \"SmolLM2: When Smol Goes Big--Data-Centric Training of a Small Language Model.\" arXiv. 2025."}, {"review_id": "Ws6ZrP1WJO", "reviewer": "Reviewer_yjb1", "summary": "This submission address low-rank adaptation in federated learning (FL) under the use of differential private stochastic gradient descent (DP-SGD). The authors tackle a known problem in literature, which is related to the noise amplification of DP when applied to both matrices (A,B) of the low-rank reparametrization. Similarly to previous works, they propose to explicitly train locally only the B matrix, while A ramains fixed during each round. Differently from other works, the A matrix changes across rounds, as an effect of further singular value decomposition at the server side. The authors provide a motivation for the proposed approach, highlighting the orthonormalization of A as a factor that promotes better conditioned optimization.\nExperiments show promising results on 5 GLUE datasets, with/without DP.", "questions": "* Can the authors explain why other intuitive strategies for avoiding the noise amplification (see example above) are not equally valid ad the proposed approach?\n* Can the authors better motivate why the proposed approach does not limit the expressiveness of the decomposition, even if the matrix A is effectively never directly updated by training?\n* Can the authors discuss theoretically how the techniques relates to priors works and guarantess with low-rank training?", "limitations": "Yes, even if the limitations appear significant for large language models (e.g. computing SVD very often)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strenghts\n\n* **The paper is well written:** the paper is easy to follow, sections are clear and reasonable coverage of closely related approaches is provided. Some details appear superflous for a research paper, and risk to be not accurate, such as eq. 1, which is just the GD update rule at each local step and the def. of the cross-entropy loss. This could be a typo, since the algorithms should use SGD, and not GD locally (as per eq. 1, the loss is calculated on the entire local dataset $\\mathcal{D}_k$).\n* **The proposed approach is simple:** conceptually, the approach is of straightforward understanding, and it is trivial to implement. Under full comprehension of the induced dynamics, which is not trivial at all (see weaknesses on that), the approach could have immediate practical implementation.\n\n## Weaknesses\n1. **Weak motivation for the proposed approach:** the aim of the work is to avoid the noise amplification that follows the use of DP-SGD in FL. To do so, authors rely on what other works do, that is train locally only the B matrix, and add on top a further SVD at the server after aggregation with the purpose of not limiting the expressiveness of A. However, the motivation behind the precise choice of recomputing an SVD after training only B is not clear, it seems an heuristic similar to other intuitive ideas one can come up. For example, clients could optimize A or B depending on the round: this would similarly avoid the noise amplification and effectively train both matrices, with the same communication complexity and without the burden of the additional SVD. Without stronger motivations for the precise steps of the proposed algorithm, this work does not offer a clear advancement of our understanding on low-rank (federated) training.\n2. **Insufficient theoretical rationale:** The main point of the theoretical discuss points to the orthonormality of A as a way to have a stricter upper bound on the condition number of the problem. This analysis is insufficient in explaining the characteristics of the method for two reasons: (i) It holds on a very restrictive setting (logistic regression and one layer network) and (ii) does not reflect where the real advatage comes from experimentally. Indeed, the orhonormality of A does not appear as the main factor improving the performance (see Table 3). Better analysis should be provided to offer relevant insights on the induced dynamics of low-rank training\n3. **Missing comparison with low-rank training works:** this paper compares with closely related lora-based methods, but does not discuss the relationship with related FL papers on low-rank training. For example [1] exploits low-rankness of the subspace spanned by gradients to drastically reduce the number of parameters transmitted through the network. That algorithm should be compliant with the premises of this work (avoid DP noise amplification or restricting expressiveness), and so it should be considered as competing baseline. \n\n\n[1] Azam et. al, Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?, ICLR 2022", "was_revised": true, "final_justification": "The authors sufficiently addressed my core concerns, so I turn my recommendations towards acceptance. The author provided detailed responses and additional experiments following the suggestions arising in our discussion.\nAmong those, the controlled experiments that show the lower condition number are greatly appreciated, since serve as empirical demonstration of the main theoretical claim. Other experiments provided in their last message show that simpler baselines do not achieve a comparably good result even with additional tuning, which further strenghtens the empirical part.\n\nThose motivations and evidence do clarify theoretical motivation of the method and show why refactorization of BA product is necessary for superior performance, and for this reason I am revising my score from 2 to 4.\n\nThere are mangin for improvement in clarity (I trust authors will enhance incorporating suggestions) and in proposing controlled experiments directly tied to measure the distance from the proposed solution (i.e. refactorize BA via SVD) and what should be optimal (but infeasible in practice in FL) one.", "rebuttal": "We sincerely appreciate you for your time, effort, and recognizing the clear presentation and straightforward implementation, and we kindly acknowledge that other reviewers noted the **simple and effective design** (z79n, toPF, w3wU), **robust and consistent empirical results** (z79n, toPF, w3wU), **theoretical depth** (toPF), and **practical impact and timeliness** (toPF, w3wU). We address your concerns below and would be glad to dicuss additional points during discussion period:\n\n---\n> **[Q1-1]** The motivation for recomputing SVD after training only B is not clear.\n- We respectfully note that the SVD step of our $\\texttt{FedSVD}$  is **not an arbitrary choice**. It is the **mathematically optimal way** to factorize the aggregated knowledge from clients. \n- After the server aggregates the client updates to form $B_{i+1}$, the complete collective update is represented by the product $B_{i+1}\\hat{A}_i$. $\\texttt{FedSVD}$ uses SVD to refactorize this product. \n- As established by the **Eckart-Young-Mirsky theorem** [1], the truncated SVD provides the best possible rank-$r$ approximation of a matrix. \n- Since the rank of $B_{i+1}\\hat{A}\\_{i}$ is at most $r$, our SVD procedure is a **lossless operation** that perfectly preserves the aggregated update information while reparameterizing it into a new, synchronized pair of matrices, $\\hat{A}\\_{i+1}, \\hat{B}\\_{i+1}$ for the next round. \n- This is a **principled method for information preservation**, not a heuristic.\n- We will clarify the above in the revision.\n---\n> **[Q1-2]** Clients could optimize A or B depending on the round.\n- We appreciate your insightful suggestion. Following your suggestion, we perform experiments where clients alternate between training either the matrix $A$ or $B$ in each round. We refer to this method as **FedRand**.\n- This approach has the same communication complexity as our proposed method and avoids the SVD computation. However, our empirical results in the following table show that this alternating training scheme leads to **a significant drop in performance** across most datasets.\n\n    **[Table R1. Comparison with FedRand.]**.\n    |Method|MNLI (matched)|MNLI (mismatched)|SST-2|QQP|QNLI|\n    |-|-|-|-|-|-|\n    |FedAvg|$65.44\\pm6.14$|$67.02\\pm5.93$|$89.41\\pm1.35$|$58.58\\pm5.26$|$60.69\\pm6.04$|\n    |FFA|$55.56\\pm8.58$|$56.39\\pm8.93$|$91.41\\pm0.86$|$64.35\\pm 0.86$|$72.38\\pm4.96$|\n    |FedRand|$42.14\\pm7.20$|$41.73\\pm7.02$|$90.68\\pm0.59$|$55.15\\pm5.57$|$49.46\\pm0.00$|\n    |$\\texttt{FedSVD}$|$\\textbf{71.68}\\pm3.18$|$\\textbf{73.03}\\pm2.89$|$91.31\\pm0.85$|$\\textbf{72.41}\\pm2.36$|$\\textbf{75.49}\\pm4.20$|\n- Regarding the computational burden of SVD, we argue this cost is **manageable and can be made negligible**. First, as shown in `Fig. 5`, $\\texttt{FedSVD}$ 's performance is robust to less frequent SVD computations. Second, we conduct new experiments during the rebuttal period using a randomized low-rank SVD approximation [2]. \n- As shown below, this approach retains full accuracy while reducing the average **server-side runtime by over 98%** on the MNLI dataset.\n\n    **[Table R2. Experiments with low-rank SVD.]**\n    |Method|Matched Acc|Mismatched Acc|Avg. Run-time|\n    |-|-|-|-|\n    |$\\texttt{FedSVD}$ w/ full SVD|$71.57\\pm3.18$|$73.03\\pm2.89$|$9.07\\text{s}\\pm0.07$|\n    |$\\texttt{FedSVD}$ w/ low-rank SVD|$72.76\\pm1.82$|$72.74\\pm1.64$|$0.13\\text{s}\\pm0.06$|\n---\n> **[Q2-1]** The theoretical analysis holds on a very restrictive setting.\n- Yes, this is a restricted setting, but at least provides **insight into why orthonormality can facilitate optimization**. This simplification of deep learning models into linear ones has been indeed explored in prior works [3, 4]. \n- We believe that extending this theoretical analysis to more complex non-convex optimization settings remains an **important direction for future research**.\n---\n>**[Q2-2]** Othonormality of A is not the main factor improving the performance (Table 3).\n- Thank you for pointing it out. In addition to the merit of orthonormality, we explain why periodically aligning the matrix $A$ with the principal direction of the aggregated update is crucial. \n- It ensures the model's updates are applied in the most effective and impactful way, preventing the learning process from being constrained by outdated or random directions.\n- We can formalize this core benefit. At the beginning of round $i+1$, the server has the aggregated client matrix, $B_{i+1}$ and the previous round's matrix $\\hat{A}_{i}$. \n- The complete collective knowledge from the previous round is captured in the product $\\Delta\\_{i+1}=B\\_{i+1}\\hat{A}\\_{i}$. The server's objective is to apply a rank-$r$ update, $\\hat{B}\\_{i+1}\\hat{A}\\_{i+1}$, that best reflects this collective knowledge. This is equivalent to finding the rank-$r$ matrix $\\hat{B}\\_{i+1}\\hat{A}\\_{i+1}$ that minimizes the approximation error with respect to $\\Delta\\_{i+1}$: \n$$\n\\min\\_{\\hat{A}\\_{i+1}, \\hat{B}\\_{i+1}}\\lVert \\Delta\\_{i+1}- \\hat{B}\\_{i+1}\\hat{A}\\_{i+1} \\rVert_F^2 \\; \\text{ subject to } \\text{rank}(\\hat{A}\\_{i+1}) \\leq r, \\text{rank}(\\hat{B}\\_{i+1}) \\leq r \n$$\n- The Eckart-Young-Mirsky [1] theorem provides the optimal solution to this problem. It states that the **best rank-$r$ approximation of a matrix $\\Delta_{i+1}$ is obtained by its truncated SVD**. Our $\\texttt{FedSVD}$  method implements exactly this solution. \n- Since $\\texttt{FedSVD}$  defines the new matrices as $\\hat{A}\\_{i+1}=V^\\top[:r, :]$ and $\\hat{B}\\_{i+1}=U[:, :r]\\Sigma[:r, :r]$, where $U\\Sigma V^\\top = B\\_{i+1}\\hat{A}\\_{i}$. Since the rank of $\\Delta\\_{i+1}$ is at most $r$ by the design of LoRA, the rank-$r$ SVD perfectly reconstructs the matrix, meaning $\\Delta\\_{i+1}=\\hat{B}\\_{i+1}\\hat{A}\\_{i+1}$. \n- Therefore, the SVD reparameterization is not just an arbitrary update rule. It is the **mathematically optimal strategy for preserving and propagating the aggregated knowledge** for all clients into the new low-rank adapters for the next round.\n---\n>**[Q3]** A missing baseline [5] exploits low-rank of the subspace spanned by gradients.\n- We thank you for highlighting the highly relevant work by Azam et al [5]. While both our $\\texttt{FedSVD}$ and their LBGM algorithm leverage the low-rank properties, we believe they **address orthogonal problems, making LBGM not a direct experimental baseline**. Specifically, our work focuses on **private federated learning in resource-constrained environments**, while LBGM is a general-purpose compression technique for full gradients.\n- We agree this is an important connection to make, and we will gladly add a **discussion of Azam et al. to `Section 2`** in a revision to better contextualize our contributions. The reasons why it is more suitable for discussion rather than direct experimental comparison are twofold:\n- $\\texttt{FedSVD}$ is designed for the parameter efficient fine-tuning (PEFT) setting where fine-tuning large models makes the **computation or transmission of the full gradient infeasible** for clients. Our method operates exclusively on small LoRA adapter matrices. In contrast, LBGM's core mechanism requires clients to periodically **compute and transmit the entire gradient vector**, representing a fundamentally different operational paradigm. Comparing a PEFT method to a full-gradient compression technique would be an apples-to-oranges comparison.\n- Our primary contribution is to **mitigate the DP noise amplification** that is specific to the LoRA architecture in a private FL setting. The **LBGM framework is not designed for this context**.\n- Applying differential privacy to LBGM would necessitate **adding noise to the full, high-dimensional gradient vectors** during its periodic updates. This approach is known to severely degrade model performance [6], which is precisely the problem that private PEFT methods like $\\texttt{FedSVD}$ are designed to overcome.\n- As noted, this will be incorporated into the revised `Section 2` for clarity.\n---\n>**[Q4]** Other intuitive strategies for avoiding noise amplification.\n- Please refer to **[Q1-2]**.\n---\n>**[Q5]** Better motivate why FedSVD does not limit the expressiveness of decomposition.\n- We appreciate this insightful question, which **highlights the core novelty of our approach**. While it's true that matrix $A$ is not trained directly with gradients, it is not fixed. Instead, it is adaptively reparameterized in each communication round. This dynamic updating process is precisely why our method avoids limiting the expressiveness of the low-rank decomposition, unlike methods that use a static matrix $A$.\n- Our method, $\\texttt{FedSVD}$, updates $A$ on the server by performing a SVD on the aggregated update product from the previous round ($BA$). The new matrix $A$ is then re-initialized using the orthonormal $r$ right singular vectors ($V^{\\top}$) obtained from this decomposition.\n- This means that $A$ is set to an orthonormal basis that captures the principal directions of the aggregated updates from all clients.\n---\n>**[Q6-1]** Relation to priors works.\n- Please refer to **[Q3]**.\n---\n>**[Q6-2]** Guarantee with low-rank training.\n- Please refer to **[Q2-2]**.\n---\n### References\n[1] Eckart, Carl, and Gale Young. \"The approximation of one matrix by another of lower rank.\" Psychometrika. 1936.\n\n[2] Halko, Nathan, Per-Gunnar Martinsson, and Joel A. Tropp. \"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.\" SIAM. 2011.\n\n[3] Kawaguchi, Kenji. \"Deep learning without poor local minima.\" NeurIPS. 2016.\n\n[4] Achour, El Mehdi, François Malgouyres, and Sébastien Gerchinovitz. \"The loss landscape of deep linear neural networks: a second-order analysis.\" JMLR. 2024.\n\n[5] Azam, Sheikh Shams, et al. \"Recycling model updates in federated learning: Are gradient subspaces low-rank?.\" ICLR. 2021.\n\n[6] Hölzl, Florian A., Daniel Rueckert, and Georgios Kaissis. \"Equivariant differentially private deep learning: Why DP-SGD needs sparser models.\" ACM. 2023."}, {"review_id": "nf5PFYPS3K", "reviewer": "Reviewer_w3wU", "summary": "This paper proposes a new algorithm, called FedSVD, which allows efficient LoRa fine-tuning in the federated setting. More precisely, a difficulty in the federated setting is that local updates cannot be easily summed up while preserving the low-rank structure. One way to mitigate this, for the update rule \\$W\\_0 + \\Delta W = W\\_0 + BA\\$, is to keep \\$A\\$ fixed over time, as was previously done by FFA-LoRA. FedSVD improves upon this existing method by proposing to update \\$A\\$ on the server side as well. After presenting the algorithm, the authors theoretically analyze the algorithm, deriving privacy properties and Hessian properties, and demonstrate empirically that FedSVD improves upon FFA-LoRA.", "questions": "- Do you think FedSVD is optimal for federated LoRA fine-tuning in terms of the utility of the final result? Is it possible to consider other update rules that would lead to better results?\n- The numerical experiments only test a very small number of clients (maximum 12). Can the method scale? Did you have a specific reason to keep it very small? In particular, would the algorithm perform well if only a small portion of the clients is updated at a given round (\\$K' \\ll K\\$)?\n- I am not sure I understand the heterogeneity you studied (l.277). Can you explain your method? Could you test more diverse heterogeneity settings (covariate shift, concept drift, unbalanced clients), or explain how your method would behave?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "+ The paper is very clearly written, easy to follow, with clear explanations of the setup, the theoretical results, and the experiments.\n+ The topic is very timely, as adapting federated learning to foundation models and using techniques such as low-rank updates could interest many people in the community.\n+ The proposed method is simple and could be implemented in practice.\n\n----\n\n- The optimality of the approach is not clear; see questions.\n- The experiments and the privacy part could be more developed; see questions.\n- The paper might feel a bit incremental with respect to FFA-LoRA. However, I don't believe this to be a major issue, as the new algorithm still needed to be properly written up and has some theoretical results, but people more familiar with the field might disagree.", "was_revised": true, "final_justification": "The author did a very impressive rebuttal with additional experiments that addressed all the questions I could think of when writing my review. To the extent of my knowledge in the specific topic, it seems that the proposed method indeed provides significant improvement with respect to existing methods and its presentation is very clear.", "rebuttal": "We sincerely thank you for your time, effort, and the clarity of your feedback. We are grateful that other reviewers appreciated the **simple and effective design** (z79n, yjb1, toPF), the robust and consistent experimental results (z79n, toPF), the **clear presentation** (z79n, yjb1), the **theoretical contribution** (toPF), and the **practical relevance** (toPF, yjb1). Below, we address your concerns and would be happy to discuss further during the discussion period. \n\n>**[Q1]** The paper might feel a bit incremental with respect to FFA-LoRA. However, I don't believe this to be a major issue, as the new algorithm still needed to be properly written up and has some theoretical results, but people more familiar with the field might disagree.\n- We are very grateful to the reviewer for their positive evaluation and for the thoughtful comment regarding the paper's relationship with FFA-LoRA. This is an excellent point, and we appreciate the opportunity to clarify the key distinctions and contributions of our work.\n- We agree that $\\texttt{FedSVD}$  is motivated by the same challenges identified by prior work like FFA-LoRA. Our hope with $\\texttt{FedSVD}$  was to introduce a new mechanism that addresses these challenges in a more fundamental and adaptive way.\n- The key distinction of $\\texttt{FedSVD}$  is its shift from FFA-LoRA's **static, random projection** to a **dynamic and adaptive** one. Instead of using a fixed $A$ matrix, $\\texttt{FedSVD}$ periodically refactorizes the aggregated $BA$ update via SVD, allowing $A$ to evolve and **align with the principal directions of the client updates**. \n- This adaptive orthogonalization provides a **strong theoretical basis** for our method's success, leading to a better-conditioned optimization landscape (`Theorem 2.2`) and superior gradient signal preservation under DP-SGD (`Eq. 8`).\n- These advantages are most evident in our empirical results. In the challenging private setting, $\\texttt{FedSVD}$ achieves an **8.77% improvement over FFA-LoRA** (`Table 2`), demonstrating a substantial and non-incremental gain. We believe this significant improvement, backed by our theoretical analysis, highlights that our adaptive approach is a meaningful step forward.\n \n---\n\n>**[Q2]** Do you think $\\texttt{FedSVD}$  is optimal for federated LoRA fine-tuning in terms of the utility of the final result? Is it possible to consider other update rules that would lead to better results?\n- This is an excellent question. We thank you for encouraging us to think about the broader context and future directions for this work.\n- Regarding optimality, we would position $\\texttt{FedSVD}$ as a **principled and highly effective approach** rather than a universally optimal one. Its strength lies in its specific design: by using SVD to periodically reparameterize the LoRA matrices, it enforces a beneficial orthonormal structure on the A matrix without requiring it to be trained. \n- As our analysis and results show, this leads to a better-conditioned optimization landscape and superior performance, particularly under the constraints of differential privacy.\n- The question about other possible update rules points to several exciting avenues for future research. We completely agree that exploring alternatives is a promising direction.\n- One could explore combining $\\texttt{FedSVD}$ with methods like **DoRA** [1], which decouples the update's magnitude and direction. Our SVD-based approach could define the orthonormal directions ($A$ matrix), while a separate mechanism could learn the optimal scaling magnitude, potentially offering another layer of adaptability.\n- Furthermore, prompted by your question and our own limitations in `L320-329`, we've already run preliminary experiments to address the computational cost of SVD. We compare our standard $\\texttt{FedSVD}$ with a version using a **highly efficient low-rank SVD approximation** [2]. The results are very promising:\n\n    **[Table R1. Experiments with low-rank SVD.]**\n    |Method|Matched Acc|Mismatched Acc|Avg. Run-time|\n    |-|-|-|-|\n    |$\\texttt{FedSVD}$ w/ full SVD|$71.57\\pm3.18$|$73.03\\pm2.89$|$9.07\\text{s}\\pm0.07$|\n    |$\\texttt{FedSVD}$ w/ low-rank SVD|$72.76\\pm1.82$|$72.74\\pm1.64$|$0.13\\text{s}\\pm0.06$|\n- These results show we can dramatically reduce the server-side computation time by over $\\times69$ (from $9.07$s to $0.13$s) while maintaining on-par accuracy. This confirms that the benefits of $\\texttt{FedSVD}$  can be achieved with **significantly improved practical efficiency**, strengthening its viability for real-world deployment.\n\n---\n\n>**[Q3]** The numerical experiments only test a very small number of clients (maximum 12). Can the method scale? Did you have a specific reason to keep it very small? In particular, would the algorithm perform well if only a small portion of the clients is updated at a given round ($K' \\ll K$)?\n\n- We thank you for this very important question, which is a critical consideration for any federated learning algorithm. \n- The choice to test with up to 12 clients was primarily driven by the significant computational resources required to simulate federated fine-tuning of language models like RoBERTa-large, for which we used 3 NVIDIA RTX A6000 GPUs. However, we also specifically designed our experiments to be **more challenging than those in the baseline work** we build upon (e.g., FFA-LoRA, which used $K=3$ clients).\n- Regarding performance when only a small fraction of clients participate per round ($K'\\ll K$). We designed the experiment in `Fig. 4b` of our paper to investigate this exact scenario. In that analysis: We kept the number of participating clients fixed at K′=3. We then increased the total number of clients $K$ from 6, to 9, and then to 12, which directly simulates a decreasing participation ratio. \n- The results in `Fig. 4b` show that while all methods experience some performance degradation as the client pool grows, $\\texttt{FedSVD}$ 's performance degrades much less  than both FedAvg and FFA-LORA. This suggests that our adaptive orthogonalization method is indeed more robust and better suited for scalable deployments where only a small subset of clients is active in any given round.\n- Therefore, while a massive-scale experiment would be the ultimate validation, the combination of $\\texttt{FedSVD}$ 's observed robustness in our tests and its inherent design for efficient, principled aggregation gives us strong reason to believe it would maintain its advantages in more drastic, large-scale settings.\n---\n\n>**[Q4]** I am not sure I understand the heterogeneity you studied (l.277). Can you explain your method? \n\n- Thank you for this question, which gives us the opportunity to clarify our methodology and highlight the robustness of $\\texttt{FedSVD}$ in more diverse settings.\n\n- To emulate non-i.i.d. data conditions with total $K$ clients, client data proportions are sampled using a Dirichlet distribution. Specifically, for each class, we sample a size‑$K$ simplex vector from a Dirichlet distribution with parameter $\\alpha$ (i.e., $\\text{Dir}(\\alpha, \\ldots, \\alpha)$), which determines the proportion of that class assigned to each client. This method induces a **label distribution skew**, where each client receives a different mixture of data from the various classes.\n\n- Our results in `Fig. 4a` show that $\\texttt{FedSVD}$  consistently outperforms baselines across various levels of this label skew, demonstrating its robustness to this type of heterogeneity. \n\n- We will include the clarification in the revised version of the paper.\n\n> **[Q5]** Could you test more diverse heterogeneity settings (covariate shift, concept drift, unbalanced clients), or explain how your method would behave?\n\n- Thank you for pointing this out. **Covariate shift occurs when clients have different distributions** of input data $p(x)$. Concept drift is a more challenging scenario where the **underlying relationship between inputs and outputs** $p(y∣x)$ differs between clients. \n- For both heterogeneity settings, we believe the adaptive nature of $\\texttt{FedSVD}$ is crucial. By periodically re-computing the $A$ matrix, $\\texttt{FedSVD}$ **can track changes and find a compromise representation** that better accommodates the different concepts present in the network. While severe either covariate or concept drift is difficult for any global model, $\\texttt{FedSVD}$ 's ability to evolve its low-rank subspace makes it inherently more capable of finding a robust, generalized solution than a static method like FFA-LoRA. \n- However, we respectfully consider these two heterogeneity settings to be beyond the scope of this paper. \n- Furthermore, our paper already covers **imbalanced clients (some clients have much more data)** through the aggregation scheme. As described in `Alg. 1` and `Eq. 3`, we perform a weighted average of the client $B$ matrices, where each client's update is weighted by the size of its local dataset ($n_k$). This ensures that clients with more data have a proportionally larger influence on the global model, which is a standard and effective technique for handling data imbalance. \n- Empirically, by sampling client data proportions from a Dirichlet distribution with parameter $\\alpha$ (`L218–219`), **we simulate varying degrees of client imbalance**.\n- `Fig. 4a` shows that all methods fail to effectively learn in the most severe case ($\\alpha=0.2$), while $\\texttt{FedSVD}$ demonstrates its robustness to unbalanced clients.\n\n---\n\n## References\n[1] Liu, Shih-Yang, et al. \"DoRA: Weight-decomposed low-rank adaptation.\" ICML. 2024.\n\n[2] Halko, Nathan, Per-Gunnar Martinsson, and Joel A. Tropp. \"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.\" SIAM review 53.2 (2011): 217-288."}]

Author rebuttal:

Meta-review: This paper studies the private federated learning with LoRA. More specifically, the authors propose to reparameterize the LoRA update through SVD, which can effectively reduce the amount of the additive noise.

All the reviewers agree the proposed method is novel, and their concerns are successfully addressed during the discussion.

I encourage the authors to include the promised change and the new experiments (e.g., complex task, approximation of SVD, small privacy budgets, role of orthonormal) in the revision.
Decision: Accept (poster)