Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Authors: Raghav Singhal
Kaustubh Ponkshe
Rohit Vartak
Lav R. Varshney
Praneeth Vepakomma
Venue: Accepted by TMLR ()

Abstract: Low-Rank Adaptation (LoRA) has become ubiquitous for efficiently fine-tuning foundation models. However, federated fine-tuning using LoRA is challenging due to suboptimal updates arising from traditional federated averaging of individual adapters. Existing solutions either incur prohibitively high communication cost that scales linearly with the number of clients or suffer from performance degradation due to limited expressivity. We introduce Fed-SB, a novel approach for federated fine-tuning of LLMs using LoRA-SB, a recently proposed low-rank adaptation method. LoRA-SB optimally aligns the optimization trajectory with the ideal low-rank full fine-tuning projection by learning a small square matrix ($R$) between adapters $B$ and $A$, keeping other components fixed. Direct averaging of $R$ guarantees exact updates, substantially reducing communication cost, which remains independent of the number of clients, and enables scalability. Fed-SB achieves state-of-the-art performance across commonsense reasoning, arithmetic reasoning, and language inference tasks while reducing communication costs by up to 230x. In private settings, Fed-SB further improves performance by (1) reducing trainable parameters, thereby lowering the noise required for differential privacy and (2) avoiding noise amplification introduced by other methods. Overall, Fed-SB offers a state-of-the-art, efficient, and scalable solution for both private and non-private federated fine-tuning. Our code is available publicly at: https://github.com/CERT-Lab/fed-sb.

Published in Transactions on Machine Learning Research (02/2026)
Fed-SB: A Silver Bullet for Extreme Communication Effi- ciency and Performance in (Private) Federated LoRA Fine- Tuning
Raghav Singhal*!, Kaustubh Ponkshe") Rohit Vartak?, Lav R. Varshney', and Praneeth Vepakomma
'Mohamed bin Zayed University of Artificial Intelligence, UAE
Duke University, USA
*University of Illinois Urbana- Chaampaii,,,,,,n, USA
*Massachusetts Institute of Technology, USA
Reviewed on /
Abstract
Low-Rank Adaptation (LoRA) has become ubiquitous for efficiently fine- finetuning foundation models. However, federated fine-tuning using LoRA challenging due to suboptimal updates arising from traditional federated averaging of individual adapters. Existing solutions either incur prohibitively high communication cost that scales linearly with the number of clients or suffer from performance degradation due to limited expressivity, We introduce Federated Silver Bullet (Fed-SB), novel approach for federated fine-tuning of LLMs using LoRA-SB, a recently proposed low-rank adaptation method. LoRA-SB optimally aligms the optimization trajectory with the idcal low-rank full finc-tuning projection by learning a small square matrix (R) between adapters and A, keeping other components fixed. Direct averaging of guarantees exact updates, substantially reducing communication cost, which remains independent of the number of clients, and enables scalability. Fed-SB achieves state-of- the-art performance across commonsense reasoning, arithmetic reasoning, and language inference tasks while reducing communication costs by up to 230x. In private settings, Fed-SB further improves performance by (1) reducing trainable parameters, thereby lowering the noise required for differential privacy and (2) avoiding noise amplification introduced by other methods. Overall, Fed-SB offers a the efficient, and scalable solution for both private and non-private federated fine-tuning. Our code publicly available at: https:/ /tt///gittt www.ithu..oooooeeeeeooooooo com/CERT-Lab/fede
1 Introduction
Large language models (LLMs) have demonstrated remarkable generalization across a wide range of tasks (2; 49; 46; 40). Fine-tuning (FT) remains the most effective approach for aligning LLMs to specific data distributions and reinforcing desired properties. However, as model sizes scale, full FT becomes increasingly prohibitive due to its substantial computational cost. address this, parameter-efficiet fine-tuning (PEFT) techniques, such as low-rank adaptation (LoRA, (21)), have emerged as viable alternatives, offering a favorable trade-off between computational efficiency and performance. Variants of LoRA, including QLoRA (14), DoRA (32), AdaLoRA (60), and LoRA-SB (39), further refine this paradigm by optimizing memory efficiency, training dynamics, and generalization.
Federated learning (FI. is a popular method for training models in settings where data is siloed across multiple entities (26; 24; 7). Federated FT extends this paradigm by enabling large models, pre-trained on
Equal Contributions. Author ordering determined randomly.

---

Published in Transactions on Machine Learning Research (02/2026)
public data, to be efficiently adapted to private, distributed datasets without requiring clients to share their local data. Existing methods predominantly rely on LoRA-basee techniques to learn client-specific adaptations (58). However, optimizing federated aggregation often involves tradeoffs between model performance (44) and communication efficiency (52; 43), necessitating careful design choices to balance these competing objectives.
Arithmetic
Commonsense
Mistral-7B (GSMBK)
Gemma-2 9B (MATH)
Liama 3.2-3B
Ped 200
37.5
Ted -201
Fee--Sily 2001
/B
Fedt
FedEx LoFAy
37,0
SB
FedExtoRM
76
220
Fl FLoRAy
Fod SBy
FlufAAy kn
36.5
Feu-5B, LIM
FedITy ad
Fedi
54
3o 201
36.0
72
53
35.5
FFA LoRAir
FFA
10
10a
10a
10
10a
10a
70
10a
10a
Private: SNLI (BERT)
red 50,-
hall-tty
73
Fed
Performance
74
71
Fed-SByr
Fad-SBy
FFA LoRAy
70
69
Comm. Params
68
FedEx=LoRA FLORA
70
TodEx=LoRAu--o.
FLORAy-
10a
10a
10a
Figure 1: Performance vs. communicated parameter cost (log scale) for Fed-SB and other federated fine-tuning methods in both non-private and privacy-preserving federated settings. Fed-SB achieves state-of-the-at accuracy while siguificantly reducing communication cost. Communicated parameters are in thousands for BERT and millions for other models.
LoRA-SB (39), optimally simulates full fine-tuning in low-rank spaces by learning an matrix between the low-rank adapters A and B while keeping other components fixed. This design reduces trainable parameters and enables better updates through its initialization strategy. Morcover, LoRA-SB demonstrates that this optimal approximation not achievable with standard LoRA-based methods. LoRA-SB learns higher-rank updates with 2-4x greater rank than LoRA while requiring 45-90x fewer parameters. We propose Fed-SB, a federated variant of LoRA-SB, providing an ideal framework for (private) federated FT. Fed-SB overcomes limitations in LoRA-based federated FT while being significantly more computation- and Communication-effiiien Notably, it enables exact and optimal aggregation by simply averaging the learnable matrix R.
Differential privacy (DP) is a well-established framework for ensuring strong privacy guarantees (17; 18), which is particularly crucial in federated settings. DP-SGD is a widely used privacy-preserving optimization method (1), but its challenges are exacerbated in federated FT, where noise injected for privacy amplifies divergence across client models (44). Learning in DP-SGD more effective when the number of learnable parameters is reduced, as the magnitude of noise added for privacy guarantees scales with the parameter count. Fed-SB mitigates this issuc to yield improved performance, since it inherently has fewer learnable parameters and thus less noise injection. Furthermore, we show that Fed-SB avoids noise amplification introduced by other methods, further enhancing privacy-preerving learning.
Fed-SB offers an extremely efficient and scalable solution for both private and non-private federated FT, as shown in Figure consistently has superior performance while substantially reducing communication overhead than other methods. Our key contributions are summarized follows:
2

---

Published in Transactions on Machine Learning Research (02/2026)
We propose Fed-SB, federated fine-tuning method that achieves exact and optimal aggregation in low-rank adaptation without incurring prohibitive communication costs performance degradation.
Fed-SB consistently achieves state-of-the-art results while significantly reducing communication cost, by up to 230x, by requiring only an r X r matrix to be transmitted per aggregation.
We demonstrate that Fed-SB is particularly well-suited for privacy-preservnng (federated) fine-tuning, as it minimizes noise by reducing the number learnable parameters and leveraging lincarity in the aggregate update.
Extensive experiments on models across diverse benchmarks show that Fed-SB consistently outperforms existing methods while drastically reducing communication overhead in both private and non-private fedcrated settings.
Table 1: Advantages of Fed-SB over various SOTA federated fine-tuning methods (c clients). Fed-SB achieves exact aggregation and high expressivity with extremely low communication cost constant with the number of clients. In private settings, Fed-SB offers additional advantages by minimizing noise through reducing learnable parameters and leveraging linearity to avoid noise amplification.
FedIT
FLoRA
BcdEx-LoRa
FFA-LoRAFed-SI
Exact aggregation
Learnable params.
( w)r)
((m + n)r) ( u)r)
(mer O(mr)
(v3)
Communication cost
O((m m)r) (c((m n)r, mm))e(min(((m + njn, mn))
(rr))
No noise ampl.
Privacy (less params.)
x
/
Optimal expressivity
d
2 Preliminaries and Motivation
Federated Fine-Tuning Given pretrained weight matrix W e Rm*n, the objective in FT is to learn update AW for given dataset. LoRA (21) remains the preferred method, where low-rank adapter matrices A E Rexn and B Rxxr are Iearned such that AW BA. In federated learning, the dataset is distributed across e clients, and the goal is to learn AW without sharing local data with a central server. To achieve this, each client learns its own adapter matrices A; and B, The server aggregates these updates to refine W, along with globally beneficial reprosentations of A and B, ultimately producing shared aggregate model WHeE, Next, each client continues the local FT process, followed by aggregation at the end of each round. This cycle repeats over multiple rounds. We summarize some of the state-of-the-art federated FT methods below.
Fed-IT (58) updates the adapters A and B using the standard FedAvg (35) algorithm:
A agg
1
A
BagE
M
B,
(1)
FedEx-LoRA (43) follows the same aggregation but introduces an additional error correction matrix War of rank min(cr, min(cr,m,,)) n):
W OFT 1 M A,B.)
- Σ AJ) ΣB B.).
FLoRA (52) follows the same principle as and reinitializes them randomly at the end of each communication round. FFA-LoRA (44) keeps A fixed while training (and aggregating) only B matrices.
but achieves by stacking the adapter matrices,
BaKK
B
(3)
1=1
3

---

Published in Transactions on Machine Learning Research (02/2026)
Waolobol Wo + B,
1 k
A
*
Wo E
(B.,))
Wallobo!
(4)
1=1
Parameters after aggregation with LoRA + FedAvg (FedIT)
Ideal parameters following modellaverrggnng
(Approximate) Differential Privacy. DP, introduced by (17), is widely adopted mathematical framework for privacy preservation. A randomized mechanism M D - R, mapping a domain D to a range R, satisfies (e, 8)-differential privacy if, for any two adjacent imputs d, E D and any subset of outputs - R, the following holds:
Pr[M(() 8] $
(5)
b B A
k Σ 1=1
k
k
B
W A 1
1+1
t Wo
k B A
B
X
Σ
A i=1
(6)
h
i=1
Residual
DP-SGD. DP-SGD (1) is a privacy-preservngg variant of stochastic gradient descent (SGD) designed to ensure DP during training. It enforces privacy by clipping per-sample gradients to a fixed norm C to limit their sensitivity and then adding isotropic Gaussian noise V (0, o²C²T), where controls the noise magnitude. The cumulative privacy loss over iterations quantified using the moments accountant (51) and Rényi DP (38), which offer tight bound on the final privacy parameter e.
Exact Aggregation in Fed. LoRA: Tradcoff b/w Performance and Communication Costs.
Standard federated averaging of individual LoRA adapters (FedIT (58)) introduces inezaclness in aggregation, as the ideal update should be the average of client updates.
1 B Wo + B
A
Wo Σ (BA.).
(7)
Vanilla aggregation LoRA (FedIT)
Ideal aggregation
The inexactness arises because the ideal averaged updates, given by Σi=1 B,A; often exceed rank r, violating the low-rank constraint imposed by LoRA. To address this, FedEx-LoRA and FLoRA introduce War as a higher-rank correction term within the pre-trained weight matrix Wo, which is inherently high-rank. This correction ensures exact aggregation, leading to consistently improved performance over FedIT.
This, however, comes at the cost of increased communication. Since the error matrix is high rank, it substantially increases the amount of data transmitted per round. The communication cost is determined by the number of parameters sent during aggregation, which, for an m x n matrix, is proportional to its rank. As a result, in FedEx-LLRR and similar methods that enforce exact aggregation, communication cost scales lincarly with the number clients relative to Fed-IT. This bocomes particularly concerning when the number of clients grows large, potentially requiring the transmission of the entire model's weights.
FFA-LORA addresses inexact aggregation by keeping only B trainable while fixing A uniformly across clients. However, this comes at the cost of reduced expressivity and limits the benefits of jointly optimizing A and B. As a result, performance degrades, as demonstrated previously (43). This stems from two factors: suboptimal individual updates and the need for higher-rank adaptations. Freezing A leads to suboptimal updates, even in centralized training, where FFA-LoRA underperforms compared LoRA. Additionally, recent work (34) shows that models trained using FFA-LoRA progressively deviate from the optimal hypothesis. Empirical evidence shows that the advantages of exactness are outweighed by the degradation caused by these factors.
4

---

Published in Transactions on Machine Learning Research (02/2020)
Server
R
:-1
Ra
R
Ra
R
R.
R
Q
Client
Client
Client
Wr
AL
Wo
@
Figure 2: Fed-SB: Our method achieves optimal exact aggregation by averaging only the r x matrices R.
significantly reducing communication costs.
Private Fine-Tuning. Pre-training on public data followed by FT on user-specific private data" is a common approach for adapting models under privacy constraints (54; 45). This two-stage process enhances performance in private learning while preserving user data privacy. FL naturally improves privacy by keeping data decentralized. However, even without direct data sharing, client-specifie model updates can still leak sensitive information (50). Thus, developing strong privacy guarantees while maintaining performance.
FT methods for FL essential ensure
Training model with DP-SGD introduces noise inlo the gradient, and consequently, into the model update itself. In the case of LoRA, this deviation from the ideal update is more pronounced than in full FT due to second-order noise terms. To illustrate this, let A and B represent the adapter updates learned without privacy. Under DP-SGD, these updates are perturbed by noise terms E. and εB: respectively. The difference between the ideal update AW and the noisy update A AWn pp is:
AW Dr AW (B EB) + +A) BA EBA + BEA EB
(8)
The first-order noise term, & AE is expected and occurs even in full FT with DP-SGD. However, the second-order noise term, & A: causes noise amplification, leading to further performance degradation in LoRA-based methods (44). This issue is exacerbated in FL, as individual client updates deviate even further from the ideal global update. FFA-LoRA avoids this problem by freezing A, preventing the introduction of additional noise terms.
The bilinear parameterization in LoRA introduces two key challenges: inexact aggregation and noise amplification. FedEx-LoRA/FLoR. addresses the inexactness issue by enabling exact aggregation, but at the cost of communication overhead that scales prohibitively with the number of clients. FFA-LoRA mitigates inexact aggregation and excessive communication but sacrifices performance, as it operates in a low-rank space and has reduced expressivity. An ideal method would efficiently learn higher-rank updates while inherently enabling exact aggregation without increasing communication costs. However, any LoRA-based formulation that attempts resolve these challenges must incvitably trade off expressivity, ultimately compromising performance. We prove that LoRA-SB provides an optimal reparameterization of the updates, effectively overcoming all limitations of LoRA in both non-private and Srivacy-preeevvinn federated settings.
5

---

Published in Transactions on Machine Learning Research (02/2026)
3 Method
LoRA-SB for Fine-Tuning LoRA-SB (39) optimally approximates full FT gradients in low-rank spaces and demonstrates that its entire optimization trajectory aligns with the ideal low-rank projection of the full FT path. To achieve this, LoRA-SB fixes A and B while introducing a new trainable adapter R of size r x r. Since has rank r, it updates the pre-trained weight while maintaining rank r, making it highly parameter efficient. As a result, LoRA-SB consistently outperforms LoRA (and variants) across benchmarks while using 45 90x fewer trainable parameters.
Fed-SB: A Silver bullet for (Private) Federated Fino-Tuning. We propose Fed-SB, extremely and high-performing federated adaptation of LoRA-SB. Instead of reparameterixing updates as a low-rank decomposition with learnable adapters, the server distributes frozen adapters B and A, while clients train only small matrix R (Figure 2). This enables exact aggregation, as the global update is simply the average of R across clients. Formally, given a pre-trained weight Wo and data distributed across e clients, each client learns updates of the form:
AW, = BR,
(9)
The server then aggregates the updates by computing the global R. matrix:
R
ARR
Σ i=1
A = R. R.A.
Σ R A.
(10)
We show that Fed-SB effectively resolves all challenges in (private) federated FT while achieving state-of- the-art communication efficiency and performance. Table 1 highlights the advantages of Fed-SB over other methods. Since Fed-SB fixes the adapter matrices A and B throughout training, their initialization is crucial for effective lcarning. We adopt the update-based initialization strategy from LoRA-SB, which we detail in Appendix C.
Fed-SB: Exact Aggregation. Since only R is trainable, simple averaging of R. across clients ensures exact aggregation without any updates to any other matrix. Further, the lincarity of the global update with respect to the client-specific matrices R. guarantees that exact aggregation occurs within rank r, preventing communication costs from scaling with number of clients. This is because the server only needs to aggregate and transmit R, which can be proven by computing the global update A Wags;
A Wass
B
R
A,
(11)
1=1
c
1
A W
BR,
AW,
(12)
Since the global update is simply the average of the individual updates, the aggregation is exact. The key advantage here that this exact aggregation does not incur additional communication overhead like FedEx-LoRA, nor does it compromise individual update quality like FFA-LoRA.
FT with Fed-SB has two key advantages: 1) Fed-SB avoids noise amplification, which is a common issue in LoRA-based methods. 2) Since Fed-SB inherently requires fewer learnable parameters, the amount of noise added to enforce DP guarantees is significantly lower.
Fed-SB: Privacy.
Avoids Noise Amplification DP-SGD training in Fed-SB avoids second-order noise terms, as only is trainable. This prevents the introduction of cross terms, thereby climinating noise amplification. The difference between the updates with and without privale training is given by:
AW DP AW B (R + Eg) BRA = AW Dr AW = B
(13)
Although pre-training data may public, it often contains sensitive proprietary information, raising privacy concerns. However, any privacy loss from pre-training has already occurred upon the model's release.

---

Published in Transactions on Machine Learning Research (02/2026)
Since the private update remains linear in R, Fed-SB achieves the same benefits in private settings FFA-LORA, while avoiding its limitations.
Fewer Learnable Parameters. The noise added to gradients for DP enforcement increases with the number trainable parameters (4; 1; 9), potentially distorting learning and degrading performance. Reducing trainable parameters improves DP performance, provided the model retains sufficient task-specific expressivity.
Lemma 1. Consider a model with d learnable porameters trained using DP-SGD. The privacy parameter e for 5-approzimate differentiol privacy, given T training steps and a batch size of 9, is expressed as:
Proof. See Appendix A. = O = =
(14)
Lemma establishes that reducing the number of learnable parameters enhances privacy guarantees under the same training setup. Specifically, achieving equivalent level of privacy requires injecting less noise per parameter when fewer parameters are trained. Since LoRA-SB optimally approximates full fine-tuning gradients, its updates remain effective as those in LoRA while benefiting from lower noise per update, resulting in a superior privacy-utility tradeoff. More generally, any reparameterization that reduces trainable parameters leads to a smaller accumulated privacy parameter e, thereby improving performance, provided the reduction does not compromise learning.
Fed-SB. Fed-SB has significantly less commuication costs than other federated FT methods. This is due to two key reasons: 1) L-RA-SB achieves performance comparable to better than LoRA while requiring 45-90x fewer trainable parameters. 2) Fed-SB aggregates only the X trainable matrix R, ensuring exact aggregation without additional communication overhcad. This allows Fed-SB to leverage higher-rank updates without increasing communication costs. LoRA-SB typically operates al ranks 2-4x higher than LoRA, enabling Fed-SB to capture richer updates. Retaining high-rank information is crucial in FL (34) and key factor in the superior performance of FedEx=LoRRA/LLoRA over FFA-LoRA/Pes /Fed-IIT beyond just aggregation exactness.
While main focus is the rank-homogencous setting (where all clients use the same adapter rank), also extend Fed-SB to support rank-heterogeneous clients, where each client trains with its own local rank budget. Additional details and results are provided in Table 9 (Appendix D), where we show that the rank-heterogennous setup achieves performance comparable to the homogeneous rank settings.
4 Experiments & Results
Table 2: Federated fine-tuning of Llama-3.2 3B across eight commonsense reasoning datasets. # Comm. denotes the number of parameters communicated per round (in M). Best results are in bold.
Accuracy (t)
Method
Rank # Comm. (L)
BoolQ PIQA SIQA HellaS. WinoG. ARC-e ARC-c OBQA
Avg.
FedIT
32
48.63
62.99
81.50
73.13
76.83
71.51
84.89
70.65
70.62
74.02
FFA-LoRA
32
24.31
62.87
80.03
68.53
70.02
65.56
82.95
66.38
66.85
70.40
ReREx-LoRA
32
243.15
65.05
82.81
74.67
81.84
76.01
86.32
71.42
73.81
76.49
FLoRA
32
243.15
65.05
82.81
74.67
81.84
76.01
86.32
71.42
73.81
76.49
Fed-SB
120
2.83
64.86
81.66
74.87
81.67
75.22
86.03
70.56
72.25
75.89
Fed-SB
160
5.02
65.57
82.37
76.15
84.10
77.98
86.62
72.10
73.63
77.32
Fed-SB
200
7.85
66.66
83. 79
77.22
85.42
79.56
87.46
72.53
76.02
78.58
Overview. We evaluate across throe diverse NLP benchmarks, covering models that span from BERT-base (110M) to Gemma-2 (9B), thereby encompassing both masked and autoregressive architectures. Specifically, fine-tune Mistral-7B (23), Gemma-2 9B (47), I.lama3.22 3B (16), and BERT-base (15). Our experiments

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Federated fine-tuning of Llama-3.2 3B across eight commonsense reasoning datasets, in a highly hata-hhterogenoous setting, where each client trained on a distinct dataset. # Comm. denotes the number of parameters communicated per round (in M). Best results are in bold.
Accuracy (1)
Method
Rank # Comm. (1)
BoolQ PIQA
SIQA HellaS.
WinoG. ARC-e ARC-c OBQA
Avg.
Fed[T
32
48.63
60.89
78.22
69.92
73.18
67.88
81.21
67.04
66.91
70.80
FFA-LoRA
32
24.31
60.73
76.91
65.37
65.18
61.89
79.41
62.92
63.12
67.17
FedEX-LoRA
32
243.15
62.55
79.36
71.41
78.12
72.45
82.89
67.88
70.25
73.13
FLoRA
32
243.15
62.55
79.36
71.11
78.12
72.45
82.89
67.88
70.25
73.13
Fed-SB
120
2.83
61.41
78.13
71.02
78.24
71.78
82.45
67.12
68.83
72.65
Fed-SB
160
5.02
62.34
79.05
72.39
80.52
74.67
83.18
68.64
70.12
73.98
Fed-SB
200
7.85
63.28
80.34
73.56
82.07
76.01
84.01
69.02
72.46
75.21
Table 4: Federated fine-tuning of Mistral-7B and Gemma-2 9B GSM8K and MATH. # Comm. denotes the number of parameters communicated per round (in M). Best results are in bold.
Accuracy (1)
Model
Method
Rank # Comm. (1)
GSM8K
MATH
FedIT
32
83.88
52.91
12.26
FFA-LoRA
32
41.94
53.67
12.46
FedEx-LoRA
32
2097.34
54.28
12.92
Mistral-7B
FLoRA
32
2097.34
54.28
12.92
Fed-SB
120
3.22
54.44
14.06
Fed-SB
160
5.73
54.81
13.74
Fed-SB
200
8.96
56.18
13.76
FedIT
32
108.04
74.22
36.30
FFA-LoRA
32
54.02
75.06
35.44
FedEx-LoRA
32
2701.12
74.68
36.70
Gemma-2 9B
FLoRA
32
2701.12
74.68
36.70
Fed-SB
120
1.23
74.75
36.36
Fed-SB
160
7.53
76.88
36.94
Fed-SB
200
11.76
77.03
37.56
consider both performance and communication efficiency. Detailed experimental and dataset specifications are provided in Appendix G and H, respectively. For federated data distribution, we adopt a standard protocol where client datasets are randomly sampled, following established practice in FL (44; 19; 29). We conduct experiments single NVIDIA A6000 GPU (48 GB) and report the average results from three independent. runs.
We evaluate against several SOTA federated FT approaches described previously, considering both private and non-private settings. Specifically, we compare it with FedIT, FodEx-LoRA, FLoRA, and FFA-LORA. Where applicable, we also include comparisons with standard LoRA (21).
Baselines.
4.1 Instruction Tuning
Details. We conduct experiments in the federated non-private setling across two reasoning lasks: com- monsense reasoning and arithmetic reasoning. For commonsense reasoning, we fine-tune Llama-3.2 3B on CoMMONSRNSEI700F a dataset aggregating eight commonsense reasoning corpora (22), and evaluato its effectiveness across all constituent datascts. The experiments are performed cross-silo federated learning setup involving 5 clients.
8

---

Published in Transactions on Machine Learning Research (02/2026)
We also evaluate Fed-SB under extreme data heterogeneity. Instead of randomly sampling examples for each client, assign each constituent dataset to distinct client, resulting in a highly non-IID 8-client setup. Each client trains on a distinct distribution, with varying dataset sizes.
For arithmetic reasoning, we fine-tune Mistral-7B (23) and Gemma-2 9B (47) on 20K samples from the MetaMathQA dataset (55) and assess their performance on the GSM8K (13) and MATH (20) benchmarks. In this setup, we distribute the federated training across 25 clients. In both cases, we apply LoRA modules to the key, query, value, attention output, and all fully connected weights.
Results (Tables 2, 3, 4). Our method achieves state-of-the-art performance, outperforming all previous baselines in both accuracy and communication efficiency across all models and benchmarks. Figure further illustrates this significant improvement. Additional results on the effect of varying rank are reported in Table 10 in Appendix E.
Table 5: Centralized (Cent.) private fine-tuning of BERT-base on SNLI for varying values of c. smaller indicates stricter privacy budget. # Params. denotes the number of trainable parameters (in K). Best results are in bold.
Accuracy (t)
Method
Rank # Params. (1)
6= 3
< 5
6 7.5
10
Cent. LoRA
32
1181.96
66.49
67.79
68.17
70.78
70.81
Cent. FFA-LoRA
592.13
74.40
75.02
75.02
76.14
76.60
Cent. Fed-SB
32
26.88
73.99
75.09
74.45
77.01
76.24
Cent. Fed-SB
57.59
75.98
75.70
76.58
76.77
77.96
Cent. Fed-SB
64
100.61
75.81
77.07
77.59
78.75
78.08
Table 6: Federated private fine-tuning of BERT-base on SNLI for varying values of e. A smaller indicates a stricter privacy budget. # Comm. donotes the number of parameters communicated per round (in K). Best results are bold.
Accuracy (t)
Method
Rank # Comm. (1)
3
7.5
10
FedIT
32
1181.96
49.57
51.29
48.53
55.63
60.96
FFA-LoRA
32
592.13
70.11
71.49
72.69
73.27
74.02
FodEx-LoRA
3541.26
67.38
69.68
72.92
71.89
74.33
FLoRA
32
3541.26
67.38
69.68
72.92
71.89
74.33
Fed-SB
32
26.88
70.33
72.68
73.57
73.62
73.85
Fed-SB
48
57.59
73.7
74.74
73.66
74.75
75.02
Fed-SB
100.61
73.83
74.88
76.27
75.75
75.86
Commonsense Reasoning (Table 2). Fed-SB (r 200) achieves average improvement of 4.56% over FedIT while requiring 6x lower communication cost. Additionally, Fed-SB (r 200) surpasses the previous SOTA performance methods FedEx Miriix-LRR///LLon impressive 31x. Notably, while the communication cost of FedEx-LoRA/APe FLLR scales linearly with the number of clients, our method maintains a constant, dieent-nndeendder communication cost. These results are obtained with just clients, implying that the full extent of our method's 's communication efficiency is not fully depicted here. As the number of clients increases, the relative advantage of Fed-SB over existing methods grows even further.
by 2.09%, while reducing communication cost by an
Highly Data- Hetor-ontoossnouu Setting (Table 3). Fed-SB significantly outperforms all other methods even in this highly non-lID setting. Specifically, Fed-SBB (r = 200) surpasses the previous state-of-the-art methods, FedEx-LoRA and FLoRA, by 2.08% in accuracy while achieving a remarkable 31 x reduction in communication cost.
Arithmetic Reasoning (Table 4). For Mistral-7B, Fed-SB (r = 200) outperforms FedEx-LoRA FFLOAA GSMSK by 1.90%, while achieving impressive 234x reduction in communication cost. Additionally,

---

Published in Transactions on Machine Learning Research (02/2026)
Fed-SB (r = 200) surpasses FFA-LoRA GSM8K by 2.51%, with approximately 5x lower communication cost. For Gemma-2 9B, Fed-SB (r 200) outperforms FedEx-ToRA/FLooR on MATH by 0.86%, while reducing communication cost by 230x.
4.2 (Federated) Private Fine- Fine-Tuning
Details. We fine-tune BERT-base (15) on SNLI (8), standard benchmark for natural language inference. Following L2RA we apply LoRA modules only the self-attention layers. Our evaluation considers DP settings: centralized private setup and federated private setup. To enforce DP guarantees during training, use the Opacus library (53) with the DP-SGD optimizer (1). In the federated setting, training is conducted in a cross-silo setup with 3 clients. We conduct experiments across range of privacy budgets, varying from 1 to 10.
Results (Tables 5, 6). Fed-SB consistently outperforms all prior baselines both accuracy and communica- tion/parameter efficiency across all privacy budgets in both settings. Figures 5, 6, and further illustrate this significant improvement. Further experiments analyzing the impact of rank variation are given in Table 11 (Appendix E).
Centralized Private (Table 5). Fed-SB showcases significant improvement over other methods while using only a fraction of the parameters, across all values. For instance, at 3, Fed-SB (r = 64) surpasses ralizel LoRA and centralized PFA-LoRA by 9.28% and 2.05%, respectively, while using s 12x and 6x fewer parameters.
Federated Private (Table 6). Fod-SB consistently outperforms all previous methods across all values of 6; while significantly reducing communication costs. For instance, al 1, Fed-SB (r =r= 64) outperforms FedIT, FedEx-LoRA/FLLn LoRAAFLOA,, and FFA-LoRA by 24.26%, 6,48%, and 2.72%, respectively, while reducing communication cost by approximately 12x, 35x, and 6x. FedIT performs significantly worse in the federated privale setting compared to the federated non-private setting. We hypothesize that this is due to increased deviation in updates under DP constraints and added noise, leading to greater divergence from the ideal.
4.3 Memory and Training Time
Memory, Fed-SB needs lower per-client training memory relative all other baselines by substantially reducing the number of trainable parameters. Notably, this advantage holds even when Fed-SB is trained with higher rank (r 200), where it still requires less memory than competing methods at a lower rank (r 32). We note that the peak memory usage of Fed-SB never exceeds that of any other federated LoRA-based baseline. We benchmark the peak per-client training memory for all models and configurations used in our study in Table
Table 7: Peak per-client training memory (in GB) for different methods across the various models used in this work. Fed-SB consistently exhibits lower memory usage across all model configurations.
Peak Memory (GB)
Method
Rank
Mistral-7B
Gemma-2 9B
Llama-3.2 3B
FedIT
15.92
19.99
7.71
FFA-LoRA
32
15.51
19.44
7.46
FodEx-LORA
15.92
19.99
7.71
FLoRA
32
15.92
19.99
7.71
Fed-SB
200
15.18
19.03
7.30
Training Time. Fed-SB introduces a negligible training time overhead compared to other methods, primarily due to its lightweight initialization process. To quantify this, we measure the additional training time introduced by Fed-SB relative the average per-epoch training time per client in bascline methods. These measurements are conducted across the various experimental settings described in our paper. As shown in Table 8, the overhead remains consistently minimal, approximately 2%, across multiple model configurations.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Training time overhead introduced by Fed-SB (r= = 200) relative to the average per-epoch training time per client in baseline methods. The overhead is minimal (= 2%) across different model configurations.
Model
Fed-SB Overhead (mmsss)
Avg.
Epoch Time Client (mm:ss)
Mistral-7B
00:13
09:22
Gemma-2 9B
00:16
12:43
Llama-3.2 3B
01:43
62:54
5
Conclusion
Existing LoRA-based federated FT methods either suffer from suboptimal updates or incur prohibitively high communication costs. We introduce Fed-SB, a federated adaptation LoRA-SB that ensures exact aggregation while maintaining high communication efficiency. By training only a small x r matrix and leveraging direct averaging, Fed-SB eliminates high-rank updale costs and achieves communication efficiency independent of the number of clients. Fed-SB is particularly well-suited for private FT, as its linearity prevents noise amplification, and its reduced parameter count minimizes noise required for enforcing DP guarantees. It consistently achieves new stale-of-the-arl across all models and tasks while reducing communication costs by up to 230x. These advantages establish Fed-SB as an efficient and scalable solution for (private) federated FT.
6 Limitations
While our approach is easily adaptable to other architectures, such as VLMs and VITs, we have not evaluated on such architectures, or on vision/ masion/mulimmdall settings. Given the strong performance of our method in non-IID settings, also holds promise for adapter merging, where adapters are independently trained on distinct tasks, scenario have not yet evaluated.
7 Acknowledgements
This work was supported by funding from Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) and an ADIA Lab fellowship.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 808-318, 2016.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint arXiv:2303.08774, 2023.
[3] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee- Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. Xiv preprint arXiv:230s.0652., 2023.
[4] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464-473. IEEE, 2014.
5 Klaudia Balazy, Mohammadreza Banaci, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters, 2024.
11

---

Published in Transactions on Machine Learning Research (02/2026)
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Procerdings of the AAAT conference on artificial intelligence, volume 34, pages 7432 7439, 2020.
[7] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design, 2019.
[8] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.0553 05556, 2015.
[9] Mark Bun, Jonathan Ullman, and Salil Vadhan. Fingerprinting codes and the price of approximate differential privacy. In Proceedings of the forty-siath annxual ACM symposium on Theory of computingg pages 1 10, 2014.
[10] Yac Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Hotcrogeneous lora for federated fine-tuning of on-device foundation models, 2024.
[11] Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arxiv: 1905.104400104 2019.
[12] Peter Clark, Isaac Cowhey Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint orviv:1803. 1553.05577, 2018.
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworck, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized 1mss. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
[15] Jacob Devlin. Bert: Pro-training of deep bidircctional transformers for language understanding. arXiv preprinl 1xXx:1810.04800 2018.
[66 Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint orXiv:2407.2178. 2024.
[17] Cynthia Dwork. Differential privacy. In International colloguium on automala, languages, and progrum- ming, pages 1-12. Springer, 2006.
[88] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and TrendsM in Theoretical Computer Science, 9(3 4):211-407, 1171-007,4110 2014.
[19] Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A rescarch library and benchmark for federated machine learning. ar preprint arXiv:2007.13518 2020.
[2] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXte preprint arXiv:2106. 09685, 2021.
12

---

Published in Transactions on Machine Learning Research (02/2026)
[22] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-effiiiet fine-tuning of large language models, 2023.
[23] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnicr, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[24] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trendso in machine learning, 14(1 2):1 210, 2021.
[25] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023.
[26]
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency, 2017.
[27] Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-basod random matrix adaptation, 2024.
[28] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-IImt A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 5260-5271, 2024.
[29] Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning at scale. In International conference machine learning, pages 11814-11888 11827. PMT.R, 2022.
[30] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data, orXiv preprint arXiv:1907.02118, 2019.
[31] Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou. Federated matrix factorization with privacy guarantec. Proc. VLDB Endow., 15(4)3900-913, December 2021.
[32] Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024.
[33] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[34] Navyansh Mahla, Kshitij Sharad Jadhav, and Ganesh Ramakrishnan. Exploring gradient subspaces: Addressing and overcoming lora's limitations in federated finc-tuning of large language models, 2025.
[35] Brendan McMahan, Eider Moore, Daniel Ramage, Scth Hampson, and Blaise Aguera y Arcas Communication--fflieee learning of deep networks from decentralized data. In Arlificial intelligence and PMLR, 2017.
statistics, pages
[36] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
[37] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint
2018.
[38] Tla Mironov. Rényi differential privacy. In 2017 IEFF 30th computer security foundations symposium (CSF), pages 263 275. IEEE, 2017.
[39] Kanstubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexcy Tumanov, Samuel Horvath, and Pranccth Vepakomma. Initialization using update approximation silver bullet for extremely efficient low-rank fine-tuning. Xixvv preprint arXiv:24 11. 11.11441 2024.
13

---

Published in Transactions on Machine Learning Research (02/2026)
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text. transformer. Journal of machine learning research, (1(14)): 67, 2020.
[41] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9))99 106, 2021.
[2] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense
reasoning about social interactions. arXiv preprint arXiv:1904.09778, 2019.
[43] Raghav Singhal, Kaustubh Ponksho, and Pranecth Vepakomma. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models. arXiv preprinl urXiv:2410.09432 2025.
[44] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-prcserving federated learning, arxiv preprinl Xiv:24003. 12313, 2024.
[45] Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Private fine-tuning of large language models with zeroth-order optimization. arXiv preprinl arXi::2411.04 2024.
[46] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXic preprint 2iv: 11805, 2023.
[47] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408. 10X.:2208800011,, 2024.
[48] Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. Fedbert: When federated learning meets pre- pre-training. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-26, 2022.
[49] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini vxXiv2237700288 2023.
[5]] Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy proservation in federated learning: An insightful survey from the gdpr perspective. Computers & Securily, 110:102402, 2021.
[1]] Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd internalional conference on arlificial intelligence and statistics, pages 1226 1235. PMLR, 2019.
[2] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint or Xiv: 05976, 2024.
[3] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint 0tth::2009122999 2021.
[54] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500 2021.
[55] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024.
[66] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? Xiu preprint X 100500003 2019.
14

---

Published in Transactions on Machine Learning Research (02/2026)
[57] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method, 2024.
[58]
Jianyi Zhang, Saced Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6915-6915 IEEE, 2024.
[59] Jianyi Zhang, Saced Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning, 2024.
[60] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tho Zhao. Adalora: Adaptive budget allocation for parameter-eficient fine-tuning, 2023.
[61] Zhuo Zhang, Yuanhang Yang, Yong Dai, Lizhen Qu, and Zenglin Xu. When federated learning meets pre-trained language models' parameter-efficint tuning methods. arXiv preprint ar Xiv:22211 2022.
[62] Puning Zhao, Lifeng Lai, Li Shen, Qingming Li, Jiafei Wu, and Zhe Liu. A huber loss minimization approach to mean estimation under user-level differential privacy. Advances in Neural Informalion Processing Systems, 37:130018 1100106, 2024.
[63] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra, Federated learning with non-iid data. arXiv preprint orXiu: 005800 2018.
[64] Shiyuan Zuo, Xingrun Yan, Rongfei Fan, Han Hu, Hangguan Shan, and Tony QS Quek. Byzantine- resilient federated learning with adaptivity to data heterogeneity. arXiv preprint (ix::2403.1337, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
Contents
A Proof of Lemma 1
16
B Related Work
17
C Initialization in Fed-SB
18
D Extensions to Rank-HHeterogeneous Setting
18
D.1 Rane-eettrogenneous Fed-SB
18
D.2 Experiments
19
E Effect of Varying Rank on Fed-SB Performance
19
F Loss Curves
19
G Experiment Details
20
Dataset Details
21
Additional Results
22
Additional Plots
22
A Proof of Lemma
Lemma. Consider a model with d learnable parameters trained using DP-SGD, The privacy parameter for ô-approximate differential privacy, given T training steps and batch size of q, is expressed as:
(15)
Proof. The following result (1) describes the relationship between noise variance, privacy parameters, number of optimization steps, batch size, and sample size in DP-SGD.
Theorem. There exist constants and e2 such that, given the sampling probability q = L/N and the number of optimization steps T, for any < CT4T, DP-SGD is (e, difffereentl private for any 8 > 0 if the noise scale satisfies:
4V Tlo(11/))
22
(16)
Each DP-SGD step introduces noise following N (0, LL) and satisfies (a, ()22"))))) (Rényi DP) for the Gaussian mechanism. For function with fssensitivvty Ag, the Gaussian mechanism satisfics (a, RDP with:
a4
<(a)
(17)
20 noise
16

---

Published in Transactions on Machine Learning Research (02/2026)
Since DP-SGD has A₂ C and Juiie = oC, applying privacy amplification due to sampling probability q results in each step satislying (),P)-RDP, where, for small
q:a
0
(18)
Using composition over T steps, the total RDP privacy parameter becomes:
q T
Protal 0
(19)
Converting this RDP bound back to (e,8)-DP and setting a proportional to 1/ Vd, given that the E2 form of the gradient scales as vd, we obtain:
T log(1/5)
c=
(20)
0²
a
Substituting a x 1/Vd, we derive:
= = = o(vd)
(21)
B
Related Work
Parameter-Efficies Fine- Tuning (PEFT). LoRA (21) has become ubiquitous for fine-tuning LLMs (57) by modeling weight updates as product of low-rank matrices. Several variants have been proposed to improve efficiency, stability, and adaptability. QLoRA (14) enables efficient fine-tuning through quantization strategies, reducing memory usage while maintaining performance. AdaLoRA (60) dynamically allocates a layer-specific rank budget by assigning importance scores to individual weight matrices. LoRA-XS (5) further reduces trainable parameters by inserting a trainable matrix between frozen LoRA matrices. VeRA (27) enhances parameter efficiency by learning shared adapters across layers. DoRA (32) decomposes the pre-trained matrix into two parts mants--de and direction-ane and applies LoRA modules only the direction component. PiSSA (36) improves adaptation by initializing adapters using the singular value decomposition (SVD) of pre-trained weights. rsLoRA (25) introduces a rank-scaling factor to stabilize learning. LoRA-SB (39) provably approximates gradients optimally in low-rank spaces, achieving superior performance with significantly higher parameter efficiency.
Federated Fine-Tuning Federated Learning (FL) consists of centralized global model and multiple clients, each with its own local dataset and computational capacity. The global model is updated by aggregating client updates (24). FedBERT (48) focuses on federated pre-training, while other methods work on federated fine-tuning (61; 28; 3). Fed-IT (59) aggregates low-rank adapters across clients using standard federated averaging (35) before updating the global model. To address inexact aggregation, LoRE (43) introduces an error matrix to correct residual errors, ensuring more precise updates. FLoRA (52) follows the same exact aggregation principle by stacking matrices and extends this approach to heterogeneous rank settings. FFA-LoRA (44) mitigates aggregation inexactness by freezing A and updating only the trainable low-rank adapter, averaging the latter to compute the global update. In some scenarios, clients require heterogeneous LoRA ranks due varying computational budgets (63; 30). Methods like Het (10) enable rank heterogeneity through self-pruning and sparsity-aware aggregation strategics, but incur significant overhead. Beyond data heterogeneily, FL systems must also address Byzantine atlacks, where malicious clients send corrupted updates. RAGA (64) propose a geometric median- based aggregation method that provides convergence guarantoes under both Byzantine attacks and data heterogeneity for non-convex objectives, offering complementary robustness considerations to the privacy-focused approaches in Fed-SB.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Differential Privacy (DP) and FL. A common limitation of standard FL frameworks is their susceptibility to privacy attacks, as clients publicly share model updates with a central server. To address this issue, DP is incorporated into FL methods to ensure the privacy of client updates. This work follows the approximate DP framework (17; 18), which provides formal privacy guarantoes for model updates. Privacy is enforced during training using the DP-SGD optimizer (1), which applies gradient clipping and noise injection to protect individual contributions.
Prior work (62) demonstrates that Huber loss minimization can improve mean estimation under user-level DP by reducing bias from clipping operations, particularly for heavy-tailed distributions and imbalanced user contributions. This approach complements the DP-SGD framework used in Fed-SB by addressing challenges in robust aggregation under privacy constraints. Since DP is preserved under composition and post-processing (17; 31), the final global model update also retains DP guarantees. Prior methods, such as Fed-IT and FodEx-LoRA, did not explicitly incorporate DP. This study extends these approaches DP sottings and benchmarks them alongside FFA-LoRA and the proposod method.
C Initialization in Fed-SB
Fed-SB adopts the initialization strategy introduced in LoRA-SB to fix the adapter matrices B and A. Proper initialization is crucial, since B and A remain frozen during training, For instance, if B were initialized to zero (as in standard LoRA), the product BRA would remain zero throughout, preventing any learning, In contrast, initializing B and A as orthonormal matrices ensures well-scaled gradients and allows Fed-SB to nearly match the performance of full fine-tuning.
To construct B and A, we approximate the optimal update by averaging the first-step update across a small set of samples. A truncated SVD of this estimated update is then used to initialize the adapters. This requires only a small fraction of the training data (typically 0.1%), leading to negligible overhead in computation and time. Since the update is computed layerwise, memory usage during initialization never exceeds that of subsequent Fed-SB fine-tuning and remains below that of LoRA. Empirical analysis in LoRA-SB (39) shows that even 0.1% of the samples is sufficient for stable initialization.
Extensions to Hetereeetoo Setting
In real-world federated deployments, client devices often operate under diverse computational budgets and memory constraints. This naturally leads to runk-heterogeneouus settings, where different clients cannot. train adapters of the same rank. Supporting such heterogeneity is important for practical adoption: while high-resource clients can benefit from richer low-rank subspaces, low-resource clients should still be able to participate meaningfully without being excluded from collaboration.
D.1 Hatt--eetrogeeeoous Fed-SB
We extend Fed-SB to explicitly handle rank heterogeneous clients while preserving its guarantees of exact aggregation. The key idca is to align all clients in a shared basis, chosen as the top Tmax singular vectors of a reference weight matrix. Each client then selects a local rank budget r, Tmax and optimizes within its most informative subspace:
A = vil,
B, Bl: rs.
R; = R(: res ri].
During aggregation, each client's update R2 zero paro-padded (along rows and columns) to match the global dimension Tmax X Tmax- This ensures that all updates are aligned the same coordinate system and can be averaged exactly:
Ragg
(R.), AW = B.
In this formulation, low-rank clients contribute updates restricted to their subspaces, while high-rank clients provide richer information, and all updates combine scamlessly. Thus, Fed-SB can support heterogencous client capabilities without loss of information, while maintaining exactness of aggregation.
18

---

Published in Transactions on Machine Learning Research (02/2026)
D.2 Experiments
Table 9: Comparison of homogencous and heterogencous Fod-SB configurations for federated finc-tuning of Llama-3.2 3B on cight commonsense reasoning datasets,
Method
BoolQ PIQA SIQA HellaS, WinoG. ARC-e ARC-c OBQA
Avg.
Homogeneous (all ranks 120)
64.86
81.66
74.87
81.67
75.22
86.03
70.56
72.25
75.89
Heterogeneous (effective rank 120)
64.34
81.50
74.23
81.02
74.88
85.89
70.65
71.62
75.52
Homogeneous (all ranks 160)
65.57
82.37
76.15
84.10
77.98
86.62
72.10
73.63
77.32
Heterogeneous (effective rank 160)
64.83
82.05
76.43
83.92
77.53
85.96
71.90
72.98
76.95
assess the effectiveness of our federated rank-heeterogeneous approach, we extend the commonsense reasoning experiments with Llama-3.2 3B heterogeneous rank settings. For fair comparison, we match the total rank budget of the homogeneous baselines (120² and 1602) by assigning client-specific ranks of {40, 40, 120, 120, 200} and {60, 60, 180, 200, 220), respectively. As shown in Table 9, Fed-SB achieves performance comparable to its homogeneous counterparts in both cases, demonstrating strong robust.ess to rank heterogeneity.
E Effect of Varying Rank on Fed-SB Performance
To further investigate the role of the rank parameter т, conduct ablation studies of Fed-SB in both standard federated and privacy-preserving settings. In the non-private setting, we evaluate Mistral-7B and Gemma-2 9B fine-tuned on a subset of MetaMathQAA across a wide range of rank values (r = 32- 32-240), with results reported in Table 10. While selecting an optimal rank remains an open problem for all LoRA-basse methods, our experiments show that intermediate values (r = 120- 120-200) generally offer the best trade-off between performance and efficiency.
In the privacy-preservie setting, we evaluate centralized private Fed-SB using BERT-base fine-tuned on SNLI across ranks ranging from 16 to 80, with results presented in Table 11. Here, obscrve that ranks in the range of 48 80 consistently achieve the strongest performance across different privacy budgets.
Overall, owing to Fed-SB's lightweight design, we can scale to higher ranks when resources allow, yielding further performance improvements without incurring memory bottlenecks.
Table 10: Effect of varying Fed-SB rank (r) on federated fine-tuning performance of Mistral-7B and Gemma-2 9B, evaluated on GSM8K and MATH. Best results are in bold.
Mistral-7B
Gemma-2 2 9B
Rank
GSMSK (f)
MATH (t)
GSM8K (t)
MATH (t)
32
53.76
12.88
73.78
35.92
53.93
13.31
74.32
36.05
96
54.38
13.56
74.66
36.23
120
54.44
14.06
74.75
36.36
160
54.81
13.74
76.88
36.94
200
56.18
13.76
77.03
37.56
240
56.32
13.74
77.14
37.34
F
Loss Curves
We present the training loss curves for the different methods in Figure 3 on Llama-3.2 3B for commonsense reasoning. All methods exhibit stable convergence, indicating that they are trained effectively.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Effect of varying Fed-SB rank (r) on centralized private fine-tuning performance of BERT-base, evaluated on SNLI, under various privacy budgets (c). A smaller indicates stricter privacy budget. Best results are in bold.
Accuracy (t)
Rank
1
c
6
16
73.26
74.21
73.68
76.23
75.80
24
73.65
74.78
73.92
76.88
76.02
32
73.99
75.09
74.45
77.01
76.24
48
75.98
75.70
76.58
76.77
77.96
64
75.81
77.07
77.59
78.75
78.08
75.93
76.87
77.35
78.81
78
Train Loss
feddsb_rraaaeeeeeeeeee fod-sb_rankk rankk-1660
fed sdd-bbaank-120
ffallor fed-it
0.2
0.15
0.1
0.05
Steps
50
100
150
200
250
300
350
Figure 3: Training loss curves for various methods for representative single client Llama-3.2 3B for commonsense reasoning.
G
Experiment Details
We conduct experiments single NVIDIA A6000 GPU (48 GB) and report the average results from three independent runs. All non-private models are trained using the AdamW optimizer (33). To optimize memory efficiency, all base models (except BERT) are loaded in torch. bfloat16. In line with LoRA-SB (39), we initialize the adapter matrices using just 1/1000 (0.1%) of the respective training datasct size.
Instruction Tuning. Table 12 presents the key hyperparameters and configurations for Mistral-7BB, Gemma-2 9B, and Llama-3.2 3B. Our setup closely follows previous works (22; 39), ensuring consistency with established best practices. For the baseline experiments, we further set = 16, consistent with prior literature (43; 44). We additionally perform sweep over the learning rate for our experiments.
(Federated) Private Fine-Tuning, Table 13 outlines the key hyperparameters and configurations for BERT-base in both centralized private and foderated private settings. We train our models using the Opacus library (53) with the DP-SGD optimizer (1). Following standard DP practices, set the privacy parameter 8 [rainset] To ensure adherence to best practices, adopt hyperparameter choices from prior works
20

---

Published in Transactions on Machine Learning Research (02/2026)
(43; 21). For baseline experiments, we additionally set 16, aligning with previous literature (43; 44). We additionally perform sweep over the learning rate and maximum gradient norm in DP-SGD for our experiments.
Table 12: Hyperparamcter scttings for Mistral-7B, Gemma-2 9B, and Llama-3.2 3B.
Mistral-7B
Gemma-2 9B
Llama-3.2 3B
Optimizer
AdamW
AdamW
AdamW
Learning Rate
5e-4
5e-4
2e-4
LR Scheduler
Cosine
Cosine
Linear
Warmup Ratio
0.02
0.02
0.02
Batch Size
1
8
Grad Acc. Steps
32
32
24
Max. Seq. Len
512
512
256
Dropout
0
# Clients
25
25
5
Local Epochs
2
2
Rounds
1
1
Table 13: Hyperparameter settings for BERT-base in centralized private and federated private setups.
BERT-base (centralized)
BERT-base (federated)
Optimizer
DP-SGD
DP-SGD
Learning Rate
5e-4
5c
LR Scheduler
Warmup Ratio
0
0
Batch Size
32
32
Max. Phy. Batch Size
8
Max. Seq. Ten
128
128
Dropout
0.05
0.05
Max. Grad. Norm
0.1
0.1
Epochs
3
# Clients
3
Local Epochs
6
Rounds
H Dataset Details
CommonSense1 70K is a large-scale dataset that brings together eight benchmarks designed to assess various aspects of commonsense reasoning (22). Below is an overview of its constituent datasets:
PIQA (6) evaluates physical commonsense by asking models to determine the most reasonable action in a given scenario.
2. A R.C Easy (ARC-e) (12) consists of clementary-level science questions, serving as fundamental Lest of a model's reasoning abilities.
3. OBQA (37) presents knowledge-intensivg open-book multiple-choice questions that require multi-step reasoning and retrieval.
HellaSwag (56) tests contextual reasoning by asking models to predict the most plausible continuation of a passage from set of candidates.
21

---

Published in Transactions on Machine Learning Research (02/2026)
5. SIQA (42) examines social intelligence, requiring models predict human actions and their social consequences.
6. ARC Challenge (ARC-c) (12) includes difficult multiple-choice science questions that demand deeper logical inference beyond statistical co-occurrence.
7. BoolQ (11) consists of naturally occurring yes/no questions, requiring models to infer relevant information from provided contexts.
8. WinoGrande (41) assesses commonsense knowledge through binary-choice sentence completion tasks that require resolving ambiguities.
The MetaMathQQ dataset (55) constructs mathematical questions by reformulating them from different viewpoints while preserving their original knowledge content. We assess its performance using two well-ostablished benchmarks: (1) GSM8K (13), collection of grado-school-level math problems requiring step-by-step reasoning lo reach solution, and (2) MATH (20), which consists of high-difficulty, competition-style problems designed to test advanced mathematical skills.
Stanford Natural Language Inference (SNLI) is widely used benchmark for assessing textual entailment models in natural language understanding. It contains approximately 570,000 sentence pairs, each categorized into one of three classes: entailment, contradiction, neutral, requiring models to infer the relationship between a given premise and hypothesis.
Additional Results
We present the standard deviation results across multiple runs in Tables 14, 15, 16, 18, and 17.
Table 14: Standard deviation results on Llama-3.2 3B across commonsense reasoning benchmarks.
Std. Dev. (Accuracy)
Method
Rank # Comm.
BoolQ PIQA SIQA
HellaS. WinoG. ARC-e ARC-c OBQA Avg.
FedIT
48.63
0.83
0.91
1.18
0.97
1.21
0.74
0.98
FFA-LORA
32
24.31
0.95
0.79
1.24
1.05
1.13
0.86
1.17
0.93
1.01
FedEx-LoRA
32
243.15
0.88
0.77
1.02
1.01
0.94
0.73
1.08
0.80
0.90
FLoRA
32
243.15
0.76
0.88
1.07
1.02
0.99
0.69
1.04
0.90
0.92
Fed-SB
120
2.83
0.71
0.85
1.10
0.92
1.05
0.68
1.00
0.84
0.89
Fed-SB
160
5.02
0.89
0.72
1.15
0.94
1.08
0.66
0.97
1.01
0.93
Fed-SB
200
7.85
0.79
0.93
0.96
0.89
1.12
0.65
0.92
0.98
0.90
Table 15: Standard deviation results on Llama-3.2 3B across commonsense reasoning benchmarks under a
highly dala-heterogenous setting.
Std. Dev. (Accuracy) HellaS. WinoG. ARC-e ARC-c OBQA Avg.
Method
Rank # Comm.
BoolQ PIQA SIQA
FedIT
32
48.63
0.88
0.97
1.21
0.93
1.18
0.72
1.05
0.90
0.98
FFA-LORA
32
24.31
1.02
0.84
1.26
1.09
1.11
0.88
1.14
0.95
1.04
FodEx-LoRA
32
243.15
0.90
0.81
1.03
1.02
0.93
0.75
1.07
0.83
0.92
FLoRA
32
243.15
0.81
0.91
1.08
1.03
0.96
0.69
1.01
0.87
0.92
Fed-SB
120
2.83
0.74
0.89
1.12
0.91
1.04
0.70
0.98
0.86
0.91
Fed-SB
160
5.02
0.93
0.76
1.17
0.95
1.09
0.66
0.94
1.00
0.94
Fed-SB
200
7.85
0.82
0.96
0.97
0.88
1.15
0.68
0.90
0.99
0.92
J Additional Plots
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 16: Standard deviation results on Mistral-7B and Gemma-2 9B across arithmetic reasoning benchmarks.
Std. Dev. (Accuracy)
Model
Method
Rank # Comm.
GSMSK
MATH
FedIT
32
83.88
0.96
1.18
FFA-LORA
32
41.94
0.82
1.25
FedEx-LoRA
32
2097.34
1.07
1.02
Mistral-7B
FLoRA
32
2097.34
0.91
1.11
Fed-SB
120
3.22
0.77
1.05
Fed-SB
160
5.73
0.89
0.94
Fed-SB
200
0.74
1.00
FedIT
32
108.04
1.03
1.16
FFA-LoRA
32
54.02
0.89
1.30
FedEx-LoRA
32
2701.12
0.83
1.08
Gemma-2 9B
FLoRA
32
2701.12
0.99
1.17
Fed-SB
120
4.23
0.86
1.02
Fed-SB
160
7.53
0.79
1.11
Fed-SB
200
11.76
0.93
0.99
Table Standard deviation results for centralized private line-tuning on SNLI
Std. Dev. (Accuracy)
Method
Rank # Params.
6 3 <
7.5
10
Cent. LoRA
1181.96
0.78
0.71
0.69
0.63
0.61
Cent. PFA-LoRA
32
592.13
0.66
0.60
0.58
0.54
0.52
Cent. Fed-SB
32
26.88
0.64
0.59
0.61
0.53
0.55
Cent. Fed-SB
48
57.59
0.59
0.57
0.54
0.52
0.50
Cent. Fed-SB
100.61
0.57
0.52
0.50
0.48
Table 18: Standard deviation results for federated private fine-tuning on SNLI
Std. Dev. (Accuracy)
Method
Rank # Comm.
e= 3 = 5 == 7.5
= 10
FedIT
32
1181.96
0.98
0.91
0.95
0.88
0.82
FFA-LoRA
32
592.13
0.86
0.80
0.78
0.75
0.73
FoRA
32
3541.26
0.90
0.84
0.79
0.82
0.76
FLoRA
32
3541.26
0.89
0.83
0.78
0.81
0.75
Fed-SB
32
26.88
0.84
0.79
0.76
0.74
0.72
Fed-SB
57.59
0.80
0.76
0.74
0.71
0.70
Fed-SB
64
100.61
0.78
0.74
0.71
0.69
0.68
23

---

Published in Transactions on Machine Learning Research (02/2026)
red-SBy 200)
red-SB
37.5
56
37.0
Fed-SB
Accuracy (%)
Accuracy (%)
redCx-LoRAir 321
55
=
FLoRAy 32)
Fed-SBir-
FedFx-I oR
36.5
Fed-SBI: =20))
FedIT (-=1
= 32)
54
FFA-LORA -
36.0
53
FediT (r= 321
35.5
FFA-LORA
10a
102
10a
10a
102
10a
# Communication Parameters (M)
Communication Parameters (M)
(a) Mistral-7B (GSM8K)
(b) Gemma-2 9B (MATH)
red-SB
78
=ed-SBy=1en
Fed 32)
Accuracy (%)
76
Fed B 120)
FLoRA(r=33) 321
redi ((r= 32)
74
FFA-LORA,
70
10a
102
Communication Parameters (M)
(c) Llama-3.2 3B (Commonsense)
Figure 4: Performance vs. number of communccttd parameters (in log scale) for various methods in federated fine-tuning across multiple models on arithmetic and commonsense reasoning tasks.
76
78
76
74
Accuracy (%)
Accuracy (%)
74
72
72
70
Cent. LoRAy-331
FedEx-LRAAFLORAy--st
Cent. (TA-LoRAy=19)
FFA-LoRAy--a1
Fed-SBy-45t
68
Fed-SB, mi
Cent. Fed-5B,
Fed-SB,
66
1.0
3.0
5.0 (Privacy Param.)
7.5
10.0
1.0
3.0
5.0
7.5
10.0
(Privacy Param.)
(a) Centralized Private
(b) Federated Private
Figure Performance comparison various methods in centralized (Cent.) private and federated private fine-tuning (BERT-base) SNLI across varying values of e.
24

---

Published in Transactions on Machine Learning Research (02/2026)
76
Fed-SBir-Ge
Fed-SBir
Fed-SBB 48)
76
74
Fed-SB 1r=
FFA-LORA(r
r 37)
FFA-LORAIr 22)
Fed-SBly 32)
Accuracy (%)
Accuracy (%)
74
72
72
70
70
68
LoRA
LoRAi 32)
68
102
10
102
10a
# Trainable Parameters (K)
# Trainable Parameters (K)
(b) e =
78
Fed-SBi -4) 64)
Fed-SBB Ted-SB(r-Ga)
78
76
Fed-S Bod-SBira SB
Fed-SB 48)
Accuracy (%)
74
Fed-SB
FFA-LORAy 32)
Accuracy (%)
76
Fed-SB
HFA-LORAy 32]
72
74
70
72
LoRA 32)
LoRAy, 32)
68
102
103
102
10a
Trainable Parameters (K)
# Trainable Parameters (K)
(c)
7.5
78
Fed-SBir 84)
Fed-SBir
77
Accuracy (%)
76
Fed-SBB 32)
FFA-LORA
75
74
73
72
LoRAi
71
10a
10a
# Trainable Parameters (K)
(e) 10
Figure 6: Performance vs. number of trainable parameters (in log scale) for various methods in centralized private fine-tuning (BERT-base) across different privacy budgets (e).
25

---

Published in Transactions on Machine Learning Research (02/2026)
74
Fed-SBly o4
75
red red 64)
73
Fed r
Fed-SBi- 481
72
Accuracy (%)
Accuracy (%)
71
Fed-SB(r- 321
Fed-SBir-
FFA-LORA
72
FFA- FFA-LORAW/s! 32)
69
71
68
LUBEX-LORAIr=32 32)
70
Fedbx-LoRAir- FLoRA(
FLoRA(r332
102
10a
102
10a
# Communication Parameters (K)
# Communication Parameters (K)
(a)
(b) =
Fed-SBt Lod-SBir-Ga 64)
76
red red-Stir 64)
76.0
75.5
75
Accuracy (%)
75.0
Accuracy (%)
Fed-SB
74
74.5
Fed-SB 321
FFA LoRAir 32)
74.0
73
Fed-SB ex0
73.5
FedEX-LoRAy, LoRA|r= FLoRA
Ped-SB:r=
FedEx-LoRe LolEx-LoRAir-a3
72
73.0
FLoRA
FFA-LORAY
71
102
103
10a Communication Parameters (K)
102
Communication Parameters (K)
(c)
(d) 7.5
76.0
=ed
75.5
Accuracy (%6
75.0
Fed-SB 48)
74.5
FedEX-LoRAy =LoRAy=32)
FFA-LORA
74.0
Fed-S8/r=3al
10a
103
# Communication Parameters (K)
(e) 10
Figure 7: Performance vs. number of communicated parameters (in log scale) for various methods in federated private fine-tuning (BERT-base) across different privacy budgets (e).
26


Reviews:
[{"review_id": "7NbJwdQg1y", "reviewer": "Reviewer_3AEH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Exact aggregation with minimal communication: By training only a small r×r matrix R while freezing adapters  A and B, Fed-SB enables exact model aggregation via simple averaging, reducing communication costs by up to 230× compared to prior methods.\n\nEnhanced privacy preservation: The reduced parameter count lowers the noise required for differential privacy, and the linear update structure avoids noise amplification—a common issue in bilinear LoRA-based methods.\n\nStrong empirical performance: Extensive evaluations on commonsense reasoning, arithmetic reasoning, and language inference tasks show that Fed-SB outperforms existing federated fine-tuning methods while being significantly more communication-efficient.\n\nExtensions to heterogeneous settings: The method is adapted to support clients with different rank budgets, maintaining exact aggregation and robust performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The study covers multiple model families (BERT, Mistral, Gemma, Llama), diverse benchmarks, and both IID and non-IID data distributions.", "audience_yn": "Yes", "audience_explanation": "1.Clear problem framing: The paper nicely motivates the trade-offs between exact aggregation, communication cost, and expressivity in federated LoRA.\n\n2.Privacy-aware design: The integration of differential privacy is thoughtful, with analysis showing how Fed-SB naturally reduces noise injection and avoids second-order noise terms.\n\n3.Practical insights: The inclusion of memory and training time analyses strengthens the practical relevance of the work.", "requested_changes": "The authors may consider discussing the following related works to provide additional context and strengthen the privacy and robustness perspectives:\n\nFor privacy-aware mean estimation and user-level DP:\n\n[1] Zhao, Puning, Lifeng Lai, Li Shen, et al. \"A huber loss minimization approach to mean estimation under user-level differential privacy.\" Advances in Neural Information Processing Systems 37 (2024): 130018-130056.*\nThis work offers relevant insights into robust and private aggregation, which could complement the differential privacy analysis in Fed-SB.\n\nFor Byzantine-resilient federated learning under data heterogeneity:\n\n[2] Zuo, Shiyuan, Xingrun Yan, Rongfei Fan et al. \"Federated learning resilient to byzantine attacks and data heterogeneity.\" IEEE Transactions on Mobile Computing (2025).\n\nA brief discussion of  this paper would help position Fed-SB within the broader literature on robust federated learning, especially regarding adversarial client scenarios.", "broader_impact_concerns": "I have not found any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PBTdctTQtx", "reviewer": "Reviewer_PLri", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a method for achieving exact gradient updates in parameter efficient fine-tuning for LLMs. The key idea is to build upon a recent work LoRA-SB whose structure is compatible with exact gradient updates compared to prior works. While LoRA-SB considers a centralized setting the application to the federated setting is natural and appears to provide significant gains in communication efficiency over prior approaches. The authors demonstrate that the proposed method is also a natural candidate for introducing differential privacy during the training process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "While the extension of LoRA-SB to the federated setting is natural, the paper provides a comprehensive set of experiments showing impressive performance gains. In particular the non-iid setting is handled in a natural way and shows impressive gains.", "audience_yn": "Yes", "audience_explanation": "The topics on federated fine-tuning of LoRA models and differential privacy are important and timely and of interest to the TMLR audience.", "requested_changes": "Lemma 1 is the only theoretical proof in the paper and the proof is unfortunately not very clear. \n- how do you go from (20) to (21)? What is the choice of alpha used? \n- in practice if the learning parameters are reduced why should other parameters e.g. T, q etc be the same accross different schemes? \nI am not an expert in the topic of differential privacy and I do not understand the whole proof. It is not written in a very rigorous manner. Some variables do not seem to be defined. What is \\sigma_noise? \n\nWhy is the initialization used similar to what is done in LoRA-SB?  If you have different heterogenous clients, better intializations could be used to improve the performance. \n\nThere are no error bars presented in the experimental results. Are the results statistically significant? \n\nCan you provide any theoretical convergence guarantees especially in the non-iid setting to demonstrate further novelty over prior works?", "broader_impact_concerns": "none that I can consider.", "additional_comments": "The proposed work seems to be a natural extension of prior work on LoRA-SB to the federated setting. The the empirical results look interesting. I would request the authors to address the questions raised above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LCFGvqdrT8", "reviewer": "Reviewer_ktdP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a reparametrisation of LoRA that is more suitable for federated learning. The paper claims that the work outperforms existing federated variants of LoRA while incurring lower overhead.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The empirical evidence is not very robust as it stands. The paper states that the results are an average of three runs. It is not enough runs to get a stable result. Additionally, the results should be reported as mean+std and not just the average. Furthermore, for the most all results are quite close; hence, depending on the standard deviations they might be statistically insignificant, and that should be tested for and reported.\n2. The method outperforms everything else even at significantly lower capacity. This casts doubt whether all methods have been properly trained and not over/under-trained. The paper could really use some presentation of the loss landscape and convergence analysis.\n3. Similarly, I'm not sure why such extensive discussion about the rank is needed. As the authors note, in standard LoRA the trainable part **W=AB** where the inner dimension *r* give us the efficiency, and here, it's replaced with **W=ARB** where **R** is *r x r*. Why do we care about it and not just the number of trainable parameters?\n4. Given the abovementioned reduced capacity, I am not certain whether the selection of benchmarks faithfully represents the efficacy of the method. There should exist settings/benchmarks that require more trainable parameters than is afforded by a set of **R** matrices, which in turn would show how one needs to scale up **R** to meet them. This likely ties in with (1) and (2).\n5. I don't understand why differential privacy (DP) is such a central part of the presentation of this work. It is imperative that mechanisms with fewer trainable parameters work better with DP. If we can trust all the results in this work (see above points), then it follows naturally that this method would be more suitable for private learning. However, I fail to see what algorithmic **specifically to private learning** this work introduces. \n\nWithout this information, I cannot make a definitive statement about the strengths of the paper where the core contribution boils down to training a smaller intermediate matrix in LoRA reparameterisation.", "audience_yn": "Yes", "audience_explanation": "Federated learning is an active area of machine learning research. Federated learning of performance efficient adapters for large models naturally follows, given their popularity in centralised customisation of (large) language models.", "requested_changes": "1. Loss landscape and convergence analysis for the compared methods\n2. Example of a benchmark where a lower rank matrix cannot effectively learn the finetuning dataset due to insufficient capacity -- with experiments.\n3. More runs with reported mean+std, statistical significance. \n4. This paper has no algorithmic contribution to differential privacy. It boils down to \"lower rank -> less noise -> better DP trade-off\". The boilerplate can be moved to the appendix or just made into a 1-2 paragraph discussion piece, to make space for (1).\n5. Streamlined writing (see additional comments).", "broader_impact_concerns": "N/A", "additional_comments": "The paper is difficult to follow because of the sensationalised writing and visual clutter. \"Silver bullets\", \"pushing pareto frontier\", excessive bolding in every single paragraph. It reads like a press release.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)