Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Authors: Arian Raje
Baris Askin
Divyansh Jhunjhunwala
Gauri Joshi
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
LoRA
Fine-Tuning
Efficiency
Primary area: deep_learning
TL;DR: We propose a parameter-efficient fine-tuning method that improves the performance of LoRA in the presence of computational and data heterogeneity in FL.

Abstract: Large Language Models (LLMs) have yet to effectively leverage the vast amounts of edge-device data, and Federated Learning (FL) offers a promising paradigm to collaboratively fine-tune LLMs without transferring private edge data to the cloud. To operate within the computational and communication constraints of edge devices, recent literature on federated fine-tuning of LLMs proposes the use of low-rank adaptation (LoRA) and similar parameter-efficient methods. However, LoRA-based methods suffer from accuracy degradation in FL settings, primarily because of data and computational heterogeneity across clients. We propose Ravan, an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity by reparameterizing the weight updates as the sum of multiple LoRA heads, $s_i\textbf{B}_i\textbf{H}_i\textbf{A}_i$, in which only the $\textbf{H}_i$ parameters and their lightweight scaling factors $s_i$ are trained. These trainable scaling factors let the optimization focus on the most useful heads, recovering a higher-rank approximation of the full update without increasing the number of communicated parameters since clients upload $s_i\textbf{H}_i$ directly. Experiments on vision and language benchmarks show that Ravan improves test accuracy by 2–8\% over prior parameter-efficient baselines, making it a robust and scalable solution for federated fine-tuning of LLMs.

RAVAN: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning
Arian Raje" Baris Askin Divyansh Jhunjhunwala Gauri Joshi
Department of Electrical and Computer Engincering Carnegie Mellon University
Abstract
Large language models (LLMs) have not yet effectively leveraged the vast amounts of edge-device data, and federated learning (FL) offers promising paradigm collaboratively fine-tune LLMs without transferring private edge data to the cloud. To operate within the computation and communication constraints of edge devices, recent literature on federated fine-tuning of LLMs proposes the use of low-rank adaptation (LoRA) and similar parameter- effficeent methods. However, LoRA- based methods suffer from accuracy degradation in FL settings, primarily because of data and computational heterogeneity across clients. We propose RAVAN, an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity by reparamcterizing the weight updates as the sum of multiple LoRA heads B, A,A, in which only the core matrices H, and their lightweight scaling factors 5; are trained. These trainable scaling factors let the optimization focus on the most usefuu heads, recovering higher-rank approximation of the full update without increasing the number of communicated parameters since clients upload s;H, directly. Experiments vision and language benchmarks show that RAVAN improves test accuracy by 2-8% over prior parameter-efficite baselines, making it robust and scalable solution for federated fine-tuning of LLMs.
Introduction
In recent years, the amount of data available on edge devices has increased exponentially, opening the doors for applications that perform machine learning (ML) at the edge. One such paradigm is federated learning (FL), model training regime where edge devices, or "clients", collaboratively train a model without sharing their local data with a central server 277. FL training offers a way to perform large-scale MI. on a distributed network of clients by utilizing on-device data while reducing potential privacy risks. The primary challenge in FL is to design methods that are robust in the presence of both
data heterogeneity 24 20 23 I -vriiaiion in clients' local data distributions- computational differences clients' computing capacities.
heterogeneity [10 30
Recent literature has begun exploring the integration of large language models (LLMs) into FL frameworks, driven by the surge in
edge applications and the resultant need leverage on-device data for training 40 41. Unfortunately. naively training LLMs in FL settings is intractable as a result of the memory constraints of edge devices and communication constraints wireless networks. As consequence, these works have examined the impact of parameter-efficient fine-tuning (PEFT) for L.LMs in federated settings 17 These methods reduce the computational load of fine-tuning pretrained LLMs by scaling down the number of trainable parameters. A particularly
"Coresponding Author, araje0andrew. cmu cmu.edu edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

important PEFT method in FL is low-rank adaptation (LoRA) [16], where the update AW is re- parameterized as BA, the product of two low-rank matrices and A. The original pretrained model parameters are frozen throughout fine-tuning, and only the LoRA B and A parameters ever receive gradient updates, resulting in fine-tuning that vastly more parameter-efficien than full- full-parameter fine-tuning. LoRA-based methods are promising alternative to fall-parameter fine-tuning in FL since clients only have to train and communicate the LoRA parameters, simultaneously addressing computation and communication bottlenecks.
Table 1: Accuracy comparison CIFAR-100
However, LoRA-based methods are highly affected
22 non-L.LD. clients (Dirichlet a = 0.3)
by client data heterogeneity (see T) because re-
stricting updates a low-rank subspace deprives the
Method
L.D.D
Non-I.I.D.
model of the capacity needed to fit the diverse direc- tions introduced by heterogeneous data. Moreover, directly extending LoRA to FL., as done in FedIT 41, leads to an inexactness problem during aggrega- tion. Since BA is proxy for the true model update W. averaging the B and A parameters separately would be inconsistent with the true model update:
Fu
89.78
85.17
FedIT
81.75
68.15
FedEX-LoRA
77.82
66.98
FFA-LORA
78.17
59.89
1 |C(0|
1
B cEC()
Σ
1
A DEJOCEI
Σ
+
B A DEJOCE1
Σ
(I)
|CO]]
|CO1
where C(O) is the selected client set at round L. Previous works that seek to address this exact aggregation issue suffer from accuracy loss and poor scalability in practical FL settings due to data and computational heterogeneity. FFA-LoRA (34] manages exact updates by freezing the A parameter at initialization but reduces the model ex expressivity relative to vanilla LoRA. FedEx- LoRA [32] adds
|cto| Execto ( to the original pretrained weights W to get exact update every round. However, the method substantially increases the communication cost of fine-tuning since the updated model weights also have to be communicated every round. Critically, these LoRA-based methods can afford only small ranks within limited parameter budget. When the true update is high-rank, this approximation disregards most of the update's 's variance and limits accuracy. Fed-SB [3), motivated by the update approximation from LoRA-XS [5], introduces a third LoRA parameter between the standard B and A parameters and only fine-tunes this additional parameter. However, it necessitates an initial round of full- full-parameter fine- tuning to initialize B and A, which is prohibitively expensive in FL.. Furthermore, the initialization may become stale training progresses due to data heterogeneity and partial participation.
the inexact residual [c(*)) Eeeco B(O A ( [c(o) Execto) B 1
Computational heterogeneity is an additional scalability challenge in practical FL systems. Clients may vary significantly in their hardware resources and computational capabilities, making it difficult for all clients to fine-tune models at the same scale and speed. The methods described above do not allow for LoRA parameters of different sizes across clients. HetLoRA [9] and FlexLoRA [4] allow clients to train varying-rank LoRA parameters, but the methods struggle in the presence of data heterogeneity and do not ensure exact aggregation. In this vein, our goal is to design method for FL that 1) performs efficient computation and communication throughout the training procedure, 2) remains robust in the presence of heterogeneity, and 3) retains the property of exact aggregation.
To this end, we propose RAVAN P an adaptive multi-head LoRA method that sharply reduces the number of trainable parameters while maintaining accuracy in the presence of data and computational heterogeneity. We take inspiration from multi-head approaches such as HydraLoRA (35]; however, when naively ported to federated settings, those methods fail to guarantee exact aggregation and cannot raise the effective rank of the updates under a fixed parameter budget. Our design meets both requirements. RAVAN ce-parameterizes each weight update AW weighted sum of low-rank heads s,B, H,H.A. where the bases B, and A, are frozen at initialization and only the H, parameters and lightweight scaling parameters s; are trained. We choose B, and A, with mutually orthogonal column and row spaces, respectively, and thus the heads combine to achieve a higher effective rank without exceeding the original LoRA parameter budget. When clients differ in resources, the most constrained devices can fine-tune only a subset of heads and leave the rest frozen. Uploading the products s; H, preserves exact aggregation and the method incurs no extra communication cost. Extending prior efforts, RAVAN introduces an integrated framework that maintains parameter-efffcient computation
PRAVAN derives its name from the mythical 10-headed villain from the Hindu epic, Ramayana

---

Top-64 Singular Values of AW
Top-64 64 Singular Values AW
10
Stagnias Vaiue tlg (og-scalo)
Singular Villue llg--cccio)
10e
10-1
10-1
Centralizdd
Fl. LLD. Clients
FLW Nond LLD. Cliens
Contralized
Fl. LLD. Clients
w/ Non LLD. Clients
(a) CIFAR-100 100 Spectrum
(b) SVHN Spectrum
Figure 1: Singular value spectra of the weight updates AW for CIFAR- and SVHN 29 in three different training regimes. We display only the 64 largest values (hence the truncated plots). Moving from centralized learning - FL (L..D. clients) - FL (non-L.L.D clients), the median shifts up and the distribution becomes broader, meaning larger fraction of singular values remains near the higher end of the spectrum. The effective rank is, therefore, higher in the federated, non-I..D. setting.
and communication and demonstrates robustness across diverse data and computational heterogeneity in federated settings. Across all benchmarks, RAVAN outperforms related federated PEFT methods in both I.I.D. and non-L.LD, settings, demonstrating its strength in the presence of heterogeneity and client diversity.
2 Problem Setup and Motivation
LoRA is PEFT method that reparameterizes the weight updates reduce the number of trainable parameters. It contends that the full-parameter weight update Wiall can be approximated as follows:
full
BA
(2)
Weight Update
LoRA Parameters
For notational ease, we write each weight matrix W and its corresponding update AW full square matrix in Rdxd, but all derivations extend directly to the general rectangular case in which W m= The method contends that A full exists in a low-rank subspace and can therefore be represented as the product of two low-rank matrices. In this context, the low-rank matrices, referred to as the LoRA parameters, have dimensions the pretrained model weights W are frozen throughout training and only B and A receive gradient updates. Since r & d, the number of trainable parameters decreases from da to 2rd.
Rd and A E Rrxd To perform LoRA fine- mhe-tuning
Importance of Higher-Rank Update Approximation. A key limitation of LoRA is that straining the approximation of the update A Willl to the low-rank subspace spanned by BA can limit its expressive capacity. When the rank is set too low, the approximation of AW full may fail to capture the full complexity and variation present in the full-rank gradient updates. This limitation amplified when perform fine-tuning in federated setting, shows the spectra of singular values of the full-parameter weight updates, Wiall in three different training regimes (centralized learning, FL with I.I.D. clients, and FL with non-I.I.D. clients). The model is trained to a target accuracy, and the weight update is decomposed using singular value decomposition (SVD). Figure [ demonstrates that the "effective rank" of the weight updates is larger in the federated non-I.L.D. setting. Intuitively, the greater the diversity among client updates, the more the spectral mass is spread across singular vectors, thereby increasing the effective rank. This suggests that low-rank approximations of the weight updates discard more information and fail to capture many of the significant intrinsic dimensions of the true updates.
observe in Figure [ which
Improving the Effective Rank and Expressivity of LoRA. A naive way to capture the richer spectrum of weight updates is to raise the LoRA rank r, but that linearly increases the number of trainable parameters. To better approximate the higher-rank update, as proposed in LoRA-XS,

---

Imprroiing the rank of the update approximation LoRAA
Tinimable
Selective freazing for resource baterogsnssig
Frowm
LoHA
Aucucuted LoRA
Malti Hend Auquented LoRA
Client
Ti
Cliiett 3
V
H
Ay
Cliint
H
Cllimnd
H
Max Edfective Rank
Max Elfertiive Rank
11
Azy
Total trainelake persmclere remmiino N but méximuun effective rank increases from 2)
Maxe Ffferrive Rank
VAA
Figure Left: Within the same parameter count, the effective rank of the LoRA parameters increases when using an augmented third parameter and multiple heads. Right: Clients with various computational constraints can freeze certain heads reduce memory consumption.
can augment the traditional LoRA approximation with an additional parameter, H, as follows:
full
BHA
Weight Update
LoRA Parameters
where B and remain frozen and only is trained. Suppose we have trainable parameter budget of N parameters. In the case of vanilla LoRA, N 2rd and r = ((1). With the augmented version of LoRA that has parameters BHA and frozen B and A, we can instead use the much larger rank of VN = ((vd), Additionally, if used FL, this setup would avoid inexactness in the aggregation of the disparate client models since only the H parameter is averaged across clients.
Multiple Heads for Further Rank Improvements. We further improve the effective rank of the update approximation by using multiple concurrent augmented LoRA heads. Again, suppose we have a trainable parameter budget of N parameters. We use / heads, where each head E [1.....h) has structure B, B,H,A, and each B, and A, is frozen at initialization. With this reparameterization
of the weight update, each head has rank V h Using the sub-additivity of rank, we instead have:
rank (B.HA,)
N vi [I......]]
rank Σ
h B.HAA S h
N V h
VNh
(4)
Within the same trainable parameter budget, by using h heads, we can improve the rank expressivity of the augmented version of LoRA by a factor of Vh. Furthermore, using heads of the form B H,A,, where B, and A; fixed, retains the property of exact aggregation because the following is true:
Σ
B
B: ΣB.
Σ
HOO CF
CCO
CC
CECO 1=1
CECU
Note that methods like HydraLoRA and LoRAMoE [22 that use multiple vanilla LoRA heads the form B,A, (instead of the augmented B, A,H,A; form) do not confer these benefits of increased rank expressivity and exact aggregation. Suppose we have N trainable parameters; each vanilla
LoRA head would have dimensions B, E Rdx 2 and A; E 2 xd We would then have the same maximum effective rank as vanilla LoRA because of the following inequality:
N
N
rank (B.A)
Vi [1....,hh h] rank Σ B.
< hh
(6)
2dh
2dh
2d
Since the number of trainable parameters N (d)) (recall that N 2dr for standard LoRA), this
rank is N/2d @(1), which is much smaller than the rank / Nhh (vvih) achieved by multiple augmented LoRA heads.

---

Addressing Computational Heterogeneity. In realistic resource-heterogeneee federation, clients may have vastly different computational capacities. PEFT scheme that forces every device to train the same size LoRA parameters will exclude the weakest clients or throttle the strongest. In the FL. setting specifically, using multiple heads has the additional benefit of providing a way to manage computational heterogeneity. Clients with more significant resource limitations can freeze subsets of the heads and only fine-tune the remaining heads (Figure 2 Right). This further reduces the memory requirements of local fine-tuning and prevents clients from having to drop out of the FL procedure due to stricter resource constraints, This partial freezing scheme still avoids inexactness in the aggregate updates, unlike previous heterogeneous- rank works in FL.
3 Proposed Method
In this section, we present RAVAN, method that uses multiple augmented LoRA heads to perform efficient LLM fine-tuning in the pres- ence of data and computational heterogeneity. For pretrained weights W in the model M, the forward pass is replaced by the following:
Algorithm RAVAN
Require:
Clients C, Model M, Rounds T, Local Steps S, LR f, Rank r, Heads h
1: Initialization:
......,)) h] for M 3: Freeze original model parameters, B,, A,
2: INIT(B,, H A.), [1....,
4: Model Training:
W Σ S.H.H.A æ
5: for t = 1 to T' do
Reset s 1- i [1, .......h
RAVAN Forward Pass
7: Select active client subset C()
8: Broadcast {H (H)) M
to C())
The pretrained W along with each B, and A; are frozen at the start training. As consequence, only the H, parameters and the lightweight scaling factors 8; are updated
9:
for all ((+) in parallel do
10:
H <- -EELECTHEDS((...)
and communicated throughout the FL train-
11:
for T = 1 to do
12:
Update 8 H. for E H!! end for sends {52 H
ing procedure. The pseudocode of the pro-
posed method RAVAN is given in Algorithm 13:
(1)
Il and the following sections highlight key
14:
(o) to server
components of our framework. Specifically,
15:
end for
we examine the importance of initialization
16:
for i 1 to h do
improving the update approximation (Sec-
Update H! ((+1)
Σ
tion B.I.. We additionally analyze strategies for per-elient head subset selection and aggre- gation for computational heterogeneity (Sec- tion 3.2). Together, these design choices let RAVAN match the communication cost of vanilla LoRA, while delivering higher-rank, resource-aware updates that preserve exact aggregation.
17:
(c00
Sc,
rEC!!)
18: end for
19:
end for
3.1 Parameter Initialization
Fine-tuning efficiency hinges on the initialization of the LoRA parameters. The standard LoRA initialization sets B O and draws A ~ (0 o²). In RAVAN, this initialization cannot be used since each B,H,A, would remain 0 throughout training Therefore, we must draw non-zero B, and A, and set H, 0 so that fine-tuning starts from the original pretrained weights but updates the LoRA parameters throughout the training procedure. In this section, we provide methods for effective initializations for the B, and A; parameters. These initializations do not require performing full- nar-mmareeteer fine-tuning of the original LLM weights such as the initialization presented in Fed-SB. Since each B; and A; parameter are fixed, the expressive power of the sum Σn= s; A, is limited by the subspaces spanned by the column spaces of the B, and row spaces of the A,. We test the following two methods to obtain orthogonal subspaces:
B, and A, are frozen at initialization.
Set B, ~ N() and A, ~ (0) In high-dimensional space, their column and row spaces are orthogonal with high probability.
Random Normal:
Gram-Schmidt: For the B, parameters, we concatenate the rh columns of random normal initialized B....., By] E Idxrk and apply the Gram-Schmidt procedure in the column space.
5

---

This yields an orthonormal set {bx 1b= The orthonormal set can be sliced back into h blocks of width r to form the B,. For the A, parameters, we can apply the Gram- -chhiit procedure in the row space of the concatenated Ass. With this initialization, orthogonality holds deterministically.
We benchmark our initializations against constant initialization where B; B,, A =j Vi, We test an additional more flexible initialization benchmark where B; MR, A, R,N for normally distributed M E _dxr N grxd and invertible R !R"x" which are different for each head. We refer to this baseline as "shared subspace" since the initialization ensures that the column and row spaces of the B, and A; parameters are identical. On both vision and language tasks, the random normal and
initializations deliver the highest test accuracy, confirming that mutually orthogonal B, and A, increase the effective rank of the update approximation which translates directly into better downstream performance. Full numbers are reported in Section 4.3
3.2 Head Selection Strategies
RAVAN allows clients to choose subsets of the LoRA heads to fine-tune. This is particularly advanta- geous in FL where client devices often possess widely varying computational capacities. Suppose we have a participating client set in communication round t, ((O), For client E c(t), we define
H
Vi E (1..... htj. ..). At the beginning of each local training step, each client evaluates (1)
scoring function Pr. = score H Dz) Vi E [1...., h] on its local data De. In the following, || ||r the Frobenius norm of the input matrix. The Frobenius norm of a matrix is calculated the square root of the sum of the squares of all its entries. We employ the following three scoring functions:
Random Scoring: Pa. Unif(0, 1). Heads receive random scores, so clients form their fine-tuning subset by uniformly sampling heads random.
Weight-Based Scoring: P H c.i Fl. Heads whose weights have the largest magnitude assigned a higher score and deemed more influential.
Gradient- Scoring: Pe.i = V i fell for a single mini-batch with all other heads frozen.
Heads whose gradients have the largest magnitude are deemed more influential.
Client selects the top Kg heads ranked by Pe,i and forms the selection set H = {i E [I.... A|] is among the top Ke heads} where the value of Ke depends on client c's computa- tional constraints. During local fine-tunin tine-tuning client e only updates heads in H!!) Let c( denote the set of clients that fine-tuned head in communication round t. The server performs the update:
IC
O,
(+1)
Ic
(8)
I (1) 0.
Equation 8 ensures exact aggregation of the 8; and H, parameters by directly averaging their
product and reinitializing each s; vi [1, 11,... h] the start of every communication round. An alternative measure to ensure exact aggregation is to fix each 8; = 1 at the start of training. We would
then directly average the individual H, 's so that H (++1) | Σ Erec(e)
H
We compare these
two aggregation schemes in Section 4.3 and find consistent improvements with scaling factors.
Experiments
4.1 Experimental Setup
Dataset and Model Usage, For image classification, we adopt ViT-B/16 [1] (85 M parameters)
and fine-tune on two benchmarks: (i) CIFAR-100 (50,000 train / 10,000 test images, 100 classes) and (ii) SVHN (73, 73,250 train 26,032 test digits, 10 classes). For language tasks, we fine-tune Ba- 31 (224 M parameters) on (i) 20 Newsgroups (28 (11 (11,300 train / 77,322 test articles, 20 topics) and (ii) MRQA [4 (516,800 train 58,221 test examples). The MRQA corpus the union of six sources (HotpotQA, NaturalQuestions, NewsQA, SearchQA, SQUAD, TriviaQA).

---

Federated Partitioning, We create federated splits with |C|= 20 or |C|= 50 clients. For I.I.D. partitions, clients receive an equal-sized random subsample of the global training set. For non-I.I.D. partitions, we draw client-specific class proportions from Dirichlet distribution with a 0.3. For MRQA, which lacks class labels, the Dirichlet split is performed over the constituent sub-datasets.
Table 2: Performance comparison on CIFAR-100 and SVHN.
CIFAR 100 (Ace. %)
SVHN (Acc. %)
20 Clients
50 Clients
20 Clients
50 Clients
Method
Rank
LLD.
Non-LLD
LLD.
Non-LLD.
LLD.
Non-LLD.
LLD.
Non-L.DD
Full-FT
NIA
89 89.89
86 86
89.78
85.17
95,06
90.29
94.90
89.49
Naul = 2.4M.Nu. = 1.2M
FediT
32
83.49
68 66
81.75
68.15
88. 28.66
84.00
91.67
77.53
FedEx-LoRA
32
80.56
47.45
77.82
66.58
91.94
84.30
91.51
81.63
FFA-LoRA
64
78.82
56.34
78.17
59.98
91.53
86,03
91.82
83 33.30
Fed-SB
221
79.27
71.48
79.06
69.51
90.94
82.25
92.74
85.30
RAVAN
110
84.42
76.22
84.02
73.80
94.13
90.02
93.75
89.17
FedIT
64
83.82
71.01
84.04
T3.23
91.39
84.68
92.06
79.31
FedEX-LoRA
64
79.38
50.47
79.42
57.86
91.16
74.04
92.01
74.84
HFA-LORA
128
81.39
70.31
82.13
66.81
91.95
88.06
92.07
84.24
Fed-SB
73.12
83.90
71.13
92.29
86.89
92.78
82.46
RAVAN
156
85.04
77.20
85.55
77.81
93.92
89.41
94.28
84.34
Table 3: Performance comparison on 20 Newsgroups and MRQA.
20 Newsgroups (Acc. %)
MRQA (FI %)
20 Clients
50 Clients
20 Clients
50 Clients
Method
Rank
L.D.
Non-L.LD.
LLD.
Non-LL.D.
LLD.
Non-LLD.
L.D.
Non-LL.D.
Full-FT
N/A
71.34
69.29
71.71
70.13
62.19
62.25
62.41
62.51
Noa = 4.7M. Nam = 2.4M
FedIT'
69.07
61.98
67.99
60.67
61.00
60.57
61.24
60.52
FedEx-LoRA
32
69.04
62.52
68.19
63.33
60.99
60.68
61.40
60.56
FFA-LORA
64
68.11
62.36
68.00
64.86
60.31
60.40
61.21
60.14
Fed-SB
221
67.15
63.10
66.69
63.98
59.93
59.73
59.96
60.01
RAVAN
110
68.96
65.73
68.18
65.67
61.18
60.45
61.33
61.55
FedIT
64
69.36
64.41
68.12
62.67
61.25
60.75
61.39
60.26
FedEX-LORA
64
68.59
65.11
67.75
64.31
61.23
60.36
61.43
60.06
FFA-LORA
128
69.33
66.22
68.42
64.86
61.50
60.50
61.66
60.12
Fed-SB
313
68.07
64.18
67.58
65.59
60.22
60.11
60.28
60.60
RAVAN
156
69.29
66.45
68.89
66.85
61.82
61.33
61.73
61.26
4.2
Main Results:
Vision and Language
We consider an FL setting with partial client participation where, in each communication round, the server samples three clients uniformly at random. Every selected client performs 50 local training iterations before uploading its update. Note, we intentionally train for 50 bini-batches and not 50 entire traversals of the client's training dataset so that each client performs exactly the same number of forward-backen backward passes. We evaluate two separate trainable parameter budgets. The upper half of Tables 2 and correspond to the lower budget and the lower half to the higher budget. The RAVAN configuration uses heads where each head H, E Rexr and the specified rank. All results displayed in the following sections are the averages across 3 random seeds.
RAVAN achieves the best performance among all PEFT methods in all vision configurations and in 1 11/16 of the language configurations. A key finding is that its advantage widens systematically in the statistically heterogeneous regime. For CIFAR-100 with 50 non-L.L.D. clients, RAVAN exceeds the performance of FedEx-LoRA by 7.2% and FedIT by 5.6% at the lower budget, whereas the corresponding I.I.D. gains are 6.2% and 2.3% respectively. The language results also display larger improvements in the non.I.L.D paradigm. On 20 Newsgroups, the gap over Fed-SB reaches 2.6% with 20 non-I.L.D. clients the lower parameter budget. On MRQA, the pretrained T5-Base already attains a strong Fl score, so all PEFT methods yield only modest absolute gains. Consequently, the scores of different approaches are tightly clustered in the lower-budget rows. Even in this regime, the higher parameter budget setting, RAVAN outperforms every baseline in every MRQA configuration.
7

---

Accuracy Computational Heterogeneity
Accuracy Computational Heterageneity
0.95
0.95
MeiLoRA
(Wuihht)
0.85
0.85
0.75
0.75
0.65
0.65
8.60
0.55
0.55
Brdtt Shaped
Uniform
Skownd Ruyht
Belt Sbaped
Unlform
Skowed Right
(a) CIFAR- 100 Performance Comparison
(b) SVHN Performance Comparison
Figure Clients draw trainable parameter budget from bell-shaped, uniform, or skewed right distributions. All RAVAN variants outperform the baselines in every distribution.
4.3 Ablation Studies and Analysis
Initialization Comparison. Table 4 compares the
Table 4: Initialization comparison with 20 non-L.LD. clients and lower parameter budget.
initializations for the fixed bases B,, A; as described in Section 3.1 The fully orthogonal Gram-Schmidt initialization is consistently best on the vision tasks,
Method
CIFAR-100
SVHN
adding 1.2% over random normal SVHN and out- performing the constant baseline by 20% on CIFAR- 100. On language tasks, the random normal initial- ization outperforms the constant baseline by 8.9% on 20 Newsgroups and the shared subspace baseline by 0.26% on MRQA. Thus, the proposed orthogonal ini- tializations maximize the effective rank the update and yield the strongest accuracy across all domains. While Gram-Schmide outperforms other schemes on the vision tasks, the procedure is more computationally expensive in righ--imenssonal space. How- ever, since this initialization is one-time server-run operation at the start of training, amortized across the entire FL. procedure, it adds virtually no overhead to the fine-tuning workload.
Random Normal
76.22
90.02
Trum-SShumidh
78.75
91.25
Constant
58.12
88.01
Shared Subspace
57.39
84.54
Method
20 Newsgroups
MRQA
Random Normal
65.
60.45
Gram-Schmidt
61.83
59.71
Constant
Shared Suhspace
55.64
60.19
Computational Heterogeneity. We emulate devices with unequal trainable parameter budgets by drawing each client's trainable parameter budget from three fixed distributions (bell-shaped, uniform, and skewed right). Details on the individual distributions can be found in the Appendix. As displayed in Figure 3 in these settings, all R AVAN variants outperform the rank-adaptive baselines HetLoRA and FleXLoRA on both CIFAR- 100 and SVHN. On CIFAR 100, the strongest baseline loses 11% of overall accuracy when moving from the bell-shaped distribution to the skewed right distribution. In comparison, RAVAN only loses 2% on average across its 3 variants. A key reason is that Aoe performs per-client SVD and HetLoRA performs hard-rank truncation when redistributing global model to individual clients. RAVAN avoids these approximation operations, so its updates remain accurate even in the extreme skewed right case. We note, however, that weight- based scoring consistently lags behind the other two scoring mechanisms because it always selects the same -iggmtag heads across all clients. Random scoring and gradient- pradeent-based scoring more evenly distribute updates across all heads and are hetter suited for heterogeneous fine-tuning in FL.
Table 5: Effect of using trainable scaling factors with non-I.I.D, clients and lower parameter budget.
CIFAR-100
SVHN
20 Newsgroups
MRQA Clients
Method
20 Clients
50 Clients
20 Clients
Clients
20 Clients
50 Clients
20 Clients
Constant
74.93
71.53
90.35 90.02
89.58
65.24 65.73
65.39
60.45
61.44
Trainable
76.22
73.80
89.17
65.67
60.60
61.53
Influence of Scaling Factors. Table compares the setting where scaling factors are set to constant 8; = Vi E [1....; throughout the fine-tuning procedure in comparison the standard RAVAN algorithm where the scaling factors are trainable. Keeping the scaling factors trainable boosts

---

Effect of Number of Heads
Effect of Number of Heads
14
3.5
V
Aecursa Galo Hicd
1.2
3.0
1.0
2.5
0.8
2.0
0.6
1.5
0.4
1.0
0.2
0.5
0.0
0.0
20
Hleads
(a) Lower Parameter Budget Comparison
Higher Parameter Budget Comparison
Figure 4: Comparison of performance when using different numbers of RAVAN heads at two different parameter budgets (SVHN: Nuxal = 1.2 M/2.4 M, 20 Newsgroups: Nusal = M/4.7 M).
accuracy on three of the four datasets while never hurting performance by more than 0.4%. The improvement is largest when client updates most diverse, specifically in CIFAR-I00, evidenced by Table 2 These lightweight scaling factors upweight useful heads and diminish the importance of heads whose fine-tuning subspaces provide less utility. Notably, we can achieve these performance gains without increasing the per-round communication cost or breaking exact aggregation.
Effect of Number of Heads. From Section 2 know that with a per-layer trainable parameter
budget of N, the effective rank of the RAVAN update is VNh. However, the maximum effective rank is still bounded by d, where the pretrained weights W E [gdxd, Hence, adding heads improves the effective rank only while the value of /i increases and remains below the dimension d. Figure 4 confirms this behavior as accuracy generally only improves while VNh < d. In the lower parameter budget setting, this happens with a larger number of heads as can assume larger value while still meeting this condition. In the higher parameter budget setting, we reach this saturation point much sooner. After V exceeds the value of d, the effective rank of each individual head becomes smaller while the actual overall update does become more expressive. This reduces representational power, especially at much larger values of h, and weakens performance. Thus, optimally choosing the number of heads critical criterion for effective federated fine-tuning using RAVAN.
Table Accuracy comparison on GLUE benchmark with LLaMA3.2-1B.
Method
MNLI-MM
MNLI-M
QNLI
QQP
SST-2
RTE
Average
FedIT
84.24
84.62
82.74
85.96
94.61
65.70
82.97
FedEX-LoRA
84.15
84.70
82.74
86.07
94.61
65.34
82.94
FFA-LORA
85.05
85.78
82.07
84.40
94.38
62.46
82.36
Fed-SB
84.88
85.23
82.84
84.23
94.95
67.15
83.21
RAVAN 85.24
85.65
84.00
86.11
95.18
67.15
83.90
Scaling to Larger Model Architectures. We demonstrate the scalability of RAVAN for larger model architectures by benchmarking the method against prior baselines on the GLUE benchmark 37 using LLaMA3.2- 13 (see T) For each subtask, use C 20 clients and follow the training procedure described in Section 4.2 We use a trainable parameter budget of N = 15 M, which corresponds to the rank configurations in the upper half of Tables 2 and On average, RAVAN outperforms the next best baseline by 0.7% with the maximum gain on a single subtask being the 1.2% improvement over Fed-SB the QNLI dataset. This demonstrates that RAVAN scales smoothly from 85 M to billion-parameter L.LMs and complements state-of-the-arr on-device models.
Conclusion and Future Work
RAVAN offers new avenue for performing FL. fine-tuning in the presence of data and computational heterogencity. By using multiple augmented LoRA heads and per-head scaling factors, RAVAN

---

improves the rank of the update approximation within parameter budget, allowing the method to better approximate the paramper update without exceeding a device's memory constraints. Partially freezing subsets of the heads allows clients to adaptively manage their own computational restrictions without sacrificing the accuracy of the aggregated model update. We helieve that RVAN and similar methods will open the doors for LLMs to capitalize on the vast amounts of edge data, virtually untapped resource for model training and a critical future direction for the frontier of ML.
While RAVAN outperforms existing PEFT benchmarks in FL, we identify three limitations and potential directions for improvement in the current method. First, while clients can fine-ture subsets of the heads to address device-level constraints and computational heterogeneity, the current framework necessitates that the same number of heads be selected in each layer of the original model. This reduces flexibility and can be remedied by cross-layer scoring scheme that considers all RAVAN heads simultaneously. Second, RAVAN has yet to be tested in the context of differentially-pritn learning, and further study is required to validate its performance with stricter privacy guarantees. Finally, data-aware initializations of the B, and A, parameters may improve the performance of the method while retaining the current improvements in the rank of the update approximation.
(DP)
6 Acknowledgements
This work was supported in part by NSF grants CCF 2045694, CNS-2112471, CPS-2111751, SHF-2107024, ONR grant N00014-23-1-22 1--219,,, Google Research Scholar Award, the CyLab loT Enterprise Security Initiative, and the CMU Prabhu and Poonam Goel fellowship. This work used Bridges-2 GPU at the Pittsburgh Supercomputing Center through allocation CIS250011 from the Advanced Cyberinfrastructtre Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants #3122259, #2138286, #2138307, #2137603, and #2138296 (7]. We would like to thank Pranay Sharma, Siddharth Shah, Kevin Kuo, Aneesha Sampath, and Akash Dhasade for providing feedback and contributing to discussions for the project.
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In International Conference on Learning Representations, 2021. URL
id B7V40MM6299
[2] Paul Albert, Frederic Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. RandloRA: Full rank parameter-efficient fine-tuning of large models. In The Thirteenth International Conference on Learning Representations, 2025. URL Bttptttt///ppereeiieee wttp://ww..e//toommidtiiiioaaoae
[3] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song. Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models, 2023. https
org/ 22008 2308.06522
[4] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https
//openreview net/ net/forummid-ee http//orrmmdddggooooiiBBUBB
[5] Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters, 2024. URL httpt:tttts:/iti// org/ aba/2405. 17604
[6] Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Lora-fair: Federated Iora fine-tuning with aggregation and initialization refinement, 2025. URL https://arxiv...rg watg/bb///// abbs/211
14961
[7] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023:
10

---

Computing for the Common Good, PEARC '23, page 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. doi: 11
httpp::/p:///i..... org/ 10. 3599995
[8] Zheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex, Ali Anwar, Nathalie Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, and Yue Cheng. Tifl: A tier-based federated learning system. In Proceedings of the 29th international symposium on high--erformance parallel and distributed computing, pages 125-136, 2020.
[9] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pages 12903-1299 12913, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1/653//1/2224.mmml main.717. URL
https://acll hct actp/holoa org/2024. emnlp-main. 717/
[10] Enmao Diao, Jie Ding, and Vahid Tarokh. Hetero Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021. URL httpps://ppereevieee
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http://wwweeet..ett/./oooooaaaaa
[12] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932- 1945, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/w12022..cchh 106. URL
org 2024
acl acl-long. 106/
[13] Aaron Grattafiori et al. The llama 3 herd of models, 2024. URL httthtt////arxiitt .rg abs/
[14] Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen, editors, Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China, November 2019, Association for Computational Linguisties. doi: 10. 10.1866////D1195800. URL https://aclannthologg org/ SXg/DD995881/
[15] Pengxin Guo, Shuang Zeng. Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selec- tive aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025. URL htttp:://openreiiiii www.:ewwwwee/ooonin.o
id=1xenEsGas0
[16] Edward Hu. yelong shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URLphttss h
het//forumiiddnZeeWeeee
[17] Jiahui Hu, Dan Wang, Zhibo Wang, Xiaoyi Pang, Huiyu Xu, Ju Ren, and Kui Ren. Federated large language model: Solutions, challenges and future directions. IEEE Wireless Communica- tions, 2024.
[18] Nam Wreon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficic federated learning. International Conference on Learning Representations, 2022. URL httpps://openrevie: http://wwwwweeeeeeeeeoeoeeeeeaee 1nAfttd
11

---

[19] Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei. Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-effiient fine-tuning, 2024. htRl.htts::// btttps://arxi...ee wwwpxxi..or/ab//////// 12130
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132-5143, 5143. PMLR, 2020.
[21] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfih International Conference on Learning Representations, 2024. URL
3f
[22]
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https www.eaattiicccoll..rrrrrrooaaaae
18268744
[23] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10713-10722 2021.
[24] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[25] Yongle Li, Bo Liu, Sheng Huang. ZHeng ZHang, Xiaotong Yuan, and Richang Hong. Communicationn-eficceenn and personalized federated foundation model fine-tuning via tri-matrix adaptation, 2025. URL
org/ h/g//bb//5500 .
[26] Shih- Shih-Yang Liu, Chien-Yt Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Timg Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. URL htttt:://aratttte
orooooggg/oo.oooooooooooe abt///////22////////////////// 09353.
[27] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas commumication-efffcciee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[28] Tom Mitchell. Twenty Newsgroups. UCI Machine Learning Repository, 1997. DOI:
https///bb.or/11022444//CCC
[29] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011. http: stanf ord
wet.hhussenmmerr//aa https//iiinppii111 housenunbers, httpoibrrrrrr.frr........
[30] John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek. and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International conference on artificial intelligence and statistics, pages 3581-3601 PMLR, 2022.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(14)):1-67, 2020.
[32] Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. Fedex-loRA: Exact aggregation for federated parameter-efficient fine-tuning of foundation models. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. URI https
//openreview.c wtt.://wwwneet/toomdddd/ddoeeeee
[33] Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Lav R Varshney, and Praneeth Vepakomma. Fed-sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning. arXiv preprint arXiv:2502. /5436, 2025.
[34] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. The Twelfth International Conference on Learning Representations, 2024.
URL http:pp////pppeeeeeeeeeeete/ttns
12

---

[35] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng zhong Xu. HydraloRA: An asym- metric loRA architecture for efficient fine-tuning. In The Thirty-eightt Anmual Conference on Neural Information Processing Systems, 2024, URL httpp:://ppeurreeee..eeee wwwp:/nn//////meeoooo qEp18uWX3N
[36] Van-Tuan Tuan Tran, Le Huy Khiem, and Viet Quoc Pham. Revisiting sparse mixture of experts for resource-adaptive federated fine-tuning foundation models. In ICLR 2025 Workshop Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025. URL
https://openreviesz http://rrrrraaeeeeeeouoorruoaaaa
[37] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.
GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP
Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353- 355, Brussels. Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/w1/W18 5446. URL /ttpt///////////ahhllooggg.. -rtp:///oooooooooo------------0- W094W115
[38] Ziyao Wang. Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li.
FL.RA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In The Tiiitt--iihht Annual Conference on Neural Information Processing Systems, 2024. URL
[39] Yuxuan Yan, Qianqian Yang, Shumpu Tang, and Zhiguo Shi. Federatefficient fine-tuning of language models in federated learning leveraging weight decomposition, 2024. URL https
//arxiv.org/ abs/ 2404. 18848
[40] Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Ang Li, et al. Federated large language models: Current progress and future directions. arXiv preprint arXiv:2409.1572) 2024.
[41] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6915-6919 IEFE, 2024.
[42] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parfmeter-efficient fine- tuning. In The Eleventh International Conference on Learning Representations, 2023. URL /sttps://openeviine net/forrmmidd= let//forumn/dd=1l@22WWJ...11
13

---

A Technical Appendices and Supplementary Material
A.1 Related Works
Federated Learning. Federated learning (FL) enables distributed set of clients collaboratively train single global model using their local data [27]]. The primary challenges in FL are data heterogeneity and computational heterogeneity induced by the statistical and hardware differences across clients, respectively. Several works aim to address these challenges by altering either the client training procedure or the server aggregation algorithm. FedProx [24]. SCAFFOLD (0), and FedDyn [1] add a corrective regularization term to each client's 's local Snrective-whethee a proximal penalty (FedProx), control-variate correction (SCAFFOLD)) or dynamic regularizer (FedDyn))to keep local updates from drifting too far from the global model, phenomena more often referred to "client drift". An alternative approach is to adjust the server aggregation scheme to reduce client drift. For example, FedVARP [4] incorporates historical client updates into the current round's d's aggregation step to reduce the variance caused by partial client participation and heterogeneity. FedExP [50] varies the server learning rate by using extrapolation rule that increases the server step size when consecutive aggregated updates point in similar directions and shrinks it when they diverge.
Parameter-F.fiiien Fine- Fine-Tuning. Recently, many works have adopted the pretrain-thes tthe--fie- tune framework, in which a general-purpose large language model (LLM) is adapted to a smaller downstream task [ 31 46 48) The excessive computational cost of fine-tuning LLMs has led to parameter-efficient fine-tuning (PEFT) methods that fine-tune a fraction the overall parameters of the model. Adapter tuning 47). BitFit [43, and low-rank adaptation (LoRA) [T6] have emerged as effective PEFT methods that can significantly reduce the number of parameters while maintaining task performance. Since its conception, many works have improved upon the initial LoRA formulation in various ways. Works such as QLoRA [45] and LoftQ [51] quantize the pretrained model weights to further improve the memory efficiency of LoR A-bsed fine-tuning. LORA-XS (5). LoRA-SB (52), and MoRA [9] introduce an additional LoRA parameter while freezing both the model backbone and the original LoRA parameters during fine-tuning. HydraLoRRA B5) LoRAMoE [12, and MoLE
55 employ a mixture-of-experts architecture to traditional LoRA-basee fine-tuning frameworks.
PEFT Methods for FL. As newer applications look to use on-device data to fine-tume LLMs, PEFT methods for FL have become increasingly relevant. FedPETuning [54], FedPrompt [55), and FedIT incorporate adapter tuning, prompt tuming, and LoRA into federated frameworks, respectively, More recently, methods like FFA-LORA 34 FedEx-LORA 32), Fed-SB (33] and RoL.R [44] optimize LoRA in homogeneous-compute by addressing the inexactness in LoRA aggregation caused by separately averaging the B and A parameters. An orthogonal direction is explored by works such as HetLoRA [9, FlexLoRA 44, and FLoRA (38] that address computational heterogeneity in LoRA- FL fine-tuning by allowing clients to train LoRA parameters with different ranks. However, an examination of cross-device fine-tuning remains limited in this context as methods like FLoRA have communication costs that scale linearly with the number of clients and communication rounds, making fine-tuning particularly difficult in large-scale FL systems. The goal of RAVAN is to design PEFT method for FL that addresses data and computational heterogeneity. while scaling effectively to cross-device settings with a large number of clients and communication rounds. In this way, we overcome shortcomings in prior works and enable resource-awar fine-tuning.
41
A.2 Broader Impacts
RAVAN enables accurate, efficient fine-tuning in federated setting. While RAVAN has not yet been integrated into a real-world FL system, we identify two potential impacts of utilizing RAVAN:
Edge Data for LLM Fine-Tuning: RAVAN provides opportunity for LLMs to utilize the data collected by edge devices. As edge applications become increasingly critical, capitalizing on these specialized datasets
make these applications more effective without forfeiting data locality. Efficiency and Improved Performance: LLM fine-tuning is an expensive, energy-consuming procedure. RAVAN improves the efficiency of the process while retaining performance. Applied to realistic FL training regimes, RAVAN addresses some of these prior concerns.
RAVAN should be implemented with safeguards to prevent misuse, privacy leaks, and harmful content. While RAVAN does not exacerbate these issues, they remain concerns in LLM usage more broadly.
14

---

A.3 Experimental Settings
Hyperparameters and Optimization Details. In this section, we highlight hyperparameter choices used in our experiments that were not discussed in Section 4 These descriptions, in addition to the provided code, should aid in the reproducibility of the stated results.
Table 7: FL hyperparameter settings used for each
pair.
NTT-B-16CCEAR-100
VIT-B-16SVVII
15-Base/20 Newsgroups
I3-Base/MROA
LLAMA32-IBGGLUEE
32
Max Sequence ength
236
256
128
Local lterations
50
50
50
50
30
Communication
Total Epochs (per round)
For RAVAN and each baseline, we run learning rate hyperparameter sweep across the values (5e-5, 1e-5, 5e-4, 1e-4, 5e-3, 1e-3, 5e-2, 1e-2, 5e-2} and choose the most performant learning to represent in our results. Table represents the optimal choices for each baseline in all settings. The following results each use the ADAM optimizer with momentum
0.9.
Table Optimal learning rate configurations for all baselines.
(a) Lower parameter budget L.D., clients.
Model/Datased
Method
VIT-B-16CIIFA-1000 VIT-B-1/SVHN
TS-Base/00 Newsgroups T5-BascMMRQA LLaMA32-IB/GLUE
FedIT
5x10
1x10
1x10
1x10
1x10
FedEx-LoRA
1x10-3
1x10-3
1x10
1x10-3
1x10
FFA FEA-LORA
1x10-2
1x10-2 1x10-4
5x10-a
1x10-2
Fed-SB
1x10-3
6x10-3
5x10-
1x10-1
RAVAN
5x10-4
5x10
5x10-4
1x10
5x10-6
(b) Higher parameter budget LLID. clients.
Model/Dataset
Method
VIT-B-16/CIFA-1100 WT-B-1G/SVHN
TS-Base/20 Newsgroups T5-Base/MRQA
LLaMA32-IB/GLUE
FedIT
5x10-3
1x10-3
1x10-3
5x10-4
1x10-4
LoRA
1x10-3
1x10-3
1x10-
5x10-4
1x10-1
FFA-LoRA
1x10-2
1x10-2
1x10-2
1x10-2
1x10-3
Fed-SB
1x10-3
1x10-3
5x10
5x10
1x10-4
RAVAN
5x10-
5x10-
1x10-
1x10-4
5x10-b
Lower parameter budget non-LL.D. clients.
Model/Dataset
Method
ViT-B-16/CIEAR-100
ViT-B-16/SVHN T5-Base/20 Newsgroups T5-Base/MRQA
FedIT
5x10-3
1x10-3
1x10-3
1x10-3
FodEX-LoRA
1x10-5
1x10-3
1x10-3
1x10-3
FFA-LoRA
1x10
1x10-2
1x10
5x10-3
Fed-SB
5x
5x
1x10-*
5x10
RAVAN
5x10-
5x10-4
5x 10
1x10-4
(d) Higher parameter budget non-I.L.D clients.
Model/Dataset
Method
VIT-B-16/CIFAR-1000
VIT-B-16/SVHN TS-Base/20 Newsgroups T5-Base/MRQA
FedIT
5x
1x10-3
1x1003
5x10
FedEx-LoRA
1x10
5x10
1x10
5x10 -4
FFA-LORA
1x10
1x10
1x10
5x10
Fed-SB
5x10
1x10
1x10
5x10
RAVAN
5x
5x10-4
1x10
1x10-4
15

---

Baseline Descriptions. We provide details for each of the baselines used in our experiments. We highlight how each baseline initializes, trains, and communicates the individual LoRA parameters.
FedIT: FedIT initializes the LoRA parameters with (0) 0 and (0) ~ (0, o²). communi- cation round I, each client E (()) locally trains the LoRA parameters resulting local parameters B(d) A After communicating these parameters back to the central server, the server performs the following aggregation to generate the global model:
B ((+1)
1
B CEC(
Σ
A (1+1)
A
C
(CO
cECO
LedEx-LLRAA: FedEx-LoRA initializes the LoRA parameters with B(O) = 0 and A (0) (0,, o²). In communication round f, each client E (()) locally trains the LoRA parameters resulting in local parameters B ( A To address the exact aggregation issue, the server updates both the global LoRA parameters as well as the model backbone:
(
Σ
(
(f+1)
Σ
A
CC00
CCO
CEC()
(10)
w(++))
w(O
1
Σ CECO)
B
(4) B
Σ CECO)
A
CC00
[CO0]
1000
cECO
While this ensures exact updates in every round, the updated model backbone ((++1) also has be communicated from the central server, increasing the communication overhead of the procedure.
FFA-LoRA: FFA-LORA initializes the LoRA parameters with (
and (00 o²).
However, the A parameter remains frozen at initialization and is never locally trained by the clients and communicated throughout the procedure. Thus, the only update throughout training is the update to the LoRA B parameter:
B
Σ D CEC(O)
(11)
|C01
Fed-SB: Fed-SB uses three LoRA parameters which, for the sake of consistency with prior notation, we call B E Rdxr, H E Rx A Igrdd The weight update is reparameterized BHA. To initialize the LoRA parameters, FB performs an initial round of full-parameter fine-tuning to obtain a full-parameter weight update A The weight update is decomposed using SVD to get A W full UEV B, H. and A are then initialized A V [ :.:r:. B and are frozen at initialization, the only update the following:
= U|: r). H E[ [[:r, 1r1::00
H (++1)
Σ OECLO
(12)
CC
FlexLoRA: Foe allows each client e E c(O) to train LoRA parameters with client-specific 1
ranks re- To aggregate the LoRA parameters, the server performs SVD on |cto] Eeecio A
UEVT. To redistribute the LoRA parameters back to the clients, the central server sends each client e C(++1) the following LoRA parameters:
B (*+11) U 1: r.] E re. re). A (t+1)_ = V re::
(13)
HetLoRA: Let Tmax be the highest rank supported by any client. Each client pads its local parameters this common shape (with zeros in the unused columns and rows) before upload, so aggregation is still dimensionally consistent. The server weights the individual LoRA parameters based on their relative Frobenius norms:
S
sO
|
A
P!0
Ecsci
S:O
(14)
B ((+1) = Σ
(t+1)
Σ p!0 A
P
cEC
The server then truncates the new global LoRA parameters B (t++) and A (++1) for each client
c(++)) so that B (((+)) ((+1)) [ 1 re] and ((+1) A (t+1) [1 Te.
16

---

Trainable Parameter Distributions
0.5
Bell-Shaped
Uniform
Proportion Clients
Skewed- Bkewed-Right
0.3
0.1
0.0
Fraction of Maximum Trainable Parameters
Figure 5: Fraction of clients assigned to each trainable parameter budget in each distribution. Skewed-left is omitted because it is never used in our experiments.
Computational Heterogeneity Setup. To emulate computational heterogeneity in FL setup, we vary the number of trainable parameters at each client. Let Nmax denote the largest number of trainable parameters that any client can afford. Each client is constrained to trainable parameter budget No E (2 2: 1/NNmax. This value is held constant throughout the FL procedure to mirror fixed hardware limits. No simply determines the trainable parameter budget per-client; all other hyperparameters are identical to the compute-homogeneouu experiments. The bar plot in Figure provides visual summary of the client mix. Uniform serves neutral baseline where there an equal proportion of clients at every trainable parameter budget; bell-shaped concentrates clients around medium ranks, reflecting the case where most devices have moderate capability; skewed-right stresses the system by placing a large share of the population at the lowest rank, leaving only a small fraction high-capacity contributors. HetLoRA, FlexLoRA, and RAVAN each accommodate clients with different trainable- parameter -rainable-parametee budgets in distinct ways:
(Nee/ /NNe/mms Nmax) Tmma where Tmmx is the maximum rank trained by any client. Since the number of trainable parameters scales linearly with the rank, this scaling ensures that every client keeps its update within the allotted budget N while allowing higher-capacity devices to contribute proportionally higher-rank updates.
HetLoRA and FlexLoRA: The LoRA rank for each client c 1
RAVAN: RAVAN uses H LoRA heads per weight matrix. A client with budget No, fine-tunes only [NNe Nmax) H heads and leaves the remaining heads frozen (e.g. for N Nmax the client trains one quarter of the heads).
Table 9: Layers equipped with LoRA adapters in each model backbone.
Model
LoRA Target Modules
ViT-B-16
query, value
T5-Base
SelfAttention.g q, SelfAttention.v q-proj, v_proj
LLaMA3.2-1B
LoRA Implementation Details. For every model backbone, insert LoRA adapters only in the self-attention projection matrices. The exact parameters for which we apply LoRA are described in All other parameters are frozen and do not have associated LoRA parameters.
Table 9
Compute Details and Cluster Description. All experiments were executed on GPU cluster managed by SLURM. Each training job used a single NVIDIA V100 32GB GPU with 256 GB RAM. Our environment used Pytorch 2.5.1 and Huggingface 4.47.1 for all experiments. With this setup, each experimental run took GPU hour with ViT-B-16 for both CIFAR- 100 and SVHN, x2 GPU hours with T5-Base for 20 Newsgroups, x3 GPU hours with T5-Base for MRQA, and x22 GPU hours with LLaMA3.2-1B for each GLUE subtask. baselines were trained with identical hardware, batch sizes, optimizers, and communication rounds to ensure fair comparison.
17

---

A.4 Additional Experiments
Accuracy with Computational Hoterogeneity
with Computational Beterogeneity
4000 (Raviooi
BAWAN (Oradivnt)
(Maugau
0.85
Netumic
0.80
0.75
8.20
0.5
8.65
0.55
Bell Shapad
Underr
Slowed Right
Bell Shaped
Undoem
Roowad Right
(a) CIFAR-100 lower parameter budget
(b) CIFAR-100 higher parameter budget
Accuraey with Compatational Hoterogencity
Accuracy with Computational Heterogenetty
0.45
0.95
0.90
0.90
0.85
0.85
0.60
0.90
0.55
0.55
Sell Shapad
Uniiora
Stawed Right
BAll Staged
Gasfort
Skowwed Right
(c) SVHN lower parameter budget
(d) SVHN higher parameter budget
Accuracy with Compatational Heterogenetty
Accuracy with Computational Heterogeneity
Aucumucy
0.55
0.56
0.50
0.50
0.15
0:45
Shapad
Uoilora
Skowed Right
Showwed R0ght
Hall
(e) Newsgroups lower parameter budget
20 Newsgroups higher parameter budget
Compulational Heterogezeiiy
Computsticnal Heterogeneity
0.610
0.62
0.005
0.61
Accurey
0.585
0.59
Showedd Right
Unttere
Skownd Right
Hall
Sell
(g) MRQA/ lower parameter budget
(h) MRQA higher parameter budget
Figure 6: Impact of computational heterogeneity baselines and RAVAN across four datasets. Each row shows single dataset (left: lower parameter budget, right: higher parameter budget), All settings match the descriptions from Section
Computational Heterogeneity Experiments. We evaluate methods with 20 non-I.I.D. clients whose usable parameter budgets are drawn from three distributions. Across vision and language tasks, RAVAN's variants consistently outperform the competing LoRA baselines. These results underscore RAVAN's robustness to computational heterogeneity across different tasks and parameter budgets.
18

---

Table 10: LoRA ranks under lower vs. higher parameter budgets.
Method
Lower Budget
Higher Budget
FedIT
32
64
FedEx-LoRA
32
64
FFA-LoRA
64
128
Fed-SB
221
313
RAVAN
110
156
Table 11: Accuracy comparison GLUE benchmark with LLaMA3.2-IB.
(a) 20 clients lower parameter budget
Method
MNLI-MM
MNLI-M
QNLI
QQP
SST-2
RTE
Average
FedIT
84.24
84.62
82.74
85.96
94.61
65.70
82.97
FedEX-LoRA
84.15
84.70
82.74
86.07
94.61
65.34
82.94
FFA-LoRA
85.05
85.78
82.07
84.40
94.38
62.46
82.36
Fed-SB
84.88
85.23
82.84
84.23
94.95
67.15
83.21
RAVAN
85.24
85.65
84.00
86.11
95.18
67.15
83.90
(b) 20 clients / higher parameter budget
Method
MNLI-MM
MNLI-M
QNLI
QQP
SST-2
RTE
Average
FedIT
83.74
83.24
87.72
85.60
95.30
68.95
84.09
FedEX-LoRA
83.95
83.41
87.79
85.65
95.41
70.04
84.38
FFA-LORA
85.27
84.69
89.51
87.10
95.18
68.23
85.00
Fed-SB
85.85
84.76
89.53
86.09
94.95
66.79
84.66
RAVAN
86.20
85.34
90.35
87.22
95.18
70.04
85.72
(c) 50 clients / lower parameter budget
Method
MNLI-MM
MNLI-M
QNLI
QQP
SST-2
RTE
Average
FedIT
84.22
84.24
87.53
85.87
94.61
61.73
83.03
FeRAX-LoRA
84.25
84.15
87.77
85.81
94.61
62.09
83.11
FFA-LoRA
85.92
85.05
89.33
87.40
95.30
60.29
83.88
Fed-SB
85.71
84.65
88.05
86.08
94.15
64.98
83.94
RAVAN
86.03
85.53
88.91
86.95
95.41
62.09
84.15
(d) 50 clients higher parameter budget
Method
MNLI-MM
MNLI-M
QNLI
QQP
SST-2
RTE
Average
FedIT
84.66
84.26
88.38
85.87
95.18
63.58
83.66
FoREX-LORA
84.74
84.02
88.50
85.82
95.41
58.12
82.77
FFA-LoRA
85.35
84.64
87.21
87.20
94.50
61.37
83.38
Fed-SB
85.91
85.24
87.68
86.30
93.58
67.15
84.31
RAVAN
86.17
85.35
88.87
87.39
95.87
64.62
84.71
LLaMA Experiments. In these experiments, we use the same hyperparameter settings described in Section 4 but vary the number of total clients and the the ranks of each baseline. Across four GLUE confguratioos, RAVAN consistently matches or exceeds the performance of the strongest PEFT baselines using LLaMA3.2-1 (see Taalee Additionally, while other PEFT baselines vary in performance across settings, RAVAN'S performance remains consistent in all configurations. This suggests that RAVAN maintains robust performance, demonstrating ability to scale effectively to larger models and diverse FL scenarios.
19

---

Training
Training
Accummy
Fedux- LaiIA
RAVAN
FedtT
PFA-LaRA
10
20
30
40
30
10
20
30
40
50
Communication Bounds
Communication Rounds
(a) CIFAR- 1EFAR-100/ 20 clients / lower parameter budget
(b) CIFAR- 100 50 clients lower parameter
budget
FL Training
Training
0.8
0.8
0.6
0.6
0.4
0.0
0.2
0.2
0.0
0.0
16
30
40
50
10
20
40
50
Communtration Hounds
Communication Rounds
(c) CTFAR- 1-00 / 20 clients higher parameter budget
(d) CIFAR-100 50 clients higher parameter
budget
EL Training Curves
FL. Training Carves
0.8
Acorse
0.0-
0.6
0.4
0.4
0.2
0.2
10
40
10
20
Communication Bounds
Communication Rounds
SVHN 20 clients/ lower parameter budget
SVHN clients/ lower parameter budget
FL Training Curves
Training
0.8
Accurary
0.0
0.6
0.4
0.4
0.2
0.2
1.0
40
50
10
20
30
40
50
Commomication Rounds
Communiattion Rounds
(g) SVHN 20 clients / higher parameter budget
(h) SVHN / 50 clients / higher parameter budget
Figure 7: FL training curves for CIFAR-100 100 and SVHN for all benchmarks.
Training Curves. Figure 7 displays the training curves for the various PEFT benchmarks on CIFAR100 and SVHN using varying number of I.I.D. clients and trainable parameter budgets. In comparison to the other PEFT methods, AVANN converges faster and to a better overall performance, suggesting that it requires fewer communication rounds to reach optimal performance.
20

---

Appendix References
[43] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 12.18866/11/22222.2 URL httts://aclanthhllee wrg/2022..c...... acl acl-short.1/
[44] Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish J Khisti. Robust federated finetuning of foundation models via alternating minimization of IoRA. In Workshop Efficient Systems for Foundation Models II @ ICML2024, 2024. URL https:///pperree forum?id-xTOacyogg
net/
[45] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs.. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpps://ppenreeiee..nttt wttp:/wwwww...e.eooommmmdddrrooe
[46] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pages 1171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18633/1//11--142 1423. URL https: / http//aclaaaa acllanhhloggy org/N19-1423/
[47]
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019. URL hnttps:/prooeeeding: mml press/v97/houlsbee 19 html
[48] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text clas- sification. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1031. URL /ttp://t..ttttttt////aaaaattttaa
[49] Divyansh Jhunjhunwala, Pranay Sharma, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tack- ling the variance due partial client participation federated learning. In Uncertainty in Artificial Intelligence, pages 906-916. PMLR, 2022.
[50] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated aver- aging via extrapolation. The Eleventh International Conference on Learning Representations, 2023. URL TtRLsttps:t http::::///ppeeevviii www.eeew.eeettteeeeeeeeeee mett/forumidd=PPPaale
[51] Yixiao Li, Yifan Yu. Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: LoRA-fine-tning tunne---rn--- quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:: peattt::://ppeneevvie het:forummIdd-zPPPP....
[52] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation a silver bullet for extremely efficient low-rank fine-tuning, 2025. https://:///t org/ abs 2411, 19557
[53] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts. In The Twelfth International Conference on Learning Representations, 2024. htR.htttss /tttt:::///ppeeeieeee net/forun id=uWykBCYh4S
21

---

[54] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. FedPETuning: When federated learning meets the ceaumeter--fficiem tuning methods of pre- trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 9963-9977, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v//2022 findings-acl.632. URL https: http:::::///aaaaaholoosccoo org//023.... 2023. findings--acee acl. 632/
org/
[55] Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication efficient and priiacy--reserviine prompt tuning in federated learning. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023. doi: 10.1 N0.119//AASSP44955..2222 10095356.
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We justify the contributions our method in Sections 3 and 4 which discuss the proposed method and experiments using the proposed method, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The conclusion Section 5 mentions three limitations of our work and potential directions for future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification: Our work is empirical and therefore does not require proofs of key theoretical findings.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4 and the Appendix provide all necessary hyperparameters and config- urations necessary to reproduce the experiments listed in the results. We highlight the most important values used in our setting in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code base is provided in the supplementary material zip file with README that includes instructions on run commands for our method and all baselines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cc/pwwppccccuceeeeeeoooooeee htte://wiisss/eeee wwwweoewwisssoonnooooooooooooeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and the Appendix provide all necessary hyperparameters and config- urations necessary to reproduce the experiments listed in the results. We highlight the most important values used in setting in Section 4
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results are reported as the average across runs with random seeds. For the sake of horizontal space we don't include the values directly in the tables, but provide standard deviation numbers in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The Appendix contains details on the resources required to run each experiment. All experiments were run using a single V100 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tt httppp/bbilctt.cccsceeeeeeeeee
Answer: [Yes]
Justification: We reviewed the code of ethics and ensured that research followed these guidelines. We avoid privacy violations, have documented the research thoroughly, and all involved parties have been fairly compensated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
26

---

Justification: Our paper highlights the potential impacts in this ongoing field of research in Sections I [ These sections highlight the bigger picture of work and place in the context of the larger field of research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our work uses general-purpose purpose models and datasets that have been adequately cited and referenced. We have avoid privacy violations and other security risks.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All original material is created with the consent of the authors. Any other material has been cited and referenced.
Guidelines:
The answer NA means that the paper does not existing assets.
27

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We release no new models/datasets in this project.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do include human subjects any experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: We do not use human subjects in any experiments.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLM usage declared in the initial submission. We used LLMs for grammar edits and to generate the icon in the intro line.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standardd components.
Please refer to our LLM policy for what should or should not be described.
29


Reviews:
[{"review_id": "gZZmvJL6Bu", "reviewer": "Reviewer_pUXf", "summary": "This paper introduces RAVAN, a multi-head, low-rank adaptation method for the federated fine-tuning of large models. The proposed method addresses the challenges of data and computational heterogeneity in federated learning by reparameterizing weight updates as a sum of multiple augmented LoRA heads. By freezing the outer matrices and only training the small inner matrices and lightweight scaling factors, RAVAN aims to improve the expressive power of the model updates while maintaining parameter efficiency and ensuring mathematically exact aggregation.", "questions": "While RAVAN offers a high theoretical rank, does the optimization process for non-I.I.D. updates actually utilize this full capacity? I'm interested in an empirical analysis of the singular values of the learned update matrix which demonstrates how much of the available rank is used in practice. This would confirm if the increased rank is a fully leveraged resource in heterogeneous settings or primarily a theoretical upper bound.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. **Exact Aggregation**: RAVAN's design provides an elegant solution to the inexact aggregation problem that affects many federated LoRA methods. By having clients only train and upload $s_iH_i$, the server can perform a direct, weighted average that is mathematically equivalent to averaging the full updates.\n2. **Communication and Computation Efficiency**: The method maintains the low communication cost of standard LoRA by only requiring clients to send the small, trained parameters. Furthermore, it directly addresses computational heterogeneity by allowing resource-constrained clients to train only a subset of the available heads, making the framework flexible and scalable in realistic FL environments. \n\nWeaknesses:\n\nOne of the paper's core argument is that RAVAN solves the non-I.I.D. issue, based on the observation that the spectrum of weight updates is broader in such settings.  While the empirical results are strong, this connection feels more like an intuition than a direct, proven solution. The observation in Figure 1 that $\\delta W$ has a more distributed rank does not necessarily mean that a low-rank approximation is the primary bottleneck for non-I.I.D. training. A more rigorous theoretical justification, perhaps related to convergence guarantees or generalization bounds, would be appreciated to formally connect the spectral properties of non-I.I.D. updates to the necessity of a higher-rank adaptation space.", "was_revised": true, "final_justification": "The author addresses most of my concerns. I will maintain my positive score.", "rebuttal": "Thank you for the meaningful feedback and the acknowledgement of Ravan's strengths and practical utility. We would like to address your individual concerns. \n\n**W1. A more rigorous theoretical justification would be appreciated to formally connect the spectral properties of non-I.I.D. updates to the necessity of a higher-rank adaptation space.**\n\nThank you for bringing up this discussion; we believe this is a valid point about the theoretical rigor of our proposed method. Unfortunately, there exists limited theory about the spectral properties of LoRA optimization, especially in distributed or federated settings. As such, our work closely aligns with previous works in federated fine-tuning that focus primarily on the empirical advantages of methodological innovations ([1], [2], [3]). That being said, there are a few prior works ([4], [5], [6]) that tie the convergence of LoRA directly to the rank and suggest that when the actual desired update is high rank, LoRA may underperform or even fail to converge. This is particularly important given our observation that the true update is high rank in non-I.I.D. settings. In this context, we believe there are two critical pieces of evidence that suggest that the rank of the update approximation contributes to the improvement in performance in federated settings. \n\n* Increasing the number of heads, and subsequently the rank of the update approximation, improves the performance of the method *only until* the rank of the update approximation stops increasing. In Section 4.3, we include an ablation that demonstrates the impact of changing the number of heads while leaving the number of trainable parameters constant. We find that increasing the number of heads is beneficial as long as the rank of the update approximation remains less than the embedding dimension, which is the maximum possible rank of the update approximation. However, once the rank of the update approximation equals the embedding dimension and stops increasing, these performance gains vanish. Since we hold the number of trainable parameters constant in this experiment, this conveys that the rank of the update approximation is associated directly with the performance of the method.\n* Initialization schemes that do not increase the effective rank of the update approximation perform worse on all benchmarks. In Section 3.1, we propose two initialization benchmarks that do not increase the effective rank of the update approximation. In the constant initialization $\\textbf{B}_i = \\textbf{B}_j$ and $\\textbf{A}_i = \\textbf{A}_j \\ \\forall i,j$, and in the shared subspace initialization $\\textbf{B}_i = \\textbf{MR}_i, \\textbf{A}_i = \\textbf{R}_i\\textbf{N}$ for normally distributed $\\textbf{M}, \\textbf{N}$ and invertible $\\textbf{R}_i \\ \\forall i$. We highlight that the constant initialization and shared subspace initialization underperform relative to the random normal initialization and Gram-Schmidt initialization. Despite using multiple heads, the constant and shared subspace initializations have an update approximation that is the same rank as a single head. In contrast, the random normal initialization and Gram-Schmidt initialization have an effective rank that is (number of heads * rank of each head). Since each initialization uses the same number of trainable parameters, the same rank for each head, and the same number of heads, the only variable that changes is the effective rank of the entire update approximation which qualifies the importance of increasing the rank of the update approximation. \n\nWhile we agree that our work presents limited theoretical insights into this behavior, we believe these experiments and prior theoretical work combined present convincing evidence that the improved rank of Ravan's update approximation is a critical component of the method's performance. \n\n**Q1. While Ravan offers a high theoretical rank, does the optimization process for non-I.I.D. updates actually utilize this full capacity?**\n\nYes, the optimization process actually utilizes this full capacity as demonstrated by the following experiment. For CIFAR-100, we calculated the rank of the update approximation $\\sum s_i\\textbf{B}_i\\textbf{H}_i\\textbf{A}_i$ for all 4 initialization schemes (constant, shared subspace, random normal, Gram-Schmidt) after communication round 1 and communication round 50. For the sake of calculating the rank, any singular value less than $1 \\times 10^{-6}$ is considered 0. We follow the same hyperparameter setup in Tables 2 and 3 by using 4 heads of rank 156 each.\n\nAfter communication round 1 -\n\n| Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- |\n| 624     | 624     | 156     | 156     |\n\nAfter communication round 50 -\n\n| Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- |\n| 624     | 624     | 156     | 156     |\n\nThese results establish that Ravan achieves the upper bound theoretical rank of 624 ($4 \\times 156$) throughout the training procedure when using initializations that result in orthogonal $\\textbf{B}_i$ and $\\textbf{A}_i$. It also lends further credence to the idea that the differentiating factor in performance between the various initialization schemes is a large difference in effective rank. We have included the performance for the different initialization schemes on CIFAR-100 here for convenience. \n\nResults for CIFAR-100 -\n\n| Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- |\n| 78.75    | 76.22     | 58.12    | 57.39    |\n\n[1] \"FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models\" Singhal et al. \n\n[2] \"FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations\" Wang et al. \n\n[3] \"Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models\" Cho et al. \n\n[4] \"Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation\" Malinovsky et al. \n\n[5] \"GeLoRA: Geometric Adaptive Ranks for Efficient LoRA Fine-Tuning\" Ed-dib et al. \n\n[6] \"LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)\" Kim et al."}, {"review_id": "7UXS5ONb1u", "reviewer": "Reviewer_SFDe", "summary": "This paper proposes a contribution to the field of Federated Learning with LLM models. A shortcoming of FL for LLMs is the high communication budget between clients and the server. Previous works in the literature, such as FedIT, proposed using LoRA to reduce this communication cost. This work refines that method, proposing a multi-head LoRA adaptation in the fashion of HydraLoRA. Here, however, the authors propose an augmented version si​Bi​Hi​Ai​ that allows for exact aggregation and increased expressivity. In this version, only si​ and Hi​ are trained and communicated, while Ai​ and Bi​ remain frozen. Experimental results advocate for good efficiency on a diverse set of architectures and datasets.", "questions": "- Could the authors provide more grounded explanations for the strategies proposed in parts 3.1 and 3.2? Specifically, why does Gram-Schmidt perform better, and is there a theoretical justification for this choice?\n\n- Can the authors perform new experiments with a more reasonable number of local epochs for client training? This would be necessary to position the results in the context of federated learning and make them comparable with other works in the literature.\n\n- How are the \"scaling factors\" learned in the proposed method? A clearer explanation of this process would be beneficial for understanding the overall approach.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The method is well-grounded and motivated, with a solid overview of related work. It is integrated into the broader context of federated learning and low-rank adaptation.\n- The method is evaluated on CIFAR-100, which is a challenging dataset in the context of federated learning.\n- The comparison to other methods in the same scenario is well-conducted, and the results are good for the proposed method.\n- RAVAN is easy to implement, which is a plus for practical applications.\n\nWeaknesses:\n\n- The claim \"To our knowledge, RAVAN is the first federated PEFT technique that simultaneously maintains parameter-efficient computation and communication and addresses both data and computational heterogeneity\" is too strong. The idea has been around for a long time, with works like FedPara and others. The authors should be more cautious with such claims.\n- Parts 3.1 and 3.2 of the paper raise questions about the underlying assumptions and motivations for the proposed strategies. For part 3.1: why does Gram-Schmidt perform better? Is there a theoretical justification? It would also be interesting to compare these results with a pre-trained initialization on the same domain using a random dataset.\n- The choice of 50 local epochs for client training is very problematic for comparing results with other works in the literature. In federated learning, it is common to see clients training for 1, 5, 10, or 20 epochs. Training for 50 epochs is almost equivalent to letting each client train their own model, which undermines the federated learning paradigm. It is absolutely necessary to perform new experiments with a more reasonable number of local epochs.\n- The paper mentions that the \"scaling factors\" are learned, but it is not clear how this learning process occurs. This aspect needs to be clarified for better understanding.", "was_revised": true, "final_justification": "Authors have addressed my few concerns in the rebuttal. I changed my evaluation to 5: Accept.", "rebuttal": "Thank you for the detailed critique of our work and the recognition of its value in the field of federated fine-tuning. We aim to thoughtfully address the specific points you have raised.\n\n**W1. The claim \"To our knowledge, RAVAN is the first federated PEFT technique that simultaneously maintains parameter-efficient computation and communication and addresses both data and computational heterogeneity\" is too strong. The idea has been around for a long time, with works like FedPara and others.**\n\nWe appreciate the feedback on the framing of our contributions and will revise this sentence in the final draft of our paper to better reflect the benefits of Ravan in the context of current research. The following will be included in the paper should it be accepted \"Extending prior efforts such as FedPara, Ravan introduces an integrated framework that maintains parameter‑efficient computation and communication and demonstrates robustness across diverse data and computational heterogeneity in federated settings.\" \n\n**W2. For part 3.1: why does Gram-Schmidt perform better? Is there a theoretical justification?**\n\nSection 3.1 discusses various strategies for initializing the Ravan parameters. In particular, we find that the Gram-Schmidt initialization and random normal initialization are effective initializations for the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters. The random normal initialization yields orthogonality for the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters in expectation, while Gram-Schmidt ensures that the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters are orthogonal deterministically. This orthogonality is important because the column space of $\\sum  s_i\\textbf{B}_i\\textbf{H}_i\\textbf{A}_i$ is a subspace of the span of the $\\textbf{B}_i$'s and the row space of $\\sum s_i\\textbf{B}_i\\textbf{H}_i\\textbf{A}_i$ is a subspace of the span of the $\\textbf{A}_i$'s. Therefore, if we constrain all the $\\textbf{B}_i$'s and $\\textbf{A}_i$'s to share the same subspace, the rank of $\\sum s_i\\textbf{B}_i\\textbf{H}_i\\textbf{A}_i$ collapses to that of a single head. This is the case for both the constant initialization, where $\\textbf{B}_i = \\textbf{B}_j$ and $\\textbf{A}_i = \\textbf{A}_j \\ \\forall i,j$, and in the shared subspace initialization, where $\\textbf{B}_i = \\textbf{MR}_i, \\textbf{A}_i = \\textbf{R}_i\\textbf{N}$ for normally distributed $\\textbf{M}, \\textbf{N}$ and invertible $\\textbf{R}_i \\ \\forall i$. For these initializations, we would not benefit from using multiple heads at all since the effective rank of the update approximation would not actually improve. Section 4.3 demonstrates that initializations that do improve the rank of the update approximation (random normal, Gram-Schmidt) outperform methods that do not improve the rank of the update approximation (constant, shared subspace). \n\n\nFurther empirical evidence of the improvement to the rank of the update approximation can be demonstrated by the following experiment. For CIFAR-100, we calculated the rank of the update approximation $\\sum s_i\\textbf{B}_i\\textbf{H}_i\\textbf{A}_i$ for all 4 initialization schemes (constant, shared subspace, random normal, Gram-Schmidt) after communication round 1 and communication round 50. For the sake of calculating the rank, any singular value less than $1 \\times 10^{-6}$ is considered 0. We follow the same hyperparameter setup in Tables 2 and 3 by using 4 heads of rank 156 each.\n\nAfter communication round 1 -\n\n| Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- |\n| 624     | 624     | 156     | 156     |\n\nAfter communication round 50 -\n\n| Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- |\n| 624     | 624     | 156     | 156     |\n\nThese results establish that Ravan achieves the upper bound theoretical rank of 624 ($4 \\times 156$) throughout the training procedure when using initializations that result in orthogonal $\\textbf{B}_i$ and $\\textbf{A}_i$.\n\n**W3. It would also be interesting to compare these results with a pre-trained initialization on the same domain using a random dataset.**\n\nThis is an exciting idea and a potential alternative initialization scheme to the ones we include in the paper. We have benchmarked this initialization in which we pretrain the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters prior to the FL fine-tuning procedure. The $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters are subsequently frozen at this pretrained initialization. The following are results on CIFAR-100 and SVHN in which the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters are pretrained on random subsets of CIFAR-100 and SVHN, respectively. The hyperparameter settings follow the ones listed in Table 4. \n\nResults for CIFAR-100 -\n\n| Pretrained | Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- | -------- |\n| 80.49    | 78.75    | 76.22     | 58.12    | 57.39    |\n\nResults for SVHN -\n\n| Pretrained | Gram-Schmidt | Random Normal | Constant | Shared Subspace |\n| -------- | -------- | -------- | -------- | -------- |\n|  89.53    | 91.25    | 90.02   | 88.01  | 84.54    |\n\nThe results of this initialization are mixed with improvements on our proposed initializations in CIFAR-100 but a slight decrease in performance on SVHN (though we admit that testing is limited given the time constraints of the rebuttal period). While this direction is certainly interesting, and a potential direction for future work, we would like to bring up that this initialization would likely require pretraining the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters at the server since they are quite a bit larger than the $\\textbf{H}_i$ parameters and might be difficult to train given device resource constraints. This would require a public dataset at the server that could be used to pretrain the $\\textbf{B}_i$ and $\\textbf{A}_i$ parameters. Thus, there are some limitations that might make this initialization more difficult in practice. The initializations that we propose are light-weight, effective, and data-independent. That being said, we do believe that the initialization you propose is an intriguing direction for future study. \n\n**W4. The choice of 50 local epochs for client training is very problematic for comparing results with other works in the literature. In federated learning, it is common to see clients training for 1, 5, 10, or 20 epochs.**\n\nWe believe there is a misunderstanding here -- we train for 50 *local iterations* and not 50 *local epochs*. This means that for each client, we only train on 50 mini-batches and *not* 50 entire traversals of the client's training dataset. 50 local iterations, depending on the dataset size and batch size, is approximately 1 local epoch (client dataset size / batch size = local iterations per epoch). We use local iterations instead of local epochs since clients may have different amounts of training data, so using local iterations means that each client performs exactly the same number of forward-backward passes. We believe that this is a fair amount of compute for resource-constrained devices and does not differ from most modern FL literature in a significant way. We will highlight the difference between local iterations and local epochs in the final draft of our paper to avoid similar confusion in the future. However, to demonstrate the robustness of our method, we include the results for CIFAR-100 and SVHN using 1 or 2 local epochs below. \n\nResults for CIFAR-100 with 1 Local Epoch - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 75.21    | 61.32    | 68.47     | 73.34    | 79.12    |\n\nResults for CIFAR-100 with 2 Local Epochs - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 72.36    | 77.36    | 74.66     | 77.61    | 80.88    |\n\nResults for SVHN with 1 Local Epoch - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 92.70    | 91.40    | 92.77     | 92.82    | 93.15    |\n\nResults for SVHN with 2 Local Epochs - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 93.91    | 92.43    | 93.37     | 94.02    | 94.53    |\n\n**W5. The paper mentions that the \"scaling factors\" are learned, but it is not clear how this learning process occurs. This aspect needs to be clarified for better understanding.**\n\nEffectively, the scaling factors $s_i$ are trainable parameters just like any other trainable parameter in the network. The only difference is that they are scalars and not multi-dimensional tensors. However, they receive gradients, just like the $\\textbf{H}_i$ parameters, and are trained with these gradient updates. The advantage to keeping these $s_i$ factors trainable is that the network naturally learns which heads are more significant throughout the fine-tuning procedure and as a consequence adjusts the value of $s_i$, in essence learning the scaling factors most beneficial for model prediction."}, {"review_id": "jr50vLtt1v", "reviewer": "Reviewer_1RFB", "summary": "This paper presents a new approach to federated learning (FL) based on parameter efficient fine-tuning (PEFT) which works with computational and data heterogeneity while allowing for maximum effective rank with low communication overhead and exact aggregation across clients. Specifically, it proposes a multi-head variant of LoRA whereby the weight updates are factored as in LoRA-XS (i.e. \\Delta W = B H A) and each client updates a set of H matrices and their associated scalar scaling factors i.e. (\\Delta W = \\sum_{i=1}^{h}  s_i B_i H_i A_i), and the coefficients A_i and B_i are frozen. For a given parameter budget N, the method allows for a higher effective rank \\sqrt(N*h) compared to LoRA (N/2d). Also, it allows for exact aggregation across clients along with communication efficiency. Results on vision and language tasks show that the approach improves performance over other PEFT approaches (e.g. FedIT/Fed-SB). Ablations are shown to compare performance as a function of the mode of initialization of the B/A matrices, head selection strategies, computational heterogeneity, scaling factors, number of heads and scaling.", "questions": "* Sec 4.3 / Figure 3: Computational Heterogeneity: Could you provide more details about how the clients choose  # of heads based on their computational capacity? Also, it would be useful to show the distribution of # of heads chosen across clients for each of the 3 distributions: Bell-shaped/Uniform/Skewed right\n* Effect of number of heads (Figure 4): It would be good to understand what happens when \\sqrt(N*h) exceeds d. Why does the effective rank of each head become smaller in this setup? Could you provide further insight?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Proposes a new PEFT method for FL which is able to tackle computational and data heterogeneity while allowing for a large effective rank.\n* Shows that the method outperforms various FL PEFT approaches on vision and language tasks.\n* Reports a number of ablations which show the impact of key factors such as initialization, scaling factors\n* Approach improves upon prior approaches which either have inexact aggregation (FedIT) or exact aggregation with high communication costs (FFA-LoRA).\n\n\nWeaknesses:\n* Results are reported on simulated data, where the heterogeneity has been introduced artificially (e.g. by drawing client specific class proportions based on a Dirichlet distribution with known parameters).  It is unclear if the method can capture heterogeneity in real datasets. It would be useful to report performance on alternative datasets used in FL research e.g. FedJAX (https://github.com/google/fedjax) contains prepackaged datasets such as EMNIST-62, Stack Overflow grouped by writers/username which would allow experimentation to study data heterogeneity.", "was_revised": true, "final_justification": "The authors' responses have adequately answered my questions - Hence, I am increasing the score.", "rebuttal": "Thank you for the thorough assessment of our paper and valuable feedback. We hope to comprehensively respond to each of your concerns. \n\n**W1. Results are reported on simulated data, where the heterogeneity has been introduced artificially. It is unclear if the method can capture heterogeneity in real datasets. It would be useful to report performance on alternative datasets used in FL research e.g. FedJAX (https://github.com/google/fedjax) contains prepackaged datasets such as EMNIST-62, Stack Overflow**\n\nThank you for the suggestion; based on this comment we have included results on federated EMNIST-62 where the dataset has a natural non-I.I.D. partition based on the writer responsible for specific handwritten characters. We use the ViT-B/16 model and the hyperaparameter setup described in the lower halves of Tables 2 and 3 in Section 4 of our paper. We train 3 different settings in which we randomly sample a total of 20 clients, 50 clients, or 100 clients for model fine-tuning and where each client corresponds to a specific writer and their dataset corresponds to their handwritten characters. \n\nFEMNIST-62 with 20 Clients - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 78.97    | 79.01    | 78.54     | 77.75    | 80.51    |\n\nFEMNIST-62 with 50 Clients - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 78.96    | 77.99    | 79.37     | 76.99    | 82.05    |\n\nFEMNIST-62 with 100 Clients - \n| FedIT | FedEx-LoRA | FFA-LoRA | Fed-SB | Ravan |\n| -------- | -------- | -------- | -------- | -------- |\n| 81.07    | 80.07    | 79.65     | 79.97    | 81.91    |\n\nAs is evident, even when using a dataset with a natural non-I.I.D. partition, Ravan outperforms related federated PEFT methods. Thus, the method is not just applicable in simulated non-I.I.D. datasets but also practically useful in real-world settings. We will include this experiment in our final paper. \n\n**Q1. Could you provide more details about how the clients choose # of heads based on their computational capacity? Also, it would be useful to show the distribution of # of heads chosen across clients for each of the 3 distributions.**\n\nFor the sake of space, the discussion on how clients choose the number of heads for fine-tuning was included in the Appendix as opposed to the main text of the paper. The distribution of number of heads chosen across clients for the 3 distributions can be found in Figure 5 of the Appendix. We define $N_{\\max}$ as the largest number of trainable parameters that any client can afford. Each client $c$ is constrained to a trainable parameter budget $N_c \\in \\\\{\\frac{1}{4}, \\frac{1}{2}, \\frac{3}{4}, 1\\\\} N_{\\max}$. A client with budget $N_c$, fine-tunes only $\\bigl\\lfloor \\left(N_c \\\\, \\/ \\\\, N_{\\max}\\right) \\cdot H \\bigr\\rfloor$ heads and leaves the remaining heads frozen (e.g. for $N_c = \\frac{1}{4} N_{\\max}$ the client trains one quarter of the heads).\n\n**Q2. It would be good to understand what happens when $\\sqrt{N*h}$ exceeds d. Why does the effective rank of each head become smaller in this setup? Could you provide further insight?**\n\nAn important thing to note for Figure 4 is that while we increase the number of heads, we are still keeping the number of trainable parameters constant. In order to do so, the size of each individual head must decrease, otherwise the number of trainable parameters would also increase. As a consequence, the effective rank of each individual head goes down. This is fine as long as the effective rank of the overall sum continues to increase. However, beyond a certain number of heads, the effective rank of the overall sum equals the embedding dimension $d$, which is the maximum possible rank of the update approximation, and no longer increases. At this point, as each individual head becomes less expressive, model performance begins to degrade. This phenomenon can be observed in our results figure."}]

Author rebuttal:

Meta-review: This paper introduces a novel multi-head, low-rank adaptation method for addressing the data and computational heterogeneity challenges in the setting of federated fine-tuning of large models. The effectiveness of the proposed approach is verified by extensive experimental evaluations.  One weakness is the lack of theoretical justifications, also pointed out by one of the the reviewers.

The paper receives all positive scores from the reviewers initially,  and the scores are further increased after rebuttal as the reviewers are satisfied with the authors' rebuttal. Considering the technical novelty and the consensus positive reviews, I would recommend the acceptance of this paper, while encouraging the authors to further improve the manuscript by providing rigorous analysis on theoretical justification of convergence and generalization bounds.
Decision: Accept (poster)