Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation
Authors: Zikai Zhang
Rui Hu
Jiahao Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Low-Rank Adaptation
Large Language Models
Resource Heterogeneity
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large Language Models (LLMs) have demonstrated remarkable effectiveness in adapting to downstream tasks through fine-tuning. Federated Learning (FL) extends this capability by enabling collaborative fine-tuning across distributed clients using Low-Rank Adaptation (LoRA), while preserving data privacy by avoiding raw data sharing. However, practical deployments face challenges when clients have heterogeneous resources and thus adopt different LoRA ranks, leading to substantial initialization and aggregation noise that undermines performance. To address these challenges, we propose Fed-PLoRA, a novel lightweight heterogeneous federated fine-tuning (FFT) framework. Fed-PLoRA introduces Parallel One-Rank Adaptation (PLoRA), a new LoRA variant that replaces the classic multi-rank LoRA module with multiple parallel one-rank modules, and a novel Select-N-Fold strategy that folds untrained PLoRA modules into the pre-trained weights before local training, thereby accommodating heterogeneous client resources. We provide a unified analysis of initialization and aggregation noise of Fed-PLoRA and demonstrate how it addresses the limitations of state-of-the-art methods. Extensive experiments on diverse LLM fine-tuning tasks demonstrate that Fed-PLoRA consistently outperforms existing methods in both accuracy and efficiency. The code is available at \url{https://github.com/TNI-playground/Fed-PLoRA}.

Published as a conference paper at ICLR 2026
HETEROGENEOUS FEDERATED FINE-TUNING WITH PARALLEL ONE-RANK ADAPTATION
Zikai Zhang Rui Hu Jiahao Xu University of Nevada, Reno, USA {zikaiz, ruihu, jiahacx)eunr:
ABSTRACT
Large Language Models (LLMs) have demonstrated remarkable effectiveness in adapting to downstream tasks through fine-tuning. Federated Learning (FL) extends this capability by enabling collaborative fine-tuning across distributed clients using Low-Rank Adaptation (LoRA), while preserving data privacy by avoiding raw data sharing. However, practical deployments face challenges when clients have heterogeneous resources and thus adopt different LoRA ranks, lead- ing to substantial initialization and aggregation noise that undermines perfor- mance. To address these challenges, we propose Fed-PLoRA, novel lightweight heterogeneous federated tinettnnig (FFT) framework. Fed-PLoRA introduces Parallel One-Rank Adaptation (PLoRA), new LoRA variant that replaces the classic multi-rank LoRA module with multiple parallel one-rank modules, and novel Select-N-Fold strategy that folds untrained PLoRA modules into the pre-trained weights before local training, thereby accommodating heterogeneous client resources. We provide a unified analysis of initialization and aggregation noise of Fed-PLoRA and demonstrate how it addresses the limitations of state- of-the-art methods. Extensive experiments on diverse LLM fine-tuning tasks demonstrate that Fed-PLoRA consistently outperforms existing methods in both accuracy and efficiency. The code available TNI- "NI-playground Ped-r
com/
INTRODUCTION
Federated Learning (FI.) has emerged as a crucial paradigm for fine-tuning Large I Language Models (LLMs) across multiple clients while preserving data privacy by avoiding raw data sharing. Low- Rank Adaptation (LoRA) Hu et al. (202I) is a widely used parameter-efficits method, which adapts LLMS by injecting trainable low-rank matrices into pretrained weight spaces. Specifically, LoRA factorizes a weight update matrix AW E d as AW = BA, B E RRdxr A E R. where r < min(d, k) denoting the rank. Combining FL and LoRA provides an ef- fective framework for collaborative LLM fine-tuning on downstream tasks WWag et al. (2018). A representative example FedrT /hang et al. (2023), which extends the classical Federated Averaging (FedAvg) McMahan et al. (2017) algorithm to the LoRA setting. In FedIT, each client i locally trains its LoRA modules (A,, B.) while keeping the pretrained backbone frozen. At the end of each round, the client uploads its LoRA matrices to the server. The server then aggregates them by weighted averaging (i.c., A = Σ W; and B = W; with weights w:). The resulting global LoRA modules are distributed back to all clients and serve the initialization for the next training round. This iterative process continues until convergence of the global adapted model.
fine-tuning (PEFT)
However, client resource heterogeneity poses major challenge for federated LoRA-based fine- tuning. In practice, clients differ significantly in computational capacity, which directly constrains the LoRA ranks they can afford train Wang et al. (2024); Bai al. (2024). Prior works Haobo et al. (2024): Ren et al. (2024) demonstrate that the LoRA rank critically influences both fine-tuning performance and resource consumption. Higher ranks generally enhance adaptation capacity, but at the cost of greater computational and communication overhead. As result, resource-constrain clients are often unable adopt the larger ranks that more capable clients can sustain. This im-
"Corresponding author.

---

Published as a conference paper at ICLR 2026
balance undermines global aggregation, leading to degraded model performance and limiting the effective participation of weaker clients. These challenges highlight the urgent need for methods that can accommodate heterogeneous LoRA ranks across clients, thereby enabling effective and inclusive federated fine-tuning of L.LMs.
Existing studies Wang al. (004): Cho et al. ) Bai al. ) Zhang et al. 2024b 2025
on federated fine-tuning (FFT) with heterogeneous LoRA ranks primarily address the challenge of aggregating local LoRA modules with varying dimensions. For example, FLoRA Wang et al. (2024) proposes a stacking-basee -ased aggregation scheme, where local LoRA modules are concatenated to construct global modules, thereby accommodating heterogeneous ranks during aggregation. Beyond aggregation, local initialization poses another key challenge in heterogeneous-rann FFT. Since the global aggregated module may have a different rank from that of clients, mismatches naturally arise. For instance, HETLoRA Cho et al. (2024) initializes local modules by truncating the global LoRA matrices from lower to higher rank indices according to each client's rank, whereas FL.RA resorts to random re-initialization in each round. Both strategies introduce substantial initialization noise, in contrast to homogeneous settings (e.g., FedIT), where local modules can be directly initialized with the global ones of identical rank.
In this paper, we provide unified analysis of initialization and aggregation noise across several rep- resentative methods for heterogeneous FFT. Building these insights, we propose novel heteroge- neous FFT framework, Fed-PLoRA, which incorporates Parallel One-Rank Adaptation (PLoRA). Unlike the classical LoRA approach that relies on single multi-rank matrix, PLoRA constructs modules of any desired rank by combining multiple parallel one-rank matrices. This design natu- rally supports client resource heterogeneity through a novel Select-N-Fold strategy. Our theoretical results show that Fed-PLoRA achieves near-optimal local initialization while minimizing aggrega- tion noise, thereby enabling effective and inclusive fine-tuning in heterogeneous federated settings. Our main contributions are summarized as follows:
We present a unified analysis of initialization and aggregation noise in heterogeneous FFT with varying LoRA ranks. Guided by this analysis, we propose Fed-PLoRA, a lightweight framework that naturally accommodates heterogeneous client capacities while ensuring sta- ble initialization and low aggregation noise. Fed-PLoRA is lightweight and can be seam- lessly integrated into existing LoRA and FL pipelines.
Fed-PLoRA incorporates PLoRA, new LoRA variant that substitutes a single multi-rank LoRA module with multiple parallel one-rank modules. Building on this design, we pro- pose the Select-N-Fold strategy, where each client trains only a subset of PLoRA modules according to its computational budget, while folding the remaining modules into the frozen pretrained weights.
Through extensive experiments on diverse LLM fine-tuning tasks, we demonstrate that Fed-PLoRA consistently outperforms existing heterogeneous LoRA-based FFT methods across varying client resource and data settings.
FEDERATED
SYSTEM
We consider an FFT system consisting of a central server and v clients. The server maintains a global transformer-base pre-trained LLM, denoted by O", while each client E [u] owns local dataset D,. For LoRA-based FFT, LoRA can be applied to pre-trained target module W e gax (c.g., query projection) within a transformer block e
LoRA constrains the update of the target module, AW, through low-rank factorization such that W + AW = W + BA, where A E R B E Radxr, with the rank r < min(d, k). In practice, LoRA is applied to L distinct target modules within O", denoted by w (W 0.11 T-r We define the corresponding collection of LoRA parameters 0 = {A, BJ with A ) M and B- {B') { and each pair {AL, B'} represents the low-rank matrices associated with the 1-th target module. Given this setup, the problem of FFT can be formulated as follows:
min L(0) := 2,
(1)
0
i=1
2

---

Published as a conference paper at ICLR 2026
Intormation
AdE ogation
Frozan
Truercateed information
SEIIVER
Trainable
Locally
Insistizzed
Locally
Replaceable
Start From
CLENN
Scratch
C
Intermaciate
[8] FLORA
(b) TexLLRA
Aggrogation
Interaction
Tern Nolse
Aggro Aogrogotinn
Asrrogation
One Rank Weight
Truncoted
Stocking
SERVER
Llement-Wise
X
Menrix
Fini
Mini
CUIETT
C
CUENTT
CUENT
(c)
HETLORA
(d) Fed- FroR-PLoRA, (Ours)
Figure 1: Framework of FLORA, FlexLoRA, HETLoRA, and method Fed-PLoRA.
where (0; @O) EzED, ( O .z]] denotes the local objective function of client i, and 1(; @0, x) is the loss of the model on a datapoint x sampled from D;.
In heterogeneous rank setting, each client employs LoRA modules with its own LoRA rank
To solve the optimization problem in Equation equation [ FFT methods Cho et al. (2024); Wang et al. (2024); Bai et al. (2024) typically follow three-step procedure in each training round [T]:
1. Broadcast and Initialization: The server broadcasts the global model to clients, and client i initializes its local LoRA modules with rank via 0-- Init (0'-1 ,r).
Local Client Update: Each client i then fine-tunes its initialized LoRA parameters 0--1 using its local dataset D., while keeping the pretrained backbone t frozen. This process typically involves multiple steps of Stochastic Gradient Descent (SGD) or its variants Kingma (2014), resulting in updated local LoRA parameters: 0 LocalUpdate (0)-1, Di, @²) The updated parameters 0 are then transmitted back to the server.
3. Server-Side Aggregation: The server collects the set of updated parameters (0) (1) and aggre- gates them form the global LoRA parameters 0f for round 0c {A', B') Agg(( heeu)) where A E Rxc and B Rd Here, the aggregation operator Agg(-) must be capable of combining local LoRA modules with heterogeneous ranks, and the resulting global rank R, which satisfies R 2 max(r:), is determined by the choice aggregation rule.
3 PARALLEL ONE-RANK ADAPTATION FOR HETEROGENEOUS FFT
In this section, we introduce Fed-PLoRA, a heterogeneous FFT framework that incorporates new LoRA variant (PLoRA) and a novel local initialization strategy (Select-N-Fold) to eliminate initial- ization noise and mitigate aggregation noise.
3.1 MOTIVATION
Existing LoRA-based heterogeneous FFT methods Cho et al. (2024); Bai et al. (2024): Wang et al.
20244 that follow the above training framework differ primarily in their initialization and aggrega- tion strategies. To analyze their effects, we formally define the notions of initialization noise and aggregation noise follows:
Initialization Noise: During the initialization step of round t, clients with limited local rank r; may be unable to fully accommodate the information contained in the global LoRA modules as r; < R.
3

---

Published as a conference paper at ICLR 2026
This mismatch introduces initialization noise, defined as the total initialization gap across all clients:
Wint =
E(1A=_'AA:11 B (Il)),
(2)
RE[D]
where ||-1| denotes the Frobenius norm, and the operator @ represents a rank-wise subtraction. This metric quantifies the information that constree-n clients fail retain when initializing from the global model. In homogeneous rank settings (i.e., = R for all i E [v]), the initialization noise vanishes since all clients can directly adopt the global LoRA modules.
Aggregation noise: After local training. the server receives updated parameters O from each client Following prior works Wang et al. (2024); [Sun al. (2024), we define an ideal model update AW", as the average of the local model updates, and the deviation from this ideal model update defines the aggregation noise, i.e.,
N Agg 1Aw - -MW with AW = - _ AW,
= Σ BAA,
(3)
V iEiv
xEfe]
where AW" B' A' represents the actual model update obtained by specific aggregation method. Ideally, perfect aggregation method yields Not Agg
0.
3.2 PARALLEL ONE-RANE ADAPTATION
Before presenting the Fed-PLoRA framework, we first introduce its fundamental building block, PLoRA. The key idea to replace a single multi-rank LoRA module with multiple parallel one- rank modules. Concretely, consider applying LoRA module with rank R to a target weight matrix W pd In classical LoRA, this is parameterized by matrices A E I Rxx and B E Rd contrast, PLoRA decomposes this rank-R module into
yielding the update AWLORA == BA. R parallel one-rank components. For each j [R]. PLoRA component consists of pair of one- rank matrices Ag) E IR. 1x and B,y) E Rdxx (see Figure Id), producing an individual update AW = B Ag) The overall update is simply the sum of these contributions:
R AWPLORA = Aww = Σ A j=1
R
R
= E BBJ = AWLoRA J=1
which is mathematically equivalent to the classic multi-rank LoRA formulation, since B(j) B(:jj] and A(j) = :]. Thus, PLoRA achieves the same adaptation effect and parameter ef- ficiency as standard multi-rank LoRA, while enabling modular decomposition that is naturally suited to heterogeneous FTT.
3.3 FED-PLORA: HETEROGENEOUS FFT WITH PLORA
As in existing approaches, resource-constrained clients can reduce their LoRA ranks to fit limited computational budgets, which naturally results in heterogeneous ranks across clients. Within our framework, this corresponds to adjusting the number of parallel one-rank matrix pairs in the PLoRA modules. However, this adjustment alone does not eliminate initialization and aggregation noise.
To overcome these issues in heterogeneous settings, we propose a novel Select-N-Fold strategy within the Fed-PLoRA framework. This strategy is specifically designed to handle rank hetero- geneity, ensuring zero initialization noise and minimizing aggregation noise. Assume the server maintains global PLoRA parameters = ([A') Re(L); (B') [e())), where for each target module I E [], A' : (A hie(R] and B' == {B', bie(M) represent the R parallel one-rank PLoRA components. For client the local PLoRA parameters
0, ([A(hej))) (B/h4 where A := (Al,) and B² == {Bi (ie,))) with local rank 7;- The global rank chosen such that Z max4e[") ri. Fed-PLoRA follows the three-step training framework described in Section and the useudo-code is shown in Algorithm In each training round t:
The server broadcasts the global PLoRA parameters f-- to clients. Each client i then randomly selects a subset K; of r; parallel one-rank PloRA modules for local
Broadcast and Initialization:
4

---

Published as a conference paper at ICLR 2026
training. For clarity, omit the index / in the notation, but note that this selection and the subse- quence operations are performed independently for each target module / E [L]. The local PLoRA --1
parameters of client are initialized as 0 = {) Bi Jee; The remaining (R (R-r.) parallel
one-rank PLoRA modules are temporarily folded into the corresponding pre-trained target module W yielding the local target module
W = W" AW = Wo
Σ JEKC
B (
fold,
which then remains frozen during local training. This procedure allows the client to preserve infor- mation from the full set of global PloRA modules while training only a subset. Consequently, client i operates with an effective rank of Tú, matching the parameter count of standard rank-r; LoRA configuration, but without incurring initialization noise.
Randommess in Select-N-Fole Our objective is to obtain a well-trained global PLoRA that can be applied the pretrained backbone for downstream tasks. When local resource sufficient (i.e., Rt), all local A modules can be trained, ensuring that every global PLoRA module is updated. When r; < R. trainable modules are selected independently across clients and target modules; hence, in expectation, each global module is updated by some subset of clients in every round. This randomness mitigates the risk of long-term staleness in global PLoRA modules under the Select-N-Fold strategy.
During local training, each client updates only the selected PLoRA modules in ICJ. After optimization, the updated local PLoRA parameters are 0 {A B which are then transmitted to the server.
Local Client Update:
Server-Side Aggregation: The server aggregates these local PLoRA modules in rank wise man- ner. For each [R], the global PLoRA modules are computed A ر Eieg, Al. 1
B
Eieg; B(.)) (.())* where L'y; = {ili E [u], E KC}} denotes the set of clients that updated module j in round t. The aggregated global PLoRA parameters 0 = {A', B' are then used for the next round training. The process itcrates until the global model converges.
3.4 ANALYSIS OF INITIALIZATION AND AGGREGATION NOISE
0.8
Here, analyze the initializa- tion and aggregation noise of Fed- PLoRA, comparing it with three SOTA methods, FLORA Wang et al. (2024), FlexLoRA Bai al. (2024), and HETLORA Cho el al. (2024). Due to page limitations, we pro- vide the detailed derivation in Ap- pendix E
10
10
0.0
-0.2
(6) 12
Index
Round
Fed-PLoRA. Because each client either trains or folds the globally initialized one- rank modules into its frozen target weights, there is no discrepancy between local and global modules. Hence, N
Cosine similarities between parallel one-rank modules of PLoRA. (a) Low, random similarities at initial- ization; (b) Increasing within-rank similarity across clients, while cross-rank similarity remains low, indicating that modules capture distinct knowledge at different ranks but converge similar knowledge across clients, despite dif- ferences in resource limitations and data distributions (see Appendix Section C.3
Figure 2:
0. Under our aggregation rule, the aggrega- tion noise is Nous = || Eieg (B - (O) - A))
where ( = (1/10) Ei AL. and B = (1/10) Eiso; Blu This noise vanishes when the cross-client covariance between Ai,0) and Bl.0 is zero for every rank. More generally, by the Cauchy-Schwarz inequality, it can be bounded as Nours S E 10; EiEr; |B BBD)lz + (|A 122 As shown in the
5

---

Published as a conference paper at ICLR 2026
heatmap in Figure the diagonal entries represent the similarity of the j-th PLoRA module (j E [R] across clients, averaged over all client pairs. We observe that these similarities increase significantly as training progresses, indicating that local PLoRA modules become more aligned across clients. This alignment reduces deviations from the mean, thereby tightening the upper bound of the aggregation noise.
FLoRA. As shown in Figure I (a), FLoRA employs a stacking- slackig-bassed aggregation method to accom- modate LoRA modules from clients with heterogeneous ranks, Each client initializes its trainable LoRA modules {A} BB } with rank r; from scratch (e.g.. A (- drawn from a normal distribu- tion and B set to zeros). Before local training, the local target module is replaced with the latest global target module from the server. After local training, the updated LoRA modules (A B are sent to the server, which concatenates all client updates along the rank dimension to form two large intermediate matrices: {A' R Zecto) x2) B' E RRd (Eico
"*) The global update is then com-
puted as AW FLORA B' //A//v. Because local LoRA modules (A B are freshly initialized each round, the initialization noise is FL Init Exe(s) ||A--1|] --11 + [1()) + o², where o² reflects the variance of the random initialization for A. This noise can be substantial and may degrade local training performance. On the other hand, the stacking operation in FLoRA preserves 0.
the local updates without distortion, leading to zero aggregation noise, i.c., VI Agg
FleXLoRA, As shown in Figure (b). FlexLoRA applies singular value decomposition (SVD) to the global target module update Aw" to construct global LoRA modules (A'-1,''-)) with rank R. i.e., AW x svd svd
svd = V Here, U'-1 E Rddxit contains the top-It left singular vectors, S E Rxx is the diagonal matrix of singular values, and Vt-1 E Rgi holds the top-f right singular vectors. For client with rank ri. the local LoRA modules are initialized by truncating to the top r; components: A t-1
--1 where syd == U and A f-1
VIT. [Ir..]] . = U [..r..) [+r::]] After local fine-tuning, the server aggregates client updates to form the global target module update: AW" + -iE(()) AW; - Eiein B Because each client only retains the top-r, singular components, the initialization noise is -Viec = (le t... 1I + ||B[-] (1.1-1:81] [)) which increases as r, decreases. The server obtains the exact average update AW", but when reconstructing global LoRA modules of rank R via SVD, decomposition error is introduced: -V AAW'' U'S'v [[.. This error increases as global rank R increases.
HETLORA. As shown in Figure (c). HETLORA initializes each client's LoRA modules by trun- cating the global LoRA modules, i.e., A == A t-1 Brr.d B B 1..r.1 [[ After local fine-tuning, the updated modules { BJ} are sent to the server, where they are expanded to rank by zero- padding. The padded modules A t. B'd are then averaged to form the global LoRA modules: A' = 1 ExE(s) A, B' = ! Eie(n) ponents, the initialization noise A/ METLORAAIIi = -EET-) ( (r. |I
B Because each client discards the bottom (R-- - r:) com- + f ||B[-1
[.,,+1.R| [|r), which
grows as r; decreases. For aggregation, the global update is computed as AW B' but because A and B' are averaged separately, this introduces a mathematical bias relative to the opti- mal update 1 >, A Wang et al. (024); Sun et al. (2024). The resulting aggregation noise is NiiETTOAA
[1/ ((v 1) Eee B A
ΣSE(M] Exe(s), A
In summary, FLoRA suffers from substantial initialization noise because each client's LoRA mod- ules are randomly re-initialized. FlexLoRA and HETLoRA reduce this randomness by truncating global modules. but still incur initialization noise that grows more clients operate with smaller ranks. Moreover, HETLORA introduces structural aggregation bias, while Foe suffers de- composition error from SVD. In contrast, Fed-PLoRA eliminates initialization noise entirely and minimizes aggregation noise by integrating PLORA with the Select-N-Fold strategy, leading to more stable and accurate FFT under heterogeneous client capacities.
3.5 COMMUNICATION COMPUTATION, AND MEMORY OVERHEAD
We analyze the communication, computation, and memory overhead of Fed-PLoRA in comparison with FLORA, LORA and HETLORA.

---

Published as a conference paper at ICLR 2026
Communication. In each round, every client in Fed-PLoRA uploads its local PLoRA update with rank r; to the server. The uplink payload scales as D((d+A)r) which is identical to the uplink cost FLoRA, FlexLoRA, and HETLORA. Thus, Fed-PLoRA introduces additional uplink overhead. After aggregation, the server sends the global PL.RA module with rank R to every client, giving downlink cost of ((dd R). In comparison, HETLoRA and Fle send each client personalized global LoRA module of rank ri, resulting in per-client downlink cost O((d+ + k)r:). FLoRA incurs larger downlink cost of (dk) as its server sends the updated target module. Thus, the downlink cost of Fed-PLoRA is higher than HETLoRA and FlexLoRA by O(d k) r;)) per client, but it reduces the downlink cost relative FLoRA by O(dk- (d + R)R)
Computation. All methods use an identical local tine-tuning procedure with local rank for client i after initialization, so Fed-PLoRA incurs no additional model training cost on the client side. The only difference in local computation arises during model initialization, whose cost is negligi- ble compared with model training cost. For completeness. we also discuss the initialization cost of Fed-PLoRA compared with other methods. During initialization, all methods update their local PLoRA/LORA parameters, either by using the received global PLoRA/LORA module or by ran- domly initializing them. Compared to FlexLoRA and HETLORA, Fed-PLoRA and FLORA require an additional step. In Fed-PLoRA, client i folds the remaining R - r; global one-rank PLoRA mod- ules into the frozen model weights, which incurs extra computational cost of Odd (R rz)). In FL,RA, every client updates its local target module using the received global target module, resulting in an additional computational cost of (dk)). On the server side, Fed-PLoRA performs rank-wise averaging directly on the received local updates, which is lightweight. In HETLORA, the server first expands each local update to rank R before averaging, introducing slightly more computation than simple rank-wise averaging. FlexLoRA requires an SVD operation to obtain a global low-rank update, which computationally expensive. FLoRA concatenates all local updates and computes full update to the global target module, which also incurs substantial cost. Thus, Fed-PLoRA introduces no additional server-side computational overhead compared with the other methods and is in fact the most lightweight among them.
During local model training. Fed-PLoRA has the same memory footprint for storing model parameters, optimizer states, activations, and other training-related tensors as HETLORA, FLORA, and FOA.LoRA. This is because all methods follow the same local fine-tuning procedure to update only the LORA/PLORA module top of the frozen backbone. The only difference in memory usage arises during model initialization, whose cost is negligible compared to the overall fine-tuning memory footprint. During local initialization, each client requires small temporary memory buffer to hold the parameters received from the server. These parameters are immediately discarded once initialization completes, resulting in no persistent memory cost. Specifically, in Fed-PLoRA, client i temporarily stores the global PLoRA module of rank R. In HETLoRA and Foe, client instead stores the global LoRA module of rank r,, In contrast, FLORA requires client i to temporarily store the full global target module, whose size scales as (dd)). Thus, Fed- PLoRA incurs a temporary memory overhead of O(d (R compared with HETLoRA and Fle,LoRA, but reduces temporary memory usage relative to FLORA by O - (d + k)).
Memory.
Overall, Fed-PLoRA introduce negligible overhead communication, computation, and memory. Detailed numerical results and measurements are provided in Appendix F2
4 EVALUATION
Models, Datasets, Baselines, and Experimental Settings. We employ six models with different scales in our experiments: BERT-baseDDevin et al. (2019). Llama- IB Chen al. (2024). Llama-3.1
8B Meta AT (224)) OPT-1.3B Zhang et al. (2022a). Qwen3-4B-A3 A3 2507 Team (025)
and Mistral-7B-vdl v-3 MistralAl (203). Following the configurations in the original LoRA paper Hu
et al. (2021). the LoRA modules are applied to the self-attention layers only. We use datasets across
multiple domains. For general instruction following, we use Natural Instructions Wang et al. (2022)
and Dolly-15K| [Conover et al. (2023) for training and evaluation. For general natural language understanding, we adopt the GLUE benchmark Wang et al. (2018). 8). In the finance domain, we train on FinGPT FinGPT 2023) and evaluate FPB Malo al. (2014). FIQA Maia et al. (2018). and
TFNS Neural Magic (2022). In the medical domain, we train on MedAlpaca Han et al. (2023 and
evaluate on PubMedQA Jin (2019), MedMCQA Pal et al. (2022), MedQAJin et al.
2021 and
7

---

Published as a conference paper at ICLR 2026
Method
Natural Instructions (IID)
Natural Instructions (non-IID)
Untuned Model
33.8220.12
Homogeneous
FedIT RRank=R/2)
FedIT (Rank=R)
-6.52-
06.88 10.00
64.52 10.11
0:30
61.28 10:67
60.57 -0.21
0.41
FLoRA
33.82
33. 13..22"5
Heterogeneous
FlexLoRA
59.07 2.665
53.51 ++5.27
R Avg Rank=8
HETLORA
58.07 1:45
58.81 81
Fed-PLoRA
64.96+1.65
60.76 -0.46
Table 1: Comparison of Fed-PLoRA with baselines on IID and non-III Natural Instructions dataset. FedIT as the baseline for FFT under homogeneous settings without resource constraints.
Method
CoLA
SST-2
MRPC
QQP
QNLI
RTE
Avg
Untuned Model
1.20
10.10
0.00
49.08 0:43
31.61
10.85
0.80
63.09 1.25
0:57 49.95
0.52
52 70' 10.95
41.27
0.45
FedIT (Rank-R)
61.68*
92.47
87.581044
86.861
89.53
0.3
68.35 19.34
81.08
0.50
Homo.
FedIT (Rank=R/2))
59:05 12:27
92.23 8.05
86.47 1.02
89.32 1.08
65.23
70.59
0:30
FLaRA
4.08 01
91.70
43.877133 51
53.88 6.79
50.47
R 16 Avg Rank=7
FlexLoRA
13. 13.12f
87.88
57.88-1:11
70. 2.00
T4. T1.299333)
90.01
-0.43
$5.8322..0
HETLORA
48 09 1.04
91.74 0:25
78.59 3:70
77.
85.30 0.20
60. 60:04
1200
73.55
1.50
Fed-PLoRA
59.38 00.78
92.35 0.15
86.35 0.57
87.25 21.25"1..
1.40
88.54 1.74
65.46 0..8
79.89 0.57
Table 2: Comparison of Fed-PLoRA with baselines on GLUE benchmark.
CareQA Arias-Duart al. (2025). In the math domain, we use MATH Hendrryks et 202 for training and evaluation. For IID data settings, we evenly split the dataset across clients. For non-IID data partitioning methods.
settings, we apply pathological/Dirichhe
We compare Fed-PLoRA against four baselines. FedIT Zhang et al. (2023) represents a classic FFT approach that combines FedAvg with LoRA. Since designed for homogeneous LoRA, we only apply it in homogeneous experiments, where it serves the baseline representing FFT without resource constraints. For heterogeneous LoRA, we consider FLORA Wang al.] (0024) FlexLoRA Bai et al. (2024), and HETLoRA Cho et al. (2024) (see Section 3.4 for their details). For the heterogencous setting, clients are (by default) evenly divided into three groups with ranks (1, Rt). where the maximum R is set as the optimal rank in homogeneous setting, and the middle rank mm E (1, R) is chosen so that the average rank r/// is at most R/2, reflecting realistic resource-limited clients. The number of clients across tasks ranges from 50 200. In each train- ing round, 10% of clients are sampled uniformly at random. The main experimental results are reported the average with upper and lower deviations over three repeat experiments. Additional experimental configurations are provided in Appendix
4.1 MAIN EXPERIMENTAL RESULTS
Due to page limitations, we present the main results on Natural Instructions (IID and non-IID) with Llama- IB, the GLUE benchmark (IID) with BERT-base. and financial datasets (IID) with Llama- 3.1-8B, while leaving the remaining results other datasets and non-IlI) settings in Appendix [
Results on Natural Instructions. Table I reports averaged Rouge-L scores of the fine-tuned global model under both IID and non-lID settings (the latter simulated by assigning 20 out of 613 distinct tasks per client). The FedIT results under homogeneous settings show that a larger LoRA rank leads better fine-tuning performance, improving over the untuned model by +33.06% under the IID setting and by +27 .466 under the non-lID setting on average. Fed-PLoRA consistently outperforms heterogeneous baselines. the IID setting. achieves average Rouge-L gains of a31.14444 +5.89%, and +6.89% over FLORA. FleXLoRA, and HETLoRA, respectively, highlighting the effectiveness in eliminating initialization noise and reducing aggregation noise. We also observe that Fed- PoRA slightly outperforms the homogeneous FedIT with rank R/ 2 by +0.19%. Similar trends appear on other datasets, and some cases Fed-PLoRA even surpasses FedIT with rank R. These effects are often more pronounced under non-lID settings. This suggests that Fed-PLoRA's parallel and randomized module updates ensure that all R global PLoRA modules are updated by different clients across rounds. As result, rank-wise updates stay aligned across clients, allowing each one-rank module to gradually learn consistent features even under data heterogeneity, consistent with our observations in Figure
8

---

Published as a conference paper at ICLR 2026
Method
FPB
FIQA
TENS
Avg
Untuned Model
50.57_6:31
0:42
29.33.0.44
1051
49.87 1:22
40. 40.12_0 00.00
FedIT (R
63.532635 20
28. 28.9411.2 3.33
64.1120.03
64.11
22.1916
Homogeneous
FedIT (Rank=f/2))
62.70
29.93
2.33
63.71 0-44
52.11 0.778
FLORA
52.33 0.08
31.42"0:24
0:24
50.53+0:48
0:40
44.76 0.26
Hederogencous
FoeXLoRA
62.79.0.74 0:54
31. 7617666000 0.90
62.60* 1,30
L.05
52.3811 +0.00
R Avg Ranke4
HETLORA
60.06 0:55
32.01 +2.52
60.09
t..59
50. 000.7210:5
50.72-0.92
Fed-PLoRA
63.9411.18
1.18
31.68190x
64.19
53.27 2--37
-0.85
Table 3: Comparison of Fed-PLoRA with baselines on financial datasets.
Results on GLUE As shown in Table 2 our method, Fed-PLoRA, demonstrates substantial im- provements over these baselines on IID GLUE benchmark. Fed-PLoRA improves untuned model by +38.6299 on average. Compared to FLORA, Fed-PLoRA achieves average improvements of +63.46% CoLA and +42.48% on MRPC. This significant margin shows the detrimental impact of FLoRA's random initialization, which incurs significant perturbation to the local training at every round. Compared to FlexLoRA, Fed-PLoRA achieves notable average improvement of +4.47% on the SST-2 dataset. This suggests that while FlexLoRA aims to provide representative low-rank matrices with SVD, can still suffer from significant information loss that leads to initialization and aggregation noise, particularly when some clients are constrained to very small LoRA ranks. Fed-PLoRA also outperforms HETLORA by +9.72% on QQP and +5.42 +5.42% on RTE. This advantage can be attributed to the large initialization and aggregation noises in HETLoRA A from zero-padding and truncation, which PLoRA is designed to mitigate more effectively. Note that FlcxLoRA outperforms FedIT with rank R on QNLI, likely because this relatively simple binary QA task can be well captured using a low rank, making SVD-based aggregation particularly effective.
Results Financial Datasets. As shown in Table 3 Fed-PLoRA achieves average gains of +13.15% over untuned model, +8.51% over FLoRA, +0.89% over FlexLoRA, and +2.55% over HETLORA. Additionally, both A and FlexLoRA here outperforms the homogeneous FedIT baselines. As we discussed before, Fed-PLoRA A achieves this by maintaining rank-wise align- ment of local module updates across clients. FlexLoRA benefits from its SVD-based initialization, which aligns low-rank modules with the most informative global directions. When the task is inher- ently low-rank, this targeted representation may outperform homogeneous FedIT, which treats all directions equally.
4.2 ADDITIONAL DISCUSSIONS
Adaptability of the PLORA to Homogeneous Settings
and
LoRA
Variants.
Theoretically, PLoRA achieves the same adaptation effect and parameter efficiency as standard multi-rank LoRA (see Section 3.2 and can be extended to other LoRA variants. We adapt it to rsLoRA Kalajdzievski (2023). forming Fed- PrsLoRA, and compare with FedIT and another homogeneous method FFA-LOR A et al. (2024) (which only shares B ma- trices). We evaluate on IID CoLA dataset, the hardest GLUE task, under homogeneous settings with different ranks. From Figure 3 we observe that Pro-PAsL consistently outper- forms Fed-PLoRA, leveraging rsLoRA's improvements over LoRA. Both methods also achieve substantial gains over FFA- LoRA (e.g.. +10.63% and +10.89% rank 16) and in some cases surpass FedIT. This advantage becomes more pronounced as the rank decreases, underscoring the effectiveness of PLORA's paral- lelization resotrd settings.
Ped-Prsanki
fash-11 18
Figure Performance of PLoRA and Fed-PrsLoRA in homoge- neous settings.
Weight Norm 633 Fixed
Select-N-Drop
We also evaluate other strate- gies for selecting trainable one-rank PLoRA modules in Fed- PLORA on IID CoLA dataset. We consider: Weight Norm, which picks modules with the largest weight norms; Fixed, which always selects the first 7; modules; and Seroct-N-Drop. which randomly selects r; modules like Select-N-Fol but dis-
Effectiveness of Select-N-Fold.
COLA Witttree wittentss Comm
Select-Ni--C Notect-N--oodd...
cards the rest. As shown in Figure 4. both random strategies
Figure 4: Select-N-Fold vs. other selection methods.

---

Published as a conference paper at ICLR 2026
(Select-N-Drew and Select-N-Fold) outperform the deterministic ones, since randomness ensures that all global PLoRA modules are eventually updated. Select-N-Fold achieves the highest perfor- mance by additionally reusing the latest unselected modules during local training.
Communication and Computational Efficiency. Figure 5 shows the accuracy of four heteroge- neous methods the IID QQP dataset with respect to communication (uplink and downlink) cost and computation (wall-clock training time) cost. Overall, Fed-PLoRA achieves higher communica- tion efficiency although it incurs additional
r; downlink cost compared others. Both Fed- PLoRA and HETLoRA achieve good computational efficiency as they are lightweight. FlexLoRA incurs substantial computational overhead due to the use of SVD, nearly doubling the training time. A detailed overhead analysis is given in Appendix Section F2
5 RELATED WORK
LoRA-based FFT has recently emerged as popular paradigm for adapting I.LMs while pre- serving data privacy /Zhang et al.] (2023); [Babakniya et al.] (2023); [Sun et al. (2024); 7a- wad et al. (2025). Early approaches such as FedIT Zhang et al. (2023) and FFA-LoRA Sun al. (2024) combine LoRA with FedAvg modify update sharing to mitigate aggre- gation noise, but they assume homogeneous client resources. Other work like FedSA-
LoRA Guo et al. (2024) explores selective aggregation, y. To address resource heterogeneity, several methods allow clients to fine-tune with different LoRA ranks. HET- LoRA Cho et al. (2024) aggregates heterogeneous updates through zero-padding and truncation. FLoRA Wang et al.
Accuracy (MDD)
0.7
(204)) stacks full-weight updates before reinitialization, and FlexLoRA Bai al. (202)) applies SVD-based ag- gregation with truncation. While effective to some extent, these approaches introduce substantial initialization or ag- gregation noise, degrading global performance.
0.6
FLORA
FlettRA
HETLORA
0.5
Ours
50
100
150
200
250
300
Communication Cost (MB)
FloRA
The use of multiple parallel LoRA modules has been ex- plored, for enhancing model capacity in centralized train- (Habbo ing, c.g., Capaboost Haobo et al. (2024) and MELoRA Ren
Accuracy [IID)
0.8
FiceLORA
HETLRRA
0.7-
eet al. (2024) rather than addressing the unique challenges of federated settings. In contrast, our work designs PLoRA mechanism to mitigate initialization and aggregation noise under heterogeneous FFT.
0.0
2.5
15.0
Training Time (Hours)
Finally, other efficiency-orrinted techniques such as quan- tization |Frnntar et al.) [2022); Lin et al. (2024), zeroth-order optimization Xu al. (2024); Zhang al. (2024a). and system-level methods like activation checkpointing Chen et al. (0016) provide complementary di- rections for resource-efficient training. These are orthogonal to our design and could be combined with Fed-PLoRe in future work. Due to the limited space, we provide comprehensive literature review in Appendix Section
Figure Training efficiency of Fed- PLoRA.
CONCLUSION
In this paper, we introduced Fed-PLoRA, novel heterogeneous FFT framework designed tackle the fundamental challenges of initialization and aggregation noise in LoRA-based fine-tuning. By leveraging PLoRA's parallel one-rank modules together with the Select-N-Fole strategy, Fed- PLoRA aligns client updates more effectively and preserves consistency under resource heterogene- ity. We conducted a unified analysis of initialization and aggregation noise of our method, comparing with state-of-thh-ar heterogeneous LoRA- Abbsed FFT methods. Through extensive experiments on multiple tasks, we justified the effectiveness of PLoRA and Select-N-Fold and showed that Fed- PLoRA consistently outperforms existing state-of-the-ar methods in both accuracy and efficiency. PLoRA's parallelization design opens new opportunities for integration with other LoRA-like meth- ods, potentially extending its benefits beyond the current framework. In the future, will work on further reducing the noise that arises during the aggregation low-rank adaptations and investigat- ing how parallelization can be leveraged better align model updates under data heterogeneity.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation under the Harnessing the Data Revo- lution for Nevada Fire Science (HDRFS) Seed Grant NSHE-24-37.
REFERENCES
Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urce- lay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, and Dario Surcia-Gasulle Automatic evaluation of healthcare Ilms beyond question-answering arXiv preprint arXiv:2502.06666. 2025.
Sara Babakniya, Ahmed Roushdy Elkordy, Yabya H Ezzeldin, et al. Slora: Federated parameter efficient fine-tuning of language models. In International Workshop in Conjunction with NeurIPS 2023, 2023.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, et al. Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International Conference on Management of Data, pp. 120-134, 2024.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604.66/4444 2016.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for fed- erated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12903-12913, 2024.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 1
2018
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 1lm. Company Blog of Databricks, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088-110 11100000011111111111111111111111 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, et Bert: Pre-training deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the NAACL: Human Language Technologies, Volume 1, 2019.
FinGPT. fingpt-sentiment-train.
senprtt--stiimett-taaaae 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732.. 2022.
Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri, Chong Shen Ng, Dimitris Stripelis, Zexi Li, Tao Shen, Jiamu Bai, Daoyuan Chen, et al. Flowertune: A cross-domain benchmark for federated fine-tuning of large language models. arXiv preprint arXiv:2500602961. 2025.
Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu. Selec- tive aggregation for low-rank adaptation in federated learning. arXiv preprint 44X1v:2410001463, 2024.
11

---

Published as a conference paper at ICLR 2026
Tianyu Han, Lisa C Adams, Cens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca an open-source collection of medical conversational ai models and training data. arXiv preprint arXhv:2304.082477 2023.
SONG Haobo, Hao Zhao, Soumajit Majumder, and Tao Lin. Increasing model capacity for free: A simple strategy for parameter efficient fine-tuning. In The Twelfth International Conference on Learning Representations, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- case does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint 1rXiv:1909.06/ 2019.
Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. arXiv preprint
arXiv:2312.03732, 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- weight quantization for on-device IIm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www' 18 open challenge: financial opinion mining and question answer- ing. In Companion proceedings of the the web conference 2018, pp. 1941-1943 2018
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Informa- tion Science and Technology, 65(4):782-796 2014.
Brendan McMahan, Eider Moore, Daniel Ramage, et al. Communication-efficici learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 2017.
Meta AI. Introducing Ilama 3.1: Our most capable models to date, July 2024.
URL https:
//ai.meta. www.met...comm www.c//oooo......ooooooooooooooo www.tta..o//oooometttaaaaaaaaaaa
MistralAI. Mistral-7b-v0.33
3. 2023.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-aware training. In International Conference on Machine Learning, pp. 16318-16330. PMLR, 2022.
Neural Magic. Twitter financial news sentiment dataset. https::/hugging co/
financial neeansiilllewsss.. sell-newws-ettt--- iment 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248-2601 PMLR, 2022.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, and Jiahuan Pei. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine- tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pp. 3052-3064, 2024.
12

---

Published as a conference paper at ICLR 2026
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preservee federated learning. In The Twelfth International Conference on Learning Representations, 2024. URL
net/ forummid-NLL216::..
Qwen Team. Qwen3 technical report, 2025. URL
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1XX4:1804.0076/ 2018.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Generalization via declarative instructions on 1600+ nip tasks. Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 5085-5109, 2022.
huper-natuuralinttuctiio
Ziyao Wang. Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In The eiiht Annual Conference on Neural Information Processing Systems, 2024.
Hongda Wu and Ping Wang. Node selection toward faster convergence for federated learning on non-iid data. JEEE Transactions Network Science and Engineering, 955):099--31 -1199,1-111111111111111111111111 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099, PMLR, 2023.
Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, and Shangguang Wang. Fwdllm: Efficient federated finetuning of large language models with perturbed inferences. In USENIX ATC, 2024.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6137-6147, 2024.
Syed Zawad, Xiaolong Ma, Jun Yi, Cheng Li, Minjia Zhang, Lei Yang, Feng Yan, and Yuxiong He. Fedcust: Offloading hyperparameter customization for federated learning. Performance Evaluation, 167: 102450, 2025.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, et al. Towards building the federatedgpt: Federated instruction tuning. In International Workshop in Conjunction with NeurIPS 2023, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01006 2022a.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient IIm finettunig: A benchmark. arXiv preprint arXiv:2402.1 11592, 2024a.
Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Ron- grong Ji. Learning best combination for efficient n: M sparsity. Advances in Neural Information Processing Systems, 35:941 9513, 2022b.
Zikai Zhang, Rui Hu, Ping Liu, and Jiahao Xu. Fed-pilot: Optimizing lora allocation for efficient federated fine-tuning with heterogeneous clients. arXiv preprint arXiv:2410.10200 2024b.
Zikai Zhang, Suman Rath, Jiahao Xu, and Tingsong Xiao. Federated learning for smart grid: A sur- vey on applications and potential vulnerabilities. ACM Transactions on Cyber-Physical Systems, 2024c.
Zikai Zhang, Ping Liu, Jiahao Xu, and Rui Hu. Fed-hello: Efficient federated foundation model fine- tuning with heterogeneous lora allocation. IEEE Transactions on Neural Networks and Learning Systems, 1756-177777 2025. doi: 10.
13

---

Published as a conference paper at ICLR 2026
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. ICLR, 2021.
Shenglong Zhou al. arXiv:2208. / /1231, 2022.
Exact penalty method for federated learning.
arXiv preprint
14

---

Published as a conference paper at ICLR 2026
APPENDIX
This appendix presents additional discussions and experimental details to support the main text. The content is organized follows:
Appendix Section code snippet illustrating how our method can be easily integrated into existing frameworks, as well as pseudo-code of Fed-PLoRA.
Appendix Section Extended literature review on related work.
Appendix Section Detailed description of experimental settings.
Subsection
Dataset and model descriptions.
Subsection
Settings for the main experiments, including FL configuration and
infrastructure.
Subsection C.3
Settings for the experiments in Figure and Figure 2
Appendix Section Additional experimental results.
Subsection
D.1
Results on the GLUE benchmark under the non-lID setting.
Subsection
D.2
Results on the Dolly-15K dataset under both IID and non-IID set-
tings.
Subsection
D.3
Results on the Medical Question Answering datasets.
Subsection
D.4
Results on the MATH reasoning datasets.
Subsection
D.5
Ablation studies.
* Subsection D.5.1
Effectiveness of different selection methods for PLoRA mod-
ules.
Subsection D.5.2
Empirical observations on initialization and aggregation noise.
Subsection D.5.3
Impact of the number of clients, heterogeneity ratio, and local
rank per client.
Subsection
D.5
Effects different PLoRA ranks.
Subsection
D.5.5
Effects of Dropout.
Subsection
D.5.6
Additional visualizations of cosine similarity across PLoRA
modules.
Appendix Section Analysis of initialization and aggregation noise existing methods.
Subsection
Derivations for FLoRA.
Subsection
Derivations for FlexLoRA.
Subsection
Derivations for HETLoRA.
Subsection
E.4
Detailed derivations of the aggregation noise for Fed-PLoRA.
Appendix Section
Additional Studis.
Subsection Demonstration of the efficiency of the rank-based method on resource- constrained devices.
Subsection Theoretical and numerical analysis of the resource overhead for exist- ing methods and Fed-PLoRA.
USE OF LLM STATEMENT
For the preparation of this manuscript, Al assistants are utilized to aid in checking grammar, spelling, and punctuation.
A CODE SNIPPET AND PSEUDO-CODE
$ Implementation
Just line of code apply Parallel One-Rank IoRA in PyTorch. model AutoModdelRorCausalIMM, from pretrained (model_nanc) name)
15

---

Published as a conference paper at ICLR 2026
con config = LoraConfig (r=8,-.)) contig PLoraConfig (r=1, NunModule=8,... get peft_ (model, config)
We implement Fed-PLoRA using the Huggingface-stye LoRA initialization API (as shown above), making it easy to use by replacing just single line in the original code.
To better illustrate the proposed method, we present the pseudo-code of Fed-PLoRA as follows.
Algorithm 1 Fed-PLoRA
Require: Number of communication rounds T, global rank R, local ranks (rihele)) pre-trained backbone @ initial global PLoRA parameters 0
1: Server sends 6 to all clients
2: for t 1 to T do
3: Server samples set of clients S S [u]
4: Server sends global PLoRA parameters 0 to St
5: for each client i E St in parallel do
6: Randomly sample a subset KC; of size r, from index set [R]
7:
Initialize local trainable PLoRA modules: 0 + {A 1-1 B
(J) (] bee
8:
Fold unselected modules into the frozen target weights: W + W +EzeKt (T) 14-1 to obtain the local frozen backbone θ Obtain new local PLoRA parameters: 0 < LocalUpdate(0) D;, O end for
9:
11: Rank-wise aggregation of PLoRA modules: 0f + Agg((O))ies) (Ki)es))
12: Update global model: $
{00, 0*}
13:
end for
14: return T. 9
1 0T}
B COMPREHENSIVE LITERATURE REVIEWS
Federated LoRA- Fine-tuning. The application of FL to LoRA-based fine-tuning of LLMs has become prominent paradigm, enabling multiple clients to collaboratively adapt models with-
out sharing their raw data Zhang et al. (0023); [aaaknniy et al. (2023); Sun et al. (2024); Zhang jet al. (2024c). Several prior works have investigated this combination: FedIT Zhang al. (2023) introduces naive federated fine-tuning method with LoRA and FedAvg, serving
effective benchmark for further research. FFA-LORA Sun et al. (2024) introduces a strategy of freezing the LoRA A matrix during local training help mitigate aggregation noise at the server. Furthermore, FedSA-LoRA Guo et al. (2024) explores selective aggregation for a personalized FL. system, which designed only share the A matrix with the server for aggregation, due its specific role in learning general knowledge. However, these methods assume homogeneous client resources, which often do not align with the varied resources present in practical FL systems.
Federated Fine- Tuning with Heterogeneous LoRA Ranks. Resource heterogeneity remains a significant challenge in federated fine-tuning. To address this, HETLoRA Cho al. (2024) allows clients to train with different LoRA ranks based on their capacity. For aggregation, local model modules from smaller ranks are pero-padded, and the resulting global LoRA module is subsequently truncated for client LoRA module initialization. However, this process introduces both initialization and aggregation noise due to the limited LoRA rank. FLoRA Wang et al. (2024) employs a stacking- based aggregation method, which is designed to accurately aggregate LoRA module updates from clients with heterogeneous ranks FLoRA then applies this update to the client's 's target module, and the trainable LoRA modules for the next round are typically initialized from scratch, thereby incurring significant initialization noise. LoRA Bai et al. 2024) utilizes SVD during aggregation, and then initializes LoRA modules for
full-weight space. While this aims for optimal aggregation,
16

---

Published as a conference paper at ICLR 2026
clients with varying ranks by truncation. These processes lead to both aggregation and initialization noise.
Parallel Low-Rank Adaptation and Sparse LoRA Fine-Tuning. The concept of employing mul- tiple parallel LoRA modules has been investigated primarily for enhancing model capacity. For in- stance, Capaboost (Haobo al. (2024) applies different random masks single large rank LoRA module during training, effectively simulating an ensemble of parallel LoRA modules to increase model capacity without incurring additional parameter costs. MELORA RRnn et al. (2024) adopts a group of mini LoRA modules to obtain sparse A and B matrices. They claim that each LoRA module is designed to capture different features. These approaches leverage the parallelism of the LoRA module for capacity scaling. In contrast, our framework introduces parallel one-rank LoRA modules specifically as a way to improve initialization and aggregation within the Fl. system.
Another related line of work Zhou et al. (202T): : Zhang et al. (2022b) studies sparse LoRA fine- tuning, where only subset LoRA rows is updated while the remaining rows are simply frozen. In principle, such sparsification yields similar training dynamics on the active rows, since the inac- tive rows receive zero gradients. However, this design has several system-level limitations. First, the frozen rows must still be stored in GPU memory and participate in the forward pass, which in- creases activation storage and peak memory consumption compared with physically removing these rows. Second, the resulting sparsity pattern is typically unstructured: it does not match the N M structured sparsity patterns required by modern accelerators (for example, the 2:4 pattern used by NVIDIA A100 sparse tensor cores Zhou et al.] (2021)). As result, simply freezing R - r; out of R rows cannot exploit hardware sparsity support or lead to real speedups. In contrast, Fed-PLoRA folds unused one-rank modules into the backbone weights, preserving the same effective training behavior on active ranks while avoiding extra computation and memory overhead, which makes it more suitable for large-scale federated deployment.
Other Resourve-Efficient Tanne-Tuinggaa Pathways. Quantization is a widely studied technique for improving inference efficiency by reducing the precision of model weights and activations. Common methods include post-training quantization (PTQ). quantization- aware training (QAT), and activa- tion quantization. PTQ Frantar et al.] (2022); x Lin al. (2024): Xiao et al.] (2023) applies quan- tization to a pretrained model without modifying its training process. It is simple and efficient but may lead to noticeable accuracy degradation, especially for low-bit quantization or in sensitive tasks. QAT Nagel et al. (2022) is particularly focused preserving inference-time accuracy by simulating low-precision operations (e.g., 8-bit or 4-bit) during training through the insertion of fake quantization nodes in the computation graph, allowing the model to adapt to quantization ef- fects. However, QAT typically targets deployment-time efficiency and does not substantially reduce training-time memory or compute costs. Activation quantization Choi et al. (2018) focuses on ducing the bit-width of intermediate activations in addition to weights. While this offers additional memory savings during training, it also introduces additional quantization noise and quantization and de-quantization costs with extra computation. This can make training more unstable and often requires careful calibration to preserve performance.
Zeroth-order optimization Xu et al. (2024) reduces memory usage by avoiding explicit gradient computation. Since it does not require backpropagation, eliminates the need to store intermediate activations during the forward pass. However, this benefit comes with notable trade-offs: zeroth- order methods typically require many more function evaluations (i.c., higher query complexity) to estimate gradients indirectly, and the resulting gradient estimates tend to be noisier. These fac- tors can significantly slow down convergence, particularly for large, high-dimensional models like LLMs, where accurate and efficient gradient information is crucial for effective training ZZang et al. 2024a).
In addition to algorithmic approaches, several system-level engineering strategies have been devel- oped to address the memory and compute bottlenecks of large model fine-tuning. These methods aim to optimize training pipelines without modifying model architectures learning objectives. For example, activation checkpointing Chen et al. (2016) reduces GPU memory usage by storing only a subset of activations during the forward pass and recomputing the rest during backpropaga- tion. While it enables training deeper models under tight memory budgets, the trade-off is increased computational overhead due to repeated forward computations. This may lengthen training time significantly, especially in models with expensive forward passes.
17

---

Published as a conference paper at ICLR 2026
All the above techniques orthogonal and potentially complementary to our approach and could be integrated in specific scenarios. However, in the context of our work, these comparisons are less directly relevant to serve as comparisons. We will consider exploring such combinations in our future work.
C EXPERIMENTAL. SETTINGS
C.I DATASETS AND MODELS
General Language Understanding Task, For evaluating general language understanding profi- ciency, we follow Hao et al. HHaobo et al. (2024) that utilize six well-established datasets from the General Language Understanding Evaluation (GLUE) benchmark, as detailed by Wang et al. [aang] eet al. (018)). Specifically, the tasks included the Corpus of Linguistic Acceptability (CoLA), the Stanford Sentiment Treebank (SST-2), the Microsoft Research Paraphrase Corpus (MRPC), the Quora Question Pairs (QQP) dataset, Question NLI (QNLI), and the Recognizing Textual Entail- ment (RTE) dataset. Across all these GLUE tasks, the BERT-base model, introduced by Devlin et al. Devlin et al. (2019) is employed as the foundational pre-trained language model.
General Instruction Following Task. To assess the framework's 's performance on tasks requiring general instruction following, we benchmark on two prominent datasets, following the approach of Bai et al. Bai et al. (2024). The first dataset is Natural Instructions (NT), large-scale collection of diverse NLP tasks structured as instructions, developed by Wang et al. [Wang et al. (2022). For this dataset, we utilized Llama-1D model, with weights sourced from the Data-Juicer project by Chen et al. Chen et al. (2024). The second dataset is Dolly- 15K, an open- dataset of instruction- followed records created by Databricks Conover et al. (2023). for which the OPT-1.3B model from Zhang et al. Zhang et al. (2022a) is employed.
Domain-Specific Tasks. We further extend our evaluation to domain-speciff applications, focus- ing on the challenging medical and financial domains. Medical Domain: Adhering to the exper- imental design outlined by Ye et al. Ye et al. (2024) and Flower Gao al. (2025), utilize the MedAlpaca dataset, curated by Han et al. Han et al. (202)). for the training phase of our medical domain tasks. The framework's s effectiveness is then evaluated on four widely recognized medi- cal question answering benchmarks: PubMedQA by Jin et al. Jin et al. (2019) MedMCQA by Pal al. Pal al.) (2022). MedQA (USMLE-style questions) also by Jin al. Jin et al. (202I). and CareQA Duart et al. 2025). For these demanding medical tasks, we employ the Mistral-7B- v0.3 model MistralA| (2023). leveraging QLoRA for 4-bit precision fine-tuning, as proposed by Dettmers et al. Dettmers et al. (202)). Financial Domain: For the financial domain, training is conducted on a financial sentiment analysis dataset derived from the FinGPT project by Yang et al. FinGPT (2023). Evaluation is performed on three established financial NLP benchmarks: the Financial PhraseBank (FPB) by Malo et al. Malo et al. (2014), the Financial Sentiment Analysis on News Headlines (FIQA) dataset by Maia et al. Maia et al. (2018). and the Twitter Financial News Sentiment (TFNS) dataset Neural Magic (2022) These financial experiments are carried out using
the 1lama-3.1 1-B8B model from Meta Meta Al A 20244. also fine-tuned with QLoRA in 4-bit precision.
Reasoning Tasks. We additionally evaluate our framework on mathematical reasoning, using the MATH benchmark introduced by Hendrycks et al. [Hendrycks al. (2021). This dataset contains 12,500 competition-style math problems, each tagged by subject (algebra, counting and probabil- ity, geometry, number theory, etc.) and difficulty level from to 5. Unlike standard word-problem benchmarks, MATH emphasizes multi-step deduction and symbolic manipulation rather than direct pattern matching, making it strong stress test for federated parameter efficieet tuning. Every prob- lem includes full step-by-step solution, enabling training signals beyond final answer supervision. For these math tasks, we employ the Swve33-4--Instrut-22555 model, leveraging QLoRA for 4-bit precision fine-tuning, similar to the previous model settings.
Data Heterogeneity. To evaluate the performance under diverse data distributions, we specifically assess task heterogeneity using the Natural Instructions, D5lly-15K, and GLUE datasets, they naturally consist of varied instruction-basee or classification- based tasks. We apply both pathologi- cal and Dirichlet partitioning methods: the pathological method skews the number of tasks or label categories available to each client, while the Dirichlet method primarily controls the number of sam- ples per client. For Natural Instructions, we use an extreme case where each client is assigned 20
18

---

Published as a conference paper at ICLR 2026
out of 612 tasks; for Dolly- 15K, each client receives data from only 1 out of 7 tasks. In the GLUE setup, we apply Dirichlet partitioning with a = 0.01, resulting in a highly imbalanced distribution of samples across clients. For all other datasets, adopt IID partitioning.
C.2 MAIN RESULT SETTINGS
Table 4: FL settings for all experiments.
Medel
LaRA
Clint
Training Round
Learning Rate
batch size
CoLA
tu liea, 11816 tegeet pooduiient "gaay""
"valac]
(000)
200
0.001
16
28T-2
tork touuppaas 19916 www...ngggnononeeoooooooooo
100 (1001
85
01001
16
MRP::
101k-14416. lin,, sppee wegggneeeeeeeeeitte.oooooo
100 (10()
150
01001
16
OUP
bERT
1941-194766 aibpl 16766 www.s.oollelee
"valac"
(00(100)
16
Notural Instrrtioo
Lime (Iun--rtions itly
mek:1//016 kmraapta--tntt target pf "1ppo
10:1195)
Dells- 15k
OPT
18/52. tou suget sugettnodsks=""" "4.pr""""""""" "X.pe0""
50:3009)
25
0100000
Medical
Mistra--B--d..3
made- 1228. kea alptam1/25, toteet modaiient penr pnn"
50:1199)
30
0100005
Fiaance MATH
Liketa 3.1-B
1ado-122 ke 1229 taaaiaaede "Nami""
59:109)
15
0.00035
Qweni- peemi-til-laomme (011------mm--255000
make 1228 kms krs taree prceiienn "4,0e/
50:1195)
0.00005
the FL configurations for all experiments are comprehensively detailed in Table 4 This table outlines the specific models, LoRA parameters, client setups, and training hyperparameters employed for each dataset.
FL Settings, As shown in Table 4
For the GLUE benchmark tasks (CoLA, SST-2, MRPC, QQP, QNLI, and RTE), the BERT-base
model is utilized. The heterogeneous ranks are set using R = 16, rm = 4. These experiments involve 100 clients. Instruction fine-tuning tasks on Natural Instruction and Dolly-15K employed Llama (Data-Juicer-IIB and OPT-1.3B models, respectively. For Natural Instruction, R = 16 and I' 8, while for Dolly-15K, R- 16 and I'm 8. These setups use 50 clients. Domain-speciife experiments on Medical and Finance datasets also involve 50 clients. Medical tasks use Mistral- 7B.v0.3, Finance tasks use Llama-3.1-88
and both have R = 16 and I'm = 2. For mathematical 4e-3--4B-Intructt
reasoning, we further evaluate on the MATH dataset using federated client setup. All experiments sample 10% of clients uniformly at random per round.
under the same
Across these diverse setups, the number of training rounds was task specific,,,, varying from 15 to 200, as detailed Table 4 Learning rates are selected for each task group after evaluating a common search space of {0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001 J: specifically, GLUE tasks utilize a learning rate of 0.001. Natural Instruction and Dolly- 15K tasks are trained with 0.00001, and both Medical, Finance, and MATH experiments employ a learning rate of 0.00005. Batch sizes are also tailored: 16 for GLUE tasks, 4 for instruction fine-tuning, and 4 for the domain-specific experiments.
Justification heterogeneous rank settings: We empirically evaluate performance across wide range of LoRA ranks { 2,2,4, 8, 16, 32, 64). For the highest-resource clients, we select the largest rank that does not lead to overparameterization, following standard practice in FL to ensure balance between model capacity and resource capacity. For clients with mid-level resources, we adopt middle ranks so that the average rank does not excced the half of the highest rank, reflecting that majoraty of the clients have low resources.
Justification for rank settings: In homogeneous settings, we evaluate LoRA ranks (1, 2, 8, 16, 32, 64) to identify the optimal rank. In heterogencous settings, high-resource clients are assigned this optimal rank to avoid werrparramterrization while maintaining a balance between model capacity and device constraints. Low-resource clients are fixed at rank 1. Mid-resource clients are assigned intermediate ranks such that the average rank does not exceed half of the maximum, reflecting the realistic case where most clients are resource-limited.
We evaluate performance across a broad range of LoRA ranks (1, 2, 4, 8, 16, 32, 64) in homoge- neous settings and identify the best rank. For high-resource clients in heterogeneous settings, we assign the best rank that avoids over-parameterization ensuring a balance between model capacity and device constraints. For mid-resource clients, we choose intermediate ranks such that the average rank does not exceed half of the maximum rank, reflecting the realistic scenario where most clients have limited resources.
Implementation Details. Our framework was implemented using PyTorch version 2.6.0 (built with CUDA 12.4 support). Key libraries included Hugging Face fransformers version 4.51.3,
19

---

Published as a conference paper at ICLR 2026
datasets version 3.6.0, and peřt version 0.9.0 for LoRA and QLoRA functionalities. Exper- iments were conducted using CUDA 12.4. All experiments were carried out on a server equipped with an AMD EPYC 7763 64-Core Processor, 1.0 TB of system RAM, and 8 x NVIDIA RTX A6000 GPUs. The total GPU hours for running all the experiments are over 5,000.
C.3 OTHER EXPERIMENTAL SETTINGS
The experiments illustrated in Figure 7 are conducted on the QQP, MRPC, and RTE datasets using BERT-based model. A key aspect of this setup is the use of homogeneous LoRA rank across all clients (i.e., every client uses the same rank). Two distinct homogeneous rank configurations evaluated: 'Large Rank' setup with uniform rank of 16 per client, and 'Small Rank' setup with a uniform rank of I per client. These FL experiments involved total of 100 clients, with 10% of clients participating in each training round. The training is conducted for total of 150 rounds. In Figure 2 we report the average cosine similarities calculated for LoRA A matrices on the Query target module of Layer 1. These calculations based on experiments performed on the RTE dataset, with a non-IID data distribution achieved by assigning data from only one category to each client.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 RESULTS ON NON-IID GLUE DATASET
Table 5: Evaluation results on GLUE datasets under a non-IID setting (Dirichlet 0.01).
CoLA
SST-2
MRPC Acc.
QQP
QNLI
RTE
Method
Avg
Matthews Corr.
Acc.
Acc.
Acc.
Untuned Model
1.20
49.08
31.61
63.09
49.95
52.70
41.27
FedIT (Rank=R)
56.27
90.25
85.29
87.07
87.07
64.62
78. 78.28
Homogeneous
FedIT (Rank=R/2)
51.80
90.25
79.31
84.23
88.01
61.42
75.67
FLoRA
-2.07
88.97
68.38
38.75
47.37
47.29
48.18
Heterogeneous R = 16 Avg Rank=7
FlexLoRA
38.51
90.25
68.39
79.91
70.98
53.79
66.97
HETLORA
40.08
89.97
76.35
81.71
85.96
58.
72.03
Fed-PLoRA
52.33
90.71
79.65
84.79
88.01
61.01
76.08
Table 5 presents evaluation results on the GLUE dataset under a non-lID setting, where client data distributions are skewed using a Dirichlet distribution with 0.01, representing highly im- balanced and extreme case. this challenging scenario, our method Fed-PLoRA consistently out- performs all baselines. Compared to FLoRA, Fed-PLoRA yields substantial gains of +54.40% on CoLA and 1.27% on MRPC, underscoring the limitations of FLoRA's random initialization and emphasizing the benefit of more carefully designed initialization strategy in our method. Compared FlexLoRA, Fed-PLoRA achieves a notable improvement +17.03% on QNLI and +0.46% on SST-2. Although FlexLoRA leverages SVD to derive representative low-rank matrices, it remains susceptible to information loss, which can lead to significant initialization and aggregation noise, especially when der non-llD cases. When compared to HETLORA, Fed-PLoRA improves performance by +3.08% on QQP and +2.89% on RTE. These gains come from Fed-PLoRA's more effective handling of initialization and aggregation noise, which HETLORA introduces through zero-padding and rank truncation. Furthermore, Fed-PLoRA surpasses FedIT with fixed rank of 8 by average of +0.41 %, despite using a lower average rank of 7. This demonstrates the efficiency and adaptability of our approach that incorporate all resouree-constraaine clients into training with zero initialization noise and low aggregation noise.
clients operate with much lower ranks than the global configuration and un-
D.2 RESULTS ON DOLLY-15K DATASET
Table presents the main results on the Dolly-15K dataset. We report Rouge-L scores the fine- tuned global model under both IID and non-lll settings, where the non-IlI scenario is simulated
20

---

Published as a conference paper at ICLR 2026
Table Evaluation results on IID and non-lID Dolly- 15K datasets with OPT-1.3B model.
Dolly-15K (Rouge-L)
Method
IID
non-IID
Untuned Model
40.02
FedIT (Rank=R)
60.05
59.37
Homogeneous
FedIT (Rank=R/2)
59.75
59.07
FLoRA
40.01
40.01
Heterogeneous
FlexLoRA
57.40
48.82
R = 32 Avg Rank=13
HETLoRA
59.38
58.36
Fed-PLoRA
60.07
59.69
by assigning each client data from only out of 7 distinct tasks. Our method, Fed-PLoRA, con- sistently outperforms all baseline approaches under both data and resource heterogeneity. Notably, Fed-PLoRA surpasses FlexLoRA in the non-lID setting by large margin of + +10.87% highlighting the critical impact of initialization noise in federated fine-tuning. Moreover, Fed-PLoRA even out- performs the homogeneous case with full rank 32, achieving gains of +0.02% and +0.32% under IID and non-IID conditions, respectively. This suggests that our parallel one-rank LoRA modules offer greater flexibility than conventional high-rank LoRA, and can potentially lead to better over- all performance. These results further demonstrate the effectiveness of Fed-PI A in mitigating initialization and aggregation noise via its parallelized design.
D.3 RESULTS ON MEDICAL DATASETS
Table 7: Evaluation results on medical datasets with Mistral-7B-v0.3 model.
Method
PubMedQA
MedMeQA
MedQA
CareQA
Avg
Untuned Model
55.77
38.43
40.92
41.28
44.10
FedIT (Rank=F)
70.60
41.16
46.19
47.35
51.32
Homogeneous
FedIT (Rank=R/2)
68.60
41.19
46.19
46.55
50.63
FLoRA
55.80
40.30
45.70
46.10
46.98
Heterogeneous
FlexLoRA
58.40
41.16
46.26
46.26
48.02
A = Avg Rank=3
HETLORA
60.80
41.23
46.26
46.10
50.85
Fed-PLoRA
70.20
41.50
46.11
46.85
51.16
Table 7 presents the main results (in accuracy) on medical and financial datasets. It is noteworthy that due to the inherent capabilities of the LLMs employed and the potential domain shift between the training and test sets, the observed performance gaps among different federated methods remain relatively small. Fed-PLoRA demonstrates improvements over FLoRA, achieving accuracy gains of +14.40% on PubMedQA, +1 +1.20% on MedMcQA, +0.75% on CareQA, + +12.29% on FPB, and +13.15% on TFNS. However, when compared to FlexLoRA and HETLORA on the MedQA dataset and to FLoRA on the FIQA dataset, our method shows slight accuracy drops. This may be attributed the nature of these comain-specific instruction fine-tuning tasks with large LLMs requiring only small LoRA rank. In such scenarios, the initialization noise in LoA and HETLoRA can be relatively small.
D.4 RESULTS ON MATH DATASETS
Table g summarizes the performance of various federated fine-tuning methods across seven MATH cub-categories. The baseline (untuned model) exhibits limited reasoning capability across most cat- egories, with overall average accuracy remaining below 10%. Under homogeneous settings, FedlT with full rank (R) yields consistent gains over the R/2 configuration across all sub-tasks, demon- strating the benefit of maintaining richer update capacity during global aggregation.
21

---

Published as a conference paper at ICLR 2026
Table 8: Evaluation of Qwenn--B-AAB---nttru----5 MATH datasets.
Algehra
Counting&
Gommetry
Algehra
Numberr Theory
PreAlgehra
Ang
Uatuned
12.58
5.90
3.75
1.32
5.74
15.61
4.57
7.94
Homogeneeus
(Rank-R)
26.79
26.11
FedIT (/))k-RR/2))
36.47
22,57
16.70
11,02
21.48
57.08
12.27
24,02
FLoRA
12:72
6.75
5,27
Heterrgeneous
I S Avg Rank-4
FlesLoRA
38.83
19.62
15.65
9.52
22.03
40.41
10,62
24.88
HETLORA
28.47
17.27
Fed-PLoRA
43.99
24.89
18.99
14.17
27.77
42.59
12.82
28.96
Under heterogeneous environments where the average rank is fixed to R 4, Fed-PLoRA achieves the strongest overall results, outperforming FLoRA, FlexLoRA and HETLORA with significant im- provements in Algebra, Counting & Probability, and Number Theory, where gains exceed +5-15% on average. Notably, Fed-PLoRA also maintains competitive accuracy on Geometry and Intermedi- Algebra, indicating enhanced adaptability even under rank-diverse client configurations.
D.5
ABLATION STUDIES
D.5.1
COMPARING SELECT-N-FOR WITH OTHER STRATEGIES
Figure 6: Visualization of rank selection results using different selection strategies.
Here, provide visualization of the rank selection results in Figure 6 comparing Select-N-Fold with the Weight Norm and Fixed methods in Figure Overall, random strategies yield a more uniform and unbiased distribution across layers.
D.5.2 EMPIRICAL OBSERVATIONS ON INITIALIZATION AND AGGREGATION NOISE IN SOTA METHODS
As discussed in Section 3.4 existing federated LoRA-based fine-tuning methods suffer from dif-
ferent levels of initialization and aggregation noise. To isolate these effects, Figure 7 illustrates optimization trajectories and empirical results. In this setup, fix the LoRA rank across clients to remove initialization noise for FlexLoRA and HETLoRA, allowing a clearer focus on their aggre- gation noise (detailed settings in Appendix Section
22

---

Published as a conference paper at ICLR 2026
Method/Data set (Acc.)
FlaxLoRA (Small Rank)
QQP
MRPC
RTE
FlaxLoRA (Large Bank)
Optimal
FLORA
53.80 -1513
43.87
-12.26
51.02*
FlexLoRA
84.64-0.0
79.65
64.25"
(Large Rank)
(Small Rank)
-0.12
2,06
FlexLoRA
(Large Rank)
86.94-0.15
0.11
86.84
1.31
66.90*
1.56
FLoRA (Large Rank)
HETLORA
86.86 0.02
87.58
0033
68.35*
Figure 7: The intuitive convergence trajectories of existing methods and empirical results three datasets homogeneous settings.
The results show that FLoRA, while free of aggregation noise, struggles due to random initialization, yielding low accuracies of 53.88% 43.87%, and 51.029 ..00% on QQP, MRPC, and RTE. FlexLoRA, by contrast, is mainly affected by its rank-dependent SVD-based aggregation noise: at large ranks (c.g.. T2 16) performs well (86.94% on QQP), but its accuracy drops at smaller ranks (c.g., 84.64% at performance, with 87.58% on MRPC and 68.35% on RTE, suggesting its noise can be no worse than FlexLoRA's. Finally, FlexLoRA incurs heavy computation from repeated SVD, whereas HETLoRA uses lightweight averaging scheme.
1). HETLoRA introduces bias through its averaging rule, yet achieves strong
D.5.3 IMPACT OF HYPERPARAMETERS
Table 9; The impacts of the number of TC and clients' HR.
CoLA Dataset
Method
TC=100
TC=200 HR=1:1:1
TC=100 HR-6:3:1
HR=1:1:1
FLoRA
-4.08
-2.07
-2.07
13.42
-4.73
-2.5
HETLORA
48.09
50.54
51.04
Fed-PLoRA
59.38
55.56
58.28
The Impact of Total Clients and Resource Heterogeneity. We study how the total number of clients (TC) and the heterogeneity ratio (HR) of client resources affect performance. Table 9 re- ports results on CoLA. Across all configurations, Fed-PLoRA consistently outperforms basclines (FLoRA, FORA HETLoRA). Increasing TC from 100 to 200 (with balanced HR=1:1:1) slightly reduces Fed-PLoRAY score (59.38% to 55.56%) but still keeps a clear margin over HETL (50.54%). When shifting HR from balanced (1:1:1) to skewed (6:3:1, more low-resource clients) with TC=100, Fed-PLoRA remains robust (59.38% to 58.28 58.28%). These results show that Fed-PLoRA scales well with both client population and resource imbalance.
Table 10: The impacts of the heterogeneous rank setting.
CoLA Dataset
Method
Ranks 1/2/4
Ranks 1/2/16
Ranks
Ranks 2/8/16
1/4/16
FLoRA
-2.07
-2.07
-4.08
-2.07
A
1.01
4.16
13.42
-3.76
HETLoRA
47.44
46.70
48.09
49.54
Fed-PLoRA
58.58
60.26
59.38
60.41
23

---

Published as a conference paper at ICLR 2026
The Impact of Client Rank Settings. evaluate how different distributions of LoRA ranks across clients influence performance, with results on CoLA shown in Table 10 Rank configurations tested include {1, 2, 4). {1, 2, 16), {1, 4, 16), and {2. 8, 16), under the same settings as the main exper- iments. Fed-PI.oRA consistently achieves the highest Matthews Correlation scores, outperforming FLORA, FOR,, and HETLoRA across all cases. For example, Fed-PLoR attains 58.58% with ranks 1/2/4 and 60.41% % with ranks 2/8/16, demonstrating strong robustness and adaptability to heterogeneous client resources.
Table 11: Performance and resource usage of different methods under extremely large global model rank.
Methed
CaLA Matthews Curr.
SST-2 Accurncy
MRPC Accurscy
RTE Accuracy
Client ELOPS GFLOPS
Server ELOPS MFLOPS
Thronghput seconds 100 tnken
Uplink/Demmiini MB/ Round
Hontogenoous
FedIT (Rank= R0
02.94
91,97
88.72
73.25
695,50
0,50
2.33
430.05 / 430.05
Heterogenesus
FLaRA
-1.95
70.52
31.51
90
74.14
25,52 2835
FlesLoRA
50.91
202.97
620.20
23.52
mux/rs)
HETLORA
47.65
71,02
90,88
59.55
202.37
1.71
0.89
23,02/ 21.32
Arg Rank
Fdd-
28.62/129.16
The Impact of Extremely Large Global Rank R Learning Efficiency. We evaluate the effect of increasing the global model rank to 128 while capping local ranks at 16 to reflect realistic resource constraints. Experiments are conducted on four GLUE tasks under the IID setting with a batch size of 16. Input sequence length is set to 128 for CoLA and SST-2, and 256 for MRPC and RTE.
Client-side FLOPS (including initialization and LoRA operations) are averaged over all clients and rounds on per-sample basis. Server-side FLOPS measure aggregation costs. Throughput is re- ported as the average seconds to process 100 training tokens on clients. Communication volume denotes uplink and downlink per round per client in MB under 32-bit precision.
As shown Table III Fed-PLoRA consistently outperforms other heterogeneous baselines under this extreme R. Client-side FLOPS and throughput remain identical across methods since all use the same model structure and simulated hardware. On the server side, Fed-PLoRA matches HETLoRA's efficiency by aggregating one-rank modules independently. Uplink costs equally low across all methods, as only trainable LoRA modules are transmitted. The main trade-off is in downlink cost, which scales with R and equals that of FedIT. However, excessively large R is rarely practical under heterogeneous constraints and may even cause divergence, as observed for FLoRA and FleX on Col.A. Since LoRA is designed for parameter efficiency, setting R close to the hidden dimension provides little benefit and often harms stability.
D.5.4 THE IMPACT OF DIFFERENT CONFIGURATIONS OF PLORA
We evaluate how the rank size of individual PLoRA modules influences performance. Specifically, compare our default one-rank design (Fed-PLoRA (Parallel One-Rank)) with two-rank variant (Fed-PLoRA (Parallel Two-Rank)). both configured to yield the same total effective LoRA rank across clients with heterogeneous rank settings (2, 8, 16). The one-rank configuration achieves a slightly higher Matthews Correlation score (60.41% than the two-rank configuration (59.35%). This result supports our design choice.
D.5.5 THE IMPACTS OF DROPOUT
Table 12: The impacts of dropout in the PLoRA.
Dolly-15K (Rouge-L)
Method
Rank: 1/8/32
IID
non-IID
Fed-PLoRA (Unfold, w/ Dropout)
60.07
59.70
Fed-PLoRA
60.07
59.69
We investigate the impact of applying LoRA dropout within the Fed-PLoRA framework. Typi- cally, dropout layers are utilized for regularization before input features pass into LoRA modules
24

---

Published as a conference paper at ICLR 2026
(specifically, before the LoRA A matrix). In Fed-PLoRA approach, the Select-N-Fold strategy involves merging non-selected parallel LoRA modules into the main target module. When these modules are folded, any dropout layers specifically associated with these folded LoRA paths are ef- fectively bypassed for those components in subsequent forward passes. This characteristic motivates evaluation understand how performance is affected by dropout under folding mechanism. Table 12 presents a comparison on the Dolly-15K dataset (Rouge-L score), with client ranks set to (I, 8, 32). We compare variant termed "Fed-PLoRA (Unfold, w/ Dropout)", where we assume modules are frozen but kept unfolded, allowing standard LoRA dropout to be active on all PLoRA modules against our standard Fed-PLoRA configuration, but with higher computational costs. The results show minimal differences between the two approaches across both IID and non-IID data dis- tributions. In the IID setting, both configurations achieve a Rouge-L. score 60.07%. the non-III setting, Fedd-PLoRA (Unfold, w/ Dropout)" achieves 59.70%, while our method achieves 59.69%. These nearly identical results suggest that not using dropout on frozen layers does not affect the performance.
D.5.6
MORE VISUALIZATION ON COSINE SIMILARITIES ACROSS PLORA MODULES.
Layar
Luyar
Layer
Layer
LORA-A
-75
LORA-B
075
402
-035
LoRA
Figure 8: Cosine similarity heatmaps of PLoRA modules on the non-lID RTE dataset (averaged over all rounds). Each block reports pairwise similarities of modules across clients for LoRA-A and LoRA-B, spanning layers 0 through 11.
To analyze the aggregation noise in Fed-PLoRA, we visualize the cosine similarity heatmaps across clients' PLoRA modules for the Query weights in all 12 transformer layers of BERT-base, averaged over all rounds. Both A and B module similarities are shown. According to Section when
AL.G) Agy Bi.G) B aggregation noise Fed-PLoRA minimized. Figure reveals consistently high similarity values (near 1.0) along diagonal and cross-wise patterns, particularly highlighted by the red arrow in Layer 0''s LoRA-A, suggesting strong similarity among certain ranks. Moreover, we observe that some consecutive layers maintain high inter-client similarity, indicating that even under non-lID settings, certain ranks learn overlapping redundant features, potentially pointing to shared semantic structures across clients.
This pattern emerges naturally from the design of Fed-PLoRA. Since each client receives the full set of R global PLoRA modules and selects a subset for local training. all modules remain aligned in initialization across rounds. The random selection mechanism ensures that time, every rank-j module trained by a diverse subset of clients, leading to gradual synchronization of pa- rameters across the population. Furthermore, the frozen folded modules help preserve the global
25

---

Published as a conference paper at ICLR 2026
structure clients with limited capacity, reducing noise accumulation across rounds. As result, frequently selected ranks tend to exhibit strong inter-client alignment, while occasionally se- lected modules still retain structural consistency due to their shared initialization and partial update history. This design promotes implicit coordination among clients and contributes to the observed high cosine similarity, thereby reducing the aggregation noise without explicit regularization ordination.
While our method does entirely eliminate aggregation noise. This a characteristic shared with several other federated LoRA-based fine-tuning methods, such as FedIT and HETLoRA. However, we note that there is growing body of research in the literature Sun (2024); Guo et al. (2024) focused on improving the aggregation of LoRA in the homogeneous setting. We will explore the integration of these methods with Fed-PL.RRA in heterogeneous settings our future research.
E DETAILED ANALYSIS OF INITIALIZATION AND AGGREGATION NOISE
This section investigates the noise arising from initialization and aggregation processes in heteroge- neous federated LoRA- Abased fine-tuning methods.
We use the following general definitions for noise as established in the main paper: Let 6--1
(A l r-1 1) be the global LoRA parameters from the server at the end of round (with global rank R). Let 0:-1 (A!-1 B) be the local LoRA parameters with which client (from a set of clients) starts round (with local rank r,). The Initialization Noise at the beginning of round t is:
N Init
-E(
o " | e |
Here, X & signifies the part of X not captured by Y.
After local training, client i produces updated LoRA parameters 0f = (A By). The ideal ag- gregated target module update is Aw = Ei B(A, The actual aggregated target module update by specific method is AW B' A', where A B¹ the global LoRA parameters after aggregation at round l. The Aggregation Noise at the end of round is:
Agg
AW AW
E.1 FLORA
FLoRA Wang et al. (2024) employs a stacking-bascd aggregation. For initialization, client LoRA modules A, are randomly initialized (c.g., Gaussian Distribution), and B is initialized to zeros at each round
Initializatinn Noise (N (WFLORAIni :mn): Given A is random and B 1 O, these local parameters do not retain information from the previous global LoRA parameters ' and B For B matrices: Since B O, the part of B not captured by B is B itself. Thus, B e B = ||= F. For A matrices: A is random and independent of A It does not systematically
capture any part of A The initialization noise is formulated
Not NLLOAIIIN
(1
e A A
+
Σ
+ | +0²
where o² denotes the magnitude of the random noise used for initializing A matrix. This noise is substantial.
A Aggregation Noise (NELORA FLORA stacks client LoRA modules. Let client i have A
gri.k and B Rdd., The server forms:
A
A'
E R KA) and
B = (Bi
BJ) e (x))
26

---

Published as a conference paper at ICLR 2026
The aggregated update is AW actual aggregated update is:
AB
The product is B'A
Eielo! B So, the
FLORA
AW 4E
B
iE/o
The ideal target module update AW = EiEuj) B. Therefore, the aggregation noise
NELORAAEE
AWW AW MORAlE
B
Σ
B
=0
iEjul
iE(0]
E.2 FLEXLORA
FlexLoRA aggregates LoRA modules in full weight space, then uses SVD for local LoRA modules.
Clients truncate these for initialization.
Initialization Noise (N FleLLOAAIiii The server has global LoRA modules A * (from SVD, rank R) and B (from SVD, rank R). Client (rank r; S R) initializes by truncation: A
A t-1
(first rows of A 1) B
B B [.ary] (first columns of B'-1). The part of A not captured is The initialization noise is then formulated
(-1 as,
The part of B not captured is Bt=1 br++1:))
FiexLORA
Σ iE(ol
1-1
e
A
+
B'-l e BB" (1(1)
Σ
A t-1 11
+
B
B (p)
iE(M]
This noise is non-zero if any r; < R.
Aggregation Noise (N Av) Clients send A B/. The server computes the ideal average update:
AW
B
However, FlexLoRA then performs SVD on AW, to get rank- R approximation:
AW FlexLoRA = SVD(AW5) U'S
The aggregation noise is the SVD truncation error:
N MiLLORAAAE =
AW' - AW FIceLoRAl
AW' U'S
E.3 HETLORA
HETLoRA zero-pads LoRA modules uniform rank for aggregation, and then clients truncate for initialization.
Initialization Noise (Ni WEELLORAMMD: Server has global LoRA modules
A (rank R), B (rank
R). Client initializes by rank 73 truncation: A f-1 A t-1 B noise is formulated as,
B(11 The initialization bar.)
Σ ((AAMTERA)) + +|B B (l))
NO
HETLORA Init
Aggregation Noise (WHETLORA ) Client sends A (rank r;), B (rank rς). Server zero-pads to EjE(1) A/y,
rank R: A. B Global LoRA modules averages of padded matrices: A'
27

---

Published as a conference paper at ICLR 2026
B' = -.EE(u) BJ Actual update: A WH AWHETLORA B'A' 1 Eseny Exe(ol B A Ideal
HETLORA
update: AW" Eielel BiA Eiely B A
The aggregation noise is:
N
WHETLORA
AW
AW WMNETTORRA
B
By B
[E(u]
jElo kE[u]
B
E
Σ
B A
2
[E(e]
SElo] AE(O]
F E
B
A
Σ
A
BY
LE(e]
LE/vl
jElvl,iA
1)
Σ
B AW
E
B
(E(v)
jE(oji*A
E.4 FED-PLORA
Client i updates its selected modules (A B B bex; and sends them to the server. The server aggregates each j-th parallel module independently. Let 2/ = {i I i E [v],] E 1C;) be the set of clients that trained the j-th module. The server computes the average for each PLoRA module's A and B matrices:
1 A 6)
Σ 1E
ALooy
B
Σ
Bloo)
(
The actual global update from Fed-PLoRA is the sum of the products of these averaged components:
R
AW Fod-PLORA
Σ
B
A
(
j=1
Σ
1
Σ
B B
Σ iEE
Alr
10t
10t
1100
R
Σ iEE
B
Al.
12
ME
The optimal aggregation in this context, as defined for Fed-PLoRA is an average of client products per parallel module path:
AW =
Σ ==1
Σ Bl.0)
[0/
28

---

Published as a conference paper at ICLR 2026
The aggregation noise is then the difference between these two global updates. We the formula- tion consistent with the main paper. which is ||AW" -rr PLORA I
N N Aed-
Aw - AW Fed-PLORA
R Σ
Σ
Σ
B (1.0) 22.00
Σ
(20)
())
(1 i20
J=1
NED
R
Σ
Σ 11 12E
Σ
B 40.
Σ
B
O
6))
kE
Let T be the term inside the parenthesis for module j:
T Σ Σ
A -
ارر
Σ
Ak.)
k kEO
We can expand the double summation:
E B PA.u)A
Σ
Bi,0)
Σ
A
KERi
Substituting this back into Ty:
TJ= =
Σ
B
A
Σ
B A1)
- -19
Σ
Al,))
KE
i1
Σ
B 4.0)
A
2.00
-1)
Σ
B BB A
K0
kE KEO
i1712
So the aggregation noise can also be expressed by substituting this form of Tj:
n
N2t Nood-PLORALAAS
Σ
B AL (10tn) -1)
Σ REO
B Bl
2
K17i2
This noise term Tý (and thus the overall noise) becomes zero if, for each module j. the condition
Rin EREO; (B (A)) Any)) holds, which typically occurs when client up- dates (Be.)) A ( for a given module j are highly similar or aligned across clients 22)
ADDITIONAL STUDIES
F.1 EFFICIENCY OF RANK-BASHI FEDERATED TUNN-TNIIG METHOD
In this section, demonstrate the efficiency of the rank-based method for resource-constraing devices by analyzing the FLOPs of LoRA modules, overall training throughput, and communication costs under different rank settings. We the 3.LaMA-3.1-88 model the Finance dataset representative case (detailed dataset description is shown in Section C.I). with a hidden size of 8192, batch size of 4, and assuming float32 precision, where each parameter occupies bytes.
To compute the FLOPs introduced by LoRA modules, we begin with the forward computation 2 B. Ar, which requires 2r(dm + d operations per token, where is the LoRA rank and dn d
29

---

Published as a conference paper at ICLR 2026
Reduddov FLOPS
Seconda Tokrr
Horeeon 0112
Rad udan
Rad
1119%
Incwaas 2.27
Reduclor
Natadias
FLOPS
$2.19
Radket
Radeed
Ranse!
Ranked
Rasie
Ranee!
Rasted
Rasied
Rensit
Ranke] Pankid Hanii1
[b] Throughout Incroo
Communicatiion
Peduuction
Reduct on on Finance Dataset
(d) Averaged Acturnt
Figure 9: Visualization on the efficiency of the rank-based method.
d = 8192 is the hidden dimension. Since the backward pass roughly doubles the compute cost, the total per-token FLOPs for each LoRA module becomes Grd. The total LoRA FLOPs per training step can thus be estimated as 6rd x batch size x sequence length X number of LoRA modules. For instance, with a sequence length 512, 32 transformer layers, target modules per layer, result in total 64 LoRA modules, the total becomes 6 x r x 8192 4 x 512 x 64 = 196, 196,608 x FLOPs. This value increases linearly with rank. On the training dataset, order to train one epoch with the full dataset, it requires approximately 10,000 steps with this batch size (i.e., 4), such that the total FLOPs introduced by LoRA modules under rank settings
8, 2, and 1 are respectively 1.57 X 10th x 10,000 1.57 x 10t0, 3.93 x 10a x 10,000 3.93 10 and 1.97 x 10s 10,000 1.97 x 10 FLOPs per epoch for the LLaMA-3.1-20 model. We visualize the FLOPs reductions for
three levels of rank settings in Figure 9 (a).
To assess the training efficiency under different LoRA rank settings, we measure the throughput in terms of seconds per 100 tokens. The actual wall-clock times for the whole training process in
setting are 3.07, 2.84, and 2.41 days under rank settings r = 8, 2, and 1, respectively. The experi- mental and hardware settings are reported in Sections C.I C.2 This results in throughput values of approximately 0.864, 0.799, and 0.678 seconds per 100 tokens, respectively. These measurements demonstrate that reducing the rank leads to improved computational efficiency during training. We visualize the throughput increase for three levels of rank settings in Figure (
To estimate the communication volume for one training epoch, we assume that only the LoRA pa- rameters are transmitted every 10 steps. With 10,000 steps per epoch and transmission frequency every 10 steps, there are 1,000 transmissions total. Each transmission involves sending and re- ceiving 128rd LoRA module parameters, where is the LoRA rank and d = 8192 is the hidden size. Assuming float 32 precision (4 bytes per parameter), this yields total communication vol- ume of 512,000rd bytes per epoch. For ranks are approximately 3.91 GB, 7.81 GB, and 31.25 GB, respectively. We visualize the communication volume reductions for three levels of rank settings in (i)
1, 2, and 8, the resulting communication costs
From the resource consumption perspective, the rank-based method provides a practical solution for efficient federated fine-tuning of LLMs,, evidenced by the reduction in LoRA module FLOPss and communication volume, as well as improved training throughput. Although the rank-based approach not specifically designed for memory savings, our method orthogonal wide range of existing memory-efficient techniques and can be easily combined with them (as discussed in Section B).
These resource consumption reductions make low-rank configurations particularly practical for de- ployment in heterogeneous federated settings. However, as shown in Figure 9 (d), we provide an average performance comparison on Finance datasets under three levels of rank settings. The av- erage accuracies are 52. 19%, 51.04%, and 50.05% for ranks r = 8, 2. and 1, respectively. This demonstrates that different LoRA rank settings lead to a noticeable performance gap, where even a % drop in accuracy is considered significant and typical in LLM instruction fine-tuning tasks on domain-specific downstream datasets. This observation motivates the design of our proposed method, Fed-PLoRA which aims to incorporate clients experiencing degraded performance under low-rank configurations into heterogeneous-rame federated training process and improve the over- all performance of the global model.
30

---

Published as a conference paper at ICLR 2026
F.2 COMPUTATION, COMMUNICATION AND MEMORY OVERHEAD OF FED-PLORA
We analyze numerical resource overhead in this section. Here, we use HETLoRA Cho et al. (2024) as a reference to calculate overhead, since it is the naive extension of FedAvg to heterogeneous resource settings. We provide the results following:
During the FL process, the computation overhead can be analyzed in the steps of local parameter initialization, local training, and server-side model aggregation Wu & Wang p002); Zhou et al. (2022). To ground the discussion, we report the per-client, per-round computational workloads under BERT- Devlin et al. (211)) with PF16 precision. The results (in terms of FLOPs) are reported using 100 clients with a global rank R=16 and local rank configuration of {1, 4, 16), representing three tiers of resource heterogeneity. The number of clients in each tier are 34, 33, and 33, respectively. Each client assigned 100 training samples. The result are summarized in Table 13 Note that, because local initialization, local training, and local overhead vary across clients, we report the results for the weakest client with = 1, as it represents the most resource-constrained setting.
Computation Overhead.
Method
Local Iniiialization (FLOPS)
Local Troining (FLOPS)
Model
Local Overhead OP))
Overhead (PLOP)))
HETLORA
3.68
1014
107
FLoRA
8.68
+885
PleLLORA
3,58 10f
1.45 1014
7.71 x 101
Noae
17.70 20
PLARA
1.41
1014
10
Table 13: Computation overhead of Fed-PLoRA compared with SOTA methods.
Overall, the dominant computation cost arises from local training, which on the order of 10 FLOPs. Our method adds only negligible overhead during local initialization (on the order 107 FLOPs), and reduces computational cost for model aggregation compared to HETLoRA.
Communication Overhead. The communication cost accounts for both uplink and downlink traffic.
Here, report the per-client, per-round uplink and downlink volumes. The measurements obtained using FP16 BERT-base Devlin et al. (2019) with a global rank of R=16. Table [4] report the results for the weakest client with r, which reflects the largest communication overhead incurred by our method.
Method
Uplink (MB)
Downlink (MB)
Overhead (MB)
HETLORA
0.04
0.04
FLoRA
0.04
13.54
+13.50
FleLLoRA
0.04
0.04
Nome
Fed-PL.RRA (Ours)
0.54
+0.50
Table 14: Communication overhead of
compared with SOTA methods.
We can see that our method incurs only 0.50 ME overhead per round for the weakest client, which is negligible
in constrained network settings. We further extend evaluation to larger models when analyzing the downlink traffic of Fed-PL.RR. As shown in Table 15] in Fed-PLoRA. the downlink traffic is only few megabytes per round per client, even for models with billions of parameters, making it easily deployable over low-bandwidth networks.
Model
Downlink (MB)
LLaMA-IB
2.79
O.T-1.3B
2.62
Mistral-7B-v0.3
7.08
Llama-3.1-8B
7.09
Owenn-44D-Instuux-25000 Instruct-2507
5.24
Table 15: Downlink traffic of Fed-PLoRA across model scales.
31

---

Published as a conference paper at ICLR 2026
Persiticnt Parameders (MB)) Reupeeary Piruncters
(EB))
2727.30
13.54
+13.0
RosLaRA
210.00
004 0
2727.90
Noxe
Fol-PLoKA (Ours)
210.00
0.64
0.4 1
2737.90
10.50 1.0
Table 16: Memory overhead of Fed-PLoRA compared with SOTA methods.
Memory Overhead. Here, we study the memory usage for the storage of the base model and local trainable LoRA/PLORA parameters, the temporary memory for the received global parameters during local initialization, and the memory during local training. The results are obtained using FP16 BERT-base Devlin et al. (2019) with global rank of R=16. The batch size for local training 16, and the input size is 512. We report the memory footprint on the weakest client with rank r, in Table I6 The persistent base model and local LoRA/PLORA parameters occupy the same amount of memory across all methods. When dealing with the received global parameters during local initialization, our method requires addtional 0.50 MB of temporary memory, which is negligible compared to the base model's ''s and local training memory costs.
32


Reviews:
[{"review_id": "AMLLyqbZjK", "reviewer": "Reviewer_HFN8", "summary": "Fed-PLoRA addresses federated fine-tuning with heterogeneous LoRA ranks by proposing a new framework to mitigate both issues where rank mismatches introduce substantial initialization and aggregation noise. It introduces PLoRA, which replaces a single rank-R LoRA module with R parallel rank-1 modules that are mathematically equivalent to standard LoRA. Combined with a Select-N-Fold strategy, the method achieves zero initialization noise and reduces aggregation noise under heterogeneous budgets. Experiments across multiple LLM fine-tuning tasks (e.g., GLUE and instruction-following) show consistent accuracy gains over FLoRA, FlexLoRA, and HETLoRA, while avoiding the heavy SVD overhead that hurts communication and training time. It’s easy to adopt in practice though broadcasting all R modules does add some downlink cost.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper precisely defines initialization and aggregation noise in heterogeneous LoRA settings and shows Fed-PLoRA removes the former and reduces the latter.\n2.  PLoRA’s parallel rank-1 decomposition is mathematically equivalent to standard LoRA yet naturally supports heterogeneity; paired with Select-N-Fold, it guarantees zero initialization noise while curbing aggregation noise. \n3. Strong and robust empirical results. Fed-PLoRA consistently outperforms FLoRA/FlexLoRA/HETLoRA across tasks (e.g., GLUE), and remains robust as client counts and rank distributions vary, including challenging non-IID scenarios.", "weaknesses": "1. The method broadcasts all R parallel rank-1 modules to every client and asks clients to keep folded modules, downlink traffic and on-device storage could become non-trivial in weak-network or mobile scenarios.\n2. The paper’s empirical validation relies on relatively small or outdated and non-unified base models, which limits generalizability; it would be stronger to standardize on modern backbones like Qwen3 and Llama 3.2 across multiple sizes.\n3. Its benchmark suite skews toward easier tasks (e.g., GLUE and basic instruction following) and should incorporate more rigorous reasoning/knowledge evaluations such as MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH.\n4. The authors do not clearly report the untuned base-model performance, obscuring absolute gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IBtvWzcK9G", "reviewer": "Reviewer_G2mv", "summary": "This paper targets a problem of resource heterogeneity in Federated Learning (FL) for fine-tuning LLMs. A common solution for federated fine tuning for clients with different computational resources is to adapt LoRA modules of different ranks. The authors argue that this heterogeneity introduces two primary issues: initialization noise (when low-resource clients must truncate or discard parts of the global model) and aggregation noise (when the server attempts to combine modules of different dimensions).\n\nTo solve this, the paper proposes Fed-PLORA, a novel framework built on two core ideas: 1) Parallel One-Rank Adaptation  or PLORA where instead of having a standard rank-R LoRA module, they propose utilizing the  sum of R parallel one-rank modules. 2) Select-N-Fold Strategy which is A new initialization and training protocol.\n\nThe authors claim that the \"Select-N-Fold\" strategy completely eliminates initialization noise, as no information from the global model is \ndiscarded. They analyze the remaining aggregation noise and argue it is minimal. Through extensive experiments, they demonstrate that Fed-PLORA outperforms existing heterogeneous FFT methods.", "questions": "Can you make a table (at least for one setting) to show all the new costs of your method and compare with the prior works?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and the authors did a good job explaining the existing problems. \n\n* Through various settings and different empirical results the authors show the merits of their algorithms. \n\n* The authors did a proper ablation study, explaining the importance of each component.", "weaknesses": "One important aspect of the paper is the Downlink Communication cost. The \"Select-N-Fold\" strategy has one clear limitation that is understated: downlink communication cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DeVT4YXLAb", "reviewer": "Reviewer_AmKz", "summary": "This paper tackles the critical challenge of heterogeneous LoRA ranks in Federated Fine-Tuning (FFT). The authors identify that current methods suffer from initialization noise and aggregation noise due to rank mismatches. Their proposed solution, Fed-PLORA, uses PLORA (Parallel One-Rank Adaptation) to re-parameterize LoRA modules as a sum of parallel rank-1 components . This enables a novel Select-N-Fold strategy, where resource-constrained clients train a random subset of modules and \"fold\" the rest into the frozen weights. This method is theoretically shown to achieve zero initialization noise and is empirically demonstrated to consistently outperform existing heterogeneous FFT methods like FLORA, HETLORA, and FlexLoRA.", "questions": "PLoRA folds the remaining R − ri untrained rank modules into the pretrained weight. Is this different from sparse LoRA tuning where the corresponding R − ri untrained rows are frozen?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Clear and Compelling Motivation:** The paper is exceptionally well-motivated. It formalizes the *specific failure modes* of prior art: initialization noise and aggregation noise. The entire paper is a clear and focused effort to solve these two problems.\n- **Strong Theoretical Analysis:** The noise analysis theoretically proves that the proposed Fed-PLORA framework eliminates initialization noise and provides a powerful and fundamental justification for the method's design.\n- **Simple and Effective Methodology:** The proposed solution is both elegant and practical.\n- **Comprehensive Empirical Validation:** The experimental results are strong and thorough. The authors test on a wide variety of models and diverse datasets.", "weaknesses": "- Adding pseudocode for the algorithm would improve clarity.\n- It appears that PLoRA requires downloading the entire global LoRA model. In contrast, other methods—if rank is publicly available—can use much smaller downloads. Although Section 4.2 addresses this, the R − ri downlink cost could be quite high when R is large and ri is small, potentially causing synchronization issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A47rfd21Kl", "reviewer": "Reviewer_x1y8", "summary": "The paper proposes Fed-PLoRA, a framework for heterogeneous federated fine-tuning (FFT) of large language models. The key idea is to decompose each LoRA module into multiple parallel one-rank modules (PLoRA) and introduce a Select-N-Fold strategy that folds untrained modules into the frozen backbone to mitigate initialization and aggregation noise. The paper provides a unified noise analysis and reports performance gains over prior methods such as FLoRA, FlexLoRA, and HETLoRA on GLUE, Natural Instructions, and other datasets.", "questions": "1. Please describe in more detail how the non-IID data distribution is constructed, and the definition of heterogeneity ratio.\n2. For a fair comparison, the number of trainable parameters should be explicitly reported in the main experimental results. For instance, what are the trainable parameter counts for each baseline method presented in Table 1, Table 2, and Table 3? This information is critical for interpreting both the performance and communication efficiency.\n3. How is the rank $r_i$ for the $i$-th client determined? Can $r_i$ be adaptively assigned according to the complexity of the client’s local task, or it only depends on the client resources. \n4. In Table 2, FLoRA performs near random guessing on the CoLA benchmark. Could the authors provide an explanation for this? Even though there exists large initialization noise in the stage of broadcast and initialization. \n5. What is the definition of the communication cost in Figure 5, please clarify it or provide the clear reference. Does the communication occur every local training iteration？", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses a timely problem of heterogeneous resource constraints in federated fine-tuning of LLMs. It proposes a modular reformulation, PLoRA, that can in principle generalize to other LoRA-based methods, and introduces a Select-N-Fold strategy to mitigate initialization and aggregation noise. The experiments span multiple models and benchmarks, providing empirical evidence of the proposed method’s efficiency.", "weaknesses": "1. The paper overlooks the training FLOPs and storage overhead of the frozen base model on the client side. From my understanding, each client fine-tunes only its LoRA adapters while keeping the base model parameters fixed. However, even though the base model is frozen, it is still fully involved in the forward passes. When the base model is large (e.g., LLaMA-3.1-8B), its forward FLOPs can dominate the total computation, likely exceeding the resource capacity of many local clients. This makes the proposed setup impractical for real-world federated environments.\n\n2. The implementation code is currently unavailable. Releasing it would greatly improve reproducibility and allow the community to validate its efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)