Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
Authors: Lei Wang
Jieming Bian
Letian Zhang
Jie Xu
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
LLM fine-tuning
Primary area: deep_learning

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but fine-tuning them for domain-specific applications often requires substantial domain-specific data that may be distributed across multiple organizations. Federated Learning (FL) offers a privacy-preserving solution, but faces challenges with computational constraints when applied to LLMs. Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient fine-tuning approach, though a single LoRA module often struggles with heterogeneous data across diverse domains. This paper addresses two critical challenges in federated LoRA fine-tuning: 1. determining the optimal number and allocation of LoRA experts across heterogeneous clients, and 2. enabling clients to selectively utilize these experts based on their specific data characteristics. We propose FedLEASE (Federated adaptive LoRA Expert Allocation and SElection), a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain-specific LoRA experts. It also introduces an adaptive top-$M$ Mixture-of-Experts mechanism that allows each client to select the optimal number of utilized experts. Our extensive experiments on diverse benchmark datasets demonstrate that FedLEASE significantly outperforms existing federated fine-tuning approaches in heterogeneous client settings while maintaining communication efficiency.

Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
Lei Wang University of Florida Gainesville, FL 32611 Heiwang1@uf] @du edu
Jieming Bian
University of Florida Gainesville, FL 32611 j jieming, bian@ufl.ee.mm edu
Letian Zhang
Jie Xu University Florida Gainesville, FL 32611 jie. wee.xuuuffle..
Middle Tennessee State University
Murfreesboro, TN 37132 letian. wwang@mtsu. edu
Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but fine-tuning them for domain-specific applications often requires substantial domain-specific data that may be distributed across multiple organi- zations. Federated Learning (FL) offers a
solution, but faces challenges with computational constraints when applied to LLMs. Low-Rank Adap- tation (LoRA) has emerged as parameter-efficien fine-tuning approach, though a single LoRA module often struggles with heterogeneous data across diverse do- mains. This paper addresses two critical challenges in federated LoRA fine-tuning: 1. determining the optimal number and allocation of LoRA experts across hetero- geneous clients, and 2. enabling clients to selectively utilize these experts based on their specific data characteristics. We propose FedLEASS (Federated adap- tive LoRA Expert Allocation and SElection), a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain- specific LoRA experts. It also introduces adaptive top-M mechanism that allows each client to select the optimal number of utilized ex- perts. Our extensive experiments diverse benchmark datasets demonstrate that FedLEASE significantly outperforms existing federated fine-tuning approaches heterogeneous client settings while maintaining communication efficiency.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, from natural language understanding and generation to reasoning and problem solving [12 38 [ 39 36). Despite their impressive general abilities, these models often require fine-tuning -specic applications and specialized tasks [19]. Fine- tuning adapts pre-trained LLMs to particular domains, enhancing their performance targeted tasks by incorporating domain-specific knowledge and patterns. This process has proven essential for applications in healthcare, finance, law, and science, where specialized expertise is required [1]. However, effective fine-tuning typically requires large volumes of high-quality, domain-specific data, which may be distributed across multiple organizations. In many real-world scenarios, such data cannot be centralized due to privacy concerns or regulatory restrictions [43]. Federated Learning 47 22 25 has emerged as promising solution to this challenge, enabling
to achieve optimal performance
(FL) 3I 28 27
"The first two authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

collaborative model training across distributed data sources without sharing the raw data. In FL, clients train models locally using their private data and share only model updates with a central server, thereby preserving data privacy while leveraging the collective knowledge embedded in the distributed datasets.
The application of FL to LLM fine-tuning presents significant challenges due to the computational and communication constraints inherent in federated settings. Full fine-tuning of LLMs, which typically contain billions of parameters, is prohibitively expensive for many FL clients with limited resources. This has led to growing interest Parameter-Efficte significantly reduce the number of trainable parameters. Among these PEFT approaches, Low-Rank Adaptation (LoRA) [20] has gained substantial traction due to its simplicity and effectiveness. LoRA introduces small trainable low-rank matrices alongside the frozen pre- pre-trained weights, substantially reducing the number of parameters that need to be updated during fine-tuning in Fl. settings 23
Fine- Tuninn (PEFT) methods 19, which
Although LoRA enables efficient domain adaptation, recent studies 15
33 show that a single LoRA
module often falls short in handling heterogeneous domains and complex tasks- especially in FL
settings where clients hold data from distinct domains. Existing FL methods [45] 35 I7 largely rely on a single shared LoRA module across all clients. While some personalized approaches MEl combine global and local LoRA modules to address heterogeneity, such binary designs cannot capture nuanced client similarities. where some clients share domain traits while others diverge significantly, This oversimplification leads to suboptimal knowledge sharing and underutilization of the collective learning potential. On the other hand, using too many LoRA experts- one per client- introducces computational overhead and risks representational collapse due to redundancy [8] These competing constraints give rise to two key research questions: (1) Given heterogeneous client distributions, what is the optimal number of LoRA experts to allocate, and how should clients contribute to their training? Furthermore, client heterogeneity suggests that different clients may benefit from different expert combinations, leading to a second question: (2) Given allocated experts, how can each client dynamically determine the optimal number of experts to use based on its data characteristics?
To address these questions, we conducted extensive empirical analysis with heterogeneous clients in a federated learning environment. Our analysis yielded two significant observations: First, clients with similar domain characteristics should collaboratively train shared LoRA experts, while clients with dissimilar data distributions should contribute to distinct experts. Second, different clients require different numbers of experts to achieve optimal performance, necessitating an adaptive approach to expert utilization rather than a fixed selection strategy. Based on these insights, we propose FedLEASE (short for Federated adaptive Lora Expert Allocation and SElection), novel framework for federated LoRA fine-tuning that systematically addresses both research questions. For thefitt problem of optimal expert allocation, we introduce principled data-driven approach that determines both how many experts are needed and which clients should collaborate on each expert. FedLEASE implements brief initial training phase followed by mathematical clustering of clients based on their LoRA parameter similarity. This process leverages the silhouette coefficient to identify the optimal number of experts while ensuring clients with similar task characteristics contribute to shared experts. For the second problem, we introduce a novel adaptive top-M mechanism that transforms the conventional Mixture-of-Exxe Expetss (MoE) paradigm (T.. While traditional MoE approaches require manually specifying a fixed number of experts (top-k) for all inputs- a significant limitation in heterogeneous federated settings-our adaptive mechanism automatically determines the optimal number of experts for each client based on their specific data characteristics. Through an innovative router architecture that expands the output space from RM Mx to (2M our approach enables dynamic expert selection ranging from single expert to the full ensemble while guaranteeing the inclusion of each client's 's assigned expert. Together, these innovations create a comprehensive solution to the dual challenges of expert allocation and selection in federated LoRA fine-tuning. Our contributions can be summarized as follows:
We identify and formalize two key challenges in federated LoRA fine-tuning: allocation of LoRA experts, and enabling clients to selectively utilize them based on data characteristics.
We propose FedLEASE, novel framework that clusters clients train domain domain-specific LoRA experts and enables flexible expert selection via an adaptive top-M MoE mechanism.
Extensive experiments on diverse benchmarks demonstrate that FedLEASS consistently outperforms existing federated fine-tuning methods, achieving superior performance in heterogeneous settings while maintaining communication efficiency.

---

2 Related Works
Parameter-Ffficient Fine-Tuning: Parameter-effiiien fine-tuning (PEFT) reduces the cost of adapting large language models by updating only a small subset of parameters while freezing the rest [9]. Common PEFT techniques include adapters [13 [4], prefix-tuning [24 26), and low-rank adaptation (LoRA) 120 [29]. LoRA injects trainable low-rank matrices into pre-trained weights, significantly cutting trainable parameters and computation. However, single LoRA module can struggle with diverse domains and complex tasks [I5] 29 37,. prompting Mixture-od of of-Experts (MoE)
extensions that combine multiple small LoRA modules (29 37]. Conversely, too many experts may introduce redundancy and collapse representations (8). These centralized findings motivate our study of optimal LoRA deployment under heterogeneous data distributions in federated learning-
PEFT in Federated Learning. PEFT methods have become particularly suitable for resource- constrained federated learning settings by adjusting only small number of lightweight parameters while keeping most pre-trained parameters unchanged. Various PEFT approaches have been integrated within FL frameworks (3, such as prompt-based fine-tuning 46 [8] and adapter-based tuning techniques 77 6 In this paper, we focus specifically on LoR A-based approaches in FL. FedIT 45 pioneered this direction by combining LoRA with the standard FedAvg algorithm, demonstrating its viability in distributed settings. Subsequent works like [42 [S] attempted to further enhance LoRA in FL by addressing challenges related to inexact server aggregation. Other research efforts [T7] have investigated LoRA's application in data heterogeneous settings, but primarily focused on the relatively simpler label distribution non-IID scenario, where clients share the same underlying task but differ in their label distributions. Our work addresses more complex and realistic scenario where clients may possess data from both similar and different tasks, representing true domain heterogeneity, Unlike prior works that typically employ binary global-local architecture [44], we investigate the fundamental question of determining the optimal number and allocation of LoRA experts given heterogeneous client distributions. Additionally, while existing approaches apply the same aggregation strategy to all clients regardless their data characteristics, our method adaptively determines client groupings and enables adaptive expert selection based on client-specifie needs.
3 Preliminary and Motivation
3.1 LoRA and MoE Integration
Low-Rank Adaptation (LoRA) 220 has been proven to achieve comparable performance to full fine-tuning by inserting trainable low-rank matrices into each layer pre-trained model. For pre-trained model with parameters Wo E Rixd, where d the input dimension and is the output dimension, LoRA introduces two sequen- tial low-rank matrices A E xd and gixr to fit the residual weights for adaptation, where r < min(d, 1). The forward computation is expressed y Wo + BAx, where A is typ- ically initialized with random Gaussian values,
Module with LoRA-MoE
Transformer
Layer
Transformer
Leyer
trained Weight Wo
Tokan Embedding
Router
Input
Figure Illustration of LoRA-MoE mechanism. while B is initialized to zero to ensure stable start to the fine-tuning process. Although LoRA performance is comparable to full fine-tuning in many scenarios, its effectiveness can significantly deteriorate when applied to heterogeneous data containing multiple tasks with different corpora. The performance gap between LoRA and full fine-tuning widens in such complex setting [2].
Recent research (37 29 has explored integrating LoRA with MoE to address multi-domain adaptation challenges. In this integration, each expert in the MoE framework is implemented as a separate LoRA module rather than full neural network. A router network computes routing probabilities and the forward computation for such a LoRA-MoE system can be expressed as:
y = Wo + E((()) B,A,T,
(1)
where px(x) is the routing probability for expert i, and A; E RRrx and B, E Rlxr are the low-rank matrices for the i-th LoRA expert. A extended top-k MoE mechanism selects the LoRA experts
3

---

based on top-k routing probabilities for each input, where i is fixed and pre-defined number. This integration offers significant advantages for handling diverse domains by leveraging different LoRA experts for different input types while maintaining the parameter efficiency of LoRA. In centralized settings, this approach has shown promising results for multi-domain adaptation [37]. However, applying LoRA-MoE in federated learning introduces unique challenges, particularly in determining the optimal number of experts, their allocation across hetcrogeneous clients, and how many experts each client should utilize based on their specific data.
3.2 Heterogeneous Federated Fine-tuning Scenario
Consider a system with N clients, where each client i E (1, N) possesses a local dataset
-. - (1), w)) PPl u with each dataset potentially originating from similar or heterogeneous tasks, The goal of heterogeneous federated fine-tuning is to obtain models for each client that perform well on their respective data distributions. This can be formulated as the following optimization problem: miny L(W)- Σin [D]
Del (W.), where L; the local loss function for client i, |D] = Σ [D:]
the total size of data across all clients, and W (W.) denotes the set of fine-tuned models.
3.3 Observations
The objective of this work is to address two key problems of LoRA in complex heterogeneous federated learning settings: (1) determining the optimal number and allocation of LoRA experts, and (2) enabling each client to selectively utilize these experts according to their specific data characteristics. To investigate these issues, we conduct empirical studies that yield important insights.
Observation 1: Clients with similar tasks//oomins should contribute to the same LoRA expert through averaging, while those with different ones should be assigned to different LoRA experts.
In realistic federated learning settings, client heterogeneity often goes beyond sim- ple label distribution shifts and encom- passes fundamental task differences. To ex- amine this, we designed two experimental scenarios. Scenario 1 involved four clients, each assigned a different GLUE task (SST- 2, QNLI, MRPC, QQP) 40 represent- ing a task- seteroeetrogeeuuu setting. Scenario 2 used four clients all holding data from the same task (QNLI), thereby represent- ing hask-homogeneous setting. For each scenario, we compared two methods: (1) FedIT [45), where each client trains a sin- gle shareable LoRA module that is averaged at the server, and (2) FedLoRA-MoE, where clients train individual LoRA modules without averaging. Instead, all modules are shared, and each client trains a MoE router to dynamically combine its own module with others'.
fe aboite
(a) Scenario 1 task hetero- geneity. Each client holds data from different tasks (SST-2, QNLI, MRPC, QQP).
(b) Scenario task homo- geneity. Each client holds data from the same task (QNLI).
Figure 2: Performance comparison between FedIT and Ood under two scenarios with dif- ferent clients' task heterogeneity.
As shown in Figure 2 our experimental results indicate that in Scenario 1, FedIT performs signifi- cantly worse than FedL.oRA-MoE, suggesting that clients with different tasks struggle to contribute effectively single shared L module. This observation is consistent with findings in centralized settings 15 37. where a single LoRA module proves insufficient for handling diverse domains. Conversely, in Scenario 2, FedIT outperforms FedLoRA-Moff suggesting that using separate LoRA experts for homogeneous clients may be redundant and can degrade performance while increasing inference overhead, which aligns with results from [8) These findings offer key insights into expert allocation strategies under varying task heterogeneity in FL.
Observation 2: Task heterogeneity among clients can be detected through representation similarity of LoRA B matrices after brief local training.
We observe that task similarity between clients can be effectively assessed by computing the cosine similarity of their LoRA B matrices after short period of local training. To validate this, we conducted experiment with four clients: two using the SST-2 dataset and two using the QNLI

---

1.0
10
10
Cosine Smilany
Cosine Smilant
Cosine Smilant
0.542
2
0.0430
0.0303
Clere 2
Class: m
9.4
Clask
622
Layer
Layer
Layer
(a) LoRA matrices.
(b) LoRA matrices.
(c) LoRA BA matrices.
Figure 3: Visualization result of cosine similarity among clients using different LoRA matrices. Note that client 0 and hold data from SST-2 and client 2 and 3 hold data from QNLI.
dataset. As shown in Figure clients working on the same task develop highly similar LoRA B matrices, while those working different tasks exhibit significantly lower similarity. Interestingly, this pattern is exclusive to the LoRA matrices; the A matrices show no consistent relationship with task similarity. This observation supports findings in (37 177, which suggest that the output transformation matrix B captures task-specific information, whereas the input matrix A tends to encode general linguistic features shared across tasks. Although similar task-specific pattern be observed by analyzing the product BA, this requires matrix multiplication and full-rank projection recovery, which incurs significantly higher computational cost. In contrast, using the / matrices alone provides a lightweight yet effective proxy for task similarity, making it more practical in FL.
Observation 3: Clients utilize varying numbers of LoRA experts for optimal performance.
To investigate our second research question, we conducted additional experiments under the task- heterogeneous setting (Scenario 1) using the approach. We varied the top-k parameter in the MoE router, testing values of k 2, 4 (where k 4 corresponds to using all available experts). As shown in Figure 4 different clients achieve optimal performance at different values-sone benefit most from k 2, while others perform better with & or k = 4. This result highlights that even the total number of trained LoRA experts is fixed, clients have varying needs regarding how many experts they should utilize. A static top-k se- lection strategy is therefore suboptimal across all clients. These findings motivate the design of an adaptive expert selection mechanism that dynamically determines the optimal number of experts for each client based on its specific data characteristics.
SST-2 (Client
QNL (Client
89.4
93.3
93.9
93.6
93
91
06
86.0
90
Top-2
Top-4
Top-2
Top-3
MRPC (Cliien
QQP (Cillen 3)
90
83.6
88
87.8
83
87.0
07
85.6
80.9
80.0
Top-2
Top-3
16p4
re
Top-2
100-3
Top-4
Top-2 (wrg: 87.1)
Top-3 (mwg: 87.0)
Top-4 (wwg: 87.1)
Figure 4: Comparison on accuracy of clients using different top-k strategies under the task- heterogeneous setting.
4 Proposed Method
In this section, we describe the framework design of our proposed method FedLEASE by explaining how it addresses the two important challenges in complex federated fine-tuning settings. A theoretical analysis of the proposed method can be found in Section
4.1 Adaptive LoRA Experts Allocation
A fundamental challenge in federated LoRA fine-tuning is determining the optimal number of experts and identifying which clients should contribute to each expert. We address this challenge through a systematic data-driven approach that analyzes similarity patterns in client-specific adaptations. Our method begins with a brief initialization phase where each client E (1, 2, N) independently trains a LoRA module (A,, B,) for E epochs using its local dataset D, This phase serves to capture initial task-spec adaptations the LoRA parameters. Upon completion, each client transmits its LoRA parameters to the central server.
5

---

Initialization
Adaptive LoRA Experts Allocation
Adaptive Top-M Router
Vanilla Top-2 Router
Local Fine-tuning
LORA
LoRA Activated
Clienti
O
Cluster-lavel LallA Expert Router Aggregation
Clustering Similarity Patterns
LoR
LoRA
Clients
Server
Averaged Single LORA
Individual LoRA
Client
Input
Input
Trainable
Active
Inactive
LORA Module
Router
Router Outpuu
Figure 5: Illustration of our proposed adaptive LoRA experts allocation and top-M experts selection mechanism. Average Singe LoRA and Individual LoRA shows the LoRA experts allocation strategies employed by FedIT and FedLoRA- respectively described in Section 3.3 Vanilla Top-2 Router is an example of the MoE-based fixed top-k LoRA experts selection strategy.
Based on the observations in Section 3.3 the B, matrices exhibit distinct similarity patterns that align with underlying task relationships. Accordingly, we define a distance metric between clients and using cosine similarity across all layers:
B B B B
d(i,
L
JEL
where L is the set of model layers, B is the flattened B; matrix layer I, and || denotes the Euclidean norm. To identify the optimal expert allocation, we systematically evaluate all potential clustering configurations. For each possible number of clusters & (2, ........ Mmar), where Mmaz should be set to satisfy the maximum number of LoRA modules given the limited computation resources of clients, we apply Agglomerative Hierarchical Clustering 332 partition clients C* Cluster[[B,]] N d, k), resulting in & clusters C* = {C+, Cg,...,, C
We evaluate the quality of each clustering configuration using the silhouette coefficient 34 which measures how well each client fits within its assigned cluster relative to others. The average silhouette score for a k-cluster configuration is defined as 5(k) = 1/2 Σi=1 (i), where (i) is the silhouette
coefficient for client i given by si) ("(0-a((0) (t), Here, (i) denotes the average dissimilarity between client i and all other clients in the same cluster (intra-clused cohesion), while (1()) is the minimum average dissimilarity between client and clients in other clusters (inter-cluster separation).
The optimal number of experts M arg max (kk is selected the that maximizes the average 2gkgM score. This approach ensures that we identify both the optimal number of experts needed and the most coherent grouping of clients based on their adaptation patterns. A high silhouette score indicates that clients within the same cluster exhibit similar adaptation characteristics, while clients in different clusters show distinct patterns. Once the optimal clustering CMM (CM C.M ..... C. determined, we initialize each LoRA by aggregating the clients within the corresponding cluster:
A
expert
Σ Ar iEC
eexpert
B iECM
Σ
ICM
ICM
The server then distributes the experts along with their cluster assignment information. This approach effectively balances between having too few experts (which would fail capture domain diversity) and too many experts (which would lead to redundancy and inefficient parameter usage).

---

4.2 Adaptive top-M LoRA Experts
Having addressed the first critical question of expert allocation in the previous section, we now turn our attention to the second challenge: how can each client selectively utilize these experts based on its specific data characteristics? This question is particularly important as our observation 3 demonstrated that different clients require different numbers of experts to achieve optimal performance.
After receiving the allocated LoRA experts, each client enters the main training phase with access to all M experts. During this phase, each client only updates its assigned LoRA expert while leveraging knowledge from other experts enhance performance on its local data distribution. The challenge lies in determining how many and which experts each client should selectively utilize, as a fixed top-k selection strategy proves suboptimal across heterogeneous clients. A standard Mixture-of-Expere approach with top-k selection would formulate the forward computation as:
Wox +
Σ
w;B.A,T
(4)
where (w)..... war) denotes the routing weights computed as wi softmax (nftmax(G,z)) (G,z) with G1 E I Mxd being the trainable router.
This standard approach, however, presents two limitations in context: (1) it requires manual tuning of k for each client, which is impractical federated settings, and (2) it does not guarantee the inclusion of the client's assigned expert, which is essential for effective parameter updates. We address these limitations through an innovative adaptive routing mechanism that ensures the client's is assigned expert is always selected while dynamically determining the appropriate number of additional experts to utilize. Instead of employing a conventional router with dimensions RM x we expand the router's output space to R(2M (2 where the first M outputs are connected to the client's s assigned expert, while the remaining M - 1 outputs correspond to the other experts.
Formally, our adaptive routing mechanism is expressed as:
y
Wor +
Σ
{ BJAJæ,
if < M if 2 M
(5)
VE (G.
where w = softmax(Gaat E 22M-1 and denotes the expert index assigned to the client.
An illustration of the proposed adaptive top-M mechanism is shown in Figure 5 The proposed router à E R2M-1 allows each client to decide, for every input, how many and which experts contribute, instead of relying on globally fixed k. When the top-ranked scores lie among the first M entries of W, the computation is dominated by the client's 's own assigned expert Ey. When large scores appear in the remaining M-1 positions. the client leverages additional experts. Intermediate cases arise naturally: if the router selects of the first M entries (i.e., internal components of the assigned expert Fi²) together with M- entries from the other experts, then in effect M M-p+1 unique experts participate in the forward computation.
To make the mechanism more concrete, consider whose assigned expert is Ej. The router produces
with M-3 experts {Ei, E2, Es} and a client
w af
Ei
En
Ea
E
W2
W3
connected to assigned expent E, connected to other E2,
where the first three entries are independently learned routing scores corresponding to distinct internal components of the assigned expert E, (all routed to B, Ajz), and the last two correspond to the other experts Eg and E3 (routed to A2 and Agar). With different input samples inducing different routing score distributions, our proposed top-M mechanism can accordingly select different expert combinations:
Fi Sample 1: if the top-3 are 0f ,0 .6g then only the assigned expert E₁ contributes.
Sample 2: if the top-3 are 4 ,022 F, Fa then two unique experts { (F,, F2) are selected.
Sample 3: if the top-3 are aF. a .F. then all three experts (F1, E2, Ea) participate.
This mechanism guarantees that each client's designated expert F. always participates participates-faciliiiii facilitating stable local updates-ww while allowing flexible, data-driven cooperation with other experts. By enabling
7

---

the router to balance the internal components of Ej with the contributions from the remaining (Emim the method adapts to both imput complexity and inter-client heterogeneity, avoiding any manual tuning of k and achieving effective expert utilization for federated fine-tuning.
4.3 Algorithm Workflow
FedLEASE operates in two phases: initialization and iterative training. A detailed Algorithm | can be found in Section
Server Operations. In the initialization phase, the server clusters clients based on the similarity of their initial B, matrices using the silhouettt-baseed method in Section 4.1 yielding M clusters C
{CI...., Car.. For each cluster Cj. expert parameters (Af 3 D are initialized by averaging LoRA modules within the cluster and distributed to all clients along with cluster assignments.
In each communication round, the server receives updated expert parameters and router networks from clients and performs within-cluster aggregation. The updated experts and routers are then broadcast to clients for the next training round.
Client Operations. Each client begins with E epochs of local fine-tuning obtain (A,, B.). After clustering. the client receives the full set of experts and its assigned cluster ID. During local training, client updates only its assigned expert and the corresponding router G E R22M-1)xd keeping all other experts fixed. Using the adaptive top-M strategy in Section 4.2 each client dynamically determines how many experts to utilize, ranging from just one to all M, based on its local data. Upon completion, only the updated expert and router uploaded to the server.
Experiment
In this section, we evaluate the performance of FedLEASE, against baseline methods on two types of datasets: natural language understanding (NLU) and natural language generation (NLG).
5.1 Training Details
For the NLU task, benchmark [40]. For the NLG task, we adopt LLaMA2 33 as the pre-trained model and fine-tune on the FLAN dataset [10]
RoBERTa 30) the pre-trained model and fine-tune it on the GLUE
NLU Task. We consider 16 clients total, with four clients assigned to each of the four GLUE datasets. Each client's data is randomly partitioned from the corresponding full dataset. RoBERTa- Large (355M) 30 (24 transformer layers) from HuggingFace used as the base model. AdamW adopted as the optimizer for all methods, with batch size of 128, local epochs set to 2, and total of 25 communication rounds. Following (35), LoRA is applied to the query and value projections in the attention layers, and the classification head is frozen after initialization. For our method, the upper bound of experts Maaz is set to 8 and the LoRA rank to 4. Baselines are configured to ensure comparable computational workloads. Learning rates are selected via grid search from E (1E- 4, 3E 4, 5E 4, 1E 3, 3E- 3, 5E-3). Accuracy utilized as the evaluation metric.
NLG Tasks. For NLG tasks, use LLaMA-2-7B 339 with 8-bit quantization (32 transformer layers) from Hugging Face as the base model and select four FLAN TeaassttsTexx Editing, Struet to Text, Sentiment Analysis, and Commonsense Reasoning- to construct heterogeneous client setting. A total of 8 clients are considered, with each dataset assigned to two clients. Each client has 600 training samples and 200 test samples. All methods use AdamW as the optimizer, with a batch size of 8, local epochs set to 2, 10 communication rounds and the upper bound of experts Mmar is set to 8. LoRA is applied to the query and value matrices in the attention layers, with LoRA rank of 8. Learning rates are selected via grid search from E {1E-4, 3E 4, 1E-3, 3, 3E 3, 1E 2). Followed by [4). choose ROUGE-1 the evaluation metric.
Baseline Methods. To assess the effectiveness of FedLEASE, we compare it with the following
state-of-the-art federated LoRA fine-tuning methods: FedIT 45 FedSA [17, FFA-LORA 35), and
FedDPA 44. Additionally, we include clustered federated learning method, IFCA 16 and adapt it with LoRA fine-tuning bascline, denoted IFCA LoRA.

---

5.2 Natural Language Understanding
We evaluate our method on four GLUE 40 benchmark datasets: SST-2, QNLI, MRPC, and QQP. Unlike prior works such as FedSA, which assume clients differ only in label distribution, we adopt more realistic heterogeneous setting, where each client is assigned a different dataset from the four. The setup includes 16 clients in total, with four clients per dataset. Each client's 's data is randomly partitioned from the full corresponding dataset. We use RoBERTa-Laage (355M) 30 from the HuggingFace library as the base model. Additional training details are provided in Section 5.1
Performance Comparison. Table I summarizes the results on the NLU tasks, FedLEASE con- sistently outperforms all baselines, both in terms of average performance and on each individual dataset. Notably, achieves an average improvement of 3.16% over the strongest baseline across the four GLUE tasks. These gains are attributed to two key innovations: (1) clients with similar data distributions are grouped to collaboratively train shared expert, while clients with distinct data contribute to different LoRA experts; and (2) an adaptive top-M expert selection strategy that enables each client to personalize expert usage based on its local data. Although IFCA+LORA incorporates client clustering, it falls short of FedLEASS due to its lack of cross-cluster knowledge transfer, resulting in isolated learning and reduced generalization. In contrast, FeALE allows clients to dynamically leverage experts, facilitating effective cross-task knowledge sharing. These improve- ments are achieved without additional computational or communication overhead, highlighting the scalability and efficiency of FedLEASE.
Ablation on Adaptive Expert Allocation. FedLEASE assigns 16 clients to 4 experts based on clustering results (illustrated in Section B).
Table 2: Ablation on adaptive expert allocation.
Method
Experts
Param
Performance
To evaluate the impact of expert training allo-
Fedl Single
0.1 1.15
82.00
Fedl
0.4426%
83.84
cation, we compare against the following alter- natives: FedLoRA-Sing! trained with contributions from all clients. To ensure fair comparison, we test two variants with different LoRA A ranks: FedLoR A-Single (r = 4) and FedLoRA-Single (r = 16). FedLoRA-Individee Each client trains a separate expert, resulting in 16 experts and one-to-one client-expert mapping. FedLEASE (w/o adaptive top-M): Uses the same expert allocation as FedLEASE but replaces adaptive M router with vanilla fixed 2.p Results in Table 2 show that our clustering- based expert allocation achieves the best performance, even without the adaptive top-M mechanism. Both FedLoRA-Single variants underperform due to limited capacity to model heterogeneous data, while Anddiiii-ndi despite higher computational cost, still lags behind FedLEASE, validating the effectiveness of our clustering strategy in balancing knowledge sharing and task specificity.
80.60
A single expert is
FeaLEASE adaptive top 3d) EdELEASE (Ours)
0.138316
85.91
0.207755
87.76
We also evaluate router aggregation strategies. Compared to maintaining individual routers per client, our aaproach--avragiee router networks within each souu-achieves better performance (Figure 6a confirming the benefit of shared routing among clients with similar data.
Ablation on Adaptive top-M Mechanism. We assess the effectiveness of our adaptive top- M mechanism by comparing it with fixed top-k strategies (top-1 through top-4). As shown in Figure 6b different clients achieve optimal performance with different k values-fre for example, clients with QQP dataset perform best with 2 whereas those with MRPC dataset benefit more from top-4. Notably, all fixed top-k strategies except top-1 achieve comparable performance and outperform baseline methods reported in Table also highlighting the importance of the expert allocation strategy. In contrast, our adaptive top-M mechanism consistently surpasses all fixed strategies across clients, demonstrating its capability to dynamically select the optimal number of experts per input. To further illustrate this adaptivity, Figure 6c visualizes expert selection across layers for 16 clients in our main experiment. The patterns show substantial variation both across clients and across layers within the
Table 1: Performance on GLUE dataset (RoBERTa-Large -arge-3355M.
Methods
% Param
SST-2
QNLI
MRPC
QQP
Average
A
FedIT
45
0.2213%
93,33 : 0.38
85,43
+ 1.41
76.35
73.82 +: 4,01
82,23 +: 2.10
FFA-LORA
35
0.1107%
90.32 + 0.83
77.53 + 2.18
78.45 0.84
77.95
2.15
81.06 + 1.50
-1.17
FedDPA 14
0.2213%
91.90
0.43
13.1300.0
81.0
1.61
81.35
+ 1.22
1.99
+2.26
FedSA [17]
0.2213%
91.97
: 0.81
82,70 1 0.53
82.08
+ 1.51
81.65
+ 1.37
84,60
+ 1.05
+2.37
ORA
16
0.2213%
92.95
0.50
85.90
+ 0.64
78.63
+ 2.38
80.42
+ 1.30
84.48
+ 1.21
+2.25
FedLEASE
0.2075%
93.33
87.22 :: 1.16
86.93
0.68
83.57
0.96
87.76
0.78
+5.53

---

MRPC
FedLEAdE
Lma Fed andiisul tozten
Avergy Accurnty
02
87-22
07.70
SST-2
QNLI
Top
12
3512
007
Awerage
(a) Comparison router ag- (b) Comparison between adaptive (c) Visualization results of ex- gregation strategies.
top-M and top-k mechanisms. pert selection.
Figure 6: Ablation on router aggregation strategies and adaptive top-M mechanism.
same client- client-some some layers rely on a single expert, while others combine multiple. Client ID 0-3, 4-7, 8-11 and 12-15 correspond to SST-2, QNLI, MRPC and QQP, respectively. Specifically, we can observe that: Vertical trend: As layers deepen, the average number of activated LoRA experts tends to increase; 2. Horizontal trend: As task difficulty increases (reflected by decreasing accuracy from SST-2 to QQP in Table D the average number of activated LoRA experts also increases; Cluster-level: Since aggregate the router within each cluster group during server aggregation, rather than maintaining each client's 's individual one, clients within the same cluster tend to exhibit similar but not identical expert selection patterns. This fine- ane-ggained adaptivity underscores the limitations of fixed top-k approaches and confirms the necessity of our adaptive top-M mechanism.
We also conduct sensitivity analysis to evaluate the robustness of our method under various settings. Due to space constraints, results examining the effects of local epochs, LoRA rank, number of clients, data heterogeneity, and the expert upper bound M provided in Section
5.3 Natural Language Generation
In addition to NLU tasks, we evaluate our method on NLG tasks. We adopt LLaMA-2-7B 339 as the base model and use four FLAN datasets-Text Editing, Struct to Text, Sentiment Analysis, and Commonsense Reason- ing-te to construct a heterogeneous client setting. We consider 8 clients in total, with each dataset assigned to two clients. Training details are provided in Section 5.1 outperforms all baselines on NLG tasks. Compared to the strongest overall baseline, FedLEASE achieves gains of 2.26%, 0.46%, 1.43%, and 1.85% on Text Editing, Struct to Text, Sentiment Analysis, and Commonsense Reasoning, respectively. On average, FedLEASE improves by 1.50%, highlighting its ability to handle heterogeneous client data across both classification and generation tasks, demonstrating its generalizability beyond NLU.
Table 3: Performance on FLAN dataset (LLaMA-2-7BB.
Methods
% Param
Text Editing
Struct to Test
Sentiment Anilysis
Commonsense Ressoning
Averuge
FedIT [S
0.0622%
39.30
52.14
43.95
73.95
57.33
HEA-LRRA
85
0.0311%
59.37
50,85
41.25
72.61
56,02
FaIDPA [4
0.06229
65.30
53.40
47.68
72.84
59.81
FedSA
[
0.0622%6
64.82
54,48
46.70
74.31
60.20
16
0.06229
FedLEASE (Ours)
0.0584%
54.94
As shown in Table FedL consistently
6 Conclusion
We presented FedLEASE, novel framework addressing key challenges in federated LoRA fine- tuning for heterogeneous clients. Our approach combines intelligent client clustering for optimal expert allocation with an adaptive top-M mechanism that dynamically determines expert selection based on client-specific needs. Extensive experiments on NLU and NLG tasks demonstrate that FedLEASE consistently outperforms existing approaches across diverse datasets while maintaining communication efficiency. Our method effectively balances knowledge sharing and domain specificity. Future work could explore dynamic clustering techniques, additional parameter-efficien fine-tuning methods, and further communication optimizations for resource-constraing federated settings.
10

---

Acknowledgments
The work Lei Wang, Jieming Bian and Jie Xu partially supported by NSF under grants 2433886, 2505381 and 2519982. The work of Letian Zhang is partially supported by NSF under grant 2348279 and also supported by MTSU Stark Land project.
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I. Aleman, F.L., Almeida, D., Altenschmidr, J., Altman, S., Anadkat, S., al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
[2] Babakniya, S., Elkordy, A.R.. Ezzeldin, Y.H., Liu, Q., Song, K.B., El-Khamy, M., Aves- timehr, S.: Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint PXXi:2338.06622 (2023)
[3] Bian. J., Peng, Y.. Wang. L., Huang. Y., Xu, J.: A survey parameter-efficient fine-tuning for foundation models federated learning. arXiv preprint arXiv:2504.21099 (2025)
[4] Bian, J., Wang, I.., Yang. K., Shen, C., Xu, J.: Accelerating hybrid federated learning con- vergence under partial participation. IEEE Transactions on Signal Processing 72, 3258-3271 (2024). https://doi.org/1... 109/TSP220243400600
[5] Bian, J., Wang, I.., Zhang, I.., Xu, Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement. arXiv preprint ar arXiv:2411. 14961 (2024)
[6] Cai, D., Wu, Y., Wang, S., Lin, F.X., Xu, M.: Fedadapter: Efficient federated learning for modern nip. arXiv preprint arXiv:2205. 10162 (2022)
[7] Chen, H., Zhang, Y., Krompass, D., Gu, J., Tresp, V.: Feddat: An approach for foundation model finetuming in multi-modal heterogeneous federated learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 11285-11299 11293 (2024)
[8] Chen, T., Zhang, Z., Jaiswal, A., Liu, S., Wang, Z.: Sparse moc as the new dropout: Scaling dense and self-slimmable transformers. arXiv preprint arXiv:2303001610 (2023)
[9] Chen, Z.Z.. Ma, J., Zhang, X., Hao, N.. Yan, A., Nourbakhsh, A., Yang. X., McAuley, J., Petzold, L., Wang, W.Y.: survey on large language models for critical societal domains: Finance, healthcare, and law. arXiv preprint arXiv:2405.01769 (2024), https://aarat org/ abs abs/2405.. 01769
[10] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling inlrructionfinnttunn language models. Journal of Machine Learning Research 25(70), 1-53 (2024)
[11] Cui, J., Zhang, W., Tang, J., Tong, X., Zhang, Z., Wen, J., Wang, R., Wu, P., et al.: Anytasktune: Advanced domain-specifie solutions through aas-fine-tuning. arXiv preprint arXiv:2407.0777 .07094 (2024)
[12] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies. volume (long and short papers). pp. 4171-4188 (2019)
[13] Ding, N., Qin, Y., Yang. G.. Wei, F., Yang, Z., Su. Y., Hu, S., Chen, Y., Chan, C.M., Chen, W., et al.: Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5(3), 220-235 (2023)
[14] Fu, Z.,, Yang, H., So, A.M.C., Lam, W., Bing, L., Collier, N.: On the effectiveness of parameter- efficient fine-tuning. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 12799-12880 (2023)

---

[15] Gao, C., Chen, K., Rao, J., Sun. B.. Liu, R., Peng, D., Zhang, Y.. Guo, X., Yang. J., Subrahma- nian, V:: Higher layers need more lora experts, arXiv preprint arXiv:2402.08562 (2024)
[16] Ghosh, A., Chung, J., Yin, D., Ramchandran, K.: An efficient framework for clustered federated learning. Advances in neural information processing systems 33, 19586-19597 (2020)
[17] Guo, P., Zeng, S., Wang, Y., Fan, H., Wang. F., Qu, L.: Selective aggregation for low-rank adaptation in federated learning. In: International Conference on Learning Representations (2025)
[18] Guo, T., Guo, S., Wang, J., Tang, X., Xu, W.: Promptfl: Let federated participants coopera- tively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions on Mobile Computing 23(5), 5 5179-5194 (2023)
[19] Han, Z., Gao, C., Liu, J., Zhang, J., Zhang, S.Q.: Parametee--fffiieet fine-tuning for large models: A comprehensive survey, arXiv preprint 2XXv:2443311468 (2024)
[20] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z.,, Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint urXiv:2106.0968 (2021)
[21] Jordan, M.I. Jacobs, R.A.: Hierarchical mixtures of experts and the em algorithm. Neural computation 6(2), 181-214 (1994)
[22] Karimireddy, S.P., Kale, S.. Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: International conference on machine learning. pp. 5132-13555 5143. PMLR (2020)
[23] Kuang, W., Qian, B., Li, Z., Chen, D.,, Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., Zhou, J.: Federatedscope-IIit A comprehensive package for fine-tuning large language models in federated learning. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 5260-5271 (2024)
[24] Lester, B., AI-Rfou, R., Constant, N.: The power of scale for parameter-efffiient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
[25] Li, T. Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37(3), 50-60 (2020)
[26] X.L., Liang, P.: Prefix-tuning Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
[27] Liu, J., Liu, Y., Shang, F., Liu, H., Liu, J., Feng, W.: Improving generalization in federated learning with highly heterogeneous data via momentum-base stochastic controlled weight averaging. In: Forty-second International Conference on Machine Learning (2025)
[28) Liu, J., Shang, F., Liu, Y., Liu, H., Li, Y., Gong, Y.: Fedbegd: Communication-effnn -ffficiie accelerated block coordinate gradient descent for federated learning. In: Proceedings of the 32nd ACM International Conference on Multimedia. pp. 2955-2963 (2024)
[29] Liu. Q., Wu, X., Zhao, X., Zhu, Y., Xu, D., Tian, F., Zheng, Y.: Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310. 183399 (2023)
[30] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.. Stoyanov, V.: Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 1692 (2019)
[31] McMahan, B., Moore. E., Ramage. D.., Hampson, S., y Arcas, B.A.: Communication-effien efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. pp. 1273-1282. PMLR (2017)
[32] Müllner, D.: Modern hierarchical, agglomerative clustering algorithms. arXiv preprint 109.2:110 (2011)
12

---

[33] Qing. P.. Gao, C., Zhou, Y., Diao, X., Yang, Y., Vosoughi, S.: Alphalora: Assigning lora experts based on layer training quality. arXiv preprint arXiv:2410.10054 (2024)
[34] Rousseeuw, PJJ.: Silhouettes: A graphical aid to the interpretation and valida-
tion of cluster analysis. Journal of Computational and Applied Mathematics 20, 53 65 (1987),
https://wns
sciencedirect cww/ http:///ww.....aa.ao///a/aaaaaan
[35] Sun, Y., Li, Z.,, Li, Y., Ding, B.: Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403. 12313 (2024)
[36] Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805 (2023)
[37] Tian, C., Shi, Z., Guo, Z., Li, L.., Xu, C.: Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404. 19245 (2024)
[38] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.. Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.
(2023)
[39] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0999 .09888 (2023)
[40] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Glue: A multi-task bench- mark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0746 (2018)
[41] Wang, L., Bian, J., Zhang. L., Chen, C., Xu, J.: Taming cross-domain representation variance in federated prototype learning with heterogeneous data domains. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J.. Zhang, C. (eds.) Ad- vances in Neural Information Processing Systems. vol. 37, pp. 88348 88372. Curran As- sociates, Inc. (2024), https: https::/prooeeddingg neurips cc/ facaaer_fflls_ http//ppprrr222/ ffle//aleee225776cc666666663f
Confor--oooeeeeee Conf Confereeeee
[42] Wang, Z., Shen, Z., He, Y., Sun, G., Wang. H., Lyu, L., Li, A.: Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint AXiv:2409.00977 (2024)
[43] Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10(2), 1-19 (2019)
[44] Yang. Y., Long, G., Shen, T., Jiang, J., Blumenstein, M., et al.: Dual-personalizing adapter for federated foundation models. Advances in Neural Information Processing Systems 37, 39409-39433 (2024)
[45] Zhang, J., Vahidian, S., Kuo, M., Li, C.,, Zhang, R., Yu, T., Wang, G., Chen, Y.: Towards building the federatedgpt: Federated instruction tuning. In: ICASSP 2024-2024 IEBE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6915-6919. IEEE (2024)
[46] Zhao, H., Du W., Li, F., Li, Liu, G.: Fedprompt: Communication--ffiiiene and privacy- preserving prompt tuning in federated learning. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp- 1-5. IEEE (2023)
[47] Zhao, Y., Li, M., Lai, I.., Suda, N., Civin, D., Chandra, V: Federated learning with non-iid data. arXiv preprint aivi 808.00682 (2018)
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions- (1) clustering- based LoRA expert allocation strategy and (2) an adaptive top-M expert selection mechanism.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We include limitation disucssion in section E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiaiinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
section presents formal assumptions (smoothness, convexity, bounded sensitivity, and cluster stability) and provides a complete convergence proof for the proposed method.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
we detail the model architectures, datasets, hyperparameters, optimizer settings, and evaluation metries needed reproduce the results.
Justification: In section 5.T
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include our proposed method's 's code in the supplemental material. The official implement can be found at htttps://githubb wwtpshhwu...mmml..maeeemeee wwww/ooommaaaaeeee wang- wttpeiiwwwngninn////eem Liink/FeeLEASS
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc/puulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The training and test details can be found section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Tables 1 and 3 include standard deviations for all performance metrics, and in Section 5 we reference statistical variance across runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The sufficient information on the computing resources can be found sec- 5.1
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper is foundational research which has no negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
18

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper appropriately cites and references all datasets (GLUE, FLAN) and pre- pre-trained models (RoBERTa, LLaMA2).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasesssppprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include the code for proposed method in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hatoge//nearris.cc/oooffeeeeeeee for what should or should not be described.
20

---

A The Algorithm of FedLEASE
The algorithm of proposed FedLEASE is summarized in Algorithm I
Algorithm FedLEASE: Federated Low-Rank Expert Learning
Initialization Phase:
2: Server initializes model parameters (A,, B,J
3: for each client i {L..... ] do
4: Client performs local training on (A,, B,) for E epochs
5: Client i sends trained parameters (A,, to the server
6: end for
7: Server computes distance d(i, j) using cosine similarity:
B
8: d(i,j)= - [
|
9: Server determines optimal number of experts M using silhouette scores
10: Server clusters clients using Agglomerative Hierarchical Clustering:
[Ci....,Cay, f Cluster( (B, [B,) .. M))
12: Server aggregates expert parameters per cluster:
13:
A
[ Eiec, A, B + [ep Eiec, B;
14: Iterative Training Phase:
15: for each communication round t= = 1,2...... do for each client i {1...., N} with i C, do
16:
Client i receives all expert parameters {(A pape BB )
MM
Client uses local router G; E (22M-1)xd
19:
Client i trains assigned expert parameters and router G, locally
Compute adaptive routing with weights às softmax G,æ)) E 22 Output:
21:
expert A, eapen H,
y < M if 2 M
22:
t Woz I EpeTop app
expert B M p- M412, (A BJ) to server
expent
23:
Client uploads updated parameters end for
24:
25:
for each expert j = 1,..., M do
26:
Server aggregates expert parameters: Ba
27:
A, A TCy1 Eiec, A B
[cl +cgl 1
Eiec, Bj
end for end for
29:
B Clustering Results
Our clustering analysis reveals natural groupings of clients based on the cosine similarity of their LoRA B matrices. Figure [ presents comprehensive visualization of the clustering results using three complementary approaches.
Figure 7 aa shows the Silhouette scores for different numbers of clusters, ranging from A: = 2 to = 8. The Silhouette score, which measures how similar objects are to their assigned cluster compared to other clusters, peaks at k 4, indicating that 4 is the optimal number of clusters for the main experiments setting. This finding suggests that clients naturally form 4 distinct groups based on their model adaptations. In Figure (b)), visualize the distance matrix derived from cosine similarity between client LoRA B matrices. The heatmap reveals clear block diagonal structures, which further supports the existence of distinct client clusters. The darker squares along the diagonal represent groups of clients with high intra-cluster similarity, while lighter colors indicate greater dissimilarity between different clusters.
Finally, Figure (c) presents hierarchical clustering dendrogram based on the same cosine similarity measure. The dendrogram provides alternative view of client relationships, illustrating how clients
21

---

progressively merge into larger groups. The vertical axis represents the distance at which clusters are combined, with longer vertical lines indicating greater separation between clusters.
These clustering results provide strong evidence for natural groupings of clients, suggesting an optimal clustering of clients into 4 distinct groups. To ensure robustness of our findings, all results presented are averaged across 5 independent runs with different random initializations.
Distance Cosine Similerity)
Costre Simitamt umtianbty
1.0
Sthowetta
0.08
Distance
0.07
02
1000
(a) Silhouette scores different numbers of clusters, showing opti- mal clustering at 4 clusters.
(b) Heatmap of distance matrix de- rived from cosine similarity between client LoRA matrices.
(c) Hicrarchical clustering dendro- gram based on cosine similarity of client LoRA B matrices.
Figure Visualization client clustering results based on cosine similarity LoRA matrices.
While clustering is indeed an essential component of our proposed method-playing play-ig a key role in expert allocation based client similarity the specific choice of clustering algorithm is not the focus our contribution. Our observations in Section 3.33 pairwise similarity between clients (e.g., via LoRA B matrices), the overall performance is relatively robust to the particular clustering strategy.
3.3 suggest that as long as the method captures
We adopt Agglomerative Hierarchical Clustering due to its ability to operate directly pairwise distances without requiring pre-defined centroids. To validate the generality of our approach, we also applied Spectral Clustering, which similarly supports pairwise similarity inputs, and observed comparable performance. Tables 4 and bellow demonstrate that both clustering methods achieve similar silhouette scores and downstream performance, reinforcing that our performance gains stem primarily from the expert allocation and adaptive top- selection mechanisms, rather than the specific clustering algorithm used.
Table 4: Silhouette Scores Comparison
Clustering Method
2
4
5
6
7
8
Spectral Clustering Agglomerative Hierarchical Clustering
0.0585
0.0820
0.1023
0.0739
0.0637
0.0549
0.0218
0.0599
0.0884
0.1066
0.0965
0.0854
0.0747
0.0660
Table 5: Performance Comparison
Clustering Method
SST2
QNLI
MRPC
QQP
Average
Spectral Clustering
93.97
86.63
86.48
83.40
87.62
Agglomerative Hierarchical Clustering
93.33
87.22
86.93
83.57
87.76
C Sensitivity Analysis
In this section, we perform multiple sensitivity analysis to demonstrate the robustness of our proposed method under different settings.
We vary the number of local training epochs to examine its effect on performance. The results in Figure g show that proposed method consistently outperforms all baseline methods across different local epoch settings (E = 5), confirming that FedLEASE's effectiveness is not dependent on specific epoch configurations.
Impact of LoRA Rank, We test the performance with different LoRA ranks, adjusting the rank parameter for both method and baselines. As shown in Table 6 FedLEASE maintains superior
22

---

07.18
SST-2
QNLI
MRPC
QQP
1.0
Averooc nocuracy
60
Label Distribution
0.8
0.6
76
E-
Number of Local Epochs (E)
2
8 10 11 13 14
FFA-LORA
BBF
FedSA
FedLEASE
Client ID
Figure 8: Impact of Local Epochs.
Figure 9: Illustration of Non-IID Label Distribution.
performance across all tested rank values. The performance gap is particularly notable at lower ranks, highlighting our method's efficiency in parameter utilization.
Table 6: Impact of Rank.
Methods
SST-2
QNLI
MRPC
QQP
Average
SST-2
QNLI
MRPC
QQP
Average
FedIT
45
93.20
80.37
77.35
78.95
82.47
83.88
80.68
77.45
83.74
FFA-LoRA
35
91.83
74.48
74.10
77.10
79.38
91.63
75.50
78.07
74.60
79.95
FedDPA
44
92.63
81.80
80.93
80.30
83.91
91.65
82.25
83.18
78.52
83.90
FedSA 10
92.05
82.13
78.68
80.42
83.32
92.05
82.72
81.10
80.75
84.16
IFCA+LoRA
16
93.33
84.95
77.05
81.82
84.29
93.40
85.90
79.60
82.38
85.32
FedLFASE
93.80
87.32
84.63
83.30
87.26
93.43
86.12
87.42
84.10
87.77
Table 7: Impact of Number of Clients.
==8
N
Methods
SST-2
QNLI
MRPC
QQP
Average
SST-2
QNLI
MRPC
QQP
Average
FedIT 451
93.50
79.63
80.70
78.25
83.02
92.72
81.65
76.67
72.70
80.94
FFA-LoRA
35
91.70
71.10
82.23
75.15
80.05
90.42
79.82
73.53
77.12
80.22
FedDPA
44
92.13
84.17
84.22
80.60
85.28
91.65
78.30
73.75
79.37
80.77
FedSA 17
92.65
84.35
77.10
78.93
83.26
92.25
81.12
75.45
78.73
81.89
LFCA+LORA 16
93.82
85.83
81.07
79.92
85.16
93.75
84.25
81.15
78.73
84.47
FedLEASE
86.88
87.14
87.72
93.25
87.50
86.07
Impact of Client Numbers. To evaluate scalability, we change the number of clients the system to and 32. Table 7 demonstrates that proposed method maintains its performance advantage with different number of clients, indicating robust scalability to federated networks.
Impact of Data Heterogeneity. We evaluate method across varying degrees of data heterogeneity, focusing on realistic task heterogeneity rather than simple label distribution shifts. We consider three unbalanced task distribution settings: (1) Least heterogeneous: 16 clients having 2 kinds of NLU datasets (10 with QNLI and with QQP): (2) Mildly heterogencous: 16 clients having 3 kinds of NLU datasets (4 with SST-2, 7 with QNLI 5 with QQP); and (3) Most heterogeneous: 16 clients having 4 kinds NLU datasets (3 with SST2, 6 with QNLI, with MRPC, 5 with QQP). The results in Table demonstrate that FedLEASE consistently outperforms baseline methods across all heterogeneity levels, with the performance advantage becoming more pronounced as heterogeneity increases.
We further conduct the additional experiments under both task and label non-i.i.d setting, and label distribution is illustrated in Figure main experiment. shown Table
9 Note the task distribution is same as what we used in the 9 our proposed mthod still outperforms other baselines in this
both task and label Non-IID setting.
23

---

Table 8: Impact of Task Heterogeneity.
Methods
(2NTT(
Least Helerogeneous
QQ2 (6)
SST-2 (4)
Mikdly Heterogeneonn
Q2NT (7)
122 (5)
Avg.
SST-2
(2)
Most Helerggenouss
MRR (2)
Q22P (3)
Avg.
FediT
71.78
69.76
92.30
81.71
FeIDPA
$4.20
81.85
83.03
91.78
83.67
79.00
84.82
93.50
82.37
79.90
83.32
84.77
FedSA [x]]
81.74
80.35
82.05
91.10
N0.07
80.50
83.89
91.55
81.92
X3.10
77.92
83.61
JRRA 16
87.75
78.22
82.98
93.50
87.77
82.16
87.81
92.50
86.75
77.50
84.16
85.23
FedE ..S.
87.77
$4.18
85.98
93.85
89.23
84.66
89.25
92.93
88.60
33.05
$4.74
87.33
Table 9: Performance under Task and Label
Non-IID.
Table 10: Performance Comparison of Differ-
Methods
SST-2
QNLI
MRPC
QQP
Average
ent Model Configurations.
FFA-LoRA A5
FediT 555
89.77
93.37
74.93
83.12
76.70
80.73
79.72
72.25
$0.28
82.37
Expert Upper Boand Fixal Nurter Eepors
351-2
QN&I
NIR.C
QOF
Aneexes
79.77
09.25
8036
06.33
$3.50
83.43
$2.11
FedSA 10
87.25
77.85
73.50
74.43
78.26
IFCA+LORA 16
92.95
83.88
80.07
79.00
83.98
(Uval)
98.3
89.22
9639
83.57
97.76
(Ours)
93.60
84.13
Then we perform additional experiments under label non-i..d. setting and follow the exact setup used in FedSA under a Dirichlet (a = 0.5) partitionaing scheme using 20 clients on the QQP dataset. Results in Tables I1 14 demonstrate that FedLEASE consistently outperforms existing methods even under hebeell-ettrogeeouss conditions, further confirming the robustness and generality of our proposed approach.
Table 11: Label Distribution under Dirichlet(O.5) on QQP
Client
Class (S) 0.7
0.4
85.8
92.3
90.9
33.2
99.9
20.6
81.8
85.9
24.6
988
80.5
92.1
93.1
73.2 6.9
Class 1 (9))
99.3
81.9
29.7
92.0
99.5
14.2
1.7
$1.3
9.1
06.8
0.1
79.4
18.2
14.1
75.4
90.2
19.5
7.9
24.8
Table 12: Silhouette Scores under Only Label
Table 13: Cluster Groups under Only Label Non-IID
Non-IID
Clusters
Group
Clients
0.1829
00053
0,0527
Group 0 2,5, 6, 10, 12, 13, 16, 17, 18,
Group 0, 1,3,,,47, 4, 7, 11,
Impact of Expert Upper Bound Mmax- In our main experiments, we set the expert upper bound to Miax 8. To investigate the sensitivity of our approach this parameter, we conducted additional experiments with lower upper bounds (M (Muax 2. 3. 4) using the same experimental setup: 16 clients with data from 4 GLUE datasets. As shown Table 10 we observe comparable performance between M = 8 and Mmax = 4 configurations. This aligns with our clustering analysis in Section B which indicates that this particular configuration requires only 4 experts. This finding validates that method can efficiently determine the appropriate number of experts needed even when the budget (Mmax) exceeds the system's actual requirements. When restricted to Mmax < 4, we observe performance degradation compared the M,ax or Mmax settings, further confirming the importance of allocating an adequate number of experts. Nevertheless, it is noteworthy that even with this constrained expert budget, our proposed method still outperforms all baseline methods. This demonstrates the robustness of our approach and its ability to make efficient use of even limited expert resources through effective allocation and adaptive selection.
D
Computational Overhead
Our clustering step is performed only once during the initialization phase and not repeated during iterative training. Thus, its runtime impact is negligible. As observed in Section 3.3 using only the LoRA matrices offers an efficient and lightweight proxy for task similarity, given their small size compared to full model weights or BA products. As shown in Table we measured the clustering time (3.11 seconds on Intel Xeon Platinum 8570 CPU), which is significantly shorter than the total training time (193.49 seconds with local training the NVIDIA B200 GPU);

---

Table 14: Performance under Only Label Non-IID
Method
FedIT
FFA-LORA
FedDPA
FedSA
IFCA+LORA
FedL.FASE
Accuracy (%)
83,52
83.05
85.78
86.85
84.73
89.23
Table 15: Comparison of Computing Time
Time (S)
FedIT
FFA-LoRA
FedDPA
FedSA
IFCA+LORA
FedLEASE
Local Per-Epochh Training Time
3.75
3.41
3.82
3.72
3.85
3.78
Glohal Aggregation Time
0.048
0.038
0.051
0.042
0.061
0.055
Clustering Time
2.77
3.11
Total Training Time
188.70
171.45
192.28
187.05
263.28
193.49
E
Limitations
While RedLEASE achieves strong performance in heterogeneous federated fine-tuning, it has lim- itation. The current framework assumes a static client population and fixed expert assignments throughout training. In practical federated environments where client availability and data distri- butions evolve over time, this rigidity may limit adaptability, Future work could explore dynamic clustering or meta-routing strategies to accommodate such non-stationary conditions.
Convergence Analysis
In this section, we analyze the convergence properties of our proposed FedLEASE method.
We define the following notation:
C (C1, C2, Cr, Cur} represents the partition of clients into M clusters
0f = {A B,, G.) denotes the trainable parameters for client at round
0 (Agapertit B Bexpert,t Guupert.t) denotes the aggregated parameters for cluster round é
For any client E Cj, we define j(i) = j as the cluster it belongs to
The cluster-level parameters are computed by averaging the parameters of all clients in the cluster:
0 = C iEC
0
We denote the client-level loss function as f:(0:11(x)M ,) and the cluster-level loss function as:
1((0)(())M))
C, iEc
F.1 Assumptions
To establish convergence, we make the following assumptions: Assumption F.1 (Client-Level Smoothness). For each client i, the loss function fi is p-smooth with respect 0, i.e., for any 07, 02:
(f.()) V f(( S m|10 -0?11
(8)
Assumption F.2 (Client-Level Strong Convexity). For each client the loss function fo is X-strongly convex with respect to 0,, i.e., for any 0.. 07:
f.(07) 2 f.(0)) (V.(())) 67 07) > 2 0:
(9)
25

---

Assumption F.3 (Bounded Expert Sensitivity). The optimal parameters for client are sensitive changes in the expert parameters with a bounded Lipschitz constant. For any two of expert parameters [0 and (0)) k=:
- s
Σ O - fll
(10)
where ((00)) represents the optimal parameters for client i given fixed expert parameters. Assumption E4 (Cluster Assignment Stability). After the initial clustering phase, the assignment of clients to clusters remains stable throughout the training process.
Theorem E5 (Convergence of FedLEASE). With the assumptions of client-level smoothness, client- level strong convexity, limited inter-cluster influence, and cluster assignment stability, we can derive:
2e
1 - S 1- 3M
(3MY' max [10] - -
If 3M < 1 and the local training at each round converges neighborhood of the optimal solution, then the sequence of cluster models {θ]} generated by FedLEASE converges to a stable point for each cluster j.
F.2 Proof
At round f, each client performs local training to update its parameters. Let 0:²0 represent the parameters of client after steps of local training within round
The gradient descent update rule for client at step
0:0011 00 n SA ((0
(11)
From Assumption (client-level smoothness), we have:
f1 $ f((0)-)) (VI(())")) 0: -::00) + 11 "1100.011
1100
8/:011
(12)
2
- - - um?
IV [(
(13)
2
= - (111 2 (1
(14)
Let 0: 07 ((O))) denote the optimal parameters for client given the fixed expert parameters round f. From Assumption (client-level strong convexity), we have:
f(0)) 2 f((0.-) ( (V())))) - 07:0) + $2 10 1
(15)
Then we can establish:
V f - f((11))
(16)
Substituting this into our earlier inequality:
1(005*+1= SA(07) S S((0)") - 11(0)) - n111 pm V [/A(0:'))
(17)
2
< f((00))) - f((0)) - 271(1 1m7 (f((0 f((0)))
(18)
(1 2m 19 )) f((07))
(19)
Denoting p == (1 2n)2111111111 )), with properly chosen learning rate 1 < 2, # recursively applying this incquality for s steps:
we have p E (0, By
f((0,-)) f((0t) (f((())) f((0)))
(20)
26

---

From strong convexity (Assumption 2). we can relate the optimality gap in function value to the parameter distance:
A
$10 -1 - f.(0%-) - f(((,)
(21)
2
Therefore:
2p" 100 01112 - X (S((0, - S((2)))
(22)
With sufficient number of local training steps S, we can ensure:
||05+1 8 ((+))) S
(23)
where 0 = 0g. and E can be made arbitrarily small by increasing S.
This establishes that each client's model converges approximate optimal solution for fixed expert parameters.
Now we analyze the stability of cluster models across communication rounds. For cluster j. with the assumption of Cluster Assignment Stability, the aggregated parameters after round t are:
e 1
0,
(24)
Cg
VEC
The difference between consecutive cluster models is:
1
10,
0
(25)
CCJ
IEC
REC,
110
0:
(26)
C2
SEC
For each client i E Cj. we have:
|10+t1
81 S |10 (T0 (10 (7(10) + (0:((0 "1) - 0:11 (27)
S E 8
1 0%
1|1 e
(28)
M
= 22 + 3
10/ - 0
(29)
k=1
Let max; G and A maxy |10/|1 - 0/1l Substituting into cluster difference bound:
1
10 5
(2e +
3
10 - -
(30)
[2
iEC
M
2r + ,
1 1
(31)
k=1
2e + 3MA 1-1
(32)
Taking the maximum over all clusters:
A < 2e + -3M
(33)
When 3M 1, this is a contraction, and by iterating:
[-1
A < 2c
(3M) + (3M))
(34)
i=U
< 2c
+ (BM)'
(35)
3M
27

---

As t + 00, A + 2e
I-BM which can be made arbitrarily small by increasing local training steps
(reducing c).
Our analysis demonstrates that FedLEASE converges both client and cluster levels:
1. Each client's model converges approximate optimal solution with error bounded by ca
2. The cluster models stabilize with a maximum change between rounds bounded by 1-9M
2c
The convergence is guaranteed when:
The learning rate is appropriately chosen (1 < 3
The inter-cluster influence is limited (3M
Sufficient local training steps performed (to reduce e)
28


Reviews:
[{"review_id": "YBsJfSrCdu", "reviewer": "Reviewer_Wn44", "summary": "This paper proposes FedLEASE, a federated LoRA fine-tuning framework that tackles two key challenges in heterogeneous federated learning. FedLEASE clusters clients using the similarity of their LoRA B matrices to determine expert allocation. Additionally, it introduces an adaptive top-M mechanism that enables each client to dynamically select the number of experts to use. Experimental results across various NLU and NLG datasets show that FedLEASE consistently outperforms existing baselines.", "questions": "1. How does FedLEASE compare to baseline methods in terms of computational cost and training time?\n2. Is the router network maintained locally by each client or distributed from the server? Algorithm 1 indicates that clients reinitialize the router at each communication round.\n3. In Table 5, when the number of clients changes (e.g., from 8 to 32), does the optimal number of experts adjust accordingly?\n4. In addition to the number of experts each client uses, it would be helpful to show which experts are selected by each client. A discussion on the frequency and distribution of expert selection across clients would provide deeper insights.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper identifies two major challenges in federated LoRA fine-tuning and supports them with insightful preliminary experiments. \n2. The adaptive top-M routing mechanism enables client-specific expert selection, which improves personalization over fixed top-k strategies.\n3. FedLEASE is evaluated on both NLU and NLG tasks, and the paper includes comprehensive experiments that validate the effectiveness and individual contributions of each component.\n\nWeaknesses\n1. The clustering phase performs clustering based on the LoRA B matrices from all clients, which could introduce computational overhead, especially in large-scale settings.\n2. The ablation studies are not sufficiently thorough. For example, alternative clustering methods are not explored, making it unclear whether Agglomerative Hierarchical Clustering is the optimal choice.", "was_revised": true, "final_justification": "The authors have responded thoughtfully to my concerns. In particular, their clarification on computational overhead and training time, which is critical in the context of federated learning, was appreciated. Based on these improvements, I have decided to maintain my score.", "rebuttal": "Thank you for your valuable comments!\n\n**W1: LoRA B clustering computational overhead.**  \nThe clustering step is performed **only once** during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As observed in §3.3, using only the **LoRA B matrices** offers an efficient and lightweight proxy for task similarity, given their small size compared to full model weights or BA products.  \nTo further support this, we measured the clustering time (**3.11** seconds on Intel Xeon Platinum 8570 CPU), which is significantly shorter than the total training time (**193.49** seconds with local training on the NVIDIA B200 GPU):\n\n| Time (s)            | FedIT | FFA-LoRA | FedDPA  | FedSA   | IFCA+LoRA | FedLEASE |\n|---------------------|---------------|----------|---------|---------|-----------|----------|\n| Local Per-Epoch Training Time | 3.75          | 3.41     | 3.82    | 3.72    | 3.85      | 3.78 |\n| Global Aggregation Time     | 0.048         | 0.038    | 0.051   | 0.042   | 0.061     | 0.055 |\n| Clustering Time        | -             | -        | -       | -       | 2.77  | 3.11 |\n| Total Training Time     | 188.70    | 171.45 | 192.28 | 187.05 | 263.28 | 193.49 |\n\n\n---\n\n\n**W2: Alternative clustering methods are not explored.**  \nWhile clustering is indeed an essential component of our proposed method—playing a key role in expert allocation based on client similarity—the specific choice of clustering algorithm is not the focus of our contribution. Our observations (§3.3) suggest that as long as the method captures pairwise similarity between clients (e.g., via LoRA B matrices), the overall performance is relatively robust to the particular clustering strategy.\n\nWe adopt Agglomerative Hierarchical Clustering due to its ability to operate directly on pairwise distances without requiring pre-defined centroids. To validate the generality of our approach, we also applied Spectral Clustering, which similarly supports pairwise similarity inputs, and observed comparable performance (within 0.14% accuracy difference).\n\nThe tables below demonstrate that both clustering methods achieve similar silhouette scores and downstream performance, reinforcing that our performance gains stem primarily from the expert allocation and adaptive top-$M$ selection mechanisms, rather than the specific clustering algorithm used.\n\n**Silhouette Scores:**\n\n| Method / Number of Clusters                        | 2      | 3      | 4 (best) | 5      | 6      | 7      | 8      |\n|------------------------------|--------|--------|----------|--------|--------|--------|--------|\n| Spectral Clustering           | 0.0585 | 0.0820 | 0.1023   | 0.0739 | 0.0637 | 0.0549 | 0.0218 |\n| Agglomerative Hierarchical   | 0.0599 | 0.0884 | 0.1066   | 0.0965 | 0.0854 | 0.0747 | 0.0660 |\n\n**Final Accuracy Comparison:**\n\n| Method / Task                      | SST-2 | QNLI  | MRPC  | QQP   | Average |\n|----------------------------|-------|-------|-------|-------|---------|\n| Spectral Clustering         | 93.97 | 86.63 | 86.48 | 83.40 | 87.62   |\n| Agglomerative Hierarchical | 93.33 | 87.22 | 86.93 | 83.57 | 87.76   |\n\n---\n\n**W3: Comparison of computational cost and training time.**  \nAs discussed in our response to W1, FedLEASE introduces only minimal additional overhead. The clustering step is performed only once during initialization, taking only 3.11 seconds on CPU, which is negligible compared to the total training time (193.49 seconds with local training on GPU, as shown in the above Table in W1).  \n\nDuring training, each client updates only its assigned LoRA expert and router, keeping the per-round computation costs on par with baselines such as FedIT and FedSA.  Importantly, in both GLUE and Flan experiments (**Table 1** and **Table 3** in the main paper), we ensured that the total number of trainable parameters in FedLEASE is comparable to or even smaller than those of baseline methods. Despite this, our method consistently achieves superior performance, validating the efficiency of our expert allocation and adaptive top-$M$ design.\n\nFurther, ablation studies (**Table 2** and **Figure 6**) confirm that the adaptive top-$M$ mechanism provides substantial performance gains without increasing training workload, demonstrating both the computational efficiency and effectiveness of our proposed approach.\n\n---\n\n**W4: Clarification on router reinitialization.**  \nThank you for pointing this out. The router is indeed maintained locally by each client throughout training. The statement in Algorithm 1 suggesting per-round reinitialization was a typo, in fact clients initialize their router only once at the beginning.\n\nWe will correct this in the final version of the paper. Additionally, our ablation study in **Figure 6(a)** explicitly demonstrates that maintaining the router locally leads to better performance, supporting our design choice.\n\n---\n\n**W5: Does the optimal number of experts vary with number of clients?**  \nIn the experiments where the number of clients varies, the underlying data distribution remains fixed across four distinct datasets (SST-2, QNLI, MRPC, QQP). As a result, our clustering procedure consistently identifies 4 clusters, which aligns with the task/domain-level heterogeneity rather than the client count.\n\nImportantly, the clustering is based on pairwise similarity of LoRA B matrices, a core component of our method motivated by prior observations that LoRA B captures task-specific information (§3.3). Overall, the number of experts is determined by the **intrinsic task-level differences** across clients. If additional tasks or datasets are introduced, our clustering procedure would naturally detect a different number of clusters as needed.\n\n---\n\n**W6: Frequency and distribution of expert selection.**  \nWe provide additional analysis showing **normalized expert selection percentage** at different layers for each client. This allows us to capture relative expert usage, especially when clients assign multiple experts.\n\nOur findings reveal several important patterns that aligns with Figure 6(c) in our paper:\n\n- **Deeper layers** (e.g., Layer 23) show strong reliance on the expert corresponding to the client’s assigned cluster, indicating more specialized and task-specific behavior as depth increases.\n- **Shallower layers** (e.g., Layer 0) tend to involve a more diverse set of experts, reflecting greater need for knowledge transfer across domains in earlier representations.\n- Despite local updates of the router networks clients within the same cluster show highly similar expert selection patterns, validating the effectiveness of the initial clustering based on LoRA B similarity.\n\nThus we can find that the number of experts selected varies across layers—shallow layers often draw from more experts than deeper ones. This confirms the flexibility and effectiveness of our adaptive expert routing mechanism. Unlike traditional fixed top-$k$ MoE strategies—which constrain every data sample, every layer to a fixed number of experts—our method enables fine-grained adaptivity across samples and layers, offering a much more flexable approach to expert utilization in federated fine-tuning.\n\n| Client ID | Cluster Group | Layer 0                             | Layer 11                            | Layer 23                            |\n|-----------|---------------|-------------------------------------|-------------------------------------|-------------------------------------|\n| 0         | 2          | [0.146, 0.157, 0.602, 0.095]        | [0.031, 0.011, 0.932, 0.026]        | [0.001, 0.003, 0.993, 0.002]        |\n| 1         | 2          | [0.134, 0.143, 0.632, 0.090]        | [0.033, 0.012, 0.927, 0.028]        | [0.001, 0.003, 0.994, 0.002]        |\n| 2         | 2         | [0.129, 0.139, 0.642, 0.089]        | [0.033, 0.012, 0.925, 0.029]        | [0.001, 0.003, 0.993, 0.002]        |\n| 3         | 2  | [0.131, 0.141, 0.638, 0.090]        | [0.031, 0.014, 0.927, 0.028]        | [0.001, 0.003, 0.994, 0.002]        |\n| 4         | 0  | [0.514, 0.158, 0.166, 0.162]        | [0.875, 0.082, 0.015, 0.029]        | [0.885, 0.073, 0.014, 0.028]        |\n| 5         | 0    | [0.513, 0.158, 0.166, 0.162]        | [0.877, 0.082, 0.013, 0.029]        | [0.884, 0.075, 0.014, 0.027]        |\n| 6         | 0    | [0.505, 0.160, 0.169, 0.165]        | [0.872, 0.085, 0.014, 0.029]        | [0.884, 0.072, 0.015, 0.029]        |\n| 7         | 0       | [0.510, 0.159, 0.168, 0.164]        | [0.876, 0.082, 0.014, 0.028]        | [0.885, 0.075, 0.013, 0.028]        |\n| 8         | 3             | [0.133, 0.102, 0.083, 0.682]        | [0.057, 0.079, 0.083, 0.780]        | [0.072, 0.031, 0.134, 0.763]        |\n| 9         | 3     | [0.124, 0.100, 0.079, 0.696]        | [0.057, 0.077, 0.084, 0.783]        | [0.073, 0.031, 0.136, 0.760]        |\n| 10        | 3             | [0.133, 0.102, 0.083, 0.682]        | [0.056, 0.079, 0.083, 0.782]        | [0.074, 0.031, 0.134, 0.762]        |\n| 11        | 3             | [0.131, 0.101, 0.082, 0.685]        | [0.056, 0.078, 0.083, 0.783]        | [0.074, 0.031, 0.133, 0.763]        |\n| 12        | 1             | [0.098, 0.618, 0.181, 0.103]        | [0.150, 0.795, 0.043, 0.011]        | [0.027, 0.878, 0.006, 0.089]        |\n| 13        | 1             | [0.098, 0.617, 0.182, 0.103]        | [0.148, 0.798, 0.041, 0.012]        | [0.028, 0.878, 0.005, 0.090]        |\n| 14        | 1             | [0.097, 0.631, 0.172, 0.100]        | [0.148, 0.794, 0.045, 0.013]        | [0.028, 0.877, 0.006, 0.089]        |\n| 15        | 1             | [0.099, 0.614, 0.184, 0.104]        | [0.151, 0.799, 0.038, 0.011]        | [0.026, 0.876, 0.005, 0.093]        |\n---\n\nWe thank the reviewer for the valuable comments and will incorporate these clarifications and additions in the final version."}, {"review_id": "06EG5z8PQO", "reviewer": "Reviewer_zQ88", "summary": "This paper presents FedLEASE, an innovative federated learning framework designed to address the challenges of LoRA-based fine-tuning in heterogeneous client environments. FedLEASE mainly includes two key innovations: optimizing expert allocation by grouping clients based on parameter similarity; and a dynamic top-M mechanism that adaptively determines the optimal number of experts per client, overcoming the limitations of fixed selection strategies.  Extensive experiments demonstrate FedLEASE's superior performance across both natural language understanding (NLU) and generation (NLG) tasks while maintaining communication efficiency.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-structured, with clear explanations of the methodology and experiments.\n- The proposed adaptive top-M MoE mechanism is innovative and well-justified by empirical results.\n- The paper provides the theoretical analysis in the Appendix, ensuring the robustness of the proposed method.\n- The ablation studies validate the proposed method's effectiveness and scalability.\n\n**Weaknesses**\n\n- Some parts of the writing in the paper are confusing.  \n    - First, regarding the adaptive top-M MOE scheme, according to the paper's description, M clusters are obtained through hierarchical clustering, and the protocol of each cluster is used as the initialized LoRA experts. On each client, the LoRA belonging to itself is replicated M times to obtain 2M-1 output routers, thereby allowing each client to dynamically determine the final number of experts used based on its local data. So, what does the \"fixed Top-K\" mentioned in the paper refer to? Based on the above explanation and my understanding, it should result in K clusters and K LoRA experts, with each client obtaining K output routers. However, in the top-right corner of Figure 5, there are 3 experts and 2 output routers, which is perplexing. Perhaps I missed something, or maybe the confusion arises from inconsistent notation in the method description.\n\n    - Second, regarding the 2M-1 output routers, why is the expert belonging to the client replicated M times? The authors could provide a more intuitive explanation rather than just presenting empirical results. Additionally, each client essentially still selects M experts; the unequal number of experts utilized is due to the replication of its own expert M times. Therefore, I believe attributing the dynamic selection of expert numbers as the reason for achieving optimal personalized performance is not entirely appropriate. The authors might need to adopt a more suitable explanation.\n\n- Concerning the selection of baselines in the experiments. The proposed method has two contributions: one is hierarchical clustering with the silhouette score as the metric to determine the optimal number of experts, and the other is the adaptive top-M MOE. However, the baselines are either FL+LoRA or clustered FL (IFCA)+LoRA, neither of which includes MOE LoRA techniques. Thus, the experimental comparisons are not entirely fair. For example, in the comparison with IFCA+LoRA, it is unclear whether the performance improvement comes from hierarchical clustering, MOE, or the adaptive top-M MOE, making it difficult to quantify these incremental contributions. Nevertheless, it is commendable that the authors demonstrate the advantages of hierarchical clustering for obtaining LoRA experts and the adaptive top-M MOE in Table 2.\n\n- The experimental section only reports the number of parameters, but does not present runtime or communication efficiency. Since the proposed method uses hierarchical clustering to determine the optimal number of experts, this process could be highly time-consuming when the number of clients is large. \n\n- The dataset partitioning in the experiments reveals a natural clustering structure, which explains why the proposed (clustering-based) method achieves strong performance. However, data heterogeneity in FL is typically complex and may exhibit very ambiguous clustering structures. If possible, I would like to know how the proposed method performs under more commonly used Dirichlet partitioning (or other heterogeneous partitioning schemes).", "was_revised": true, "final_justification": "The authors have added useful results.", "rebuttal": "Thank you for your thoughtful comments!\n\n**W1: Clarification on \"fixed top-$k$ MoE\" vs. our proposed \"adaptive top-$M$ MoE\".**  \nTo clarify, the term \"fixed top-$k$ MoE\" in our paper does **not** refer to the number of clusters or LoRA experts being $k$. Instead, it follows the conventional MoE paradigm where, given $M$ experts, each input is routed to the **top-$k$ experts** based on routing scores (i.e., the same fixed number $k$ is used for all data across all clients.)\n\nIn contrast, our proposed ''adaptive top-$M$ MoE'' introduces a more flexible and input-adaptive routing mechanism. Each client is first assigned to a primary expert via clustering. Then, the client’s router network $G_i \\in \\mathbb{R}^{(2M-1) \\times d}$ produces **$2M - 1$** routing scores:\n- The **first $M$ scores** correspond to **distinct internal components of the assigned expert**, i.e., they are all routed to the same expert but model different aspects (not different experts).\n- The **remaining $M - 1$ scores** are associated with the other experts.\n\nThese $2M - 1$ values are learned via a router network $G_i \\in \\mathbb{R}^{(2M-1) \\times d}$, and selection is conducted via softmax followed by a top-$M$ selection.\n\nTo illustrate with an example where $M = 3$ with experts $[E1, E2, E3]$ and $E1$ is the assigned expert, the router may output scores like:  \n$[ω₁^{E1}, ω₂^{E1}, ω₃^{E1}, ω₁^{E2}, ω₁^{E3}]$\n\nHere:\n- $ω₁^{E1}$, $ω₂^{E1}$, and $ω₃^{E1}$ are three independently learned scores for the **assigned expert E1**, capturing different aspects of its contribution.\n- $ω₁^{E2}$ and $ω₁^{E3}$ correspond to the other experts.\n\nDepending on the input samples, the output scores can change and the top-M selection might yield:\n- **Sample 1**: ${ω₁^{E1}, ω₂^{E1}, ω₁^{E2}}$ are the top-$M$ highest scores → 2 **unique** experts $E1, E2$ are selected\n- **Sample 2**: ${ω₁^{E1}, ω₂^{E1}, ω₃^{E1}}$ are the top-$M$ highest scores → only 1 **unique** expert $E1$ is selected\n- **Sample 3**: ${ω₂^{E1}, ω₁^{E2}, ω₁^{E3}}$ are the top-$M$ highest scores → all 3 experts are selected\n\nThis design has two main benefits:\n1. It **guarantees participation of the assigned expert** for every client.\n2. It allows **fine-grained and input-dependent expert allocation**, supporting different levels of specialization across inputs and across layers.\n\nWe appreciate the reviewer’s comment and acknowledge that the explanation in the current version could be confusing. In the final version, we will revise the description of this mechanism to make it explicitly clear.\n\n---\n\n**W2: Choice of MoE + FL baselines.**  \nWe appreciate the reviewer’s concern regarding baseline selection. To our knowledge, there is currently no prior work that integrates MoE with federated LoRA fine-tuning. Given this gap, we selected recent state-of-the-art FL + LoRA methods for comparison, including:\n- FedIT (ICASSP 2024), which uses single global LoRA\n- FFA-LoRA (ICLR 2024), which freezes LoRA A to reduce server aggregation noise\n- FedSA (ICLR 2025), which applies selective aggregation\n- FedDPA (NeurIPS 2024), which introduces dual-personalized adapters\n\nThese methods are all published in top-tier conferences. Besides of these FL + LoRA methods, we also modified a tradtional clustered FL method (i.e. IFCA) with LoRA as an additional baseline. Our goal is to go beyond these works by addressing both (1) **expert allocation** (via clustering), and (2) **expert utilization** (via adaptive top-$M$ routing).\n\nTo further isolate and demonstrate the benefit of our adaptive top-$M$ MoE, we conducted direct comparisons in the **“Ablation on Adaptive top-M Mechanism”** section (see **Figure 6(b)**), where we compare our full method against a variant that uses **fixed top-k MoE + adaptive expert allocation**. This ablation serves as the closest proxy to a potential MoE + FL baseline, and results show that our adaptive routing mechanism consistently outperforms all fixed-k alternatives, regardless of the value of $k$. \n\nAdditionally, **Table 2** in the main paper provides a clear breakdown of performance contributions from clustering alone (i.e., expert allocation).\n\n---\n\n**W3: Runtime and clustering computational overhead.**  \nRegarding the training and clustering overhead, the clustering step is performed **only once** during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As described in §3.3, using only the **LoRA B matrices** provides an efficient and lightweight proxy for task similarity, due to their significantly smaller size compared to full model weights or LoRA A–B combinations.\n\nTo further support this, we measured the clustering time as **3.11** seconds on the Intel Xeon Platinum 8570 CPU, which is substantially shorter than the total training time (**193.49** seconds with local training on an NVIDIA B200 GPU). The detailed timing comparisons are provided below:\n\n| Time (s)                        | FedIT | FFA-LoRA | FedDPA | FedSA  | IFCA+LoRA | FedLEASE |\n|---------------------------------|-------|----------|--------|--------|------------|----------|\n| Local Per-Epoch Training Time   | 3.75  | 3.41     | 3.82   | 3.72   | 3.85       | 3.78     |\n| Global Aggregation Time         | 0.048 | 0.038    | 0.051  | 0.042  | 0.061      | 0.055    |\n| Clustering Time | –     | –        | –      | –      | 2.77          | 3.11     |\n| Total Training Time             | 188.70| 171.45   | 192.28 | 187.05 | 263.28     | 193.49   |\n\nWe will include the table in the final version to clearly demonstrate that FedLEASE maintains high performance while incurring only modest runtime overhead.\n\n---\n\n**W4: Dataset partitioning: Task heterogeneity vs. label non-IID.**  \nWhile we acknowledge that traditional federated learning often uses Dirichlet partitioning to simulate label non-IID settings, our task-heterogeneous experimental setup is designed to more accurately reflect real-world scenarios for large language model fine-tuning and training. In practical deployments, clients often hold data from fundamentally different **domains and tasks**, such as:\n- Customer service platforms with dialogue transcripts  \n- Educational institutions with exam question-answering datasets  \n- Medical research labs with clinical trial summaries  \n- Legal departments with contract summarization tasks  \n- E-commerce companies with product review classification\n\nSuch **task-level heterogeneity** presents a more **challenging and realistic** federated fine-tuning setting (similar setting also considered by FedDPA) than simple label skew within the same task.  \n\nWe also jointly evaluated our method under **label non-IID conditions**, as shown in **Figure 9** and **Table 7**, where FedLEASE consistently outperforms all baselines even in these skewed scenarios. For these binary classification tasks (e.g., SST-2, MRPC), we followed the **non-IID setup used in prior work** (FedSA, FFA-LoRA) instead of applying Dirichlet partitioning, which may be less meaningful for such binary classification tasks.  \n\n---\n\nWe thank the reviewer for these thoughtful comments and hope our detailed clarification addresses each point clearly. We will include these clarifications and results in the final version of the paper."}, {"review_id": "1bEtjpS0gc", "reviewer": "Reviewer_j5zx", "summary": "This paper introduces FedLEASE, a novel framework for federated fine-tuning of large language models (LLMs) using adaptive LoRA. FedLEASE tackles two key challenges in federated settings: (1) how to allocate LoRA experts across heterogeneous clients, and (2) how clients can selectively use relevant experts. The method clusters clients by representation similarity and assigns domain-specific LoRA experts accordingly. It also employs an adaptive top-M Mixture-of-Experts mechanism to optimize expert selection per client. Experiments show that FedLEASE outperforms existing methods on diverse datasets, achieving better performance under data heterogeneity while maintaining communication efficiency.", "questions": "1) System Overheads (Computation, Memory, Communication): The proposed FedLEASE framework introduces additional mechanisms such as expert clustering, multiple LoRA experts, and dynamic expert selection. However, the paper does not provide a quantitative analysis of the resulting overheads. \n\n* Can the authors elaborate on the computational, memory, and communication overheads introduced by FedLEASE? \n\n* Specifically, how do these overheads compare to other baselines? \n\n* Providing metrics such as peak memory usage, training time, and convergence rate (e.g., in wall-clock time or communication rounds) would help assess practical deployability. \n\n2) Adaptivity and Sensitivity of the Cluster Count (M): The framework relies on clustering experts, with the number of clusters M which is fixed. \n\n* Can the authors discuss the implications of using a static value for M? \n\n* Is there a principled way to choose or adapt M dynamically based on the client distribution or similarity structure? \n\n* If fully adaptive selection is not feasible, an empirical sensitivity analysis showing how performance varies with different M values would be highly valuable.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n+ Valuable empirical insights: The paper offers several meaningful insights drawn from rigorous experimentation. It shows that training a single shared LoRA module is ineffective under heterogeneous data distributions, while training completely separate modules is suboptimal for homogeneous settings. The study also highlights the effectiveness of using only the LoRA B matrix for representation similarity and demonstrates that allowing a variable number of experts per client leads to improved performance. \n\n+ Comprehensive evaluation on accuracy: The experiments are thorough, covering a range of datasets and tasks with relevant baseline comparisons. The inclusion of ablation studies in the appendix adds depth and supports the claims. The overall evaluation convincingly demonstrates the effectiveness of the proposed FedLEASE framework in diverse federated settings. \n\nWeaknesses\n- Lack of quantitative analysis/evaluation on efficiency metrics: While accuracy is well covered, the evaluation lacks quantitative results on key efficiency metrics such as computational overhead, memory usage, and time-to-accuracy. For example, reporting convergence in terms of both communication rounds and wall-clock time would help assess the practical cost and scalability of FedLEASE, especially in real-world deployments with resource-constrained clients.", "was_revised": true, "final_justification": "Thank you for your thoughtful responses, which answered my questions well.", "rebuttal": "Thank you for your meaningful suggestions!\n\n**W1: System Overheads**  \nRegarding the training and clustering overhead, the clustering step is performed **only once** during the initialization phase and is not repeated during iterative training. Thus, its runtime impact is negligible. As described in §3.3, using only the **LoRA B matrices** provides an efficient and lightweight proxy for task similarity, due to their significantly smaller size compared to full model weights or LoRA A–B combinations.\n\nTo further support this, we measured the clustering time as **3.11** seconds on an Intel Xeon Platinum 8570 CPU, which is substantially shorter than the total training time (**193.49** seconds with local training on an NVIDIA B200 GPU). The detailed timing comparisons are provided below:\n\n| Time (s)                        | FedIT | FFA-LoRA | FedDPA | FedSA  | IFCA+LoRA | FedLEASE |\n|---------------------------------|-------|----------|--------|--------|------------|----------|\n| Local Per-Epoch Training Time   | 3.75  | 3.41     | 3.82   | 3.72   | 3.85       | 3.78     |\n| Global Aggregation Time         | 0.048 | 0.038    | 0.051  | 0.042  | 0.061      | 0.055    |\n| Clustering Time | –     | –        | –      | –      | 2.77         | 3.11     |\n| Total Training Time             | 188.70| 171.45   | 192.28 | 187.05 | 263.28     | 193.49   |\n\nWe also add a table summarizing convergence speed, demonstrating that FedLEASE achieves superior accuracy under the same round.\n\n**Accuracy Performance Across Training Rounds (%)**\n\n| **Method**     | **Round 1** | **Round 6** | **Round 11** | **Round 16** | **Round 21** | **Round 25** |\n|----------------|-------------|-------------|--------------|--------------|--------------|--------------|\n| FedIT          | 51.02       | 57.68       | 71.57        | 77.45        | 82.23        | 82.23        |\n| FFA-LoRA       | 50.66       | 59.84       | 70.53        | 77.79        | 81.08        | 81.06        |\n| FedDPA         | 51.33       | 59.36       | 75.99        | 81.94        | 84.23        | 84.49        |\n| FedSA          | 51.24       | 59.41       | 76.25        | 81.39        | 84.71        | 84.60        |\n| IFCA+LoRA      | 51.31       | 54.20       | 74.00        | 82.12        | 84.29        | 84.48        |\n| **FedLEASE**   | **50.82**   | **63.68**   | **81.75**    | **85.64**    | **87.36**    | **87.76**    |\n\n---\n\n**W2: Adaptivity and Sensitivity of the Cluster Count ($M$)**  \nWe clarify that $M$ is not manually fixed at first in FedLEASE. Instead, we only set an upper bound $M_{\\text{max}}$ to limit computational and communication costs. The actual number of clusters $M$ is determined automatically during the initialization phase through hierarchical clustering with the silhouette score as the evaluation metric (see §4.1). This procedure allows $M$ to adapt to the underlying client distribution and similarity structure without manual tuning.\n\nOur sensitivity analysis in **Table 8 (Appendix C.2)** demonstrates robustness across different $M_{\\text{max}}$ values: when $M_{\\text{max}}=2,3,4,8$, our method achieves 85.49%, 87.21%, 87.85%, and 87.76% average performance respectively, with the silhouette-based selection consistently identifying the optimal $M=4$ for our 16-client, 4-task setting.\n\nThese results highlight two important observations:\n- The performance degradation when $M_{\\text{max}} < M$ (85.49% vs 87.76%) confirms the importance of adequate expert budget. **Notably, even under limited expert capacity, our method still outperforms all baselines**, showcasing its effectiveness in constrained scenarios.\n- Comparable performance between $M_{\\text{max}} = 4$ and $M_{\\text{max}} = 8$ validates our method’s ability to **avoid over-clustering**, even when given excess capacity.\n\n---\nWe are grateful for the reviewer's constructive suggestions and hope our explanations resolve the raised issues."}, {"review_id": "mea796nzNo", "reviewer": "Reviewer_VvHY", "summary": "The authors propose a novel PEFT methods for federated learning, FedLEASE. They consider two key challenge: the optimal number and allocation of LoRA experts in heterogeneous clients, how clients choose to use these LoRA. FedLEASE firstly clusters clients based on LoRA similarity, and then adopts an adaptive top-M MoE mechanism to determine the local LoRA number. Experiments are conducted to evaluate the performance of the proposed mechanism.", "questions": "Please refers to weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The structure of this paper is clear and reasonable.\n\n2. This paper is easy to follow and generally well presented.\n\n3. The proposed method shows impressive accuracy improvement on the datasets.\n\n\nWeakness:\n1. Clustering based on LoRA parameter similarity makes less of a difference than traditional FL clustering based on the whole model’s parameters. The special feature of the B matrix has been discovered in the FedSA.\n\n2. The motivation and setting of heterogeneity are strange. Why do clients of different tasks need to collaborate? General FL considers label, feature, and model heterogeneity. However, contrary to the method's motivation, the experiments adopt different datasets in a benchmark with the same task.\n\n3. Implement details of the experiments should be in the main paper.\n\n4. The citation is not standard enough, such as FedDPA [37] is in the proceedings of NeurIPS 2024. The published version should be cited.", "was_revised": true, "final_justification": "Thanks for the authors' detailed response. My concerns have been addressed.", "rebuttal": "We appreciate your insightful comments!\n\n**W1: Use of LoRA B for Clustering vs. Prior FL Clustering Approaches**\n\nWhile FedSA also recognizes the distinct roles of LoRA $A$ and $B$ matrices, its method primarily uses this insight for *selective aggregation*—aggregating only the $A$ matrices while keeping $B$ matrices local to preserve client-specific information.\n\nIn contrast, **our approach uniquely leverages the LoRA $B$ matrices for clustering clients into distinct groups**, a fundamentally different purpose from FedSA. We apply hierarchical clustering on the $B$ matrices to capture task similarity and dynamically allocate personalized LoRA experts, which enables expert-aware routing and fine-tuning in federated settings.\n\nBeyond differentiating from FedSA, our method also offers clear advantages over traditional FL clustering methods like IFCA. Traditional clustering FL methods clusters clients using full model updates or losses, which incurs significant computational and communication cost. In contrast, FedLEASE uses only the small LoRA $B$ matrices, which are significantly more lightweight yet sufficient for capturing task-level differences. Moreover, our method couples this clustering with an **adaptive top-$M$ expert selection mechanism**, allowing sample-level flexibility across experts—something IFCA lacks.\n\nAs shown in our empirical results (§3.3), clustering based solely on LoRA $B$ matrices yields comparable result with LoRA $BA$-based clustering, with far less overhead. Our experiments further show that FedLEASE outperforms IFCA+LoRA by a clear margin in both performance and efficiency.\n\nTherefore, while prior works have considered either clustering (e.g., IFCA) or LoRA-specific aggregation (e.g., FedSA), we think **FedLEASE is the first to both address expert allocation and utilization challenges by LoRA $B$-based clustering with flexible top-$M$ expert selection**, providing a novel and effective solution for federated LoRA fine-tuning..\n\n---\n\n**W2: Justification for Task-Heterogeneous Setting**\n\nWhile we acknowledge that traditional federated learning often adopts Dirichlet partitioning to simulate label non-IID settings, our task-heterogeneous setup is explicitly designed to better reflect real-world scenarios of federated fine-tuning for large language models.\n\nIn practical deployments, clients typically possess data from fundamentally different domains and tasks, rather than working on the same task with skewed labels. For example:\n- Customer service platforms with dialogue transcripts\n- Educational institutions with question-answering datasets\n- Medical research labs with clinical trial summaries\n- Legal departments with contract summarization tasks\n- E-commerce companies with product review classification\n\nSuch **task-level heterogeneity** introduces more **challenging but realistic federated conditions** compared to simple label imbalance, and has also been adopted in prior work such as FedDPA (NeurIPS 2024).\n\nIn our experiments, we emulate this realistic setup by using benchmarks such as:\n- **GLUE**, which combines diverse NLU tasks including sentiment analysis (SST-2), natural language inference (QNLI), paraphrase detection (QQP), and textual similarity (MRPC);\n- and the **Flan Collection**, which aggregates NLG tasks across question answering, summarization, reasoning, and more.\n\nThese benchmarks are not a single dataset with label skew, but rather **collections of distinct tasks**, validating our claim of **task heterogeneity** in both motivation and implementation.\n\nMoreover, we also conducted evaluations of our method under **label non-IID conditions**, with results presented in **Figure 9** and **Table 7**. The findings demonstrate that FedLEASE maintains superior performance compared to all baseline methods, even when operating under these data distribution imbalances.\n\n---\n\n**W3&W4: Implement details and citation format**\n\nWe thank the reviewer for these helpful suggestions. We agree that placing critical implementation details in the main paper will enhance readability and reproducibility, and we will reorganize accordingly in the final version. Additionally, we acknowledge that certain citations, such as FedDPA, should reference their official published versions. We will correct and standardize all citations in the final manuscript to ensure accuracy and clarity."}]

Author rebuttal:

Meta-review: This paper introduces FedLEASE, a federated LoRA fine-tuning algorithm powered by mixture of adaptive number of LoRA experts. To construct multiple experts, FedLEASE clusters the LoRA from clients. The reviewers found the proposed method to be well-motivated and effective, providing a meaningful advancement over the prior art. The main criticisms have been on relatively weak efficiency analysis, ablation studies, plus the experimental setup. During the rebuttal period, the authors have addressed the concerns comprehensively, and now reviewers are unanimously positive toward the acceptance of the manuscript.
Decision: Accept (poster)