Title: Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

LLM fine-tuning
Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but fine-tuning them for domain-specific applications often requires substantial domain-specific data that may be distributed across multiple organizations. Federated Learning (FL) offers a privacy-preserving solution, but faces challenges with computational constraints when applied to LLMs. Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient fine-tuning approach, though a single LoRA module often struggles with heterogeneous data across diverse domains. This paper addresses two critical challenges in federated LoRA fine-tuning: 1. determining the optimal number and allocation of LoRA experts across heterogeneous clients, and 2. enabling clients to selectively utilize these experts based on their specific data characteristics. We propose FedLEASE (Federated adaptive LoRA Expert Allocation and SElection), a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain-specific LoRA experts. It also introduces an adaptive top-$M$ Mixture-of-Experts mechanism that allows each client to select the optimal number of utilized experts. Our extensive experiments on diverse benchmark datasets demonstrate that FedLEASE significantly outperforms existing federated fine-tuning approaches in heterogeneous client settings while maintaining communication efficiency.


Peer reviews:
comments: SUMMARY:
The authors propose a novel PEFT methods for federated learning, FedLEASE. They consider two key challenge: the optimal number and allocation of LoRA experts in heterogeneous clients, how clients choose to use these LoRA. FedLEASE firstly clusters clients based on LoRA similarity, and then adopts an adaptive top-M MoE mechanism to determine the local LoRA number. Experiments are conducted to evaluate the performance of the proposed mechanism.

STRENGTHS AND WEAKNESSES:
Strength:
1. The structure of this paper is clear and reasonable.

2. This paper is easy to follow and generally well presented.

3. The proposed method shows impressive accuracy improvement on the datasets.


Weakness:
1. Clustering based on LoRA parameter similarity makes less of a difference than traditional FL clustering based on the whole model’s parameters. The special feature of the B matrix has been discovered in the FedSA.

2. The motivation and setting of heterogeneity are strange. Why do clients of different tasks need to collaborate? General FL considers label, feature, and model heterogeneity. However, contrary to the method's motivation, the experiments adopt different datasets in a benchmark with the same task.

3. Implement details of the experiments should be in the main paper.

4. The citation is not standard enough, such as FedDPA [37] is in the proceedings of NeurIPS 2024. The published version should be cited.

QUESTIONS:
Please refers to weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FedLEASE, a novel framework for federated fine-tuning of large language models (LLMs) using adaptive LoRA. FedLEASE tackles two key challenges in federated settings: (1) how to allocate LoRA experts across heterogeneous clients, and (2) how clients can selectively use relevant experts. The method clusters clients by representation similarity and assigns domain-specific LoRA experts accordingly. It also employs an adaptive top-M Mixture-of-Experts mechanism to optimize expert selection per client. Experiments show that FedLEASE outperforms existing methods on diverse datasets, achieving better performance under data heterogeneity while maintaining communication efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ Valuable empirical insights: The paper offers several meaningful insights drawn from rigorous experimentation. It shows that training a single shared LoRA module is ineffective under heterogeneous data distributions, while training completely separate modules is suboptimal for homogeneous settings. The study also highlights the effectiveness of using only the LoRA B matrix for representation similarity and demonstrates that allowing a variable number of experts per client leads to improved performance. \n\n+ Comprehensive evaluation on accuracy: The experiments are thorough, covering a range of datasets and tasks with relevant baseline comparisons. The inclusion of ablation studies in the appendix adds depth and supports the claims. The overall evaluation convincingly demonstrates the effectiveness of the proposed FedLEASE framework in diverse federated settings. \n\nWeaknesses\n- Lack of quantitative analysis/evaluation on efficiency metrics: While accuracy is well covered, the evaluation lacks quantitative results on key efficiency metrics such as computational overhead, memory usage, and time-to-accuracy. For example, reporting convergence in terms of both communication rounds and wall-clock time would help assess the practical cost and scalability of FedLEASE, especially in real-world deployments with resource-constrained clients.\n\nQUESTIONS:\n1) System Overheads (Computation, Memory, Communication): The proposed FedLEASE framework introduces additional mechanisms such as expert clustering, multiple LoRA experts, and dynamic expert selection. However, the paper does not provide a quantitative analysis of the resulting overheads. \n\n* Can the authors elaborate on the computational, memory, and communication overheads introduced by FedLEASE? \n\n* Specifically, how do these overheads compare to other baselines? \n\n* Providing metrics such as peak memory usage, training time, and convergence rate (e.g., in wall-clock time or communication rounds) would help assess practical deployability. \n\n2) Adaptivity and Sensitivity of the Cluster Count (M): The framework relies on clustering experts, with the number of clusters M which is fixed. \n\n* Can the authors discuss the implications of using a static value for M? \n\n* Is there a principled way to choose or adapt M dynamically based on the client distribution or similarity structure? \n\n* If fully adaptive selection is not feasible, an empirical sensitivity analysis showing how performance varies with different M values would be highly valuable.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FedLEASE, an innovative federated learning framework designed to address the challenges of LoRA-based fine-tuning in heterogeneous client environments. FedLEASE mainly includes two key innovations: optimizing expert allocation by grouping clients based on parameter similarity; and a dynamic top-M mechanism that adaptively determines the optimal number of experts per client, overcoming the limitations of fixed selection strategies.  Extensive experiments demonstrate FedLEASE's superior performance across both natural language understanding (NLU) and generation (NLG) tasks while maintaining communication efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well-structured, with clear explanations of the methodology and experiments.\n- The proposed adaptive top-M MoE mechanism is innovative and well-justified by empirical results.\n- The paper provides the theoretical analysis in the Appendix, ensuring the robustness of the proposed method.\n- The ablation studies validate the proposed method's effectiveness and scalability.\n\n**Weaknesses**\n\n- Some parts of the writing in the paper are confusing.  \n    - First, regarding the adaptive top-M MOE scheme, according to the paper's description, M clusters are obtained through hierarchical clustering, and the protocol of each cluster is used as the initialized LoRA experts. On each client, the LoRA belonging to itself is replicated M times to obtain 2M-1 output routers, thereby allowing each client to dynamically determine the final number of experts used based on its local data. So, what does the \"fixed Top-K\" mentioned in the paper refer to? Based on the above explanation and my understanding, it should result in K clusters and K LoRA experts, with each client obtaining K output routers. However, in the top-right corner of Figure 5, there are 3 experts and 2 output routers, which is perplexing. Perhaps I missed something, or maybe the confusion arises from inconsistent notation in the method description.\n\n    - Second, regarding the 2M-1 output routers, why is the expert belonging to the client replicated M times? The authors could provide a more intuitive explanation rather than just presenting empirical results. Additionally, each client essentially still selects M experts; the unequal number of experts utilized is due to the replication of its own expert M times. Therefore, I believe attributing the dynamic selection of expert numbers as the reason for achieving optimal personalized performance is not entirely appropriate. The authors might need to adopt a more suitable explanation.\n\n- Concerning the selection of baselines in the experiments. The proposed method has two contributions: one is hierarchical clustering with the silhouette score as the metric to determine the optimal number of experts, and the other is the adaptive top-M MOE. However, the baselines are either FL+LoRA or clustered FL (IFCA)+LoRA, neither of which includes MOE LoRA techniques. Thus, the experimental comparisons are not entirely fair. For example, in the comparison with IFCA+LoRA, it is unclear whether the performance improvement comes from hierarchical clustering, MOE, or the adaptive top-M MOE, making it difficult to quantify these incremental contributions. Nevertheless, it is commendable that the authors demonstrate the advantages of hierarchical clustering for obtaining LoRA experts and the adaptive top-M MOE in Table 2.\n\n- The experimental section only reports the number of parameters, but does not present runtime or communication efficiency. Since the proposed method uses hierarchical clustering to determine the optimal number of experts, this process could be highly time-consuming when the number of clients is large. \n\n- The dataset partitioning in the experiments reveals a natural clustering structure, which explains why the proposed (clustering-based) method achieves strong performance. However, data heterogeneity in FL is typically complex and may exhibit very ambiguous clustering structures. If possible, I would like to know how the proposed method performs under more commonly used Dirichlet partitioning (or other heterogeneous partitioning schemes).\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FedLEASE, a federated LoRA fine-tuning framework that tackles two key challenges in heterogeneous federated learning. FedLEASE clusters clients using the similarity of their LoRA B matrices to determine expert allocation. Additionally, it introduces an adaptive top-M mechanism that enables each client to dynamically select the number of experts to use. Experimental results across various NLU and NLG datasets show that FedLEASE consistently outperforms existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper identifies two major challenges in federated LoRA fine-tuning and supports them with insightful preliminary experiments. \n2. The adaptive top-M routing mechanism enables client-specific expert selection, which improves personalization over fixed top-k strategies.\n3. FedLEASE is evaluated on both NLU and NLG tasks, and the paper includes comprehensive experiments that validate the effectiveness and individual contributions of each component.\n\nWeaknesses\n1. The clustering phase performs clustering based on the LoRA B matrices from all clients, which could introduce computational overhead, especially in large-scale settings.\n2. The ablation studies are not sufficiently thorough. For example, alternative clustering methods are not explored, making it unclear whether Agglomerative Hierarchical Clustering is the optimal choice.\n\nQUESTIONS:\n1. How does FedLEASE compare to baseline methods in terms of computational cost and training time?\n2. Is the router network maintained locally by each client or distributed from the server? Algorithm 1 indicates that clients reinitialize the router at each communication round.\n3. In Table 5, when the number of clients changes (e.g., from 8 to 32), does the optimal number of experts adjust accordingly?\n4. In addition to the number of experts each client uses, it would be helpful to show which experts are selected by each client. A discussion on the frequency and distribution of expert selection across clients would provide deeper insights.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}