Title: PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Natural Language Processing
Abstract: Parameter-efficient fine-tuning (PEFT) methods have shown promise in adapting large language models, yet existing approaches exhibit counter-intuitive phenomena: integrating either matrix decomposition or mixture-of-experts (MoE) individually decreases performance across tasks, though decomposition improves results on specific domains despite reducing parameters, while MoE increases parameter count without corresponding decrease in training efficiency. Motivated by these observations and the modular nature of PT, we propose PT-MoE, a novel framework that integrates matrix decomposition with MoE routing for efficient PT. Evaluation results across 17 datasets demonstrate that PT-MoE achieves state-of-the-art performance in both question answering (QA) and mathematical problem solving tasks, improving F1 score by 1.49 points over PT and 2.13 points over LoRA in QA tasks, while improving mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA. Our analysis reveals that while PT methods generally excel in QA tasks and LoRA-based methods in math datasets, the integration of matrix decomposition and MoE in PT-MoE yields complementary benefits: decomposition enables efficient parameter sharing across experts while MoE provides dynamic adaptation, collectively enabling PT-MoE to demonstrate cross-task consistency and generalization abilities. These findings, along with ablation studies on routing mechanisms and architectural components, provide insights for future PEFT methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new variant of Prompt Tuning that combines matrix decomposition (DPT - Shi et al. 2024) with mixture of experts (SMoP - Choi et al. 2023). Following Shi et al. 2024, each soft prompt is decomposed into a product of two matrices whereby one of matrices is shared among inputs and the second is chosen through a mixture of experts. Experimental results show that on average  the proposed PT-MoE method outperforms prompt tuning variants (PT, DPT, SMoP) and LoRA variants for QA tasks. For Math reasoning tasks, it obtains a better performance than prompt tuning variants and performs similarly to LoRA for Math Problem solving tasks. The paper presents ablation studies showing the importance of factors such as number of experts, prompt length, routing mechanism, number of parameters etc.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Proposes a new Parameter Efficient Fine Tuning approach that combines Prompt Tuning with matrix decomposition and Mixture-of-Experts.\n* Proposed method outperforms regular prompt tuning and other PEFT variants on average for QA tasks. On Math tasks, it outperforms prompt tuning variants and obtains a comparable performance to LoRA variants. This is notable because prior prompt tuning approaches typically underperform LoRA for math tasks.\n* Reports ablation studies showing the impact of various factors such as routing mechanism, number of parameters etc.\n\nWeaknesses:\n* For one of the Out-of-Domain QA task (TB), PT-MoE underperforms PT (Table 3). It would be good to present an error analysis.\n* It would be good to see a comparison of the number of hallucination errors between PT,PT-MoE and LoRA on at least a single task. Though Table 5 presents a case study, a quantitative study over a full test set would be more useful.\n* Though the conclusions section discusses some limitations, it would be useful to see a more comprehensive discussion of limitations in a separate section.\n\nQUESTIONS:\n* For one of the Out-of-Domain QA task (TB), PT-MoE underperforms PT (Table 3). It would be good to present an error analysis.\n\nLIMITATIONS:\nNo. While the authors have mentioned some limitations in the conclusion section, it would be useful to discuss this a bit more in a separate Limitations section. Also, the authors have not discussed the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PT-MoE, a novel parameter-efficient finetuning framework that combines matrix decomposition with a Mixture-of-Experts (MoE) routing mechanism within prompt tuning. Unlike previous methods where each component alone may degrade performance, PT-MoE leverages their complementary strengths to achieve state-of-the-art results on 17 QA and math datasets, with significantly fewer trainable parameters. The framework preserves model modularity and efficiency, and includes extensive ablation studies exploring prompt length, expert count, and routing strategies. PT-MoE demonstrates strong generalization across domains, offering a practical and scalable approach for efficient adaptation of large language models.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes PT-MoE, a novel PEFT framework that combines matrix decomposition with mixture-of-experts (MoE) routing to improve parameter efficiency in prompt tuning. A key strength lies in the extensive empirical evaluation across 17 datasets in QA and mathematical reasoning, where PT-MoE achieves consistent gains over baselines like PT, DPT, and SMoP. The ablation studies are thorough and help isolate the contribution of architectural components such as prompt length, expert count, and routing strategies.\n\nThe paper is generally clear in motivation and structure, though some sections—especially methodological details—could benefit from greater exposition for accessibility. While the results are solid, the originality is somewhat limited; the core idea of combining decomposition with MoE is incremental, building directly on prior lines of work without introducing fundamentally new techniques. Moreover, theoretical insights are minimal, and the novelty mainly lies in empirical synthesis.\n\nIn sum, the work offers a meaningful contribution to the PEFT literature, with strong empirical performance and reasonable clarity, but it lacks a deeper conceptual or algorithmic innovation to be considered highly original.\n\nQUESTIONS:\n1. While the authors demonstrate that PT-MoE outperforms other methods, it remains unclear why these two techniques exhibit complementary effects when integrated. Could the authors provide more analytical or empirical evidence to explain this interaction?\n\n2. The paper’s results are focused on QA and math problem-solving. Given the claimed general-purpose nature of PT-MoE, can the authors provide evidence of performance on other task types, such as dialogue, code generation, or classification? Alternatively, can they elaborate on any task-specific limitations observed in PT-MoE?\n\n3. While PT-MoE achieves strong performance with fewer parameters, it introduces additional computation through routing. Could the authors quantify inference-time overhead more precisely, and compare it with other PEFT methods?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PT-MoE, a parameter-efficient fine-tuning framework that combines matrix decomposition with mixture-of-experts (MoE) routing for prompt tuning. The key motivation is addressing counter-intuitive observations where applying matrix decomposition or MoE individually to prompt tuning decreases performance, but combining them yields complementary benefits. PT-MoE decomposes prompt matrices using low-rank factorization and employs dynamic routing to select among expert prompts. Evaluated on 17 datasets spanning QA and mathematical tasks, PT-MoE achieves state-of-the-art performance while using 25% fewer parameters than LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1、The observation that individual components (matrix decomposition, MoE) hurt performance but their combination helps is genuinely interesting and well-motivated.\n\n2、Consistent improvements across diverse tasks (QA and math) with fewer parameters than baselines.\n\n3、Thorough experiments on 17 datasets with proper ablation studies on key components (prompt length, expert count, routing mechanisms).\n\n4、Well-designed architecture with principled initialization using SVD and effective routing strategy.\n\nWeaknesses: \n\n1、The core components (matrix decomposition, MoE routing) are well-established; the main contribution is their combination in prompt tuning.\n\n2、Only tested on LLaMA models; generalization to other model families unclear. Mathematical tasks show large performance gaps between methods, suggesting potential dataset-specific effects.\n\n3、Adding routing mechanisms increases training complexity compared to simple prompt tuning, though the paper claims efficiency benefits.\n\nQUESTIONS:\n1、Why does the combination work? The paper observes that matrix decomposition and MoE individually hurt performance but help when combined. What is the underlying mechanism? Is this related to optimization dynamics or something else?\n\n2、The 3B model experiments show different trends than 1B results. How does PT-MoE scale to larger models (7B+), and do the benefits persist?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProblem Statement: This work introduces PT-MoE, a novel parameter-efficient fine-tuning framework that combines matrix decomposition with mixture-of-experts routing to address limitations of existing PEFT methods.\n\nMethods Compared: DPT (LoRA+PT), SMoP (MoE+PT), and HydraLoRA (LoRA-FT+MoE).\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\nThe paper makes four important observations:\n\n(i) Integrating MoE or LoRA alone reduces the average performance on tasks\n(ii) LoRA reduces training parameters and improves results on some tasks (task-dependent optimisation) \n(iii) MoE increases training parameters, but training efficiency remains stable (training convergence remains stable)\n(iv) PT: Improves performance on QA tasks, LoRA: Improves performance on mathematical reasoning\n\nCons:\n\n(i) The novelty is very limited. The work implemented the HydraLoRA (LoRA+MoE) for prompt tuning, which was originally implemented for fine-tuning. The same idea of a shared matrix B, along with routing prompt-specific matrices A, is used. No additional contributions to methodology can be seen. Also, the integration of LoRA or MoE alone to Prompt Tuning has already been studied (DPT and SMoP).\n\n(ii) I have seen in the appendix that the authors have taken rank=1 for the implementation of LoRA and HydraLoRA. The rank is too low. To evaluate the prompt tuning method based on LoRA, DST, and the proposed PT-MoE, they have taken ranks 39 and 36. The evaluation doesn’t seem fair to me.\nHydraLoRA performs better than LoRA with the same number of parameters, but the authors have taken the HydraLoRA implementation with such a high number of parameters. Further, PT-MoE has fewer trainable parameters than DPT, which seems unintuitive. I am interested in seeing the performance comparison of all with the same number of parameters. \n\nFurther, ablation for performance on different ranks is missing. The performance of the base model (without tuning) is missing from the tables.\n\n(iii) The key points presented as the main contributions are not backed by solid analysis and discussion.\n“Integrating MoE or LoRA alone reduces the average performance on tasks”: earlier works have not experienced this; instead, the opposite is observed, where performance improves on integration to prompt tuning (DST, DeST, SMoP). The paper is missing a good explanation, other than the result backing. \nFurther, this is contradicted by the next point, which states LoRA improves performance on some tasks.\n“MoE increases training parameters, but training efficiency remains stable”: how this reason favours the integration of MoE is not explained. \n“PT: Improves performance on QA tasks than LoRA.” is difficult to comprehend unless the fair comparison doubt is addressed. Moreover, PEFT optimisation (LoRA, MoE or PT) is task agnostic in nature. A more thorough understanding and evaluation are required.\n\n(iv) In the analysis part, the authors have just repeated the trends and scores observed from the tables without presenting any significant explanation or findings. \n\n(v) Writing issue:  Except for the related works and datasets sections, there are no citations in the rest of the paper. Please add appropriate references throughout the method, experiments, analysis, and conclusion sections to support your claims and clearly position your contributions within the existing literature.\n\n(vi) Confusing writing: \n\nThe abstract itself is confusing. \nA. Initially, it is written that integrating LoRA decreases performance, and later it is written that it increases performance on some tasks, to favour the integration. \nB. “Integrating either matrix decomposition or mixture-of-experts (MoE) individually decreases performance across tasks”: Is it in general statement, or only related to prompt tuning?\n\nAlso, please replace the term “parameters” with the term “training parameters” in the paper.\n\n(vii) Since prompt-tuning involves inference time overhead, comparison on inference cost (FLOPs) has to be provided.\n\n(viii) Main results are shown on the 1B parameter model, while in order to show scalability 3B parameter model has been taken. It is difficult to judge the scalability of the method on low-parameter models.\n\nQUESTIONS:\nQ1. Please clarify exactly how PT‑MoE differs methodologically from HydraLoRA (LoRA + MoE) and other known combinations like DPT and SMoP. In particular, focus on what specific architectural or training innovations make PT‑MoE novel for prompt tuning, beyond reusing the shared matrix and routing idea?\n\nQ2. Please explain why rank = 1 was chosen for LoRA and HydraLoRA in your implementation? It would be good to add an ablation on different rank settings, and include the base model performance (without tuning) for proper context.\n\nQ3. The paper currently just restates numerical trends without explaining the underlying reasons for its key claims (e.g., why MoE keeps training efficiency stable despite more parameters, or why LoRA hurts average performance). Please add more in-depth discussion or diagnostic experiments to support these observations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}