PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
Authors: Zongqian Li
Yixuan Su
Nigel Collier
Venue: NeurIPS 2025 poster ()
Keywords: Natural Language Processing
Primary area: deep_learning
TL;DR: A novel prompt tuning method that efficiently combines matrix decomposition and mixture-of-experts.

Abstract: Parameter-efficient fine-tuning (PEFT) methods have shown promise in adapting large language models, yet existing approaches exhibit counter-intuitive phenomena: integrating either matrix decomposition or mixture-of-experts (MoE) individually decreases performance across tasks, though decomposition improves results on specific domains despite reducing parameters, while MoE increases parameter count without corresponding decrease in training efficiency. Motivated by these observations and the modular nature of PT, we propose PT-MoE, a novel framework that integrates matrix decomposition with MoE routing for efficient PT. Evaluation results across 17 datasets demonstrate that PT-MoE achieves state-of-the-art performance in both question answering (QA) and mathematical problem solving tasks, improving F1 score by 1.49 points over PT and 2.13 points over LoRA in QA tasks, while improving mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA. Our analysis reveals that while PT methods generally excel in QA tasks and LoRA-based methods in math datasets, the integration of matrix decomposition and MoE in PT-MoE yields complementary benefits: decomposition enables efficient parameter sharing across experts while MoE provides dynamic adaptation, collectively enabling PT-MoE to demonstrate cross-task consistency and generalization abilities. These findings, along with ablation studies on routing mechanisms and architectural components, provide insights for future PEFT methods.

PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
Zongqian Li University of Cambridge ac uk
Yixuan Su University of Cambridge ys484@cam. ac uk
Nigel Collier University of Cambridge nhc30@cam. ac uk
Abstract
Parameter-efficient fine-tuning (PEFT) methods have shown promise in adapting large language models, yet existing approaches exhibit counter-intuitive phenom- ena: integrating either matrix decomposition ually decreases performance across tasks, though decomposition improves results specific domains despite reducing parameters, while MoE increases parameter count without corresponding decrease in training efficiency. Motivated by these observations and the modular nature of PT, we propose PT- novel framework that integrates matrix decomposition with MoE routing for efficient PT. Evaluation results across 17 datasets demonstrate that PT-MoE achieves state-of-the-art perfor- mance both question answering (QA) and mathematical problem solving tasks, improving F1 score by 1.49 points over PT and 2.13 points
mixtture-of-expetr (MoE) individ-
LoRA in QA tasks, while improving mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA. Our analysis reveals that while PT methods generally excel in QA tasks and --bsedeed methods in math datasets, the integration of matrix decomposition and MoE PT-MoE yields complementary benefits: decomposition enables efficient parameter sharing across experts while MoE provides dynamic adaptation, collectively enabling PT-MoE to demonstrate cross-task consistency and generalization abilities. These findings, along with ablation studies on routing mechanisms and architectural components, provide insights for future PEFT methods.
Introduction
Background. Large language models (LLMs) have shown remarkable capabilities but require resources for fine-tuning [41 18 19). PEFT methods address this challenge by updating only small subset of parameters B unique advantages: minimizing trainable parameters through soft prompt optimization, enabling modular deployment through task-specific prompts without model modifications, and supporting flexible knowledge composition [14 (20). These properties make it particularly effective for low- resource and multi-task applications where efficient adaptation is essential [17].
21 22 Prompt tuning (PT) stands out among PEFT approaches with its
Motivation. Despite these advantages, observe three counter-intuitive phenomena in prompt tuning. First, applying either matrix decomposition or MoE routing individually leads to performance decrease in both QA and mathematical tasks compared to standard PT (PT vs DPT, PT vs SMoP: Figure I). Second, matrix decomposition, while reducing parameter count, improves performance on specific subsets of tasks (PT DPT; Tables dynamics. Third, integrating additional routing components or multiple experts increases parameter count without corresponding decrease in training efficiency (PT SMoP, LoRA Hydral HydraLoRRA, Figure These phenomena indicate that the relationship between parameter efficiency and model
revealing task-dependent optimization
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ratio of Trainable Parameters
Figure 1: Performance and parameter ef- ficiency comparison of PEFT methods on QA and mathematical tasks. The upper sub- graph shows average Fl scores on 12 MRQA benchmark datasets, while the lower subgraph shows average accuracy on 5 mathematical datasets. The x-axis represents the number of trainable parameters. with corresponding parameter ratio shown at the top. t indi- cates higher is better; J. indicates lower is bet- ter. Red arrows indicate method transforma- tions: +MD (matrix decomposition), +MoE (mixture-of-eeperts))
0.0065%
0.0068%
0.0081%
0.0162%
58.5
PE-MA
Average FI Score
HytnialA
57.5
-MD-MGE
57.0
56.5
+M0
56.0
$5.5
DFT
QA
55.0
PF-Mat
Average Accuracy 1
55-
Mail
their combination. PT excels in QA tasks while LoRA demon- strates advantages mathematical tasks. PT- MoE achieves the best performance both task types while using fewer parameters than alternative methods, demonstrating that com- bining matrix decomposition and MoE yields complementary benefits despite each com- ponent individually decreasing performance when applied to PT.
50-
-MD-MM
45-
+M0
SMar
DFT
ATTEMPT
Math
S1K
B4K
100K
200K
Number of Trainable Parameters
PT
SMoP
X
LoRA
HydraLoRA
ATTEMPT
effectiveness in prompt tuning more nuanced than previously understood, motivating the need for a more sophisticated approach prompt optimization.
Based on these observations, we propose a novel framework, Prompt Tuning with Efficient Mixture- of-Exxerts (PT-MoE),, that combines matrix decomposition with MoE routing. As shown in Figure I our approach not only achieves statc-of-the-ee performance, but also keeps modular and uses minimal trainable parameters and moderate training steps.
Contributions. Our work offers three key developments:
Novel finetuning framework: We propose PT-MoF, integrating matrix decomposition with MoE for prompt tuning. Our framework achieves state-of-the-arl performance with fewer parameters while outperforming either technique alone, demonstrating their complementary benefits.
Design dynamics: We analyze key variables influencing the performance PT-MoF, including prompt length, expert count, routing mechanisms, and model size. Our findings provide design guidelines for future parameter-efffiien tuning approaches.
Key insights: Our comprehensive analysis across diverse tasks reveals several important find- ings: First, prompt tuning methods excel in QA tasks while LoRA-based methods demonstrate advantages in mathematical reasoning; Second, matrix decomposition reduces parameters while potentially improving domain-specific performance, whereas MoE integration increases parameter count without compromising training efficiency; and Third, combining matrix decomposition and MoE enables PT-MoE to achieve superior performance across all tasks while maintaining minimal parameter count and moderate training costs, whereas applying either of them individually can decrease average performance.
Organization. The remainder of this paper is organized as follows: Section reviews related work in prompt tuning, covering both direct tuning approaches and transfer learning methods. Section presents our PT-MoE framework, detailing the matrix decomposition strategy, dynamic router design. and training methodology. Section problem-sol ving tasks. Section presents comprehensive results, including detailed ablation studies analyzing the influences of prompt length, parameter count, expert number, routing mechanisms, and model size, followed by efficiency analysis. Section concludes with key findings and future directions.
DestionnJeeccee describes our experimental design across QA and mathematical

---

2
Related Work
To contextualize our approach, we review existing prompt tuning methods, which fall into two categories: direct prompt tuning approaches focusing on architectural innovations, and transfer leamnnng methods enabling cross-task knowledge sharing.
Direct prompt tuning methods have evolved into four main branches: (1) General approaches that directly optimize prompt parameters, including Prompt Tuning that prepends trainable vectors to input while freezing the language model [14] XPrompt that employs pruning identify and retain important prompt tokens [25] and P-Tuning v2 that introduces deep prompts across all transformer layers [23]: (2) Encoder-based methods that leverage additional additional modules, such P- Tuning that incorporates an encoder to learn dependencies between continuous embeddings [24], Residual Prompt Tuning (RPT) that employs residual part with Nown//ppprojjcteee layers for stable optimization [31], and Prefix Tuning that prepends trainable key-value pairs at each layer through a reparameterization section [T6]: (3) Decomposition methods that decompose prompt embeddings, including Decomposed Prompt Tuning (DPT) that applies low-rank matrix decomposition to reduce parameter count (37], and DePT that combines shorter soft prompts with low-rank updates word embeddings 3 and (4) MoE approaches such as Sparse Mixture-of- tixture-of-Proopts (SMoP) that employs multiple shorter prompts with a dynamic gating mechanism to route inputs to the most suitable prompt representations (2)
Transfer learning approaches in prompt tuning have developed into three categories: (1) General approaches that directly transfer prompt knowledge, including SPoT that introduces both generic transfer through multi-task pre-training and targeted transfer via task similarity matching (34, and ATTEMPT that dynamically combines multiple source prompts through an attention based mixing mechanism with instance-level adaptation M: (2) Encoder-based methods that facilitate knowledge transfer through additional architectures, such as TransPrompt that employs parallel task-specific and universal encoders with balancing mechanisms for obtaining both task-dependent and task agnostic knowledge 35 and Cross-Task Prompt Tuning (CTPT) that leverages multi-head attention for cross-task knowledge transfer with dimension reduction and derivative-fres optimization [8]; and (3) Decomposition methods exemplified by Multitask Prompt Tuning (MPT) that decomposes prompts into shared and task-specific components through knowledge distillation, enabling efficient transfer while preserving task-specific adaptability through a rank-one decomposition strategy (36]
3 Methods
Building upon the insights from prior work, we propose new parameter-efficient prompt tun- ing framework, PT-MoE, shown in Figure 2 and Algorithm [
LLM
Framework Overview. PT-MoE integrates ma- trix decomposition and dynamic routing. Given an input sequence x, our framework first gener- ates routing weights through router network R: w = R(x). These weights determine the selection among N decomposed prompts, where each prompt P; is decomposed as P; = A,B, with A, being prompt-spe speeifi and B being shared across all prompts. The final soft prompt Di=1 A,A.B, which
Rouler
Soft Prompt (SP)
Input
Decomposed SP
Output
P is computed as P E is then prepended to the input sequence for the frozen language model.
Architecture of PT-MoF. Each soft prompt is decomposed into an input-specific trix A, and a shared matrix B, with a router adap- tively selecting and combining prompt compo- nents based on input. The resulting soft prompt is prepended to the input for the frozen LLM.
Figure 2:
Matrix Decompasition. To achieve parame- ter efficiency, we decompose each prompt ma- trix P. E R into a prompt-specific matrix A, E TRR and a shared matrix B E R where T. H, and R denote the prompt length, hidden dimension, and low-rank dimension respectively. This reduces parameters from H(NTH) to
3

---

Algorithm Pseudo code of PT-MoB
Require:
Base model. Jo; inti busch X
parameters
Notation:
basch size; sequence length: number prompes tokens per prompt: di rank dimension: hidden dimension
Get input ambbeddings E (zubee(() where E 2
Calculase embeddingg mean(E, dim where je RRXA
We buheree W ERXXA, 6 2 pxxe Get rotter weights vo = soflmax(() where w E for each sample y in batch de indices lup- weights: 'toph anpuril 2 [-kk weights except lop-k: my fforall E Stopk
Compute
ghxw
cnd for
Iniisalize embeddinngs P P ghxaxá weight
Compute weighted pronuee P P A,A,, where A, C glox end for
Project dimension
Combine C concat(P. wher idxa Generate through model: y M(C)
E (*((k)))
cnd for
Fasure: Model predictions
MRQA (Extractive QAA
In-domain
SQuAD 30
TriviaQA
9
SearchQA [5]. HotpoiQA 39 NaturalQuestionn
Out-of-domain
BioASQ
DROP DuoRC [2]. RACE 13 RelationExtraction [5].
Text-
bookQA 10
Mathematics (Problem Solving)
In-domain
GSM8K 3
Out-of-domain
SVAMP: Subtraction, Addition, Common--Dvissoon Multiplication
ASDIV
MAWPS
[[:: MATH PROBLEMS 27
Table 1: Overview of training and evaluation datasets. The experiments span two task categories: extractive QA (MRQA benchmark with 12 QA datasets) and mathematical problem solving (GSM8K and specialized mathematical datasets). For each category, datasets are divided into in-domain sets used for training, validation, and evaluation, and out-of-domain sets used exclusively for testing generalization capability.
O(NTR - RH) for N prompts. The low-rank dimension either manually specified or auto- matically computed maintain parameter efficiency. For initialization, first encode task-relevant text to obtain embeddings E T then perform SVD: E UEVT. Each A; is initialized as U: R/ and the shared B as /2V R: where subscript R indicates truncation to the first R components. This approach ensures the initial prompts encode tesk-relevant information while maintaining the parameter efficiency of the decomposition.
The router network adaptively selects prompts based on input context. Given an input sequence embedding x E H (obtained by averaging token embeddings), the router computes logits through linear projection: = Wx b, where W E R and b E RV.. During training, we apply multiplicative Gaussian noise to encourage exploration: I' 1 (1 e)). where (N0 o²). The routing weights are computed a one-hot vector with at the position of the maximum value. This hard selection strategy reduces interference between prompts while maintaining end-to-end differentiability through traight-througee estimation.
Dynamic Router.
softmax(1') 1 Largmax. where Largmax is
Training and Prediction. During training, we optimize both the router parameters and decom- posed prompt matrices while keeping the base model frozen. For language model training, we use negative log-likelihood loss computed only non-prompt positions using binary mask: f. - Pog((u (r where M denotes non-prompt positions. We employ AdamW opti- mizer with warmup followed by constant learning rate schedule, and gradient accumulation for stable optimization. At inference, noise is not added in the router, ensuring deterministic prompt selection.
Experimental Design
Datasets. We conduct evaluations across 17 diverse datasets, as shown in Table I where in- domain datasets split into training, validation, and test sets, while out-of-domain datasets

---

used exclusively for testing. For QA, we utilize 12 MRQA datasets [6]. with in in-domain sets like SQuAD [30] testing information extraction abilities and out-of-domain sets like DROP 4 evaluating domain adaptation. For mathematical problem solving, we use GSM8K 3 from MetaMath [40]
as our in-domain benchmark, complemented by specialized out-of-domain datasets including the subject-specific subsets of SVAMP (29), ASDIV (26). MAWPS [1. and MATHPROBLEMS [27].
Gold Standard and Baselines. We employ full model fine-tuning as our gold standard, which updates all parameters but requires computational resources. Our baselines include representative methods from prompt tuning categories: For direct prompt tuning, we select (1) PT from general approaches, (2) DPT from decomposition methods, and (3) SMoP from MoE approaches. While transfer learning methods like (4) ATTEMPT typically involve multi-turn training, we also evaluate its architecture under similar training for comprehensive comparison. We additionally compare against other PEFT methods including (5) LoRA and (6) Hydral.RRA, with Hydral.oRA adopting a MoE-like architecture that uses shared down- wwwn-projection matrix and multiple routed up-projection matrices. These two LoRA-based methods require model architecture modifications unlike the modular nature of prompt tuning methods.
Evaluation Metrices. We employ task-specific evaluation metrics. For extractive QA tasks from MRQA, we adopt two metrics: Fl score, which evaluates the token-level overlap between predicted and ground truth answer spans, balancing precision and recall; and Exact Match (EM), which measures the percentage of predictions that exactly match the ground truth. For mathematical problem solving tasks, we use accuracy, defined as the percentage of correctly solved problems with exact answer matches.
Models. We conduct our main experiments using LLaMA-3.2-11 1.------.2-1B-Insttre B-Inttruct as the base model for fine-tuning methods [7]. For ablation studies on model size, we additionally employ LLaMA-3.2- 3B-Instruct.
5
Results
5.1
Question Answering
The experimental results on MRQA datasets shown in Table and demonstrate the effectiveness of PT-MoE across various QA tasks. We highlight seven key findings: (1) PT-MoF. achieves superior overall performance with an average F1 score of 58.26%, outperforming SMoP (56.25%) by 2.01 points and the standard PT (56.77%) by 1.49 points, establishing a new state-of-the-art (2) This advancement is further validated by Exact Match metrics, where PT-MoE demonstrates even more gains (47. 13% for average, surpassing SMoP and PT by 2.16 and 1.61 points respectively). (3) PT-MoE exhibits strong generalization capabilities across both in-domain and out-of-domain scenarios. achieves the highest performance on four out of six in-domain datasets and three out of six out-of-domain datasets. (4) The stability of PT- MoE is evidenced by consistent improvements over PT across 11 out of 12 datasets, with only marginal decreases in the RACE dataset. In contrast, SMoP shows performance decrease on datasets compared to PT. (5) Individual architectural components show limited gains: both matrix decomposition (DPT, 55. .7.7%%% FI) and MoE (SMoP, 56.25% FI) underperform standard prompt tuning (PT, 56.77% FI). (6) PT-MoE's integration of matrix decomposition and MoE yields complementary benefits, outperforming both DPT and SMoP by 2.49 and 2.01 points for F1 respectively. This improvement over individual approaches proves the mutually beneficial nature of these techniques. than FT, it reaches comparable or even higher scores than FT on specific datasets such as DROP (48.02% vs 43.87% FI) while using only 80K parameters compared FT's 1.2B, These results collectively validate the effectiveness of the architectural design of PT-MoF and demonstrate its superior performance in accuracy and generalization across diverse QA scenarios.
on the MRQA benchmark.
(7) Notably, while PT-MoE achieves lower overall performance
5.2 Mathematical Problem Solving
The experimental results on mathematical tasks (Table 4 reveal several distinctive characteristics compared to QA tasks. We highlight six key findings: (1) PT-MoE achieves ofate-of-the-as art per-
2 All methods are controlled to have similar parameter budgets, with detailed configurations shown in Table of the Appendix.
5

---

FT
LoRA
HydraLoRA
PT
DPT
SMoP
ATTEMPT
PT-MoE
para.
278k
In-domain
SQ
78.76
69.82
74.24
72.31
70.99
74.15
74.22
73.85
48.24
Tri
71.04
70.61
71.38
65.93
65.41
66.
65.31
67.34
71.35
60.13
49.74
41,08
37.64
51.33
72.96
63.29
64,02
58.69
58.49
58,96
60.18
62.16
NQ
67.56
65.92
66.31
62.
61.65
61.17
59.59
62.95
BSQ
70.19
65.38
68.76
68.59
65.56
68.59
66.69
69.33
43.87
35.25
34.38
40.39
38.80
39.92
45.32
48.02
DRC
48.11
43.69
44,36
43.30
43.64
42.07
42.86
43.96
RC
43.44
38.04
40.00
42. 10
41.89
42,34
43.01
42.51
77.97
TB
52.71
52.00
52.44
47.34
46.62
47.85
46,91
45.71
Avg.
62.52
56.13
58.17
56.77
55.77
56.25
56.17
58.26
Table 2: Evaluation results (FI scores) for various PEFT methods on QA datasets. SQ: SQuAD; News: NewsQA; Tri: TriviaQA; Srch: SearchQA; HP: HotpotQA; NQ: NaturalQuestions; BSQ:
BioASQ; DP: DROP; DRC: DuoRC; RC: RACE; RE: Relation TB: TextbookQA. The bold values indicate the best performance among prompt tuning based methods.
LoKA 106k
HydraLoRA 278k
PT
DPT
SMoP 868
ATTEMPT
PT-MoE
#
1.28
81k
81K
90K
80k
In-domain
SQ
65.28
56.26
61.63
61.25
58.49
63.15
63.71
63.34
News
32.76
25.26
27.80
32.62
32.88
32.81
32.50
32.85
62.29
64.11
64.32
59.49
58.56
59.48
58.71
60.87
Srch
61.50
46.10
50.06
42.40
39.65
34.51
31.24
43.98
11P
56.19
47.48
47.73
44.45
44.33
43,80
45.77
47.29
NQ
49.59
Out-of-domain
BSQ
49.06
42.02
44.01
$1.79
49.46
50.06
49.26
52,06
DRC
38.84
31.24
33.57
34.64
35.64
34.11
34,84
35.64
29.52
24.92
26.11
29.82
30.26
30.56
30.41
31.75
RF
66.99
58.58
62.68
72.45
70.48
74.59
75.13
78.18
43.71
44.17
43.97
39.52
38.72
40.25
39.52
38.25
Avg.
Table 3: Evaluation results (Exact Match) for QA datasets.
formance with an average accuracy of 56.91%, % improving upon PT (46.16%) by 10.75 points, demonstrating its effectiveness in mathematical reasoning. (2) The benefits of MoE integration show method-dependent characteristics: in prompt tuning approaches, PT-MoE and SMoP demon- strate different changes over PT (by + +10.75 and -5.11 points respectively); when applied to LoRA methods, Hydral LoRA shows slightly performance decrease compared LoRA. (3) LoRA-based methods demonstrate advantages in mathematical tasks compared to their performance in QA. While LoRA underperformed PT by 5.36 points MRQA, it outperforms PT by 10.31 points in mathematical tasks, indicating task-specifi strengths of different PEFT approaches. (4) PT-MoE demonstrates unique cross-task consistency: while prompt tuning methods excel in QA tasks and LoRA-base methods in mathematical tasks, PT-MoE achieves the highest average performance both domains, indicating robust adaptability across different problem types. (5) While PEFT methods consistently underperform full fine-tuning, the performance gap is larger in mathematical tasks compared to QA tasks, with a wider performance range among different methods. Notably, PT-MoE achieves comparable higher performance full fine-tuning specific datasets such as Division and MP500. (6) PT-MoE demonstrates superior parameter efficiency, achieving higher performance than LoRA while using only 75% of its parameters (80k 106k), and outper- forming HydraLoRA which uses 3.5 times more parameters. These findings highlight both the unique challenges of mathematical tasks and the robust adaptability of PT-MoE across different problem domains.
5.3
Case Study
To better understand the performance characteristics PT-MoF, we present a detailed case study of polynomial addition in Table 5 In this example, the response of the base model suffers from

---

FT
LoRA
HydraLoKA
PT
DPT
SMoP
ATTEMPT
PT-MoE
para.
278k
81K
90k
Indomaii
GSM8K
58.15
41.77
41.31
34
26.08
27.97
27.36
35,63
Out-of-domain
Suh.
68.75
67.50
57.50
41.87
43.12
38.12
40.00
55.62
Add.
64,40
61.01
62.71
50.84
35.59
35,59
35.59
55,93
52.08
33.33
37.50
79.16
Multi.
48.48
33.33
39.39
33.33
27.27
33,33
27.27
36.36
SVAMP
61.03
53.48
52.92
48.18
42.64
35.09
35.09
56.77
ASDIV
86,04
73.42
74,08
60.13
56.14
49,50
24.91
77.74
MAWPS
82.53
70.70
76.05
57.18
43.09
65.91
49.01
71.83
MP500
30.60
43.00
33.40
31.20
18.20
26.80
14.60
42.60
Avg.
63.67
56.47
55.55
46.16
37.23
41.05
30.19
56.91
Table 4: Accuracy (%) on mathematical problem-solving tasks with the number of trainable parameters shown in the second row. The first four out-of-domain datasets are from the SVAMP dataset: Sub. (Subtraction), Add. (Addition). Div.
and Multi. (Multiplication).
MP500 denotes the first 500 questions from MAAHAAATTOSSA PROBLEMS.
3y decreasing degree. Standard Answer: y² 2y
Question: f(y)
p(v)
50 7y" Find f(v) s(v). Wrte your polynomial with terms
Tw2 +
Original:
Frompt Tuming:
PT-MoE:
AB Step I: Idenift the polynomials fly) and ply)
All Sup find the sum of the two polynomials [(x) and (1))), we reed to add their corresonn-
Step 1: Idemiify the polymomiahs Rly) ply)
The polymommal fly)
as [V)
The given polynomalss
f(v) =
3y + y 3 and the polynomial gly) is given
3 and g(v) = y² 7y
a(v)) y² Ty²
nn Step 2: The polyzomial [(x) = u 3y +
An Step Add the polynomials fly) and pty)
an Soep 2: Add the polynomials Rty) and aty)
y 3 has verms of degree 4, 3, and while the polynomial g(2) = 5² Ty 2 terms of degree 3, 7, and 1.
f(y) g(2), need
corre-
To find the sum of fty) and gty). we need to add
sponding ferms the two polysomials.
the correponding terms of each polynomial.
An Step 3: Perform the addition
an Step 3: Perfoma addition
an Step 3: When we add these two potyuomt-- which adding the coefficienss terms with the same gree.
((w) a(x) (v* By y 3)
f(g) + a(v) (y* 3y
33 +
need combine like
(u + Ty 2)
(y + Ty² 2)
Step Combine like terms
an Step 4: Combine like terms Combine the torms plafy expression.
Step terms with degrce are 1 and
Combhine temma
degree of y: " - [y 3 3, [y + (v-
degree sim--
v² their sum 2n
Step Simplitt expeession
at Step S: Simplify the expeession
Step 5: terms with degree -3y and
Simplity the expression by combining the ccefii
f(y) (g)())())) vy"3y"" By 3yy++7 7Ty an Step 6: Combine
Sy" their sum is by"
cients of like teme: y 2y" 7y² -
terms
an Step 6 The terms with degres are and 7 so their sum is ?y"
Step Write the final expressioo
Coutbure the temms with the same degree.
The expression f(2) + g(3) y 2n
an Step 7: Simplity the experssion
an Step 7: The terms with degres are 1 and -2,
Ty²
f(3) + 2(2) y² 2yd + 7
their sum -1.
nn Note: is the variable, and the inswer is y 2n" Ty²
All Suep s: Write the final answer
polynomi- all (t) and g(g) is 2y Gy To²
Step Therefore, the sum
final answer is: y* 2y 7vp
All Step 9: The answer is 2g By +7y
Table 5: A case study comparing three solution approaches (Original, Prompt Tuning, and PT-MoE) for a polynomial addition problem. Errors in the outputs are highlighted in red (incorrect terms). orange (missing terms), and blue (hallucinated terms).
information loss, specifically omitting the linear term during simplification steps, leading to an incorrect final result. The conventional prompt tuning approach exhibits hallucinations and conceptual errors, particularly in degree identification and term combination, resulting in spurious terms like 2y and -6y". PT-MoE maintains information integrity throughout the solution process and avoids hallucinations, ultimately producing the correct polynomial expression. Notably, PT-MoE achieves this with a more concise solution structure, demonstrating efficient problem-solving steps while maintaining accuracy.
5.4 Ablation Studies
To systematically evaluate the design choices in PT-MoE, we conduct ablation studics on five influ- encing variables: soft prompt length, trainable parameters, number of experts, routing mechanisms, and model size. For each experiment, we keep other variables fixed at their default values (soft prompt length=40, trainable parameters=#0KK, number of experts=2, probationary-seleeee IB base model) while varying the target component to isolate its influences on model performance.
routing,
Soft prompt length. We evaluate prompt lengths ranging from 20 to 80 tokens (Figure 3 Left). Three consistent observations emerge: (1) In-domain performance exceeds out-of-domain across all lengths, maintaining 5-6% Fl score margin; (2) Both domains achieve optimal performance at 40 tokens, with peak F1 scores of 60.66% and 55. 28% respectively; and (3) Performance both domains follows similar trajectory, improving up to 40 tokens then declining. These findings
7

---

47
domann
Iedreaain Out-ol dammnn
Outok-domma
45.P
FI Soore (N)
Score (Ng
Fi Scoke (16)
-.oore
12
54
In-ooman
Cut oi domaan
20
40
50 D0
43K
23K
1834
Sub Prompt Length (taken)
Number Experts
Tranable Parameters
10 onD
Figure 3: Ablation studies on key components of PT-MoE, showing the influences of (Left) prompt length, (Center left) number of experts, (Center right) trainable parameters, and (Right) routing mechanisms ((N)S: (Non-)Selectees (N)P: (Non-)Probationrry on in-domain (ID) and out-of-domain (OOD) performance.
indicate that the optimal prompt length is domain-agnostic, though the absolute performance levels remain Nomain-deppndenn
Number of experts. We investigate the influences of expert count by varying it from 8 (Figure Center left). There are three key points: (1) Single-expert configuration yields the poorest performance 55.90% and 52. 64% F1 for in-domain and out-of-domain), demonstrating the necessity of MoE; (2) Performance exhibits an initial increase followed by decrease, with in-domain peaking at N=2 (60.66% F1) and out-of-domain at N=4 555 FI), suggesting different optimal routing capacities for each domain; (3) In-domain tasks consistently outperform out-of-domain scenarios by a 4-6% Fl margin across all expert counts. These observations demonstrate that the optimal number of experts varies by domain type and highlight the importance of balancing expert specialization with routing complexity.
Trainable parameters. We vary the parameter count from 18K to 163K to analyze its influence on model performance (Figure3CCente Right). Three key observations emerge: (1) Performance consis- tently improves with increasing parameters, from 57.51 % to 61.04% Fl for in-domain and 53.96% to 55.38% F1 for out-of-domain tasks, and notably maintains stability even at higher parameter counts, contrasting with conventional prompt tuming methods; (2) While both in-domain and out-of-domain tasks show positive scaling, they exhibit distinct parameter sensitivity behaviours, in-domain tasks demonstrate rapid improvement before 8OK parameters, while out-of-domain tasks show accelerated growth in the 40K-80K range; (3) In-domain performance maintains a consistent advantage over out-of-domaite tasks across all parameter settings, with Fl scores differing by approximately 4-6%. These findings suggest that PT-MoE effectively leverages additional parameters to achieve continuous performance gains.
Routing mechanisms. We examine two key routing design choices (Figure 3 Right): selective routing, which activates only the highest-weighted expert versus non-selective routing that utilizes all experts with their respective weights, and probationary routing, which scales the output by the router's selection probability versus probbataiii routing that uses unscaled outputs. Our experiments reveal four key findings: (1) The combination of selective and probationary routing (S, P) consistently outperforms other configurations (NS, P and S, NP) across both in-domain (60.66% vs 59.24% and 58.78% F1) and out-of-domain tasks (55.28% vs 53.41% 1 and 52.64% F1), suggesting the complementary benefits of focused expert utilization and maffidence-basee output scaling; (2) Probationary routing demonstrates superior performance over its non probpabbatioaary counterpart, indicating the value of incorporating router confidence in the final output; (3) Under probationary conditions, selective routing achieves 1 1.42% higher F1 score while reducing active parameters compared to non-selective routing, highlighting the effectiveness and efficiency of specialized expert knowledge; (4) All routing configurations maintain higher performance on in-domain tasks compared to oft-oi-domain scenarios, though the relative performance remains consistent across domains. These findings collectively demonstrate that the selective probationary routing mechanism achieves an optimal balance between model performance and computational efficiency.

---

Model size. We conduct additional experiments using 3B version of the base model, compar- ing PT-MoE with PT and the MoF-integrated method, SMoP (Table (6). Three key findings emerge: (1) PT-MoE maintains its competitive- ness at larger scales, achieving the highest aver- age accuracy of 71.11% surpassing standard PT (66.43%) and SMoP (69.61% %).. (2) SMoP shows scale-dependent behavior: while underperform- ing PT on the 1B model (56.77% vs 56.25%), surpasses PT on the 3B model (69.61% vs 66.43%). (3) PT-MoF demonstrates robust per- formance outperforming the baselines three
PT
SMoP
PT-MoE
GSM8K
56.70
61.78
59.74
SVAMP
69.36
74.69
72.81
ASDIV
76.41
80.06
81.39
MAWPS
70.70
70.70
78.02
MP500
59.00
60.80
63.60
Average
66.43
69.61
71.11
Table 6: Performance comparison (accuracy %) of standard and MoE-based prompt tuning methods on mathematical problem solving tasks using a 3B base model.
out of five mathematical datasets. These findings collectively validate the scalability and stability of PT-MoE across different model sizes.
5.5 Efficiency Analysis
Results in Figure 4 demonstrate PT-MoE's efficiency across both computational and parametric dimensions. PT-MoE achieves the highest performance with only moderate training steps and minimal parameters (80k). In contrast, LoRA and HydraL.oRA require steps to achieve comparable performance. Other prompt tuning methods such as PT, SMoP, and DPT converge fast but achieve lower performance, suggesting potential trade-off between training efficiency and model effectiveness. This evidence validates that PT-MoE balances the computational cost, parameter efficiency, and model performance.
parameters and training
Figure 4: Parameter and training ef- ficiency comparison across different methods. The x-axis shows training steps for the highest performance af- ter training parameter search, while the y-axis shows the average accuracy on math datasets. Circle sizes indicate the number of trainable parameters, with larger circles representing more param- eters. Circle markers represent mod- ular methods (PT-based) that preserve model architecture, while markers in- dicate LoRA- methods requiring model structure modifications. Though PT-MoE introduces router, it's imple- mented as a shallow layer with negligi- ble inference overhead compared to gen- eration time.
Average Accuracy t
45
40
35
30
400
1000
1400
2000
Training Steps
PT
DPT
LoRA
SMoP
HydraLoRA
(Ours)
ATTEMPT
6 Conclusions
This work introduces PT-MoE, a novel parameter-efficient framework that integrates matrix decom- position with MoE routing for prompt tuning. Our experiments across 16 datasets demonstrate that PT-MoE achieves state-of-the of--th--th--- art performance while maintaining parameter efficiency, outperforming existing methods in both QA and mathematical tasks. Through ablation studies, we identify optimal configurations for prompt length, expert count, and routing mechanisms, providing insights for future parameter-efficiet tuning approaches.
Future directions include using different routers to manage different task, and extending PT-MoE to continual learning scenarios for efficient adaptation and knowledge transfer across tasks.

---

References
[1] Akari Asai, Mohammadreza Salchi, Matthew Peters, and Hannanch Hajishirzi. ATTEMPT: Parameter-efficien multi-task tuning via attentional mixtures of soft prompts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 0455-66722 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[2] Joon-Young Young Choi, Junho Kim, Jun Huung Park, Wing-Lam Mok, and SangKeun Lee. SMop: Towards efficient and effective prompt tuning with sparse poaturreof-promppss In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[4] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2368- 2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[5] Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: new q&a dataset augmented with context from a search engine, 2017.
[6] Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP, 2019.
[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Zhiyu Ma. The Ilama 3 herd of models, 2024.
[8] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024.
[9] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Anmual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[10] Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Han- nanch Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5376- 5384, 2017.
[11] Rik Kedziorkedii.skki.. Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152-1157, San Diego California, June 2016. Association for Computational Linguistics.
[12] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoc Lc, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466. 2019.
[13] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural
10

---

Language Processing. pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguisties.
[14] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[15] Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and I Lucia Specia, editors, Proceedings of the 2 ss Conference on Computational Natural Language Learning (CoNI.I 2017), pages 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics.
[16) Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics.
[17] Zonggian Li and Jacqueline Cole. Auto-generating question-answeriigg datasets with domain- specific knowledge for language models in scientific tasks. Digital Discovery, 4(4):998- 1005, 2025.
[18] Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey, In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7182-7195, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
[19] Zonggian Li, Ehsan Shareghi, and Nigel Collier. ReasonGraph: Visualization of reasoning methods and extended inference paths. In Pushkar Mishra, Smaranda Muresan, and Tao Yu, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 140-147, Vienna, Austria, July 2025. Association for Computational Linguistics.
[20] Zonggian Li, Yixuan Su, and Nigel Collier. 500xCompressor: Generalized prompt compression for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 2508 25081-25091, Vienna, Austria, July 2025. Association for Computational Linguistics.
[21] Zongqian Li, Yixuan Su, and Nigel Collier. A survey prompt tuning, 2025.
[22] Zongqian Li, Yixuan Su, Han Zhou, Zihao Fu, and Nigel Collier. Flexi-loRA: Efficient loRA finetuning with input-adaptive dynamic ranks. In ES-FoMo IlI: 3rd Workshop on Efficient Systems for Foundation Models, 2025.
[23] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning: Prompt tniig can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68, Dublin, Ireland, May 2022. Association for Computational I Linguistics.
[24] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2023.
[25] Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang. Wei Wu, Xiaojun Quan, and Dawei Song. XPrompt: Exploring the extreme of prompt tuning. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11033-11047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

---

[26] Suen-yun Miao, Chao-Chun Liang, and Keh- Kih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online, July 2020. Association for Computational Linguistics.
[27] Nebrelbug. Math problems. Hugging Face Hub, 2024.
[28] Ioannis Partalas, Eric Gaussier, Axel-Cyrille Ngonga Ngomo, et al. Results of the first bioasq workshop. In BioASQ@ CLEF, pages 1-8, 2013.
[29] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur,, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings the
the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2080-2094, Online, June 2021. Association for Computational Linguistics.
[30] Pranay Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2392-2392, Austin, Texas, November 2016. Association for Computational Linguistics.
[31] Anastasiaa Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: improving prompt tuning with residual reparam- eterization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 6740-6757, Toronto, Canada, July 2023, Association for Computational Linguistics.
[32] Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1683-1693, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[33] Zhengxiang Shi and Aldo Lipani. DePT: Decomposed prompt tuning for parameter-efficient fine-tuning. In The Twelfth International Conference on Learning Representations, 2024.
[34] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5039-5059 Dublin, Ireland, May 2022. Association for Computational Linguistics.
[35] Chengyu Wang, Jianing Wang. Minghui Qiu, Jun Huang, and Ming Gao. TransPrompt: Towards an automatic transferable prompting framework for few-shot text classification. Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2792-2802, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[36] Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efffcient transfer learning. In The Eleventh International Conference on Learning Representations, 2023.
[37] Yao Xiao, Lu Xu, Jiaxi Li, Wei Lu, and Xiaoli Li. Decomposed prompt tuning via low-rank reparameterization. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13335-13347, Singapore, December 2023. Association for Computational Linguistics.
12

---

[38] Yige Xu, Zhiwei Zeng, and Zhiqi Shen. Efficient cross-task prompt tuning for few-shot conversational emotion recognition. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11654-11666, Singapore, December 2023. Association for Computational Linguisties.
[39] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujü,, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[40] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[41] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2024.
13

---

A Appendix
A.1
Implementation Details
We provide implementation details, including training hyperparameters in Table inference parame-
ters in Table 7 and method-specific configurations in Tables to facilitate reproducibility.
QA
Math
max_ max_new tokens
T00
768
num beams
do sample temperature top_p pad_token_id pad_token cos_token_id True
False
False
1.0
1.0
1.0
1.0
pad_token
cos_token_id carly_stopping
eos_token_id
True
Table 7: Inference parameters for QA and mathematical tasks.
QA
Math
Train steps
{500, 1000, 1500] for Pi-based methods
(500,
1000, 1500)
(200, 600, 1000) for LoRA A-based methods
Optimizer
AdamW
AdamW
Max length
512
768
warmup_stest learning rate
500
500
2e-5
2e-5
per_device_trad
bactoe__trinn_bachhhhee batch_
32
jr_scheduler_tyie
constant_ with warmup
constant _arrhpppruuppppp
gradient_ www.aaccccuctoonttteeee steps
2
2
Table 8: Training hyperparameters for QA and mathematical tasks. { means parameter search.
Method
Details
LoRA
r=1; lora a6pha=16; target modules=["g, "rojj proj"]: lora dropout=:; bias=" noon":;
lask_type=TaskkTye CAUSAL_LM
HydraLoRA
r=1; alpha=16: trrrrgetmmoollsss"s_rerrr
"y_proj"ll dropout=0.0; num_ cuun_b_matrice==2
Router: mm.init.kamming.. uniform a=math.sqr((5)), B: nn.init.zeros_0
num b_matrccs));; Initialization: A:
Soft prompt length: 40; Initialization: Specific words
DPT
Soft prompt length: 40; low low_rank____aaoo
39; Initialization: Specific words;
Decomposition method: SVD
SMoP
Total soft prompt length: 40; Number of experts: Initialization: Specitic
words; Noise: *(1 "(Ittorc.rramn (indd_
ATTEMPT
Total soft prompt length: 40; Number of experts: 2: Encoder: nn. iinear(mbedddingddin dim, projectio_ddi==),,.m nn. pinear(projection _imiiionndimml.... embedding dim), LaayerNoommmmbecdiiggdii words
Initialization: Specific
PT-MoE
Soft prompt length: 40: Number of expert: 2; Rank: 36: Router:
im.Linearfembeddnng dim, num prompts); Noise: <(*oorch.ramdn like0)+0.01)
Table Method configurations for various PEFT methods.
Prompt structure for MRQA:
<|start_header_n </de hewdeeddddddeddddddeeaeeeee hedreeeede../eeeteEeeetttt the given context that directly answers the question. without modifying combining multiple parts of the text. text.An AnVnContxx:: OAAAnnQusstion: eot_eo htt..e/eeeeeddeeessstttttttttee
exact text span from
Prompt structure for Math datasets:
14

---

<|start_header ber end Seaaerr_id[Annnnnso.. the question and response should end with \"The answer is: [answer "naawerr"".Annneee Antnquestion:: [X|ccttidd ( (An.nnnswrr...
Texts used to initialize soft prompt for finetuning on MRQA:
"Read the following context and answer the question. "Extract the answer from the context.
"The answer is span the context." "Answer the question directly." "Use the original wards in the context." "Da not introduce words nat present in the context."
Texts used to initialize soft prompt for finetuning on Math datasets:
"Read the question carefully and nake beginning.
understand it before
"Pay clase attentien the details and requirements of the question.
"Answer the question, ensuring your response is relevant to what is asked
"Ensure your answer is both accurate and correct."
A.2 Environment
python==3, 11.5
3. 1+cu118
transformerrs=:4446
datasets==2.18. 18.0
nuggingface__uub=ee 24.2
deepspeed==0.1 15.3
1ondb==0.14.2
numpy==1.23.5 tqdm==4. 6
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper shows relevant content (Section Abstract, Introduction).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work (Section Conclusions).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The paper provides related content (Section Methods),
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides all the information (Section Methods, Experimental Design, Appendix).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
The paper provides related codes (Section Appendix and supplemental mate-
rial).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public//uiides/Codeeuiiieee www.immsiooooliyyooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ bttips.cc/pbbl.. http:ubllc/gguie htte:/guidee//ooeSuue wwt.suuumiisiiiiiiiii.iiioiiioio ssionsolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides related details (Section Methods, Experimental Design, Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The paper provides relevant information (Section Results, Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides relevant information (Section Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps///aarppss..pppo///tttcccl
Question:
Answer: [Yes]
Justification:
The paper follows code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper follows the instructions (Section Appendix, supplemental material).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |paper swi thcode. com/ conasaatssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper follows the instructions (Section Appendix, supplemental material).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve content with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve content with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper follows the instructions (Section Appendix).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM policy httppp////arrrrss.ccccoorrrrrrrs for what should or should not be described.
22


Reviews:
[{"review_id": "ZZYuJU3TRn", "reviewer": "Reviewer_MobP", "summary": "Problem Statement: This work introduces PT-MoE, a novel parameter-efficient fine-tuning framework that combines matrix decomposition with mixture-of-experts routing to address limitations of existing PEFT methods.\n\nMethods Compared: DPT (LoRA+PT), SMoP (MoE+PT), and HydraLoRA (LoRA-FT+MoE).", "questions": "Q1. Please clarify exactly how PT‑MoE differs methodologically from HydraLoRA (LoRA + MoE) and other known combinations like DPT and SMoP. In particular, focus on what specific architectural or training innovations make PT‑MoE novel for prompt tuning, beyond reusing the shared matrix and routing idea?\n\nQ2. Please explain why rank = 1 was chosen for LoRA and HydraLoRA in your implementation? It would be good to add an ablation on different rank settings, and include the base model performance (without tuning) for proper context.\n\nQ3. The paper currently just restates numerical trends without explaining the underlying reasons for its key claims (e.g., why MoE keeps training efficiency stable despite more parameters, or why LoRA hurts average performance). Please add more in-depth discussion or diagnostic experiments to support these observations.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\nThe paper makes four important observations:\n\n(i) Integrating MoE or LoRA alone reduces the average performance on tasks\n(ii) LoRA reduces training parameters and improves results on some tasks (task-dependent optimisation) \n(iii) MoE increases training parameters, but training efficiency remains stable (training convergence remains stable)\n(iv) PT: Improves performance on QA tasks, LoRA: Improves performance on mathematical reasoning\n\nCons:\n\n(i) The novelty is very limited. The work implemented the HydraLoRA (LoRA+MoE) for prompt tuning, which was originally implemented for fine-tuning. The same idea of a shared matrix B, along with routing prompt-specific matrices A, is used. No additional contributions to methodology can be seen. Also, the integration of LoRA or MoE alone to Prompt Tuning has already been studied (DPT and SMoP).\n\n(ii) I have seen in the appendix that the authors have taken rank=1 for the implementation of LoRA and HydraLoRA. The rank is too low. To evaluate the prompt tuning method based on LoRA, DST, and the proposed PT-MoE, they have taken ranks 39 and 36. The evaluation doesn’t seem fair to me.\nHydraLoRA performs better than LoRA with the same number of parameters, but the authors have taken the HydraLoRA implementation with such a high number of parameters. Further, PT-MoE has fewer trainable parameters than DPT, which seems unintuitive. I am interested in seeing the performance comparison of all with the same number of parameters. \n\nFurther, ablation for performance on different ranks is missing. The performance of the base model (without tuning) is missing from the tables.\n\n(iii) The key points presented as the main contributions are not backed by solid analysis and discussion.\n“Integrating MoE or LoRA alone reduces the average performance on tasks”: earlier works have not experienced this; instead, the opposite is observed, where performance improves on integration to prompt tuning (DST, DeST, SMoP). The paper is missing a good explanation, other than the result backing. \nFurther, this is contradicted by the next point, which states LoRA improves performance on some tasks.\n“MoE increases training parameters, but training efficiency remains stable”: how this reason favours the integration of MoE is not explained. \n“PT: Improves performance on QA tasks than LoRA.” is difficult to comprehend unless the fair comparison doubt is addressed. Moreover, PEFT optimisation (LoRA, MoE or PT) is task agnostic in nature. A more thorough understanding and evaluation are required.\n\n(iv) In the analysis part, the authors have just repeated the trends and scores observed from the tables without presenting any significant explanation or findings. \n\n(v) Writing issue:  Except for the related works and datasets sections, there are no citations in the rest of the paper. Please add appropriate references throughout the method, experiments, analysis, and conclusion sections to support your claims and clearly position your contributions within the existing literature.\n\n(vi) Confusing writing: \n\nThe abstract itself is confusing. \nA. Initially, it is written that integrating LoRA decreases performance, and later it is written that it increases performance on some tasks, to favour the integration. \nB. “Integrating either matrix decomposition or mixture-of-experts (MoE) individually decreases performance across tasks”: Is it in general statement, or only related to prompt tuning?\n\nAlso, please replace the term “parameters” with the term “training parameters” in the paper.\n\n(vii) Since prompt-tuning involves inference time overhead, comparison on inference cost (FLOPs) has to be provided.\n\n(viii) Main results are shown on the 1B parameter model, while in order to show scalability 3B parameter model has been taken. It is difficult to judge the scalability of the method on low-parameter models.", "was_revised": true, "final_justification": "While I acknowledge that the rebuttal has addressed several of my concerns, my most critical reservations still remain. The four key observations presented in the paper are indeed intriguing; however, I continue to have concerns regarding the clarity of the writing and the implementation of the baselines, particularly LoRA and HydraLoRA. The authors have stated that the paper has been revised, but unless it has undergone substantial rewriting, it remains difficult to consider it fully acceptable. In particular, the methodology section lacks sufficient motivation and appropriate citations when introducing key components, which hinders the reader’s ability to follow the development of the ideas and understand the novelty of the approach.  Nevertheless, I appreciate the authors’ thoughtful and thorough responses to each of my concerns, and in recognition of their efforts, I am raising my score to a 3.\"", "rebuttal": "## **Response to Reviewer Comments**\n\nThank you for your exceptionally insightful and constructive feedback. We are genuinely grateful for the depth of analysis you've provided, which demonstrates outstanding scholarly rigor and expertise. Your comprehensive evaluation and the specific concerns you've raised regarding methodological novelty, experimental fairness, and analytical depth are particularly valuable and reflect the high standards that elevate scientific discourse. The thoughtful nature of your critique helps us better position our contributions within the broader research landscape. We address your concerns systematically below and believe our responses demonstrate the significance and innovation of this work.\n\n> ### **Methodological Novelty vs. HydraLoRA**\n\n- **HydraLoRA (baseline)'s Innovation**: HydraLoRA differs from LoRA by using a router to select different B matrices (up-projection), but still operates on attention weights within model layers.\n\n- **PT-MoE (ours)'s Architectural Innovation**: PT-MoE differs from standard PT by decomposing soft prompts into Ai×B matrices where B is router-selected, discovering that this combination creates complementary effects absent in individual components - decomposition enables efficient parameter sharing while routing provides dynamic adaptation.\n\n- **PT-MoE's Training Innovation**: PT-MoE uses SVD-based initialization for task-relevant decomposition and introduces multiplicative Gaussian noise (ε~N(0,σ²)) during routing training to encourage exploration, enabling straight-through estimation for end-to-end differentiability.\n\n- **Compared to HydraLoRA**: PT-MoE demonstrates richer architectural and methodological innovations - targeting prompt representations rather than attention weights, enabling cross-task parameter sharing through decomposed prompts, and incorporating principled SVD initialization with noise-based exploration. HydraLoRA's innovation is limited to router-based matrix selection within existing LoRA framework.\n\n- **Compared to LoRA-family methods**: LoRA-family methods require model architecture modifications, while PT-MoE maintains modularity - task-specific prompts can be deployed without altering the frozen LLM.\n\n- **Research Principle**: Innovation could be better to be elegant and effective, not artificial. Our approach demonstrates that principled combination of complementary techniques (parameter sharing via decomposition and dynamic adaptation via routing) yields superior results where individual integration fails - a counter-intuitive finding requiring methodological sophistication, not simple concatenation, validated through comprehensive ablations across 17 datasets.\n\n> ### **Fair Parameter Comparison & Rank Settings**\n\n- **Fundamental Architectural Differences**: LoRA modifies model architecture by adding trainable parameters to every attention layer (q,k,v,o projections), while PT methods add parameters only once at input level, maintaining complete modularity without structural modifications.\n\n- **Non-Equivalent Rank Concepts**: LoRA applies rank decomposition across all model layers with parameters distributed throughout every attention module (q,k,v,o projections × all layers), while PT-MoE applies decomposition only once at input level - these represent fundamentally different parameter allocation strategies that cannot be directly compared via rank values.\n\n- **Established Parameter Budget Consistency**: Current LoRA rank follows standard PEFT literature conventions for fair comparison within ~80-100K parameter range, as used in multiple published works.\n\n- **Parameter Range Comparability**: Higher LoRA ranks would exceed 800K+ parameters, departing from PT methods' parameter-efficient scope.\n\n- **Empirical Rank Validation**: Our preliminary experiments show LoRA performance plateaus while parameters increase exponentially, confirming current rank sufficiency for this parameter regime and validating our choice for controlled comparison. Recent studies further demonstrate that excessively high ranks introduce noise and decrease performance, supporting our conservative rank selection.\n\n- **Different Method Categories**: LoRA-based methods fundamentally differ from PT approaches - comparing HydraLoRA (architecture modification) against PT-MoE (modular prompting) is inherently not proper given their distinct deployment ways. We include these comparisons solely to demonstrate PT-MoE's capabilities and provide comprehensive experimental validation.\n\n> ### **Deeper Analysis of Key Claims**\n\n- **Limited Prior Exploration**: PT-domain applications of matrix decomposition and MoE remain underexplored. Our comprehensive literature review (Section 2) demonstrates that all existing baselines (DPT, SMoP, ATTEMPT) and their variants have been systematically evaluated in this work, representing the complete landscape of current approaches.\n\n- **Fundamental Methodological Distinction**: PT-based matrix decomposition fundamentally differs from LoRA - while LoRA modifies attention weights (qkvo) across model layers, our approach operates exclusively on soft prompt representations. These constitute categorically different mechanisms with distinct optimization dynamics.\n\n- **Task Complexity Limitation in Prior Work**: Previous studies (DPT, ATTEMPT, SMoP) focused mainly on simple classification tasks (GLUE, SuperGLUE) with limited reasoning requirements. Our evaluation encompasses complex QA and mathematical reasoning tasks that expose the inadequacy of existing approaches.\n\n- **Previous Methods Failure on Complex Tasks**: Prior methods exhibit substantial performance decrease on mathematical reasoning tasks (Table 4), revealing their inability to handle tasks requiring multi-step inference. This problem, unaddressed in previous papers, constitutes a core contribution of our analysis.\n\n- **Task-Specific Improvements and Overall Performance Decrease Reflect Poor Generalization**: Individual task improvements coupled with overall performance decrease indicate insufficient generalization and multi-task learning capabilities. Our method's design simultaneously enhances both generalization and cross-task consistency, as evidenced by superior performance across diverse domains.\n\n- **Parameter-Efficient MoE Innovation**: Our architectural innovation enables MoE integration to reduce total parameters in prompt tuning through matrix decomposition and sharing. The decomposition Pi = AiB with shared matrix B reduces parameters from O(NTH) to O(NTR + RH), while MoE routing leverages sparse training advantages, achieving both parameter efficiency and enhanced training speed.\n\n- **Theoretical Foundation for Complementary Integration**: The efficacy of combining matrix decomposition with MoE is from complementary regularization and capacity enhancement mechanisms. Matrix decomposition Pi = AiB induces structured low-rank constraints that eliminate spurious correlations and noise in the prompt embedding space, effectively implementing spectral regularization through SVD initialization. Simultaneously, MoE routing introduces adaptive specialization through conditional computation P = Σᵢ wᵢ(x)AᵢB, where routing weights wᵢ(x) = softmax(Wx) enable dynamic expert selection based on input characteristics. This architecture achieves optimal trade-off: decomposition reduces model variance through dimensionality reduction while MoE increases model capacity through specialized pathways, collectively enhancing both training efficiency and representational power.\n\n> ### **Comprehensive Evaluation & Enhanced Analysis**\n\n- **Complete efficiency analysis provided**: Figure 4 comprehensively analyzes parameter count, training efficiency, and model performance. The router in methods like SMoP, ATTEMPT, and PT-MoE consists of only two layers - negligible compared to billion-parameter models. For prompt tuning methods, introducing 10+ soft prompt tokens has minimal inference impact compared to hundreds or thousands of output tokens, while our method uses the fewest soft prompts, maintaining superior inference efficiency.\n\n- **Superior model scale over prior work**: DPT, ATTEMPT, and SMoP still rely on smaller models like BERT and T5. The counter-intuitive phenomena we discovered are largely from model scale changes - our 1B and 3B experiments demonstrate consistent phenomena, proving scalability. Research shows 3B-14B models exhibit similar trends; our recent supplementary experiments further confirm this and will be included in the camera-ready version within days.\n\n- **Interpretability complexity acknowledged**: Joint interpretability analysis of matrix decomposition and router is more complex than individual components. We observe relevant phenomena by combining attention values, router selections, and semantics, but rigorous verification is ongoing - this constitutes a complete independent research topic requiring separate publication.\n\n- **Citations comprehensively added**: Over a dozen additional references have been incorporated throughout Methods, Analysis, and Conclusion sections as requested.\n\n- **Writing clarity fully addressed**: Abstract and all paper issues have been revised per your requirements, clarifying the misunderstanding between improvements in specific tasks versus decreased overall performance, and standardizing the terminology \"trainable parameters\" throughout the paper.\n\nWe sincerely appreciate your detailed feedback and the opportunity to clarify these important points. Your concerns have helped us better articulate the novelty and significance of our contributions. We believe PT-MoE represents a meaningful advancement in parameter-efficient fine-tuning through its principled combination of complementary techniques, validated across diverse and challenging tasks. We look forward to further discussion and are committed to addressing any remaining concerns to strengthen this work."}, {"review_id": "nOU54Ki5cC", "reviewer": "Reviewer_cYZ8", "summary": "This paper proposes PT-MoE, a parameter-efficient fine-tuning framework that combines matrix decomposition with mixture-of-experts (MoE) routing for prompt tuning. The key motivation is addressing counter-intuitive observations where applying matrix decomposition or MoE individually to prompt tuning decreases performance, but combining them yields complementary benefits. PT-MoE decomposes prompt matrices using low-rank factorization and employs dynamic routing to select among expert prompts. Evaluated on 17 datasets spanning QA and mathematical tasks, PT-MoE achieves state-of-the-art performance while using 25% fewer parameters than LoRA.", "questions": "1、Why does the combination work? The paper observes that matrix decomposition and MoE individually hurt performance but help when combined. What is the underlying mechanism? Is this related to optimization dynamics or something else?\n\n2、The 3B model experiments show different trends than 1B results. How does PT-MoE scale to larger models (7B+), and do the benefits persist?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1、The observation that individual components (matrix decomposition, MoE) hurt performance but their combination helps is genuinely interesting and well-motivated.\n\n2、Consistent improvements across diverse tasks (QA and math) with fewer parameters than baselines.\n\n3、Thorough experiments on 17 datasets with proper ablation studies on key components (prompt length, expert count, routing mechanisms).\n\n4、Well-designed architecture with principled initialization using SVD and effective routing strategy.\n\nWeaknesses: \n\n1、The core components (matrix decomposition, MoE routing) are well-established; the main contribution is their combination in prompt tuning.\n\n2、Only tested on LLaMA models; generalization to other model families unclear. Mathematical tasks show large performance gaps between methods, suggesting potential dataset-specific effects.\n\n3、Adding routing mechanisms increases training complexity compared to simple prompt tuning, though the paper claims efficiency benefits.", "was_revised": true, "final_justification": "I’m sticking with borderline accept.  The rebuttal does a nice job unpacking why matrix decomposition and MoE click together, and the math checks out. The authors also took the comments seriously and revised carefully. That said, the paper still feels like a smart mash-up of known tricks rather than a fresh idea, and every experiment is still inside the Llama family. The promised 14B run and extra ablations never showed up.", "rebuttal": "## **Response to Reviewer Comments**\n\nThank you for your exceptionally thorough and insightful review. We are genuinely grateful for the depth of analysis you've provided and the constructive nature of your feedback. Your careful examination of our methodological contributions, experimental design, and analytical rigor demonstrates the high standards of scholarly review that elevate the quality of scientific discourse. The specific concerns you've raised are precisely the kind of critical evaluation that helps strengthen research contributions. We deeply appreciate the time and expertise you've invested in understanding our work and providing such detailed guidance for improvement.\n\n> ### **Limited Prior Methods, Model Size and Evaluation Scope**\n\n- **Limited Prior Exploration**: PT-domain applications of matrix decomposition and MoE remain underexplored. Our comprehensive literature review (Section 2) demonstrates that all existing baselines (DPT, SMoP, ATTEMPT) and their variants have been systematically evaluated in this work, representing the complete landscape of current approaches.\n\n- **Task Complexity Limitation in Prior Work**: Previous studies (DPT, ATTEMPT, SMoP) focused mainly on simple classification tasks (GLUE, SuperGLUE) with limited reasoning requirements. Our evaluation encompasses complex QA and mathematical reasoning tasks that expose the inadequacy of existing approaches.\n\n- **Previous Methods Failure on Complex Tasks**: Prior methods exhibit substantial performance decrease on mathematical reasoning tasks (Table 4), revealing their inability to handle tasks requiring multi-step inference. This problem, unaddressed in previous papers, constitutes a core contribution of our analysis.\n\n- **Superior model scale over prior work**: DPT, ATTEMPT, and SMoP still rely on smaller models like BERT and T5. The counter-intuitive phenomena we discovered are largely from model scale changes - our 1B and 3B experiments demonstrate similar phenomena, proving scalability. Research shows 3B-14B models exhibit similar trends; our recent supplementary experiments further confirm this and will be included in the camera-ready version within days.\n\n- **PT-MoE (ours)'s Architectural Innovation**: PT-MoE differs from standard PT by decomposing soft prompts into Ai×B matrices where B is router-selected, discovering that this combination creates complementary effects absent in individual components - decomposition enables efficient parameter sharing while routing provides dynamic adaptation.\n\n- **PT-MoE's Training Innovation**: PT-MoE uses SVD-based initialization for task-relevant decomposition and introduces multiplicative Gaussian noise (ε~N(0,σ²)) during routing training to encourage exploration, enabling straight-through estimation for end-to-end differentiability.\n\n- Our task selection strategically targets the performance gaps where current PEFT methods struggle most, providing more diagnostic value than superficial coverage across easier domains, while our 1B-3B model range represents the optimal balance between computational accessibility and methodological representativeness for systematic PEFT evaluation.\n\n> ### **Explanation for Complementary Benefits**\n\nWe appreciate this fundamental question and provide a deeper mechanistic analysis:\n\n**Mathematical Foundation**:\n\nTraditional prompt tuning optimizes P ∈ ℝ^(T×H) directly, while PT-MoE decomposes this as:\n\nP = Σᵢ wᵢ · Aᵢ · B\n\nwhere:\n- Aᵢ ∈ ℝ^(T×R): Expert-specific low-rank matrices\n- B ∈ ℝ^(R×H): Shared projection matrix\n- wᵢ: Router-assigned weights\n\n**Noise Reduction via Matrix Decomposition**:\n\nThe key insight is that prompt optimization in full-dimensional space ℝ^(T×H) suffers from the curse of dimensionality. By applying SVD-based decomposition:\n\nP = UΣV^T ≈ U_R Σ_R V_R^T\n\nWe retain only the top-R singular values, corresponding to principal semantic directions. This acts as an implicit regularizer:\n\n- **Signal concentration**: Task-relevant information concentrates in high singular value components\n- **Noise suppression**: Random optimization noise distributes across all dimensions and gets filtered when truncating to rank R\n- **Spectral filtering**: The decomposition P = AB constrains prompts to an R-dimensional subspace, preventing overfitting\n\n**Knowledge Reparameterization via Shared-Unique Decomposition**:\n\nThe combination of decomposition with MoE enables reparameterization of knowledge:\n\n**Theorem**: PT-MoE induces a natural decomposition into shared and task-specific components.\n\n**Proof sketch**: \n- Shared matrix B spans subspace S ⊆ ℝ^H of dimension R\n- Each expert Aᵢ defines transformation Tᵢ: ℝ^R → ℝ^T\n- Combined representation: P = (Σ wᵢAᵢ) · B = A_eff · B\n\nThis enables:\n- B captures **task-invariant linguistic patterns** (syntax, common semantics)\n- Aᵢ encodes **task-specific adaptations** (domain terminology, reasoning patterns)\n- Router learns **input space partition** for expert assignment\n\n**Complementary Optimization Dynamics**:\n\nThe complementarity emerges from the optimization landscape:\n\n- **Without decomposition**: MoE prompts interfere due to high-dimensional redundancy\n- **Without MoE**: Decomposition over-constrains expressiveness\n\n**With both**: The gradient flow naturally separates:\n- ∂L/∂B aggregates cross-task gradients → learns universal patterns\n- ∂L/∂Aᵢ receives task-specific gradients → specializes per expert\n- ∂L/∂w learns input-conditional routing → enables dynamic selection\n\n**Information-Theoretic Perspective**:\n\nFrom information theory:\n- Mutual Information: I(P; Y|X) = I(B; Y|X) + I(A_eff; Y|X, B)\n- Decomposition maximizes **information compression** in B while preserving **task-discriminative information** in {Aᵢ}\n- Achieves optimal **rate-distortion tradeoff** for prompt representations\n\n**Empirical Evidence**:\n\nWe will add:\n- **Singular value analysis**: Noise concentration in low singular values\n- **Expert specialization metrics**: KL divergence between Aᵢ matrices\n- **Routing pattern visualization**: Task-specific expert activation patterns\n- **Rank ablation**: Optimal R=36 balances compression vs. expressiveness\n\nThis mechanistic understanding explains the \"negative + negative = positive\" phenomenon: each technique alone over-constrains optimization, but together they create a structured, navigable parameter space enabling both efficiency and expressiveness.\n\nWe are profoundly grateful for your meticulous review and the opportunity to engage with such thoughtful criticism. Your feedback has not only helped us better articulate the significance of our contributions but has also provided valuable insights that will strengthen future iterations of this work. We believe your concerns have led to a more comprehensive understanding of PT-MoE's methodological innovations and its position within the broader PEFT landscape. The rigor of your evaluation exemplifies the scholarly standards that drive scientific progress, and we are honored to have received such detailed attention to our research. We remain committed to addressing any remaining questions and look forward to continued dialogue that advances our collective understanding of parameter-efficient fine-tuning methodologies. Thank you once again for your invaluable contribution to improving this work."}, {"review_id": "lqyUTNcK9M", "reviewer": "Reviewer_4bWr", "summary": "This paper proposes PT-MoE, a novel parameter-efficient finetuning framework that combines matrix decomposition with a Mixture-of-Experts (MoE) routing mechanism within prompt tuning. Unlike previous methods where each component alone may degrade performance, PT-MoE leverages their complementary strengths to achieve state-of-the-art results on 17 QA and math datasets, with significantly fewer trainable parameters. The framework preserves model modularity and efficiency, and includes extensive ablation studies exploring prompt length, expert count, and routing strategies. PT-MoE demonstrates strong generalization across domains, offering a practical and scalable approach for efficient adaptation of large language models.", "questions": "1. While the authors demonstrate that PT-MoE outperforms other methods, it remains unclear why these two techniques exhibit complementary effects when integrated. Could the authors provide more analytical or empirical evidence to explain this interaction?\n\n2. The paper’s results are focused on QA and math problem-solving. Given the claimed general-purpose nature of PT-MoE, can the authors provide evidence of performance on other task types, such as dialogue, code generation, or classification? Alternatively, can they elaborate on any task-specific limitations observed in PT-MoE?\n\n3. While PT-MoE achieves strong performance with fewer parameters, it introduces additional computation through routing. Could the authors quantify inference-time overhead more precisely, and compare it with other PEFT methods?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper proposes PT-MoE, a novel PEFT framework that combines matrix decomposition with mixture-of-experts (MoE) routing to improve parameter efficiency in prompt tuning. A key strength lies in the extensive empirical evaluation across 17 datasets in QA and mathematical reasoning, where PT-MoE achieves consistent gains over baselines like PT, DPT, and SMoP. The ablation studies are thorough and help isolate the contribution of architectural components such as prompt length, expert count, and routing strategies.\n\nThe paper is generally clear in motivation and structure, though some sections—especially methodological details—could benefit from greater exposition for accessibility. While the results are solid, the originality is somewhat limited; the core idea of combining decomposition with MoE is incremental, building directly on prior lines of work without introducing fundamentally new techniques. Moreover, theoretical insights are minimal, and the novelty mainly lies in empirical synthesis.\n\nIn sum, the work offers a meaningful contribution to the PEFT literature, with strong empirical performance and reasonable clarity, but it lacks a deeper conceptual or algorithmic innovation to be considered highly original.", "was_revised": true, "final_justification": "The authors have addressed all my concerns, I decide to raise my score to 4.", "rebuttal": "## **Response to Reviewer Comments**\n\nThank you immensely for your exceptionally thorough and insightful review. We deeply appreciate your constructive feedback and the careful analysis you've provided. Your critical evaluation demonstrates the high standards that strengthen scientific research, and we're truly grateful for the time and expertise you've invested in improving our work.\n\n> ### **Limited Prior Methods, Model Size and Evaluation Scope**\n\n- **Limited Prior Exploration**: PT-domain applications of matrix decomposition and MoE remain underexplored. Our comprehensive literature review (Section 2) demonstrates that all existing baselines (DPT, SMoP, ATTEMPT) and their variants have been systematically evaluated in this work, representing the complete landscape of current approaches.\n\n- **Task Complexity Limitation in Prior Work**: Previous studies (DPT, ATTEMPT, SMoP) focused mainly on simple classification tasks (GLUE, SuperGLUE) with limited reasoning requirements. Our evaluation encompasses complex QA and mathematical reasoning tasks that expose the inadequacy of existing approaches.\n\n- **Previous Methods Failure on Complex Tasks**: Prior methods exhibit substantial performance decrease on mathematical reasoning tasks (Table 4), revealing their inability to handle tasks requiring multi-step inference. This problem, unaddressed in previous papers, constitutes a core contribution of our analysis.\n\n- **Superior model scale over prior work**: DPT, ATTEMPT, and SMoP still rely on smaller models like BERT and T5. The counter-intuitive phenomena we discovered are largely from model scale changes - our 1B and 3B experiments demonstrate consistent phenomena, proving scalability. Research shows 3B-14B models exhibit similar trends; our recent supplementary experiments further confirm this and will be included in the camera-ready version within days.\n\n- Our task selection strategically targets the performance gaps where current PEFT methods struggle most, providing more diagnostic value than superficial coverage across easier domains, while our 1B-3B model range represents the optimal balance between computational accessibility and methodological representativeness for systematic PEFT evaluation.\n\n> ### **Explanation for Complementary Benefits**\n\nWe appreciate this fundamental question and provide a deeper mechanistic analysis:\n\n**Mathematical Foundation**:\n\nTraditional prompt tuning optimizes P ∈ ℝ^(T×H) directly, while PT-MoE decomposes this as:\n\nP = Σᵢ wᵢ · Aᵢ · B\n\nwhere:\n- Aᵢ ∈ ℝ^(T×R): Expert-specific low-rank matrices\n- B ∈ ℝ^(R×H): Shared projection matrix\n- wᵢ: Router-assigned weights\n\n**Noise Reduction via Matrix Decomposition**:\n\nThe key insight is that prompt optimization in full-dimensional space ℝ^(T×H) suffers from the curse of dimensionality. By applying SVD-based decomposition:\n\nP = UΣV^T ≈ U_R Σ_R V_R^T\n\nWe retain only the top-R singular values, corresponding to principal semantic directions. This acts as an implicit regularizer:\n\n- **Signal concentration**: Task-relevant information concentrates in high singular value components\n- **Noise suppression**: Random optimization noise distributes across all dimensions and gets filtered when truncating to rank R\n- **Spectral filtering**: The decomposition P = AB constrains prompts to an R-dimensional subspace, preventing overfitting\n\n**Knowledge Reparameterization via Shared-Unique Decomposition**:\n\nThe combination of decomposition with MoE enables reparameterization of knowledge:\n\n**Theorem**: PT-MoE induces a natural decomposition into shared and task-specific components.\n\n**Proof sketch**: \n- Shared matrix B spans subspace S ⊆ ℝ^H of dimension R\n- Each expert Aᵢ defines transformation Tᵢ: ℝ^R → ℝ^T\n- Combined representation: P = (Σ wᵢAᵢ) · B = A_eff · B\n\nThis enables:\n- B captures **task-invariant linguistic patterns** (syntax, common semantics)\n- Aᵢ encodes **task-specific adaptations** (domain terminology, reasoning patterns)\n- Router learns **input space partition** for expert assignment\n\n**Complementary Optimization Dynamics**:\n\nThe complementarity emerges from the optimization landscape:\n\n- **Without decomposition**: MoE prompts interfere due to high-dimensional redundancy\n- **Without MoE**: Decomposition over-constrains expressiveness\n\n**With both**: The gradient flow naturally separates:\n- ∂L/∂B aggregates cross-task gradients → learns universal patterns\n- ∂L/∂Aᵢ receives task-specific gradients → specializes per expert\n- ∂L/∂w learns input-conditional routing → enables dynamic selection\n\n**Information-Theoretic Perspective**:\n\nFrom information theory:\n- Mutual Information: I(P; Y|X) = I(B; Y|X) + I(A_eff; Y|X, B)\n- Decomposition maximizes **information compression** in B while preserving **task-discriminative information** in {Aᵢ}\n- Achieves optimal **rate-distortion tradeoff** for prompt representations\n\n**Empirical Evidence**:\n\nWe will add:\n- **Singular value analysis**: Noise concentration in low singular values\n- **Expert specialization metrics**: KL divergence between Aᵢ matrices\n- **Routing pattern visualization**: Task-specific expert activation patterns\n- **Rank ablation**: Optimal R=36 balances compression vs. expressiveness\n\nThis mechanistic understanding explains the \"negative + negative = positive\" phenomenon: each technique alone over-constrains optimization, but together they create a structured, navigable parameter space enabling both efficiency and expressiveness.\n\n> ### **Inference Overhead Quantification**\n\n**Model Architecture Details**:\n- **Base Model**: LLaMA-3.2-1B-Instruct\n- **Hidden Dimension (H)**: 2048\n- **Number of Layers (L)**: 16\n- **Attention Heads**: 32\n- **FFN Hidden Dimension**: 5632\n- **Vocabulary Size**: 128,256\n\n**PT-MoE Configuration**:\n- **Number of Experts (N)**: 2\n- **Soft Prompt Length (T)**: 40\n- **Low-rank Dimension (R)**: 36\n- **Router Architecture**: Linear(2048, 2)\n\n**Router Parameter Calculation**:\n\nRouter weights: W ∈ ℝ^(N×H) = ℝ^(2×2048)\nRouter bias: b ∈ ℝ^N = ℝ^2\nTotal router parameters = 2 × 2048 + 2 = 4,098 parameters\n\n**Detailed FLOPs Analysis**:\n\n**Router Forward Pass (Per Batch)**:\n\nStep 1: Input embedding averaging\n\nInput: E ∈ ℝ^(B×S×H) where B=batch_size, S=sequence_length\n\nOperation: mean(E, dim=1) → μ ∈ ℝ^(B×H)\n\nFLOPs: B × S × H additions + B × H divisions\n\nFor B=1, S=512: 512 × 2048 + 2048 = 1,050,624 FLOPs\n\nStep 2: Linear projection\n\nOperation: μ @ W^T + b → l ∈ ℝ^(B×N)\n\nFLOPs: B × H × N multiply-adds + B × N additions\n\nFor B=1: 1 × 2048 × 2 + 1 × 2 = 4,098 FLOPs\n\nStep 3: Softmax computation\n\nOperation: softmax(l) → w ∈ ℝ^(B×N)\n\nFLOPs per element: 2 exp + 1 div + N-1 adds ≈ 5N\n\nFor B=1, N=2: 1 × 10 = 10 FLOPs\n\nTotal router FLOPs = 1,050,624 + 4,098 + 10 = 1,054,732 FLOPs\n\n**LLaMA-3.2-1B Forward Pass (Per Token)**:\n\nPer Transformer Layer:\n\nMulti-Head Attention:\n\nQ,K,V projections: 3 × (H × H) = 3 × 2048² = 12,582,912 FLOPs\n\nAttention scores: S × H = 512 × 2048 = 1,048,576 FLOPs\n\nAttention output: H × H = 2048² = 4,194,304 FLOPs\n\nTotal MHA: ~17,825,792 FLOPs\n\n\nFeed-Forward Network:\n\nUp projection: H × FFN_dim = 2048 × 5632 = 11,534,336 FLOPs\n\nDown projection: FFN_dim × H = 5632 × 2048 = 11,534,336 FLOPs\n\nTotal FFN: 23,068,672 FLOPs\n\nLayer Norm (×2): 2 × 3H = 12,288 FLOPs\n\nTotal per layer: 17,825,792 + 23,068,672 + 12,288 = 40,906,752 FLOPs\n\nFor 16 layers: 16 × 40,906,752 = 654,508,032 FLOPs per token\n\n**Relative Overhead Calculation**:\n\nFor single forward pass:\n\nRouter overhead: 1,054,732 FLOPs\n\nSingle token through LLM: 654,508,032 FLOPs\n\nRelative overhead: 1,054,732 / 654,508,032 = 0.161%\n\nFor 1000-token generation:\n\nRouter overhead: 1,054,732 FLOPs (computed once)\n\nLLM computation: 1000 × 654,508,032 = 654.5B FLOPs\n\nRelative overhead: 1,054,732 / 654,508,032,000 = 0.000161%\n\n**Comparison with Prompt Processing Overhead**:\n\nStandard Prompt Tuning (40 tokens):\n\nExtra tokens processed: 40 tokens\n\nFLOPs per prompt token: 654,508,032\n\nTotal prompt overhead: 40 × 654,508,032 = 26,180,321,280 FLOPs\n\nPT-MoE routing vs prompt processing:\n\nRouting: 1,054,732 FLOPs\n\nPrompt processing: 26,180,321,280 FLOPs\n\nRatio: 26,180,321,280 / 1,054,732 = 24,812× less overhead\n\n**Memory Overhead Analysis**:\n\nRouter memory footprint:\n\nParameters: 4,098 × 4 bytes (FP32) = 16,392 bytes = 16 KB\n\nActivations (batch_size=32):\n\nInput embeddings: 32 × 2048 × 4 = 262,144 bytes\n\nRouter output: 32 × 2 × 4 = 256 bytes\n\nTotal activation: 262,400 bytes = 256 KB\n\n**Model memory footprint**:\n\nParameters: 1.2B × 4 bytes = 4.8 GB\n\nRouter percentage: 16 KB / 4.8 GB = 0.00033%\n\n**Training Overhead Analysis**:\n\nForward + Backward Pass:\n\nForward router computation: 1,054,732 FLOPs\n\nBackward gradient computation:\n\n∂L/∂W: B × H × N = 1 × 2048 × 2 = 4,096 FLOPs\n\n∂L/∂b: B × N = 1 × 2 = 2 FLOPs\n\n∂L/∂μ: B × N × H = 1 × 2 × 2048 = 4,096 FLOPs\n\nTotal backward: 8,194 FLOPs\n\n**Total training overhead per step**:\n\nForward + Backward: 1,054,732 + 8,194 = 1,062,926 FLOPs\n\nRelative to model training: 1,062,926 / (2 × 654,508,032) = 0.081%\n\n**Wall-clock Time Analysis**:\n\nTheoretical computation time (assuming 100 TFLOPS GPU):\n\nRouter forward: 1,054,732 / 10¹⁴ = 0.00001 ms\n\nLLM forward (1 token): 654,508,032 / 10¹⁴ = 6.5 ms\n\nOverhead percentage: 0.00015%\n\nMemory-bound reality:\n\nRouter computation is completely masked by memory transfer latency\n\nActual overhead: unmeasurable in practice\n\nTheoretical computation time (assuming 100 TFLOPS GPU):\n\nRouter forward: 1,054,732 / 10¹⁴ = 0.00001 ms\n\nLLM forward (1 token): 654,508,032 / 10¹⁴ = 6.5 ms\n\nOverhead percentage: 0.00015%\n\nMemory-bound reality:\n\nRouter computation is completely masked by memory transfer latency\nActual overhead: unmeasurable in practice\n\n**Summary**:\n\nThe routing mechanism adds:\n- **0.000161%** computational overhead for inference\n- **16 KB** memory (0.00033% of model size)\n- **Unmeasurable** wall-clock time impact\n\nThis negligible cost enables:\n- **10.75%** improvement in mathematical accuracy\n- **1.49 F1** improvement in QA tasks\n- Dynamic task-specific adaptation"}, {"review_id": "r4qpSn64ch", "reviewer": "Reviewer_CM3Y", "summary": "This paper proposed a new variant of Prompt Tuning that combines matrix decomposition (DPT - Shi et al. 2024) with mixture of experts (SMoP - Choi et al. 2023). Following Shi et al. 2024, each soft prompt is decomposed into a product of two matrices whereby one of matrices is shared among inputs and the second is chosen through a mixture of experts. Experimental results show that on average  the proposed PT-MoE method outperforms prompt tuning variants (PT, DPT, SMoP) and LoRA variants for QA tasks. For Math reasoning tasks, it obtains a better performance than prompt tuning variants and performs similarly to LoRA for Math Problem solving tasks. The paper presents ablation studies showing the importance of factors such as number of experts, prompt length, routing mechanism, number of parameters etc.", "questions": "* For one of the Out-of-Domain QA task (TB), PT-MoE underperforms PT (Table 3). It would be good to present an error analysis.", "limitations": "No. While the authors have mentioned some limitations in the conclusion section, it would be useful to discuss this a bit more in a separate Limitations section. Also, the authors have not discussed the potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Proposes a new Parameter Efficient Fine Tuning approach that combines Prompt Tuning with matrix decomposition and Mixture-of-Experts.\n* Proposed method outperforms regular prompt tuning and other PEFT variants on average for QA tasks. On Math tasks, it outperforms prompt tuning variants and obtains a comparable performance to LoRA variants. This is notable because prior prompt tuning approaches typically underperform LoRA for math tasks.\n* Reports ablation studies showing the impact of various factors such as routing mechanism, number of parameters etc.\n\nWeaknesses:\n* For one of the Out-of-Domain QA task (TB), PT-MoE underperforms PT (Table 3). It would be good to present an error analysis.\n* It would be good to see a comparison of the number of hallucination errors between PT,PT-MoE and LoRA on at least a single task. Though Table 5 presents a case study, a quantitative study over a full test set would be more useful.\n* Though the conclusions section discusses some limitations, it would be useful to see a more comprehensive discussion of limitations in a separate section.", "was_revised": true, "final_justification": "My final rating for the paper is 'Accept'. It is a solid paper worthy of acceptance, and my original rating was 'Accept'. \nThe authors have provided clarifications to all my questions; however, these responses do not warrant raising the rating to Strong Accept, which I reserve for papers that are either exceptionally well-written or groundbreaking. I am therefore maintaining my rating at Accept.", "rebuttal": "## **Response to Reviewer Comments**\n\nWe sincerely thank the reviewer for their thorough and insightful review of our work. We are delighted that you found our paper to be of excellent quality, clarity, and significance. Your positive assessment of PT-MoE's contributions to parameter-efficient fine-tuning is greatly appreciated.\n\n> ###  **Error Analysis for TextbookQA (TB) Performance**:\n\nThank you for pointing out the performance gap on the TextbookQA dataset. This is indeed an interesting observation that warrants further investigation. We analyzed this case and found that TextbookQA's unique characteristics may require different routing strategies. We will do a detailed error analysis, examining:\n- Distribution mismatch between TB and training domains\n- Router activation patterns on TB samples\n- Potential improvements through domain-specific fine-tuning\n\n> ### **Quantitative Hallucination Analysis**:\n\nWe appreciate your suggestion for a comprehensive hallucination study. You raise an excellent point about the importance of quantifying hallucination rates across methods. We will do:\n- Systematic hallucination metrics across all test sets\n- Comparative analysis between PT, PT-MoE, and LoRA\n- Statistical significance testing of hallucination reduction\n\nThis quantitative analysis will complement our case study and provide stronger empirical evidence for PT-MoE's reliability.\n\n> ### **Comprehensive Limitations Section**:\n\nWe acknowledge that our current discussion of limitations could be expanded. Following your recommendation, we will do a dedicated limitation analysis.\n\n> ### **Minor Clarifications**:\n\nWe also appreciate your careful reading that identified areas for improvement. Your feedback on the ablation studies and their implications has been particularly valuable in helping us understand how to better present our findings.\n\nThank you again for your constructive review and strong support for our work. We are committed to addressing all your suggestions to further strengthen the paper's contribution to the PEFT community. Your insights have been invaluable in helping us improve both the technical content and presentation of our work."}]

Author rebuttal:

Meta-review: This paper introduces PT-MoE, a PEFT framework centered on the counter-intuitive and interesting discovery that combining matrix decomposition and Mixture-of-Experts—two techniques that individually degrade prompt tuning performance—yields better results. The method is validated across 17 datasets, showing significant improvements, particularly on complex reasoning tasks where prompt tuning has traditionally lagged behind methods like LoRA.

The review process involved significant discussion, particularly regarding the novelty of combining existing components and the fairness of baseline comparisons. However, the authors provided a thorough and technically deep rebuttal that offered a compelling mechanistic explanation for the method's success. This response successfully addressed the concerns of a majority of the reviewers, leading to a clear consensus for acceptance.

While some reservations about the degree of novelty remain, the paper's strong empirical contribution and its valuable central finding justify its publication.
Decision: Accept (poster)