One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning
Authors: Minh Le
Bao-Ngoc Dao
Huy Nguyen
Quyen Tran
Anh Nguyen
Nhat Ho
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Prefix Tuning
Mixture of Experts
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strategy in this paradigm assigns a dedicated subset of prompts to each task, which, while effective, incurs substantial computational overhead and causes memory requirements to scale linearly with the number of tasks. Conversely, approaches employing a single shared prompt across tasks offer greater efficiency but often suffer from degraded performance due to knowledge interference. To reconcile this trade-off, we propose **SMoPE**, a novel framework that integrates the benefits of both task-specific and shared prompt strategies. Inspired by recent findings on the relationship between Prefix Tuning and Mixture of Experts (MoE), SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse MoE architecture. For each input, only a select subset of relevant experts is activated, effectively mitigating interference. To facilitate expert selection, we introduce a prompt-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling dynamic and sparse activation. Additionally, we propose an adaptive noise mechanism to encourage balanced expert utilization while preserving knowledge from prior tasks. To further enhance expert specialization, we design a prototype-based loss function that leverages prefix keys as implicit memory representations. Extensive experiments across multiple CL benchmarks demonstrate that SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches, all while significantly reducing parameter counts and computational costs.

Published as a conference paper at ICLR 2026
STRIKES BACK: SPARSE MIXTURE OF EXPERTS FOR PROMPT-BASED CONTINUAL LEARNING
Minh Le Bao Ngoe Dao* Huy Nguyen', Quyen Tran?, Anh Nguyen', Nhat Ho
Trivita AI 2 The University of Texas at Austin
Northeastern University
4 Hanoi University of Science and Technology
Rutgers University
ABSTRACT
Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strat- egy in this paradigm assigns a dedicated subset of prompts to each task, which, while effective, incurs substantial computational overhead and causes memory requirements to scale linearly with the number of tasks. Conversely, approaches employing single shared prompt across tasks offer greater efficiency but often suffer from degraded performance due to knowledge interference. To reconcile this trade-off, propose SMoPE, novel framework that integrates the benefits of both task-specific and shared prompt strategies. Inspired by recent findings on the relationship between Prefix Tuning and Mixture of Experts (MoE), SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse MoE architecture. For each input, only select subset of relevant experts is activated, effectively mitigating interference. To facilitate expert selection, we introduce a prompt-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling dynamic and sparse activation. Additionally, we pro- pose an adaptive noise mechanism to encourage balanced expert utilization while preserving knowledge from prior tasks. To further enhance expert specialization, we design a prototype-basee loss function that leverages prefix keys as implicit memory representations. Extensive experiments across multiple CL benchmarks demonstrate that SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-arr approaches, all while significantly reducing parameter counts and computational costs. Our code pub- licly available at /ittts::/// httpa//wcc/////aaaaaeeeeeeoooona
INTRODUCTION
Continual Learning (CL) is critical research area focused on enabling neural networks to learn from a sequence of tasks in dynamic environments while retaining knowledge from previous tasks (Aljundi
al.
2017 De Lange al. 202T: Zhang al. 2023 Tran al., 2024). A primary challenge
in CL is catastrophic forgetting, where performance on earlier tasks deteriorates as new ones are
learned (McCloskey Cohen 1989, French 1999; Mchta et al. (023). Recently, prompt-based
approaches have gained attention as a promising direction in CL. offering strong performance and high memory efficiency (Wang al. 2022b a. Smith et al. 2023 Le et 2024a). These methods adapt pre-trained models using a small set of learnable parameters, referred to as prompts, which function as task-specific instructions to guide adaptation and alleviate forgetting. Several studies have demonstrated the effectiveness of prompt-based methods, reporting state-of-the-art results across range of CL benchmarks.
A common strategy in prompt- craeet-based based CL is to allocate distinct subset of prompt parameters each task (Wang et al.) 2022a 2023) Le et al., 2024a Tran et al.) (025)). This task-specific partitioning helps mitigate interference by isolating knowledge within separate prompt modules. While effective, such approaches face several notable limitations. First, when task identity is unknown at inference time, the model must infer the appropriate prompt for each input, Existing methods often rely
"Equal Contribution

---

Published as a conference paper at ICLR 2026
forwarding the imput through the full pre- pre-trained model to compute query, introducing non-negligible computational overhead (Huang et al. 2024 Kim et al. 2022).. Second, assigning dedicated prompts to each task hinders scalability and limits knowledge sharing. As the number of tasks increases, the number of learnable prompi parameters grows linearly, making this approach inefficient for long- term continual learning. Moreover, striet prompt partitioning restricts transferability by preventing the reuse or adaptation of previously learned prompts, thereby limiting positive knowledge transfer across tasks.
In contrast to task-specific prompt methods, Huang el al. (2044) recently introduced OVOR, a highly aarameter-effiieen and computationally lightweight approach that employs a single shared prompt across all tasks. Despite its efficiency, OVOR underperforms relative to state-of-the-ar task-specific prompt methods. We hypothesize that this performance gap arises from excessive knowledge interference: because the same prompt is continually updated across tasks, it struggles to retain task-specific information, leading to degraded performance. This raises a key question:
(2) Can we strike a balance between these two paradigms, retaining the parameter efficiency of single prompt while achieving performance competitive with task-specific approaches?
Most prompt-basee -baee CL methods rely on Prefix Tuning (Li & Liang 202T) to integrate prompts into pre-trained models. However, recent work by Le et al. (2024a) offers a new perspective: each attention head can be viewed composition of multiple Mixture of Experts (MoE) models (Jacobs) et al. 1991 Shazeer et al., 2017), and prefix tuning effectively introduces new prompt experts into these models. Building on this insight, we propose SMoPE (Sparse Mixture of Prompt Experts), a novel method that retains a single shared prompt while structuring it as multiple prompt experts within a sparse MoE framework.
SMoPE introduces a prompl-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling sparse and dynamic expert selection for each input. Like OVOR, SMoPE maintains a single shared prompt across tasks for high parameter efficiency. However, unlike OVOR, which updates all prompt components uniformly. S.MPPE selectively activates and updates only a small, relevant subset of experts for each input. This targeted update mechanism reduces interference and promotes positive transfer by reusing previously learned experts across tasks. A common challenge in sparse MoE architectures is the risk of imbalanced expert utilization. To address this while preserving learned knowledge, we introduce an adaptive noise mechanism that encourages the use of underutilized experts without overwriting important ones. Furthermore, to enhance expert specialization in the CL setting, we propose a novel prototype loss that leverages prefix keys as an implicit memory of past tasks. Extensive experiments on standard CL benchmarks show that SMoPE significantly outperforms task-specific prompt methods despite using a single shared prompt. Moreover, SMoPE matches or exceeds the performance of current state-of-the-art methods while reducing computational cost by up to 50% % and requiring substantially fewer Icarnable parameters.
Contributions. The primary contributions of this work are: We propose SMoPE, a novel approach that integrates a sparse mixture of experts architecture into the prefix tuning framework, featuring a prompt-attenin score aggregation mechanism for efficient and dynamic expert selection. 2. We introduce an adaptive noise mechanism to encourage balanced expert utilization without overwriting prior knowledge, and prototype-baad baseed loss that leverages prefix keys as implicit memory. 3. We show that SMoPE achieves of-the-art results on multiple CL. benchmarks, while using significantly fewer parameters and reducing computation compared to existing prompt-based methods.
2
BACKGROUND AND RELATED WORK
2.1 CONTINUAL LEARNING
Continual Learning (CL) involves training a model on a sequence of tasks [Di...., Dr Each task D. = D w()) contains N, samples, where æi E X is an input and yf E e is its corresponding label. The primary objective is for the model, after being trained on task L, to perform well on the set of all classes encountered up to task t, denoted by yol:t = U yr We focus on the challenging
class-incremental learning scenario (Van de Ven & Tohas 2019 Wang et al. 2022b 2023), where the label spaces of different tasks disjoint, i.e., y 25" CL is that data from previous tasks Di....,22 ......,D-11 is unavailable when training on the current task D,, This sequential training process, without access past data, makes the model susceptible to
Ø for any # t'. A standard constraint in
2

---

Published as a conference paper at ICLR 2026
catastrophic forgettig (McCloskey & Cohen 1989 Nguyen et al. 2019 Mehta et al. 2023) wherein
performance on previously learned tasks degrades significantly as the model adapts to new ones.
2.2 PROMPT-BASE CONTINUAL LEARNING
In vision tasks, prompting is typically applied to the Vision Transformer (ViT) (Dosovitskiy et al. 2020), which consists of a sequence of Multi-Head Self-Artention (MSA) (Vaswani et al., 2017)) blocks. To illustrate how prompts the input to an MSA layer be a sequence of token embeddings (x)...., &N] E RRNxd xd where N is the sequence length and d is the embedding dimension. The MSA operation is defined follows:
integrated, first describe the standard MSA mechanism. Let
MSA(X 1 .²) Concat(hh,,..... ....... ",,,,/mW" ER
h: Attention (eteenttionx""""" X x XVW"), i i.
(1)
where X² = = X = [ ØN are the query, key, and value matrices, respectively;
m is the number of attention heads; and Wo E IRddx xd W.O E Rdxdd, W.K E Rddd, and wy E Rdxd, are projection matrices, with dg = dg d.
Most prompt-based methods (Wang al. 2022a 2023 Jiao et al. 2024) adopt Prefix Tuning (L & Liang 2021 Le et al. 2024b), which introduces learnable prefix key p E RN xd and prefix value parameters, prepended the input key and value matrices of the MSA layer:
p
pk'
pV
)
fPee [X9 X K ) = MSA x
W (2)
X
x
During training, only the prompt parameters pk and pV are updated, while the pre-trained VIT parameters, including WO, W.O. W.K, and W.Y. remain frozen.
Conventional prompt-based continual learning methods, such as DualPrompt (Wang et al. 2022a). HiDe-Prompt (Wang et al. 2023), and NoRGa (Le et al. 2024a), typically allocate separate set of prompt parameters for each task. However, recent work by Huang et al. (2024) demonstrates that single prompt shared across tasks can achieve performance comparable to that of task-specific prompting, raising questions about the efficiency and utilization of prompts in prior approaches. While subsequent works have attempted address related issues, they still present limitations. For instance, the mechanism proposed by Gao et al.] (2024) to mitigate incorrect prompt selection at test time still results linear growth of prompts with the number of tasks. Similarly, the strategy of Roy et al. (2024) to manage prompt growth by employing Large Language Models (LLMs) introduces significant computational overhead due to its reliance on external model.
2.3 MIXTURE OF EXPERTS
Mixture of Experts (MoE) extends classical mixture models by introducing an adaptive gating mechanism (Jacobs et al. 1991 Jordan & Jacobs 1994). For a given input h E IRd, the MoB model computes outputs from N' experts fy R² R. which are then combined using weights from
learned gating function IRd + R as follows:
N
N'
exp (x)(h)
ÿ = Σ (5()
(h),
(3)
N t= exp exp (se(h)
J=1
where 8j Rdd IR denotes the score function for expert fz- To enhance scalability, Shazeer et al. (217)) introduced the Sparse Mixture of Experts (SMoE), variant that activates only the top-K experts with the highest scores. The of these indices, denoted Kn, is formally defined as:
Kn.
arg mar SC NC jES
Σ sh)...
(4)
The final output is then computed as:
Σ (x)
JEKn ΣIEKn exp (se (se(h)) fh).
By routing only a small subset of experts per input, SMoE achieves high model capacity with limited computational overhead. This property has driven adoption in large-scale systems, including language
models (Fedus et al.) 2022 Jiang et al.) 2024 Comanici et al. 2025). computer vision (Riquelme) et al. 2021 Xue et al. 2023), and multi-task learning (Ma et al.
2018 Yang et al. 2024).
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 MIXTURE OF EXPERTS AND PREFIX TUNING
We adopt prefix tuning as primary prompting strategy, following prior work. Furthermore, a single prompi shared across all lasks, similar to Huang et al. (2024). Recent findings by Le et al. (2024a 2025) show that each attention head in VIT can be viewed structured composition of multiple MoE models, framing prefix tuning as way to add new experts to this structure.
Specifically, consider the output of the I-th head in Equation denoted as h; = [h v Nxd. Let X [a 2 layer, and let pk pl
E IR Nd denote the concatenated input embeddings to the MSA
ER xd and P =
RRxx
PN,
We interpret each attention head as comprising N pre-trained experts fj RRNd - Rd., together with Np prompt experts fN+3' Rd + Rd introduced via prefix tuning:
fx( W. zj W/ EjX, (x+ Pjr j =1, l,.... N. and j' = l...., Np- Here, E, E R.dxNd is selection matrix such that E,y X Z.j. The corresponding score functions are defined as:
V
$1.j (X) = 2; W,W I
X N.W.WWY E,X
VT
Van X X E, WOWK W P
8 +2 (X) x; W WOWKK P
dd
Vd.
for =, N. = l...., , N, and j' = l,..., Sp. With these definitions, the output the i-th token in head I can be expressed an MoE model:
exp(si,(X))
Ek exp(si, Exn exp(s,,+*(X)) (X)
j=1
exp($t,N+)(X) (X))
k= exp(ssx(X)) - Ex exp(s..++ (N+
(6)
Thus, each attention head can be regarded multi-gate MoE model, where each output token h is itself an MoE model. Crucially, only the prefix parameters pK and pV are learnable. Adaptation is therefore restricted to training the prompt experts fN+j'' and their score functions st,N+''. These experts complement the pre-trained experts in the attention head, enabling efficient task adaptation.
3.2 SPARSE MIXTURE PROMPT EXPERTS
The preceding analysis reveals that even when single prompi is shared across tasks, multiple prompt experts are still added to the pre-trained model. Their simultancous activation leads to repeated updates across tasks, inducing inter-task interference. To address this, propose SMoPE, novel sparse mixture of experts architecture built on prefix tuning. By selectively activating only subset of relevant prompt experts, SMoPE introduces implicit parameter partitioning that mitigates interference.
Prompt-Attention ttention Score Aggregation. Unlike standard MoEs where each expert has one score function, the MoE architecture within the attention head utilizes a multi gattiggtteeeee mechanism. Specifically, under prefix tuning, each prompt expert SN+j' has score functions & for i = 1,..., N, ren- dering direct application of the standard SMoE framework non-trivial and computationally intensive. To reduce this complexity, we introduce a unified proxy score that aggregates these individual scores:
Prompt-A
SL,NJ N
X
W KT p
&TWO
Vd,
Vd
where E = = E E, is the mean extraction matrix and ã + Em x, denotes the average token representation. Importantly, this formulation eliminates the need compute all individual
4

---

Published as a conference paper at ICLR 2026
scores Si,N-j'; computing the single vector I't suffices to obtain the proxy scores 5yy These proxy scores are then used to determine expert selection and output composition:
N
exp(st)(X))
exp(s; (X)) + Ex exp($x (x)) {x)
k=
exp($)((X)
Σ j'Exx exp(ex Ex exp(sx (X))
(N+5(())
(8)
Sample Complexity of Estimating Prompt Experts. Although the prompt-attention
aggrega- tion is designed to activate only a subset of relevant prompt experts, we show Appendix A that the SMoPE model in Equation (8) maintains the same sample complexity for estimating prompt experts as the standard MoE in Equation ( Specifically, estimating the prompt experts fN+ (X) in each MoE variant with an estimation error of of the order (rr=)) Therefore, SMoPE is as sample- aample-fffcientt as the standard MoE for prefix tuning.
requires most a polynomial number of data points
Sparse Prompt Expert Selection. Each prompt expert fN+ is now associated with a single unified score 83, enabling the efficient selection of the top-K most relevant experts using Equation This yields the active expert set K x, and Equation (8) becomes:
exp(si,(X))
11.
E k= exp(si, + ENEKX cxp(S (X)) fx(
exp(3,((X)
+
Σ J'EKx Ek exp(ss,(()) + ΣNEKX exp(sw (X)) [N+)((X).
(9)
Similar OVOR (Huang al. 2024). SMoPE employs single prompt shared across all tasks. ensuring high parameter efficiency. However. unlike OVOR, SMoPE activates only a sparse subset of K prompt experts per input within each MSA layer, introducing implicit parameter partitioning. By updating only the relevant parameters, SMoPE reduces knowledge interference and mitigates catastrophic forgetting. Importantly, this parameter selection relies solely on the current layer's input X and does not require precomputed query representations from a full model forward pass, as in prior task-specific prompting methods (Wang et al. 2022a| 2023) Smith et al. 2023 Le et al.. 2024ap. As result, SMoPE offers substantial reductions in computational overhead compared to these approaches (Kim et al., 2024). Furthermore, by dynamically reusing relevant experts across tasks, SMoPE naturally promotes knowledge sharing and transfer.
Implementation Attention Layers. We now describe how the MoE formulation in Equation 9 integrates with the attention mechanism. In prefix tuning, the attention matrix A₁ for the 1-th head is constructed as follows:
xOW,WK T [pKT X
A
Aprompi
A.T trained
Vd
X X W T dd,
PKT
XOW XKT dd,
(10)
This decomposition shows that the full attention matrix comprises scores from both the prompt experts (Aprompy and the original pre-trained attention (Apreetrainedd Since SMoPE modifies only the prompt components, we adjust A Apropp while keeping Apre-trained unchanged. Based on Equation ( this adjustment requires only the computation æ. The SMoPE-adjusted attention matrix is then:
( @TT&&w W.RKK T PK
A = Promps
A pre-trained]
Aprompe TopK
eexpand(N, -1). (11)
Vd
Here, the Top-K operator selects the K most relevant prompt experts based on the proxy scores By, which are computed once per input and shared across all output tokens h Notably, while conventional prefix tuning computes N scores per prompt expert at a cost of ((Nd)), SMoPE reduces this to single proxy score (dz)), yielding up an N-fold reduction in complexity.
5

---

Published as a conference paper at ICLR 2026
The I-th Attention Head
Topk
Softmax
Pramppt Exparts
Add noise
pv
Expand by
A
prompt
A
pree ained
Prestrinne Exports
Activation Frequency
Prompt Experts
pk
w/
w
W
componant
Pre-trinnad companent
Score sciedd noina
Hyss Embreding
X
Figure 1: SMoPE Implementation Attention Layers. The attention mechanism for each head is composed of both pre-trained and prompt components. standard self-attene seff-attention. To construct the prompt attention matrix Aprompt first calculate the average input representation σ. and evaluate the scores for prompt experts. During training. frequently activated prompt experts are penalized by applying an adaptive noise to their scores, which promotes exploration underutilized experts for new tasks while preserving essential knowledge in critical experts. A Top- K selection operator then identifies the most relevant experts based on these adjusted scores. The selected scores row-expanded to form A Finally, A is concatenated with A Apre-trained to produce the final attention matrix, which is applied to the expert representations via dot product, similar to the standard self- attention mechanism.
pre-trained attention matrix A! trained computed using
3.3 BALANCING EXPERT UTILIZATION WITH ADAPTIVE NOISE
A well-known challenge in training SMoE models is the imbalance in expert utilization, where small subset of experts dominates routing decisions while others remain underutilized or inactive (Mu & Lin 2025). This imbalance reduces the diversity of learned representations and ultimately limits model effectiveness. We find that SMoPE exhibits a similar issue: during training, only a small group of prompt experts is consistently activated across tasks. While SMoPE mitigates interference by selectively activating experts, repeatedly relying on the same subset still risks knowledge interference.
Adaptive Noise Mechanism. To address this issue, introduce novel adaptive noise mechanism that promotes more balanced expert utilization in continual learning, as follows:
Kx
arg max
Σ (X) - Gr).
(12)
SC(1,...,,:11111-
J'ES
Cy' (max, 3x(X) min, 8x())
if trainn and Fg N, Σラ N₂
(13)
otherwise
where E [0, 1] is hyperparameter, and F3- denotes the accumulated activation frequency of prompt expert fN+ Lee, the proportion of instances in which expert fN+j' was selected across previous tasks. We define prompt expert as important its activation frequency exceeds the average across all prompt experts within its attention head. These experts, having been frequently activated, are expected to encode essential prior knowledge. The adaptive noise mechanism specifically targets these important experts, applying the noise penalty only to those with activation frequencies above the mean. This penalization encourages the selection of underutilized experts when adapting to new tasks, while protecting knowledge stored in the important experts that are already highly activated. Please refer Figure [ for illustration.

---

Published as a conference paper at ICLR 2026
24
Expert
Expert
Expert
Figure 2: Activation Frequencies of Prompt Experts. Results CUB-200 with the prompt length N₂ 25 and K = 5. We show representative attention head and visualize the frequeney with which prompt experts are activated after training on tasks under different values
As shown in Figure setting e 0.0 results in the repeated activation of small subset of experts. while ı = 1.0 spans the entire dynamic score range, heavily penalizing frequently used experts and enforcing full expert utilization. Intermediate values provide a smooth trade-off between exploration and stability. By balancing expert utilization in this way, SMoPE can more effectively adapt to new tasks while preserving the knowledge encoded in important experts, thereby mitigating catastrophic forgetting. Further discussion and experimental results can be found in Appendix D.3
3.4 PREFIX KEY PROTOTYPES FOR EXPERT SPECIALIZATION
Promoting Expert Specialization. During both training and inference, each MSA layer receives an input X and selects subset of prompt experts. To facilitate this selection, we encourage expert specialization, whereby each expert focuses on distinct regions of the input space. This enables more reliable routing, as specialized experts are more easily identifiable for given input. To promote such specialization, we introduce the following objective:
exp (5 (X))
Lroater =
Σ ΣNEKX exp (8
(14)
x)) + EK'KK exp (S (X))
J'EKx
This objective encourages higher scores (x)) for experts in the selected set Kx while suppressing the scores of unselected experts. As a result, experts develop more distinct input-space specializations, reducing redundancy and improving routing accuracy.
Prefix Keys as Prototypes. From Equation optimizing By X) corresponds to updating the prefix key p... Since the score function determines expert activation, the prefix key effectively defines the region of the input space in which each expert specializes. Thus, optimizing Croneer requires updating both selected and unselected prefix keys.
However, in continual learning, this poses challenge: past-task data unavailable, and up- dating unselected prefix keys may lead to overwriting prior specializations. To address this, we propose treating prefix keys from earlier tasks as prototypes, which serve implicit memory representations of past input distributions. Specifically, we define the prototype set: D
(pda.s 1 S S N,, Fy 2 E } where p denotes the prefix key after training on the previous task. Only frequently activated experts are retained to avoid noisy uninformative prototypes. We then define the prototype based objective:
exp (p p3))
Eproto
Σ E
(15)
p DmD pm J'EK, EN exp (p p
where Kp
NC Eyes (p p/)) denotes the top-K experts activated by prototype p.
argmaa max
While Crouee promotes specialization using current-task data, Cporo preserves specializations learned from previous tasks, thereby mitigating catastrophic forgetting without relying on past inputs.
Additional Training Strategies. continual learningg the MLP classifier head often develops a bias
toward newly introduced classes (Hou et al., 2019 Belouadah & Popescu 2019). To mitigate this, adopt task-add adaptive prediction, following prior work (Wang et al. 2023 Jiao et al. 2024 Le et al.
2024a). This technique adjusts predictions using the representation statistics of previously learned
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on ImageNet-R, CIFAR- 100, CUB-200 200 (10-task splits). 1 indicates
higher is better. FAA and CAA are averaged over runs. Bold denotes the best results, excluding joint training.
ImageNet-R
CIFAR- 100
CUB-200
Method
FAA (t)
CAA (1)
FAA (t)
CAA
FAA (1)
CAA (T)
Joint-Train
82.06
91.38
88.00
1.2P++
69.29 + 0.73
78.30 0.69
82.50 1.10
88.96 + 0.82
77.18 0.71
84.31 0.17
Deep L2P++
71.66 + 0.64
79.63
1.90
84.30 1.03
00.50
78.64 0.60
86.14 0.57
DualPrompt
71.32 0.62
78.94 0.72
82.37
0.29
87.10
77.27 0.31
81.82 0.15
CODA-Prompt
7
0.10
7
0.23
OVOR
75.25
79.78 0.65
86.91
91.02 + 1.00
77.45 - 0169
85.81 +..1+11 1.56
HiDe-Prompp
74.25
79.64 0.53
0.59
91.38 + 0.78
85.60 1:00=:0 0.05
90.16
NoRGa
74.39+0..
79.68 19.68+0.11 01.41
8.13
92.06 + 0.51
85.78 +..8++0.21 01.24
00.63+0.33
ConyPrompt
+.45
81.46+0..5 0.67
0.36
92.49 0.89
22.56-0.66
+.99
CPrompt
77.14
87.82
92.53 = 0.23
80.35
1 0.44
85.66 0.59
VQ-Prompt
78.71 0.22
83.24 0.68
88.73
0.27
92.84 + 0.73
86.72 0.94
90.33 1.03
SMoPE
79.32 + 0.42
84.39 0.77
89.23 1 0.12
93.67 + 0.82
87.43 0.39
91.11 0.55
Table 2: Performance comparison on ImageNet-R (10-task split) for self-supervised pre-training, iBOT-IK, and DINO-1K. t indicates higher better. FAA and CAA are averaged over runs. Bold highlights best results.
BK
DINO-IK
Method
FAA (t)
CAA (f)
FAA (t)
CAA (t)
DualPrompt
61.51 + 1.05
67.11 + 0.08
58.57 + 0.45
64.89 + 0.15
CODA-Prompy
66.56 + 0.68
73.14 0.57
63.15 0.39
69.73 + 0.25
HiDe-Prompl
65.34 0.37 0.37
71.67 + 0.56
#2.94
70.04
NoRGA
65.81 + 0.52
72.11
63.02
70.25 + 0.61
OVOR
69.46 + 0.46
74.95 0.63
66.55
72.89 0.71
VQ-Prompt
71.68 + 0.72
76.66
0.40
68.42
74.43 +: 0.58
SMoPE
72.17 1 0.36
7..24
68.61
1
classes to correct classifier bias and stabilize prompt learning. Additionally, inspired by advances in SMoE training (Wu al., 2022 Lin et al. 2024 Cai et al. 2025), we apply dense expert training (i.e. without sparse selection) during the initial epochs of the first task. This aids in establishing stable expert representations before enabling sparse routing.
Final Optimization Objective. The complete loss function for our SMoPE framework is:
Lee Orooner Prouter + Q'proto
Cproto
(16)
where Lee is the standard cross-entropy loss, and O'youter and Oproooo are weighting hyperparameters. During training, only the prefix parameters and classifier head are updated, while the pre-trained backbone remains frozen. Please refer to Appendix B for further details on the training algorithm.
4
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datasets. Following Jiao et al. 2024). we evaluate SMoPE on three representative CL benchmarks: ImageNet-R (Boschini et al. 2022). CIFAR- (Krizhevsky et al., (2009), and CUB-200 (Wah et al.) 2011}. ImageNet-R consists of 200 challenging classes, including difficult samples from ImageNet and newly collected data with diverse stylistic variations, and is split into 5, 10, or 20 disjoint tasks. CIFAR- 100 contains 100 classes, which we randomly partition into 10 tasks. CUB-200 comprises fine-grained images of 200 bird species, randomly divided into 10 tasks with 20 classes each.
Evaluation Metrics. We adopt two widely used CL metrics: Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA). FAA is the average accuracy after training on all tasks, while CAA represents the mean of the FAA values recorded after each task.
Baselines. We compare approach against several representative prompt- based continual learn- ing methods, including L2P (Wang et al.) 20226), DualPrompt (Wang et al.) (222a)) and CODA- Prompt (Smith al. 2023 Additionally, evaluate against recent state-of-thh-art methods, such
8

---

Published as a conference paper at ICLR 2026
as Conv (Roy et al. 2024), CPrrmpt (Gao al. ) and VQ-Prompi (Jiao et al. 2024).
We also consider OVOR (Huang et al., 2024), which employs a single prompt, and state-ot of the-hhr art task-specifie prompt methods such as HiDe-Prompt (Wang al. 2023) and NoRGa (Le et al. 2024a).
For I.2P, we follow Smith et al. (023) and include two variants: "I.2+++,, which replaces prompt tuning with prefix tuning, and " "Deep I.2P++", which extends 1,2++ by inserting prompts into the first five MSA blocks. Finally, we report the performance of "Joint-Tran Soint-Train", an offline upper bound obtained by training on all tasks simultaneously.
Implementation Details. All experiments are conducted on a single NVIDIA A100 GPU. To ensure a fair comparison, all baseline methods are re-implemented using the configurations reported their original papers. Following Diao et al. (2024). we adopt a ViT-B/16 backbone pretrained on ImageNet- 1K (Russakovsky al. 2015) and ImageNet-21K (Ridnik et al., 202T) as the backbone. We also evaluate self-supervised ViT-B/16 models from iBOT-1K (Zhou et al. (021)) and DINO-1K (Caron al. (02T). The prompt length is fixed at N, 25, and the number of selected prompt experts set to K = 5 for all experiments. Prompts are inserted into the first six MSA blocks. SMoPE is optimized using Adam W (Loshchilov & Hutter 2017) with a cosine learning rate decay schedule. The batch size is set to 64 for ImageNet-R, and 128 for both CIFAR- 100 and CUB-200, Additional implementation details are provided in Appendix
4.2 EXPERIMENTAL RESULTS
Overall Comparison. Table summarizes the performance of SMoPE compared prompt-based continual learning baselines. SMoPE achieves the best overall results across all three benchmarks, consistently surpassing prior methods on both FAA and CAA, demonstrating strong resistance to forgetting while effectively adapting to new tasks. On CIFAR-100 and CUB-200, OVOR, which employs a single shared prompt, performs competitively with L2P, DualPrompt, and CODA-Prompt but falls short of task- task-specific methods such HiDe-Prompt and NoRGa. Notably, SMoPE breaks this trade-off: despite using only a single shared prompt, it not only outperforms OVOR by a large margin but also surpasses task-specific methods, indicating that its structured design overcomes the limitations of shared prompting. Its performance is also comparable to that of VQ-Prompt, a recent state-of of-the-art method. These results highlight that SMoPE mitigates forgetting while maintaining the efficiency of shared prompt, achieving favorable balance between effectiveness and efficiency without relying on task-specific model expansions.
Performance with Different Pre-training Paradigms. In line with prior work (Wang et al.,
Table 3: Comparison of computational cost on ImageNet-R (10-task split), including lcarnable parame- ters (millions), training and inference costs (GFLOPs), and relative cost (%) to L.P.
2023 Jiao et al. (024). we conduct experiments on IetgeNet-R R using two self-supervised pre-
training paradigms: iBOT-1K 1K (Zhou et al.] [202T) and DINO-1K K (Caron al. 2021). The results, presented in Ta demonstrate that SMoPE surpasses all other prompt-based continual learn- ing baselines, highlighting the generalizability and robustness of SMMPEE's ''s design across differ- pre-training paradigms.
Method
Params (M)
Train GFLOPs
Test GFLOPs
Deep L2P++
4.78
(100%)
67.44 (100%)
DualPrompt
1.10
33.72 (50%)
67.44 (100%)
CODA COmA-Prompt
3.99
67.44 (100%)
67.44 (100%)
HiDe-Prompt
4.21
33.72 (50%)
67.45 (100%)
NoRGa
4.21
33.72 (50%)
67.45 (100%)
OVOR
0.26
33.72 (50%)
33.72 (50%)
VQ-Prompt
0.50
67.44 (100%)
67.44 (100%)
SMoPE
0.38
33.72 (50%)
33.72 (50%)
Computational Cost Analysis. To evaluate the efficiency of SMoPE, compare the number of learnable parameters, training GFLOPs, and inference GFLOPs, as shown in Table 3 By using a single prompt for all tasks, SMoPE sig- nificantly reduces the number of parameters. It also avoids passing the input through the full pre-trained model to compute a query, unlike prior approaches. Instead, each MSA layer se- lects prompt experts based on the input at that
Table 4: Ablation study of the proposed SMoPE on CUB-200, split into 10 tasks. FAA and CAA av- eraged over 5 runs. indicates that higher values are better. Bold highlights the best results.
Training Strategy
FAA (1)
CAA (1)
One Prompt
75.23 0.17
83.61 1 0.48
Prompt Score Aggregation
75.49
83.65
Sparse Expert Selection
7..1
87.16 0.56
layer, resulting in a substantial reduction in com-
+ Adaptive Noise
89.12 8..3
+ Task Adaptive Prediction
86.03
90.09
0.43
putational cost up to 50% %. These design choices highlight SMoPE's efficiency over spsk-spe prompting methods while maintaining compet-
+ Initial Dense Training
0.446
90.23 1 0.52
+ Router Loss Lowwer
87.05 0.40
90.
+ 0.40
Prototype Loss Lpmnto
87.43 0.39
91.11 0.55
itive performance.
9

---

Published as a conference paper at ICLR 2026
Ablation Studies. To evaluate the contribution of individual components in SMoPE, conducted a series of ablation experiments (see Table(4). The baseline configuration, "One Prompt", employs standard prefix tuning with single prompt shared across all tasks. Removing all SMoPE components results a significant performance drop, primarily due to the continual updating of shared parameters. Introducing the prompt-attention score aggregation mechanism alters the attention computation and achieves performance comparable to the bascline, consistent with our analysis in Appendix [ Adding sparse expert selection further improves performance, although the gains are limited by imbalanced expert utilization. Notably, integrating the adaptive noise mechanism leads to substantial performance improvements, underscoring its role in reducing interference and mitigating forgetting. Incorporating all components yields the highest overall performance, confirming the complementary nature of SMoPE's design. These ablation results collectively validate that each component contributes mean- ingfully to the final model performance, and their integration is essential for SMoPE's effectiveness.
5 DISCUSSION AND CONCLUSION
In this paper, we introduced SMoPE, prompt-based CL method that balances the paradigms of using single shared and task-specific prompts. By integrating sparse MoE architecture into prefix tuning, SMoPE reduces knowledge interference by updating only task-relevant parameters. We also proposed an adaptive noise mechanism and a prototype loss function that leverage prefix keys as an implicit memory of previous tasks, further enhancing performance. Extensive experiments demonstrate that SMoPE effectively reduces catastrophic forgetting while maintaining adaptability to new tasks, with significantly improved parameter and computational efficiency compared to existing methods.
Despite these promising results, several avenues for future work remain. Although our approach shared prompt structure across tasks, which mitigates forgetting more effectively than prior methods, interference may still arise as the number of tasks grows. Future research could explore dynamically expanding the prompt length increasing the number prompt experts to better address continual learning demands. Investigating how SMoPE scales with the number of prompt experts may also yield valuable insights. While our experiments focused on ViT, extending SMoPE to other foundational models would help assess its generalizability. Finally, though designed for continual learning, SMoPE's architecture may have broader applications warranting further exploration.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
In order to facilitate the reproduction of our empirical results, we provide detailed descriptions of the experimental setup in Section 4 and Appendix All datasets used in this study are publicly available, enabling full replication of our experiments.
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 3366-3375, 2017.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 583-592, 2019.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. European conference on computer vision, pp. 692-709. Springer, 2022.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 9660, 2021.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208. 028/3, 2022.
Gheorghe Comanici. Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626: 2025.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jía, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-338 3336-3355,, 2021.
Nghiem Tuong Diep, Huy Nguyen, Chau Nguyen, Minh Le, Duy Minh Ho Nguyen, Daniel Sonntag, Mathias Niepert, and Nhat Ho. On zeno-initialized attention: Optimal prompt and gating factor estimation. In Forty-second International Conference on Machine Learning, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint P2Xiv:2010111920 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 13(1 2022.
Qian Feng, Da-Wei Zhou, Hanbin Zhao, Chao Zhang, and Hui Qian. LW2g: Learning whether to grow for prompt-based continual learning, 2025. URL hatp:::tt///cc. httt:::://ppeezeviee wttpeeewwwwweeeeeeeeeeeeeeeeeeee (d=0ZuZmfLLRRE
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal- freeelffeeeee continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024.
11

---

Published as a conference paper at ICLR 2026
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 831-839, 2019.
Wei-Cheng Huang, Chun-Fu Chen, and Hsiang Hsu. Ovor: Oneprompt with virtual outlier regular- ization for rehearsal-free stass-nccemental learning. arXiv preprint arXiv:2402.04129, 2024.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Li Jiao, Qiuxia Lai, Yu Li, and Qiang Xu. Vector quantization prompting for continual learning. Advances in Neural Information Processing Systems, 37:34056-34077 2024.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures experts and the em algorithm. Neural computation, 6(2): 10::181-221 214, 1994.
Youngeun Kim, Yuhang Li, and Priyadarshini Panda. One-stage prompt-basee based continual learning, In European conference on computer vision, pp. 163-179. Springer, 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, 37:119025-11966,, 2024a.
Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning Statistical benefits of reparamcterization among prompts. arXiv preprint arXiv:2410.02200, 2024b.
Minh Le, Tien Ngoc Luu, Thanh-Thien Le, Trang Nguyen, Tung Thanh Nguyen, Linh Ngo Van, Thien Huu Nguyen, et al. Adaptive prompting for continual relation extraction: a within-task variance perspective. In Proceedings the AAAI Conference Artificial Intelligence, volume 39, pp. 24384-2433999 2025.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001 101000990,,, 2021.
Bin Lin, Zhenyu Tang, Yang Yc, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-Ilava: Mixture of experts for large lisoun-language models. arXiv preprint ar arXiv:2401.15947, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed Chi. Modeling task relationships
in multi-task task learning with multi- gate mixture-offexperte In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1930-1939. 2018.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Sanket Vaibhav Mchta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214): 1-50, 2023.
Siyuan Mu and Sen Lin. A comprchensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07/37, 2025.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXXiv/9088 01091, 2019.
12

---

Published as a conference paper at ICLR 2026
Huy Nguyen, Pedram Akbarian, Trang Pham, Trang Nguyen, Shujian Zhang, and Nhat Ho. Statistical advantages of perturbing cosine router in mixture of experts. arXiv preprint 2024.
Hang Qi. Matthew Brown, and David Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5822-5830, 2018.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretrainieg for the masses. arXiv preprint 2tt1::2004.19972 2021.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583 85955, 2021.
Anurag Roy, Riddhiman Moulick, Vinay Verma, Saptarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211 11-2222,, 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- offture-of-exppets layer. arXiv preprint arXiv: 1701006880,. 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Fascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal- free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11909-11919, 2023.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Quyen Tran, Nguyen Xuan Thanh, Nguyen Hoang Anh, Nam Le Hai, Trung Le, Linh Van Ngo, and Thien Huu Nguyen. Preserving generalization of language models few-shot continual relation extraction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1377 1-3744, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 18.18653/v1/ 2024.emnlp www.e-mainlmmnnnn.. main.main.......
URL https: http::::///claatthoogo..o org. 2024 emn coannppmman 1w-mmin....n..... 7
Quyen Tran, Tung Lam Tran, Khanh Doan, Toan Tran, Dinh Phung, Khoat Than, and Trung Le. Boosting multiple views for netrained-based continual learning. The Thirteenth International Conference on Learning Representations, 2025. URL /tttts:///ppereeeeeeee
net/ for forum?
Ad=AZRRR31w7?
Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, and Nhat Ho. Replora: Reparam- eterizing low-rank adaptation via the perspective of mixture of experts. In Proceedings of the 42st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vancouver, Canada, 2025. PMLR.
van de Geer. Empirical processes in M-estimation. Cambridge University Press, 2000.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 1rXXvv:90.007733 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
13

---

Published as a conference paper at ICLR 2026
Liyuan Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36:69054-6990 69076, 2023.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free -ree continual learning. European conference on computer vision, pp. 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Lemeng Wu, Mengchen Liu, Yinpeng Chen, Dongdong Chen, Xiyang Dai, and Lu Yuan. Residual mixture of experts. arXiv preprini TrXiv:2204.09630 2022.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, (6641693-417066 2023.
Yuqi Yang. Peng-Tai Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 27927-27933 2024.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19148-19158, 2023.
Linglan Zhao, Xuerui Zhang, Yan, Shouhong Ding, and Weiran Huang. Safe: Slow and fast parameter-efficient tuning for continual learning with pre-trained models. Advances in Neural Information Processing Systems, 37:113772-113796, 2024.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1.00833 2021.
14

---

Published as a conference paper at ICLR 2026
Supplement to "One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning"
This supplementary material provides: theoretical analysis of prompt estimation rates when using prompt-attention score aggregation (Appendix AA); the detailed training algorithm for SMoPE (Ap- pendix [); additional experimental details (Appendix C and results (Appendix D): and discussion on the use of large language models in this work (Appendix E
A THEORETICAL ANALYSIS OF PROMPT-ATTENTIOS SCORE AGGREGATION
In this appendix, we aim to compare the sample complexity of estimating prompt experts with and without prompt-attention score aggregation. To this end, conduct convergence analysis of prompt expert estimation. The results of the analysis reveal that employing prompt-attention score aggregation is sample-efficient as not using it, which indicates that this choice does not affect the model's s ability to adapt to new tasks. Before stating the problem formally, let us introduce the notation that we use throughout this appendix.
Notation. For any natural number n E N, let [n] = {1, (1,2,...,n)) n). For a vector u E IRd, we use both v = (n(2), (2) ..... uld)) and и = (u1, u ua) interchangeably. Given a multi-index = (a1, ag, .........,,..................... ..).....)))))... E Nd, write ut" uj un u " |u| 01 + U2 + ..... + Udd and al aj ay!a?!... org. The Euclidean norm of u is denoted by the term ||2]], while [S] refers to the cardinality of any set S. For two positive sequences (an)n21 and (bm)n21. we write a, = O(b,) or a, & b, there exists constant C' > 0 such that an S Cb, for all n. The notation a,, = Op(b,) means that an Bn is stochastically bounded. We further write an Or(b,) when an (r,,, (b.)). for some c > 0.
As established in Section 3.2 the MoE models f ... hi in each attention head share common structure of experts and score functions. Therefore, to simplify our analysis while maintaining rigor, we focus on the first head (i.e., = 1) and the first row of its attention matrix (i.e. i = 1). Within this framework to analyze the convergence of prompt
simplified setting, we present a regression-basee expert estimation in an MoE model for prefix tuning. This framework has previously been used to study the asymptotic behavior of other MoE-baser
-ffficieeefffiient fine-tuming methods, namely, low-rank adaptation (Truong al. 2025) and I.LaMA-Adapte (Diep et al., 2025). We begin by defining the problem setting with prompt-attention score aggregation.
Problem Setting with Prompt-Attention Score Aggregation. Suppose we have i..dd data (X2, (Xn, Yn) Rad R generated from the following model:
Y, ga- (X.)+ i=
(17)
where the noise terms Ei,.... En independent Gaussian random variables with mean zero and variance v?2 The covariates X Xy...., X,, are drawn i.i.d. from some probability distribution μ. The regression function go- is composed of N pre-trained experts and N, learnable prompt experts:
Σ/ cxp(X B 2) h(X. n
9G- (X) =
Σ exp X B + +c4) + Ex' exp (BiTe) T WT X + 3
Σx= expl exp(( ((3 TW X (op)) ((X,((X (x,))
(18)
N -k-1 exp X x + +c2) E exp (( TW X + (ōx))
N₂ exp((ts)) 660 m3)) represents the mixing measure, i.e., weighted sum of Dirac Here, G* Ey= measures associated with the parameters S nf..) of the prompt experts. The matrix B9 plays the same role as the matrix
E.. w. E, in the score function 81.5( (X), while the vector Big and Vd.
the matrix W. respectively, correspond to the components ET B"W?w:? 2 used in the modified score J.. function šyr (X). The expert functions h(X,, n7²) correspond the pre-trained experts fX),, while h(X, nn.)) correspond to the new prompt experts fN+y/ (X). In general, the expert functions can have flexible parametric forms, not restricted to the simple forms used in the linear- linear-gating gating-prefix MoE model (Le et al. 2024a).
15

---

Published as a conference paper at ICLR 2026
Least Squares Estimation. We use the least squares method (van de Geer 2000 to estimate the unknown parameters (Ghy: Miy. n No estimator, denoted by Gn. is defined as:
or, equivalently, the true mixing measure G". The
Gn = arg
min
Σ 1=1
(X, go gx())))
(19)
GEONG (0)
where
9x₂(0) = Gx= Σ exp( (op( ((.)))) : -Ss No (Bor Br n) E {)
the set of mixing measures with at most Ny components. Since the true number of experts, Npr is typically unknown practice, we assume that the maximum number components, N₁, chosen to be an upper bound such that N2 2 Np.
Definition A.1 (Strong Identifiability). We say an expert function ht., 17) is strongly identifiable and only if it is twice differentiable with respect to its parameters, and if for any k 2 and any set of pairwise distinct parameters T,,....,,l T.j., the collection of functions
0l h X
(X, m.) :jE [K], VENN 7EN", 05 1 θmp
is linearly independent for almost every X E Rd
Intuitively, this condition prevents unwanted interactions between the parameters of the expert func-
tion h(:, n). In practice, it guarantees that the regression difference ÿo (x) 9c- (X) can be expanded into linearly independent terms via a Taylor expansion of expressions i.e., exp(3] W X)h(X X 1)). Thus, the above condition guarantees that all the derivative terms in the Taylor expansion are linearly independent. The following example illustrates class of expert functions lt,, 77) that satisfy this condition.
Example. Consider a neural network expert of the form 4(X, b)) = o(a" X + b), where the parameters
(a, b) R x IR and the activation function is (ReLU, GELU, 2 + 2²). This form of expert function satisfies the strong identifiability condition under mild assumptions on the activation function á.
Voronoi Loss. To quantify the discrepancy between an estimated measure G and the target measure G*, we define
D)G
Σ
Σ
exp(Boi) (I
AAm 123
3 y |21 V
Σ
Σ exp(33s) (14311511 ||
N
+
exp(Bni) - exxp exp(3
(20)
KEV.
Here, the notation Abpy' == B1 Bi and Anyy' := 1) - nf. represents the difference between estimated and true parameters. The partitioning of the estimated components is based on Voronoi cells. For each true component wy. = (Bi nffr), the corresponding Voronoi cell V/ = (G) contains the indices of the estimated components that are closest to it. Formally, it is defined as
V/- ==
{ i
- w < ||ا w VL
(21)
where wi := (B1,, TR) denotes an estimated component from G. Consequently, the cardinality of cell |V,1 indicates how many components w; from G are used to approximate the true component w in G*
By construction, a small value of DGG, G*) implies that the estimated parameters are close to the true parameters. Based on the observation, although this loss function D(G, G.) is not symmetric, it
16

---

Published as a conference paper at ICLR 2026
suitable measure for analyzing the convergence of the least squares estimator G,, With this loss function defined, we can now present the sample complexity estimating prompt experts with prompt-attention score aggregation.
Theorem A.2. Under the strong identifiailiiyy condition for the expert function ((X.1), m)), the least squares estimator GR converges to the true measure G* at the following rate:
D(G,o G*) Op(nn112)
(22)
The proof of Theorem A.2 is deferred to Appendix A. Several remarks on this result are in order.
(i) Convergence rate of prompt parameter estimation. This theorem, combined with the definition of the Voronoi loss in Equation (20), implies that the convergence rates for the prompt parameters Tij range from Op(n 1/4) to Op 1/2), depending on the cardinality the corresponding Voronoi cells V...
(ii) Convergence rate of prompt expert estimation. Denote G, := -i
( (8.. .1) is Lipschitz continuous with respect to its parameter 17 for almost every the expert function (xX,1)
X, ie.,
[x(X, fn,;)) h(X,((XX) nf.)) I -
(23)
then this property. combined with the parameter convergence rates from remark (i), implies that the rates for estimating prompt experts ((X,, ng.)) admit the same orders those for estimating prompt parameters ni.. standing at Öp(n 1/4) or O Op(nn1/
(iii) Sample complexity of estimating prompt experts. As a consequence, when using the prompt- attention score aggregation, need polynomial number of data points, either O(r-a) O(r-2)
to estimate the prompt experts with given error T > O.
Problem Setting without Prompt-Attention Score Aggregation. In this setting, assume the i..d. data (X,, Yi), (X2, Y2),..., (X., Yn) E R x R are generated from the same regression model (7)) but with the following modified regression function:
Σ exp X B (9) n(X, n7))
Ic-
Σ k== exp X B
Ex' exp ((Px.)) T W. X + Pixe)
Σ exp ( T X 9ú,,) ((,77))
(24)
E exp(X B X + 2 Σx' exp ((3 T.T (0R
The key difference between this function and the in Equation (18) is that the weight matrix W is no longer shared across prompt experts. Instead, because there is no prompt-attention score aggregation, each expert j' has its own matrix, Wy, Accordingly, the least squares estimator is now defined as:
Gn = arg mün
(V i=1
GEGN (9)
Theorem A.3. Under the strong identifiability condition for the expert function h h(X, (1,)), the least squares estimator Gn converges to the true measure G* at the following rate:
D(G,, =pn
(25)
It should be noted that the proof arguments for the dense gating have been included in Appendix A. Furthermore, since the weight matrices W are frozen during training, their dependence on the expert index j' do not affect the proof arguments in Appendix A. In other words, the proof of Theorem A.3
can be done similarly to that of Theorem A.2
omitted here.
Comparison of Sample Complexity with and without Prompt-Attention Score Aggregation. Comparing the convergence rates Equations (22) and (5).. we find that they are identical. This indicates that the estimation rate for the prompt parameters is unaffected by the use of prompt- attention score aggregation. As direct consequence of the Lipschitz continuity in Equation (23). the estimation rates for the prompt experts also preserved. We therefore conclude that incorporating prompt-attenton score aggregation into the MoË model for prefix tuning does not change the sample complexity of prompt expert estimation.
17

---

Published as a conference paper at ICLR 2026
A.1 PROOF OF THEOREM A.2
Roadmap. The result follows once we establish the following two inequalities:
119. - 1GG-99.l0 Rec.IlZ(m)) > 0,
inf G (9)
(26)
G.)
119.. -9.. ((P Op(Vlog(n)/n)
(27)
We will prove Equations (26 and in turn.
Proof of Equation (26). We split the argument into local and global part.
Local part. We use proof by contradiction show that:
lga - 9G_ _G_1lL D(G, G.)
lim c
inf GERN; (0): D(G,G.)
> 0.
(28)
Suppose, for the sake of contradiction, that the claim false. Then there exists a sequence of mixing measures, Gn Ein exp( (pp E GNy (0), such that Dn == D(Gn, G.) + 0 and
lge. gG. 12(()) 0.
(29)
D
Let VT = (G.) be the Voronoi cell corresponding to the j-th true component, Since D, - 0. the components of G, converge those of G.- For sufficiently large, the Voronoi partition stabilizes, so we can drop the superscript n and write V, = V for simplicity. The loss D, is then given by
D Σ Σ
(xp((3k) (IAST, 11 I!²)
HAEE REV,
Σ
Σ
exp(3()) (1 (14 + |
exp()6)) exp(365) (30)
;:V1-11 SEV
/=1
NEV
where AB;; S4 B and Am = my. Since D, - 0, we have (3 n7¹) + (8i nj) for i Vj, and the weights aggregate correctly: Eve,, cxp( exp(66j). We now split the proof of the local part into three steps.
Step - Taylor expansion. In this step, we want to decompose the following quantity into a combination of linearly independent elements:
N₂
(x)
Σ cxp(X
X
B
X
+ 2
Σ
"xp((Bt))TTW)
T X [h5y)] [9c., (X) - Ic. (X)
5'=1
(31)
We first rewrite it as follows:
N (x
cxp(
(ex((())))T)) TWT x)h(X; exp((17,) Tw T x)((X:n))
j=n
N
exp (3)) ( TWT x) exp(((ii)) TW x) go. (X)
1iv,
(W exp( - exp(31,)) exp(((8))) TW x) ((x:n))) - IG. (x)
+
1=1
iEV;
A,
(X) (x) + (X).
(32)
18

---

Published as a conference paper at ICLR 2026
Decomposition of (x).. Let denote E 31) == exp( W X), then An can be separated into two terms as follows:
An (x)
Σ
exp(
; (1,) E(X; B mj)]
Jil, =
Σ
F M (x:)) -(X;)) m3]]
=
1 + (X).
By means of the first-order Taylor expansion, we have
exp(Pot)
(M (An))) )0 I E
gloath
(x,1
Σ
(1
(X: (x:31))
0mya (X:n;) + (X)
a!
0.9
Aliy/=LiEV,
|a|=1
Ilai F2
glazsh h (X:nī) (x),
Σ
Σ
Snj,ar
(X:
;
0.
0rp0
j-11/=11 |as1=1
where R (X) is Taylor remainder such that Rn,
exp( 30 () - 0 as n + xo, and
Sajjar.
(A Lig )a (An
a!
On the other hand, by applying the second-order Taylor expansion, we get that
g E 0
g
An,2(X) = Σ Σ
Suijana
(X:
0mpe (X:n)) + Rx),(X),
A.V 15 +aass52
in which (x) is a Taylor remainder such that R,, (X))(X))D, - 0 n oc. Decomposition of B,, Recall that we have
(x(
Σ
Σ
gG. (GG,(3. F(X: F(X:; ) (X)
exp(X%))
HiV,,=
Σ
exp( (0
Fx(X; (3.)
Fla;; go.
9a. (X)
h: H
Bn, (X)
B (X).
By invoking first-order and second-order Taylor expansions to (X) and 2x,2 it follows that
Bx, = Σ Σ
X; (ig)),, (X) + (X),
03
31 =1 11=1
@i E
Bx, Σ
Σ
Tagi
(X; 3i; 1i, (X) + Rm,
03
ARA 15
where we define
exp(
Ta
(A
SEV,
Additionally, R,,3(X) (X) and ( Taylor remainders such that R and Rn,3(X ///,, - 0 as n + co. Collect the above results together,
represent (x) as
N Σ
ala1 E (X;
Glaz
(x
Σ
anjaanaz 08
(X:nj),
j=l
05|
|02
@ld E
Tng.t
(X: )g. (X)
(x:
(X),
(33)
1=1 0511/52
19

---

Published as a conference paper at ICLR 2026
where we define Sn.j.Oxxa.D, nn.x Eiev, exp( exp(96,)
for any j [Np]-
Step 2 - Non-vanishing coefficients. Here, show that not all normalized coefficients
vanish. Specifically, we prove that at least one of the ratios Sn.jaannaz/DD Dr, or Tn.j.t/ /D, does not converge to 0 + DC. Assume by the contrary: for every j [Ny) and |oib [02]: |4] - 2,
Sp.j.01.22 +0
Tage - 0. D,
D
In particular,
Np
N₂
Σ
cxp(Phi)
exp(3%;))
= -
+ O.
(34)
D
Dn
LEV;
j=1
First, consider indices j for which the Voronoi cell is singleton, i.e., V3l = 1.
Fix u, v E [Nd]], take aj E Nd with aj (uv) = 1 and a2 = 0g. Then
1
exp (AAT) (wv)) 1 0.
Dm
Dn
SEV;
Summing over u, v and using equivalence of and la norms yields
Dn exp(2%) | +
(35)
Fix u E [g]. take 01 ON Nd and 0g No with 02 (w) 1. Then
Σ exp( (An [0] Sn.j.an.az
+ 0,
Dn iEV;
Dr
hence
1
exp( |An +
(36)
Dn REV;
Combining (35)m and (36 yields
Σ
Σ
exp( || ||nn |] + O.
(37)
D
J-Vy VEV,
Next, consider indices with multi-element cells, i.e., |y > 1.
Fix u, E [Nd]. let ai (uv) and a² = 0g. Then
exp( ( (A.%)) ( Sn
+ 0,
D
Dn
EV
hence
Σ exp( || 1|? + O.
(38)
D n iEV,
Fix a E (g). take an Od and 0x2 (u) Then
1
Σ exp( e (An 2w)p=
Sn.j.yy.2
- 0,
D,
Dnn
3EV
giving
Σ exp( || O.
(39)
D
iE
20

---

Published as a conference paper at ICLR 2026
Combining these gives
1
Σ
expl (Po) |1 || |A
||
- 0.
(40)
D
J:
3:12/>>1 ieV;
Finally, summing the terms in 34). and (40) yields
Dn/DD - O,
which implies 1 0, contradiction. Therefore, initial assumption was false: at least ratio among Sn.j.au /2./ Dr, and Tn.j.e/DD D' does not converge to 0 as n + co.
Step 3 - Application of Fatou's lemma. In this step, show that all normalized coefficients Su.j.an. 0z/ DD and n.j.e/DD 'D, must, in fact, converge to zero as n - co. This will contradict the conclusion of Step 2, completing the proof by contradiction. We denote m, as the maximum of the absolute values of those ratios. The result of Step implies that 1/m,, A 00.
have ||9G. gG. (w)))())) /D, - 0 - co, which indicates that ||9G. gG. (/) DD, + O. Applying Fatou's lemma, get
From the hypothesis in Equation 29
0 lim 19a.. 9G. [[()(m)
lim inf 9G.. (X) Ja. (X) n-000
du((X) 2 0.
mn Da
Mi Dn
This implies that Dr 19c, (X) ga. (X) + 0 as + oo for p-almost surely X. Looking
the formulation of Qn in equation 31 since the term
Σ Di=1 exp(X B X og)
exp(( (xp((())) TW X + ( is bounded, we deduce that the term surely X.
(x m. Da + for p-almost
Let us define the normalized limits
To.je m, D,
- iPj.e.
m, Dn
with note that least among them is non-zero. Then, from the decomposition of (.(() in Equation (33 we have
N₂
1+10,
I E
glozsh Orpar
Σ
Σ
(x;
(X:15),
05
j=1 [a1l||2//00
N₂ (M
a E
(x; (;BBi)) (x))) 0,
03
for p-almost surely X. It is worth noting that the term
8 F
(X;
gluztn (X;1)) can be D1
03
explicitly expressed as
When |a1| = 0, |22 0: exp((3t,) T W X))(X;n;):
When [ar]
= 1, |a2| 0: (WT X) (4) exp(( X)h(X;n1)
When la: 0, |22l 1: exp((3i,) ) W X) ah an
(X (xmn);;
When |a: 1, |02| 1: (WT X exp( W X
Hh
(X;nf);
When |o,| 2, |02| 0: (W X ( X (v) exp( (51,) X ( n?);
When |a1| 0, |02| 2: exp((3t,)
X
Bry (yy X (xn7))
8%
Recall that the expert function / satisfies the strong identifiability condition in Definition A. condition implies that the set of functions
This
x
(WT x) Om
(X, (x, : (Np), 0 4
21

---

Published as a conference paper at ICLR 2026
is linearly independent for almost every X. Therefore, obtain that Jj,a,,02 G,,e 0 for all j E [Npl, Σ |0.| + |ay] [ S 1 + (11, This contradicts the fact that at least one of these coefficients is non-zero. This completes the proof by contradiction for the local part, thereby establishing the inequality in Equation (28).
Global part. From the local inequality
28
infer that there exists constant g such that
l gG_ll 0. D(G, G.)
G (0) P(G.
It remains to show that the inequality also holds for measures far from G.:
1l9c 10a-9ch -a.. 0. Ge ()).G..)))) D(G,GA G.)
inf
(41)
Suppose, for the sake of contradiction, that (41) is false. Then there exists sequence of mixing measures {G,} C. GNy (9) with D(G G.) > for all n, such that
lla. - -9G 0, D G.)
lim
which in turn implies that I19G, 9G. || - + oc.
Since the parameter space θ compact, the sequence {Gr) } admits convergent subsequence with limit G' E Gxy(O), Moreover, because D(G,, G.) > e', it follows that the limit satisfies D(G", G.) 2 e'.
Applying Fatou's 's lemma, we obtain
{limin [lmmin inf 19G; (X) - 9G. (x) (x(
lim |9a. - -9c. 2
This implies that 96 (X) ga. (X) for ja-almost every X.
By identifiability property of the MoE model (to be established at the end of this proof), this functional equality implies that the measures themselves must be identical, i.e., G" = G.. This leads ( G.) = 0, which contradicts our earlier conclusion that D(G', G.) g 0. This contradiction establishes (41) and completes the proof of Equation (26).
Identifiable. We now establish the identifiability of the MoE regression function gg. Specifically, we show that if (X) = gG. (X) for almost every X, then the measures must be identical, i.e., G G..
For clarity, define
exp(u)
softmaxc(u)
Ev=1 exp X B X + c
='=1 TWT X + Boj')
N
exp(1t
softmaxG, (u")
N ΣN Σv= exp X B$ X c)) Σ3= exp((3i,) TW X 163
with
" (X B X + C0 (B1x) T W X + for : E [N],J' E (NN]),,
"² (X B + Ci (giz) T W x + 5h :i' E [N],j' e [Np)).
Since 96 (X) = gc. (X) for almost every X, we have
N
softmax
X B:X + 2) h/ x +
softmaxd (B1g) X + Bos) 1((x nz)
N
N
softmaxg. (X B,X (XT + 4)
1(X n +
softmaxc. ((8i)) 065) 1(X,
(42)
22

---

Published as a conference paper at ICLR 2026
As the expert function h satisfies the conditions in Definition A.1 the set {h(X,, 1/.) : E [K]], where are distinct vectors for some k' E 1, is linearly independent. Nf + Np. then there exists
The nh....,
some i E [AV]] such that 7) # nř for any j E [Npp]. This implies that Σ softmaxG (11) (os) 1 X mb) O. which is a contradiction. Thus, must have that N, N₂. As result,
ssoftmax (31) x + Boy) j E [N softmaxG. (191)) X + (hn : E [Npl]
for almost every X. Without loss of generality,
softmaxc((91)) X Soj) softmaxc. ((91)) X + 18g):
(43)
for each j [Np]]- Since the softmax function is invariant to translation, this forces B1j = and Boj = 8o; + vo for some up E Given the normalization SoN; SON, = 0, we conclude
3oz = Bög.
Substituting back, the equation becomes:
N (xp(3a))exp(((3 X)h(X,n) m1) Σ
exp(56,) exp((t))))p(((tt) x))(X,n))
"x)(( nj)..
for almost every X.
Partition the index set [Ppp] into groups Pi...., Pm (with m S Np) such that within each group the weights exp(3o:) agree, and across groups they differ. Then the above equality reduces groupwise to
Σ iEP
exp((oi) exp((31.) T x)h(X (1((X,m) m)
Σ sEP;
exp( exp(((1.) X)h(X, nn))
for almost every X. Since B1s = 3is and Bo: = Büx this equality implies that (m i E P3} = tn% i P5} for all j.
Therefore,
m2 T=
exp(8o))0(11..)
=
-E 3= f=
Σ
= G.,
j=1 iEP
which proves the identifiable property.
We begin the proof by introducing some key notations. Let Ryy (9) be the class of regression functions corresponding to the mixing measures in GN; (0):
Proof of Equation (27).
Rxy (0) = {gG(< (x) G E GN; (0)).
For any 8 >> 0, we define the (2) neighborhood around the true function gG. (X| intersected with R.y (9) as
R,y, (6, {g Rxy(() 11g 9a. (Ip()) I 8).
To bound the complexity of this function class, van de Geer 2000 introduced the functional
(8 (9:
H1 (t. Ryy t). Il- ll<at)) dt 8, (44)
8
where HB(.) denotes the bracketing entropy of R,y (θ, t) with respect to the (p) norm, and V b means max(a, b). By adapting the proof techniques of Theorems 7.4 and 9.2 from van de Geer (2000), we can establish the following lemma:
Lemma A.4. Suppose (r))) 2 JB RNy (0, 6)) and that (6) is non-increasing in d. Then there exists a universal constant e and a sequence (5n) satisfying / 8% 2 c (6,,). such that
TP P(Il, - 9G. 9G. (cw > =6) S cexp ng
for all 8 ' 8n.
23

---

Published as a conference paper at ICLR 2026
Next, we show that if the expert functions are Lipschitz continuous, then Hn(e, RN; (8), l| (())(w) X log(1/s);
(45)
for any < & S 1/2. To see this, note that for any 9G E Rx, (0) the boundedness of ensures (xX 1) s M for all X, for some constant M.
Let and consider (-cover ( of RN; (0) (o) under the L norm, where N Rx; Il [| Construct brackets [X), (x) for each i E [N] by
( == max n. X + M.M By construction, Rxy(0) C U N [L, U3] and the width satisfies U; L, min M). Thus,
(x) == max nT. (X) 5,0},
I Lell² f dy(() s 4ç²;
so that ||U - Lill $ 2Ç. Hence, by the definition of bracketing entropy,
H H2(2 RN; (9), -1) No (S, Rx; (0), 1allIlllaa -11=)).
(46)
Therefore, we need provide an upper bound for the covering number N. In particular, we denote A := {(31, Po) E R R. Nd x W.
R (31,80, }} and Ω == {17 E RR" (31. (o E }}. Since compact set, A and are also compact. Therefore, we can find (-covers D and Ωg for A and S2, respectively. We can check that
|A < Op(r (NA+I)N))))
[Agl S Op(-
For each mixing measure G Σ i (( GNy (e), we consider other two mixing measures:
N -=
N
cxp(Bu))0(A..)))
G== exp 3 0i
Σ
16
(7,57)))
Here, 17, Pr such that ni is the closest 17 in that set, while (B Box) A is the closest to (B1r, Roi) in that set. From the above formulations, we get that
exp(3 x + Bog) |1(X, m1) h (X, 77,1
||9c gallen = sup XEX
exp X X + c)) + Σ N exp (3T X + Boy)
N₁
exp j X + Bog) 1/ m1: h X 77,)
sup
Xex
N Σx= exp( X B x + d.) + Σy (xp((111 x + Boys)
1=1
Ny
sup
Ih(X, n3) - h h(X,7,))
XEX
Σ sup (X) ||, - 7 S N
1= XEX
Here, the second inequality occurs as the softmax weight is bounded by one, and the third inequality follows from the fact that the expert h(X,-) Lipschitz function with some Lipschitz constant L, (X) > O. Next, let us denote
N D:
Σ
exp X
0 X +
++)++ +
exp X + Boo )
j'ml
N
N
D:
exp(X X 08 x + (
B
+
exp(Bi,.X X + Boy)
3
24

---

Published as a conference paper at ICLR 2026
Then, we have
N
N exp(31 x + (b))h({ ())))((()))))) ,1)))
ll9c
20llax
sup XEX
exp X BTx x + (,))(X,,)) n
j=1
N
N
exp X T B x + (x ワリ + Σ
exp (5 15 X + (x)))
m)))
D
N
$
Σ
sup X EX
exp(X
B X + c.)h m7!)
D
D
im
N
exp
(5T + Bby)
exp(3) (3,, X + Foj))
sup XEX
|h(X .731-
(47)
D
D
Now, we will bound two terms in the above right hand side. Firstly, since both the input space X' and the parameter space O are bounded, we have that
N₂
1 D D -DD-DD
DI
Σ
exp(8Tg. X + Boy) - -ep((xxp) X + +Bor)
N2
(015 BBi)T x + (B0s - -Pos)
(BB1 B15)T T x + 1305 - -Bosl
No
$ Σ [118|] - B15ll - + BBo - Fool
j-1
$ N₁ + 1)5 S.
As result, we deduce that
- Σ
sup XEX
exp(X B x + <))((, n10) SSc.
(48)
Regarding the second term, note that
exp(3T,xx + Bay) exp(3), T
xsp(3,xx. X + Boj)
D
D
= exp($TT + Buj)
( - ) - + D
exp($TT,x + Boj) - -cxp(3,xxx x + {oo1)
D
Since both the input space and the parameter space are bounded, we have
exp(2T,x x + Bun) -
1 D exp((3]xx x + (oo)) exp(Bi, X + Boj) SB SC.
which yields that
N
exp(2Ts X + Boj)
exp(2 T x + Boj)
Σ sup J=1
-|h(X, 77)1 7,,
X
D
D
Ny sup
5 Σ
|x(X, SC (49)
-=1 Xex
25

---

Published as a conference paper at ICLR 2026
From Equations 47.. inequality, we have
(88) and
49
obtain that l19. 96ll 2 Ç. According to the triangle
l
97 -
lg golle= lla6 -clle= S.
By definition the covering number, we deduce that
(C, (N, [1_))
2 S On (( x ((n & O(n -(N(((N+)+))
(50)
Combined with Equations (46) and 500. we achieve the following result:
HB (2, Rv, ((<а()) x log(1/T).
Let Ç = 8/2, then we obtain that
Hy (xy (12)))) X log(1/s).
As result, it follows that
Jn(o, No, RN,((,0)) = l
11/2 (1, Rxy (0,0), (Ie()))) dt V / 103/21a log(1/t)dt V 82/2218
(51)
Let (() 8 [log(1/0)) (1/)) then (6)) is non-increasing function of 8. Furthermore, Equa- tion (5T) indicates that (0) 2 J6 R,; (0,()))), addition, let 8, Vlo((n)nn, then we get that vno 2 (V,)), ) for some universal constant c. Finally, by applying Lemma A.4 we achieve the desired conclusion of the theorem.
B TRAINING ALGORITHM OF SMOPE
Algorithm Training Process for Task Dz
Input: Pre-trained ViT fa. dataset D., number of epochs E, hyperparameters e, Clpouters Pprooo and accumulated expert activation frequencies {F.r
Output: Updated prompt parameters P and classifier parameters Ø
1: ift 1 then
2: Store the prompt parameters from the previous task: Pold +-
3: if then
4:
for epoch = 1, F/2 do
5: for (#), v() D. do
6:
Compute output v = go (fo =
7:
Optimize P and ø with Lee
Dense training phase
8:
for epoch = L. do
9:
for (xt;, D, do Compute output ý Do (a): P. P,K))
11:
Optimize P and & using Equation (16)
> Sparse training phase
12: for E y do
13: Estimate Gaussian distribution Ge using Equation 53
14: for epoch 1,..., E do
Optimize with the Lup from Equation 16: for (æf, vf) E D, do
D Task-adaptive prediction
17: Compute z = (a K)
18: Update accumulated frequencies of expert usage {Fy}
return (P, Φ)
For the 1-th MSA block, let PK and PV denote the prefix parameters where prompts are inserted. For clarity, the main text omits the layer index (I) in the notation. The complete set of prompt
26

---

Published as a conference paper at ICLR 2026
parameters is P ((P)) PK)),,, where L is the number of MSA blocks in the ViT. Given an input æ; its prompted representation = fa P, K), where 0 denotes the frozen pre-trained ViT weights and K is the number of selected prompt experts. Setting K = -1 activates a dense prompting mode that bypasses sparse expert selection. Using an MLP classifier Jø, the prediction is given by:
== (z) Io(So P.K)).
(52)
During training, only the prompt parameters P and the classifier parameters 6 are updated.
Following prior work on SMoE training (Wu et al.
2022 Lin et al. 2024; Cai et al. 2025), we first
adopt dense training (K' = -) for several initial epochs on the first task. This provides a good initialization of the prompt parameters before enabling sparse expert selection.
For each task t, the model is optimized on D, using the objective in Equation (6).. We then refine the classifier parameters 6 using the task-adaptive prediction (TAP) objective. following (Wang et al., 2023 Jiao et al., 2024 Le et al. 2024a), which mitigates classifier bias by modeling the Gaussian distributions of previously seen classes. Specifically, for each class с E ) from tasks
i = 1,...,- t 1, we maintain Gaussian distribution Ge - (Mep Σg), approximated from the
prompted representations D = {z fo(æ; (,,; K) (xt, c) E Dij. The distribution parameters are estimated
Me
c
Σ
Σ E.EDS
(Ze [e) Me)
(53)
D
IDo
EcED:
The classifier Io is further optimized with the loss:
explao(=)[c)
Lup = 3E llog
(54)
Ej= exp(go(z)(c)))
1=1 cE zE2,
where Zi is constructed by sampling an equal number of pseudo-represennttions from Go for each class c E yr. The full procedure is summarized in Algorithm |
C ADDITIONAL EXPERIMENTAN DETAILS
Evaluation Metrics. We report two standard metrics, Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA), as they inherently capture both plasticity and forgetting (Smith al.) 2023). FAA measures the overall performance by computing the average accuracy across all T tasks after the completion of continual learning. CAA extends this by averaging the performance across all intermediate stages. Formally, let Si, denote the accuracy on the i-th task after learning the t-th task, and define the average accuracy after / tasks A, Ei= S.,. Then, after all T tasks learned, FAA and CAA are given by FAA- = Ar, and CAA = E A.-
Data Augmentation. Input images are resized to 224 224 and augmented following the protocol of Smith et al. 2023), including random horizontal flipping and standard normalization.
Implementation Details. We use pre-trained ViT-B/16 model as the backbone. Training is conducted using the AdamW optimizer with hyperparameters 31 = 0.9 and 32 = 0.999. Following the setup in VQ-Prompt (Jiao et al. 2024), 20% of the training data reserved for validation, and grid search is used to tune hyperparameters. We evaluate L2P++. Deep L2P++, DualPrompt, CODA-Prompt, and VQ-Prompt using the official implementation provided by VQ-Prompt. For OVOR, we similarly employ the original codebase. Recent work by Feng et al. (2025) has identified an implementation issue the official ProDepPrompt codebase, specifically in its prompt retrieval mechanism. This issue introduces information leakage, resulting in overestimated performance, particularly when inference is performed with a batch size of 1, while the training batch size remains consistent with the original setup. To address this, we adopt the corrected prompt retrieval code provided by Feng et al. (2025) for both HiDe-Prompt and NoRGa, while preserving all other components from the original implementations.
Hyperparameters, For SMoPE, we fix the prompt length to N, = 25 and select K = 5 prompt experts in all experiments. Prompts inserted into the first six MSA blocks (see Table 5). The
27

---

Published as a conference paper at ICLR 2026
Table 5: Detailed prompt configurations used for our main experiments. "Location" indicates the specific MSA block where the prompt is inserted. Here, N denotes the number of prompts, and Np denotes the prompt length.
Method
Location
Dataset
Hyperparameters
1,2++
[]]
All
N = 30, N, 20
Deep L2P++
[1 [12455
All
N N0, 20
[1 2]
All
G: N-N= = N, =
DualPrompt [3 [345]
E: = 10, N, 20
CODA- Prompt
[1 53
All
N 100, N, =
[1 3: [4 5]
G: N,
OVOR
E: N,
ImageNet-R
N 10, N, = 40
HiDe-Prompt
[I (1244)5
CIFAR-100
N = 10, N = 10
CUB-200
N = 10, N₂ 40
ImageNet-R
N 10, Nr 40
NoRGa
[123455
CIFAR-100 100
N 10, N 10
CUB-200
N 10, N
40
VQ-Prompt
All
10, N
SMoPE
[] 6
All
N N, 25
Table 6: Performance comparison on ImageNet-R with varying task numbers. 1 indicates higher is better. FAA and CAA are averaged over 5 runs. Bold denotes the best results, excluding joint training.
Method
S-tack
10-rask
Morask
50-aakk
FAA
CAA (*)
FAA
CAA (1)
FAA
CAA (1)
FAA
CAA (f)
Joint: Train
82.06
82.06
$2.06
82.06
L2P++
70.83 1: 0.38
78.34 1: 0.47
69.29 = 0.73
78.30 1: 0.69
65.89 :: 1.30
77.15 1: 0.65
55.65
0.89
64.88 0.88
Deep
73.93
0.37
80.14
0.54
71.66
0.64
79.63
0.90
68.42
78.68
1.03
0.54
0.80
DualPronpp
73.05 1: 0.50
79.47
1: 0.40
71.32
0.62
78.94
1: 0.72
67.89
1.39
77.42
0.80
1.18 58.31
65.51 1.95
CODA-Prompt
76.51 0.38
82.04
75.45
0.56
81.50
0.82
1.06
66.96
75.19+00.
OVOR
75.81 = 0.16
79.31
0.11
75.25
0.21
79.78
- 0.65
72.45
0.42
77.60
1.15
65.81
0.59
74.81 0.69
HiDe-Prompp
75.00 + 0.05
79.68 + 0.43
74.25 = 0.19
79.64
+ 0.53
73.71
= 0.37
79.
0.72
70.82 0.54
76.81 0.83
NeRGa
71.39
71.122
VQ-Prompt
79.23 - 0.29
82.
82.96-2
4: 0.50
78.71
0.22
83.21
= 0.68
78.10
0.22
82.70
1.16
76.31
0.61
81.52 0.81
SMoPE
80.09 1 0.42
84.47
0.95
79.32
0.42
84.39
0.77
77.81
0.36
83.38
0.98
75.54
0.49
81.94
0.95
hyperparameter is tuned over the set {0.0, 0.1, 0.9,
1.03.
The weighting coefficients drooner
1e 13-3) For all baseline methods, we use the configurations and hyperparameters reported in their respective original papers to ensure a fair comparison.
and O'prata tuned over the set (5e - le- 5e 5 le_ 5e
D ADDITIONAL EXPERIMENTAL RESULTS
D.I PERFORMANCE ACROSS VARYING TASK LENGTHS
To assess the scalability and generalizability of SMoPE, we follow prior work (Huang al. 2024 Jiao et al. 2024) by partitioning ImageNet-R into 5, 10, and 20 sequential tasks, ensuring fair comparison with existing methods. Additionally, we introduce a 50-task split to more rigorously evaluate SMoPE's capacity to handle longer task sequences. The results shown in Table 6 Across all configurations, SMoPE consistently outperforms specifce methods such as HiDe-Prompt and NoRGa, despite using single shared prompt. Its performance remains comparable VQ-Prompt, further validating its robustness under varying task lengths. These findings suggest that SMoPE can generalize effectively as the number tasks increases. However, relying on single prompt for an indefinite number of tasks may eventually lead to knowledge interference as the task sequence grows. To address this limitation, future work could explore strategies for scaling the number of prompt experts to better support longer task sequences.
D.2 PERFORMANCE ACROSS VARYING PROMPT LENGTHS AND NUMBER OF PROMPT EXPERTS
We investigated the impact of prompt length N, and the number of selected experts on model performance, with results presented in Figure Across wide range of configurations, our method
28

---

Published as a conference paper at ICLR 2026
K=1
K=3
k=5
K=7
HiDe-Prompt
89.0
88.5
88.0
87.5
86.5
86.0
85.5
84.0
20
35
Prompt Length N,
Figure 3: Impact of Prompt Length N, and Number of Selected Experts K on Performance. Performance across different combinations of N, and K values on CUB-200 with 10-task split.
20
10
Prompt Score
0-
-10
-20
-30
Head
Head
Head
Head
Head
Head
Head
Figure Distribution Prompt Expert Scores. Box plot illustrating distribution prompt expert scores 8y across attention heads in the first MSA block on the CUB-200 dataset.
consistently surpasses task-specific prompting approaches, particularly HiDe-Prompi demonstrating the robustness SMoPE. Performance tends to degrade when N₂ is small, as indicated by lower FAA scores across various values of K. Increasing N, generally improves performance up to certain threshold. However, excessively large values of N, lead to a decline performance, likely due to the increased difficulty of effectively training all Np prompt experts. A larger number of experts also means that each receives less training data, which hinders learning and complicates the selection of relevant experts, ultimately reducing FAA scores. Regarding the number of selected experts, we find that setting K = 5 yields strong and stable performance across different prompt lengths. Based on these observations, we use Np 25 and K in all our experiments.
D.3 DETAILED ANALYSIS ADAPTIVE NOISE
Adaptive Noise Formulation. Our adaptive noise formulation is derived directly from min-max the selected expert set can be expressed as:
normalization. From Equation
Kx arg max
Σ
max, (X) min, (X) arg max
5 Cj'
Σ
(X) Eg' ming (X)
max; (x min, 85
JES
S
J'ES
3 (X) min, x (X)
'
arg mar
max; 8, (X) - min, 3;(X)
max, 3x(X) - ming 8, (x)))
FES
29

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison on ImageNet-R CUB 200 using 10-task split with different noise formulations. indicates that higher values are better. Bold highlights the best results.
ImageNet-R
CUB-200
Method
FAA (1)
CAA (1)
FAA (1)
CAA (1)
Fixed Noise
79.03 1 0.31
84,42 1: 0.68
86.86 1 0.47
90.37 1: 0.60
Uniform Noise
79.04 + 0.63
84.29 + 0.70
87.11 4.45
90.40 + 0.65
Adaptive Noise
79.32 + 0.42
84.39 + 0.77
87.43 + 0.39
91.11 + 0.55
80.0
79.5
79.32
79.21
78.98
79.0
78.62
78.5
78.27
78.43
78.38
FAA
78.0
77.5
76.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
E
Figure 5: Impact e on Performance. Performance across different e values on the ImageNet-R dataset using a 10-task split.
Thus, our adaptive noise can be interpreted performing min max normalization of the expert scores, followed by subtracting scaled from the scores of important experts. As result, the noise is adaptively scaled to the dynamic range of the scores, eliminating the need for manual tuning of noise magnitudes across different attention heads.
Previous work has typically encouraged expert exploration by injecting random noise into expert scores, either with a fixed magnitude (Nguyen et al.) 2024) or by sampling from predefined distri- butions (e.g., Uniform) (Shazeer et al. 2017 Fedus et al. 2022). However, in pre-trained models, the range of expert scores can vary considerably across attention heads (see Figure 4 making difficult to select appropriate fixed noise level. In contrast, single noise parameter, e, which is constrained to the interval [0, 1]. This approach eliminates the need for head-specific noise scaling and significantly reduces the complexity of hyperparameter tuning in large-scale models.
method requires tuning only
To assess the effectiveness of our adaptive noise approach, we compare it with the following baseline variants:
Fixed Noise: constant noise value subtracted from the scores of frequently activated prompt experts across all attention heads.
Uniform Noise: Noise sampled from uniform distribution U((-, e) and added to all
prompt expert scores across all attention heads.
The experimental results, summarized in Table 7 demonstrate that our adaptive noise formulation surpasses these baselines, highlighting its effectiveness. While noise formulation is specifically tailored for continual learning to help preserve knowledge in important experts, it may also be applicable to other settings, which we leave for future exploration.
Impact of on Performance. To evaluate the impact of e on model performance, we report FAA results on ImageNet-R across range of values, shown in Figure When 0.0, performance
30

---

Published as a conference paper at ICLR 2026
Layer 1 Head
Layer 2 Head
Layer 5 Head
02
8.0
10
Figure 6: Impact of c on Activation Frequencies. Results on CUB-200 with prompt length of N₂ 25 and K" = 5. We present the results for several representative attention heads and visualize the frequency with which prompt experts activated after training on all tasks for different values
is suboptimal due to imbalanced expert utilization. As increases, performance improves, suggesting enhanced exploration and more balanced expert engagement. However, excessively large values of can reduce expert specialization, as broader set of experts is activated more frequently, This also hinders the reuse of frequently activated trained experts, limiting knowledge transfer. Overall, moderate value achieves favorable trade-off between exploration and stability, resulting in optimal performance.
Impact of on Activation Frequencies of Prompt Experts. To investigate the influence of the regularization parameter on the activation frequencies of prompt experts, we select several representative attention heads and visualize their activation patterns across varying values of c using the CUB-200 dataset. The results are shown in Figure
At c 0.0, observe that certain attention heads activate only limited subset of prompt experts. However, as the input distribution to these attention heads shifts with the introduction of new tasks, different sets of experts may be activated. This variation is primarily driven the relevance of each expert to the input, as determined by the score function. Interestingly, even in the absence of adaptive noise, some attention heads (e.g., Layer 2, Head 2, depicted in the figure) demonstrate balanced expert utilization, with no significant imbalances in activation frequency. At 1.0, the regularization penalty reaches its maximum, substantially reducing the scores of frequently activated experts while encouraging the activation of underutilized ones. As shown in the figure, this penalty leads to more balanced distribution expert utilization across all attention heads. We find that 0.4 is the optimal value and yields the best performance. At this setting, the distribution of expert activation is relatively balanced, though some attention heads still exhibit imbalanced expert utilization. This highlights key trade-off in continual learning: activating too many experts can dilute the specialization of each expert, as they are exposed to less data during training. Furthermore, excessive number of activated experts complicates the task of selecting the most relevant experts for a given input. This challenge mirrors issues observed in methods using task-specific prompts, where increasing number of tasks makes it progressively harder identify the correct task identity and its corresponding prompt.
D.4 PERFORMANCE ACROSS VARYING Loss WEIGHTS
To enhance the selection process of prompt experts, we introduce two loss functions: Crouter and Eproto These are designed to encourage expert specialization while preserving previously acquired knowledge. We investigate the impact their corresponding loss weights, Qvouter and a as defined in Equation (16). The results, shown in Figure demonstrate that the model achieves optimal
31

---

Published as a conference paper at ICLR 2026
86,27
87.08
87.05
86.96
86.98
86.85
86.73
86.66 .66
86.
87.20
87.22
87.12
87.06
86.85
-8.2
1c-1
grrm
Se-3
76
86.80
1e-4
87.29
86.8
Se-4
78
87.09
87.21
87.05
87.01
86.83
1e3
86.71
86.78
87.11
87.12
87.07
87.13
86.85
86.6
Se**
1e
Se-*
1e:
Se
1e:
Figure 7: Impact of C'router and Oppotoo Performance. Performance across different combinations of Chroutrr and Opeoto values on CUB-200 dataset using 10-task split.
1.5
Input
Prefix Key
1.0
0.5
PCA 2
0.0
0.5
1.0
-1.5
1.5
1.0
0.5
0.0
0.5
1.0
1.5
2.0
PCA 1
Figure 8: Visualization of Prefix Keys. PCA is applied to a representative attention head to visualize the distribution of the corresponding input embeddings and prefix key vectors.
performance when these weights are sufficiently large, suggesting that both components contribute positively to overall performance. Furthermore, performance remains strong across a relatively wide range of values, highlighting the robustness and effectiveness of incorporating f.vater and f into the training objective.
D.5 DETAILED ANALYSIS OF PROTOTYPE LOSS
As discussed in Section 3.4 the activation of a prompt expert fN+5' is determined by its score function šy,, which in turn depends on the associated prefix key p Thus, the set of prefix keys implicitly defines the regions of the input space where each expert most responsive. Building this insight, we propose leveraging prefix keys from previous tasks as prototypes, serving as implicit memory representations of past input distributions. To support this intuition, select representative
32

---

Published as a conference paper at ICLR 2026
Turiig Aogropaiion
The
Thdk
Tod
2.5
g
0.5
0.0
Pack
bark
Tapc
Tock
150
1.00
0.5
Figure 9: Sample Efficiency Evaluation. Validation loss on ImageNet-R (10-task split) measured throughout the training process each task.
attention head and visualize its corresponding imput embeddings alongside the associated prefix keys using Principal Component Analysis (PCA), as shown in Figure The visualization reveals that prefix keys occupy a region of the embedding space densely populated by input samples. This proximity suggests that prefix keys encode salient features of the input distribution, thereby functioning as effective prototypes.
This perspective aligns with findings in recent literature. For example, Chen et al. (2022) showed that routers can learn cluster-center features, facilitating the decomposition complex tasks into simpler sub-problems addressable by individual experts. Furthermore, the expert selection process can be interpreted as a classification problem, where the prefix keys act as a linear classification head. In this context, recent studies have proposed that the weight vector associated with each class in classifier can be interpreted as a class prototype (Snell et al. 2017 Qi et al. 2018 Zhao et al. 2024 further supporting our hypothesis.
D.6 SAMPLE EFFICIENCY EVALUATION
As demonstrated in Appendix A applying the prompt-attention score aggregation technique yields MoE models within attention heads that maintain the same estimation rate and sample efficiency as those achieved by the standard prefix tuning formulation. To empirically support this theoretical result, we follow Le al. (2024a) and evaluate sample efficiency by tracking the validation loss on ImageNet-R throughout the training process for each new task. The results, presented in Figure show that both models exhibit comparable convergence rates. This suggests that incorporating prompt-attention score aggregation does not compromise the model's 's ability to efficiently adapt to new tasks.
D.7 ANALYSIS OF PROMPT EXPERT ACTIVATION FREQUENCIES
To provide direct evidence for our central claim that sparse activation mitigates interference, this section analyzes how expert activation patterns evolve the model encounters a sequence of tasks. As illustrated in Figure I0, which visualizes activation frequencies on the CUB-200 dataset, the model initially activates a sparse and concentrated subset of prompt experts. As training progresses across subsequent tasks, the set of activated experts becomes increasingly larger and more diverse.
This dynamic and adaptive activation pattern is fundamental how our method mitigates interfer- ence. In contrast to dense, single-promp prompt baselines (e.g., OVOR), where a single set of parameters must be continually updated for each new task-riskin knowledge- sparse approach distributes the learning load across broader pool of specialized experts. New tasks can activate and train new or underutilized experts, leaving the parameters of experts essential for earlier tasks largely unaffected. This selective activation is crucial for preserving acquired knowledge while maintaining plasticity for new learning.
the overwriting of previously acquired
Furthermore, this diversification does not preclude effective knowledge reuse. The activation mecha- nism in the SMoPE framework is input-driven rather than task-ID-driven as discussed in Section 3.2 Consequently, an expert can be reactivated for subsequent tasks if its specialized function is relevant to the current input. This mechanism facilitates positive knowledge transfer while preventing the destructive interference.
33

---

Published as a conference paper at ICLR 2026
Task
10
11
12
13
14
15
16
17
16
20
21
23
24
25
Freq
-0.00
Expert
Figure 10: Evolution of Prompt Expert Activation Frequencies Across Sequential Tasks. Results We present results for representative attention
CUB-200 with prompt length N2 and head and visualize the frequency with which prompt experts are activated across sequential tasks.
95
Annurccy
85
80
SMOFT
OVOR
VO-hommp
Co0A-hompi
HDe Promot
DuaPronppt
NoRGA
L.P+
Figure 11: Accuracy Curves Across Incremental Steps. Performance different methods on CUB-200 under the 10-task incremental learning setting.
D.8 ACCURACY CURVES ACROSS INCREMENTAL. STEPS
To provide more comprehensive understanding of the method's behavior, we present accuracy curves illustrating how FAA performance evolves across sequential tasks CUB-200 under the 10-task setting. The results are shown in Figure [T] As observed, OVOR performance deteriorates rapidly training progresses. This decline can be attributed knowledge interference arising from updating all prompt parameters simultaneously. In contrast, SMoPE maintains consistently strong performance across all tasks. Notably, its accuracy surpasses that of methods employing task-specific prompts, such as HiDe-Prompp Prompt. These results demonstrate the effectiveness and robustness of the SMoPE architecture, even when using shared prompt parameters.
E
USE OF LARGE LANGUAGE MODELS
Large language models employed solely for editorial purposes, including grammar correction and spelling refinement. They were not used for content generation, data analysis, or the design of experiments.
34


Reviews:
[{"review_id": "bNmEFBbZ01", "reviewer": "Reviewer_Uup3", "summary": "This paper proposes SMoPE, a method that integrates a sparse Mixture of Experts (MoE) architecture into prefix tuning for prompt-based continual learning. The core idea structuring a single shared prompt as multiple “prompt experts” and activating only a sparse subset per input is intuitively appealing and aims to reconcile the trade-off between parameter efficiency and performance degradation due to interference. While the paper reports strong empirical results across several benchmarks, I have some concerns regarding the novelty, methodological rigor, and experimental fairness that limit the contribution’s significance.", "questions": "Pls see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  SMoPE achieves SOTA results on multiple continual learning benchmarks while using only a single shared prompt.\n2. By activating only K out of Np prompt experts per input, SMoPE reduces both trainable parameters and FLOPs by up to 50% compared to task-specific baselines.\n3. The combination of sparse expert routing, adaptive noise, and prototype-based loss successfully balances expert utilization and preserves past knowledge, reducing catastrophic forgetting.", "weaknesses": "1. The connection between prefix tuning and MoE is leveraged as the main motivation. However, SMoPE’s key components, prompt-attention score aggregation, adaptive noise for load balancing, and a prototype loss using prefix keysbare largely adaptations of existing techniques (e.g., sparse routing in MoE, load-balancing noise, and prototype-based memory). The score aggregation mechanism simplifies computation but lacks a compelling theoretical justification beyond sample complexity equivalence (which is shown only under strong identifiability assumptions). The method feels more like an engineering combination than a conceptual advance.\n2. The paper claims superiority over task-specific prompt methods like HiDe-Prompt and NoRGa. However, it explicitly adopts a corrected implementation from Feng et al. (2025). While this correction may be valid, its use without independent verification or ablation (e.g., showing performance with and without the fix) undermines the fairness of the comparison. If the corrected baselines underperform due to implementation conservatism, SMoPE’s gains may be overstated.\n3. SMoPE relies on a fixed-size shared prompt (Np = 25) regardless of the number of tasks (up to 20 in experiments). While results are strong in this regime, the method offers no mechanism to expand capacity as the task sequence grows indefinitely, a fundamental requirement for practical continual learning.\n4. The assumption of \"strong identifiability\" in the supplementary materials is overly idealized and difficult to strictly meet in practical neural network experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryyuNHrpD8", "reviewer": "Reviewer_QtZY", "summary": "This paper introduces SMoPE, a framework for continual learning with prompt that employs a sparse mixture of prompt experts architecture within the prefix-tuning paradigm. By structuring a single shared prompt as multiple prompt experts and using sparse MoE routing, the method aims to mitigate catastrophic forgetting and knowledge interference. Experiments on continual learning benchmarks, including ImageNet-R, CIFAR-100, and CUB-200, demonstrate that SMoPE achieves superior performance to existing methods.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors carefully derive how prefix tuning maps onto a multi-gate MoE, resulting in the novel idea of sparse activation of prompt experts.\n2. The empirical results show that SMoPE can outperform both single-prompt and task-specific prompt baselines. The ablation study offers in-depth insight into which components matter, with clear quantitative impacts.", "weaknesses": "1. The literature review, though broad, primarily references earlier or closely related prompt-learning studies but omits several recent and highly relevant advances [1][2]. Incorporating these works into the related work section and including them in the comparative evaluation would strengthen the contextual foundation of the paper.\n2. While the proposed method is valuable, individual contributions such as the adaptive noise mechanism, prototype-based loss[3], and prompt score aggregation[2] extend or increment existing ideas. The presented extensions are incremental in nature, and although their integration is beneficial, the distinctiveness of each mechanism could be more forcefully argued.\n\n3. The results in Table 6 indicate that SMoPE underperforms compared to existing methods as the number of tasks increases. The authors also acknowledge that interference may persist when task numbers grow. Providing additional analyses, such as accuracy curves across incremental steps, would offer a more comprehensive understanding of the method’s behavior and limitations.\n\n[1] Roy A, Moulick R, Verma V K, et al. Convolutional prompting meets language models for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 23616-23626.\n\n[2]Gao Z, Cen J, Chang X. Consistent prompting for rehearsal-free continual learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28463-28473.\n\n[3]Zhu F, Zhang X Y, Wang C, et al. Prototype augmentation and self-supervision for incremental learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 5871-5880.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNuacl6spS", "reviewer": "Reviewer_3X9D", "summary": "This paper tackles the trade-off between efficiency and performance in prompt-based continual learning. Current methods either use task-specific prompts, which suffer from scaling issues and inference overhead, or a single shared prompt, which leads to performance degradation due to interference. The authors propose SMOPE, a method that structures a single shared prompt as a sparse Mixture of Experts. It uses a novel prompt-attention score aggregation mechanism to efficiently select a small subset of \"prompt experts\" for each input, mitigating interference. Additionally, an adaptive noise mechanism is introduced to balance expert utilization while preserving prior knowledge, and a prototype-based loss leverages prefix keys as implicit memory to enhance specialization. The work effectively combines ideas from MoE and prefix tuning into the single-prompt CL setting, showing strong results.", "questions": "-   Based on the ablation study in Table 4, the prototype loss seems to provide the final increment of performance. How critical is this component for preventing forgetting of past tasks, as opposed to simply improving expert routing for the current task? Could you provide analysis comparing forgetting metrics with and without this loss?\n-   Could you provide more details on the selection process for the number of experts K and the loss weighting hyperparameters? How sensitive is the method's performance to variations in K, alpha_router, and alpha_proto, especially concerning their relative balance?\n-   Would it be possible to visualize expert activation frequencies or provide statistics showing how expert usage changes across the sequence of tasks? This could offer more direct evidence for the claims about reduced interference and knowledge reuse.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-   The paper addresses a clear and important problem: reconciling the efficiency benefits of a single shared prompt with the performance advantages of task-specific prompts in continual learning.\n-   The core idea of interpreting the single prefix prompt pool as a Mixture of Experts and applying sparse activation is novel and interesting, building cleverly upon recent interpretations linking prefix tuning and MoE structures.\n-   The experiments are thorough and the presentation is easy to follow, demonstrating strong performance against relevant baselines across multiple datasets and pre-training paradigms.", "weaknesses": "-   The prototype loss using prefix keys is an intriguing idea, but its relative contribution compared to the benefits gained from sparse activation and adaptive noise seems modest according to the ablation study. It's unclear how crucial this component is for mitigating forgetting versus simply enabling better expert specialization.\n-   The method introduces several components and associated hyperparameters, including the number of selected experts K, the noise level epsilon, and the loss weights alpha_router and alpha_proto. While some sensitivity analyses are provided, the interplay between these hyperparameters and the overall robustness to their selection could be explored further.\n-   The central claim is that sparse activation mitigates interference. While plausible and supported by the overall performance gains, the paper lacks direct analysis or visualization showing how expert activation patterns evolve across tasks and whether overlap is indeed reduced compared to a dense single-prompt baseline like OVOR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTnBJVJftU", "reviewer": "Reviewer_L3Zd", "summary": "This paper proposes SMoPE, a framework for continual learning that combines the efficiency of shared prompts with the adaptability of task-specific prompts.SMoPE structures a single shared prompt into multiple “prompt experts” within a sparse mixture-of-experts (MoE) architecture, dynamically activating only the most relevant subset to reduce knowledge interference. It introduces a prompt-attention score aggregation mechanism for efficient expert selection, an adaptive noise mechanism to balance expert utilization, and a prototype-based loss that uses prefix keys as implicit task memory. Experiments on standard benchmarks such as ImageNet-R, CIFAR-100, and CUB-200 show that SMoPE outperforms task-specific prompt methods while cutting computational costs by up to 50%. Overall, SMoPE demonstrates that structured sparsity in shared prompts can effectively mitigate forgetting and achieve state-of-the-art continual learning performance with low computational costs.", "questions": "1. Could the authors provide additional ablation studies to clarify the contribution of each component (sparse moe, adaptive noise mechanism, prefix keys as prototypes, and task-adaptive prediction) to the overall performance?\n\n2. Would it be possible to include more experimental results under different data-split settings (such as 5-split, 20-split, and 50-split) to demonstrate the robustness of the proposed approach? Now only 5-split, 10-split, 20-split for ImageNetR are included in the Appendix.\n\n3. Could the authors elaborate on the motivation and theoretical justification for the unified proxy score in Equation (7)? In particular, how was the aggregation of individual scores determined to be appropriate?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes SMoPE, an approach that integrates a sparse mixture of experts architecture into the prefix tuning framework, featuring a prompt-attention score aggregation mechanism for efficient and dynamic expert selection. \n2. This paper introduces an adaptive noise mechanism to encourage balanced expert utilization without overwriting prior knowledge, and a prototype-based loss that leverages prefix keys as implicit memory. \n3. This paper demonstrates that SMoPE achieves state-of-the-art results on multiple CL benchmarks, while using significantly fewer parameters and reducing computation compared to existing prompt-based methods.", "weaknesses": "1. The proposed method integrates multiple components, including a sparse mixture of experts architecture built on prefix tuning, an adaptive noise mechanism, prefix keys as prototypes, and task-adaptive prediction. However, the paper does not include an ablation study to analyze the individual contribution of each component. This omission makes it difficult to assess which parts of the system are most critical to its overall performance.\n2. The rationale behind the unified proxy score that aggregates individual scores in Equation (7) is not clearly explained. Without a theoretical or empirical justification for this formulation, it is unclear whether the aggregation is mathematically or conceptually sound.\n3. The experimental evaluation is not sufficiently comprehensive. The results are primarily reported for 10-split settings, leaving out potentially informative scenarios such as 5-split, 20-split, and 50-split evaluations. Including these would strengthen the empirical validation and demonstrate the method’s robustness across different data partitioning conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)