Mixture of Experts Meets Prompt-Based Continual Learning
Authors: Minh Le
An Nguyen The
Huy Nguyen
Thien Trang Nguyen Vu
Huyen Trang Pham
Linh Ngo Van
Nhat Ho
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Pre-training
Attention
Prompt Tuning
Mixture of Experts
Primary area: online_learning
TL;DR: We explore how prompts work in continual learning by showing their connection to mixture of experts and proposes a novel gating mechanism (NoRGa) for improved performance.

Abstract: Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms. Our code is publicly available at https://github.com/Minhchuyentoancbn/MoE_PromptCL.

Mixture of Experts Meets Prompt-Based Continual Learning
Minh Le
An Nguyen
Huy Nguyen
Trang Nguyen
Trang Pham 3-
Linh Van Ngo²
Nhat Ho
'The University of Texas at Austin
2 Hanoi University of Science and Technology
VinAI Research
Abstract
Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-basee baset continual learning methods excel in leveraging prompts for state-of-the-ar performance, they often lack theoretical explanation for the effectiveness of prompting. This paper conducts theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates Ga). Through the incorporation of non-linest activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse henchmarks and pretraining paradigms. Our code is publicly avail- able at https::://// // thtb. com http:///aaaaaaaaatoooooooo/oooea PromptCC_
Introduction
Humans possess a remarkable ability to learn continuously by integrating new skills and knowledge while retaining past experiences. However, current Al models often fail to retain this ability. Unlike humans, they often suffer from catastrophic forgetting [28] 30 32
38 phenomenon where they struggle to retain knowledge from previous tasks while learning new ones. Inspired by human learning, Continual Learning (2 28 43 12] is an ongoing field that aims to train model across a sequence of tasks while mitigating this challenge. Traditional continual learning methods often rely on storing past data for fine-tuning, which raise concerns about memory usage and privacy 3. 51). To address these limitations, prompt- based approaches have emerged as a promising alternative within rehearsal-fo ffree continual learning. By attaching prompts small sets of learnable parameters to a frozen pre-trained model, these approaches enable efficient adaptation to new tasks with minimal modifications the underlying model [56 26 61]. The effectiveness of prompt-based methods has been demonstrated by several recent works achieving state-of-the-art performance on
various continual learning benchmarks 49 53
While prompt-based methods have demonstrably achieved impressive results, their emphasis largely lies prompt utility, leaving gap in our theoretical comprehension of their effectiveness. This
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

absence of a theoretical foundation hinders our ability to further refine and optimize these methods. In this work, we offer a new perspective by focusing on prefix tuning [26] and its connection mixture of experts models [9 15 [ implicitly encode a special mixture of experts architecture, revealing a surprising connection between these seemingly disparate concepts. Leveraging this connection, we propose that applying prefix tuning within pre-trained models can be interpreted as introducing new experts. The newly introduced experts collaborate with the pre-trained experts, facilitating efficient adaptation of the model to new tasks.
13 II. We demonstrate that self-attention blocks in Vision Transformers
Drawing insights from this analysis, we observe that the original prefix tuning suffers from suboptimal sample efficiency, requiring a substantial amount of data for reasonable parameter estimation. To address this challenge, we propose novel gating mechanism termed Non-linear Residual Gates (NoRGa). This architecture integrates non-linear activation functions and residual connections within the gating score functions. Our work focuses on improving within-task prediction accuracy, key component of continual learning performance as identified in previous research 22 49) We posit
that NoRGa can enhance this aspect, which contributes to improved overall continual learning performance while maintaining parameter efficiency, We further provide theoretical justification for this improvement, demonstrating how NoRGa accelerates parameter estimation rates.
Our contributions can be summarized as follows: (1) We reveal novel connection between self-attention and a mixture of experts, providing a fresh perspective on prompt- based continual learning approaches: (2) Leveraging this insight, we propose Non-linear Residual Gates (NoRGa), an innovative gating mechanism that enhances continual learning performance while maintaining parameter efficiency, and provide a theoretical justification for this improvement; (3) Extensive experiments across various continual learning benchmarks and pre-training settings demonstrate that our approach achieves state-of-the-art performance compared to existing methods.
Notation.
Next, for any set S, we let |S] stand For any n E N, we denote [ri] as the set {1, ..,,....777 n..
for its cardinality. For any vector u = (21, U2..... Ua) E Rd and a == (a1, ....... ad) E No, we u22 uf"=
ud |u| = un + 02 +.. una and a! == on lag!!..... ... atal, while ||u|| stands for its 2-norm value. Lastly, for any two positive sequences (an)n2) and (b, Jn2,, we write a, = (b,,)
let u
an S bn if on S Cb, for all n N, where C > is some universal constant. The notation a, = Op(b,) indicates that an/ 4n/b, is stochastically bounded.
2 Background and Related Works
We first provide background and related works on continual learning. Then, we define the attention mechanism, followed by discussions on prompt-based continual learning and mixture of experts.
Continual Learning (CL. addresses the challenge of training model incrementally on a sequence
T tasks, denoted by D {D1.... ..., ). Each task's training data D. {(x{", ,/ (/ ) contains pairs of input sample x (2) E xttt), and corresponding label v," yx() Notably, the class labels are distinct for each task, i..,, )0(() nytt') Ø0, vt + 1'. Consider neural network with backbone function fa and an output layer hyp The model predicts a label y = h:: E E = - y00),
where x E X = -'_, 2'(1)
is an unseen test sample from arbitrary tasks. Importantly, during training on a new task, the model can only access the current data, without access to data from previous tasks. Prior approaches often rely on storing past task samples for training new tasks, raising concerns
regarding storage and privacy 6 39 51 59
Our work focuses on the class-incremental learning (CIL) setting, where task identities are not provided during inference, unlike in task inskiincremental learning (TIL) [46]. A recent theory by [22
analyzes the CIL objective by decomposing the probability of a test sample æ of the j-th class in task t into two probabilities:
P(x 2D) = P( E 230 x E ((P))) E x())
(1)
where the first term involves within- -task prediction (WTP) and the second term pertains to task- identity inference (TII). This equation highlights that by improving either the WTP performance or the TII, we can consequently improve the overall CIL performance, shown in 22 49
Attention Mechanism. Within the Transformer architecture, the attention mechanism plays a erucial role. One prevalent variant is scaled dot-product attention (4),, formally defined as follows:

---

Definition 2.1 (Scaled
Attention). Let K E Nxde be key matrix with NN key vectors, and V RNx, be value matrix with N corresponding value vectors. Given qutery matrix Q E RMMx, Attention over (K, V) is defined as
QK V
Attention(Q. (Q, K, V) softmax
(2)
Vdk
where the softmax function acts on the rows of matrix QK E ERMXN
Vision Transformer (ViT) g employs the same attention mechanism within multiple Multi-head Self-Attention (MSA) layers, which formally defined as follows:
Definition 2.2 (Multi-head Self-Attention Layer), Let XQ xx XV denote the input query, key,
and value matrix, respectively, where X = X X 111,, NN] E R and N is the
length of the input sequence. The output is expressed as MSA(X x x == Concat(h),..... hm W E Rxd
(3)
h Attention( X & X W.K X xWW)), ie [m].
(4)
where Wo E 1m xd W Rddxd W.K E Rdxds, and W.Y E Rddds are projection matrices, and m is the number of heads in the MSA layer. In ViTs, they use d2 de = d/m.
Prompt-base based continual learning. Prompt- Prompt-based approaches have emerged as a promising alterna- tive within rehearsal free continual learning [6]] 52 44]. In vision tasks, prompt-based methods often leverage a pre-trained ViT feature extractor fo, with its parameters 0 typically frozen. These methods enhance the model by introducing prompts, small sets of learnable parameters that influence the operations of the MSA layer (53). Prompts are strategically injected into the query, key, and value matrices to guide the VIT in learning new tasks. We denote the prompt parameters by p E IR xd where Lp is the sequence length and d is the embedding dimension. Previous work (53) outlines two main prompt-based approaches: Prompt Tuning (Pro'T) 25 and Prefix Tuning (Pre'T) [26]. While Prompt Tuning directly concatenates the same prompt parameter p the query, key, and value, prefix tuning divides into prefixes {p" ,
E R - xd and appends it to the key and value vectors:
Sprompi (p. X X X MSA
x
p XK
p' X
= Concat(h1,..... W (5)
Existing prompt-based based methods in CL address catastrophic forgetting by creating new adaptive prompts for each new task. During testing, the model chooses suitable prompt combinations to handle unseen data from any encountered task [49]. I.2P (54) proposes a shared prompt pool for all tasks, utilizing a query-key mechanism for prompt selection. Instead of using the same prompt pool across tasks, DualPrompt (53) introduces G- G-Prompt and E-Prompt to capture aask-agnostic and task-specific information, respectively. S-Prompt [52] focuses on learning task-specific prompts and employs a ProT strategy similar to L2P. a weighted summation of the prompt pool using attention factors. A recent work, HiDe-Prone 44), achieves state-of-the of-the-art performance by introducing new hierarchical decomposition of CIL objectives and optimizing each component for better performance.
42 expands the prompt pool across tasks and performs
In this study, we focus prefix tuning as our primary prompt-based methodology and follow the framework presented in PromepP 49). During training. Promppt co-optimizes task-specific prompts p: and model's is output layer parameters 1/2 for each new task / using the WTP objective. These prompts are stored within a prompt pool P = {PI..... }.]. At test time, a separate lightweight auxiliary output layer h RD + R²,,, trained with the TII objective, takes the uninstructed representation So (ax) of new data point æ as input to infer the task identity, guiding the selection of the most suitable prompt pk from the prompt pool P. Subsequently, the final prediction is given as ÿ (fe(z, p.)). For further details, please refer to Appendix D
Mixture of experts (MoE) extends classical mixture models with an adaptive gating mechanism 19 21). An MoE model consists of a group of N expert networks fi : R - Rd for all i e [N], and gate function G Rd + RV Given input h E Rd, MoE computes weighted sum of expert outputs f; (h) based on learned score function 8; Rd - for each expert:
N
exp (s, (h))
y = Σ G(h); f/())
Σ
(6)
N Σ Emm exp (st(h)
j=1
3

---

followed by function sty
Wg
Each entry the matrix
X Wi
function Siy
transpose
softmax
Wy
Each value matrix is an expert fj
Figure 1: An illustrative depiction of the relationship between self-attention and MoE. Each output vector of head in the MSA layer can be viewed as the output of a MoE model. These MoE models share the same set of experts encoded in the value matrix. Each entry in the attention matrix corresponds to score function within this architecture.
where G(hh) ;= softmax(s1 (h)..... (h)).. Building on this concept, works by [0 41 established the MoE layer as fundamental building block to scale up model capacity efficiently. Please refer to Appendix C for a comprehensive discussion of related works.
Connection between Prefix Tuning and Mixture of Experts
We first explore the relationship between attention and mixture of experts Section 3.1 establishing the connection between prefix tuning and the mixture of experts in Section 3.2
3.1 Mixture of Experts Meets Attention
Following the notation established in Definition 2.2 let's consider the 1-th head within the MSA layer. Let X = [ x Rdd, which is the concatenation of input sequence embeddings into single one-dimensional vector. We define the matrix E, E dg Nd such that E;X æ, for all i E [N]. Furthermore, we introduce an MoE architecture consisting a group of N expert networks fg .R + Rd., gating functions G, .Rd - R with score function for the j-th expert of the i-th gating 8t.j Nd R. where
X E. W E,X
x. W. WWWW æ
(X)) = WY - Ej 8i,(
Vd,
Vd,
for and j E [N]. From equation
we can express the output of the /-th head as follows:
xOWWWKk X
h; = softmax
x XVWV lh.......
N E Nxd.
(7)
dd
14s
exp
Vile
W.
x. Σ
exp(si, E exp(s1,
fx(X),
(8)
xt W.W W Vd.
= exp
for E [N]. Expanding equation implicitly embodies a special mixture of experts architecture. This architecture encompasses N MoE models, each featuring its own quadratic gating function G.. However, instead of employing 2
(8).
we can discern that each attention head within the MSA layer

---

Eepert
Score function for peetralned experts
2
Wike motrle
srane funation of orginaal preitc tuning
Nondnwar
Epert
activation
Nottia
VECTORS
Figure Left: An illustrative depiction prefix tuning as the introduction of new experts into pre-trained MoE models. Right: Visualization of NoRGa implementation, integrating non-linear activation and residual connections into the prefix tuning attention matrix.
separate expert networks for each model, this architecture utilizes N shared linear expert networks Sg for j E [N], significantly reducing the number parameters. Notably, each expert network and its corresponding gating function process the entire input sequence directly, rather than individual embedding æ; as in traditional MoF layers [41]. This connection between self-attention and mixture of experts depicted in Figure In the subsequent section, we explore how prompt-base bbsed techniques can be viewed through this lens.
3.2 Prefix Tuning via the Perspective of Mixture Experts
Building on the connection between self-attention and mixture of experts, we propose that applying
prefix tuning can be interpreted as the introduction new experts to customize the pre-trained model
for specific task, as illustrated in Figure Specifically, similar to Section
3.1 we consider the I-th
head within the MSA layer. We denote pd -Ip ,p.,.... .P.] E RRdd
p = =Ip ........ P..
RLxd, where L Ly We define prefix experts fN+j R.N corresponding new score functions St,N+ Nd + E as follows:
Rdd along with their
X X F. W PF
a. W.W p
fN+5 = W. py,
(XN+s(X)=
Vd, 5). the output of the 1-th head can be expressed as:
Vd,
for E [N] and j E [Z.]. Then from equation
h4 Attention
x
p
W.K.
p
wy
hu h h
E Rxd,
(10)
X
XV
N
exp exp(s (X))
h
Ek' exp(sI.N4 fx(X)
1= exp(s ()) )
1=1
exp(s,,N4 (X))
j'=1 Ek (x)( Ek exp exp(s),++*(XX) (N+
(11)
It's worth noting that w W.K and W.Y remain fixed, with only px and p being learnable. By examining equation 8 and equation III. we can interpret each head in a multi-head self-altention layer within a pre-trained model as a mixture experts architecture with pre-trained experts f; and gating score functions St for i and [N]. Prefix tuning extends this MoE by introducing L additional prefix experts fN-z' defined by prefix vectors p., and linear score functions st,++'' for i [N] and j' E [L]. These new experts collaborate with the pre-trained ones within the MoE model, facilitating the model's adaptation to downstream tasks.
We argue that our introduction of novel connection between self- elffaatention, prefix tuning, and MoE offers fresh perspective on the design previous prompt-based continual learning methods. the context of continual learning, the pre-trained experts serve knowledge base, while prefix tuning augments it with task-specific knowledge encoded in new experts. Moreover, we draw a parallel between the pre-trained experts and the (eeneral)-Promy utilized in DualPrompt, which captures

---

task-agu -agnosii information [53] Both are shared across tasks, making them useful for prediction, especially when task identification is incorrect. Notably, the new experts achieve their efficiency through simple linear gating functions and independence from the input, unlike the pre-trained experts. For simplicity, we call the MoE. model
(11) as linear gating prefix MoE.
Statistical suboptimality. The connection between prefix tuning and the MoE within the linear gating prefix MoE model (II) allows us to theoretically explore the statistical behavior of the prefix tuning. In Appendix A by interpreting the linear gating prefix MoE as a regression problem with sample size n, we demonstrate that the convergence rate for estimating the model parameters, e-g.. prompts, can be as slow as (1/ log" (n)) where 7 > 0 is some constant. This suggests that huge amount of data is required achieve reasonable parameter estimation in the linear gating prefix MoE model, which can be discouraging in practice. To address this statistical limitation, the next section introduces novel non-linear residual gating score function to replace the linear gating function.
4 Non-linear Residual Gate Meets Prefix Tuning
As discussed earlier, prefix tuning introduces additional experts within the MoE framework, resulting in the linear gating prefix MoE model. However, outlined in Appendix A this approach suffers from suboptimal sample efficiency for parameter estimation. To overcome this and enhance overall CIL performance, we propose innovative approach that significantly improves sample efficiency while promoting WTP performance in Section 4.1 and provide theoretical explanations in Section 4.2
4.1 NoRGa: Non-linear Residual Gate
We propose simple yet effective modification to the linear gating prefix MoE model by incorporating non-linear activation and residual connection within the score functions of prefix experts as follows:
X EE W D +a.0
X E. WO P)
S1,N+ + (X)
T
dd
Vd,
(12) where a, E R are learnable sealar factors, and is non-linear activation function. The new score function in equation (12) consists of a linear and a non-linear component. We call the new prefix as non-linear residual gating prefix MoE.
S aa) -T. (X)), (M,, j JE [L].
MoE model with score functions 12
The use a non-linear activation function here is motivated by the algebraic independence condition in Definition 4.2 to theoretically guarantee the optimal sample efficiency of expert and parameter estimations (cf. Theorem 4.3). It's worth noting that removing the linear component $,,v+; (X) in the score function (12) could potentially lead to the vanishing gradient problem during training. To mitigate this challenge, we incorporate a residual connection [4] into the formulation. Our modification introduces minimal additional parameters (a and 7) compared to the original score function, ensuring parameter efficiency. This is particularly crucial in continual learning scenarios where the number of parameters grows with each new task. For implementation, we define Hi; w/Rk' From equation
(5) the attention matrix of the 1-th head can then be written as:
A H
_x
11pk T xX911X=1 [A prompt A pretraing
(13)
Vd
Vd
Here, A Apromps denotes the attention score matrix for the prompts, and A attention score matrix for the pre- pre-trained experts. To implement NoRGa, we can directly modify the final attention matrix as follows:
Apretrain represents the
A, = Arompt Apretrain]
(14)
A Aromppt prrompt + a o Aprompt).
(15)
The implementation of NoRGa is illustrated in Figure 2 Despite its simplicity, our modification can significantly enhance sample efficiency and promote more reasonable parameter estimation, as demonstrated in our theoretical analysis in Section 4.2, Within the HiDe- HiDe-Prompt framework, task-specific prompt parameters trained using the WTP objective for each new task. Consequently, our modification leads to better parameter estimation, which directly contributes to improved WTP performance, ultimately improving overall continual learning efficacy. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL because of the memory constraint. Here, we evaluated a with tanh, sigmoid, and GELU, finding tanh to perform well in most cases.

---

4.2 Theoretical Explanation for Non-linear Residual Gating Prefix MoE
Similar to the setting in Appendix A we prove that estimating parameters in the non-linear residual gating prefix MoE. model (12 is statistically efficient in terms of the number of data. To provide a fair comparison namely, I = 1 and i = 1 in equation [12 Then, we proceed to provide theoretical justification of our claim by viewing this row as an output of a regression setting. In particular, we assume that (Xi, Yi), (X2, Y2)..... (Xn, Yn) E R R are i.i.d. samples generated from model:
the linear gating prefix MoE, we focus only on the first head and its first row,
Y; 9G_ (X,) + EG i = 1, 72
(16)
where El..... En are independent Gaussian noise variables such that E|s; [X,] = and Var(s, (X,) = 2² for all 1 < i n. Additionally, we assume that X X2, X,, are i.i.d. samples from some probability distribution p. The regression function gG. (.) in equation (T6) then takes the form of a non-linear residual gating prefix MoE model with N pre-trained experts and L unknown experts,
exp(X X B X + (.
gG. (x)
h(x,1)))
T(X)
j=1
exp(((t)))) x + ao (T TX)
X): + +br))
((XXnn)))
(17)
T(X)
where T(X) = Eb= exp X ( + + + + exp(((1x) [x X + (TT T X) + (ow))
G. = Σ (xp( (o denotes a mixing measure, i.e., weighted sum of Dirac measures 6, associated with unknown parameters 18ig,,Piy nř - in Nd x R x R2, Here, the matrix By is equivalent to (ET W w Ej/vd,) in the score function $1 (X), and the vector Tiy corresponds to the vector ( w p / (V))) in (x). Furthermore, the experts h(X,, n))) and h(X, n)-)) represent (X) and (X))((),, respectively. In our formulation, for the generality of the ensuing theory, we consider general parametric forms of the experts h(X, n79) and h(X, n i... we do not only constrain these expert functions to be the forms of the simple experts in the aforementioned model. Similar to the setting in Appendix A By. cg. and the expert parameters nf are known. Our goal is to estimate the unknown prompt-re ratete parameters Piy, and n7,--
Least squares estimation. We will use the least squares method [45] to estimate the unknown parameters (86,., Gi
or, equivalently, the ground truth mixing measure G*. In particular, we take into account the estimator
G = arg min
2(x- (xx--9((X))) 9x(x (Y -
(18)
(e)
where we denote ()) (G exp(3ut)6(3m.m)) 1 I', (Bus, BiG, n:) E Q} as the of all mixing measures with at most L' atoms. In practice, since the true number of experts L is typically unknown, we assume that the number of fitted experts L' is sufficiently large, i.e., L' > L.
To begin with, we explore the convergence behavior of the regression estimator 9G. (-) to the true regression function gG. (-) under the L2(1)-norm in the following theorem:
Theorem 4.1 (Regression Estimation Rate). Equipped with a least squares estimator G, given in equation 18) the model estimation 98. (.) converges to the true model 9G. (-) at the following rate:
l1ga. 9G. ||اي Or Or((Vogg()//m)
(19)
Proof of Theorem 4.1 is in Appendix B. The bound (19) implies that the rate for estimating the regression function 9G. (-) is of order Op Iog(n)/n). which is parametric on the sample size More importantly, it also indicates that there exists a loss function among parameters C such that
l19., 9G. (ما)( 2 C(G., G.), then we would obtain the bound C(G,,, G.) = OP((/log())/n) which leads to the desired parameter and expert estimation rates.
We now turn our attention to the parameter and expert estimation problems. To understand how the non-linear residual gating affects these problems, we analyze the properties of the expert h((;,)) 4)) and the activation function (()) to determine which formulations will achieve favorable performance.
7

---

Definition 4.2 (Algebraic independence). We say that an expert function h(=, 7) and an activation function (-) are algebraically independent if they are twice differentiable w.r. their parameters. and if for any & 2 and pair-wise distinct parameters (311, 77 m)..... (B1k. Mc), the following set of functions in X is linearly independent for almost every X E VR:
O h
{ x" (1+ o' (31, + "0
(x, nh) : [ka],
Oq7
E Y E No :0 | + hi 2]
Intuitively, the algebraic independence condition ensures that there will be no interactions among parameters of the expert function h(.,7) and the activation function o.)). Technically, a key step in our argument is to decompose the regression discrepancy 92. (X) gc. (X) into combination of linearly independent terms by applying Taylor expansions to the product of the softmax's 's numerator and the expert function, i.e., exp( X ( ()) ..)). Thus, the above condition guarantees that all the derivative terms in the Taylor expansion are linearly independent. To exemplify the algebraic independence condition, we consider the following simple examples of the expert functions h(-,)) and the activation (-) that are algebraically independent.
Example. When the expert function h(., m) is formulated a neural network h(X, (a.b)) pp(a X + b) with the activation p0-) {ReLU(-), GELU(-), + 2²), where (a. R Nd R. and the activation function (-)) is one among the functions sigmoid(-), tanh("), GELU(-), then they satisfy the algebraic independence condition in Definition 4.2
Finally, we establish the rates for estimating parameters and experts in the non-linear residual gating prefix MoE model in Theorem 4.3 Prior to presenting the theorem statement, let us design loss function among parameters based on a notion of Voronoi cells [27]. which is a commonly employed
approach for the convergence analysis of expert estimation in MoE models [66 34 35 33 yet tailored to the setting of this paper. In particular, the Voronoi loss used for our analysis is defined as
Li G.) =
E E
exp((ox)
| 112 + 11
Σ Σ
exp(Sos) |1 + |
+
- -xp(2un)cxxp(66))) (20)
VEVy
where we denote A 181 == B1g Bi and Anp := 17s - nnj Above, Vy = VG) for E [L]. is
Voronoi cell associated with the mixing measure G generated by the true component w/- = (Bi nz.). which is defined follows:
Vy. E 21,2...... .......... L'): : ||: - w S ||: will ve
(21)
where we denote W/ := (S1.. 17)) as a component of G. Note that, the cardinality of each Voronoi cell Vy indicates the number of components w; of G approximating the true component w.. of G.. Additionally, since La (G, G.) 0 if and only if G G., it follows that when L1 (G, G.) becomes sufficiently small, the differences AB1y and Anjy' are also small. This observation indicates that, although L, G.) is proper metric as it is not symmetric, it is an appropriate loss function for measuring the discrepancy between the least square estimator G, and the true mixing measures G..
Theorem 4.3. Assume that the expert function h(x, 7) and the activation (-) are algebraically independent, then we achieve the following lower bound for any G E (O):
llga ga. gga.l1z(()) 2 Lz(G,
which logether with Theorem 4.1 indicutes that Li(G,, G.)G,,,.) Ōp(nn111))
Proof of Theorem 4.3 is in Appendix B.2 A few comments on Theorem 4.3 are in order: (i) From the bound L1 (G G.) = Op(n-1/2), we deduce that the estimation rates for the over-specified parameters Bi Viy-. where j' E [L] 1V3r > 1, are all of order Op( //og(n)/n)) Since the expert ((-m)) is twice differentiable over bounded domain, it is also a Lipschitz function. Thus, denote
8

---

Table 1: Overall performance comparison on Split CIFAR-100 100 and Split ImageNet-R, We present Final Average Accuracy (FA), Cumulative Average Accuracy (CA), and Average Forgetting Measure (FM) of all methods under different pre-trained models.
Split CIFAR-100 100
Split Imagenet
PTM
Method
FA (t)
CA(t)
FM(1)
FA (f)
CA(t)
FM(()
L2P
83.06 + 0.17
88.27 + 0.71
5.61
+ 0.32
67.53 + 0.44
71.98 + 0.52
5.84 + 0.38
DualPrompt
87.30 +: 0.27
91.23 1: 0.65
3.87
+ 0.43
70.93 = 0.08
76.67 +: 0.52
5.47 + 0.19
Sup-2IK
S-Prompt
87.37 4: 0.42
01.88
3.63
0.41
09.88 - 0.51
74.25-0.33
4.73
CODA-Prompt
86.94 4: 0.63
0.75
4.04
70.03 = 0.47
0.224
5.17 0.22
HiDe-Prompt
02.61
91003
0.28
75.06 = 0.12
76.00
4.09 = 0.13
NoRGa (Ours)
94.48 =: 0.13
95.83 1:33-:00 0.37
1.44 0.27
75.40
0.39
0.07 79.52
4.30 0.07
L2P
79.13 1.25
85.13
0.05
61.31
0.50
68.81 0.52
10.72
DualPrompt
:.444400.447
86.16
0.02
8.84
0.67
58.69 + 0.61
66.61 + 0.67
11.75
:BOT:21K
S Prompt
79.14 0.65
85.85 + 0.17
8.23 + 1.15
57.96 + 1.10
66.42 +: 0.71
11.27 0.72
CODA-Prompt
80.83 0.27
87.02 0.20
7.50 0.25
61.22 0.35
66.76
+ 0.37
9.66 +: 0.20
HiDe- PrDemPr
93.02 0.15
94.56
+: 0.05
1.26 +: 0.13
70.83 = 0.17
73.23
1: 0.08
6.77 = 0.23
NoRGa (Ours)
94.76 0.15
95.86 + 0.31
1.31 - 0.14
73.06 = 0.26
0.42 77.46
6.88 + 0.49
L2P
75.51 0.88
82.53 + 1.10
6,50 +: 1.70
59.43 = 0.28
66.83 + 0.92
11.33 = 1.25
DualPrompt
76.21 1.00
83.54 = 1.23
9.89 + 1.81
00.41 0.76
66.87 + 0.41
9.21 + 0.43
iBOT-IK
S-Prompt
0.61
82.89 + 0.89
8.60 + 1.36
59.56 = 0.60
66.60 + 0.13
8.83 + 0.81
Prom-PPrompi
79.11
7.69
1.57
66.56
7.22 0.38
HiDe-Promp Prompt
93.48 0.11
95.02 + 0.01
1.60 + 0.10
71.33 = 0.21
73.62 + 0.13
7.11 + 0.02
NoRGa (Ours)
94.01 + 0.04
0.35 95.11
1.61 + 0.30
0.20 72.77
0.46 76.55
7.10 + 0.39
L2P
72.23 0.33::00.355 0.35
79.71 + 1.26
8.37 :: 2.30
57.21 + 0.69
64.09 + 0.74
7.47 + 0.96
DualPrompt
73:95
81.85
0.59
9.32 1.42
57.98 0.71
65.39
0.27
9.32 +: 0.69
S- S-Prompt 74.39 0.17
81.60 + 0.74
9.07 + 1.13
57.55 = 0.72
64.90 1: 0.13
8.73 + 0.56
DINO-IK
CODA-Prompt
77.50
$481 = 0.30
8.10
63.15 + 0.39
69.73 0.25
6.86 0.11
HiDe-Prompt
92.51 +2.51++0.11
94.25
0.01
1.67 + 0.20
68.11 0.18
71.70 0.01
6.45 + 0.58
NoRGa (Ours)
93.43 =:: 0.33
94.65 : 0.62
1.65 + 0.25
71.77 - 0.44
0.49 75.76
6.42 - 0.68
L2P
77.24 +: 0.69
83.73 + 0.70
5.57 +: 0.75
54.13 + 0.67
62.09 + 0.76
4.8% + 0.42
DualPrompt
77.56 0.63
84.37
6.54 +:
54.45
0.41
5.34 +: 0.41
McCo-IK
S- S-Prompt
77.20 +.339
84.47 + 0.37
7.00 0.62
53.94 + 0.32
62.42
+ 0.51
5.16 + 0.48
Prom-Prompt
77.83 0.34
84.97 0.23
12.60 + 0.02
55.75 - 0.26
65.49
+ 0.36
10.46 + 0.04
HiDe-Prompt
91.37 0.20
1.51
63.77 = 0.49
08.26
0.01
9.37 0.71
NoRGa (Ours)
93.52 + 0.06
94.94 + 0.29
1.63
64.52 + 0.16
70.21 + 0.64
9.06 + 0.19
Gn = Eta (xap(Bn)) (3)) we achieve that
sup |A 7. h (X, nz. - Op( (/Toog(())/)))
(22)
The above bound indicates that if the experts hu, n2)) are fitted by at least two other experts, then their estimation rates are also of order Op( </log(n) (n)/n; (ii) For exactly-specified parameters Fi Ng where j' [Z] : |Vy1 1, the rates for estimating them are faster than those of their over-specified counterparts, standing at order Op( On)))og(n)/n). By arguing similarly to equation (2 the experts k(( also enjoy the faster estimation rate of order (P((log(n)/m) which is parametric on the sample size nı (iii) It follows from the above rates that we only need polynomial number of data (roughly e where is the desired approximation error) to estimate the parameters and experts of the non-linear residual gating prefix MoE. By contrast, when using the linear gating, as being demonstrated in Appendix A it requires an exponential number of data. This highlights the statistical benefits of using the non- lonerr residual gating MoE model over the linear gating prefix MoE model.
Experiments
Datasets. We evaluate various continual learning methods on widely used CIL benchmarks, including
Split CIFAR-100 (23 and Split ImageNet-R (23] consistent with prior work [49]. We further explore the model's performance on fine-grained classification tasks with Split CUB-200 (48) and large
inter-task differences with 5-Datasets ( Please refer to Appendix D for more details.
Evaluation Metrics. We utilize several established metrics described in [50] These include: final
average accuracy (FA), which represents the average accuracy after the final task; cumulative average accuracy (CA), which refers to the historical average accuracy; and average forgetting measure (FM).
We give more emphasis to FA and CA due to their comprehenssieenes, noted in [42).
Baselines. We compare our approach with several representative prompt-hased approaches including L2P (54) DualPrompt (53), CODA- [4), S-Prompt [52), and HiDe- [49]. Additionally,

---

Table 2: Final average accuracy (FA) on Split CUB-200 and 5-Datasets.
Split CUB-200
5-Datasets
Method
Sup-2IK
iBOT-21K
Sup-2IK
iBOT-21K
L2P
75.46
46.60
81.84
82.25
DualPrompt
77.56
45.93
77.91
68.03
S-Prompt
77.13
44.22
86.06
77.20
CODA-Prompt
74.34
47.79
64.18
51.65
HiDe-Prompt
86.56
78.23
93.83
94.88
NoRGa (Ours)
90.90
80.69
94.16
94.92
Table 3: Ablation study of different activation functions, measured by final average accuracy (FA).
Split CIFAR-100 100
Split CUB-200
Method
Sup-21K
iBOT-21K
Sup-21K
iBOT-21K
HiDe-Prompt Prompt
92.61
93.02
86.56
78.23
NoRGa tanh
94.36
94.76
90.87
80.69
NoRGa sigmoid
94.48
94.69
90,90
80.18
NoRGa GELU
94.05
94.63
90.74
80,54
we evaluate against state-of-the-ar pre-trained model-based continual learning methods, including ADAM [60] and RanPAC [29]. We further extend our evaluation by applying HiDe-Prompt with
parameter-efficient fine-tuning techniques like LoRA [7] and Adapters [[]] In line with [49], we utilize the checkpoints of VIT that use supervised pre-training of Imagenet-211 (denoted as
Sup-21K), and some supervuee pre-training such as iBOT-21K, iBOT-1K 62. DINO-1K 4 and MoCo-IK 77 For implementation details, see Appendix
Main Results. In Table [ we evaluate several continual learning methods Split CIFAR-100 and Split ImageNet-R using diverse pre-trained models. NoRGa achieves state-of-the-ar FA and CA across all datasets and models, consistently outperforming HiDe-Prompt. On Sup-21K, NoRGa demonstrates impressive FA results on both CIFAR-I00 and ImageNet-R. It also maintains the highest CA. with significant margins of 1.80% and 2.92% on CIFAR-100 and ImageNet-R, respectively, compared to HiDe-Prompt. These results highlight NoRGa's strong ability to retain knowledge and exhibit minimal forgetting, evidenced by the low FM values on both datasets. NoRGa also surpasses HiDe- Prompt on self-supervised models, with FA improvements up to 1.95% and 3.66%. We further investigate two scenarios: fine- tee-grained classification tasks and large inter-task differences through experiments on Split CUB-200 and 5-Datasets, respectively, as shown in Table 2 NoRGa maintains its lead, achieving FA gaps of 4.34% and 2.46% on Split CUB-200, and the highest FA on 5-Datasets. While gains in some metrics may be modest, NoRGa consistently outperforms HiDe-Prompt either FA or CA, underscoring its robustness. For example, on Split ImageNet-R with Sup-21K weights, the FA improvement is small (75.06% vs. 75,40%), but the CA gains are substantial (76.6%% vs. 79.52%), demonstrating the method's effectiveness and robustness.
HiDe-Prompt
Ablation Study. To assess the impact of non-linear activation functions on NoRGa's 's performance, we evaluated the model's ' behavior with different choices for the activation function a, including tanh, sigmoid, and GELU in Table 3 The results show that NoRGa achieves -tate-of-thh-art performance on both Split CIFAR- 100 and Split CUB-200 datasets with all three activation functions. These findings suggest that NoRGa exhibits robustness to the choice of non-linear activation within reasonable range. While all functions perform well, the tanh activation function demonstrates generally strong performance across scenarios. Further results are provided in the Appendix.
Conclusion
This paper presents an initial exploration of self-attention and prefix-tuning through the lens of mixture of experts. We find that applying prefix tuning can be viewed as introducing new prefix experts to adapt the pre-traine pre-trained model. However, limitations in sample efficiency exist. We address this by proposing NoRGa, a novel gating mechanism to enhance continual learning performance. Our results demonstrate NoRGa's effectiveness both theoretically and empirically. While the current implementation of the expert network prioritizes simplicity, future research directions could involve investigating more intricate architectures. Furthermore, the choice of activation functions in our work requires fine-tuning, which opens avenues for future research adaptively learning activation.
10

---

References
[1] R. Aljundi, Chakravarty, and T. Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2366-3375 2017.
[2] E. Belouadah, A. Popescu, and I. Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38-55 54, 2021.
[3] Y. Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online). Available: http:///yroslaavbb blogspot. #2201109hhommniss-aaaaass haml, 2, 2011.
[4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the International Conference on Computer Vision (ICCV), 2021.
[5] H. Cha, J. Lee, and J. Shin. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 9516-9525, October 2021.
[6] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. S. Torr, and M. Ranzato. On tiny episodic memories in continual learning, 2019.
[7] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers,
2021.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[9] Ebrahimi, F. Meier, Calandra, T. Darrell, and M. Rohrbach. Adversarial continual learning. arXiv preprint arXiv:2003 2020..
[10] D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in deep mixture of experts. In ICLR Workshops, 2014.
[11] Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, Z. Wang, et al. M Mixture- of-experts vision transformer for efficient multi-task learning with aocel-accelerator co-design. Advances in Neural Information Processing Systems, 11-28441-28457 2022.
[12] N. L. Hai, T. Nguyen, N. Van, T. H. Nguyen, and K. Than. Continual variational dropout: a view of auxiliary local variables continual learning. Machine Learning, 1 113(1):281-325 2024.
[13] H. Hazimeh, Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, Mazumder, L. Hong, and E. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances Neural Information Processing Systems. 44:29335-293477 2021.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[15] N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1-81, 2022.
[16] N. Houlsby, A. Giurgiu, S. Jastrzebski. B. Morrone, Q. De Laroussilhe, A. Gesmundo. M. At- tariyan, and S. Gelly. Parameter-efficien transfer learning for nlp. In International conference on machine learning, pages 2790-2799, PMLR, 2019.
[17] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106 09685, 2021.
[18] Q. Huang, Z. An, N. Zhuang, M. Tao, C. Zhang, Y. Jin, K. Xu, L. Chen, S. Huang, and Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07552 2024.

---

[19] R. A. Jacobs, M. I. Jordan, J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computationn 3, 1991.
[20] Janson, W. Zhang, R. Aljundi, and M. Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04-22 2022.
[21] M. I. Jordan and R. A. Jacobs, Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[22] G. Kim, C. Xiao, T. Konishi, Z. Ke, and B. Liu. A theoretical study on solving continual learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 5065-5079, Curran Associates, Inc., 2022.
[23] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[24] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied document recognition. Proceedings of the IEEE, 86(11) (11111)2278223220 1998.
[25] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficier prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5045-3059. Association for Computational Linguistics, Nov. 2021.
[26] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
[27] Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14979-15006. PMLR, 17-23 Jul 2022.
[28] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[29] M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. van den Hengel. Ranpac: Ran- dom projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[30] S. V. Mchta, D. Patil, S. Chandar, and E. Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research, 24(214):1-50, 2023.
[31] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[32] C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto. Toward under- standing catastrophic forgetting in continual learning. 2019.
[33] H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML, 2024.
[34] H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations, 2024.
[35] H. Nguyen, N. Ho, and A. Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICML, 2024.
[36] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023.
[37] A. Panos, Y. Kobe, D. O. Reino, R. Aljundi, and R. E. Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18820-18830, 2023.
12

---

[38] H. Phan, A. P. Tuan, S. Nguyen, N. V. Linh, and K. Than. Reducing catastrophic forgetting in neural networks via gaussian mixture approximation. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 106-117. Springer, 2022.
[39] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[40] T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, Imagenet-21k pretraining for the masses, 2021.
[41] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely- on Learning Representations (ICLR), 2017.
cexture-of--xppeee layer. In International Conference
[42] J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante- Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 1 19999-11919 June 2023.
[43] Q- Tran, H. Phan, K. Than, D. Phung, and T. Le. Continual learning with optimal transport based mixture model. arXiv preprint arXiv:221 167780 2022.
[44] Q Tran, L. Tran, K. Than, T. Tran, D. Phung, and Le. Koppa: Improving prompt-based continual learning with key-query orthogonal projection and prototype-based one-versus-all. arXiv preprint arXiv:2311.15414 2023.
[45] van de Geer. Empirical processes in M-estimation. Cambridge University Press, 2000.
[46] G. M. van de Ven, T. Tuytelaars, and A. S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 151185-11997 2022.
[47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[48] C. Wah, Branson, Welinder, P. Perona, and Belongie. The Caltech-UCSD finds-200-201 I Dataset. California Institute of Technology, 2011.
[49] L. Wang, J. Xie, X. Zhang, M. Huang. H. Su, and J. Zhu. Hierarchical decomposition of prompt-basen based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 2023.
[50] L. Wang, X. Zhang, H. Su, and J. Zhu. A comprehensive survey of continual learning: Theory, method and application, 2024.
[51] L. Wang. X. Zhang, K. Yang, L. Yu, C. Li, H. Lanqing, S. Zhang, Z. Li, Y. Zhong, and J. Zhu. Memory replay with data compression for continual learning. International Conference on Learning Representationss 2021.
[52] Y. Wang, Z. Huang, and X. Hong. S-prompts learning with pre-trained transformers; An occam's razor for domain incremental learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[53] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference Computer Vision, 2022.
[54] Z. Wang. Z. Zhang, C.-Y. Lee, H. Zhang, Sun, X. Ren, G. Su, Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139, 2022.
[55] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
13

---

[56] Y. Xin, S. Luo, H. Zhou, J. Du, X. Liu, Y. Fan, Q. Li, and Y. Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey, 2024.
[57] B. Yu. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam, pages 423-435, 1997.
[58] Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He. Boosting continual learning of vision-language models via mixture-of- Conference on Computer Vision and Pattern Recognition, pages 22219-23230, 2024.
adapters. In Proceedings of the IEEE/CVE
[59] G. Zhang, I. Wang. G. Kang, L.. Chen, and Y. Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[60] D.-W. Zhou, Z.-W. Cai, H.-J. Ye. D.-C. Zhan, and Z. Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are you need. International Journal of Computer Vision, pages 1-21, 2024.
[61] D.-W. Zhou, H.-L. Sun, J. Ning, H.-J. Ye, and D.-C. Zhan. Continual learning with pre-trained models: A survey, 2024.
[62] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
14

---

Supplement to "Mixture of Experts Meets Prompt-Based Continual Learning"
In this supplementary material, we first analyze the statistical suboptimality of the Linear Gating
Prefix MoE Model (IT) in Appendix A
Appendix B provides proofs for the theoretical results
presented in Section 4.2 In Appendix C we discuss related works on mixture of experts. Appendix [
outlines the training algorithm for HiDe-Prompt while Appendix F presents the experimental setup
and details. Further, Appendix F presents further experiments on the task-incremental learning
setting to empirically demonstrate the benefits of using our proposed Non-linear Residual Gating Prefix MoE (12) over the Linear Gating Prefix MoE Model. Appendix [ Appendix H compare NoRGa with other parameter-efficien fine-tuning techniques and pre-trained model-based methods. In Appendix DI we present the efficiency tests, while Appendix D explores the impact of learnable a and T. Finally, Appendix K| compares the training times between NoRGa and DiDe-PPrmmt.
A Statistical Suboptimality of Linear Gating Prefix MoE Model
In this appendix, we demonstrate that estimating parameters and experts in the linear gating prefix MoE model (IT) can be statistically inefficient in terms of the number of data. To simplify our findings, we particularly focus on the first head, namely, = 1 in equation (TT). and the first row of this head, namely. i = 1 in equation (II). Then, we proceed to provide theoretical justification of our claim for the suboptimality of the linear gating prefix MoE by viewing this row as an output of
the regression setting. In particular, we assume that (X1, Y1), (X2, Y2)..... (X Yn) E Nd x R an i.i.d. sample generated from the following model:
Y fa. (X.) &i i 1,2,......
(23)
where ε1; ........................... En are independent Gaussian noise variables such that Eee; XX] 0 and Var(e;
22 for all 1 S S n. Additionally, we assume that X....... Xx, are i.i.d. samples from some probability distribution Jr. Motivated by linear gating prefix MoE model (TI), the regression function
fG. (-) in equation (23) admits the form of the linear gating prefix MoE model with pre-trained N experts and L unknown experts, namely
exp X B + c.)
(X)
-hX, m))
1=1 1= exp(X T B& c + +1 Ek exp(((ix) TX + )
exp((8i,)) T T
+ Pby.)
h(X mye) (24)
k= exp X Bº x + + + Ek
X + (bxx))
where G. = Σソ= exp(Piy) M0 (
mg)) denotes a mixing measure, i.e., a weighted sum of Dirac measures 8, associated with unknown parameters (Fig, Poy. n3) in R Nd x R x R Here, the matrix B9 plays the role of the matrix E. ww in the score function $1. (X). Furthermore, the Vag
vector Bi corresponds to the vector E,T. w.T. w in the score function 81,N+ (X). Furthermore, the experts h(x, nrf)) correspond to the role of (X) and h(X, nf.)) correspond to the role of fN+J( In our formulation, we consider general parametric forms of the experts h(X, ,n?) and h(X, nf.)), i.e., we do not only constrain these expert functions to be the forms of the simple experts in the linear gating prefix MoE model.
P
Vd.,
Similar to the linear gating prefix MoE model I. the matrices Bg. the biases <g. and the expert parameters 7h are known. Our aim is to estimate the unknown gating parameters pi,, &ōp. and nīj. that correspond to the prompts.
to estimate the unknown parameters (80yr. Sig. nz. ) or, equivalently, the ground-truth mixing measure G*. In particular, we take into account the estimator
Least squares estimation: We will use the least squares method 45
G = arg min
Σ (Y-- fvx-)((x,))
(25)
(B) (8) im1
15

---

where we denote (L) := {G = Σi= exp(5us)o 1 L', (Bus, S1a. mh) E } as the set of all mixing measures with at most L' atoms. In practice, since the true number of true experts L typically unknown, we assume that the number of fitted experts L' is sufficiently large, i.c. L'
Let us recall that our main objective in this appendix is to show that using the linear gating in the prefix MoE model is not sample efficient. To illustrate that point, we consider simple scenario when the expert function takes the form h(X (a, b)) = (a X + b)", for some E N. Additionally, we also design Voronoi loss function as below to facilitate our arguments.
(2, G.) =
Σ exp(Bus) exp((6,) EV
Σ Σ
exp( |A (1 + ||Aay)l"" + |
j=
(26)
where we denote AByy B1s Fi and Any 12 nz.
Now, we are ready to state the result of parameter estimation under the linear gating prefix MoE model in the following theorem:
Theorem A.I. Assume that the experts take the form h(X, (a,b)) b)) (a T X + b)r.ffrr some p E N,
then we achieve the following minimax lower bound of estimating G.:
inf (e) Esc Ef 10)
sup
2))
for any 2 where Ee indicates the expectation taken w.r.f the product measure with fö-
There are two main implications of the result in Theorem A.I
(i) The rates for estimating parameters Pi,, 0j and by are slower than Op(n-1/21 for any r 2 1. This means that they are slower than any polynomial rates, and could be order Or log(n)). we have
Using the same reasoning described after equation
sup ((()) T X 6 (((() X + b3) S -|0 azلl (b 651.
(27)
As a consequence, the rates for estimating experts p((a)) TX + b3) are no better than those for estimating the parameters oz and bj. and could also be as slow as Op(1/ [o]]/)))))).
(ii) The above rates imply that need an exponential number of data (roughly exp(1/e") where is the desired approximation error) to estimate the parameters and experts the linear gating prefix MoE. This fact demonstrates that using the linear gating in the prefix MoE model is not sample efficient from the perspective of the expert estimation problem.
Proof of Theorem A.1 Prior to presenting the main proof of Proposition A.I lct us introduce the following key result:
Lemma A.2. the following holds for any r 2 1:
llfo fc. fi.ll 0, (G,
inf a-+ G (GGG)))
lim
(28)
then we obtain that
sup (e) GED, 10)
inf
E (2n
(29)
Proof of Lemma A.2 A.2
Indeed, from the Gaussian assumption on the noise variables 6,, we obtain
that X, N NTJa. (X.),a²) for all i E [n]. Next, the assumption in equation (28) indicates for and a fixed constant C₁ which we will choose later, we can find a mixing
sufficiently small measure G", Gz (9) such that (G,, G.) = 2s and ||fa; fo.llLa(u) Cjs. From Le Cam's
16

---

lemma 57 as the Voronoi loss function 22, satisfies the weak triangle inequality, we obtain that
inf
sup
1fa G)]
(0)
GG,
exp(-nBx~x (WCC
8
2 e exp(-xllffcl fc. (1
2 exp(xCCnnn
(30)
where the second inequality is due to the fact that
(So; (X) Sa. -Ja. (G.(())) (X))²
KL(N (x), ( (,N,
20
By choosing n- obtain that e exp( - Cine?) n n-1/2 exp(- As consequence, we achieve the desired minimax lower bound in equation (29).
Main proof. We need to prove that the following limit holds true for any 2 1:
Ilfc - fc. = 0.
lim c- (0). (G..)) C2,
inf
(31)
For that purpose, it suffices to build a sequence of mixing measures (Gx)x21 such that both C2 (G,, G+) + 0 and
llf.. - (o. 0,
(G, G.)
as 12 + oc. To this end, we consider the sequence Gn = -/=1 L++1
i=1 exp( (6,
(19 (p.))))))))>> where
exp((3)) exp( = exp(6(%) + 2 and exp(88) exp(9% for any 3 <
< L +
84 = Pi2 = and 87 = Pi((-1) for any 3 < i 1+1;
of ay
-aj and a a 1 for any i < L+1;
b9 bj + n b3 bi and b b for any 3 S S L.1.
As result, the loss function C2.- (Gn, G.) is reduced to 1
1 1 C2, G.) = n-+1 + cxp((%)) nt+1 which indicates indicates that 22, (G, G.) 0 as co.
O(n
(32)
n"
Now, we prove that llfG. - fc. fc.lle.(u)/ (Gn, Gx) + O. For that purpose, let us consider the quantity
N
Qx(X)==
Σ exp X T B x
2 + Σ Tx + P6pr Ioc,
SG. (X))-
P=1
f'=1
For simplicity, let us consider the polynomial degree p 1 as the arguments for other values of can be adapted accordingly. Recall from equation (46) that Q (X) can be decomposed as follows:
Qn(X) Σ
exp(3 exp(((()) j=liEA
Σ
x x xx
exp((3i,) T ()(())) x + b?)
Σ
exp(3 exp((3))) T X)ga (X) -ex(((xxx)) X ))9cs, (x)
j=liEd,
exp(36))) -xx(()) (exp(((1))) Tx)(a;)) x -00))exx(((fT) x TT 1),,((()
exp(3
#=1
SEA
A.
XX)
Bn (X) (X).
17

---

From the definitions of Pi oj and by, we can rewrite A, (X) as follows:
2
An(X)
Σ
exp(361) + n²tI exp(((f1))
X ()()) T x + b9) - ((aa) x bi)]
i=1
2 exp(
+ n=*1 exp(((11)
((b (T)(()))))( bi) (62 bfi
Additionally, it can also be checked that B, (X) 0, and (x) O(n O(n-(r+1)), Therefore, it follows that (X)) C2 G.) + As a consequence, Qn C2. (n, G.) - as n + 00 for almost every X.
Since the term E exp B + c.)) + Σラ= (5p + Pb.y)) is bounded, we deduce that [Sa. (X) - Sa. (X) 22, for almost every X. This result indicates that G.) -
fl. fc.. / C2,
as 72 + oc. Hence, the proof of claim (31 is completed.
B Proof of Theoretical Results
In this appendix, we present rigorous proofs for the theoretical results introduced in Section 4 namely Theorem and
and Theorem 4.3 in that order.
B.1 Proof of Theorem 4.1
For the proof of the theorem, first introduce some notation. Firstly, denote by (O) the set of conditional densities of all mixing measures in (9), that is, (9) == {go (X) ) : E Gr (9)).
Additionally, for each 5 >> 0, the L (p) ball centered around the conditional density gG, (Y (X) and intersected with the set Fz. (O) is defined as
FL, = {g E (O)
In order to measure the size of the above set, Geer et. al. 445 suggest using the following quantity:
l" (1,FF (le(w)) dt 6,
(L.
(33)
82/2218
where Ht (0, (). (((()) stands for the bracketing entropy 45 of Fz, u) under the L (n)-morm, and v max(t, 8}. By using the similar proof argument of Theorem 7.4 and Theorem 9.2 in 45 with notations being adapted to this work, we obtain the following lemma:
Lemma B.1. Take (8) 2 (6 (O, that satisfies /8) is non-increasing function of 8. Then, for some universal constant e and for some sequence (5x) such that nnō² 2 e (8,), we achieve that
P P(1sa. - -9. (L > 26) ceep
nd2²
2
for all 8 2 6n-
We now demonstrate that when the expert functions are Lipschitz continuous. the following bound holds:
HB(E,FL(O),, )())))) S log(1/=).
(34)
for any 0 < < 1/2. Indeed, for any function 9G (I), since the expert functions bounded, we obtain that h(X,, n)) < M for all X where M a bounded constant of the expert functions. Let 7 < E and { [T,,.... # be the Ç-cover under the L norm of the set Fr)(e) where N = construct the brackets of the form (X), (x( for all E [N] follows:
[ is the --covering number of the metric space (F), )) Then,
(x(x) = max ( म. X C,0).
U, (x(x) max ( , X + C.M.
18

---

From the above construction, we can validate that FL)( C C ( and (X)
(,(X) S min }. Therefore, follows that
||U - -L.llzz()) = -(UL-L di)
$ /<<))((x)= -L) 452,
which implies that ||0 - 110,-Lillag6a 2ç. By definition of the bracketing entropy, we deduce that
FL, || (lla(w)) S log N log ((C, (1-,==
(35)
Therefore, we need provide an upper bound for the covering number N. In particular, we denote := {B1, Po) E R-NdxNd x R&d x R (31, So, 7) E } and Ω == {1 E IR (31, (0: E 6}. Since compact set, A and are also compact. Therefore, we can find (-covers Âş and Ωgg for A and Ω2 respectively. We can check that
| & Op(T (0((T((Na1)))) [Pdl $ Op(T ).)).
E'' exp(Sox))(111n) E (L,( we consider other two mixing
For each mixing measure measures:
G
exp exp(Sor)
G
Σ exp 3 0i )6 (T.))))
Here, 77, Dg such that 77 is the closest 77, in that set, while (I, Dc is the closest to (Bi (0:) in that set. From the above formulations, we get that
119c -c
expl ST. X + oo ( + Buj) Buj) |n(X ( m1)
h n;²)
sup
Xex
exp X B X ( -ド - exp X + 0x (T Basr)
L'
exp( X + ax ava (T X) + Boj) 11( X 11}} - 1(x 77j}
sup
xex
Σ -'== exp( X B9 x c Σ プリ exp X + ax (T X) Bog
N
sup xex
|x(X, n1j) 1xx n7,1
sup [x) [|7 - 7lll I1
#= xex
L'CSC.
Here, the second inequality occurs the sofumax weight is bounded by one, and the third inequality follows from the fact that the expert h .X).) :) is Lipschitz function with some Lipschitz constant L (x) > O. Next, let us denote
N Σ
D exp
B
+
exp( BTs X (T BBy)),
y'=l
N
D Σ exp(X
B. X c + Σ
exp(3 X 20 Bay))
/==1
19

---

Then, we have
lla. - 9 10
N
(X B + C)) n + Σ exp( X + (T( x) + ( ,7,))
sup X
D
1 N
(Xp B +c))4(X, 11 + Σ
exp( X + (o0 + m7y))
D
j=1
<
sup XEX
exp(X T B + +,,))((Xx nn!
D
D
1/'
exp(3T X + (T x)+
+ Boj) xxx
exp(31 x + ( X) + Boo))
|In(X. n,)1-
sup
XEX
D
D
(36)
Now, we will bound two terms in the above right hand side. Firstly, since both the input space X' and the parameter space O are bounded, we have that
1 1
DIDD DI
D D
V'
exp(Bi.. X + ( X) +oo)) - -ex((iix X + ( x) + 8ogr)
(B1s - B1y) [ + a x) o(Fii Xi] + (Boz Boo)
1(81 - BBn)" x +
- (o( - 0r X)1 + 1805 - Bosl
[18i; - BIyll |X|| || + |AT| 113i; - B1pll ||||| + |Bo0 - Buy|
F=1
< L' + + + 1)c S.
As a result, we deduce that
- sup Xex
exp(X XBx + +)))( n791 SC.
(37)
Regarding the second term, note that
exp(3T X + ar X) + Bos)
exp(31, x + (T + Bo,)
D
D
exp(Biix X + (T X) + Boo) ( 1 )
+ +
1 exp((11x
x + ao ao (T x) + Boj) exp(exp(B, (B,xppB11xx + ao(TBiyx) + Boo))]
Since both the input space and the parameter space are bounded, we have
exp(3 x + (T x) + Bog) ) 1 5)
1
1 S D 5C
5 [exp(8]]x x + aoTT331 X) + Boj) - exp(3 T
x + oo (T X) + Boj)
x (B B |or| B + 1) SC.
20

---

which yields that
exp( X + ( X) + -Bo))
exp(2 X + (o(T311 + Boy)
sup Xex
-Ih(X, 77y)
D
< Σ
sup 1h(X, 7,)1 j=1 XeX
(38)
From equations (36), inequality, have
(37 and (38).
we obtain that l - 23gl1x S Ç. According to the triangle
96 - | - selle= + 119 - 9Gll SC
By definition of the covering number, we deduce that
(O), 11.=) <
A x [221 < On O( O(n On
(39)
Combine equations (35) and 339. we achieve that
HB (B(2(.FL. (1lat)))))))))))))))))))))))))) S log(1/r).
Let Ç E/2, then we obtain that
Ho(e,Fr(()).11 -1),) & log(1/E).
As result, follows that
= [ 858/213
6
H12 (t, (L, (1, de d&
log(1/t)dt V &. (40) 62/213
Let (((6) = [log( (16 then (8)/ non-increasing function of 6. Furthermore, equa- tion 40 indicates that (5())= 2 (e, In addition, let 6, Vlog(n)
(log(n/n, n,, then we get
that / n 2 (ô,)) for some universal constant c. Finally, by applying Lemma B.1 we achieve the desired conclusion of the theorem.
B.2 Proof of Theorem
Our goal also to demonstrate the following inequality:
inf 119. sc. /l 0.
(41)
GES (6)
For that purpose, divide the proof of the above inequality into local and global parts in the sequel.
Local part: In this part, we demonstrate that
lim c GEG (G,
inf
1l9a ga. 9c.llLata//i(G,GC G.) 0.
(42)
Assume by contrary that the above claim is not true, then there exists sequence of mixing measures G, = Gt (ee in Gz)(O) such that Lin Li (Gn, G.) 0 and
llg.. - gG. -GG.G.lL... (les Lin
(43)
as n oc. Let denote by V (G,). Voronoi cell of G, generated by the j-th components of G.. Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i.e., V;- = Vr. Thus, the Voronoi loss Lin can be represented
Lin = Σ
Σ
(x)) ||A 11? + An
Σ =
Σ
exp
|| || + ||
exp(31.) exp(01,) (44)
21

---

where we denote Ao% := Pi; Bi and An; == n7;- nz.
Since Ln + O, we get that (S m - G ni)) and Eiev, exp( - exp as n - oo for
any E V and j E [Z]. Now, we divide the proof of the local part into three steps as follows:
Step 1 Taylor expansion. In this step, we would like to decompose the quantity
Qx)
exp X X x + + exp(((f,))
X + ( TX)
x) + {byy)
j'=1
[9 9G (X)] (45)
into combination of linearly independent elements using Taylor expansion. In particular, the quantity Qn (X) is decomposed as follows:
exp(36:)
exp((())) x (o(((())) x))x(X:n)) -ex(((fi)) T + (T((33))) x))((x(n)))
j=lie
exp(
exp(((R)) x ao(T(())) x)) - exp(((1,)] x + ( x)) x)) go., (X)
jii,
Σ
exp( exp(3t,)) exp((31,)) X + ao((((11)) x)) ( M(X:n)) - go,, (x)
j=1
EEV
A (X) (X) (X).
(46)
Decomposition of A,(X), Let us denote E(X; B1) =xxx: x x separated into two terms as follows:
X)), then A, can be
An(X): = Σ
Σ
exp(pb)
Σ
Σ
exp(
E(X; M1,
j: 1>
An, (X)
An, (X).
By means of the first-order Taylor expansion, we have
exp(X
glaa E.
gloaah
1x)) Σ Σ
Σ (ABT.)) )a (Ant))
02
(x: 3i,)
(X:nz) + (X)
a!!
Orpas
|a|=1
alai
glual),
Σ
Σ
S Snjan
(X;37)
(Xin)) + R,,1(X))
08
0rpaz
j-V,1=1 |a2==1
where (X) is Taylor remainder such that Rn, (X) - 0 as n + 0o, and
exp(P(i
(
Snjjai.
Σ
(A (Am
al
iEV,
On the other hand, by applying the second-order Taylor expansion, we get that مlaa|
glail E 0.8
An, X)) Σ Σ in which (X) is a Taylor remainder such that ( Cin - 0 n Decomposition of B,, Recall that we have
Snijan
(X:
(X;n)) + RX),(X),
0m
;DVE 1 1g/m Haaa|s2
Bx(x) Σ
Σ
exp
E(X; E(:; )
|
gG.. (X)
HIV,-=lieh
Σ
Σ exp 3m E(X; E(x; go. go. (X)
= Br, (X) Bx,2(X).
22

---

By invoking first-order and second-order Taylor expansions to (x) and (x, it follows that
E
Bx,1(X)= = Σ
J.
(x; 51 11y),,, (X) + Rm,3(X).
dsi
J:11 1=1
e|d E
B,,,(X)= - Σ
Σ Tage
(X; (x; (o. (X) + a
0
where we define
exp(Pii) Agii))
Taj
e!
EVV
Additionally, Rn.3( X) and R are Taylor remainders such that Rn,3(X) Lin 0 and Rn,3(X))Cin + 0 as n + 20.
Collect the above results together, we can represent (x) as
L
Olai E
glazih
Qx(X) Σ
Σ
Snjana
(X;
(X;nf))
0.3
Orpo2
j=1 05 |
@ E
Σ Tnget
( gg., (X)
Rn, (x),
(47)
0
0311/2
where we define Sn.j.Oaxa.0, In.j.0axa Eiey, exp (6 exp( for any j [L].
Step Non-vanishing coefficients. In this step, demonstrate that least one among ratios of the forms Sn.j.nnn/// Lin and Tnj.t/fin Lin goes to zero as n tends to infinity. Indeed, assume by contrary that
Sn.jan.a2
Tag.t
- O,
+ 0, Lin
Lin
for any j E [Z]. S l011. J02), |e] < 2. Then, we
1
Sn.j0aaa
exp( (m)) exp((6;)
Σ
- 0.
(48)
Cin
Cin
iEV;
Now, we consider indices j [L] such that its corresponding Voronoi cell has only one element, i.e. |V>l =
For arbitrary u E [Nd], let a₁ E NddxNd and 02 O, such that a ((ur)) while other entries equal to zero. Then, we have 2 Eiey, (A Lin //L
= Dc. By taking the summation of the previous term with v E [Nd]. we achieve that Zim Σiey, exp(3))))l A AByyll Il - O. Owing to the topological equivalence between norm-l and norm-2, it follows that
[Sn.j.n.aggl/
Σ (xp()))148)) + 0.
(49)
Lin tE,
For arbitrary E [Nd], let a₁ ONaxNd and ag No such that a while other entries equal to zero. Then, we get <i. Eiev, exp(X)) )()) |Sn.j.a. 0 as n + oo. By taking the summation of the previous term with w E [g). we achieve that O, equivalently,
Zi Eiey, exp(X())
Angll [|
Σ
exp(
+ O.
(50)
L
iEV;
23

---

Combine the limits in equations 49 and (50). we obtain that
Σ
exp ( (Al
| + 0,
(51)
Lin iil
as 7 + 0c.
Next, we consider indices j E [L] such that its corresponding Voronoi cell has more than one element, Le. [V]] 1.
(ue) For arbitrary u,v E [Nd], let an E Nd and dg = og such that aj while other entries equal to zero. Then, we have Zi
1 Eiey, exp( (4 )um) =
Sn.joor.og// Lin 0 + 0c. By taking the summation of the previous term with u, U E [Nd], we achieve that
exp exp( + O.
(52)
Lin
iEV;
For arbitrary u E [Nd], let 01 = Oa and 02 E No such that 0i = while other en- 1. tries equal to zero. Then, we get Zim Liev, exp( (Ary) (w)) (10)3
|Sm.j.n // LLn + 0
+ By taking the summation of the previous term with E [g]. we achieve that
(l O.
(53)
Lin iEV;
Putting the limits in equations
449 and 500 have
Σ
exp(3.)IAAPg)) | +
(54)
In j: iilliiv,,,
as n2 + 0O. Taking the summation of three limits in equations 488 and (54). we deduce that 1 = L/n/ + 0 as n oc, which is a contradiction. Thus, at least one among ratios of the forms Sn.j.nn, Lin and Tn.jt//o Lin goes to zero as n tends to infinity.
Step 3 Application of Fatou's lemma. In this step, we show that all the ratios Sm.j.ar.9// Lin and (in.e/Lin& Tn..e/ Lin go to zero as n + 0o, which contradicts to the conclusion in Step 2. In particular, by denoting m. the maximum of the absolute values of those ratios. From the result of Step 2, follows that 1/mn A xc.
Recall from the hypothesis in equation 43) that 19g. - -G. (talt//a + 0 as n + co, which + 0. Therefore, by applying the Fatou's lemma, we get that
indicates that ||9G. 9G. (w)()/////))))))) Lin
0 = lim 199,., 9G. ((((
lim inf 19a, (X) 9G. (X) du(x)= 2 0. n-000
m.
Cin
This result implics that mh Cim [9G., (X) - 9G. (X) + 0 as n
+ oo for p-almost surely X.
Looking at the formulation of (X) in equation 45 since the term
Σ
Ei exp
ko
X A
Σ exp( T X ((f TX) for yr-almost surely X. Let us denote
X) + 16r)) is bounded, we deduce that the term
Q. (X) 17,, Cin
Sn,ja0,,0 -> 0jaa,,
T T My Lin Then, from the decomposition of (x) in
-> (Pj.e
my Cin
with note that at least one among them is non-zero. equation (47). we have
1+10
e E
gloulh 0r
Σ
Σ
Σ
Ojanaaz az
05e1
(X:31,)
(X:
(X;n)),
1=
(1
a 63
(X; (X;Bi))cc (X) 0,
03
24

---

for p-almost surely X. It is worth noting that the term
(X;
Φloz|1 (Xin)) can be
88,
81y"2
explicitly expressed as
When |ai| 0, |02] 0: exp((31,)
T T x))m(X;nf);
0: x (1 + o'((3T,) Tx)))) x)) exp((1T,) X +
When [01l = |,122
o ((31)) ( (x)3)) x))b((xnn)) n1);;
When |on| = 0, |22l = 1: exp( (x + x)) Brybo (X;1));
When |on| = 1, |a2| 1:
0h
x((1) (1+ (1 + ('((1))) (3tp(( ((3)g)) )yy(w) (s:n});
When |ai| = |2| 0:
(1)
X)) -0 ((81)) [( exp((57g) T x +o((()))) X x)))(x;n))
When |ai|
= 0, |02] 2: exp(( x + t((
Orp ( (z)))).
Recall that the expert function h satisfies the condition in Definition 4.2 i.e. the set
ahlh
x [x{(1+ (((it) x Ml + [|))=))) [(m)=2)""(($i))) ((Big) x)]
(x, m3 e [4].
0 1/
S
is linearly independent for almost every X. Therefore, obtain that @j.m. P3 0 for all j E [Z]. 0 < Jaa| + l022, [e] < + [>1- This result turns out to contradict the fact that at least one among them is different from zero. Hence, achieve the inequality in equation (42).
Global part. is worth noting that the inequality (42) suggests that there exists a positive constant e' such that
inf GE9 (GG...,)) 1190 9G, LL 0.
Therefore, is sufficient to prove that
inf GES (I:C 119. - Sc. /l..l////// 0.
(55)
Assume by contrary that the inequality (55) does hold true, then we can find sequence mixing measures C% (e) such that Li (G G.) > E and
199; 9G. [[(2(a) O, C,
lim 12-000
which indicates that ll9cr 9G. (l12(()) 0 as cc. Recall that θ is compact set, therefore, replace the sequence G², by one of its subsequences that converge to mixing measure G' e Gz (2). Since (G',,, G.) > E', we deduce that (G' >'.
Next, by invoking the Fatou's 's lemma, we have that
/liminf Liiminf n-400
9G,, (X) - 9G. (x) du(x))
0 lim 1996, - -9- -9G. 2 --+00
Thus, we get that go (X) gG. (X) for p-almost surely X. From the identifiability property of the non-linear residual gating prefix MoE (cf. the end of this proof), we deduce that G' G.. Consequently, it follows that (I" G.) 0, contradicting the fact that (G (G)) e' >
Hence, the proof is completed.
Identifiability of Non-linear Residual Gating MoE.
We now prove the identifiability of the non-linear residual gating prefix MoE. In particular, we will show that if (X) ga. (X) for almost every X, then it follows that G = G..
25

---

For ease of presentation, let us denote
exp(u)
softmaxc(u)
Σx= exp(x c.) Σ ==1 exp (515)) _ + (TT(T()))) X) -bo))
exp(u")
softmaxc,
Eア= Ex=1expl exp B. + Σ p exp( ) X + (T( (X) +oa))
where
e (X B. X + Gr, (B1r) _ x + az (T x) x) Boy :'' E [M. e [z]]
u' E [x] B x + C (giy)) _x x + ad (T (Bir.) ) x) + Poy :' [M. [4]
Since gol (X) = gG. (X) for almost every X, have
softmaxc (x B,X + (.) 1(X, m/ softmaxc
( (111)) x + (T((()) x) + (0o8) -x(X, (1)
N
softmaxc. (X B:X + c) h(x n softmaxa. (181)) X + (T(((()))) X) + (665 1(( 173)
(56)
As the expert function h satisfies the conditions in Definition 4.2 the set (h(X, n7/)) i E [K]], where ./...., T// are distinct vectors for some k E N, is linearly independent. If L' * I., then there exists
some i [L'] such that NN + ni for any j E [4]. This implies that Σド= softmaxx (BL) X
o (T X) + Bos) h(X, ni) = 0, which is a contradiction. Thus, we must have that I. T/. As result,
softmaxc (31s) X + (T( x) 80s) : j [21]] =
softmaxc. (181)) X + aTo((()) X) +bx))) :j [11].
for almost every X. WLOG, we may assume that softmaxG (1915) T X + (T((())))) X) + Bos) softmaxe. (181)) T x aT(((()))) Tx) x) + (os
(57)
for almost every X for any j E [L]. Since the softmax function invariant to translation, this result indicates that 81 = fi, and Boj Bi, + vo for some un R for any j E [L]. Recall from the universal assumption that Bor- Bot we get that Boz fū for any j E [7.]. Then,
equation (56) can be rewritten as
exp(6o)) exp ((B,)) T X (o())))))) (x)_1)))(()xx)))))
((X,,;)
exp((6g) exp (B7,) x + too(T(11)) x M(X,
( nj), (58)
for almost every X. Next, we denote Pı, Pa, P Pm as partition of the index [Z], where m S T/' such that exp(fnx) = exp(3%r) for any i, E Pj and j E [L]. On the other hand, when i and i' do not belong to the same set Pj, we let exp() # exp(Box). ). Thus, we can reformulate equation (58) as
exp(Por) exp (BB)) X + (T(3 a xx)xxxx)XX,,, m)
E
m
exp(3()) exp (BWW X + a0o(((1)) x) x)x( ()x(xx)))xxxx)))) n)),
iEP,
26

---

for almost every X. Recall that B1s 8i; and Sus = Po: for any i E [L], then the above equation leads to
{m E p2} = fnt :E Py)
for almost every X for any j E [m]. As a consequence,
G T-ΣX
ΣΣ
(pp(5))) = G..
=
j=liEp)
j=1
Hence, we reach the conclusion of this proposition.
C Discussion of related Mixture of Experts works
Recently, the MoE model has been employed to mitigate catastrophic forgetting in continual learning. For example, [58] focused on continual learning in vision-language models by adapting a pre-trained vision-language model to new tasks through learning mixture of specialized adapter modules. [58] introduced an MoE structure onto a frozen CLIP, utilizing a mixture of adapters to modify the MLP block after the MSA layer. In contrast, our work centers general continual learning with pre-trained models, leveraging the inherent MoE architecture of MSA layers. Consequently, our MoE model placement differs from that of [58]. By employing prefix tuning, we demonstrate that is analogous to introducing new prefix experts to scale and adapt these pre-trained MoE models to downstream tasks. Furthermore, while (58) utilizes task-specific routers, our approach employs task-specific prompts that encapsulate both task-specific router and expert parameters.
The parameters cost is usually considered in practical memory-consttainee continual learning sce- narios. Dynamic routing mechanism can be employed for gating-bast based neural networks [8]. To improve the parameter efficiency of the final model, method. Specifically, each head in the MSA layers comprises N MoE models, where N is the length of the input sequence. This allows for dynamic routing mechanism enhance parameter efficiency. For instance, [18] proposed a dynamic routing strategy that adaptively adjusts the number of activated experts based on the input. The computation for any MoE. model's gating is directly correlated with the corresponding row in the attention matrix, which encapsulates the MoE model's ' score functions. For example, selecting the top & experts via Top-K routing the i-th MoE model equivalent to identifying the top k largest values in the i-th row of the attention matrix. To implement [8), we first sort the elements the i-th row from highest to lowest, then find the smallest set of experts whose cumulative probability exceeds the threshold. Consequently, unselected experts remain inactive, reducing the need compute all elements of the value matrix within self-attention.
integrate this mechanism in the proposed
D Training Algorithm of HiDe-Prompt
In this appendix, we outline the detailed algorithm of BiDeePrromp, utilizing the same notation as in Section
Each previously encountered class e E ytt) 1 1 has its instructed and uninstructed representations approximated by Gaussian distributions, denoted as G. and G... respectively.
HiDe-Prompt maintains an expandable pool of task-specific prompts e,, each optimized for a specific task D, using a cross-entropy loss within the WTP objective. To prevent forgetting, previous prompts ei....y E remain frozen. Knowledge transfer across tasks is facilitated by a prompt ensemble (PE) strategy: the current prompt is initialized with the last one e, et-1 and refined using a weighted combination of all past prompts Pt Σ e, + (1 a),,, where a is prpper-parameter. Notably, HiDe-Prompt incorporates contrastive regularization within the WTP objective, pushing features of the new task away from those of past tasks represented by the prototypes of old class distributions Go- (t) Let H4 = =foo [f( .p)) i i= 1,...,N.) be the embedding transformation of Dr and l4c be the mean of g.. The contrastive loss can be written
LcR(p.)= -
exp(h Mc/T)
log(
ENER, exp(h h1/T) Σ Eeeyys exp(h /c/T)) ) (59)
NEH, i=1 x
where 7 is the temperature that is set to 0.8. The overall objective function of WTP for learning new task / defined as
LWTP(W.PM p.) ) LCE(1h, p.) + (ECR(P.)
(60)
27

---

Algorithm HiDeePrompt's training algorithm
Input: Pre-trained transformer backbone Sa. training sets Di,... Dr number of tasks T, number of epochs E, hyyper-paaameee a, 7 and X.
Output: Parameters P1...... Pr, and Initialize e1 and ei
2: =wrt,,.... do
3: for y(t) do
4: Obtain G. from fo and Dr
D Uninstructed Representations
5: i 1 then
6:
Initialize et - et-1
7:
Construct p: = Ei=1 1 e; + aree
8:
else
9:
Construct P: =
10:
for epoch ,,..., do
11:
Optimize p. and y with LwTP in (60)
D Within-Task Prediction Task-Identity Inference b Task- Mask-Addptive Prediction
12:
Optimize with Cru in Eq. 62)
13:
Optimize with CTAP Eq.
14:
for c y)()) do
Obtain Ge from fe, Pt and D. return : w)
Instructed Representations
where À is paramee. Following WTP, HiDe-Prompt performs further refinement step on the output layer parameters & using a separate objective called task adaptive prediction (TAP). TAP addresses potential classifier bias by considering the Gaussian distribution of all classes encountered so far. The final output layer hy can be further optimized for TAP objective,
CTAP(E) = -
- Σ
log(
exp(h (h)(c)
(61)
Σ/=1 Eveyon exp(h (h)lcT)
E1 ( hEH.
where Hi. is constructed by sampling equal number of pseudo representations from Ge for y00 and i =0 1,...,t.
For TII. HiDe-Prompt leverages lightweight auxiliary output layer h R R to map uninstructed representations directly to task identity. This mapping is learned explicitly through cross-entropy loss function,
(h Ever. exp hew (h)le')
Crl(w)= ΣΣ -log( REA.
(62)
where H. constructed by sampling equal number of pseudo representations from G. for (r0 and = 1,..., Please refer to Algorithm I for more details.
E Experimental Details
Datasets. We use commonly-used datasets in the field of continual learning, including (1) Split CIFAR-100 (23): This dataset comprises images from 100 classes. These classes are divided randomly into 10 separate incremental tasks, with each task featuring distinct set of classes. (2) Split IetgeNet-R [23]: This dataset is composed of images from 200 classes. includes challenging examples from the original ImageNet [40] dataset and newly gathered images representing diverse styles. These classes are also randomly divided into 10 distinct incremental tasks. (3) Split CUB-200 [8]: This dataset consists of fine-grained images of 200 different bird species. It is randomly divided into 10 incremental tasks, each comprising unique class subset. (4) 5-Datasets [9: This composite dataset incorporates CIFAR- 1-10 (3). MNIST (24) Fashion-MNIST (55). SVHN BI]. and notMNIST 3 Each of these datasets is treated as a separate incremental task, permitting for the assessment of the effects of significant variations between tasks.
Prompt-Based Approaches. We compare NoRGa against recent prompt-based continual learning approaches: L2P (54), DualPrompt [53], CODA-Prompt [2] S-Prompt [52 and HiDe-Prompt [49
28

---

Table 4: Performance comparison in task-incremental learning setting. Here we present Final Average
Accuracy (FA).
Split CIFAR-I00
Split CUB-200
Method
Sup-21K
iBOT-21K
Sup-21K
iBOT-21K
HiDe-Prompt
97.87 0.31
97.48
0.33
97.37
= 0.08
92.34 0.34
NoRGa tanh
98.55 0.45
98.26 + 0.36
97.86
+ 0.14
92.85 + 0.33
NoRGa sigmoid
98.63 + 0.35
98.15 = 0.29
97.89
= 0.14
92.85 = 0.22
NoRGa GELU
98.41 0.47
98.17 = 0.30
97.76 = 0.10
93.00 = 0.11
Table 5: Performance comparison of different PEFT methods using ViT-B/16 with Sup-21K weights.
Here we present Final Average Accuracy (FA).
Method
Split CIFAR-100
Split CUB-200
HiDe-Prompt
92.61
86.56
HiDe-LoRA
92.71
87.37
HiDe-Adapter
92.73
87.10
NoRGa
94.48
90.90
To ensure fair comparison, replicate these methods using the configurations reported in their respective papers. S-Prompt in the original paper trains a separate prompt and classifier head for each task. At evaluation, it infers domain id as the nearest centroid obtained by applying K- K-Means the training data. To adapt S-Prompt to CIL, we use one common classifier head for all tasks. For NoRGa, we adopt the same configuration HiDe-Prompt, which utilizes Prefix Tuning (26) as its prompt-based methodology. Learnable scalar factors and r are frozen after the first task's training to mitigate catastrophic forgetting. We further optimize NoRGa by selecting the best non-linear activation function a via cross-validation among tanh, sigmoid, and GELU.
Evaluation Metric. We employ three common metrics to measure the performance the methods, including final average accuracy (FA), cumulative average accuracy (CA), and average forgetting measure (FM). Let Si, denote the accuracy the i-th task after learning the t-th task, and A, represent the average accuracy as A, E Sp,t. Upon learning all T tasks, we compute T-1 FA = Ar, CA + Σ A., and FM = T=I =i=1 max (St. SLT). It's worth noting that FA and CA are prioritized over FM. as they inherently encompass both plasticity and forgetting, with FM providing supplementary context [42]
Implementation Details. We train and test on one NVIDIA A100 GPU for baselines and our method. We leverage pre-trained ViT-B/16 model as the backbone. Training employs Adam optimizer (3) = 0.9, 32 = 0.999), a batch size of 128, and a constant learning rate of 0.005 for all methods except CODA-Prompt. CODA-Prompt utilizes cosine decaying learning rate starting 0.001. Additionally, grid search technique was implemented to determine the most appropriate number of epochs for effective training.
F Task-incremental Learning Results
Because HiDe-Prone optimizes prompt parameters specifically for within-tase prediction (WTP), NoRGa inherently aligns with this objective, leading to generally better continual learning perfor- mance. We demonstrate this improvement through experiments in a task-incremental learning setting, where task labels are available during inference (as in Table well, NoRGa shows consistent improvement across all scenarios. Notably, NoRGa with sigmoid activation achieves the highest final average accuracy in both Split CIFAR- -100 and Split CUB-200 with Sup-21K training. Additionally, NoRGa demonstrates its effectiveness even with self sulf-super pre-training, further solidifying its advantage over the original prefix tuning model. Overall, NoRGa variants outperform HiDe-Prompt on both datasets and under both training conditions.
While HiDe-Promph performs
G Comparison to Different Parameter-Effiiies Fine-Tuning Methods
As the advantages of different parameter-efficien fine-tuning (PEFT) methods remain an open question, we briefly describe them through our revealed connection between self-attention and MoE.
29

---

Table 6: Performance comparison of pre-trained model-based continual learning methods using
ViT-B/16 with Sup-2IK weights. Here we present Final Average Accuracy (FA).
Method
Split CIFAR-I00
Split CUB-200
ADAM VPT-D
85.04
85.28
ADAM + SSF
85.27
85.67
ADAM Adapter
87.29
85.84
RanPAC
92.20
90.30
NoRGa
94.48
90.90
Table 7: Ablation study on the effect of learnable a and with Sup-21K weights. Here we present Final Average Accuracy (FA).
Method
Split CIFAR-100
Split CUB-200
HiDe-Promppt
92.61
86.56
Learnable a, Fixed
94.38
90,45
Fixed o, Learnable
94.42
90.48
Fixed Fixed
94.29
90.32
Learnable a, Learnable
94.48
90.90
Prefix tuning introduces additional parameters at the input of MSA layers to adapt the pre-trained model representation, contrasting with Adapter [16). which insert adaptive parameters between layers, often replacing MLP blocks. LoRA [7] approximates weight updates with low-rank matrices and adds them to the backbone weights. Our work shows that the MSA layer in pre-trained model can be seen as a pre-trained MoE architecture. Applying LoRA to the MSA layer refines both the pre-trained experts and their corresponding score functions for downstream tasks. In contrast, prefix tuning expands the pre-trained MoE models by incorporating new experts while preserving the original components, rather than modifying the pre-trained experts like LoRA.
NoRGa emerges as simple, parameter-efficient fine-tuning method and can regarded as distinct implementation of prompts. However, novel perspective the interplay between self-attention, prefix tuning, and mixture of experts enables us to thcoretically substantiate the effectiveness of NoRGa shown in Section
For empirical comparison, we integrate the framework of HiDe- Prompp with different PEFT tech- niques and K weights, evaluating performance on Split CIFAR-106 and Split CUB-200. The results are summarized in Table [5] The table shows that NoRGa consistently outperforms the other PEFT methods on both datasets, suggesting its effectiveness. Nevertheless, further investigation with LoRA and Adapter would be necessary to draw more definitive conclusions.
While exploring alternative PEFT methods might offer improvements in WTP performance, these approaches lack theoretical guarantees and could lead an increased number of parameters. In contrast, our NoRGa method modifies the original score functions of prefix tuning to enhance WTP performance with theoretical rigor. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL due to memory constraints.
H Comparison with Pre-trained Model-based Methods
Previous works have demonstrated that utilizing pre-trained models (PTM) significantly enhances performance for continual learning, often surpassing the performance of Mon-PTMMbassel methods. Moreover, studies have shown that first-task adaptation and simple PEFT-stylc tuning can achieve competitive performance (20] 37
29 with prompt-based methods. For instance, (20) demon- strated that appending a nearest class mean (NCM) classifier to VIT model's feature outputs, can serve as strong baseline. [37 60 enhanced this strategy by adapting the pre-trained model the
60
first task using the three PEFT methods for transformer networks [60] and the FiLM method for improved NCM by incorporating second-order
CNNs 37 Additionally,
and Gram matrices. However, these methods, which fine-tune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features. Our work focuses on continually adapting the backbone, utilizing task-specific prompts to consistently capture emerging
30

---

HiDe-Prompt
NoRGa
0.125
NLL Loss
0.110
0.095
0.080
0.065
20
30
40
spochs
Figure 3: Validation loss on Split CUB-200 throughout the training of the first task.
Table 8: Comparison of training times for HiDe-Prompt and NoRGa. All experiments were conducted on single NVIDIA A100 GPU.
Method
Split CIFAR-100
Split ImageNet-R
Split CUB-200
5-Dalusets
HiDe-Prompl
2.80h
2.67h
1.04h
24.06h
NoRGa
2.85h
2.70h
1.10h
24.23h
tasks' characteristics, and proposing a novel method to enhance the CL performance of previous prompting methods.
To validate our approach, we compare against state-of-the-art PTM-based continual learning methods, including ADAM [60] and RanPAC [29]. using Split CIFAR- 100 and Split CUB-200 datasets. The results are summarized in Table In comparison to other PTM-based continual learning methods, NoRGa demonstrates competitive performance across both evaluated datasets. For instance, on Split CIFAR-I00, NoRGa achieves an FA 94.48%, exceeding the next best method by over 2%. Similarly, on Split CUB-200, NoRGa delivers strong results relative to other baselines. These improvements highlight the effectiveness of our method in mitigating catastrophic forgetting and preserving knowledge retention across multiple tasks.
Efficiency Tests
We compare the validation loss NoRGa and Promppt throughout the first task on Split CUB-200, as illustrated in Figure The results demonstrate that NoRGa consistently outperforms HiDe-Prompt throughout the training process. This empirical evidence supports the theoretical advantages of NoRGa over HiDe-Prompt.
Effect of Learnable Hyperparameters
As described Section 4.1 and Appendix in our framework, a and T are learnable hyperparameters and optimized through backpropagation during the first task, eliminating the need for manual tuning. Additionally, our theoretical analysis of NoRGa's statistical efficiency in Section 4.2 holds for any values of and r, demonstrating the theoretical robustness. To further investigate, both fixed and learnable settings for these hyperparameters. For the fixed case, we set their values to 1. We report FA on Split CUB-200 and Split CIFAR-100 with Sup-21K weights. The results summarized in Table 7 Although performance slightly decreased with fixed hyperparameters, it still outperforms HiDe-Prompt, indicating our method's empirical robustness.
evaluated
K Training Times
We utilize a single A100 GPU for experiments. The training times are summarized in Table g While NoRGa exhibits slightly longer training times compared Hilee-Prompt, it consistently achieves significantly better performance. This demonstrates the effectiveness of NoRGa while maintaining competitive training efficiency.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are reflected accurately in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have already discussed the limitations of our work in the conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
The full set of assumptions and proofs is given in Section 4.2 and the Appen- dices.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental details for reproducibility are specified in Section and Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 ltaiss.cc/uullicguideee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experimental details are specified in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars and relevant information in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
34

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided sufficient information on the computer resources in Ap- pendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and followed all the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the theoretical nature of the paper, we do not think there are any positive negative societal impacts of the work performed. Experiments are conducted only for empirically justifying the theoretical results.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release any data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The data and models used in the paper are properly credited. See Section 5 and Appendix for further details.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "4d568My1e7", "reviewer": "Reviewer_cCSa", "summary": "The topic of this paper is about the prompt-based continual learning. The authors give a theoretical analysis on these prompt-based continual learning methods, and utilize a Mixture-of-Expert (MoE) architecture characterized by linear experts and quadratic gating score functions. They develop a gating mechanism Non-linear Residual Gates (NoRGa) for MoE-based continual learning. The proposed method has been evaluated on several benchmarks.", "questions": "My major concerns are included in the above weaknesses.", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow.\n+ It is interesting to theoretically analyze the effectiveness of prompt-based continual learning.\n+ Continual learning via MoE architecture is worth exploring.", "weaknesses": "-\tComparison to different Parameter-Effiecient-FineTuning methods [b] (e.g. adapter) is needed. This paper mainly focuses on the theoretical analysis of prompt-based continual learning methods.  Prompt-based continual learning belongs to PEFT methods and also add new parameters for new tasks. Can the authors further analyze the advantages of prompt-based methods theroretically?\n-\tThe parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method?\n-\tImportant MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted.\n-\tPerformance improvement on some datasets is limited. For example, FA metric (75.06%->75.40%) on Split ImageNet-R, CA metric (95.02%->95.11%) on Split-CIFAR-100 in Table 1. What are the running times of each experiment? Can the authors provide the performance variance of experimental results (e.g. the accuracy at the last incremental learning session)?\n\n[a] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR2024\n\n[b] Continual Learning with Pre-Trained Models: A Survey, IJCAI2024\n\n[c] Harder Tasks Need More Experts: Dynamic Routing in MoE Models, arxiv2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n__Q1: Comparison to Different Parameter-Efficient Fine-Tuning Methods and Theoretical Analysis__\n\nA1: Thank you for your valuable suggestion. As the advantages of different PEFT methods remain an open question, we briefly describe them through our revealed connection between self-attention and MoE. Prefix tuning introduces additional parameters at the input of MSA layers to adapt the pretrained model representation, contrasting with adapters, which insert adaptive parameters between layers, often replacing MLP blocks. LoRA approximates weight updates with low-rank matrices and adds them to the backbone weights. Our work shows that the MSA layer in a pretrained model can be seen as a pretrained MoE architecture. Applying LoRA to the MSA layer refines both the pretrained experts and their corresponding score functions for downstream tasks. In contrast, **prefix tuning expands the pretrained MoE models by incorporating new experts while preserving the original components, rather than modifying the pretrained experts** like LoRA. \n\nFor empirical comparison, we used the framework of HiDe-Prompt with different PEFT techniques and Sup-21K weights, evaluating performance using FA on Split CIFAR-100 and Split CUB-200. The results are summarized below:\n\n|Method| Split CIFAR-100| Split CUB-200|\n|--------------|-----------------|---------------|\n|HiDe-Prompt|92.61|86.56|\n|HiDe-LoRA|92.71|87.37|\n|HiDe-Adapter|92.73|87.10|\n|NoRGa|__94.48__|__90.90__|\n\nThe table shows that NoRGa consistently outperforms the other PEFT methods on both datasets, suggesting its effectiveness. However, further investigation with LoRA and adapters would be necessary to draw more definitive conclusions. We will add this discussion to the final version.\n\n__Q2: Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method?__\n\nA2: Thank you for pointing out these excellent related works. We will add the following discussion to the final version: \n\nEach head in the MSA layers comprises $N$ MoE models, where $N$ is the length of the input sequence. This allows for a dynamic routing mechanism to enhance parameter efficiency. For instance, [c] proposed a dynamic routing strategy that adaptively adjusts the number of activated experts based on the input. The computation for any MoE model’s gating is directly correlated with the corresponding row in the attention matrix, which encapsulates the MoE model’s score functions. For example, selecting the top $k$ experts via Top-K routing in the $i$-th MoE model is equivalent to identifying the top $k$ largest values in the $i$-th row of the attention matrix. To implement [c], we first sort the elements in the $i$-th row from highest to lowest, then find the smallest set of experts whose cumulative probability exceeds the threshold. Consequently, unselected experts remain inactive, reducing the need to compute all elements of the value matrix within self-attention.\n\n__Q3: Important MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted__\n\nA3: Thank you for providing these excellent related works. We will add the following discussion to the final version:\n\nRecently, the MoE model has been employed to mitigate catastrophic forgetting in continual learning (CL). For example, [a] focused on continual learning in vision-language models by adapting a pretrained vision-language model to new tasks through learning a mixture of specialized adapter modules. [a] introduced an MoE structure onto a frozen CLIP, utilizing a mixture of adapters to modify the MLP block after the MSA layer. In contrast, our work centers on general continual learning with pretrained models, leveraging the inherent MoE architecture of MSA layers. Consequently, our MoE model placement differs from that of [a]. By employing prefix tuning, we demonstrate that it is analogous to introducing new prefix experts to scale and adapt these pretrained MoE models to downstream tasks. Furthermore, while [a] utilizes task-specific routers, our approach employs task-specific prompts that encapsulate both task-specific router and expert parameters.\n\n__Q4: Performance improvement on some datasets is limited.__\n\nA4: While performance gains on certain metrics may be modest for some datasets, our method consistently outperforms the baseline, HiDe-Prompt, the current state-of-the-art in prompt-based continual learning, in terms of either FA or CA. For example, on the Split Imagenet-R dataset with Sup-21K weights, the improvement in FA is small (75.06%->75.40%), but the CA enhancement is significant (76.60%->79.52%). __This trend is consistent across various datasets and pretrained settings, underscoring our method's effectiveness and robustness.__\n\n__Q5: Running Times and Performance Variance__\n\nA5: We utilize a single A100 GPU for all experiments. The training times are summarized below:\n\n| Method| Split CIFAR-100| Split ImageNet-R| Split CUB-200| 5-Datasets|\n|-------------|-----------------|------------------|---------------|------------|\n|HiDe-Prompt|2.80h|2.67h|1.04h|24.06h|\n|NoRGa|2.85h|2.70h|1.10h|24.23h|\n\nEach experiment was conducted 3 times. While NoRGa exhibits slightly longer training times compared to HiDe-Prompt, it consistently achieves significantly better performance as indicated in Table 1. This demonstrates the effectiveness of NoRGa while maintaining competitive training efficiency. We will add the above discussion in the final version. Regarding performance variance, we have already presented the standard deviation of the results in the main results, as displayed in Table 1 of the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtYeYjyYK7", "reviewer": "Reviewer_dPHR", "summary": "This paper introduces an extension to prefix tuning, by introducing non-linear residual gating - a simple extension over existing prefix tuning. This non-linear residual gating is supported with theoretical efficiency guarantees (polynomial versus exponential) to better estimate the optimal parameters. When applied to the problem of rehearsal-free continual learning with pretrained models (over four standard benchmarks) on top of the HiDe-Prompt method, the suggested gating provides consistent and partly significant improvements.", "questions": "All relevant questions are incorporated in the previous section.\n\nI am currently leaning towards rejection - not because of the proposed method and the theoretical support (both of which I believe are sensible and meaningful), but rather its placement and consequent evaluation as a continual learning method. Neither in the methodological nature NOR the actually conducted proofs do the author account for the continual nature of the problem. \nInstead, I believe that this paper is much better suited as a contribution in the domain of parameter-efficient model finetuning. However, this in turn requires different benchmark evaluations.\n\nMoreover, I am uncertain about the significance of the report results (see above).\n\nTogether, I would be happy to raise my score if the authors can address these particular concerns (as well as those listed above).", "limitations": "The authors do not discuss limitations or societal impact explicitly.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* To the best of my knowledge in both the Continual Learning and the parameter-efficient finetuning domain, the non-linear residual gating prefix MoE is a novel extension over standard prefix tuning.\n\n* This paper is generally well written, and except for the proofs, quite easy to follow.\n\n* Assuming the regression estimator to be a fair assumption, the statistical sample efficiency improvements of the proposed NoRGa setup versus normal gating is significant (polynomial versus exponential).\n\n* The explicit improvements over HiDE-Prompt are consistent, and often significant.", "weaknesses": "My biggest issue with this work is its placement as a Continual Learning contribution, and evaluating it as such.\n\n__[1]__ Firstly, the proposed approach, whilel helping with WTP, is simply an extension on top of prefix tuning - and none of the contributions connect to the continual distribution shift nature. The improved performance is much more likely tied to an improvement in the underlying PEFT approach; which in itself is simply sufficient on standard benchmarks (see Zhou et al. 2023). This is further supported by the fact that the authors freeze both alpha and tau after the first task (see supplementary D). Moreover, on such small benchmarks, introducing two additional tunable hyperparameters does incorporate additional degrees of freedom to overfit to these benchmarks, and makes direct comparison to existing methods difficult.\n\nConsequently, the question needs to be answered: How robust is NoRGa with respect to the additional introduced alpha and temperature? These are strictly two hyperparameters more than HiDe-Prompt. As such, how much more compute went into hyperparameter optimization for NoRGa w.r.t. Hide? And given the same search budget, how does HiDe-Prompt compare relatively? This is the most crucial aspect, as the other method comparisons are somewhat redundant, since HiDe-Prompt introduces orthogonal classifier head alignment which can be applied to all other methods as well.\n\n> It would be great if the authors could address both the placement as a continual learning contribution (as opposed to a parameter-efficient finetuning paper, which would need to be evaluated on other respective benchmark tasks), and the comparability to HiDe-Prompt.\n\n__[2]__ This paper also only tackles prompt-based rehearsal-free continual learning, but misses discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning, such as: [1] Jason et al. 2022, “A simple baseline that questions the use of pretrained models for continual learning”, [2] Panos et al 2023, “First session adaptation: A strong replay-free baseline for class-incremental learning”, [3] Zhou et al. 2023, “Revisiting class-incremental learning with pretrained models: Generalizability and adaptivity are all you need”, [4] McDonnell et al. 2023: “RanPAC: Random projections and pretrained models for continual learning”. \n\n> Given that these works show that existing prompt-style CL objectives are matched by simply training on the first task, and how performance is matchable with simple parameter-efficient finetuning without explicitly accounting for the continual nature of the problem, how should the insights in this paper be rated w.r.t. These works?\n\n__[3]__ A vast part of this paper deals with the sample in-efficiency of standard prefix tuning mechanisms. However, the authors do not provide any efficiency tests (e.g. how does performance vary as a function of training iterations / samples seen), and only provide slightly improved performances on a few smaller-scale benchmarks. This is somewhat unfortunate, as it would be great to see how the efficiency estimates (e.g. Eq. 16) connect to practical convergence behaviour (e.g. L288 “polynomial number of data versus exponential number of data”).\n\n__[4]__ Both statistical efficiency proofs also do not take into account the continual nature of the problem, but rather assume a single generative model and i.i.d. Samples from said model, which are presented to the learner. This fully disconnects from the actual continual nature and sequential distribution shifts expected in such problems. Similarly, I’m having general troubles to connect the assumptions made throughout the proofs to the practical nature of the problem. E.g. why is Eq. 13 a reasonable assumption for the CL problem? Is convergence behaviour of a LS regression estimator suitably connected to practical convergence behaviour? If so, why? If not, why not? Similarly, why would I want to encourage algebraic independence - what would this give within the actual continual learning scenario?\n\n---\n\n__Smaller issues:__\n\n* I find it somewhat difficult to follow the proofs in 4.2 and Appendix A. Providing a proof-sketch as an overview would significantly help readability.\n* It is a bit problematic to have the entire proof of convergence, without even a proof sketch, in the supplementary-  as it motivates the entire section 4 and the non-linear residual gating. It would be great if the authors could provide at the very least a proof sketch in the main paper.\n* L139/140 is fairly hard to parse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response and summarize the corresponding revisions in the final version.\n\n__Q1: The placement as a continual learning contribution__\n\nA1: Our contributions encompass the introduction of a novel connection between self-attention, prefix tuning, and MoE. __This offers a fresh perspective to the design of previous prompt-based continual learning methods__. For example, DualPrompt uses two complementary prompts during training: a general prompt (G-Prompt) shared across tasks and a task-specific expert prompt (E-Prompt) per task. As illustrated in our paper, each head in the MSA layer comprises multiple MoE models. We hypothesize that the G-Prompt aims to expand the set of pretrained experts in the MSA layer, which capture generalizable knowledge, with new experts encoded within it. However, unlike pretrained experts, the experts in the G-Prompt are learnable across tasks, potentially leading to catastrophic forgetting. Conversely, E-Prompt encodes task-specific experts capturing each task's knowledge.\n\nFurthermore, we agree that NoRGa can have a broader impact to other domain, such as parameter-efficient model finetuning. However, our work builds on HiDe-Prompt, which uses task-specific prompts, i.e. each task has its own set of experts and score functions. NoRGa modifies the original score functions of prefix tuning, enhance WTP performance with theoretical guarantee. We then use the theory that improved WTP performance is sufficent and necessary condition for improved CL performance. This theory is also discussed in previous continual learning works and the original paper of HiDe-Prompt. This theory is critical and connects our contribution to continual learning. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in continual learning because of the memory constraint.\n\n__Q2: The introduction of 2 hyperparameters $\\alpha$ and $\\tau$__\n\nA2: In our framework, $\\alpha$ and $\\tau$ are learnable hyperparameters optimized through backpropagation by the objective of the first task, eliminating the need for manual tuning. __Moreover, our theory on NoRGa's statistical efficiency holds for any values of $\\alpha$ and $\\tau$, demonstrating the theoretical robustness__. We also experimented with fixed and learnable settings for $\\alpha$ and $\\tau$. For fixed hyperparameters, we set their values to 1. We report FA on Split CUB-200 and Split CIFAR-100 with Sup-21K weights. The results are summarized below:\n\n|Method|Split CIFAR-100|Split CUB-200|\n|----------------------------|-----------------|---------------|\n|HiDe-Prompt|92.61|86.56|\n|Learnable $\\alpha$, $\\tau = 1$|94.38|90.45|\n|$\\alpha = 1$, Learnable $\\tau$|94.42|90.48|\n|$\\alpha = 1$, $\\tau = 1$|94.29|90.32|\n|NoRGa|__94.48__|__90.90__|\n\nAlthough performance slightly decreased with fixed hyperparameters, it still outperforms HiDe-Prompt, indicating our method's empirical robustness. We will add this discussion in the final version.\n\n__Q3: Discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning__\n\nA3: Thank you for highlighting these excellent related works. We will add the following discussion to the final version:\n\nPrevious works have shown that first-task adaptation and simple PEFT-style tuning can achieve competitive performance [1,2,3,4] with prompt-based methods. For instance, [1] demonstrated that appending a nearest class mean (NCM) classifier to a ViT model's feature outputs, can serve as a strong baseline. [2, 3] enhanced this strategy by adapting the pretrained model to the first task using the three PEFT methods for transformer networks [3] and the FiLM method for CNNs [2]. Additionally, [4] improved NCM by incorporating second-order statistics—covariance and Gram matrices. However, these methods, which finetune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features. Our work focuses on continually adapting the backbone, utilizing task-specific prompts to consistently capture emerging tasks' characteristics, and proposing a novel method to enhance the CL performance of previous prompting methods.\n\n__Q4: Efficiency tests__\n\nA4: Thank you for your suggestion. We have added the graph to compare Validation loss of NoRGa and HiDe-Prompt throughout the first task in the attached pdf of General Response.\n\n__Q5: Statistical efficiency proofs__\n\nA5: Thanks for your questions. We use the HiDe-Prompt framework with task-specific prompts, where each task has its own experts and score functions optimized with only data from that task. Thus, it is reasonable to assume that samples within a task are generated from a single model and are i.i.d.\n\nIn Section 3.2, we show that the output attention head with prefix tuning can be expressed as a linear gating prefix MoE model. Recent works [a, b] have shown that the performance of MoE models can be improved by using a more sample efficient gating function in terms of parameter and expert estimation. Motivated by this, we propose using the non-linear residual gating prefix MoE in Section 4 and conduct the convergence analysis to justify its sample efficiency.\n\nNext, Eq.(13) is a common regression framework to study the sample efficiency of the gating function in MoE models [c], rather than an assumption. Finally, the algebraic independence condition characterizes which experts are compatible with the non-linear residual gating. It indicates that under the non-linear residual gating MoE, experts would have estimation rates of polynomial orders rather than of order $1/\\log^{\\tau}(n)$ when using the linear gating in Appendix A\n\n[a] Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?, ICML 2024\n\n[b] A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts, ICML 2024\n\n[c] On Least Square Estimation in Softmax Gating Mixture of Experts, ICML 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0O3A3Bh3YA", "reviewer": "Reviewer_MKaE", "summary": "The paper explores the theoretical underpinnings and practical implications of prompt-based methods in continual learning, aiming to enhance our understanding and optimize their effectiveness. It introduces a novel perspective by connecting prefix tuning with mixture of experts models, revealing insights into how self-attention mechanisms encode specialized architectures. The proposed Non-linear Residual Gates (NoRGa) further advances this by improving within-task prediction accuracy and overall continual learning performance while maintaining parameter efficiency.", "questions": "1. Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals?\n2. Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts.\n3. Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?", "limitations": "The paper's reliance on dense theoretical explanations without adequate visual aids may limit its accessibility. Visual representations could significantly enhance comprehension of complex concepts like the relationship between prompts and NoRGa, potentially broadening the paper's audience and impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel theoretical framework connecting self-attention mechanisms to mixture of experts models, significantly advancing the understanding of prompt-based approaches in continual learning.\n2. The theoretical insights are well-supported and complemented by empirical experiments across diverse benchmarks, demonstrating robustness and reliability.\n3. The concepts, although complex, are explained with clarity, aided by concrete examples and theoretical justifications.\n4.  Addressing the gap in theoretical understanding, the paper proposes a practical enhancement (NoRGa) that promises substantial improvements in model adaptation and efficiency.", "weaknesses": "1. The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework. \n2. While NoRGa is presented as a novel gating mechanism, its explicit connection to prompts could be elaborated further. Is it reasonable to interpret NoRGa's gating mechanism as a variant or extension of prompts in a broader sense?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n__Q1: The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework.__\n\nA1: Thank you for your valuable suggestion. As detailed in Section 2, within our framework, a prompt is a set of learnable parameters denoted by $\\mathbf{P} = [ \\mathbf{P}^K, \\mathbf{P}^V ] \\in \\mathbb{R}^{L_p \\times d}$. These parameters are utilized to fine-tune the multi-head self-attention (MSA) layer of the pretrained model, where $L_p$ is the prompt length and $d$ is the embedding dimension. Notably, the MSA layer incorporates a specialized architecture comprising multiple mixture of experts (MoE) models. Our approach leverages prompt parameters to refine these models by introducing new prefix experts. Specifically, $\\mathbf{P}^V \\in \\mathbb{R}^{\\frac{L_p}{2} \\times d}$ encodes parameters for new prefix experts appended to the pretrained MoE models. Correspondingly, $\\mathbf{P}^K\\in \\mathbb{R}^{\\frac{L_p}{2} \\times d}$ encodes parameters for the associated score functions of these new experts within the MoE models in the MSA layer. We adopt task-specific prompts for each task, ensuring that every task has its own set of experts and score functions. During inference, the task identity is inferred to select the appropriate experts and score functions. We will add the above discussion in the final version.\n\n__Q2: Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals?__\n\nA2: NoRGa's gating mechanism can be regarded as a distinct implementation of prompts. As demonstrated in Section 3, the MSA layer in pretrained models can be regarded as a specialized architecture comprising multiple MoE models. Prefix tuning finetunes these MoE models by introducing new prefix experts, utilizing prompts to encode the parameters of the new experts' components. The score functions for newly introduced experts via prefix tuning are linear functions of the input, resulting in suboptimal sample efficiency for parameter estimation as detailed in Appendix A. To mitigate this statistical limitation, NoRGa refines the score functions associated with the original prefix tuning's new experts by incorporating non-linear activation and residual connections, __substantially enhancing statistical efficiency (polynomial versus exponential) with theoretical guarantees__. We will add the above discussion in the final version.\n\n__Q3: Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts__\n\nA3: Thanks for your question. Upon careful examination, we found minimal redundancy in the theoretical explanations and definitions presented throughout the paper. Section 2 introduces the foundational concepts of MSA, prefix tuning, and mixture of experts (MoE). Building upon this foundation, Section 3 elucidates the connections among self-attention, prefix tuning, and MoE. Based on these insights, we propose a novel method termed NoRGa to enhance statistical efficiency. Our theoretical analysis considers a regression framework to demonstrate that the non-linear residual gating is more sample efficient than the linear gating in terms of estimating experts. The algebraic independence condition is employed to characterize compatible experts for the non-linear residual gating. Finally, we design a loss function based on Voronoi cells for the convergence analysis of expert estimation in MoE models. Our results indicate that under non-linear residual gating, MoE experts exhibit polynomial-order estimation rates, outperforming the $1/\\log^{\\tau}(n)$ rate observed in the original prefix tuning design (as detailed in Appendix A).\n\n\n__Q4: Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?__ \n\nA4: Thank you for your valuable suggestion. You can refer to **Q1** in General Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BPhAeYRRMf", "reviewer": "Reviewer_boEM", "summary": "The paper titled \"Mixture of Experts Meets Prompt-Based Continual Learning\" explores the integration of prompt-based continual learning methods with mixture of experts (MoE) architectures. The paper proposes a novel gating mechanism called Non-linear Residual Gates (NoRGa) to enhance the performance of prompt-based continual learning by leveraging theoretical insights and empirical evidence.", "questions": "Which dataset is the ViT-B/16 pre-trained on.", "limitations": "This paper has already discussed the limitations of our work in the conclusion section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper offers a novel connection between prompt-based-tuning and mixture-of-experts, providing a fresh perspective on prompt-based continual learning approaches.\n\n2. Introduction of NoRGa, which integrates non-linear activation and residual connections to enhance continual learning performance while maintaining parameter efficiency.", "weaknesses": "1. Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method.\n\n2. It may be better to use a additional graph to represent the final method NoRGa of the paper.\n\n3. It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n__Q1: Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method.__\n\nA1: Thank you for your valuable suggestion. As our work focuses on the continual learning of pretrained models, we have limited our comparison to pretrained model-based (PTM-based) continual learning methods. Previous works have also demonstrated that utilizing pretrained models has shown great promise and performance for continual learning, surpassing the performance upper bound of non-PTM-based methods. Specifically, we compared our method (NoRGa) against ADAM [1], RanPAC [2], and ESN [3], using ViT-B/16 with pretrained Sup-21K weights. Performance was evaluated using final average accuracy (FA) on Split CIFAR-100 and Split CUB-200. The results can be summarized as follows:\n\n| Method         | Split CIFAR-100 | Split CUB-200 |\n|----------------|-----------------|------------------|\n| ADAM + VPT-D [1] | 85.04 | 85.28 |\n| ADAM + SSF [1] | 85.27  | 85.67 |\n| ADAM + Adapter [1] | 87.29  | 85.84 |\n| RanPAC [2] | 92.20 | 90.30 |\n| ESN [3] | 86.34           | N/A |\n| NoRGa (ours) | __94.48__ | __90.90__ |\n\nAs shown, NoRGa exhibits competitive performance on both datasets. For example, on Split CIFAR-100, NoRGa achieves an FA of 94.48%, surpassing the next best method by over 2%. On Split CUB-200, NoRGa also demonstrates competitive results compared to other baselines. This improvement underscores the effectiveness of our proposed method in mitigating catastrophic forgetting and preserving knowledge across multiple tasks. A more detailed comparison will be included in the final version.\n\n__Q2: It may be better to use a additional graph to represent the final method NoRGa of the paper.__\n\nA2: Thank you for your valuable suggestion. You can refer to **Q1** in General Response.\n\n__Q3: It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.__\n\nA3: Thank you for your valuable suggestion. We compare the differences between L2P, DualPrompt, HiDe-Prompt, and NoRGa (ours) as follows:\n-  __L2P:__ Utilizes a shared prompt pool for all tasks. Each prompt is associated with a learnable prompt key. We then employ the query feature $q(\\boldsymbol{x})$ to retrieve the top K most similar prompts using cosine distance. Consequently, the most relevant keys and corresponding prompts are explicitly assigned to instances based on the query feature.\n-  __DualPrompt:__ Enhances L2P by using two complementary prompts during training: a general prompt (G-Prompt) and a task-specific expert prompt (E-Prompt) per task. The set of E-Prompts acts as an expanding pool of task-specific knowledge, similar to the L2P prompt pool, but with the key difference of growing incrementally with each new task. DualPrompt employs the same prompt selection mechanism as L2P for the E-Prompts. In contrast, the G-Prompt is shared among all tasks, requiring no prompt selection.\n-  __HiDe-Prompt:__ A recent SOTA prompt-based method that employs only task-specific E-Prompts. Prompts for each task are trained with the task's objective and a contrastive regularization that tries to push features of new tasks away from prototypes of old ones. Unlike L2P and DualPrompt, prompt selection is achieved by an additional MLP head placed atop the pre-trained ViT, which employs $q(\\boldsymbol{x})$ to determine the suitable prompt.\n-  __NoRGa:__ Utilizes the same framework as HiDe-Prompt. Specifically, NoRGa only uses task-specific prompts (E-Prompts) with an additional MLP head for prompt selection. Recognizing that MSA layers embody a mixture of experts (MoE) architecture and applying prefix tuning is the process of introducing new experts into these models, NoRGa modifies the gating mechanism of prefix tuning, addresses statistical limitations of prefix tuning, and enhances continual learning performance.\n\nWe will add the above discussion in the final version.\n\n__Q4: Which dataset is the ViT-B/16 pre-trained on?__\n\nA4: Thanks for your question. The ViT-B/16 model is pre-trained on the ImageNet dataset [4] (ImageNet-1K and ImageNet-21K). We utilize several publicly available checkpoints to demonstrate the effectiveness and robustness of our proposed method under varying pretraining settings:\n- Sup-21K: Supervised pretraining on ImageNet-21K\n- iBOT-21K: Self-supervised pretraining on ImageNet-21K\n- iBOT-1K, DINO-1K, MoCo-1K: Self-supervised pretraining on ImageNet-1K.\n\n\n[1] Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need, arxiv 2023.\n\n[2] Ranpac: Random projections and pre-trained models for continual learning, NeurIPS 2023.\n\n[3] Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference, AAAI 2023.\n\n[4] ImageNet: A large-scale hierarchical image database, CVPR 2009", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response:**

We thank all reviewers for their valuable feedback and suggestions, which have significantly contributed to the enhancement of our manuscript. We are encouraged by the endorsements that:
1. Our reveal relationship between self-attention, prefix tuning and mixture of experts is novel, significantly advancing the understanding of prompt-based approaches in continual learning (reviewer MKaE, boEM, cCSa).
2. NoRGa, which integrates non-linear activation and residual connections, can enhance continual learning performance while maintaining parameter efficiency (reviewer boEM)
3. The statistical sample efficiency improvements of NoRGa versus normal gating is significant, promises substantial gains in model adaptation and efficiency (reviewer dPHR, MKaE)
4. The theoretical insights are well-supported and complemented by empirical experiments across diverse benchmarks, demonstrating robustness and reliability (reviewer MKaE). The explicit improvements over HiDE-Prompt are consistent, and often significant (reviewer dPHR).

We address a common comment from Reviewers: 

**Q1: Additional graphs/visual aids to represent NoRGa (Reviewer MKaE, boEM)**

**Answer**: We have added some visualizations in the attached PDF. In this file, you can find visual aids illustrating the relationships between self-attention, prefix tuning, and MoE, as well as the NoRGa implementation. We plan to include these in the final manuscript, and we hope they will enhance the paper's accessibility and readability.

We have addressed all the weaknesses and questions raised by all reviewers in the respective rebuttals. We believe that most of this input is valid and greatly improves our paper. We hope that our responses to the reviewer's questions and the additional experiments will help in the review process. Please let us know if you have any further questions.

Regards,

Authors

Meta-review: Prior to rebuttal, the reviewers raised concerns regarding:

1. lack of good connection to the continual nature of CL, specifically, addressing distribution shifts in the proposed approach and the proposed theoretical grounding
2. comparison to PEFT-based CL methods
3. comparison to MoE based CL methods
4. parameter efficiency
5. limited improvement in some of the experiments
6. missing baselines
7. clarity.

the authors seem to have addressed all concerns in their rebuttal, all reviewers either increased or kept the acceptance scores.
In light of the above AC recommends acceptance and urges the authors to incorporate all the reviewer comments and subsequent discussions in the final verios of the paper.
Decision: Accept (poster)