Mixture of Experts Meets Prompt-Based Continual Learning

Authors: minh  le
an  nguyen
huy  nguyen
trang  nguyen
trang  pham
linh  van ngo
nhat  ho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:59:45

Abstract: Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms.

Main paper: 
 introduction: 
 Humans possess a remarkable ability to learn continuously by integrating new skills and knowledge while retaining past experiences. However, current AI models often fail to retain this ability. Unlike humans, they often suffer from catastrophic forgetting [28, 30, 32, 38] , a phenomenon where they struggle to retain knowledge from previous tasks while learning new ones. Inspired by human learning, Continual Learning [2, 28, 43, 1, 12] is an ongoing field that aims to train a model across a sequence of tasks while mitigating this challenge. Traditional continual learning methods often rely on storing past data for fine-tuning, which can raise concerns about memory usage and privacy [5, 39, 51] . To address these limitations, prompt-based approaches have emerged as a promising alternative within rehearsal-free continual learning. By attaching prompts -small sets of learnable parameters -to a frozen pre-trained model, these approaches enable efficient adaptation to new tasks with minimal modifications to the underlying model [56, 26, 61] . The effectiveness of prompt-based methods has been demonstrated by several recent works achieving state-of-the-art performance on various continual learning benchmarks [49, 53, 54] .While prompt-based methods have demonstrably achieved impressive results, their emphasis largely lies on prompt utility, leaving a gap in our theoretical comprehension of their effectiveness. This absence of a theoretical foundation hinders our ability to further refine and optimize these methods. In this work, we offer a new perspective by focusing on prefix tuning [26] and its connection to mixture of experts models [19, 15, 13, 11] . We demonstrate that self-attention blocks in Vision Transformers [8] implicitly encode a special mixture of experts architecture, revealing a surprising connection between these seemingly disparate concepts. Leveraging this connection, we propose that applying prefix tuning within pre-trained models can be interpreted as introducing new experts. The newly introduced experts collaborate with the pre-trained experts, facilitating efficient adaptation of the model to new tasks.Drawing insights from this analysis, we observe that the original prefix tuning suffers from suboptimal sample efficiency, requiring a substantial amount of data for reasonable parameter estimation. To address this challenge, we propose a novel gating mechanism termed Non-linear Residual Gates (NoRGa). This architecture integrates non-linear activation functions and residual connections within the gating score functions. Our work focuses on improving within-task prediction accuracy, a key component of continual learning performance as identified in previous research [22, 49] . We posit that NoRGa can enhance this aspect, which contributes to improved overall continual learning performance while maintaining parameter efficiency. We further provide theoretical justification for this improvement, demonstrating how NoRGa accelerates parameter estimation rates.Our contributions can be summarized as follows: (1) We reveal a novel connection between self-attention and a mixture of experts, providing a fresh perspective on prompt-based continual learning approaches; (2) Leveraging this insight, we propose Non-linear Residual Gates (NoRGa), an innovative gating mechanism that enhances continual learning performance while maintaining parameter efficiency, and provide a theoretical justification for this improvement; (3) Extensive experiments across various continual learning benchmarks and pre-training settings demonstrate that our approach achieves state-of-the-art performance compared to existing methods.Notation. For any n ∈ N, we denote [n] as the set {1, 2, . . . , n} . Next, for any set S, we let |S| stand for its cardinality. For any vector u := (u 1 , u 2 , . . . , u d ) ∈ R d and α := (α 1 , α 2 , . . . , α d ) ∈ N d , we let u α = u α1 1 u α2 2 . . . u α d d , |u| := u 1 +u 2 +. . .+u d and α! := α 1 !α 2 ! . . . α d !, while ∥u∥ stands for its 2-norm value. Lastly, for any two positive sequences {a n } n≥1 and {b n } n≥1 , we write a n = O(b n ) or a n ≲ b n if a n ≤ Cb n for all n ∈ N, where C > 0 is some universal constant. The notation a n = O P (b n ) indicates that a n /b n is stochastically bounded.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The paper's contributions and scope are reflected accurately in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background and related works: 
 We first provide background and related works on continual learning. Then, we define the attention mechanism, followed by discussions on prompt-based continual learning and mixture of experts.Continual Learning (CL) addresses the challenge of training a model incrementally on a sequence of T tasks, denoted by D = {D 1 , ..., D T }. Each task's training data D t = {(x(t) i , y (t) i )} Nt i=1 contains pairs of input sample x (t)i ∈ X (t) , and corresponding label y (t) i ∈ Y (t) . Notably, the class labels are distinct for each task, i.e., Y (t) Y (t ′ ) = ∅, ∀t ̸ = t ′ . Consider a neural network with a backbone function f θ and an output layer h ψ . The model predicts a label ŷ = h ψ (f θ (x)) ∈ Y = T t=1 Y (t) , where x ∈ X = T t=1 X (t) is an unseen test sample from arbitrary tasks. Importantly, during training on a new task, the model can only access the current data, without access to data from previous tasks. Prior approaches often rely on storing past task samples for training on new tasks, raising concerns regarding storage and privacy [5, 6, 39, 51, 59 ].Our work focuses on the class-incremental learning (CIL) setting, where task identities are not provided during inference, unlike in task-incremental learning (TIL) [46] . A recent theory by [22] analyzes the CIL objective by decomposing the probability of a test sample x of the j-th class in task t into two probabilities:EQUATIONwhere the first term involves within-task prediction (WTP) and the second term pertains to taskidentity inference (TII). This equation highlights that by improving either the WTP performance or the TII, we can consequently improve the overall CIL performance, as shown in [22, 49] .Attention Mechanism. Within the Transformer architecture, the attention mechanism plays a crucial role. One prevalent variant is scaled dot-product attention [47] , formally defined as follows:Definition 2.1 (Scaled Dot-Product Attention). Let K ∈ R N ×d k be a key matrix with N key vectors, and V ∈ R N ×dv be a value matrix with N corresponding value vectors. Given a query matrix Q ∈ R M ×d k , Attention over (K, V ) is defined asEQUATIONwhere the softmax function acts on the rows of matrix QK ⊤ ∈ R M ×N .Vision Transformer (ViT) [8] employs the same attention mechanism within multiple Multi-head Self-Attention (MSA) layers, which is formally defined as follows: Definition 2.2 (Multi-head Self-Attention Layer). Let X Q , X K , X V denote the input query, key, and value matrix, respectively, where X Q = X K = X V = [x 1 , ..., x N ] ⊤ ∈ R N ×d , and N is the length of the input sequence. The output is expressed asEQUATIONEQUATIONwhereW O ∈ R mdv×d , W Q i ∈ R d×d k , W K i ∈ R d×d k, and W V i ∈ R d×dv are projection matrices, and m is the number of heads in the MSA layer. In ViTs, they used k = d v = d/m.Prompt-based continual learning. Prompt-based approaches have emerged as a promising alternative within rehearsal-free continual learning [61, 52, 44] . In vision tasks, prompt-based methods often leverage a pre-trained ViT as a feature extractor f θ , with its parameters θ typically frozen. These methods enhance the model by introducing prompts, small sets of learnable parameters that influence the operations of the MSA layer [53] . Prompts are strategically injected into the query, key, and value matrices to guide the ViT in learning new tasks. We denote the prompt parameters by p ∈ R Lp×d , where L p is the sequence length and d is the embedding dimension. Previous work [53] outlines two main prompt-based approaches: Prompt Tuning (ProT) [25] and Prefix Tuning (PreT) [26] . While Prompt Tuning directly concatenates the same prompt parameter p to the query, key, and value, prefix tuning divides p into prefixes {p K , p V } ∈ R Lp 2 ×d and appends it to the key and value vectors:f Pre-T prompt (p, X Q , X K , X V ) := MSA X Q , p K X K , p V X V = Concat( h1 , ..., hm )W O (5)Existing prompt-based methods in CL address catastrophic forgetting by creating new adaptive prompts for each new task. During testing, the model chooses suitable prompt combinations to handle unseen data from any encountered task [49] . L2P [54] proposes a shared prompt pool for all tasks, utilizing a query-key mechanism for prompt selection. Instead of using the same prompt pool across tasks, DualPrompt [53] introduces G-Prompt and E-Prompt to capture task-agnostic and task-specific information, respectively. S-Prompt [52] focuses on learning task-specific prompts and employs a ProT strategy similar to L2P. CODA-Prompt [42] expands the prompt pool across tasks and performs a weighted summation of the prompt pool using attention factors. A recent work, HiDe-Prompt [49] , achieves state-of-the-art performance by introducing a new hierarchical decomposition of CIL objectives and optimizing each component for better performance.In this study, we focus on prefix tuning as our primary prompt-based methodology and follow the framework presented in HiDe-Prompt [49] . During training, HiDe-Prompt co-optimizes task-specific prompts p t and model's output layer parameters ψ for each new task t using the WTP objective. These prompts are stored within a prompt pool P = {p 1 , ..., p T }. At test time, a separate lightweight auxiliary output layer ĥω : R D → R T , trained with the TII objective, takes the uninstructed representation f θ (x) of a new data point x as input to infer the task identity, guiding the selection of the most suitable prompt p k from the prompt pool P. Subsequently, the final prediction is given as ŷ = h ψ (f θ (x, p k )). For further details, please refer to Appendix D.Mixture of experts (MoE) extends classical mixture models with an adaptive gating mechanism [19, 21] . An MoE model consists of a group of N expert networksf i : R d → R dv , for all i ∈ [N ],and a gate function G : R d → R N . Given an input h ∈ R d , MoE computes a weighted sum of expert outputs f i (h) based on learned score function s i : R d → R for each expert: where G(h) := softmax(s 1 (h), . . . , s N (h)). Building on this concept, works by [10, 41] established the MoE layer as a fundamental building block to scale up model capacity efficiently. Please refer to Appendix C for a comprehensive discussion of related works.EQUATIONQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have already discussed the limitations of our work in the conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
connection between prefix tuning and mixture of experts: 
 We first explore the relationship between attention and mixture of experts in Section 3.1, followed by establishing the connection between prefix tuning and the mixture of experts in Section 3.2.Following the notation established in Definition 2.2, let's consider the l-th head within the MSA layer.Let X = x ⊤ 1 , . . . , x ⊤ N ⊤ ∈ R N d, which is the concatenation of input sequence embeddings into a single one-dimensional vector. We define the matrixE i ∈ R d×N d such that E i X := x i for all i ∈ [N ]. Furthermore, we introduce an MoE architecture consisting of a group of N expert networks f j : R N d → R dv , N gating functions G i : R N d → R N with the score function for the j-th expert of the i-th gating s i,j : R N d → R, wheref j (X) := W V l ⊤ E j X = W V l ⊤ x j , s i,j (X) := X ⊤ E ⊤ i W Q l W K l ⊤ E j X √ d v = x ⊤ i W Q l W K l ⊤ x j √ d v for i and j ∈ [N ].From equation ( 4), we can express the output of the l-th head as follows:EQUATIONEQUATIONfor i ∈ [N ]. Expanding on equation (8) , we can discern that each attention head within the MSA layer implicitly embodies a special mixture of experts architecture. This architecture encompasses N MoE models, each featuring its own quadratic gating function G i . However, instead of employing N 2 separate expert networks for each model, this architecture utilizes N shared linear expert networks f j for j ∈ [N ], significantly reducing the number of parameters. Notably, each expert network and its corresponding gating function process the entire input sequence directly, rather than individual embedding x i as in traditional MoE layers [41] . This connection between self-attention and mixture of experts is depicted in Figure 1 . In the subsequent section, we explore how prompt-based techniques can be viewed through this lens.Building on the connection between self-attention and mixture of experts, we propose that applying prefix tuning can be interpreted as the introduction of new experts to customize the pre-trained model for a specific task, as illustrated in Figure 2 . Specifically, similar to Section 3.1, we consider the l-th head within the MSA layer. We denotep K = p K 1 , . . . , p K L ⊤ ∈ R L×d , p V = p V 1 , . . . , p V L ⊤ ∈ R L×d , where L = Lp 2 .We define new prefix experts f N +j : R N d → R dv along with their corresponding new score functions s i,N +j : R N d → R as follows:EQUATIONfor i ∈ [N ] and j ∈ [L]. Then from equation ( 5), the output of the l-th head can be expressed as:EQUATION)EQUATIONIt's worth noting that W Q l , W K l , and W V l remain fixed, with only p K and p V being learnable. By examining equation (8) and equation (11) , we can interpret each head in a multi-head self-attention layer within a pre-trained model as a mixture of experts architecture with pre-trained experts f j and gating score functions s i,j for i and j ∈ [N ]. Prefix tuning extends this MoE by introducing L additional prefix experts f N +j ′ defined by prefix vectors p V j ′ and linear score functions s i,N +j ′ for i ∈ [N ] and j ′ ∈ [L]. These new experts collaborate with the pre-trained ones within the MoE model, facilitating the model's adaptation to downstream tasks.We argue that our introduction of a novel connection between self-attention, prefix tuning, and MoE offers a fresh perspective on the design of previous prompt-based continual learning methods. In the context of continual learning, the pre-trained experts serve as a knowledge base, while prefix tuning augments it with task-specific knowledge encoded in new experts. Moreover, we draw a parallel between the pre-trained experts and the G(eneral)-Prompt utilized in DualPrompt, which captures task-agnostic information [53] . Both are shared across tasks, making them useful for prediction, especially when task identification is incorrect. Notably, the new experts achieve their efficiency through simple linear gating functions and independence from the input, unlike the pre-trained experts. For simplicity, we call the MoE model (11) as linear gating prefix MoE.Statistical suboptimality. The connection between prefix tuning and the MoE within the linear gating prefix MoE model (11) allows us to theoretically explore the statistical behavior of the prefix tuning. In Appendix A, by interpreting the linear gating prefix MoE as a regression problem with sample size n, we demonstrate that the convergence rate for estimating the model parameters, e.g., prompts, can be as slow as O(1/ log τ (n)) where τ > 0 is some constant. This suggests that a huge amount of data is required to achieve reasonable parameter estimation in the linear gating prefix MoE model, which can be discouraging in practice. To address this statistical limitation, the next section introduces a novel non-linear residual gating score function to replace the linear gating function.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
non-linear residual gate meets prefix tuning: 
 As discussed earlier, prefix tuning introduces additional experts within the MoE framework, resulting in the linear gating prefix MoE model. However, as outlined in Appendix A, this approach suffers from suboptimal sample efficiency for parameter estimation. To overcome this and enhance overall CIL performance, we propose an innovative approach that significantly improves sample efficiency while promoting WTP performance in Section 4.1 and provide theoretical explanations in Section 4.2.We propose a simple yet effective modification to the linear gating prefix MoE model by incorporating non-linear activation and residual connection within the score functions of prefix experts as follows:EQUATIONwhere α, τ ∈ R are learnable scalar factors, and σ is a non-linear activation function. The new score function in equation ( 12) consists of a linear and a non-linear component. We call the new prefix MoE model with score functions (12) as non-linear residual gating prefix MoE.The use of a non-linear activation function here is motivated by the algebraic independence condition in Definition 4.2 to theoretically guarantee the optimal sample efficiency of expert and parameter estimations (cf. Theorem 4.3). It's worth noting that removing the linear component s i,N +j (X) in the score function (12) could potentially lead to the vanishing gradient problem during training. To mitigate this challenge, we incorporate a residual connection [14] into the formulation. Our modification introduces minimal additional parameters (α and τ ) compared to the original score function, ensuring parameter efficiency. This is particularly crucial in continual learning scenarios where the number of parameters grows with each new task. For implementation, we defineH l = W Q l W K l ⊤ .From equation ( 5), the attention matrix of the l-th head can then be written as:EQUATIONHere, A prompt l denotes the attention score matrix for the prompts, and A pretrain l represents the attention score matrix for the pre-trained experts. To implement NoRGa, we can directly modify the final attention matrix as follows:EQUATIONl = A prompt l + α • σ(τ • A prompt l ).(15) The implementation of NoRGa is illustrated in Figure 2 . Despite its simplicity, our modification can significantly enhance sample efficiency and promote more reasonable parameter estimation, as demonstrated in our theoretical analysis in Section 4.2. Within the HiDe-Prompt framework, task-specific prompt parameters are trained using the WTP objective for each new task. Consequently, our modification leads to better parameter estimation, which directly contributes to improved WTP performance, ultimately improving overall continual learning efficacy. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL because of the memory constraint. Here, we evaluated σ with tanh, sigmoid, and GELU, finding tanh to perform well in most cases.Similar to the setting in Appendix A, we prove that estimating parameters in the non-linear residual gating prefix MoE model ( 12) is statistically efficient in terms of the number of data. To provide a fair comparison to the linear gating prefix MoE, we focus only on the first head and its first row, namely, l = 1 and i = 1 in equation (12) . Then, we proceed to provide a theoretical justification of our claim by viewing this row as an output of a regression setting. In particular, we assume thatEQUATIONwhere ε 1 , . . . , ε n are independent Gaussian noise variables such that E[ε i |X i ] = 0 and Var(ε i |X i ) = ν 2 for all 1 ≤ i ≤ n. Additionally, we assume that X 1 , X 2 , . . . , X n are i.i.d. samples from some probability distribution µ. The regression function g G * (•) in equation ( 16) then takes the form of a non-linear residual gating prefix MoE model with N pre-trained experts and L unknown experts,EQUATIONwhereT (X) := N k=1 exp(X ⊤ B 0 k X + c 0 k ) + L k ′ =1 exp((β * 1k ′ ) ⊤ X + ασ(τ (β * 1k ′ ) ⊤ X) + β * 0k ′ ), G * := L j ′ =1 exp(β * 0j ′ )δ (β * 1j ′ ,η * j ′) denotes a mixing measure, i.e., a weighted sum of Dirac measures δ, associated with unknown parameters(β * 1j ′ , β * 0j ′ , η * j ′ ) L j ′ =1 in R N d × R × R q .Here, the matrixB 0 j is equivalent to (E ⊤ 1 W Q 1 W K 1 ⊤ E j / √ d v )in the score function s 1,j (X), and the vector β * 1j ′ corresponds to the vector(E ⊤ 1 W Q 1 W K 1 ⊤ p K j ′ / √ d v ) in ŝ1,N+j ′ (X). Furthermore, the experts h(X, η 0 j ) and h(X, η * j ′ ) represent f j (X) and f N +j ′ (X), respectively. In our formulation, for the generality of the ensuing theory, we consider general parametric forms of the experts h(X, η 0 j ) and h(X, η * j ′ ), i.e., we do not only constrain these expert functions to be the forms of the simple experts in the aforementioned model. Similar to the setting in Appendix A, B 0 j , c 0 j , and the expert parameters η 0 j are known. Our goal is to estimate the unknown prompt-related parameters β * 1j ′ , β * 0j ′ , and η * j ′ . Least squares estimation. We will use the least squares method [45] to estimate the unknown parameters (β * 0j ′ , β * 1j ′ , η * j ′ ) L j ′ =1 or, equivalently, the ground-truth mixing measure G * . In particular, we take into account the estimatorEQUATIONwhere we denoteG L ′ (Θ) := {G = ℓ i=1 exp(β 0i )δ (β1i,ηi) : 1 ≤ ℓ ≤ L ′ , (β 0i , β 1i , η i ) ∈Θ} as the set of all mixing measures with at most L ′ atoms. In practice, since the true number of experts L is typically unknown, we assume that the number of fitted experts L ′ is sufficiently large, i.e., L ′ > L.To begin with, we explore the convergence behavior of the regression estimator g Gn (•) to the true regression function g G * (•) under the L 2 (µ)-norm in the following theorem: Theorem 4.1 (Regression Estimation Rate). Equipped with a least squares estimator G n given in equation (18) , the model estimation g Gn (•) converges to the true model g G * (•) at the following rate:EQUATIONProof of Theorem 4.1 is in Appendix B.1. The bound (19) implies that the rate for estimating the regression function g G * (•) is of order O P ( log(n)/n), which is parametric on the sample size n.More importantly, it also indicates that if there exists a loss function among parameters L such that∥g Gn -g G * ∥ L2(µ) ≳ L( G n , G * ), then we would obtain the bound L( G n , G * ) = O P ( log(n)/n),which leads to the desired parameter and expert estimation rates.We now turn our attention to the parameter and expert estimation problems. To understand how the non-linear residual gating affects these problems, we analyze the properties of the expert h(•, η) and the activation function σ(•) to determine which formulations will achieve favorable performance.Definition 4.2 (Algebraic independence). We say that an expert function h(•, η) and an activation function σ(•) are algebraically independent if they are twice differentiable w.r.t their parameters, and if for any k ≥ 1 and pair-wise distinct parameters (β 11 , η 1 ), . . . , (β 1k , η k ), the following set of functions in X is linearly independent for almost every X ∈ R N d :X ν (1 + σ ′ (β ⊤ 1j X)) |ν| + 1 {|ν|=2} σ ′′ (β ⊤ 1j X) • ∂ |γ| h ∂η γ (X, η j ) : j ∈ [k * ], ν ∈ N N d , γ ∈ N q : 0 ≤ |ν| + |γ| ≤ 2 .Intuitively, the algebraic independence condition ensures that there will be no interactions among parameters of the expert function h(•, η) and the activation function σ(•). Technically, a key step in our argument is to decompose the regression discrepancy g Gn (X) -g G * (X) into a combination of linearly independent terms by applying Taylor expansions to the product of the softmax's numerator and the expert function, i.e., exp(β ⊤ 1 X +ασ(τ β ⊤ 1 X))h(X, η). Thus, the above condition guarantees that all the derivative terms in the Taylor expansion are linearly independent. To exemplify the algebraic independence condition, we consider the following simple examples of the expert functions h(•, η) and the activation σ(•) that are algebraically independent.h(•, η) is formulated as a neural network h(X, (a, b)) = φ(a ⊤ X + b) with the activation φ(•) ∈ {ReLU(•), GELU(•), z → z p }, where (a, b) ∈ R N d × R,and the activation function σ(•) is one among the functions sigmoid(•), tanh(•), GELU(•), then they satisfy the algebraic independence condition in Definition 4.2.Finally, we establish the rates for estimating parameters and experts in the non-linear residual gating prefix MoE model in Theorem 4.3. Prior to presenting the theorem statement, let us design a loss function among parameters based on a notion of Voronoi cells [27] , which is a commonly employed approach for the convergence analysis of expert estimation in MoE models [36, 34, 35, 33] , yet tailored to the setting of this paper. In particular, the Voronoi loss used for our analysis is defined asEQUATIONwhere we denote∆β 1ij ′ := β 1i -β * 1j ′ and ∆η ij ′ := η i -η * j ′ . Above, V j ′ ≡ V j ′ (G), for j ′ ∈ [L], is a Voronoi cell associated with the mixing measure G generated by the true component ω * j := (β * 1j ′ , η * j ′ ), which is defined as follows:EQUATIONwhere we denoteω i := (β 1i , η i ) as a component of G. Note that, the cardinality of each Voronoi cell V j ′ indicates the number of components ω i of G approximating the true component ω * j ′ of G * . Additionally, since L 1 (G, G * ) = 0 if and only if G ≡ G * , it follows that when L 1 (G, G * ) becomessufficiently small, the differences ∆β 1ij ′ and ∆η ij ′ are also small. This observation indicates that, although L 1 (G, G * ) is a proper metric as it is not symmetric, it is an appropriate loss function for measuring the discrepancy between the least square estimator G n and the true mixing measures G * . Theorem 4.3. Assume that the expert function h(x, η) and the activation σ(•) are algebraically independent, then we achieve the following lower bound for any G ∈ G L ′ (Θ):∥g G -g G * ∥ L2(µ) ≳ L 1 (G, G * ), which together with Theorem 4.1 indicates that L 1 ( G n , G * ) = O P (n -1/2 ).Proof of Theorem 4.3 is in Appendix B.2. A few comments on Theorem 4.3 are in order: (i) From the bound L 1 ( G n , G * ) = O P (n -1/2 ), we deduce that the estimation rates for the over-specified parametersβ * 1j ′ , η * 1j ′ , where j ′ ∈ [L] : |V j ′ | > 1, are all of order O P ( 4 log(n)/n).Since the expert h(•, η) is twice differentiable over a bounded domain, it is also a Lipschitz function. Thus, denote Table 1 : Overall performance comparison on Split CIFAR-100 and Split ImageNet-R. We present Final Average Accuracy (FA), Cumulative Average Accuracy (CA), and Average Forgetting Measure (FM) of all methods under different pre-trained models.EQUATIONThe above bound indicates that if the experts h(•, η * j ) are fitted by at least two other experts, then their estimation rates are also of order O P ( 4 log(n)/n); (ii) For exactly-specified parameters β * 1j ′ , η * j ′ , where j ′ ∈ [L] : |V j ′ | = 1, the rates for estimating them are faster than those of their over-specified counterparts, standing at order O P ( log(n)/n). By arguing similarly to equation (22) , the experts h(•, η * j ′ ) also enjoy the faster estimation rate of order O P ( log(n)/n), which is parametric on the sample size n; (iii) It follows from the above rates that we only need a polynomial number of data (roughly ϵ -4 where ϵ is the desired approximation error) to estimate the parameters and experts of the non-linear residual gating prefix MoE. By contrast, when using the linear gating, as being demonstrated in Appendix A, it requires an exponential number of data. This highlights the statistical benefits of using the non-linear residual gating MoE model over the linear gating prefix MoE model. 
experiments: 
 Datasets. We evaluate various continual learning methods on widely used CIL benchmarks, including Split CIFAR-100 [23] and Split ImageNet-R [23] , consistent with prior work [49] . We further explore the model's performance on fine-grained classification tasks with Split CUB-200 [48] and large inter-task differences with 5-Datasets [9] . Please refer to Appendix E for more details.Evaluation Metrics. We utilize several established metrics described in [50] . These include: final average accuracy (FA), which represents the average accuracy after the final task; cumulative average accuracy (CA), which refers to the historical average accuracy; and average forgetting measure (FM). We give more emphasis to FA and CA due to their comprehensiveness, as noted in [42] .Baselines. We compare our approach with several representative prompt-based approaches including L2P [54] , DualPrompt [53] , CODA-Prompt [42] , S-Prompt [52] , and HiDe-Prompt [49] . Additionally, we evaluate against state-of-the-art pre-trained model-based continual learning methods, including ADAM [60] and RanPAC [29] . We further extend our evaluation by applying HiDe-Prompt with parameter-efficient fine-tuning techniques like LoRA [17] and Adapters [16] . In line with [49] , we utilize the checkpoints of ViT that use supervised pre-training of Imagenet-21K (denoted as Sup-21K), and some self-supervised pre-training such as iBOT-21K, iBOT-1K [62] , DINO-1K [4] , and MoCo-1K [7] . For implementation details, see Appendix E.Main Results. In Table 1 , we evaluate several continual learning methods on Split CIFAR-100 and Split ImageNet-R using diverse pre-trained models. NoRGa achieves state-of-the-art FA and CA across all datasets and models, consistently outperforming HiDe-Prompt. On Sup-21K, NoRGa demonstrates impressive FA results on both CIFAR-100 and ImageNet-R. It also maintains the highest CA, with significant margins of 1.80% and 2.92% on CIFAR-100 and ImageNet-R, respectively, compared to HiDe-Prompt. These results highlight NoRGa's strong ability to retain knowledge and exhibit minimal forgetting, as evidenced by the low FM values on both datasets. NoRGa also surpasses HiDe-Prompt on self-supervised models, with FA improvements up to 1.95% and 3.66%. We further investigate two scenarios: fine-grained classification tasks and large inter-task differences through experiments on Split CUB-200 and 5-Datasets, respectively, as shown in Table 2 . NoRGa maintains its lead, achieving FA gaps of 4.34% and 2.46% on Split CUB-200, and the highest FA on 5-Datasets. While gains in some metrics may be modest, NoRGa consistently outperforms HiDe-Prompt in either FA or CA, underscoring its robustness. For example, on Split ImageNet-R with Sup-21K weights, the FA improvement is small (75.06% vs. 75.40%), but the CA gains are substantial (76.60% vs. 79.52%), demonstrating the method's effectiveness and robustness.Ablation Study. To assess the impact of non-linear activation functions on NoRGa's performance, we evaluated the model's behavior with different choices for the activation function σ, including tanh, sigmoid, and GELU in Table 3 . The results show that NoRGa achieves state-of-the-art performance on both Split CIFAR-100 and Split CUB-200 datasets with all three activation functions. These findings suggest that NoRGa exhibits robustness to the choice of non-linear activation within a reasonable range. While all functions perform well, the tanh activation function demonstrates generally strong performance across scenarios. Further results are provided in the Appendix. 
conclusion: 
 This paper presents an initial exploration of self-attention and prefix-tuning through the lens of mixture of experts. We find that applying prefix tuning can be viewed as introducing new prefix experts to adapt the pre-trained model. However, limitations in sample efficiency exist. We address this by proposing NoRGa, a novel gating mechanism to enhance continual learning performance.Our results demonstrate NoRGa's effectiveness both theoretically and empirically. While the current implementation of the expert network prioritizes simplicity, future research directions could involve investigating more intricate architectures. Furthermore, the choice of activation functions in our work requires fine-tuning, which opens avenues for future research on adaptively learning activation. In Appendix I, we present the efficiency tests, while Appendix J explores the impact of learnable α and τ . Finally, Appendix K compares the training times between NoRGa and HiDe-Prompt.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All the experimental details are specified in Section 5, and Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report error bars and relevant information in Section 5, and Appendix F.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Given the theoretical nature of the paper, we do not think there are any positive or negative societal impacts of the work performed. Experiments are conducted only for empirically justifying the theoretical results. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not release any data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The data and models used in the paper are properly credited. See Section 5, and Appendix E for further details.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this appendix, we demonstrate that estimating parameters and experts in the linear gating prefix MoE model ( 11) can be statistically inefficient in terms of the number of data. To simplify our findings, we particularly focus on the first head, namely, l = 1 in equation ( 11), and the first row of this head, namely, i = 1 in equation (11) . Then, we proceed to provide a theoretical justification of our claim for the suboptimality of the linear gating prefix MoE by viewing this row as an output of the regression setting. In particular, we assume that(X 1 , Y 1 ), (X 2 , Y 2 ), . . . , (X n , Y n ) ∈ R N d × R is an i.i.d.sample generated from the following model:EQUATIONwhere ε 1 , . . . , ε n are independent Gaussian noise variables such thatE[ε i |X i ] = 0 and Var(ε i |X i ) = ν 2 for all 1 ≤ i ≤ n.Additionally, we assume that X 1 , X 2 , . . . , X n are i.i.d. samples from some probability distribution µ. Motivated by linear gating prefix MoE model (11) , the regression function f G * (•) in equation ( 23) admits the form of the linear gating prefix MoE model with pre-trained N experts and L unknown experts, namelyEQUATIONwhereG * := L j ′ =1 exp(β * 0j ′ )δ (β * 1j ′ ,η * j ′) denotes a mixing measure, i.e., a weighted sum of Dirac measures δ, associated with unknown parameters(β * 1j ′ , β * 0j ′ , η * j ′ ) L j ′ =1 in R N d × R × R q .Here, the matrix B 0 j plays the role of the matrixE ⊤ 1 W Q 1 W K 1 ⊤ Ej √ dvin the score function s 1,j (X). Furthermore, the vector β * 1j ′ corresponds to the vectorE ⊤ i W Q l W K l ⊤ p K j ′ √ dvin the score function s 1,N +j ′ (X). Furthermore, the experts h(X, η 0 j ) correspond to the role of f j (X) and h(X, η * j ′ ) correspond to the role of f N +j ′ (X). In our formulation, we consider general parametric forms of the experts h(X, η 0 j ) and h(X, η * j ′ ), i.e., we do not only constrain these expert functions to be the forms of the simple experts in the linear gating prefix MoE model. Similar to the linear gating prefix MoE model (11) , the matrices B 0 j , the biases c 0 j , and the expert parameters η 0 j are known. Our aim is to estimate the unknown gating parameters β * 1j ′ , β * 0j ′ , and η * j ′ that correspond to the prompts.Least squares estimation: We will use the least squares method [45] to estimate the unknown parameters(β * 0j ′ , β * 1j ′ , η * j ′ ) L j ′ =1or, equivalently, the ground-truth mixing measure G * . In particular, we take into account the estimatorEQUATION)where we denoteG L ′ (Θ) := {G = ℓ i=1 exp(β 0i )δ (β1i,ηi) : 1 ≤ ℓ ≤ L ′ , (β 0i , β 1i , η i ) ∈Θ} as the set of all mixing measures with at most L ′ atoms. In practice, since the true number of true experts L is typically unknown, we assume that the number of fitted experts L ′ is sufficiently large, i.e. L ′ > L.Let us recall that our main objective in this appendix is to show that using the linear gating in the prefix MoE model is not sample efficient. To illustrate that point, we consider a simple scenario when the expert function takes the form h(X, (a, b)) = (a ⊤ X + b) p , for some p ∈ N. Additionally, we also design a new Voronoi loss function as below to facilitate our arguments.EQUATIONwhere we denote∆β 1ij ′ := β 1i -β * 1j ′ and ∆η ij ′ := η i -η * j ′. Now, we are ready to state the result of parameter estimation under the linear gating prefix MoE model in the following theorem:Theorem A.1. Assume that the experts take the form h(X, (a, b)) = (a ⊤ X + b) p , for some p ∈ N, then we achieve the following minimax lower bound of estimating G * :inf Gn∈G L ′ (Θ) sup G∈G L ′ (Θ)\G L-1 (Θ) E f G [L 2,r (G n , G)] ≳ n -1/2 ,for any r ≥ 1, where E f G indicates the expectation taken w.r.t the product measure with f n G .There are two main implications of the result in Theorem A.1:(i) The rates for estimating parameters β * 1j , a * j and b * j are slower than O P (n -1/2r ), for any r ≥ 1. This means that they are slower than any polynomial rates, and could be of order O P (1/ log(n)). Using the same reasoning described after equation ( 22), we haveEQUATION)As a consequence, the rates for estimating experts φ((a * j ) ⊤ X + b * j ) are no better than those for estimating the parameters a * j and b * j , and could also be as slow as O P (1/ log(n)). (ii) The above rates imply that we need an exponential number of data (roughly exp(1/ϵ τ ) where ϵ is the desired approximation error) to estimate the parameters and experts of the linear gating prefix MoE. This fact demonstrates that using the linear gating in the prefix MoE model is not sample efficient from the perspective of the expert estimation problem.Proof of Theorem A.1. Prior to presenting the main proof of Proposition A.1, let us introduce the following key result: Lemma A.2. If the following holds for any r ≥ 1:EQUATIONthen we obtain that infEQUATION)Proof of Lemma A.2. Indeed, from the Gaussian assumption on the noise variables ϵ i , we obtain thatY i |X i ∼ N (f G * (X i ), σ 2 ) for all i ∈ [n].Next, the assumption in equation ( 28) indicates for sufficiently small ε > 0 and a fixed constant C 1 > 0 which we will choose later, we can find a mixing measureG ′ * ∈ G L ′ (Θ) such that L 2,r (G ′ * , G * ) = 2ε and ∥f G ′ * -f G * ∥ L 2 (µ) ≤ C 1 ε.From Le Cam's lemma [57] , as the Voronoi loss function L 2,r satisfies the weak triangle inequality, we obtain that infEQUATIONwhere the second inequality is due to the fact thatKL(N (f G ′ * (X), σ 2 ), N (f G * (X), σ 2 )) = (f G ′ * (X) -f G * (X)) 2 2σ 2 .By choosingε = n -1/2 , we obtain that ε • exp(-C 1 nε 2 ) = n -1/2 exp(-C 1 ). As a consequence, we achieve the desired minimax lower bound in equation (29) .Main proof. We need to prove that the following limit holds true for any r ≥ 1:EQUATIONFor that purpose, it suffices to build a sequence of mixing measures(G n ) n≥1 such that both L 2,r (G n , G * ) → 0 and ∥f Gn -f G * ∥ L2(µ) L 2,r (G n , G * ) → 0,as n → ∞. To this end, we consider the sequenceG n = L+1 i=1 exp(β n 0i )δ (β n 1i ,a n i ,b n i ) , where • exp(β n 01 ) = exp(β n 02 ) = 1 2 exp(β * 01 ) + 1 2n r+1 and exp(β n 0i ) = exp(β n 0(i-1) ) for any 3 ≤ i ≤ L + 1; • β n 11 = β n 12 = β * 11 and β n 1i = β n 1(i-1) for any 3 ≤ i ≤ L + 1; • a n 1 = a n 2 = a * 1 and a n i = a n i-1 for any 3 ≤ i ≤ L + 1; • b n 1 = b * 1 + 1 n , b n 2 = b * 1 -1 n and b n i = b * i-1 for any 3 ≤ i ≤ L + 1.As a result, the loss functionEQUATIONwhich indicates indicates that L 2,r (G n , G * ) → 0 as n → ∞.Gn -f G * ∥ L2(µ) /L 2,r (G n , G * ) → 0.For that purpose, let us consider the quantityQ n (X) := N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L j ′ =1 exp((β * 1j ′ ) ⊤ X + β * 0j ′ ) • [g Gn (X) -g G * (X)].For simplicity, let us consider the polynomial degree p = 1 as the arguments for other values of p can be adapted accordingly. Recall from equation ( 46) that Q n (X) can be decomposed as follows:Q n (X) = L j=1 i∈Aj exp(β n 0i ) exp((β n 1i ) ⊤ X)((a n i ) ⊤ X + b n i ) -exp((β * 1j ) ⊤ X)((a * j ) ⊤ X + b * j ) - L j=1 i∈Aj exp(β n 0i ) exp((β n 1i ) ⊤ X)g Gn (X) -exp((β * 1j ) ⊤ X)g Gn (X) + L j=1 i∈Aj exp(β n 0i ) -exp(β * 0j ) exp((β * 1j ) ⊤ X)((a * j ) ⊤ X + b * j ) -exp((β * 1j ) ⊤ X)g Gn (X) := A n (X) -B n (X) + C n (X).From the definitions of β n 1i , a n i and b n i , we can rewrite A n (X) as follows:A n (X) = 2 i=1 1 2 exp(β * 01 ) + 1 n r+1 exp((β * 11 ) ⊤ X)[((a n i ) ⊤ X + b n i ) -((a * 1 ) ⊤ X + b * 1 )] = 1 2 exp(β * 01 ) + 1 n r+1 exp((β * 11 ) ⊤ X)[(b n 1 -b * 1 ) + (b n 2 -b * 1 )] = 0.Additionally, it can also be checked that B n (X) = 0, andC n (X) = O(n -(r+1) ). Therefore, it follows that C n (X)/L 2,r (G n , G * ) → 0. As a consequence, Q n (X)/L 2,r (G n , G * ) → 0 as n → ∞for almost every X.N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L j ′ =1 exp((β * 1j ′ ) ⊤ X + β * 0j ′ ) is bounded, we deduce that [f Gn (X) -f G * (X)]/L 2,r → 0 for almost every X. This result indicates that ∥f Gn -f G * ∥ L2(µ) /L 2,r (G n , G * ) → 0as n → ∞. Hence, the proof of claim ( 31) is completed.In this appendix, we present rigorous proofs for the theoretical results introduced in Section 4, namely Theorem 4.1 and Theorem 4.3, in that order.For the proof of the theorem, we first introduce some notation. Firstly, we denote byF L ′ (Θ) the set of conditional densities of all mixing measures in G L ′ (Θ), that is, F L ′ (Θ) := {g G (X) : G ∈ G L ′ (Θ)}.Additionally, for each δ > 0, the L 2 (µ) ball centered around the conditional density g G * (Y |X) and intersected with the set F L ′ (Θ) is defined asF L ′ (Θ, δ) := g ∈ F L ′ (Θ) : ∥g -g G * ∥ L 2 (µ) ≤ δ .In order to measure the size of the above set, Geer et. al. [45] suggest using the following quantity:EQUATIONwhereH B (t, F L ′ (Θ, t), ∥ • ∥ L 2 (µ))stands for the bracketing entropy [45] of F L ′ (Θ, u) under the L 2 (µ)-norm, and t ∨ δ := max{t, δ}. By using the similar proof argument of Theorem 7.4 and Theorem 9.2 in [45] with notations being adapted to this work, we obtain the following lemma: Lemma B.1. Take Ψ(δ) ≥ J B (δ, F L ′ (Θ, δ)) that satisfies Ψ(δ)/δ 2 is a non-increasing function of δ. Then, for some universal constant c and for some sequence (δ n ) such that √ nδ 2 n ≥ cΨ(δ n ), we achieve thatP ∥g Gn -g G * ∥ L 2 (µ) > δ ≤ c exp - nδ 2 c 2 ,for all δ ≥ δ n .We now demonstrate that when the expert functions are Lipschitz continuous, the following bound holds:EQUATIONfor any 0 < ε ≤ 1/2. Indeed, for any function g G ∈ F L ′ (Θ), since the expert functions are bounded, we obtain that h(X, η) ≤ M for all X where M is a bounded constant of the expert functions. Let τ ≤ ε and {π 1 , . . . , π N } be the ζ-cover under the L ∞ norm of the set F L ′ (Θ) whereN := N (ζ, F L ′ (Θ), ∥ • ∥ L ∞ ) is the η-covering number of the metric space (F L ′ (Θ), ∥ • ∥ L ∞ ).Then, we construct the brackets of the form [L i (X), U i (X)] for all i ∈ [ N ] as follows:L i (x) := max{π i (X) -ζ, 0}, U i (x) := max{π i (X) + ζ, M }.From the above construction, we can validate thatF L ′ (Θ) ⊂ ∪ N i=1 [L i (X), U i (X)] and U i (X) - L i (X) ≤ min{2ζ, M }. Therefore, it follows that ∥U i -L i ∥ 2 L2(µ) = (U i -L i ) 2 dµ(X) ≤ 4ζ 2 dµ(X) = 4ζ 2 ,which implies that ∥U i -L i ∥ L2(µ) ≤ 2ζ. By definition of the bracketing entropy, we deduce thatEQUATIONTherefore, we need to provide an upper bound for the covering number N . In particular, we denote∆ := {(β 1 , β 0 ) ∈ R N d×N d × R N d × R : (β 1 , β 0 , η) ∈ Θ} and Ω := {η ∈ R q : (β 1 , β 0 , η) ∈ Θ}.Since Θ is a compact set, ∆ and Ω are also compact. Therefore, we can find ζ-covers ∆ ζ and Ω ζ for ∆ and Ω, respectively. We can check that|∆ ζ | ≤ O P (τ -(N d+1)L ′ ), |Ω ζ | ≲ O P (τ -qL ′ ).For each mixing measure G =L ′ i=1 exp(β 0i )δ (β1i,ηi) ∈ G L ′ (Θ), we consider other two mixing measures:Ǧ := L ′ i=1 exp(β 0i )δ (β1i,η i ) , G := L ′ i=1 exp(β 0i )δ (β 1i ,η i ) .Here,η i ∈ Ω ζ such that η i is the closest to η i in that set, while (β 1i , β 0i ) ∈ ∆ ζ is the closest to (β 1i , β 0i) in that set. From the above formulations, we get that∥g G -g Ǧ∥ L ∞ = sup X∈X L ′ j=1 exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) • |h(X, η j ) -h(X, η j )| N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L ′ j ′ =1 exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ x) + β 0j ′ ) ≤ L ′ j=1 sup X∈X exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) • |h(X, η j ) -h(X, η j )| N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L ′ j ′ =1 exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ X) + β 0j ′ ) ≤ L ′ j=1 sup X∈X |h(X, η j ) -h(X, η j )| ≤ L ′ j=1 sup X∈X [L 1 (X) • ∥η j -η j ∥] ≲ L ′ ζ ≲ ζ.Here, the second inequality occurs as the softmax weight is bounded by one, and the third inequality follows from the fact that the expert h(X, •) is a Lipschitz function with some Lipschitz constant L 1 (X) > 0. Next, let us denoteD : = N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L ′ j ′ =1 exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ X) + β 0j ′ ), D : = N i ′ =1 exp(X ⊤ B 0 i ′ X + c 0 i ′ ) + L ′ j ′ =1 exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ X) + β 0j ′ ).Then, we haveEQUATIONNow, we will bound two terms in the above right hand side. Firstly, since both the input space X and the parameter space Θ are bounded, we have that1 D - 1 D ≲ |D -D| ≤ L ′ j ′ =1 exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ X) + β 0j ′ ) -exp(β ⊤ 1j ′ X + ασ(τ β ⊤ 1j ′ X) + β 0j ′ ) ≲ L ′ j ′ =1 (β 1j -β 1j ) ⊤ X + α[σ(τ β ⊤ 1j ′ X) -σ(τ β ⊤ 1j ′ X)] + (β 0j -β 0j ) ≤ L ′ j ′ =1 |(β 1j -β 1j ) ⊤ X| + |α| • |σ(τ β ⊤ 1j ′ X) -σ(τ β ⊤ 1j ′ X)| + |β 0j -β 0j | ≲ L ′ j=1 ∥β 1j -β 1j ∥ • ∥X∥ + |ατ | • ∥β 1j -β 1j ∥ • ∥X∥ + |β 0j -β 0j | ≤ L ′ (B + |ατ |B + 1)ζ ≲ ζ.As a result, we deduce thatEQUATIONRegarding the second term, note thatexp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) D - exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) D = exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) 1 D - 1 D + 1 D exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j ) -exp(exp(β ⊤ 1j X + ασ(τ β ⊤ 1j X) + β 0j )) .Since both the input space and the parameter space are bounded, we haveEQUATIONFrom equations ( 36), ( 37) and (38) , we obtain that ∥g Ǧ -g G ∥ L ∞ ≲ ζ. According to the triangle inequality, we have∥g G -g G ∥ L ∞ ≤ ∥g G -g Ǧ∥ L ∞ + ∥g Ǧ -g G ∥ L ∞ ≲ ζ.By definition of the covering number, we deduce thatEQUATIONCombine equations ( 35) and ( 39), we achieve thatH B (2ζ, F L ′ (Θ), ∥ • ∥ L2(µ) ) ≲ log(1/τ ).Let ζ = ε/2, then we obtain thatH B (ε, F L ′ (Θ), ∥ • ∥ L 2 (µ) ) ≲ log(1/ε).As a result, it follows thatEQUATIONLetΨ(δ) = δ • [log(1/δ)] 1/2, then Ψ(δ)/δ 2 is a non-increasing function of δ. Furthermore, equation (40) indicates that Ψ(δ) ≥ J B (δ, F L ′ (Θ, δ)). In addition, let δ n = log(n)/n, then we get that √ nδ 2 n ≥ cΨ(δ n ) for some universal constant c. Finally, by applying Lemma B.1, we achieve the desired conclusion of the theorem.Our goal is also to demonstrate the following inequality:EQUATIONFor that purpose, we divide the proof of the above inequality into local and global parts in the sequel.Local part: In this part, we demonstrate thatEQUATIONAssume by contrary that the above claim is not true, then there exists a sequence of mixing measuresEQUATIONas n → ∞. Let us denote by V n j := V j (G n ) a Voronoi cell of G n generated by the j-th components of G * . Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i.e., V j = V n j . Thus, the Voronoi loss L 1n can be represented asEQUATIONwhere we denote ∆β n 1ij := β n 1i -β * 1j and ∆η n ij := η n i -η * j . Since L 1n → 0, we get that (β n 1i , η n i ) → (β * 1j , η * j ) and i∈Vj exp(β n 0i ) → exp(β * 0j ) as n → ∞ for any i ∈ V j and j ∈ [L]. Now, we divide the proof of the local part into three steps as follows:Step 1 -Taylor expansion. In this step, we would like to decompose the quantityEQUATION) into a combination of linearly independent elements using Taylor expansion. In particular, the quantity Q n (X) is decomposed as follows:EQUATIONDecomposition of A n (X). Let us denote E(X; β 1 ) := exp(β ⊤ 1 X + ασ(τ β ⊤ 1 X)), then A n can be separated into two terms as follows:A n (X) := j:|Vj |=1 i∈Vj exp(β n 0i ) E(X; β n 1i )h(x; η n i ) -E(X; β * 1j )h(X; η * j ) + j:|Vj |>1 i∈Vj exp(β n 0i ) E(X; β n 1i )h(X; η n i ) -E(X; β * 1j )h(X; η * j ) := A n,1 (X) + A n,2 (X). By means of the first-order Taylor expansion, we haveA n,1 (X) = j:|Vj |=1 i∈Vj exp(β n 0i ) α! |α|=1 (∆β n 1ij ) α1 (∆η n ij ) α2 ∂ |α1| E ∂β α1 1 (X; β * 1j ) ∂ |α2| h ∂η α2 (X; η * j ) + R n,1 (X) = j:|Vj |=1 |α1|+|α2|=1 S n,j,α1,α2 ∂ |α1| E ∂β α1 1 (X; β * 1j ) ∂ |α2| h ∂η α2 (X; η * j ) + R n,1 (X),where R n,1 (X) is a Taylor remainder such that R n,1 (X)/L 1n → 0 as n → ∞, andS n,j,α1,α2 := i∈Vj exp(β n 0i ) α! (∆β n 1ij ) α1 (∆η n ij ) α2 .On the other hand, by applying the second-order Taylor expansion, we get thatA n,2 (X) = j:|Vj |>1 1≤|α1|+|α2|≤2 S n,j,α1,α2 ∂ |α1| E ∂β α1 1 (X; β * 1j ) ∂ |α2| h ∂η α2 (X; η * j ) + R n,2 (X), in which R n,2 (X) is a Taylor remainder such that R n,2 (X)/L 1n → 0 as n → ∞.Decomposition of B n . Recall that we haveB n (X) = j:|Vj |=1 i∈Vj exp(β n 0i ) E(X; β n 1i ) -E(X; β * 1j ) g Gn (X) + j:|Vj |>1 i∈Vj exp(β n 0i ) E(X; β n 1i ) -E(x; β * 1j ) g Gn (X) := B n,1 (X) + B n,2 (X).By invoking first-order and second-order Taylor expansions to B n,1 (X) and B n,2 (X), it follows thatB n,1 (X) = j:|Vj |=1 |ℓ|=1 T n,j,ℓ • ∂ |ℓ| E ∂β ℓ 1 (X; β * 1j )g Gn (X) + R n,3 (X), B n,2 (X) = j:|Vj |>1 1≤|ℓ|≤2 T n,j,ℓ • ∂ |ℓ| E ∂β ℓ 1 (X; β * 1j )g Gn (X) + R n,4 (X),where we defineT n,j,ℓ := i∈Vj exp(β n 0i ) ℓ! (∆β n 1ij ) ℓ .Additionally, R n,3 (X) and R n,4 (X) are Taylor remainders such that R n,3 (X)/L 1n → 0 and R n,3 (X)/L 1n → 0 as n → ∞.Collect the above results together, we can represent Q n (x) asEQUATIONwhere we define S n,j,0 d×d ,0q = T n,j,0 d×d = i∈Vj exp(β n 0i ) -exp(β * 0j ) for any j ∈ [L].Step 2 -Non-vanishing coefficients. In this step, we demonstrate that at least one among ratios of the forms S n,j,α1,α2 /L 1n and T n,j,ℓ /L 1n goes to zero as n tends to infinity. Indeed, assume by contrary thatS n,j,α1,α2 L 1n → 0, T n,j,ℓ L 1n → 0, for any j ∈ [L], 0 ≤ |α 1 |, |α 2 |, |ℓ| ≤ 2.Then, we getEQUATIONNow, we consider indices j ∈ [L] such that its corresponding Voronoi cell has only one element, i.e.|V j | = 1. • For arbitrary u, v ∈ [N d], let α 1 ∈ N N d×N d and α 2 = 0 q such that α (uv) 1= 1 while other entries equal to zero. Then, we have1 L1n • i∈Vj exp(β n 0i )|(∆β n 1ij ) (uv) | = |S n,j,α1,α2 |/L 1n → 0 as n → ∞. By taking the summation of the previous term with u, v ∈ [N d], we achieve that 1 L1n i∈Vj exp(β n 0i )∥∆β n 1ij ∥ 1 → 0.Owing to the topological equivalence between norm-1 and norm-2, it follows thatEQUATION• For arbitrary u ∈ [N d], let α 1 = 0 N d×N d and α 2 ∈ N q such that α (u) 2= 1 while other entries equal to zero. Then, we get1 L1n • i∈Vj exp(β n 0i )|(∆η n ij ) (u) | = |S n,j,α1,α2 |/L 1n → 0 as n → ∞.By taking the summation of the previous term with u ∈ [q], we achieve thatEQUATION)Combine the limits in equations ( 49) and ( 50), we obtain thatEQUATIONas n → ∞.Next, we consider indices j ∈ [L] such that its corresponding Voronoi cell has more than one element, i.e. |V j | > 1.• For arbitrary u, v ∈ [N d], let α 1 ∈ N N d×N d and α 2 = 0 q such that α (uv) 1= 2 while other entries equal to zero. Then, we have1 L1n • i∈Vj exp(β n 0i )|(∆β n 1ij ) (uv) | 2 = |S n,j,α1,α2 |/L 1n → 0 as n → ∞.By taking the summation of the previous term with u, v ∈ [N d], we achieve thatEQUATION• For arbitrary u ∈ [N d], let α 1 = 0 N d×N d and α 2 ∈ N q such that α (u) 2 = 2 while other entries equal to zero. Then, we get1 L1n • i∈Vj exp(β n 0i )|(∆η n ij ) (u) | 2 = |S n,j,α1,α2 |/L 1n → 0 as n → ∞.By taking the summation of the previous term with u ∈ [q], we achieve thatEQUATION)Putting the limits in equations ( 49) and ( 50), we haveEQUATIONas n → ∞. Taking the summation of three limits in equations ( 48), ( 51) and ( 54), we deduce that 1 = L 1n /L 1n → 0 as n → ∞, which is a contradiction. Thus, at least one among ratios of the forms S n,j,α1,α2 /L 1n and T n,j,ℓ /L 1n goes to zero as n tends to infinity.Step 3 -Application of Fatou's lemma. In this step, we show that all the ratios S n,j,α1,α2 /L 1n and T n,j,ℓ /L 1n go to zero as n → ∞, which contradicts to the conclusion in Step 2. In particular, by denoting m n as the maximum of the absolute values of those ratios. From the result of Step 2, it follows that 1/m n ̸ → ∞.Recall from the hypothesis in equation ( 43) that ∥g Gn -g G * ∥ L2(µ) /L 1n → 0 as n → ∞, which indicates that ∥g Gn -g G * ∥ L 1 (µ) /L 1n → 0. Therefore, by applying the Fatou's lemma, we get that0 = lim n→∞ ∥g Gn -g G * ∥ L 1 (µ) m n L 1n ≥ lim inf n→∞ |g Gn (X) -g G * (X)| m n L 1n dµ(X) ≥ 0.This result implies that 45), since the term1 mnL1n • [g Gn (X) -g G * (X)] → 0 as n → ∞ for µ-almost surely X. Looking at the formulation of Q n (X) in equation (k0 i ′ =1 exp(X ⊤ A 0 i ′ X +c 0 i ′ )+ k * j ′ =1 exp((β * 1j ′ ) ⊤ X + σ((β * 1j ′ ) ⊤ X) + β * 0j ′) is bounded, we deduce that the term Qn(X) mnL1n → 0 for µ-almost surely X.S n,j,α1,α2m n L 1n → ϕ j,α1,α2 , T n,j,ℓ m n L 1n → φ j,ℓ ,with a note that at least one among them is non-zero. Then, from the decomposition of Q n (X) in equation ( 47), we haveL j=1 1+1 {|V j |>1} |α1|+|α2|=0 ϕ j,α1,α2 • ∂ |α1| E ∂β α1 1 (X; β * 1j ) ∂ |α2| h ∂η α2 (X; η * j ), - L j=1 1+1 {|V j |>1} |ℓ|=0 φ j,ℓ • ∂ |ℓ| E ∂β ℓ 1 (X; β * 1j )g G * (X) = 0,For ease of presentation, let us denoteEQUATION)As the expert function h satisfies the conditions in Definition 4.2, the set {h(X,η ′ i ) : i ∈ [k ′ ]}, where η ′ 1 , . . . , η ′ k ′ are distinct vectors for some k ′ ∈ N, is linearly independent. If L ′ ̸ = L, then there exists some i ∈ [L ′ ] such that η i ̸ = η * j for any j ∈ [L]. This implies that L ′ j=1 softmax G (β 1j ) ⊤ X + ασ(τ (β 1j ) ⊤ X) + β 0j • h(X, η j ) = 0, which is a contradiction. Thus, we must have that L = L ′ . As a result, softmax G (β 1j ) ⊤ X + ασ(τ (β 1j ) ⊤ X) + β 0j : j ∈ [L ′ ] = softmax G * (β * 1j ) ⊤ X + ασ(τ (β * 1j ) ⊤ X) + β * 0j : j ∈ [L] ,for almost every X. WLOG, we may assume thatEQUATIONfor almost every X for any j ∈ [L]. Since the softmax function is invariant to translation, this result indicates that β 1j = β * 1j and β 0j = β * 0j + v 0 for some v 0 ∈ R for any j ∈ [L]. Recall from the universal assumption that β 0L ′ = β 0L = 0, we get that β 0j = β * 0j for any j ∈ [L]. Then, equation ( 56) can be rewritten asEQUATION)for almost every X. Next, we denote P 1 , P 2 , . . . , P m as a partition of the index set [L], where m ≤ L ′ , such that exp(β 0i ) = exp(β * 0i ′ ) for any i, i ′ ∈ P j and j ∈ [L]. On the other hand, when i and i ′ do not belong to the same set P j , we let exp(β 0i ) ̸ = exp(β 0i ′ ). Thus, we can reformulate equation (58) asm j=1 i∈Pj exp(β 0i ) exp (β 1i ) ⊤ X + ασ(τ (β 1i ) ⊤ X h(X, η i ) = m j=1 i∈Pj exp(β * 0i ) exp (β * 1i ) ⊤ X + ασ(τ (β * 1i ) ⊤ X h(X, η * i ),for almost every X. Recall that β 1i = β * 1i and β 0i = β * 0i for any i ∈ [L], then the above equation leads to {η i : i ∈ P j } ≡ {η * i : i ∈ P j }, for almost every X for any j ∈ [m]. As a consequence,G = m j=1 i∈Pj exp(β 0i )δ (β1i,ηi) = m j=1 i∈Pj exp(β * 0i )δ (β * 1i ,η * i ) = G * .Hence, we reach the conclusion of this proposition.Recently, the MoE model has been employed to mitigate catastrophic forgetting in continual learning. For example, [58] focused on continual learning in vision-language models by adapting a pre-trained vision-language model to new tasks through learning a mixture of specialized adapter modules.[58] introduced an MoE structure onto a frozen CLIP, utilizing a mixture of adapters to modify the MLP block after the MSA layer. In contrast, our work centers on general continual learning with pre-trained models, leveraging the inherent MoE architecture of MSA layers. Consequently, our MoE model placement differs from that of [58] . By employing prefix tuning, we demonstrate that it is analogous to introducing new prefix experts to scale and adapt these pre-trained MoE models to downstream tasks. Furthermore, while [58] utilizes task-specific routers, our approach employs task-specific prompts that encapsulate both task-specific router and expert parameters.The parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks [18] . To improve the parameter efficiency of the final model, we can integrate this mechanism in the proposed method. Specifically, each head in the MSA layers comprises N MoE models, where N is the length of the input sequence. This allows for a dynamic routing mechanism to enhance parameter efficiency. For instance, [18] proposed a dynamic routing strategy that adaptively adjusts the number of activated experts based on the input. The computation for any MoE model's gating is directly correlated with the corresponding row in the attention matrix, which encapsulates the MoE model's score functions. For example, selecting the top k experts via Top-K routing in the i-th MoE model is equivalent to identifying the top k largest values in the i-th row of the attention matrix. To implement [18] , we first sort the elements in the i-th row from highest to lowest, then find the smallest set of experts whose cumulative probability exceeds the threshold. Consequently, unselected experts remain inactive, reducing the need to compute all elements of the value matrix within self-attention.In this appendix, we outline the detailed algorithm of HiDe-Prompt, utilizing the same notation as in Section 2.Each previously encountered class c ∈ Y (i) , i = 1, . . . , t -1 has its instructed and uninstructed representations approximated by Gaussian distributions, denoted as G c and Ĝc , respectively.HiDe-Prompt maintains an expandable pool of task-specific prompts e t , each optimized for a specific task D t using a cross-entropy loss within the WTP objective. To prevent forgetting, previous prompts e 1 , . . . , e t-1 remain frozen. Knowledge transfer across tasks is facilitated by a prompt ensemble (PE) strategy: the current prompt is initialized with the last one e t ← e t-1 and refined using a weighted combination of all past prompts p t = α t-1 i=1 e i + (1 -α)e t , where α is a hyper-parameter. Notably, HiDe-Prompt incorporates contrastive regularization within the WTP objective, pushing features of the new task away from those of past tasks represented by the prototypes of old class distributions G c . Let H t = {f θ (x (t) i , p t )| i = 1, . . . , N t } be the embedding transformation of D t and µ c be the mean of G c . The contrastive loss can be written asEQUATION)where τ is the temperature that is set to 0.8. The overall objective function of WTP for learning a new task t is defined as for c ∈ Y (t) do 4:EQUATIONObtain Ĝc from f θ and D t ▷ Uninstructed Representations 5:if t > 1 then 6:Initialize e t ← e t-1Construct Optimize p t and ψ with L WTP in Eq.( 60) ▷ Within-Task Prediction 12:p t = α t-1 i=1 e i + (1 -α)e t 8:Optimize ω with L TII in Eq.( 62) ▷ Task-Identity Inference 13:Optimize ψ with L TAP in Eq.( 61) ▷ Task-Adaptive Prediction 14:for c ∈ Y (t) do 15:Obtain G c from f θ , p t and D t ▷ Instructed Representations return (p 1 , . . . , p T , ω, ψ)where λ is a hyper-parameter. Following WTP, HiDe-Prompt performs a further refinement step on the output layer parameters ψ using a separate objective called task-adaptive prediction (TAP). TAP addresses potential classifier bias by considering the Gaussian distribution of all classes encountered so far. The final output layer h ψ can be further optimized for TAP objective,EQUATIONwhere H i,c is constructed by sampling an equal number of pseudo representations from G c for c ∈ Y (i) and i = 1, . . . , t.For TII, HiDe-Prompt leverages a lightweight auxiliary output layer ĥω : R D → R T , to map uninstructed representations directly to task identity. This mapping is learned explicitly through a cross-entropy loss function,EQUATIONwhere Ĥc is constructed by sampling an equal number of pseudo representations from Ĝc for c ∈ Y (i) and i = 1, . . . , t. Please refer to Algorithm 1 for more details.Datasets. We use commonly-used datasets in the field of continual learning, including , MNIST [24] , Fashion-MNIST [55] , SVHN [31] , and notMNIST [3] . Each of these datasets is treated as a separate incremental task, permitting for the assessment of the effects of significant variations between tasks.Prompt-Based Approaches. We compare NoRGa against recent prompt-based continual learning approaches: L2P [54] , DualPrompt [53] , CODA-Prompt [42] , S-Prompt [52] and HiDe-Prompt [49] . To ensure a fair comparison, we replicate these methods using the configurations reported in their respective papers. S-Prompt in the original paper trains a separate prompt and classifier head for each task. At evaluation, it infers domain id as the nearest centroid obtained by applying K-Means on the training data. To adapt S-Prompt to CIL, we use one common classifier head for all tasks. For NoRGa, we adopt the same configuration as HiDe-Prompt, which utilizes Prefix Tuning [26] as its prompt-based methodology. Learnable scalar factors α and τ are frozen after the first task's training to mitigate catastrophic forgetting. We further optimize NoRGa by selecting the best non-linear activation function σ via cross-validation among tanh, sigmoid, and GELU.Evaluation Metric. We employ three common metrics to measure the performance the methods, including final average accuracy (FA), cumulative average accuracy (CA), and average forgetting measure (FM). Let S i,t denote the accuracy on the i-th task after learning the t-th task, and A t represent the average accuracy as A t = 1 T -1 i=1 max t∈{1,...,T -1} (S i,t -S i,T ). It's worth noting that FA and CA are prioritized over FM, as they inherently encompass both plasticity and forgetting, with FM providing supplementary context [42] .Implementation Details. We train and test on one NVIDIA A100 GPU for baselines and our method. We leverage a pre-trained ViT-B/16 model as the backbone. Training employs an Adam optimizer (β 1 = 0.9, β 2 = 0.999), a batch size of 128, and a constant learning rate of 0.005 for all methods except CODA-Prompt. CODA-Prompt utilizes a cosine decaying learning rate starting at 0.001. Additionally, a grid search technique was implemented to determine the most appropriate number of epochs for effective training.Because HiDe-Prompt optimizes prompt parameters specifically for within-task prediction (WTP), NoRGa inherently aligns with this objective, leading to generally better continual learning performance. We demonstrate this improvement through experiments in a task-incremental learning setting, where task labels are available during inference (as in Table 4 ). While HiDe-Prompt performs well, NoRGa shows consistent improvement across all scenarios. Notably, NoRGa with sigmoid activation achieves the highest final average accuracy in both Split CIFAR-100 and Split CUB-200 with Sup-21K training. Additionally, NoRGa demonstrates its effectiveness even with self-supervised pre-training, further solidifying its advantage over the original prefix tuning model. Overall, NoRGa variants outperform HiDe-Prompt on both datasets and under both training conditions.As the advantages of different parameter-efficient fine-tuning (PEFT) methods remain an open question, we briefly describe them through our revealed connection between self-attention and MoE. Prefix tuning introduces additional parameters at the input of MSA layers to adapt the pre-trained model representation, contrasting with Adapter [16] , which insert adaptive parameters between layers, often replacing MLP blocks. LoRA [17] approximates weight updates with low-rank matrices and adds them to the backbone weights. Our work shows that the MSA layer in a pre-trained model can be seen as a pre-trained MoE architecture. Applying LoRA to the MSA layer refines both the pre-trained experts and their corresponding score functions for downstream tasks. In contrast, prefix tuning expands the pre-trained MoE models by incorporating new experts while preserving the original components, rather than modifying the pre-trained experts like LoRA.NoRGa emerges as a simple, parameter-efficient fine-tuning method and can be regarded as a distinct implementation of prompts. However, our novel perspective on the interplay between self-attention, prefix tuning, and mixture of experts enables us to theoretically substantiate the effectiveness of NoRGa as shown in Section 4.For empirical comparison, we integrate the framework of HiDe-Prompt with different PEFT techniques and Sup-21K weights, evaluating performance on Split CIFAR-100 and Split CUB-200. The results are summarized in Table 5 . The table shows that NoRGa consistently outperforms the other PEFT methods on both datasets, suggesting its effectiveness. Nevertheless, further investigation with LoRA and Adapter would be necessary to draw more definitive conclusions.While exploring alternative PEFT methods might offer improvements in WTP performance, these approaches lack theoretical guarantees and could lead to an increased number of parameters. In contrast, our NoRGa method modifies the original score functions of prefix tuning to enhance WTP performance with theoretical rigor. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL due to memory constraints.Previous works have demonstrated that utilizing pre-trained models (PTM) significantly enhances performance for continual learning, often surpassing the performance of non-PTM-based methods. Moreover, studies have shown that first-task adaptation and simple PEFT-style tuning can achieve competitive performance [20, 37, 60, 29] with prompt-based methods. For instance, [20] demonstrated that appending a nearest class mean (NCM) classifier to a ViT model's feature outputs, can serve as a strong baseline. [37, 60] enhanced this strategy by adapting the pre-trained model to the first task using the three PEFT methods for transformer networks [60] and the FiLM method for CNNs [37] . Additionally, [29] improved NCM by incorporating second-order statistics-covariance and Gram matrices. However, these methods, which fine-tune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features. Our work focuses on continually adapting the backbone, utilizing task-specific prompts to consistently capture emerging To validate our approach, we compare it against state-of-the-art PTM-based continual learning methods, including ADAM [60] and RanPAC [29] , using Split CIFAR-100 and Split CUB-200 datasets. The results are summarized in Table 6 . In comparison to other PTM-based continual learning methods, NoRGa demonstrates competitive performance across both evaluated datasets. For instance, on Split CIFAR-100, NoRGa achieves an FA of 94.48%, exceeding the next best method by over 2%. Similarly, on Split CUB-200, NoRGa delivers strong results relative to other baselines. These improvements highlight the effectiveness of our method in mitigating catastrophic forgetting and preserving knowledge retention across multiple tasks.We compare the validation loss of NoRGa and HiDe-Prompt throughout the first task on Split CUB-200, as illustrated in Figure 3 . The results demonstrate that NoRGa consistently outperforms HiDe-Prompt throughout the training process. This empirical evidence supports the theoretical advantages of NoRGa over HiDe-Prompt.As described in Section 4.1 and Appendix E, in our framework, α and τ are learnable hyperparameters and optimized through backpropagation during the first task, eliminating the need for manual tuning. Additionally, our theoretical analysis of NoRGa's statistical efficiency in Section 4.2 holds for any values of α and τ , demonstrating the theoretical robustness. To further investigate, we evaluated both fixed and learnable settings for these hyperparameters. For the fixed case, we set their values to 1. We report FA on Split CUB-200 and Split CIFAR-100 with Sup-21K weights. The results are summarized in Table 7 . Although performance slightly decreased with fixed hyperparameters, it still outperforms HiDe-Prompt, indicating our method's empirical robustness.We utilize a single A100 GPU for all experiments. The training times are summarized in Table 8 . While NoRGa exhibits slightly longer training times compared to HiDe-Prompt, it consistently achieves significantly better performance. This demonstrates the effectiveness of NoRGa while maintaining competitive training efficiency.NeurIPS Paper Checklist 


Decision: Prior to rebuttal, the reviewers raised concerns regarding:

1. lack of good connection to the continual nature of CL, specifically, addressing distribution shifts in the proposed approach and the proposed theoretical grounding
2. comparison to PEFT-based CL methods
3. comparison to MoE based CL methods
4. parameter efficiency
5. limited improvement in some of the experiments
6. missing baselines
7. clarity.

the authors seem to have addressed all concerns in their rebuttal, all reviewers either increased or kept the acceptance scores.
In light of the above AC recommends acceptance and urges the authors to incorporate all the reviewer comments and subsequent discussions in the final verios of the paper.

Reviews: review: paper_summary: paper_summary: The topic of this paper is about the prompt-based continual learning. The authors give a theoretical analysis on these prompt-based continual learning methods, and utilize a Mixture-of-Expert (MoE) architecture characterized by linear experts and quadratic gating score functions. They develop a gating mechanism Non-linear Residual Gates (NoRGa) for MoE-based continual learning. The proposed method has been evaluated on several benchmarks.
strength_weakness: strength_weakness: -	Comparison to different Parameter-Effiecient-FineTuning methods [b] (e.g. adapter) is needed. This paper mainly focuses on the theoretical analysis of prompt-based continual learning methods.  Prompt-based continual learning belongs to PEFT methods and also add new parameters for new tasks. Can the authors further analyze the advantages of prompt-based methods theroretically?
-	The parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method?
-	Important MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted.
-	Performance improvement on some datasets is limited. For example, FA metric (75.06%->75.40%) on Split ImageNet-R, CA metric (95.02%->95.11%) on Split-CIFAR-100 in Table 1. What are the running times of each experiment? Can the authors provide the performance variance of experimental results (e.g. the accuracy at the last incremental learning session)?

[a] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR2024

[b] Continual Learning with Pre-Trained Models: A Survey, IJCAI2024

[c] Harder Tasks Need More Experts: Dynamic Routing in MoE Models, arxiv2024
questions: questions: My major concerns are included in the above weaknesses.
limitations: limitations: The authors have addressed the limitations.
paper_summary: paper_summary: This paper introduces an extension to prefix tuning, by introducing non-linear residual gating - a simple extension over existing prefix tuning. This non-linear residual gating is supported with theoretical efficiency guarantees (polynomial versus exponential) to better estimate the optimal parameters. When applied to the problem of rehearsal-free continual learning with pretrained models (over four standard benchmarks) on top of the HiDe-Prompt method, the suggested gating provides consistent and partly significant improvements.
strength_weakness: strength_weakness: My biggest issue with this work is its placement as a Continual Learning contribution, and evaluating it as such.

__[1]__ Firstly, the proposed approach, whilel helping with WTP, is simply an extension on top of prefix tuning - and none of the contributions connect to the continual distribution shift nature. The improved performance is much more likely tied to an improvement in the underlying PEFT approach; which in itself is simply sufficient on standard benchmarks (see Zhou et al. 2023). This is further supported by the fact that the authors freeze both alpha and tau after the first task (see supplementary D). Moreover, on such small benchmarks, introducing two additional tunable hyperparameters does incorporate additional degrees of freedom to overfit to these benchmarks, and makes direct comparison to existing methods difficult.

Consequently, the question needs to be answered: How robust is NoRGa with respect to the additional introduced alpha and temperature? These are strictly two hyperparameters more than HiDe-Prompt. As such, how much more compute went into hyperparameter optimization for NoRGa w.r.t. Hide? And given the same search budget, how does HiDe-Prompt compare relatively? This is the most crucial aspect, as the other method comparisons are somewhat redundant, since HiDe-Prompt introduces orthogonal classifier head alignment which can be applied to all other methods as well.

> It would be great if the authors could address both the placement as a continual learning contribution (as opposed to a parameter-efficient finetuning paper, which would need to be evaluated on other respective benchmark tasks), and the comparability to HiDe-Prompt.

__[2]__ This paper also only tackles prompt-based rehearsal-free continual learning, but misses discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning, such as: [1] Jason et al. 2022, “A simple baseline that questions the use of pretrained models for continual learning”, [2] Panos et al 2023, “First session adaptation: A strong replay-free baseline for class-incremental learning”, [3] Zhou et al. 2023, “Revisiting class-incremental learning with pretrained models: Generalizability and adaptivity are all you need”, [4] McDonnell et al. 2023: “RanPAC: Random projections and pretrained models for continual learning”. 

> Given that these works show that existing prompt-style CL objectives are matched by simply training on the first task, and how performance is matchable with simple parameter-efficient finetuning without explicitly accounting for the continual nature of the problem, how should the insights in this paper be rated w.r.t. These works?

__[3]__ A vast part of this paper deals with the sample in-efficiency of standard prefix tuning mechanisms. However, the authors do not provide any efficiency tests (e.g. how does performance vary as a function of training iterations / samples seen), and only provide slightly improved performances on a few smaller-scale benchmarks. This is somewhat unfortunate, as it would be great to see how the efficiency estimates (e.g. Eq. 16) connect to practical convergence behaviour (e.g. L288 “polynomial number of data versus exponential number of data”).

__[4]__ Both statistical efficiency proofs also do not take into account the continual nature of the problem, but rather assume a single generative model and i.i.d. Samples from said model, which are presented to the learner. This fully disconnects from the actual continual nature and sequential distribution shifts expected in such problems. Similarly, I’m having general troubles to connect the assumptions made throughout the proofs to the practical nature of the problem. E.g. why is Eq. 13 a reasonable assumption for the CL problem? Is convergence behaviour of a LS regression estimator suitably connected to practical convergence behaviour? If so, why? If not, why not? Similarly, why would I want to encourage algebraic independence - what would this give within the actual continual learning scenario?

---

__Smaller issues:__

* I find it somewhat difficult to follow the proofs in 4.2 and Appendix A. Providing a proof-sketch as an overview would significantly help readability.
* It is a bit problematic to have the entire proof of convergence, without even a proof sketch, in the supplementary-  as it motivates the entire section 4 and the non-linear residual gating. It would be great if the authors could provide at the very least a proof sketch in the main paper.
* L139/140 is fairly hard to parse.
questions: questions: All relevant questions are incorporated in the previous section.

I am currently leaning towards rejection - not because of the proposed method and the theoretical support (both of which I believe are sensible and meaningful), but rather its placement and consequent evaluation as a continual learning method. Neither in the methodological nature NOR the actually conducted proofs do the author account for the continual nature of the problem. 
Instead, I believe that this paper is much better suited as a contribution in the domain of parameter-efficient model finetuning. However, this in turn requires different benchmark evaluations.

Moreover, I am uncertain about the significance of the report results (see above).

Together, I would be happy to raise my score if the authors can address these particular concerns (as well as those listed above).
limitations: limitations: The authors do not discuss limitations or societal impact explicitly.
paper_summary: paper_summary: The paper explores the theoretical underpinnings and practical implications of prompt-based methods in continual learning, aiming to enhance our understanding and optimize their effectiveness. It introduces a novel perspective by connecting prefix tuning with mixture of experts models, revealing insights into how self-attention mechanisms encode specialized architectures. The proposed Non-linear Residual Gates (NoRGa) further advances this by improving within-task prediction accuracy and overall continual learning performance while maintaining parameter efficiency.
strength_weakness: strength_weakness: 1. The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework. 
2. While NoRGa is presented as a novel gating mechanism, its explicit connection to prompts could be elaborated further. Is it reasonable to interpret NoRGa's gating mechanism as a variant or extension of prompts in a broader sense?
questions: questions: 1. Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals?
2. Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts.
3. Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?
limitations: limitations: The paper's reliance on dense theoretical explanations without adequate visual aids may limit its accessibility. Visual representations could significantly enhance comprehension of complex concepts like the relationship between prompts and NoRGa, potentially broadening the paper's audience and impact.
paper_summary: paper_summary: The paper titled "Mixture of Experts Meets Prompt-Based Continual Learning" explores the integration of prompt-based continual learning methods with mixture of experts (MoE) architectures. The paper proposes a novel gating mechanism called Non-linear Residual Gates (NoRGa) to enhance the performance of prompt-based continual learning by leveraging theoretical insights and empirical evidence.
strength_weakness: strength_weakness: 1. Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method.

2. It may be better to use a additional graph to represent the final method NoRGa of the paper.

3. It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.
questions: questions: Which dataset is the ViT-B/16 pre-trained on.
limitations: limitations: This paper has already discussed the limitations of our work in the conclusion section.

Comments: title: Thank you
Response to Response
Thank you
Any other questions?
Thank you
Rebuttal Response
Official Comment by Reviewer MKaE
Re: Rebuttal
comment: Thank you for your reply. We sincerely appreciate your endorsement and the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. Following your suggestion, we will include additional baselines and stronger evaluation comparisons in our revision.

Best regards,

Authors
I thank the authors for commenting on my previous response, and providing the additional experimental pointers.

The additional experiments comparing different PEFT methods within the directly comparable HiDE framework, and the comparison to first-stage adaptation baselines provide good initial pointers of the strong relative performance of NorGA.

While I still believe that NorGA is primarily a parameter-efficient finetuning contribution, the strong performance within an established CL method is quite convincing; though only evaluated on two datasets. I fully understand that the rebuttal timeframe does not provide a lot of opportunities for extended experimental studies, but would love to see a more extensive comparison at least for the final submission.

Still, I have consequently decided to raise my score and to recommend acceptance.
We would like to thank the reviewer for rating a positive score of 6. We are happy to discuss more if the reviewer still has questions.

Best regards,

Authors
Dear Reviewer dPHR,

We thank you again for your valuable and constructive comments. We already responded to your additional questions. Given that the discussion deadline is approaching (less than 36 hours), can you let us know if our responses sufficiently address your concerns and whether you still have any other questions about the paper?

We thank you again for the valuable feedback.

Best,

The Authors
We would like to thank the reviewer for rating a positive score of 6. We are happy to discuss more if the reviewer still has questions.

Best regards,

Authors
Thank the authors for their response. Since the authors have addressed all of my concerns, I decided to increase my score.
I have read the authors' response and my concerns have been addressed. I raise my rating to 6.
We thank the reviewer for maintaining the positive score “6” of the paper. Regarding the question of whether our proposed method effectively addresses core challenges in CL, particularly catastrophic forgetting, we wish to emphasize that our paper is built on the theory that improved WTP is both a necessary and sufficient condition for enhanced CL performance. This theory, which is supported by prior CL research [1] and the original HiDe-Prompt paper, is fundamental to connecting our contribution to continual learning from the perspective of continous learning.

While exploring alternative Parameter-Efficient Fine-Tuning (PEFT) methods or incorporating more complex expert models might offer improvements in WTP, these approaches lack theoretical guarantees and could lead to an increased number of parameters. In contrast, our NoRGa method modifies the original score functions of prefix tuning to enhance WTP performance with theoretical rigor. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL due to memory constraints.

We will incorporate the above discussion in the final version. If you have any further questions, please let us know.

[1] A theoretical study on solving continual learning, NeurIPS 2022.
Thank you for the author's positive response. 
Most of my doubts have been resolved, but in fact I have the same doubts as Reviewer dPHR about "the placement of this work as a Continual Learning contribution". I hope the author will seriously consider whether the proposed method can truly solve the core problems such as catastrophic forgetting from the perspective of continuous learning. In addition, the experimental settings are also compared with some popular PTM-based continuous learning methods. 
Nevertheless, I also appreciate that this work rethinks the relationship between MoE and prompt from a novel and meaningful perspective, so I still maintain the original score.
Thank you for your valuable and constructive comments. Here we explain your additional questions as below:

We agree that NoRGa can serve as a simple, parameter-efficient fine-tuning technique. However, our contributions extend beyond this by offering a novel perspective on the interplay between self-attention, prefix tuning, and the mixture of experts. This offers a fresh viewpoint on the design of previous prompt-based continual learning methods. Furthermore, this relationship enables us to theoretically substantiate the effectiveness of NoRGa.

While we acknowledge that other PEFT methods, such as LoRA and adapters, could be explored to optimize WTP performance, the relative advantages of different PEFT approaches remain an open question. In contrast, NoRGa not only demonstrates empirical superiority over prefix tuning but also offers theoretical guarantees of enhanced performance. Notably, our theoretical framework imposes no assumptions on the pretrained weights, rendering our method robust across various pretrained models. We have addressed a comparison of different PEFT methods in our response to Reviewer cCSa. Specifically, we employed the HiDe-Prompt framework with various PEFT techniques and Sup-21K weights, evaluating performance using FA on Split CIFAR-100 and Split CUB-200. The results are as follows:

|Method| Split CIFAR-100| Split CUB-200|
|--------------|-----------------|---------------|
|HiDe-Prompt|92.61|86.56|
|HiDe-LoRA|92.71|87.37|
|HiDe-Adapter|92.73|87.10|
|NoRGa|__94.48__|__90.90__|

These results highlight the effectiveness of our proposed method relative to other PEFT techniques, although further research is necessary to draw more definitive conclusions. Additionally, we compared our method with a first task-adaptation based method [3], with results summarized below:

| Method| Split CIFAR-100 | Split CUB-200 |
|----------------|-----------------|------------------|
| ADAM + VPT-D [3]| 85.04 | 85.28 |
| ADAM + SSF [3]| 85.27  | 85.67 |
| ADAM + Adapter [3]| 87.29  | 85.84 |
| NoRGa| __94.48__ | __90.90__ |

As illustrated, NoRGa achieves the highest performance across both datasets. For instance, on Split CIFAR-100, NoRGa attains an FA of 94.48%, surpassing the next best method by over 7%. This substantial improvement underscores the efficacy of our proposed method in addressing catastrophic forgetting and preserving knowledge across multiple tasks.

We thank you again for the valuable feedback. If you have any further questions, please let us know.
I thank the authors for their detailed feedback, which has helped clarified some confusion I had - particularly regarding the placement of the proposed approach in the continual learning literature. Some questions however still remain:

* If I understand the reply correctly, this paper directly builds on top of HiDE-Prompt; but does not modify any of the CL components (referring to elements that explicitly account for the continual nature; i.e. the task-specific prompts); but with an explicit focus on improving the WTP (within-task performance). But how is what the authors propose then not just simple parameter-efficient finetuning - simply applied to simplistic CL benchmarks on top of HiDE-Prompt?

* Consequently, following Zhou et al. 2023 [3]: What would happen if other PEFT approaches are used to optimize the WTP performance? Or more generally, what does the comparison to first task adaptation sanity checks look like (which are void of meaningful CL elements)? Or a comparison directly to e.g. Zhou et al? Just stating that _"However, these methods, which finetune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features."_ may not be sufficient.