Title: Mixture of Experts Meets Prompt-Based Continual Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Continual Learning

Pre-training

Attention

Prompt Tuning

Mixture of Experts
Abstract: Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms. Our code is publicly available at https://github.com/Minhchuyentoancbn/MoE_PromptCL.

Mixture of Experts Meets Prompt-Based Continual Learning

Abstract

Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms. Our code is publicly available at https://github.com/Minhchuyentoancbn/MoE_PromptCL .

1 Introduction

Humans possess a remarkable ability to learn continuously by integrating new skills and knowledge while retaining past experiences. However, current AI models often fail to retain this ability. Unlike humans, they often suffer from catastrophic forgetting [28, 30, 32, 38], a phenomenon where they struggle to retain knowledge from previous tasks while learning new ones. Inspired by human learning, Continual Learning [2, 28, 43, 1, 12] is an ongoing field that aims to train a model across a sequence of tasks while mitigating this challenge. Traditional continual learning methods often rely on storing past data for fine-tuning, which can raise concerns about memory usage and privacy [5, 39, 51]. To address these limitations, prompt-based approaches have emerged as a promising alternative within rehearsal-free continual learning. By attaching prompts - small sets of learnable parameters - to a frozen pre-trained model, these approaches enable efficient adaptation to new tasks with minimal modifications to the underlying model [56, 26, 61]. The effectiveness of prompt-based methods has been demonstrated by several recent works achieving state-of-the-art performance on various continual learning benchmarks [49, 53, 54].

While prompt-based methods have demonstrably achieved impressive results, their emphasis largely lies on prompt utility, leaving a gap in our theoretical comprehension of their effectiveness. This

absence of a theoretical foundation hinders our ability to further refine and optimize these methods. In this work, we offer a new perspective by focusing on prefix tuning [26] and its connection to mixture of experts models [19, 15, 13, 11]. We demonstrate that self-attention blocks in Vision Transformers [8] implicitly encode a special mixture of experts architecture, revealing a surprising connection between these seemingly disparate concepts. Leveraging this connection, we propose that applying prefix tuning within pre-trained models can be interpreted as introducing new experts. The newly introduced experts collaborate with the pre-trained experts, facilitating efficient adaptation of the model to new tasks.

Drawing insights from this analysis, we observe that the original prefix tuning suffers from suboptimal sample efficiency, requiring a substantial amount of data for reasonable parameter estimation. To address this challenge, we propose a novel gating mechanism termed Non-linear Residual Gates (NoRGa). This architecture integrates non-linear activation functions and residual connections within the gating score functions. Our work focuses on improving within-task prediction accuracy, a key component of continual learning performance as identified in previous research [22, 49]. We posit that NoRGa can enhance this aspect, which contributes to improved overall continual learning performance while maintaining parameter efficiency. We further provide theoretical justification for this improvement, demonstrating how NoRGa accelerates parameter estimation rates.

Our contributions can be summarized as follows: (1) We reveal a novel connection between self-attention and a mixture of experts, providing a fresh perspective on prompt-based continual learning approaches; (2) Leveraging this insight, we propose Non-linear Residual Gates (NoRGa) , an innovative gating mechanism that enhances continual learning performance while maintaining parameter efficiency, and provide a theoretical justification for this improvement; (3) Extensive experiments across various continual learning benchmarks and pre-training settings demonstrate that our approach achieves state-of-the-art performance compared to existing methods.

Notation. For any n ∈ N , we denote [ n ] as the set { 1 , 2 , . . . , n } . Next, for any set S , we let | S | stand for its cardinality. For any vector u := ( u 1 , u 2 , . . . , u d ) ∈ R d and α := ( α 1 , α 2 , . . . , α d ) ∈ N d , we let u α = u α 1 1 u α 2 2 . . . u α d d , | u | := u 1 + u 2 + . . . + u d and α ! := α 1 ! α 2 ! . . . α d ! , while ∥ u ∥ stands for its 2 -norm value. Lastly, for any two positive sequences { a n } n ≥ 1 and { b n } n ≥ 1 , we write a n = O ( b n ) or a n ≲ b n if a n ≤ Cb n for all n ∈ N , where C > 0 is some universal constant. The notation a n = O P ( b n ) indicates that a n /b n is stochastically bounded.

2 Background and Related Works

We first provide background and related works on continual learning. Then, we define the attention mechanism, followed by discussions on prompt-based continual learning and mixture of experts.

̸

Continual Learning (CL) addresses the challenge of training a model incrementally on a sequence of T tasks, denoted by D = {D 1 , ..., D T } . Each task's training data D t = { ( x ( t ) i , y ( t ) i ) } N t i =1 contains pairs of input sample x ( t ) i ∈ X ( t ) , and corresponding label y ( t ) i ∈ Y ( t ) . Notably, the class labels are distinct for each task, i.e., Y ( t ) ⋂ Y ( t ′ ) = ∅ , ∀ t = t ′ . Consider a neural network with a backbone function f θ and an output layer h ψ . The model predicts a label ˆ y = h ψ ( f θ ( x )) ∈ Y = ⋃ T t =1 Y ( t ) , where x ∈ X = ⋃ T t =1 X ( t ) is an unseen test sample from arbitrary tasks. Importantly, during training on a new task, the model can only access the current data, without access to data from previous tasks. Prior approaches often rely on storing past task samples for training on new tasks, raising concerns regarding storage and privacy [5, 6, 39, 51, 59].

Our work focuses on the class-incremental learning (CIL) setting, where task identities are not provided during inference, unlike in task-incremental learning (TIL) [46]. A recent theory by [22] analyzes the CIL objective by decomposing the probability of a test sample x of the j -th class in task t into two probabilities:

where the first term involves within-task prediction (WTP) and the second term pertains to taskidentity inference (TII). This equation highlights that by improving either the WTP performance or the TII, we can consequently improve the overall CIL performance, as shown in [22, 49].

Attention Mechanism. Within the Transformer architecture, the attention mechanism plays a crucial role. One prevalent variant is scaled dot-product attention[47], formally defined as follows:

Definition 2.1 (Scaled Dot-Product Attention) . Let K ∈ R N × d k be a key matrix with N key vectors, and V ∈ R N × d v be a value matrix with N corresponding value vectors. Given a query matrix Q ∈ R M × d k , Attention over ( K , V ) is defined as

where the softmax function acts on the rows of matrix QK ⊤ ∈ R M × N .

Vision Transformer (ViT) [8] employs the same attention mechanism within multiple Multi-head Self-Attention (MSA) layers, which is formally defined as follows:

Definition 2.2 (Multi-head Self-Attention Layer) . Let X Q , X K , X V denote the input query, key, and value matrix, respectively, where X Q = X K = X V = [ x 1 , ..., x N ] ⊤ ∈ R N × d , and N is the length of the input sequence. The output is expressed as

where W O ∈ R md v × d , W Q i ∈ R d × d k , W i K ∈ R d × d k , and W V i ∈ R d × d v are projection matrices, and m is the number of heads in the MSA layer. In ViTs, they use d k = d v = d/m .

Prompt-based continual learning. Prompt-based approaches have emerged as a promising alternative within rehearsal-free continual learning [61, 52, 44]. In vision tasks, prompt-based methods often leverage a pre-trained ViT as a feature extractor f θ , with its parameters θ typically frozen. These methods enhance the model by introducing prompts , small sets of learnable parameters that influence the operations of the MSA layer [53]. Prompts are strategically injected into the query, key, and value matrices to guide the ViT in learning new tasks. We denote the prompt parameters by p ∈ R L p × d , where L p is the sequence length and d is the embedding dimension. Previous work [53] outlines two main prompt-based approaches: Prompt Tuning (ProT) [25] and Prefix Tuning (PreT) [26]. While Prompt Tuning directly concatenates the same prompt parameter p to the query, key, and value, prefix tuning divides p into prefixes { p K , p V } ∈ R Lp 2 × d and appends it to the key and value vectors:

Existing prompt-based methods in CL address catastrophic forgetting by creating new adaptive prompts for each new task. During testing, the model chooses suitable prompt combinations to handle unseen data from any encountered task [49]. L2P [54] proposes a shared prompt pool for all tasks, utilizing a query-key mechanism for prompt selection. Instead of using the same prompt pool across tasks, DualPrompt [53] introduces G-Prompt and E-Prompt to capture task-agnostic and task-specific information, respectively. S-Prompt [52] focuses on learning task-specific prompts and employs a ProT strategy similar to L2P. CODA-Prompt [42] expands the prompt pool across tasks and performs a weighted summation of the prompt pool using attention factors. A recent work, HiDe-Prompt [49], achieves state-of-the-art performance by introducing a new hierarchical decomposition of CIL objectives and optimizing each component for better performance.

In this study, we focus on prefix tuning as our primary prompt-based methodology and follow the framework presented in HiDe-Prompt [49]. During training, HiDe-Prompt co-optimizes task-specific prompts p t and model's output layer parameters ψ for each new task t using the WTP objective. These prompts are stored within a prompt pool P = { p 1 , ..., p T } . At test time, a separate lightweight auxiliary output layer ˆ h ω : R D → R T , trained with the TII objective, takes the uninstructed representation f θ ( x ) of a new data point x as input to infer the task identity, guiding the selection of the most suitable prompt p k from the prompt pool P . Subsequently, the final prediction is given as ˆ y = h ψ ( f θ ( x , p k )) . For further details, please refer to Appendix D.

Mixture of experts (MoE) extends classical mixture models with an adaptive gating mechanism [19, 21]. An MoE model consists of a group of N expert networks f i : R d → R d v , for all i ∈ [ N ] , and a gate function G : R d → R N . Given an input h ∈ R d , MoE computes a weighted sum of expert outputs f i ( h ) based on learned score function s i : R d → R for each expert:

𝑄

𝐾

𝑉

𝑋

𝑊𝑄

𝑊𝐾

𝑊𝑉

transpose

softmax

𝑖

𝑗

score function

𝑠𝑖,𝑗

expert

𝑓 𝑗

Each row in the value matrix

is an expert

𝑓 𝑗

Each entry in the

attention matrix is a

score function

𝑠𝑖,𝑗

where G ( h ) := softmax( s 1 ( h ) , . . . , s N ( h )) . Building on this concept, works by [10, 41] established the MoE layer as a fundamental building block to scale up model capacity efficiently. Please refer to Appendix C for a comprehensive discussion of related works.

3 Connection between Prefix Tuning and Mixture of Experts

We first explore the relationship between attention and mixture of experts in Section 3.1, followed by establishing the connection between prefix tuning and the mixture of experts in Section 3.2.

3.1 Mixture of Experts Meets Attention

Following the notation established in Definition 2.2, let's consider the l -th head within the MSA layer. Let X = [ x ⊤ 1 , . . . , x ⊤ N ] ⊤ ∈ R Nd , which is the concatenation of input sequence embeddings into a single one-dimensional vector. We define the matrix E i ∈ R d × Nd such that E i X := x i for all i ∈ [ N ] . Furthermore, we introduce an MoE architecture consisting of a group of N expert networks f j : R Nd → R d v , N gating functions G i : R Nd → R N with the score function for the j -th expert of the i -th gating s i,j : R Nd → R , where

for i and j ∈ [ N ] . From equation (4), we can express the output of the l -th head as follows:

for i ∈ [ N ] . Expanding on equation (8), we can discern that each attention head within the MSA layer implicitly embodies a special mixture of experts architecture. This architecture encompasses N MoE models, each featuring its own quadratic gating function G i . However, instead of employing N 2

Expert 1

Expert 2

Expert 3

Expert 4

Expert 5

Attention matrix

New prefix

experts

Gating function

Value matrix

Value

vectors

Prefix

vectors

+

𝜎

Residual

connection

Non-linear

activation

score function

for pretrained

experts

score function

of original prefix

tuning

score function of

NoRGa

separate expert networks for each model, this architecture utilizes N shared linear expert networks f j for j ∈ [ N ] , significantly reducing the number of parameters. Notably, each expert network and its corresponding gating function process the entire input sequence directly, rather than individual embedding x i as in traditional MoE layers [41]. This connection between self-attention and mixture of experts is depicted in Figure 1. In the subsequent section, we explore how prompt-based techniques can be viewed through this lens.

3.2 Prefix Tuning via the Perspective of Mixture of Experts

Building on the connection between self-attention and mixture of experts, we propose that applying prefix tuning can be interpreted as the introduction of new experts to customize the pre-trained model for a specific task, as illustrated in Figure 2. Specifically, similar to Section 3.1, we consider the l -th head within the MSA layer. We denote p K = [ p K 1 , . . . , p K L ] ⊤ ∈ R L × d , p V = [ p V 1 , . . . , p V L ] ⊤ ∈ R L × d , where L = L p 2 . We define new prefix experts f N + j : R Nd → R d v along with their corresponding new score functions s i,N + j : R Nd → R as follows:

for i ∈ [ N ] and j ∈ [ L ] . Then from equation (5), the output of the l -th head can be expressed as:

It's worth noting that W Q l , W l K , and W V l remain fixed, with only p K and p V being learnable. By examining equation (8) and equation (11), we can interpret each head in a multi-head self-attention layer within a pre-trained model as a mixture of experts architecture with pre-trained experts f j and gating score functions s i,j for i and j ∈ [ N ] . Prefix tuning extends this MoE by introducing L additional prefix experts f N + j ′ defined by prefix vectors p V j ′ and linear score functions s i,N + j ′ for i ∈ [ N ] and j ′ ∈ [ L ] . These new experts collaborate with the pre-trained ones within the MoE model, facilitating the model's adaptation to downstream tasks.

We argue that our introduction of a novel connection between self-attention, prefix tuning, and MoE offers a fresh perspective on the design of previous prompt-based continual learning methods. In the context of continual learning, the pre-trained experts serve as a knowledge base, while prefix tuning augments it with task-specific knowledge encoded in new experts. Moreover, we draw a parallel between the pre-trained experts and the G(eneral)-Prompt utilized in DualPrompt, which captures

task-agnostic information [53]. Both are shared across tasks, making them useful for prediction, especially when task identification is incorrect. Notably, the new experts achieve their efficiency through simple linear gating functions and independence from the input, unlike the pre-trained experts. For simplicity, we call the MoE model (11) as linear gating prefix MoE .

Statistical suboptimality. The connection between prefix tuning and the MoE within the linear gating prefix MoE model (11) allows us to theoretically explore the statistical behavior of the prefix tuning. In Appendix A, by interpreting the linear gating prefix MoE as a regression problem with sample size n , we demonstrate that the convergence rate for estimating the model parameters, e.g., prompts, can be as slow as O (1 / log τ ( n )) where τ > 0 is some constant. This suggests that a huge amount of data is required to achieve reasonable parameter estimation in the linear gating prefix MoE model, which can be discouraging in practice. To address this statistical limitation, the next section introduces a novel non-linear residual gating score function to replace the linear gating function.

4 Non-linear Residual Gate Meets Prefix Tuning

As discussed earlier, prefix tuning introduces additional experts within the MoE framework, resulting in the linear gating prefix MoE model. However, as outlined in Appendix A, this approach suffers from suboptimal sample efficiency for parameter estimation. To overcome this and enhance overall CIL performance, we propose an innovative approach that significantly improves sample efficiency while promoting WTP performance in Section 4.1 and provide theoretical explanations in Section 4.2.

4.1 NoRGa: Non-linear Residual Gate

We propose a simple yet effective modification to the linear gating prefix MoE model by incorporating non-linear activation and residual connection within the score functions of prefix experts as follows:

where α, τ ∈ R are learnable scalar factors, and σ is a non-linear activation function. The new score function in equation (12) consists of a linear and a non-linear component. We call the new prefix MoE model with score functions (12) as non-linear residual gating prefix MoE .

The use of a non-linear activation function here is motivated by the algebraic independence condition in Definition 4.2 to theoretically guarantee the optimal sample efficiency of expert and parameter estimations (cf. Theorem 4.3). It's worth noting that removing the linear component s i,N + j ( X ) in the score function (12) could potentially lead to the vanishing gradient problem during training. To mitigate this challenge, we incorporate a residual connection [14] into the formulation. Our modification introduces minimal additional parameters ( α and τ ) compared to the original score function, ensuring parameter efficiency. This is particularly crucial in continual learning scenarios where the number of parameters grows with each new task. For implementation, we define H l = W Q l W l K ⊤ . From equation (5), the attention matrix of the l -th head can then be written as:

Here, A prompt l denotes the attention score matrix for the prompts, and A pretrain l represents the attention score matrix for the pre-trained experts. To implement NoRGa, we can directly modify the final attention matrix as follows:

The implementation of NoRGa is illustrated in Figure 2. Despite its simplicity, our modification can significantly enhance sample efficiency and promote more reasonable parameter estimation, as demonstrated in our theoretical analysis in Section 4.2. Within the HiDe-Prompt framework, task-specific prompt parameters are trained using the WTP objective for each new task. Consequently, our modification leads to better parameter estimation, which directly contributes to improved WTP performance, ultimately improving overall continual learning efficacy. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL because of the memory constraint. Here, we evaluated σ with tanh , sigmoid , and GELU , finding tanh to perform well in most cases.

4.2 Theoretical Explanation for Non-linear Residual Gating Prefix MoE

Similar to the setting in Appendix A, we prove that estimating parameters in the non-linear residual gating prefix MoE model (12) is statistically efficient in terms of the number of data. To provide a fair comparison to the linear gating prefix MoE, we focus only on the first head and its first row, namely, l = 1 and i = 1 in equation (12). Then, we proceed to provide a theoretical justification of our claim by viewing this row as an output of a regression setting. In particular, we assume that ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . , ( X n , Y n ) ∈ R Nd × R are i.i.d. samples generated from model:

where ε 1 , . . . , ε n are independent Gaussian noise variables such that E [ ε i | X i ] = 0 and Var( ε i | X i ) = ν 2 for all 1 ≤ i ≤ n . Additionally, we assume that X 1 , X 2 , . . . , X n are i.i.d. samples from some probability distribution µ . The regression function g G ∗ ( · ) in equation (16) then takes the form of a non-linear residual gating prefix MoE model with N pre-trained experts and L unknown experts,

where T ( X ) := ∑ N k =1 exp( X ⊤ B 0 k X + c 0 k ) + ∑ L k ′ =1 exp(( β ∗ 1 k ′ ) ⊤ X + ασ ( τ ( β ∗ 1 k ′ ) ⊤ X ) + β ∗ 0 k ′ ) , G ∗ := ∑ L j ′ =1 exp( β ∗ 0 j ′ ) δ ( β ∗ 1 j ′ ,η ∗ j ′ ) denotes a mixing measure, i.e., a weighted sum of Dirac measures δ , associated with unknown parameters ( β ∗ 1 j ′ , β ∗ 0 j ′ , η ∗ j ′ ) L j ′ =1 in R Nd × R × R q . Here, the matrix B 0 j is equivalent to ( E ⊤ 1 W Q 1 W K 1 ⊤ E j / √ d v ) in the score function s 1 ,j ( X ) , and the vector β ∗ 1 j ′ corresponds to the vector ( E ⊤ 1 W Q 1 W K 1 ⊤ p K j ′ / √ d v ) in ˆ s 1 ,N + j ′ ( X ) . Furthermore, the experts h ( X , η 0 j ) and h ( X , η ∗ j ′ ) represent f j ( X ) and f N + j ′ ( X ) , respectively. In our formulation, for the generality of the ensuing theory, we consider general parametric forms of the experts h ( X , η 0 j ) and h ( X , η ∗ j ′ ) , i.e., we do not only constrain these expert functions to be the forms of the simple experts in the aforementioned model. Similar to the setting in Appendix A, B 0 j , c 0 j , and the expert parameters η 0 j are known. Our goal is to estimate the unknown prompt-related parameters β ∗ 1 j ′ , β ∗ 0 j ′ , and η ∗ j ′ .

Least squares estimation. We will use the least squares method [45] to estimate the unknown parameters ( β ∗ 0 j ′ , β ∗ 1 j ′ , η ∗ j ′ ) L j ′ =1 or, equivalently, the ground-truth mixing measure G ∗ . In particular, we take into account the estimator

where we denote G L ′ (Θ) := { G = ∑ ℓ i =1 exp( β 0 i ) δ ( β 1 i ,η i ) : 1 ≤ ℓ ≤ L ′ , ( β 0 i , β 1 i , η i ) ∈ Θ } as the set of all mixing measures with at most L ′ atoms. In practice, since the true number of experts L is typically unknown, we assume that the number of fitted experts L ′ is sufficiently large, i.e., L ′ > L .

To begin with, we explore the convergence behavior of the regression estimator g ̂ G n ( · ) to the true regression function g G ∗ ( · ) under the L 2 ( µ ) -norm in the following theorem:

Theorem 4.1 (Regression Estimation Rate) . Equipped with a least squares estimator ̂ G n given in equation (18) , the model estimation g ̂ G n ( · ) converges to the true model g G ∗ ( · ) at the following rate:

Proof of Theorem 4.1 is in Appendix B.1. The bound (19) implies that the rate for estimating the regression function g G ∗ ( · ) is of order O P ( √ log( n ) /n ) , which is parametric on the sample size n . More importantly, it also indicates that if there exists a loss function among parameters L such that ∥ g ̂ G n -g G ∗ ∥ L 2 ( µ ) ≳ L ( ̂ G n , G ∗ ) , then we would obtain the bound L ( ̂ G n , G ∗ ) = O P ( √ log( n ) /n ) , which leads to the desired parameter and expert estimation rates.

We now turn our attention to the parameter and expert estimation problems. To understand how the non-linear residual gating affects these problems, we analyze the properties of the expert h ( · , η ) and the activation function σ ( · ) to determine which formulations will achieve favorable performance.

Definition 4.2 (Algebraic independence) . We say that an expert function h ( · , η ) and an activation function σ ( · ) are algebraically independent if they are twice differentiable w.r.t their parameters, and if for any k ≥ 1 and pair-wise distinct parameters ( β 11 , η 1 ) , . . . , ( β 1 k , η k ) , the following set of functions in X is linearly independent for almost every X ∈ R Nd :

Intuitively, the algebraic independence condition ensures that there will be no interactions among parameters of the expert function h ( · , η ) and the activation function σ ( · ) . Technically, a key step in our argument is to decompose the regression discrepancy g ̂ G n ( X ) -g G ∗ ( X ) into a combination of linearly independent terms by applying Taylor expansions to the product of the softmax's numerator and the expert function, i.e., exp( β ⊤ 1 X + ασ ( τβ ⊤ 1 X )) h ( X , η ) . Thus, the above condition guarantees that all the derivative terms in the Taylor expansion are linearly independent. To exemplify the algebraic independence condition, we consider the following simple examples of the expert functions h ( · , η ) and the activation σ ( · ) that are algebraically independent.

Example. When the expert function h ( · , η ) is formulated as a neural network h ( X , ( a, b )) = φ ( a ⊤ X + b ) with the activation φ ( · ) ∈ { ReLU( · ) , GELU( · ) , z ↦→ z p } , where ( a, b ) ∈ R Nd × R , and the activation function σ ( · ) is one among the functions sigmoid( · ) , tanh( · ) , GELU( · ) , then they satisfy the algebraic independence condition in Definition 4.2.

Finally, we establish the rates for estimating parameters and experts in the non-linear residual gating prefix MoE model in Theorem 4.3. Prior to presenting the theorem statement, let us design a loss function among parameters based on a notion of Voronoi cells [27], which is a commonly employed approach for the convergence analysis of expert estimation in MoE models [36, 34, 35, 33], yet tailored to the setting of this paper. In particular, the V oronoi loss used for our analysis is defined as

where we denote ∆ β 1 ij ′ := β 1 i -β ∗ 1 j ′ and ∆ η ij ′ := η i -η ∗ j ′ . Above, V j ′ ≡ V j ′ ( G ) , for j ′ ∈ [ L ] , is a Voronoi cell associated with the mixing measure G generated by the true component ω ∗ j := ( β ∗ 1 j ′ , η ∗ j ′ ) , which is defined as follows:

̸

where we denote ω i := ( β 1 i , η i ) as a component of G . Note that, the cardinality of each Voronoi cell V j ′ indicates the number of components ω i of G approximating the true component ω ∗ j ′ of G ∗ . Additionally, since L 1 ( G,G ∗ ) = 0 if and only if G ≡ G ∗ , it follows that when L 1 ( G,G ∗ ) becomes sufficiently small, the differences ∆ β 1 ij ′ and ∆ η ij ′ are also small. This observation indicates that, although L 1 ( G,G ∗ ) is a proper metric as it is not symmetric, it is an appropriate loss function for measuring the discrepancy between the least square estimator ̂ G n and the true mixing measures G ∗ .

Theorem 4.3. Assume that the expert function h ( x, η ) and the activation σ ( · ) are algebraically independent, then we achieve the following lower bound for any G ∈ G L ′ (Θ) :

which together with Theorem 4.1 indicates that L 1 ( ̂ G n , G ∗ ) = ˜ O P ( n -1 / 2 ) .

Proof of Theorem 4.3 is in Appendix B.2. A few comments on Theorem 4.3 are in order: (i) From the bound L 1 ( ̂ G n , G ∗ ) = ˜ O P ( n -1 / 2 ) , we deduce that the estimation rates for the over-specified parameters β ∗ 1 j ′ , η ∗ 1 j ′ , where j ′ ∈ [ L ] : |V j ′ | > 1 , are all of order O P ( 4 √ log( n ) /n ) . Since the expert h ( · , η ) is twice differentiable over a bounded domain, it is also a Lipschitz function. Thus, denote

̂ G n := ∑ L n i =1 exp( ̂ β 0 i ) δ ( ̂ β n 1 i , η n i ) , we achieve that

The above bound indicates that if the experts h ( · , η ∗ j ) are fitted by at least two other experts, then their estimation rates are also of order O P ( 4 √ log( n ) /n ) ; (ii) For exactly-specified parameters β ∗ 1 j ′ , η ∗ j ′ , where j ′ ∈ [ L ] : |V j ′ | = 1 , the rates for estimating them are faster than those of their over-specified counterparts, standing at order O P ( √ log( n ) /n ) . By arguing similarly to equation (22), the experts h ( · , η ∗ j ′ ) also enjoy the faster estimation rate of order O P ( √ log( n ) /n ) , which is parametric on the sample size n ; (iii) It follows from the above rates that we only need a polynomial number of data (roughly ϵ -4 where ϵ is the desired approximation error) to estimate the parameters and experts of the non-linear residual gating prefix MoE. By contrast, when using the linear gating, as being demonstrated in Appendix A, it requires an exponential number of data. This highlights the statistical benefits of using the non-linear residual gating MoE model over the linear gating prefix MoE model.

5 Experiments

Datasets. We evaluate various continual learning methods on widely used CIL benchmarks, including Split CIFAR-100 [23] and Split ImageNet-R [23], consistent with prior work [49]. We further explore the model's performance on fine-grained classification tasks with Split CUB-200 [48] and large inter-task differences with 5-Datasets [9]. Please refer to Appendix E for more details.

Evaluation Metrics. We utilize several established metrics described in [50]. These include: final average accuracy (FA), which represents the average accuracy after the final task; cumulative average accuracy (CA), which refers to the historical average accuracy; and average forgetting measure (FM). We give more emphasis to FA and CA due to their comprehensiveness, as noted in [42].

Baselines. We compare our approach with several representative prompt-based approaches including L2P [54], DualPrompt [53], CODA-Prompt [42], S-Prompt [52], and HiDe-Prompt [49]. Additionally,

we evaluate against state-of-the-art pre-trained model-based continual learning methods, including ADAM [60] and RanPAC [29]. We further extend our evaluation by applying HiDe-Prompt with parameter-efficient fine-tuning techniques like LoRA [17] and Adapters [16]. In line with [49], we utilize the checkpoints of ViT that use supervised pre-training of Imagenet-21K (denoted as Sup-21K), and some self-supervised pre-training such as iBOT-21K, iBOT-1K [62], DINO-1K [4], and MoCo-1K [7]. For implementation details, see Appendix E.

Main Results. In Table 1, we evaluate several continual learning methods on Split CIFAR-100 and Split ImageNet-R using diverse pre-trained models. NoRGa achieves state-of-the-art FA and CA across all datasets and models, consistently outperforming HiDe-Prompt. On Sup-21K, NoRGa demonstrates impressive FA results on both CIFAR-100 and ImageNet-R. It also maintains the highest CA, with significant margins of 1.80% and 2.92% on CIFAR-100 and ImageNet-R, respectively, compared to HiDe-Prompt. These results highlight NoRGa's strong ability to retain knowledge and exhibit minimal forgetting, as evidenced by the low FM values on both datasets. NoRGa also surpasses HiDe-Prompt on self-supervised models, with FA improvements up to 1.95% and 3.66%. We further investigate two scenarios: fine-grained classification tasks and large inter-task differences through experiments on Split CUB-200 and 5-Datasets, respectively, as shown in Table 2. NoRGa maintains its lead, achieving FA gaps of 4.34% and 2.46% on Split CUB-200, and the highest FA on 5-Datasets. While gains in some metrics may be modest, NoRGa consistently outperforms HiDe-Prompt in either FA or CA, underscoring its robustness. For example, on Split ImageNet-R with Sup-21K weights, the FA improvement is small (75.06% vs. 75.40%), but the CA gains are substantial (76.60% vs. 79.52%), demonstrating the method's effectiveness and robustness.

Ablation Study. To assess the impact of non-linear activation functions on NoRGa's performance, we evaluated the model's behavior with different choices for the activation function σ , including tanh , sigmoid , and GELU in Table 3. The results show that NoRGa achieves state-of-the-art performance on both Split CIFAR-100 and Split CUB-200 datasets with all three activation functions. These findings suggest that NoRGa exhibits robustness to the choice of non-linear activation within a reasonable range. While all functions perform well, the tanh activation function demonstrates generally strong performance across scenarios. Further results are provided in the Appendix.

6 Conclusion

This paper presents an initial exploration of self-attention and prefix-tuning through the lens of mixture of experts. We find that applying prefix tuning can be viewed as introducing new prefix experts to adapt the pre-trained model. However, limitations in sample efficiency exist. We address this by proposing NoRGa, a novel gating mechanism to enhance continual learning performance. Our results demonstrate NoRGa's effectiveness both theoretically and empirically. While the current implementation of the expert network prioritizes simplicity, future research directions could involve investigating more intricate architectures. Furthermore, the choice of activation functions in our work requires fine-tuning, which opens avenues for future research on adaptively learning activation.

References

R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3366-3375, 2017.

E. Belouadah, A. Popescu, and I. Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks , 135:38-54, 2021.

Y. Bulatov. Notmnist dataset. Google (Books/OCR), Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html , 2, 2011.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

H. Cha, J. Lee, and J. Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9516-9525, October 2021.

A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. S. Torr, and M. Ranzato. On tiny episodic memories in continual learning, 2019.

X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers, 2021.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021.

S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach. Adversarial continual learning. arXiv preprint arXiv:2003.09553 , 2020.

D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in a deep mixture of experts. In ICLR Workshops , 2014.

Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, Z. Wang, et al. M 3 vit: Mixtureof-experts vision transformer for efficient multi-task learning with model-accelerator co-design. Advances in Neural Information Processing Systems , 35:28441-28457, 2022.

N. L. Hai, T. Nguyen, L. N. Van, T. H. Nguyen, and K. Than. Continual variational dropout: a view of auxiliary local variables in continual learning. Machine Learning , 113(1):281-323, 2024.

H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, L. Hong, and E. Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems , 34:29335-29347, 2021.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.

N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for gaussian mixtures of experts. Journal of Machine Learning Research , 23(323):1-81, 2022.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Q. Huang, Z. An, N. Zhuang, M. Tao, C. Zhang, Y. Jin, K. Xu, L. Chen, S. Huang, and Y. Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07652 , 2024.

R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation , 3, 1991.

P. Janson, W. Zhang, R. Aljundi, and M. Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428 , 2022.

M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , 6(2):181-214, 1994.

G. Kim, C. Xiao, T. Konishi, Z. Ke, and B. Liu. A theoretical study on solving continual learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 5065-5079. Curran Associates, Inc., 2022.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059. Association for Computational Linguistics, Nov. 2021.

X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.

T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14979-15006. PMLR, 17-23 Jul 2022.

M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems , 36, 2024.

S. V. Mehta, D. Patil, S. Chandar, and E. Strubell. An empirical investigation of the role of pre-training in lifelong learning. Journal of Machine Learning Research , 24(214):1-50, 2023.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011.

C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto. Toward understanding catastrophic forgetting in continual learning, 2019.

H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML , 2024.

H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations , 2024.

H. Nguyen, N. Ho, and A. Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICML , 2024.

H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems , 2023.

A. Panos, Y. Kobe, D. O. Reino, R. Aljundi, and R. E. Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18820-18830, 2023.

H. Phan, A. P. Tuan, S. Nguyen, N. V. Linh, and K. Than. Reducing catastrophic forgetting in neural networks via gaussian mixture approximation. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 106-117. Springer, 2022.

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021.

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR) , 2017.

J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11909-11919, June 2023.

Q. Tran, H. Phan, K. Than, D. Phung, and T. Le. Continual learning with optimal transport based mixture model. arXiv preprint arXiv:2211.16780 , 2022.

Q. Tran, L. Tran, K. Than, T. Tran, D. Phung, and T. Le. Koppa: Improving prompt-based continual learning with key-query orthogonal projection and prototype-based one-versus-all. arXiv preprint arXiv:2311.15414 , 2023.

S. van de Geer. Empirical processes in M-estimation . Cambridge University Press, 2000.

G. M. van de Ven, T. Tuytelaars, and A. S. Tolias. Three types of incremental learning. Nature Machine Intelligence , 4:1185-1197, 2022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset . California Institute of Technology, 2011.

L. Wang, J. Xie, X. Zhang, M. Huang, H. Su, and J. Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems , 2023.

L. Wang, X. Zhang, H. Su, and J. Zhu. A comprehensive survey of continual learning: Theory, method and application, 2024.

L. Wang, X. Zhang, K. Yang, L. Yu, C. Li, H. Lanqing, S. Zhang, Z. Li, Y. Zhong, and J. Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations , 2021.

Y. Wang, Z. Huang, and X. Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. In Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision , 2022.

Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, 2022.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.

Y. Xin, S. Luo, H. Zhou, J. Du, X. Liu, Y. Fan, Q. Li, and Y. Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey, 2024.

B. Yu. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam , pages 423-435, 1997.

J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23219-23230, 2024.

G. Zhang, L. Wang, G. Kang, L. Chen, and Y. Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

D.-W. Zhou, Z.-W. Cai, H.-J. Ye, D.-C. Zhan, and Z. Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision , pages 1-21, 2024.

D.-W. Zhou, H.-L. Sun, J. Ning, H.-J. Ye, and D.-C. Zhan. Continual learning with pre-trained models: A survey, 2024.

J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR) , 2022.

Supplement to 'Mixture of Experts Meets Prompt-Based Continual Learning'

In this supplementary material, we first analyze the statistical suboptimality of the Linear Gating Prefix MoE Model (11) in Appendix A. Appendix B provides proofs for the theoretical results presented in Section 4.2. In Appendix C, we discuss related works on mixture of experts. Appendix D outlines the training algorithm for HiDe-Prompt while Appendix E presents the experimental setup and details. Further, Appendix F presents further experiments on the task-incremental learning setting to empirically demonstrate the benefits of using our proposed Non-linear Residual Gating Prefix MoE (12) over the Linear Gating Prefix MoE Model. Appendix G and Appendix H compare NoRGa with other parameter-efficient fine-tuning techniques and pre-trained model-based methods. In Appendix I, we present the efficiency tests, while Appendix J explores the impact of learnable α and τ . Finally, Appendix K compares the training times between NoRGa and HiDe-Prompt.

A Statistical Suboptimality of Linear Gating Prefix MoE Model

In this appendix, we demonstrate that estimating parameters and experts in the linear gating prefix MoE model (11) can be statistically inefficient in terms of the number of data. To simplify our findings, we particularly focus on the first head, namely, l = 1 in equation (11), and the first row of this head, namely, i = 1 in equation (11). Then, we proceed to provide a theoretical justification of our claim for the suboptimality of the linear gating prefix MoE by viewing this row as an output of the regression setting. In particular, we assume that ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . , ( X n , Y n ) ∈ R Nd × R is an i.i.d. sample generated from the following model:

where ε 1 , . . . , ε n are independent Gaussian noise variables such that E [ ε i | X i ] = 0 and Var( ε i | X i ) = ν 2 for all 1 ≤ i ≤ n . Additionally, we assume that X 1 , X 2 , . . . , X n are i.i.d. samples from some probability distribution µ . Motivated by linear gating prefix MoE model (11), the regression function f G ∗ ( · ) in equation (23) admits the form of the linear gating prefix MoE model with pre-trained N experts and L unknown experts, namely

where G ∗ := ∑ L j ′ =1 exp( β ∗ 0 j ′ ) δ ( β ∗ 1 j ′ ,η ∗ j ′ ) denotes a mixing measure, i.e., a weighted sum of Dirac measures δ , associated with unknown parameters ( β ∗ 1 j ′ , β ∗ 0 j ′ , η ∗ j ′ ) L j ′ =1 in R Nd × R × R q . Here, the matrix B 0 j plays the role of the matrix E ⊤ 1 W Q 1 W K 1 ⊤ E j √ d v in the score function s 1 ,j ( X ) . Furthermore, the vector β ∗ 1 j ′ corresponds to the vector E ⊤ i W Q l W l K ⊤ p K j ′ √ d v in the score function s 1 ,N + j ′ ( X ) . Furthermore, the experts h ( X , η 0 j ) correspond to the role of f j ( X ) and h ( X , η ∗ j ′ ) correspond to the role of f N + j ′ ( X ) . In our formulation, we consider general parametric forms of the experts h ( X , η 0 j ) and h ( X , η ∗ j ′ ) , i.e., we do not only constrain these expert functions to be the forms of the simple experts in the linear gating prefix MoE model.

Similar to the linear gating prefix MoE model (11), the matrices B 0 j , the biases c 0 j , and the expert parameters η 0 j are known. Our aim is to estimate the unknown gating parameters β ∗ 1 j ′ , β ∗ 0 j ′ , and η ∗ j ′ that correspond to the prompts.

Least squares estimation: We will use the least squares method [45] to estimate the unknown parameters ( β ∗ 0 j ′ , β ∗ 1 j ′ , η ∗ j ′ ) L j ′ =1 or, equivalently, the ground-truth mixing measure G ∗ . In particular, we take into account the estimator

where we denote G L ′ (Θ) := { G = ∑ ℓ i =1 exp( β 0 i ) δ ( β 1 i ,η i ) : 1 ≤ ℓ ≤ L ′ , ( β 0 i , β 1 i , η i ) ∈ Θ } as the set of all mixing measures with at most L ′ atoms. In practice, since the true number of true experts L is typically unknown, we assume that the number of fitted experts L ′ is sufficiently large, i.e. L ′ > L .

Let us recall that our main objective in this appendix is to show that using the linear gating in the prefix MoE model is not sample efficient. To illustrate that point, we consider a simple scenario when the expert function takes the form h ( X , ( a, b )) = ( a ⊤ X + b ) p , for some p ∈ N . Additionally, we also design a new Voronoi loss function as below to facilitate our arguments.

where we denote ∆ β 1 ij ′ := β 1 i -β ∗ 1 j ′ and ∆ η ij ′ := η i -η ∗ j ′ .

Now, we are ready to state the result of parameter estimation under the linear gating prefix MoE model in the following theorem:

Theorem A.1. Assume that the experts take the form h ( X , ( a, b )) = ( a ⊤ X + b ) p , for some p ∈ N , then we achieve the following minimax lower bound of estimating G ∗ :

for any r ≥ 1 , where E f G indicates the expectation taken w.r.t the product measure with f n G .

There are two main implications of the result in Theorem A.1:

(i) The rates for estimating parameters β ∗ 1 j , a ∗ j and b ∗ j are slower than O P ( n -1 / 2 r ) , for any r ≥ 1 . This means that they are slower than any polynomial rates, and could be of order O P (1 / log( n )) . Using the same reasoning described after equation (22), we have

As a consequence, the rates for estimating experts φ (( a ∗ j ) ⊤ X + b ∗ j ) are no better than those for estimating the parameters a ∗ j and b ∗ j , and could also be as slow as O P (1 / log( n )) .

(ii) The above rates imply that we need an exponential number of data (roughly exp(1 /ϵ τ ) where ϵ is the desired approximation error) to estimate the parameters and experts of the linear gating prefix MoE. This fact demonstrates that using the linear gating in the prefix MoE model is not sample efficient from the perspective of the expert estimation problem.

Proof of Theorem A.1. Prior to presenting the main proof of Proposition A.1, let us introduce the following key result:

Lemma A.2. If the following holds for any r ≥ 1 :

then we obtain that

Proof of Lemma A.2. Indeed, from the Gaussian assumption on the noise variables ϵ i , we obtain that Y i | X i ∼ N ( f G ∗ ( X i ) , σ 2 ) for all i ∈ [ n ] . Next, the assumption in equation (28) indicates for sufficiently small ε > 0 and a fixed constant C 1 > 0 which we will choose later, we can find a mixing measure G ′ ∗ ∈ G L ′ (Θ) such that L 2 ,r ( G ′ ∗ , G ∗ ) = 2 ε and ∥ f G ′ ∗ -f G ∗ ∥ L 2 ( µ ) ≤ C 1 ε . From Le Cam's

lemma [57], as the Voronoi loss function L 2 ,r satisfies the weak triangle inequality, we obtain that

where the second inequality is due to the fact that

By choosing ε = n -1 / 2 , we obtain that ε · exp ( -C 1 nε 2 ) = n -1 / 2 exp( -C 1 ) . As a consequence, we achieve the desired minimax lower bound in equation (29).

Main proof. We need to prove that the following limit holds true for any r ≥ 1 :

For that purpose, it suffices to build a sequence of mixing measures ( G n ) n ≥ 1 such that both L 2 ,r ( G n , G ∗ ) → 0 and

as n →∞ . To this end, we consider the sequence G n = ∑ L +1 i =1 exp( β n 0 i ) δ ( β n 1 i ,a n i ,b n i ) , where

β n 11 = β n 12 = β ∗ 11 and β n 1 i = β n 1( i -1) for any 3 ≤ i ≤ L +1 ;

a n 1 = a n 2 = a ∗ 1 and a n i = a n i -1 for any 3 ≤ i ≤ L +1 ;

As a result, the loss function L 2 ,r ( G n , G ∗ ) is reduced to

which indicates indicates that L 2 ,r ( G n , G ∗ ) → 0 as n →∞ .

Now, we prove that ∥ f G n -f G ∗ ∥ L 2 ( µ ) / L 2 ,r ( G n , G ∗ ) → 0 . For that purpose, let us consider the quantity

For simplicity, let us consider the polynomial degree p = 1 as the arguments for other values of p can be adapted accordingly. Recall from equation (46) that Q n ( X ) can be decomposed as follows:

From the definitions of β n 1 i , a n i and b n i , we can rewrite A n ( X ) as follows:

Additionally, it can also be checked that B n ( X ) = 0 , and C n ( X ) = O ( n -( r +1) ) . Therefore, it follows that C n ( X ) / L 2 ,r ( G n , G ∗ ) → 0 . As a consequence, Q n ( X ) / L 2 ,r ( G n , G ∗ ) → 0 as n →∞ for almost every X .

as n →∞ . Hence, the proof of claim (31) is completed.

B Proof of Theoretical Results

In this appendix, we present rigorous proofs for the theoretical results introduced in Section 4, namely Theorem 4.1 and Theorem 4.3, in that order.

B.1 Proof of Theorem 4.1

For the proof of the theorem, we first introduce some notation. Firstly, we denote by F L ′ (Θ) the set of conditional densities of all mixing measures in G L ′ (Θ) , that is, F L ′ (Θ) := { g G ( X ) : G ∈ G L ′ (Θ) } . Additionally, for each δ > 0 , the L 2 ( µ ) ball centered around the conditional density g G ∗ ( Y | X ) and intersected with the set F L ′ (Θ) is defined as

In order to measure the size of the above set, Geer et. al. [45] suggest using the following quantity:

where H B ( t, F L ′ (Θ , t ) , ∥ · ∥ L 2 ( µ )) stands for the bracketing entropy [45] of F L ′ (Θ , u ) under the L 2 ( µ ) -norm, and t ∨ δ := max { t, δ } . By using the similar proof argument of Theorem 7.4 and Theorem 9.2 in [45] with notations being adapted to this work, we obtain the following lemma:

Lemma B.1. Take Ψ( δ ) ≥ J B ( δ, F L ′ (Θ , δ )) that satisfies Ψ( δ ) /δ 2 is a non-increasing function of δ . Then, for some universal constant c and for some sequence ( δ n ) such that √ nδ 2 n ≥ c Ψ( δ n ) , we achieve that

for all δ ≥ δ n .

We now demonstrate that when the expert functions are Lipschitz continuous, the following bound holds:

for any 0 < ε ≤ 1 / 2 . Indeed, for any function g G ∈ F L ′ (Θ) , since the expert functions are bounded, we obtain that h ( X , η ) ≤ M for all X where M is a bounded constant of the expert functions. Let τ ≤ ε and { π 1 , . . . , π ¯ N } be the ζ -cover under the L ∞ norm of the set F L ′ (Θ) where ¯ N := N ( ζ, F L ′ (Θ) , ∥ · ∥ L ∞ ) is the η -covering number of the metric space ( F L ′ (Θ) , ∥ · ∥ L ∞ ) . Then, we construct the brackets of the form [ L i ( X ) , U i ( X )] for all i ∈ [ ¯ N ] as follows:

From the above construction, we can validate that F L ′ (Θ) ⊂ ∪ ¯ N i =1 [ L i ( X ) , U i ( X )] and U i ( X ) -L i ( X ) ≤ min { 2 ζ, M } . Therefore, it follows that

which implies that ∥ U i -L i ∥ L 2 ( µ ) ≤ 2 ζ . By definition of the bracketing entropy, we deduce that

Therefore, we need to provide an upper bound for the covering number ¯ N . In particular, we denote ∆ := { ( β 1 , β 0 ) ∈ R Nd × Nd × R Nd × R : ( β 1 , β 0 , η ) ∈ Θ } and Ω := { η ∈ R q : ( β 1 , β 0 , η ) ∈ Θ } . Since Θ is a compact set, ∆ and Ω are also compact. Therefore, we can find ζ -covers ∆ ζ and Ω ζ for ∆ and Ω , respectively. We can check that

For each mixing measure G = ∑ L ′ i =1 exp( β 0 i ) δ ( β 1 i ,η i ) ∈ G L ′ (Θ) , we consider other two mixing measures:

Here, η i ∈ Ω ζ such that η i is the closest to η i in that set, while ( β 1 i , β 0 i ) ∈ ∆ ζ is the closest to ( β 1 i , β 0 i ) in that set. From the above formulations, we get that

Here, the second inequality occurs as the softmax weight is bounded by one, and the third inequality follows from the fact that the expert h ( X , · ) is a Lipschitz function with some Lipschitz constant L 1 ( X ) > 0 . Next, let us denote

Then, we have

Now, we will bound two terms in the above right hand side. Firstly, since both the input space X and the parameter space Θ are bounded, we have that

As a result, we deduce that

Regarding the second term, note that

Since both the input space and the parameter space are bounded, we have

which yields that

From equations (36), (37) and (38), we obtain that ∥ g ˇ G -g G ∥ L ∞ ≲ ζ . According to the triangle inequality, we have

By definition of the covering number, we deduce that

Combine equations (35) and (39), we achieve that

Let ζ = ε/ 2 , then we obtain that

As a result, it follows that

Let Ψ( δ ) = δ · [log(1 /δ )] 1 / 2 , then Ψ( δ ) /δ 2 is a non-increasing function of δ . Furthermore, equation (40) indicates that Ψ( δ ) ≥ J B ( δ, F L ′ (Θ , δ )) . In addition, let δ n = √ log( n ) /n , then we get that √ nδ 2 n ≥ c Ψ( δ n ) for some universal constant c . Finally, by applying Lemma B.1, we achieve the desired conclusion of the theorem.

B.2 Proof of Theorem 4.3

Our goal is also to demonstrate the following inequality:

For that purpose, we divide the proof of the above inequality into local and global parts in the sequel.

Local part: In this part, we demonstrate that

Assume by contrary that the above claim is not true, then there exists a sequence of mixing measures G n = ∑ L i =1 exp( β n 0 i ) δ ( β n 1 i ,η n i ) in G L ′ (Θ) such that L 1 n := L 1 ( G n , G ∗ ) → 0 and

as n →∞ . Let us denote by V n j := V j ( G n ) a Voronoi cell of G n generated by the j -th components of G ∗ . Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i.e., V j = V n j . Thus, the Voronoi loss L 1 n can be represented as

where we denote ∆ β n 1 ij := β n 1 i -β ∗ 1 j and ∆ η n ij := η n i -η ∗ j .

Since L 1 n → 0 , we get that ( β n 1 i , η n i ) → ( β ∗ 1 j , η ∗ j ) and ∑ i ∈V j exp( β n 0 i ) → exp( β ∗ 0 j ) as n →∞ for any i ∈ V j and j ∈ [ L ] . Now, we divide the proof of the local part into three steps as follows:

Step 1 - Taylor expansion. In this step, we would like to decompose the quantity

into a combination of linearly independent elements using Taylor expansion. In particular, the quantity Q n ( X ) is decomposed as follows:

Decomposition of A n ( X ) . Let us denote E ( X ; β 1 ) := exp( β ⊤ 1 X + ασ ( τβ ⊤ 1 X )) , then A n can be separated into two terms as follows:

By means of the first-order Taylor expansion, we have

where R n, 1 ( X ) is a Taylor remainder such that R n, 1 ( X ) / L 1 n → 0 as n →∞ , and

On the other hand, by applying the second-order Taylor expansion, we get that

in which R n, 2 ( X ) is a Taylor remainder such that R n, 2 ( X ) / L 1 n → 0 as n →∞ .

Decomposition of B n . Recall that we have

By invoking first-order and second-order Taylor expansions to B n, 1 ( X ) and B n, 2 ( X ) , it follows that

where we define

Additionally, R n, 3 ( X ) and R n, 4 ( X ) are Taylor remainders such that R n, 3 ( X ) / L 1 n → 0 and R n, 3 ( X ) / L 1 n → 0 as n →∞ .

Collect the above results together, we can represent Q n ( x ) as

where we define S n,j, 0 d × d , 0 q = T n,j, 0 d × d = ∑ i ∈V j exp( β n 0 i ) -exp( β ∗ 0 j ) for any j ∈ [ L ] .

Step 2 - Non-vanishing coefficients. In this step, we demonstrate that at least one among ratios of the forms S n,j,α 1 ,α 2 / L 1 n and T n,j,ℓ / L 1 n goes to zero as n tends to infinity. Indeed, assume by contrary that

for any j ∈ [ L ] , 0 ≤ | α 1 | , | α 2 | , | ℓ | ≤ 2 . Then, we get

Now, we consider indices j ∈ [ L ] such that its corresponding Voronoi cell has only one element, i.e. |V j | = 1 .

For arbitrary u, v ∈ [ Nd ] , let α 1 ∈ N Nd × Nd and α 2 = 0 q such that α ( uv ) 1 = 1 while other entries equal to zero. Then, we have 1 L 1 n · ∑ i ∈V j exp( β n 0 i ) | (∆ β n 1 ij ) ( uv ) | = | S n,j,α 1 ,α 2 | / L 1 n → 0 as n → ∞ . By taking the summation of the previous term with u, v ∈ [ Nd ] , we achieve that 1 L 1 n ∑ i ∈V j exp( β n 0 i ) ∥ ∆ β n 1 ij ∥ 1 → 0 . Owing to the topological equivalence between norm-1 and norm-2, it follows that

For arbitrary u ∈ [ Nd ] , let α 1 = 0 Nd × Nd and α 2 ∈ N q such that α ( u ) 2 = 1 while other entries equal to zero. Then, we get 1 L 1 n · ∑ i ∈V j exp( β n 0 i ) | (∆ η n ij ) ( u ) | = | S n,j,α 1 ,α 2 | / L 1 n → 0 as n →∞ . By taking the summation of the previous term with u ∈ [ q ] , we achieve that 1 L 1 n ∑ i ∈V j exp( β n 0 i ) ∥ ∆ η n ij ∥ 1 → 0 , or equivalently,

Combine the limits in equations (49) and (50), we obtain that

as n →∞ .

Next, we consider indices j ∈ [ L ] such that its corresponding Voronoi cell has more than one element, i.e. |V j | > 1 .

For arbitrary u, v ∈ [ Nd ] , let α 1 ∈ N Nd × Nd and α 2 = 0 q such that α ( uv ) 1 = 2 while other entries equal to zero. Then, we have 1 L 1 n · ∑ i ∈V j exp( β n 0 i ) | (∆ β n 1 ij ) ( uv ) | 2 = | S n,j,α 1 ,α 2 | / L 1 n → 0 as n → ∞ . By taking the summation of the previous term with u, v ∈ [ Nd ] , we achieve that

For arbitrary u ∈ [ Nd ] , let α 1 = 0 Nd × Nd and α 2 ∈ N q such that α ( u ) 2 = 2 while other entries equal to zero. Then, we get 1 L 1 n · ∑ i ∈V j exp( β n 0 i ) | (∆ η n ij ) ( u ) | 2 = | S n,j,α 1 ,α 2 | / L 1 n → 0 as n →∞ . By taking the summation of the previous term with u ∈ [ q ] , we achieve that

Putting the limits in equations (49) and (50), we have

as n →∞ . Taking the summation of three limits in equations (48), (51) and (54), we deduce that 1 = L 1 n / L 1 n → 0 as n →∞ , which is a contradiction. Thus, at least one among ratios of the forms S n,j,α 1 ,α 2 / L 1 n and T n,j,ℓ / L 1 n goes to zero as n tends to infinity.

Step 3 - Application of Fatou's lemma. In this step, we show that all the ratios S n,j,α 1 ,α 2 / L 1 n and T n,j,ℓ / L 1 n go to zero as n → ∞ , which contradicts to the conclusion in Step 2. In particular, by denoting m n as the maximum of the absolute values of those ratios. From the result of Step 2, it follows that 1 /m n ̸→∞ .

Recall from the hypothesis in equation (43) that ∥ g G n -g G ∗ ∥ L 2 ( µ ) / L 1 n → 0 as n → ∞ , which indicates that ∥ g G n -g G ∗ ∥ L 1 ( µ ) / L 1 n → 0 . Therefore, by applying the Fatou's lemma, we get that

This result implies that 1 m n L 1 n · [ g G n ( X ) -g G ∗ ( X )] → 0 as n → ∞ for µ -almost surely X . Looking at the formulation of Q n ( X ) in equation (45), since the term [ ∑ k 0 i ′ =1 exp( X ⊤ A 0 i ′ X + c 0 i ′ )+ ∑ k ∗ j ′ =1 exp(( β ∗ 1 j ′ ) ⊤ X + σ (( β ∗ 1 j ′ ) ⊤ X ) + β ∗ 0 j ′ ) ] is bounded, we deduce that the term Q n ( X ) m n L 1 n → 0 for µ -almost surely X .

Let us denote

with a note that at least one among them is non-zero. Then, from the decomposition of Q n ( X ) in equation (47), we have

for µ -almost surely X . It is worth noting that the term ∂ | α 1 | E ∂β α 1 1 ( X ; β ∗ 1 j ) · ∂ | α 2 | h ∂η α 2 ( X ; η ∗ j ) can be explicitly expressed as

When | α 1 | = 0 , | α 2 | = 0 : exp(( β ∗ 1 j ) ⊤ X + σ (( β ∗ 1 j ) ⊤ X )) h ( X ; η ∗ j ) ;

When | α 1 | = 1 , | α 2 | = 0 : X ( u ) ( 1 + σ ′ (( β ∗ 1 j ) ⊤ X ) ) exp(( β ∗ 1 j ) ⊤ X + σ (( β ∗ 1 j ) ⊤ X )) h ( X ; η ∗ j ) ;

When | α 1 | = 1 , | α 2 | = 1 :

When | α 1 | = 2 , | α 2 | = 0 :

Recall that the expert function h satisfies the condition in Definition 4.2, i.e. the set

is linearly independent for almost every X . Therefore, we obtain that ϕ j,α 1 ,α 2 = φ j,ℓ = 0 for all j ∈ [ L ] , 0 ≤ | α 1 | + | α 2 | , | ℓ | ≤ 1 + 1 {|V j | > 1 } . This result turns out to contradict the fact that at least one among them is different from zero. Hence, we achieve the inequality in equation (42).

Global part. It is worth noting that the inequality (42) suggests that there exists a positive constant ε ′ such that

Therefore, it is sufficient to prove that

Assume by contrary that the inequality (55) does not hold true, then we can find a sequence of mixing measures G ′ n ∈ G L ′ (Θ) such that L 1 ( G ′ n , G ∗ ) > ε ′ and

which indicates that ∥ g G ′ n -g G ∗ ∥ L 2 ( µ ) → 0 as n →∞ . Recall that Θ is a compact set, therefore, we can replace the sequence G ′ n by one of its subsequences that converge to a mixing measure G ′ ∈ G L ′ (Ω) . Since L 1 ( G ′ n , G ∗ ) > ε ′ , we deduce that L 1 ( G ′ , G ∗ ) > ε ′ .

Next, by invoking the Fatou's lemma, we have that

Thus, we get that g G ′ ( X ) = g G ∗ ( X ) for µ -almost surely X . From the identifiability property of the non-linear residual gating prefix MoE (cf. the end of this proof), we deduce that G ′ ≡ G ∗ . Consequently, it follows that L 1 ( G ′ , G ∗ ) = 0 , contradicting the fact that L 1 ( G ′ , G ∗ ) > ε ′ > 0 . Hence, the proof is completed.

Identifiability of Non-linear Residual Gating MoE.

We now prove the identifiability of the non-linear residual gating prefix MoE. In particular, we will show that if g G ( X ) = g G ∗ ( X ) for almost every X , then it follows that G ≡ G ∗ .

For ease of presentation, let us denote

where

Since g G ( X ) = g G ∗ ( X ) for almost every X , we have

̸

As the expert function h satisfies the conditions in Definition 4.2, the set { h ( X , η ′ i ) : i ∈ [ k ′ ] } , where η ′ 1 , . . . , η ′ k ′ are distinct vectors for some k ′ ∈ N , is linearly independent. If L ′ = L , then there exists some i ∈ [ L ′ ] such that η i = η ∗ j for any j ∈ [ L ] . This implies that ∑ L ′ j =1 softmax G ( ( β 1 j ) ⊤ X + ασ ( τ ( β 1 j ) ⊤ X ) + β 0 j ) · h ( X , η j ) = 0 , which is a contradiction. Thus, we must have that L = L ′ . As a result,

̸

for almost every X . WLOG, we may assume that softmax G ( ( β 1 j ) ⊤ X + ασ ( τ ( β 1 j ) ⊤ X ) + β 0 j ) = softmax G ∗ ( ( β ∗ 1 j ) ⊤ X + ασ ( τ ( β ∗ 1 j ) ⊤ X ) + β ∗ 0 j ) ,

for almost every X for any j ∈ [ L ] . Since the softmax function is invariant to translation, this result indicates that β 1 j = β ∗ 1 j and β 0 j = β ∗ 0 j + v 0 for some v 0 ∈ R for any j ∈ [ L ] . Recall from the universal assumption that β 0 L ′ = β 0 L = 0 , we get that β 0 j = β ∗ 0 j for any j ∈ [ L ] . Then, equation (56) can be rewritten as

̸

for almost every X . Next, we denote P 1 , P 2 , . . . , P m as a partition of the index set [ L ] , where m ≤ L ′ , such that exp( β 0 i ) = exp( β ∗ 0 i ′ ) for any i, i ′ ∈ P j and j ∈ [ L ] . On the other hand, when i and i ′ do not belong to the same set P j , we let exp( β 0 i ) = exp( β 0 i ′ ) . Thus, we can reformulate equation (58) as

for almost every X . Recall that β 1 i = β ∗ 1 i and β 0 i = β ∗ 0 i for any i ∈ [ L ] , then the above equation leads to

for almost every X for any j ∈ [ m ] . As a consequence,

Hence, we reach the conclusion of this proposition.

C Discussion of related Mixture of Experts works

Recently, the MoE model has been employed to mitigate catastrophic forgetting in continual learning. For example, [58] focused on continual learning in vision-language models by adapting a pre-trained vision-language model to new tasks through learning a mixture of specialized adapter modules. [58] introduced an MoE structure onto a frozen CLIP, utilizing a mixture of adapters to modify the MLP block after the MSA layer. In contrast, our work centers on general continual learning with pre-trained models, leveraging the inherent MoE architecture of MSA layers. Consequently, our MoE model placement differs from that of [58]. By employing prefix tuning, we demonstrate that it is analogous to introducing new prefix experts to scale and adapt these pre-trained MoE models to downstream tasks. Furthermore, while [58] utilizes task-specific routers, our approach employs task-specific prompts that encapsulate both task-specific router and expert parameters.

The parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks [18]. To improve the parameter efficiency of the final model, we can integrate this mechanism in the proposed method. Specifically, each head in the MSA layers comprises N MoE models, where N is the length of the input sequence. This allows for a dynamic routing mechanism to enhance parameter efficiency. For instance, [18] proposed a dynamic routing strategy that adaptively adjusts the number of activated experts based on the input. The computation for any MoE model's gating is directly correlated with the corresponding row in the attention matrix, which encapsulates the MoE model's score functions. For example, selecting the top k experts via Top-K routing in the i -th MoE model is equivalent to identifying the top k largest values in the i -th row of the attention matrix. To implement [18], we first sort the elements in the i -th row from highest to lowest, then find the smallest set of experts whose cumulative probability exceeds the threshold. Consequently, unselected experts remain inactive, reducing the need to compute all elements of the value matrix within self-attention.

D Training Algorithm of HiDe-Prompt

In this appendix, we outline the detailed algorithm of HiDe-Prompt, utilizing the same notation as in Section 2.

Each previously encountered class c ∈ Y ( i ) , i = 1 , . . . , t -1 has its instructed and uninstructed representations approximated by Gaussian distributions, denoted as G c and ˆ G c , respectively.

HiDe-Prompt maintains an expandable pool of task-specific prompts e t , each optimized for a specific task D t using a cross-entropy loss within the WTP objective. To prevent forgetting, previous prompts e 1 , . . . , e t -1 remain frozen. Knowledge transfer across tasks is facilitated by a prompt ensemble (PE) strategy: the current prompt is initialized with the last one e t ← e t -1 and refined using a weighted combination of all past prompts p t = α ∑ t -1 i =1 e i +(1 -α ) e t , where α is a hyper-parameter. Notably, HiDe-Prompt incorporates contrastive regularization within the WTP objective, pushing features of the new task away from those of past tasks represented by the prototypes of old class distributions G c . Let H t = { f θ ( x ( t ) i , p t ) | i = 1 , . . . , N t } be the embedding transformation of D t and µ c be the mean of G c . The contrastive loss can be written as

where τ is the temperature that is set to 0.8. The overall objective function of WTP for learning a new task t is defined as

Algorithm 1 HiDe-Prompt's training algorithm

where λ is a hyper-parameter. Following WTP, HiDe-Prompt performs a further refinement step on the output layer parameters ψ using a separate objective called task-adaptive prediction (TAP). TAP addresses potential classifier bias by considering the Gaussian distribution of all classes encountered so far. The final output layer h ψ can be further optimized for TAP objective,

where H i,c is constructed by sampling an equal number of pseudo representations from G c for c ∈ Y ( i ) and i = 1 , . . . , t .

For TII, HiDe-Prompt leverages a lightweight auxiliary output layer ˆ h ω : R D → R T , to map uninstructed representations directly to task identity. This mapping is learned explicitly through a cross-entropy loss function,

where ˆ H c is constructed by sampling an equal number of pseudo representations from ˆ G c for c ∈ Y ( i ) and i = 1 , . . . , t . Please refer to Algorithm 1 for more details.

E Experimental Details

Datasets. We use commonly-used datasets in the field of continual learning, including (1) Split CIFAR-100 [23]: This dataset comprises images from 100 classes. These classes are divided randomly into 10 separate incremental tasks, with each task featuring a distinct set of classes. (2) Split ImageNet-R [23]: This dataset is composed of images from 200 classes. It includes challenging examples from the original ImageNet [40] dataset and newly gathered images representing diverse styles. These classes are also randomly divided into 10 distinct incremental tasks. (3) Split CUB-200 [48]: This dataset consists of fine-grained images of 200 different bird species. It is randomly divided into 10 incremental tasks, each comprising a unique class subset. (4) 5-Datasets [9]: This composite dataset incorporates CIFAR-10 [23], MNIST [24], Fashion-MNIST [55], SVHN [31], and notMNIST [3]. Each of these datasets is treated as a separate incremental task, permitting for the assessment of the effects of significant variations between tasks.

Prompt-Based Approaches. We compare NoRGa against recent prompt-based continual learning approaches: L2P [54], DualPrompt [53], CODA-Prompt [42], S-Prompt [52] and HiDe-Prompt [49].

To ensure a fair comparison, we replicate these methods using the configurations reported in their respective papers. S-Prompt in the original paper trains a separate prompt and classifier head for each task. At evaluation, it infers domain id as the nearest centroid obtained by applying K-Means on the training data. To adapt S-Prompt to CIL, we use one common classifier head for all tasks. For NoRGa, we adopt the same configuration as HiDe-Prompt, which utilizes Prefix Tuning [26] as its prompt-based methodology. Learnable scalar factors α and τ are frozen after the first task's training to mitigate catastrophic forgetting. We further optimize NoRGa by selecting the best non-linear activation function σ via cross-validation among tanh , sigmoid , and GELU .

Evaluation Metric. We employ three common metrics to measure the performance the methods, including final average accuracy (FA), cumulative average accuracy (CA), and average forgetting measure (FM). Let S i,t denote the accuracy on the i -th task after learning the t -th task, and A t represent the average accuracy as A t = 1 t ∑ t i =1 S i,t . Upon learning all T tasks, we compute FA = A T , CA = 1 T ∑ T t =1 A t , and FM = 1 T -1 ∑ T -1 i =1 max t ∈{ 1 ,...,T -1 } ( S i,t -S i,T ) . It's worth noting that FA and CA are prioritized over FM, as they inherently encompass both plasticity and forgetting, with FM providing supplementary context [42].

Implementation Details. We train and test on one NVIDIA A100 GPU for baselines and our method. We leverage a pre-trained ViT-B/16 model as the backbone. Training employs an Adam optimizer ( β 1 = 0 . 9 , β 2 = 0 . 999 ), a batch size of 128, and a constant learning rate of 0.005 for all methods except CODA-Prompt. CODA-Prompt utilizes a cosine decaying learning rate starting at 0.001. Additionally, a grid search technique was implemented to determine the most appropriate number of epochs for effective training.

F Task-incremental Learning Results

Because HiDe-Prompt optimizes prompt parameters specifically for within-task prediction (WTP), NoRGa inherently aligns with this objective, leading to generally better continual learning performance. We demonstrate this improvement through experiments in a task-incremental learning setting, where task labels are available during inference (as in Table 4). While HiDe-Prompt performs well, NoRGa shows consistent improvement across all scenarios. Notably, NoRGa with sigmoid activation achieves the highest final average accuracy in both Split CIFAR-100 and Split CUB-200 with Sup-21K training. Additionally, NoRGa demonstrates its effectiveness even with self-supervised pre-training, further solidifying its advantage over the original prefix tuning model. Overall, NoRGa variants outperform HiDe-Prompt on both datasets and under both training conditions.

G Comparison to Different Parameter-Efficient Fine-Tuning Methods

As the advantages of different parameter-efficient fine-tuning (PEFT) methods remain an open question, we briefly describe them through our revealed connection between self-attention and MoE.

Prefix tuning introduces additional parameters at the input of MSA layers to adapt the pre-trained model representation, contrasting with Adapter [16], which insert adaptive parameters between layers, often replacing MLP blocks. LoRA [17] approximates weight updates with low-rank matrices and adds them to the backbone weights. Our work shows that the MSA layer in a pre-trained model can be seen as a pre-trained MoE architecture. Applying LoRA to the MSA layer refines both the pre-trained experts and their corresponding score functions for downstream tasks. In contrast, prefix tuning expands the pre-trained MoE models by incorporating new experts while preserving the original components, rather than modifying the pre-trained experts like LoRA.

NoRGa emerges as a simple, parameter-efficient fine-tuning method and can be regarded as a distinct implementation of prompts. However, our novel perspective on the interplay between self-attention, prefix tuning, and mixture of experts enables us to theoretically substantiate the effectiveness of NoRGa as shown in Section 4.

For empirical comparison, we integrate the framework of HiDe-Prompt with different PEFT techniques and Sup-21K weights, evaluating performance on Split CIFAR-100 and Split CUB-200. The results are summarized in Table 5. The table shows that NoRGa consistently outperforms the other PEFT methods on both datasets, suggesting its effectiveness. Nevertheless, further investigation with LoRA and Adapter would be necessary to draw more definitive conclusions.

While exploring alternative PEFT methods might offer improvements in WTP performance, these approaches lack theoretical guarantees and could lead to an increased number of parameters. In contrast, our NoRGa method modifies the original score functions of prefix tuning to enhance WTP performance with theoretical rigor. Importantly, NoRGa maintains the same parameter count as HiDe-Prompt, which is crucial in CL due to memory constraints.

H Comparison with Pre-trained Model-based Methods

Previous works have demonstrated that utilizing pre-trained models (PTM) significantly enhances performance for continual learning, often surpassing the performance of non-PTM-based methods. Moreover, studies have shown that first-task adaptation and simple PEFT-style tuning can achieve competitive performance [20, 37, 60, 29] with prompt-based methods. For instance, [20] demonstrated that appending a nearest class mean (NCM) classifier to a ViT model's feature outputs, can serve as a strong baseline. [37, 60] enhanced this strategy by adapting the pre-trained model to the first task using the three PEFT methods for transformer networks [60] and the FiLM method for CNNs [37]. Additionally, [29] improved NCM by incorporating second-order statistics-covariance and Gram matrices. However, these methods, which fine-tune only the backbone for the initial task, may not always ensure satisfactory separation of new tasks' features. Our work focuses on continually adapting the backbone, utilizing task-specific prompts to consistently capture emerging

HiDe-Prompt

NoRGa

0.125

0.110

NLL

0.095

0.080

0.065

0

10

20

30

40

50

Epochs

tasks' characteristics, and proposing a novel method to enhance the CL performance of previous prompting methods.

To validate our approach, we compare it against state-of-the-art PTM-based continual learning methods, including ADAM [60] and RanPAC [29], using Split CIFAR-100 and Split CUB-200 datasets. The results are summarized in Table 6. In comparison to other PTM-based continual learning methods, NoRGa demonstrates competitive performance across both evaluated datasets. For instance, on Split CIFAR-100, NoRGa achieves an FA of 94.48%, exceeding the next best method by over 2%. Similarly, on Split CUB-200, NoRGa delivers strong results relative to other baselines. These improvements highlight the effectiveness of our method in mitigating catastrophic forgetting and preserving knowledge retention across multiple tasks.

I Efficiency Tests

We compare the validation loss of NoRGa and HiDe-Prompt throughout the first task on Split CUB-200, as illustrated in Figure 3. The results demonstrate that NoRGa consistently outperforms HiDe-Prompt throughout the training process. This empirical evidence supports the theoretical advantages of NoRGa over HiDe-Prompt.

J Effect of Learnable Hyperparameters

As described in Section 4.1 and Appendix E, in our framework, α and τ are learnable hyperparameters and optimized through backpropagation during the first task, eliminating the need for manual tuning. Additionally, our theoretical analysis of NoRGa's statistical efficiency in Section 4.2 holds for any values of α and τ , demonstrating the theoretical robustness. To further investigate, we evaluated both fixed and learnable settings for these hyperparameters. For the fixed case, we set their values to 1. We report FA on Split CUB-200 and Split CIFAR-100 with Sup-21K weights. The results are summarized in Table 7. Although performance slightly decreased with fixed hyperparameters, it still outperforms HiDe-Prompt, indicating our method's empirical robustness.

K Training Times

We utilize a single A100 GPU for all experiments. The training times are summarized in Table 8. While NoRGa exhibits slightly longer training times compared to HiDe-Prompt, it consistently achieves significantly better performance. This demonstrates the effectiveness of NoRGa while maintaining competitive training efficiency.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions and scope are reflected accurately in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have already discussed the limitations of our work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and proofs is given in Section 4.2 and the Appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the experimental details for reproducibility are specified in Section 5, and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental details are specified in Section 5, and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars and relevant information in Section 5, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided sufficient information on the computer resources in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and followed all the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the theoretical nature of the paper, we do not think there are any positive or negative societal impacts of the work performed. Experiments are conducted only for empirically justifying the theoretical results.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not release any data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data and models used in the paper are properly credited. See Section 5, and Appendix E for further details.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper titled "Mixture of Experts Meets Prompt-Based Continual Learning" explores the integration of prompt-based continual learning methods with mixture of experts (MoE) architectures. The paper proposes a novel gating mechanism called Non-linear Residual Gates (NoRGa) to enhance the performance of prompt-based continual learning by leveraging theoretical insights and empirical evidence.

STRENGTHS:
1. This paper offers a novel connection between prompt-based-tuning and mixture-of-experts, providing a fresh perspective on prompt-based continual learning approaches.

2. Introduction of NoRGa, which integrates non-linear activation and residual connections to enhance continual learning performance while maintaining parameter efficiency.

WEAKNESSES:
1. Comparing NoRGa with other state-of-the-art continual learning methods that do not use prompts would highlight the specific advantages of the proposed method.

2. It may be better to use a additional graph to represent the final method NoRGa of the paper.

3. It may be necessary to further clarify the differences between the proposed method and other prompt-based methods.

QUESTIONS:
Which dataset is the ViT-B/16 pre-trained on.

LIMITATIONS:
This paper has already discussed the limitations of our work in the conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the theoretical underpinnings and practical implications of prompt-based methods in continual learning, aiming to enhance our understanding and optimize their effectiveness. It introduces a novel perspective by connecting prefix tuning with mixture of experts models, revealing insights into how self-attention mechanisms encode specialized architectures. The proposed Non-linear Residual Gates (NoRGa) further advances this by improving within-task prediction accuracy and overall continual learning performance while maintaining parameter efficiency.\n\nSTRENGTHS:\n1. This paper introduces a novel theoretical framework connecting self-attention mechanisms to mixture of experts models, significantly advancing the understanding of prompt-based approaches in continual learning.\n2. The theoretical insights are well-supported and complemented by empirical experiments across diverse benchmarks, demonstrating robustness and reliability.\n3. The concepts, although complex, are explained with clarity, aided by concrete examples and theoretical justifications.\n4.  Addressing the gap in theoretical understanding, the paper proposes a practical enhancement (NoRGa) that promises substantial improvements in model adaptation and efficiency.\n\nWEAKNESSES:\n1. The paper could benefit from clearer explanations regarding the core elements of prompt-based methods, particularly in how prompts are defined and utilized within their framework. \n2. While NoRGa is presented as a novel gating mechanism, its explicit connection to prompts could be elaborated further. Is it reasonable to interpret NoRGa's gating mechanism as a variant or extension of prompts in a broader sense?\n\nQUESTIONS:\n1. Could NoRGa's gating mechanism be seen as a different interpretation or implementation of prompts, albeit operating at a different level or with distinct functional goals?\n2. Are there instances of redundant theoretical explanations or definitions throughout the paper? Streamlining these could improve clarity and focus on the novel contributions without detracting from the foundational concepts.\n3. Given the complexity of the concepts, would integrating more visual aids (e.g., diagrams illustrating the architecture of NoRGa or the relationship between prompts and gating mechanisms) enhance the paper's accessibility and readability?\n\nLIMITATIONS:\nThe paper's reliance on dense theoretical explanations without adequate visual aids may limit its accessibility. Visual representations could significantly enhance comprehension of complex concepts like the relationship between prompts and NoRGa, potentially broadening the paper's audience and impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an extension to prefix tuning, by introducing non-linear residual gating - a simple extension over existing prefix tuning. This non-linear residual gating is supported with theoretical efficiency guarantees (polynomial versus exponential) to better estimate the optimal parameters. When applied to the problem of rehearsal-free continual learning with pretrained models (over four standard benchmarks) on top of the HiDe-Prompt method, the suggested gating provides consistent and partly significant improvements.\n\nSTRENGTHS:\n* To the best of my knowledge in both the Continual Learning and the parameter-efficient finetuning domain, the non-linear residual gating prefix MoE is a novel extension over standard prefix tuning.\n\n* This paper is generally well written, and except for the proofs, quite easy to follow.\n\n* Assuming the regression estimator to be a fair assumption, the statistical sample efficiency improvements of the proposed NoRGa setup versus normal gating is significant (polynomial versus exponential).\n\n* The explicit improvements over HiDE-Prompt are consistent, and often significant.\n\nWEAKNESSES:\nMy biggest issue with this work is its placement as a Continual Learning contribution, and evaluating it as such.\n\n__[1]__ Firstly, the proposed approach, whilel helping with WTP, is simply an extension on top of prefix tuning - and none of the contributions connect to the continual distribution shift nature. The improved performance is much more likely tied to an improvement in the underlying PEFT approach; which in itself is simply sufficient on standard benchmarks (see Zhou et al. 2023). This is further supported by the fact that the authors freeze both alpha and tau after the first task (see supplementary D). Moreover, on such small benchmarks, introducing two additional tunable hyperparameters does incorporate additional degrees of freedom to overfit to these benchmarks, and makes direct comparison to existing methods difficult.\n\nConsequently, the question needs to be answered: How robust is NoRGa with respect to the additional introduced alpha and temperature? These are strictly two hyperparameters more than HiDe-Prompt. As such, how much more compute went into hyperparameter optimization for NoRGa w.r.t. Hide? And given the same search budget, how does HiDe-Prompt compare relatively? This is the most crucial aspect, as the other method comparisons are somewhat redundant, since HiDe-Prompt introduces orthogonal classifier head alignment which can be applied to all other methods as well.\n\n> It would be great if the authors could address both the placement as a continual learning contribution (as opposed to a parameter-efficient finetuning paper, which would need to be evaluated on other respective benchmark tasks), and the comparability to HiDe-Prompt.\n\n__[2]__ This paper also only tackles prompt-based rehearsal-free continual learning, but misses discussions of recent works on first-task-adaptation and simple PEFT-style tuning for Continual Learning, such as: [1] Jason et al. 2022, “A simple baseline that questions the use of pretrained models for continual learning”, [2] Panos et al 2023, “First session adaptation: A strong replay-free baseline for class-incremental learning”, [3] Zhou et al. 2023, “Revisiting class-incremental learning with pretrained models: Generalizability and adaptivity are all you need”, [4] McDonnell et al. 2023: “RanPAC: Random projections and pretrained models for continual learning”. \n\n> Given that these works show that existing prompt-style CL objectives are matched by simply training on the first task, and how performance is matchable with simple parameter-efficient finetuning without explicitly accounting for the continual nature of the problem, how should the insights in this paper be rated w.r.t. These works?\n\n__[3]__ A vast part of this paper deals with the sample in-efficiency of standard prefix tuning mechanisms. However, the authors do not provide any efficiency tests (e.g. how does performance vary as a function of training iterations / samples seen), and only provide slightly improved performances on a few smaller-scale benchmarks. This is somewhat unfortunate, as it would be great to see how the efficiency estimates (e.g. Eq. 16) connect to practical convergence behaviour (e.g. L288 “polynomial number of data versus exponential number of data”).\n\n__[4]__ Both statistical efficiency proofs also do not take into account the continual nature of the problem, but rather assume a single generative model and i.i.d. Samples from said model, which are presented to the learner. This fully disconnects from the actual continual nature and sequential distribution shifts expected in such problems. Similarly, I’m having general troubles to connect the assumptions made throughout the proofs to the practical nature of the problem. E.g. why is Eq. 13 a reasonable assumption for the CL problem? Is convergence behaviour of a LS regression estimator suitably connected to practical convergence behaviour? If so, why? If not, why not? Similarly, why would I want to encourage algebraic independence - what would this give within the actual continual learning scenario?\n\n---\n\n__Smaller issues:__\n\n* I find it somewhat difficult to follow the proofs in 4.2 and Appendix A. Providing a proof-sketch as an overview would significantly help readability.\n* It is a bit problematic to have the entire proof of convergence, without even a proof sketch, in the supplementary-  as it motivates the entire section 4 and the non-linear residual gating. It would be great if the authors could provide at the very least a proof sketch in the main paper.\n* L139/140 is fairly hard to parse.\n\nQUESTIONS:\nAll relevant questions are incorporated in the previous section.\n\nI am currently leaning towards rejection - not because of the proposed method and the theoretical support (both of which I believe are sensible and meaningful), but rather its placement and consequent evaluation as a continual learning method. Neither in the methodological nature NOR the actually conducted proofs do the author account for the continual nature of the problem. \nInstead, I believe that this paper is much better suited as a contribution in the domain of parameter-efficient model finetuning. However, this in turn requires different benchmark evaluations.\n\nMoreover, I am uncertain about the significance of the report results (see above).\n\nTogether, I would be happy to raise my score if the authors can address these particular concerns (as well as those listed above).\n\nLIMITATIONS:\nThe authors do not discuss limitations or societal impact explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe topic of this paper is about the prompt-based continual learning. The authors give a theoretical analysis on these prompt-based continual learning methods, and utilize a Mixture-of-Expert (MoE) architecture characterized by linear experts and quadratic gating score functions. They develop a gating mechanism Non-linear Residual Gates (NoRGa) for MoE-based continual learning. The proposed method has been evaluated on several benchmarks.\n\nSTRENGTHS:\n+ The paper is well-written and easy to follow.\n+ It is interesting to theoretically analyze the effectiveness of prompt-based continual learning.\n+ Continual learning via MoE architecture is worth exploring.\n\nWEAKNESSES:\n-\tComparison to different Parameter-Effiecient-FineTuning methods [b] (e.g. adapter) is needed. This paper mainly focuses on the theoretical analysis of prompt-based continual learning methods.  Prompt-based continual learning belongs to PEFT methods and also add new parameters for new tasks. Can the authors further analyze the advantages of prompt-based methods theroretically?\n-\tThe parameters cost is usually considered in practical memory-constrained continual learning scenarios. Dynamic routing mechanism can be employed for gating-based neural networks (e.g. [c]). To improve the parameter efficiency of the final model, how to integrate this mechanism in the proposed method?\n-\tImportant MoE-related continual learning methods are not included in the related works [a]. The difference between the proposed method and other related works should be highlighted.\n-\tPerformance improvement on some datasets is limited. For example, FA metric (75.06%->75.40%) on Split ImageNet-R, CA metric (95.02%->95.11%) on Split-CIFAR-100 in Table 1. What are the running times of each experiment? Can the authors provide the performance variance of experimental results (e.g. the accuracy at the last incremental learning session)?\n\n[a] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR2024\n\n[b] Continual Learning with Pre-Trained Models: A Survey, IJCAI2024\n\n[c] Harder Tasks Need More Experts: Dynamic Routing in MoE Models, arxiv2024\n\nQUESTIONS:\nMy major concerns are included in the above weaknesses.\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}