Title: Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality
Venue: NeurIPS 2023
Accepted: yes
Keywords: Continual Learning

Catastrophic Forgetting

Pre-training

Prompt Tuning
Abstract: Prompt-based continual learning is an emerging direction in leveraging pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical research reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome the exposed sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Following these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively).

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality

Abstract

Prompt-based continual learning leverages pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical study reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome such sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Based on these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our code is available at https://github.com/thu-ml/HiDe-Prompt .

1 Introduction

In the realm of artificial intelligence, continual learning has become an area of significant interest. One of the pivotal techniques that greatly facilitate this domain is pre-training, which not only delivers positive knowledge transfer but also enhances resilience to catastrophic forgetting [27, 23, 37, 26, 43]. A recent innovation is the implementation of prompt-based methodologies, which freeze a pretrained transformer backbone and employ a few prompt parameters to steer representation learning. Such approaches typically involve construction of adaptive prompts for each task and inference of appropriate prompts during the test phase. By exploring prompt architectures to accommodate task-sharing and task-specific knowledge, this emerging direction demonstrates distinct superiority, almost reaching the upper bound of continual learning performance under supervised pre-training.

Nonetheless, given that robust pre-trained models typically necessitate the learning of substantial amounts of unlabeled data in a self-supervised manner, the influence of pre-training paradigms

on the effectiveness of prompt-based continual learning represents a significant and unresolved query. To answer this question, we first perform an extensive empirical investigation, and it clearly demonstrates the sub-optimality of recent prompt-based approaches under the more realistic selfsupervised pre-training. Since self-supervised representations tend to be more general, task-specific knowledge is difficult to incorporate into instructed representations via prompt parameters, as well as predicted by uninstructed representations at test time. Consequently, the performance of many recent approaches, such as L2P [41], DualPrompt [40], S-Prompt [39] and CODA-Prompt [30], is seriously compromised. We further disclose the importance of adaptive prediction for all tasks together, which can potentially mitigate the aforementioned shortcomings to some extent.

Motivated by these observations, we provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into hierarchical components such as within-task prediction , task-identity inference and task-adaptive prediction . Thanks to the well-distributed representations resulting from adequate pre-training, the hierarchical components can be optimized explicitly by constructing an ensemble of task-specific prompts and exploiting the preserved statistics of uninstructed and instructed representations. A novel contrastive regularization is further devised to coordinate these hierarchical components. We refer to this approach as Hierarchical Decomposition (HiDe-)Prompt and demonstrate its superiority through extensive continual learning experiments, especially under the more realistic self-supervised pre-training.

Our contributions include: (1) We provide an extensive empirical study under self-supervised pretraining to demonstrate the sub-optimality of current progress in prompt-based continual learning; (2) To overcome such sub-optimality, we theoretically analyze the objective of continual learning with pre-training, and decompose it into hierarchical components for model design; (3) With task-specific prompts and representation statistics, we propose an innovative approach to optimize the hierarchical components explicitly; (4) Across various continual learning benchmarks and pre-training paradigms, our approach achieves clearly state-of-the-art performance in a rehearsal-free manner.

2 Related Work

Continual Learning: The ability of continual learning is critical for artificial neural networks to accommodate real-world changes [37, 35]. Numerous efforts in this direction have been devoted to overcoming catastrophic forgetting [22, 34, 33]. According to a recent survey [37], representative strategies include selective stabilization of network parameters, replay of a few old training samples, explicit manipulation of optimization programs, exploitation of well-distributed representations, construction of task-specific parameters, etc. The performance of such strategies varies with particular settings of continual learning. As one of the most challenging and representative settings, classincremental learning (CIL) [31, 37] requires a continual learning model to perform all old tasks (or classes) without the oracle of task identity. Strong CIL methods generally depend on storage and rehearsal of old training samples [28, 8, 38], which result in efficiency and privacy issues.

Self-Supervised Learning and Pre-Training: The exploitation of well-distributed representations, especially from the success of large-scale pre-training, brings significant benefits for downstream continual learning [37, 27, 23]. Due to the scarcity and expense of explicit labeling in many real-world applications, self-supervised learning is typically involved in the pre-training stage to cope with huge amounts of unlabeled data. In particular, instance discrimination [4, 7] with contrastive learning [25] has become the dominant strategy, which aims to maximize representation similarity of the same instance and minimize representation similarity of different instances. Besides, self-supervised paradigms have been shown less sensitive to catastrophic forgetting in upstream continual learning [9], providing a practical way to enrich pre-trained knowledge from in-the-wild data.

Prompt-Based Approach: Inspired by parameter-efficient fine-tuning techniques in NLP [11, 10], recent prompt-based approaches [3, 41, 40, 39, 30] are developed to leverage pre-trained knowledge adaptively for downstream continual learning. The basic idea includes construction and inference of adaptive prompts for each task, so as to instruct a frozen transformer backbone. The former mainly focuses on exploring prompt architectures to instruct representations with task-sharing and task-specific knowledge, closely related to the discussion of model architectures in continual learning [37, 36], while the latter attempts to predict appropriate (combinations of) prompts with uninstructed representations. Although such methods have achieved remarkably strong performance under supervised pre-training, whether these advantages are consistent under the more realistic self-

supervised pre-training remains to be explored. A concurrent study [43] observed that self-supervised pre-training is more challenging for continual learning approaches that require fine-tuning of the backbone, implying a non-trivial impact of pre-training paradigms on downstream continual learning.

3 Preliminary Analysis

In this section, we first introduce the problem formulation of prompt-based continual learning, and then evaluate the impact of pre-training paradigms with an extensive empirical study.

3.1 Formulation of Prompt-Based Continual Learning

Continual learning aims to learn a sequence of tasks on their respective training sets D 1 , ..., D T and excel on their corresponding test sets. The training set for task t typically consists of various datalabel pairs D t = { ( x t,n , y t,n ) } N t n =1 , where x t,n ∈ X t and y t,n ∈ Y t represent the sample and label elements, respectively. We will use | · | to denote the cardinality of a set and [ N ] = { 1 , 2 , · · · , N } as the set of intergers from 1 to N . Consider a neural network model with a backbone f θ parameterized by θ , and an output layer h ψ parameterized by ψ . This model seeks to learn the projection from X = ⋃ T t =1 X t to Y = ⋃ T t =1 Y t , aiming to predict the label y = h ψ ( f θ ( x )) ∈ Y of an unseen test sample x drawn from previous tasks. The backbone function f θ is assumed to be pre-trained with a substantial quantity of additional training samples external to each D t . There are commonly three distinct settings for continual learning [31]: task-, domain-, and class-incremental learning (TIL, DIL, and CIL). Specifically, Y 1 , ..., Y T are identical for DIL while disjoint for TIL and CIL. The task identity is provided for TIL at test time but is not available for DIL and CIL. Therefore, CIL is considered to be more representative and challenging in general. Of note, the continual learning process is rehearsal-free [30]-all elements of D t are available only when learning task t .

Prompt-based approaches for vision tasks further specify the backbone f θ as a pre-trained vision transformer (ViT), where multiple consecutive multi-head self-attention (MSA) layers can transform an input sample into a sequence-like output representation h ∈ R L h × D of sequence length L h and embedding dimension D . The backbone parameters θ are typically frozen to obtain generalizable representations. A few prompt parameters p ∈ R L p × D of sequence length L p and embedding dimension D are prepended to h to exploit the pre-trained knowledge adaptively. Here we denote the input of the l -th MSA layer as h l ∈ R L h l × D , which consists of query h l Q , key h l K and value h l V , and denote the prompt as p l ∈ R L p l × D . For notation clarity, we take one MSA layer as an example and omit the layer label l if not necessary. Then, the output of this MSA layer is given as

where W O , W Q,i , W K,i and W V,i are projection matrices, m is the number of heads, and h Q = h K = h V in ViT. There are two major implementations of prompt-based methodologies [40], i.e., Prompt Tuning (ProT) [18] and Prefix Tuning (PreT) [19]. Specifically, ProT prepends an identical p to h Q , h K and h V :

where [ · ; · ] denotes the concatenation operation along the dimension of sequence length, and the output in R ( L h + L p ) × D has increased dimensions. In contrast, PreT splits p into p K ∈ R L p / 2 × D and p V ∈ R L p / 2 × D only for h K and h V , respectively:

where the output dimension remains the same as the input h ∈ R L h × D . As the training samples for each task are introduced sequentially, prompt-based continual learning needs to incorporate task-specific knowledge into prompt parameters while overcoming their catastrophic forgetting. The mainstream idea is to construct adaptive prompts for each task and then infer appropriate (combinations of) prompts at test time. Here we compare state-of-the-art approaches from these two aspects, demonstrated conceptually in Fig. 1:

L2P [41] constructs a prompt pool P = { p 1 , ..., p M } potentially shared by all tasks where M is the total number of prompts, and then instructs the last MSA layer in a ProT fashion. Each prompt p i is associated to a learnable key k i ∈ R D , optimized by the cosine distance γ ( q ( x ) , k i ) of the topN keys to a query function q ( x ) = f θ ( x )[0] to incorporate knowledge. Therefore, the most relevant keys and the corresponding prompts can be selected with uninstructed representations for inference.

DualPrompt [40] constructs tasksharing prompts g l and task-specific prompts e l t to instruct different MSA layers in a PreT fashion. All e l t belonging to the same task is associated to a task-specific key k t ∈ R D , optimized by γ ( q ( x ) , k t ) , and the index of the best-matched key is selected for inference.

S-Prompt [39] constructs only taskspecific prompts e t for each task, and adopts a similar ProT strategy as L2P to instruct the last MSA layer. The inference of task identity is achieved by a simple KNN

N-do

Incremental Inputs

(2)Construction

L2P

000000000

of Prompt(s)

Task-Specific

Task-Sharing

DualPrompt

0000

Transformer

Pre-trained

Backbone

Task-Specific

S-Prompt

00000

(3) Inserted into

Fixed

MSA Layer(s)

Attention Weight

(4)Prediction

of All Tasks

CODA-Prompt

Learnable

PromptIdentity

(1) Inference of

Output Layer

strategy for the nearest task centroid. Unlike other methods, S-Prompt associates an exclusive output head ψ t to each task t = 1 , ..., T .

CODA-Prompt [30] exploits the prompt pool P by its weighted summation, i.e., p = ∑ M i =1 α i p i , where α i = γ ( q ( x ) , k i ) is the weighting factor, and adopts a similar PreT strategy as DualPrompt to instruct multiple MSA layers. The inference of α i enables construction of adaptive prompts.

3.2 Empirical Study of Pre-Training Paradigms

Either explicitly or implicitly, the above prompt-based approaches all incorporate the knowledge of each task into prompt parameters and predict their identities from uninstructed representations. To evaluate the impact of pre-training paradigms, we perform an empirical study with widely-used CIL benchmarks such as Split CIFAR-100 and Split ImageNet-R [41, 40]. In addition to supervised pretraining of ImageNet-21K [29] (denoted as Sup-21K), we consider several powerful self-supervised models that release ViT checkpoints 2 , such as iBOT [44], DINO [2] and MoCo v3 [5].

a

L06

b

75

()  a

Split CIFAR-100

Final Average Accuracy (%)

F0L

-s9

-09

65-

50-

60

45

L2P

DualPrompt

S-Prompt++

CODA-Prompt

L2P

DualPrompt

S-Prompt++

CODA-Prompt

LR Reduction

LR Reduction

C 1.2

Acquisition of Task Specificity

Task Inference Accuracy (%)

d

L08

59.12

Predictionof TaskIdentity

Final Average Accuracy (%)

e

100

Prediction of AllTasks

iBOT-21K

CKA Similarity

1.0-

0.743

0.875

0.928

0.961

0.931

0.774

0.830

0.882

10.821

60-

T

37.58

39.39

39.24

134.55

34.53

80-

-09

++ido-s

S-Prompt

■

0.8-

40

24.74

0

40-

口

DINO-1K

9:0

0.498

20-

15.71

20-

0.4

0-

Split CIFAR-100

Split CIFAR-100

Split CIFAR-100

We carefully evaluate the official implementations of all baselines for fair comparison. We follow largely the training regimes of L2P [41] and DualPrompt [40], which are basically consistent. As S-Prompt [39] is initially designed for DIL, we slightly modify its implementation by inserting task-specific prompts into the same layers as DualPrompt (i.e., layers 1-5) in a PreT manner, so as to evaluate the impact of prompt architectures. The output layer retains multiple heads associated

with the task identity (still denoted as S-Prompt), or a single head as with other baselines (denoted as S-Prompt++). CODA-Prompt [30] is officially implemented in a DualPrompt-like architecture but depends heavily on the use of a smaller learning rate with cosine decay. Here we present its performance with both default and reduced learning rates. Using the same learning rate as [41, 39], we grid search for an appropriate number of epochs (detailed in Appendix D) and report the best performance for all baselines.

As shown in Fig. 2, a, b, the above prompt-based approaches achieve outstanding performance under Sup-21K, where the use of task-specific prompts clearly outperforms task-sharing prompts (i.e., S-Prompt++ ≈ CODA-Prompt > DualPrompt > L2P) due to explicit avoidance of catastrophic forgetting. However, the four baselines suffer significant performance degradation under the more realistic self-supervised pre-training. In particular, the performance differences between prompt architectures have become much smaller, suggesting that the task-specific and task-sharing knowledge are not well differentiated. Besides, CODA-Prompt can generally achieve leading performance as a direct result of the learning rate (LR) reduction rather than the prompt architecture.

We perform two additional experiments to demonstrate the obscured sub-optimality 3 . First, we evaluate the CKA similarity of uninstructed and instructed representations by learning task-specific prompts (Fig. 2, c). The CKA similarity of self-supervised pre-training is significantly higher, suggesting a greater difficulty for prompt parameters to incorporate task-specific knowledge. Second, we evaluate the ability to predict task identity from uninstructed representations and task-specific keys, where self-supervised pre-training exhibits much lower accuracy (Fig. 2, d). Interestingly, although only less than 40% task identities are correctly predicted, S-Prompt++ can still achieve considerable (albeit sub-optimal) performance, owning to the compensation effects of using a singlehead output layer (Fig. 2, e). Together with the results in Fig. 2, c, d, it is conceivable that using an 'incorrect' prompt would not severely affect the instructed representations, which can still be correctly predicted in a well-balanced single-head output layer. In contrast, S-Prompt performs much worse than S-Prompt++, as its multi-head output layer undertakes all errors of task-identity inference.

4 Theoretical Foundation and Our Approach

In this section, we first present a theoretical analysis of the sufficient and necessary conditions for improving continual learning in the context of pre-training, and then present an innovative approach for prompt-based continual learning to achieve this objective.

4.1 Hierarchical Decomposition of Continual Learning Objective

For continual learning of sequentially arrived D t , X t and Y t are the domain and label of task t . Here we take CIL as a typical scenario for theoretical analysis where Y t ∩ Y t ′ = ∅ , ∀ t = t ′ (see Appendix A for DIL and TIL). Let X t = ⋃ j X t,j and Y t = {Y t,j } , where j ∈ [ |Y t | ] indicates the j -th class in task t . Now assume we have a ground event denoted as D = {D 1 , ..., D t } and a pre-trained model f θ . For any sample x ∈ ⋃ t k =1 X k , a general goal of the CIL problem is to learn P ( x ∈ X i,j |D , θ ) , where i ∈ [ t ] and j ∈ [ |Y i | ] . This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as P ( x ∈ X i |D , θ ) and P ( x ∈ X i,j | x ∈ X i , D , θ ) , respectively. Based on Bayes' theorem, we have

̸

Let ¯ i ∈ [ t ] and ¯ j ∈ [ |Y i | ] be the ground truth of an x w.r.t. the task identity and within-task index. Eq. (5) shows that if we can improve either the WTP performance P ( x ∈ X ¯ i, ¯ j | x ∈ X ¯ i , D , θ ) , the TII performance P ( x ∈ X ¯ i |D , θ ) , or both, then the CIL performance P ( x ∈ X ¯ i, ¯ j |D , θ ) would be improved. However, such an improvement is limited since it is upper-bounded by WTP or TII. To further improve the CIL performance, we propose a hierarchical decomposition of its objective. That is, besides the improvement of P ( x ∈ X ¯ i, ¯ j |D , θ ) , we also need to improve the performance of task-adaptive prediction (TAP), denoted as P ( x ∈ X y |D , θ ) , where X y represents the domain of

Task-ldentity Inference(Tll)

Within-TaskPrediction(WTP)

Task-AdaptivePrediction(TAP)

P(α ∈X;|D,0)

P(c∈X∈x,D,0)

P(Ex|D,0)

Contrastive

Regularization

Uninstructed

Outputof

Instructed

Outputof

Representations

TaskIdentity

Representations

AllTasks

个

Sampling

个

Sampling

Task-Specific

Save

0000

Save

Statistics

Statistics

PromptEnsemble

class y in all previous tasks, and y = Y ¯ i, ¯ j is the ground truth label of x . Then the final goal of CIL is formulated as a multi-objective optimization problem, i.e., max[ P ( x ∈ X ¯ i, ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] . Notice that the TII probability is a categorical distribution over all observed tasks upto t , while the TAP probability is over all observed classes ⋃ t k =1 Y k .

To resolve the problems above, we derive the sufficient and necessary conditions in the context of the widely-used cross-entropy loss. Specifically, we define

where H WTP , H TII , and H TAP are the cross-entropy values of WTP, TII, and TAP, respectively. The operation H ( p, q ) ≜ -E p [log q ] = -∑ i p i log q i . 1 · is a one-hot encoding function.

We now present the first theorem under the CIL scenario (see Appendix A for a detailed proof):

Theorem 1 For continual learning with pre-training, if E x [ H WTP ( x )] ≤ δ , E x [ H TII ( x )] ≤ ϵ , and E x [ H TAP ( x )] ≤ η , we have the loss error L ∈ [0 , max { δ + ϵ, η } ] , regardless whether WTP, TII and TAP are trained together or separately.

With the use of cross-entropy, the continual learning performance tends to be better as the bounds are tightened. In Theorem 1 we have shown that good performances of WTP, TII and TAP are sufficient to guarantee a good performance of CIL. For completeness, we now study the necessary conditions of a well-performed CIL model in Theorem 2.

Theorem 2 For continual learning with pre-training, if the loss error L ≤ ξ , then there always exist (1) a WTP, s.t. H WTP ≤ ξ ; (2) a TII, s.t. H TII ≤ ξ ; and (3) a TAP, s.t. H TAP ≤ ξ .

Theorem 2 suggests that if a continual learning model is well trained (i.e., with low loss), then the WTP, TII and TAP for sequential tasks are always implied to be small. It is worth noting that without the pre-trained knowledge carried by θ , Theorem 1 and Theorem 2 would degrade to the main conclusion of a previous theoretical study [13], suggesting that the presented theorems are particularly directed to the impact of pre-training for continual learning (detailed in Appendix B). Besides, the paradigm of pre-training is indeed related to the performance of continual learning, because it can affect the distribution of representations from f θ , and further WTP, TII and TAP. Previous work has demonstrated that self-supervised representations tend to be more robust to parameter changes than supervised ones [9, 21, 37], which is beneficial for accumulating pre-trained knowledge (if applicable) but challenging for adapting to downstream tasks on a continual basis (see Fig. 2, c, d).

4.2 HiDe-Prompt for Prompt-Based Continual Learning

Motivated by the above empirical and theoretical insights, we propose to optimize explicitly the hierarchical components (i.e., WTP, TII and TAP) for prompt-based continual learning, as shown in Fig. 3. Our proposal stems from a particular advantage of pre-training, where the distributions of uninstructed and instructed representations can be effectively preserved through their statistical information. In the case of classification, for example, since each class tends to have single-peaked

representations (see Appendix D, Fig. 5 and Fig. 6), we can naturally approximate them with Gaussian distributions. For generality, here we denote the approximated distributions of uninstructed and instructed representations as ˆ G c and G c for each class c ∈ Y i , i ∈ [ t -1] , respectively, and discuss their specific forms latter.

First, we improve WTP through effective incorporation of task-specific knowledge. We construct an expandable prompt pool with only task-specific prompts e t to incorporate the knowledge of D t , optimized by a cross-entropy (CE) loss for H WTP . The previous prompts e 1 , ..., e t -1 are frozen to avoid catastrophic forgetting. In order to transfer knowledge for learning each task effectively, we employ a prompt ensemble (PE) strategy, where the current prompt is initialized by the last prompt e t ← e t -1 and then optimized with a weighted combination of all previous prompts p t = α ∑ t -1 i =1 e i + (1 -α ) e t . α is a hyperparameter that controls the strength of inherited old knowledge to facilitate p t in learning the current task. Meanwhile, the instructed representations of p t , although allowing the new task to be performed well, may overlap with that of the old tasks and thus affect TAP. To overcome this issue, we exploit the old-task statistics of instructed representations (collected by f θ and p i for i = 1 , ..., t -1 ), where for classification we calculate the mean µ c of G c for each class c ∈ Y i , and design a contrastive regularization (CR):

where H t is the embedding transformation of D t with f θ and p t . τ is the temperature coefficient, which is insensitive and set to 0.8 in practice. Notably, here we use only µ c to represent each class for efficiency, which can be optionally replaced by sampling from G c for better performance.

Then, the loss function of WTP can be defined as

Therefore, the instructed representations of new classes can be well distinguished for WTP while avoiding overlap with the previous ones. λ is a hyperparamter to balance the impact of old classes.

Second, we improve TII and TAP through exploiting the approximated distributions of uninstructed and instructed representations, respectively. For TII, we construct an auxiliary output layer ˆ h ω : R D → R T parameterized by ω , learning explicitly the projection from uninstructed representations to task identity via cross-entropy (i.e., H TII ):

where ˆ H i,c is constructed by sampling an equal number of pseudo representations from ˆ G c for c ∈ Y i and i ∈ [ t ] . Unlike other baselines that freeze the projection of old tasks (i.e., the previous keys), our ˆ h ω is continually adapted for all tasks and thus greatly facilitates TII.

Similarly, the final output layer h ψ : R D → R |Y| can be further optimized for TAP (i.e., H TAP ):

where H i,c is constructed by sampling an equal number of pseudo representations from G c for c ∈ Y i and i = 1 , ..., t . As ω and ψ are usually light-weight , the optimization of TII and TAP is computationally efficient. At test time, HiDe-Prompt predicts the task identity i = ˆ h ω ( f θ ( x )) and then the label y = h ψ ( f θ ( x ; p i )) . Please refer to Appendix Algorithm 1 for more details.

Since the pre-trained representations are usually well-distributed, there are many reasonable strategies to model ˆ G c and G c . On default, the distributions of uninstructed and instruted representations can be faithfully recovered by modeling each class as a Gaussian with a dedicated mean and covariance. With adequate pre-training, the covariance can be further reduced to variance for efficiency. Alternatively, such statistical modeling can employ multiple centroids obtained from KNN and add Gaussian noise, which is also an efficient choice and is applicable to other task types.

5 Experiment

In this section, we first describe the experimental setups, and then present the experimental results.

Benchmark: We consider multiple CIL benchmarks that are widely used for prompt-based continual learning [41, 40, 30]. Specifically, Split CIFAR-100 [14] includes 100-class small-scale images, randomly split into 10 incremental tasks of disjoint classes. Split ImageNet-R [14] includes 200-class large-scale images that are hard examples of ImageNet [29] or newly collected examples of different styles, randomly split into 10 incremental tasks of disjoint classes. 5-Datasets [6] includes CIFAR-10 [14], MNIST [15], Fashion-MNIST [42], SVHN [24] and notMNIST [1] datasets, each treated as an incremental task to evaluate the impact of large inter-task differences. Split CUB-200 [32] includes 200-class fine-grained images of birds, randomly split into 10 incremental tasks of disjoint classes.

Baseline: We compare four representative prompt-based approaches as discussed in Sec. 3.1, such as L2P [41], DualPrompt [40], S-Prompt++ [39] and CODA-Prompt [30]. To evaluate the performance of continual learning, we record the average accuracy of all seen classes after learning each task, presenting the last one as the final average accuracy (FAA) and their historical average as the cumulative average accuracy (CAA) [37]. We also present the final forgetting measure (FFM) of all tasks [37]. We consider a variety of pre-training paradigms for ImageNet-21K and ImageNet-1K, including Sup-21K, iBOT-21K, iBOT-1K, DINO-1K and MoCo-1K, as described in Sec. 3.2.

Implementation: We follow similar implementations as previous work [41, 40, 30]. Specifically, we adopt a pre-trained ViT-B/16 backbone and train with an Adam optimizer ( β 1 = 0 . 9 , β 2 = 0 . 999 ), a batch size of 128, and a constant learning rate of 0.005 (except for CODA-Prompt with a cosinedecaying learning rate of 0.001), and grid search for a proper epoch number. The image inputs are resized to 224 × 224 and normalized to [0 , 1] . Please refer to Appendix C for more details.

Overall Performance: Table 1 presents the main results of all approaches. Consistent with the observations in Sec. 3.2, representative prompt-based approaches achieve outstanding performance under supervised pre-training (i.e., Sup-21K), while perform significantly worse under the more realistic self-supervised pre-training. In particular, the most recent CODA-Prompt [30] outperforms other baselines in general (here we take FAA as the primary metric), but is sensitive to learning rate (see Fig. 2, a, b and Appendix Table 6). In contrast, our HiDe-Prompt achieves generally the

highest FAA, CAA and the lowest FFM, thus greatly superior to all competitors. This advantage is more pronounced under self-supervised pre-training, e.g., up to 15.01 %and 9.61 %lead on Split CIFAR-100 and Split ImageNet-R, respectively. 4 When considering large inter-task differences and fine-grained classification (see Table 2), the performance lead of our approach under self-supervised pre-training becomes even more significant, e.g., up to 12.63 %and 30.44 %on 5-Datasets and Split CUB-200, respectively. We further evaluate the impact of pre-trained knowledge that is disjoint from downstream tasks, following the setup of a recent work [12]. Specifically, we perform Split CIFAR100 with pre-training on a subset of ImageNet in which 389 similar classes were removed. The FAAs of S-Prompt++, CODA-Prompt and HiDe-Prompt are 69.00%, 65.07% and 88.05%, respectively. All above results demonstrate the strength of our approach, thanks to the explicit optimization of the hierarchical components that overcomes potential sub-optimality.

Here we explore the characteristics of HiDePrompt in more depth. First, HiDe-Prompt requires much smaller GPU memory compared to CODA-Prompt (e.g., 16141MB vs 25325MB on Split CIFAR-100), since we optimize the construction and inference of appropriate prompts separately as hierarchical components rather than jointly in an end-to-end fashion. Compared to other baselines (except L2P that requires a much smaller epoch number), the computation cost of our approach is comparable in order of magnitude. Using the same single-card

A100 GPU, for example, the training times of L2P, DualPrompt, S-Prompt++, CODA-Prompt and HiDe-Prompt are 0.55h, 2.00h, 2.01h, 2.08h, and 2.80h on Split CIFAR-100, respectively. As for the impact of statistical modeling, reserving around 5 centroids for each class performs comparably to a single Gaussian (see Appendix Table 7), which ensures storage efficiency and can potentially be generalized to other task types.

Ablation Study: We perform an extensive ablation study in Table 3 to validate the effectiveness of hierarchical components in HiDe-Prompt. We first construct a naive architecture of task-specific prompts as the baseline and then incorporate progressively the individual designs of Sec. 4.2. In general, the optimization of each component, such as within-task prediction (WTP), task-identity inference (TII) and task-adaptive prediction (TAP), delivers clear benefits and all contribute to the strong performance of HiDe-Prompt. Interestingly, the improvement of TII only becomes apparent

with WTP+TAP rather than with WTP only, suggesting that these hierarchical components are highly synergistic instead of operating in isolation. The proposed contrastive regularization (CR) helps the instructed representations of individual tasks to be compatible with each other and avoid inter-task overlap, thus further facilitating the performance of WTP+TII+TAP. Moreover, we observe that the improvements of WTP, TII, TAP and CR are generally more significant under self-supervised pre-training , due to effectively addressing the obscured sub-optimality.

Detailed Analysis: Now we further analyze the contributions of the three components for continual learning. First, we evaluate the average performance of learning each new task in Fig. 4, a, where WTP clearly outperforms the naive architecture, indicating that the knowledge of resolving each task is better incorporated into the task-specific prompts. Second, we present the accuracy of predicting task identity from uninstructed representations in Fig. 4, b, which has been substantially improved

iBOT-21K

DINO-1K

a

96

b

100

w/WTP(Ours)

w/ TIl(Ours)

New Task Accuracy (%)

(%)

94

Task Inference Accuracy(

80

92

F09

90

40-

88

20-

86

0

Split CIFAR-100

Split CIFAR-100

(up to 67.74%) through optimizing TII explicitly. Third, we evaluate the effects of CR on only WTP and the full model of HiDe-Prompt (i.e., WTP+TII+TAP) in Table 4. Increasing the strength of CR decreases the performance of WTP since the task-specific prompt includes more knowledge about other tasks, but improves the performance of HiDe-Prompt since the inter-task representations become more compatible. These results suggest a potential trade-off between the knowledge for WTP and TAP, which can be modulated explicitly by CR.

6 Discussion and Conclusion

In this work, we analyze extensively the advanced prompt-based continual learning from both empirical and theoretical perspectives. An important finding is that, the continual learning objective (which can be decomposed into WTP, TII and TAP in the context of pre-training) is not adequately achieved, and this sub-optimality is clearly exposed under the more realistic self-supervised pre-training. By leveraging statistics of uninstructed and instructed representations, we present a strong approach to optimize explicitly the hierarchical components, which achieves superior performance across various pre-training paradigms. In particular, our theoretical analysis and the proposed approach can serve as a general framework for implementing parameter-efficient fine-tuning techniques (e.g., prompt, adapter, LoRA, FiLM, etc.) in continual learning 5 , which differ only in the form of task-specific parameters. Interestingly, our proposal is consistent with recent advances in neuroscience [17, 16], where the activation of non-memory cells and memory cells (as well as their specific populations) is internally switched. Based on these results, we expect subsequent work to further explore the architecture and optimization of continual learning with an effective use of pre-trained knowledge.

This work remains some potential limitations . First, we assume adequate pre-training to provide meaningful representations, which may not be available in some applications. Second, the prompt-based strategy is mainly applicable to the transformer backbone rather than other backbone architectures. Third, the transformer backbone is frozen and adapted to downstream tasks via prompt parameters, which prevents the pre-trained knowledge from being enriched and updated. As a fundamental research in machine learning, the potential negative societal impact is not obvious at this stage.

References

Yaroslav Bulatov. Notmnist dataset. 2011.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

Haoran Chen, Zuxuan Wu, Xintong Han, Menglin Jia, and Yu-Gang Jiang. Promptfusion: Decoupling stability and plasticity for continual learning. arXiv preprint arXiv:2303.07223 , 2023.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In European Conference on Computer Vision , pages 386-402. Springer, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 831-839, 2019.

Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? In International Conference on Learning Representations , 2021.

Zixuan Ke and Bing Liu. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.12701 , 2022.

Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In International Conference on Learning Representations , 2022.

Gyuhak Kim, Bing Liu, and Zixuan Ke. A multi-head model for continual learning via outof-distribution replay. In Conference on Lifelong Learning Agents , pages 548-563. PMLR, 2022.

Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A theoretical study on solving continual learning. Advances in Neural Information Processing Systems , 35:5065-5079, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Bo Lei, Bilin Kang, Wantong Lin, Haichao Chen, Yuejun Hao, Jian Ma, Songhai Shi, and Yi Zhong. Adult newborn granule cells confer emotional state-dependent adaptability in memory retrieval. Science Advances , 8(45):eabn2136, 2022.

Bo Lei, Li Lv, Shiqiang Hu, Yikai Tang, and Yi Zhong. Social experiences switch states of memory engrams through regulating hippocampal rac1 activity. Proceedings of the National Academy of Sciences , 119(15):e2116844119, 2022.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning. arXiv preprint arXiv:2303.09447 , 2023.

Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Representations , 2021.

James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review , 102(3):419, 1995.

Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153 , 2021.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. arXiv preprint arXiv:2303.13199 , 2023.

Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations , 2021.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11909-11919, 2023.

Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 , 2019.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5383-5392, 2021.

Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems , 34:22379-22391, 2021.

Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. arXiv preprint arXiv:2308.14991 , 2023.

Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In European Conference on Computer Vision , pages 254-271. Springer, 2022.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 , 2023.

Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations , 2021.

Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems , 35:5682-5695, 2022.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision , pages 631-648. Springer, 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, 2022.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. arXiv preprint arXiv:2303.05118 , 2023.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations , 2021.

Algorithm 1 Training Algorithm of HiDe-Prompt

Input : Pre-trained transformer backbone f θ , training sets D 1 , ..., D T , number of tasks T , number of epochs E , hyperparameters α , τ and λ .

A Theoretical Foundation

A.1 Class-Incremental Learning (CIL)

Proof of Theorem 1

Proof. For class-incremental learning (CIL) with pre-training, assume E x [ H WTP ( x )] ≤ δ , E x [ H TII ( x )] ≤ ϵ , and E x [ H TAP ( x )] ≤ η . Let y = Y ¯ i, ¯ j be the ground truth of an x , where ¯ i ∈ { 1 , ..., t } and ¯ j ∈ { 1 , ..., |Y i |} denote the task identity and within-task index, respectively.

As we defined,

Then, we have

Taking expectations on Eq. (15), we have

Taking expectations on both sides of Eq. (16), we have

Since our objective of CIL with pre-training is max[ P ( x ∈ X ¯ i, ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] , then we have the loss error

This finishes the proof.

Proof of Theorem 2

Proof. For CIL with pre-training, its loss error L ≤ ξ . Assume x ∈ X ¯ i, ¯ j ⊆ X ¯ i . According to the proof of Theorem 1, we have

In addition, we have formulated the final goal of CIL as a multi-objective optimization problem, i.e., max[ P ( x ∈ X ¯ i, ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] . Then, each objective must guarantee the loss error less than ξ , i.e.,

This finishes the proof.

A.2 Domain-Incremental Learning (DIL)

̸

For domain-incremental learning (DIL), Let X t = ⋃ j X t,j and Y t = {Y t,j } , where j ∈ { 1 , ..., |Y t |} denotes the j -th class in task t . Now assume we have a ground event denoted as D = {D 1 , ..., D t } and a pre-trained model f θ . For any sample x ∈ ⋃ t k =1 X k , a general goal of the DIL problem is to learn P ( x ∈ X ∗ ,j |D , θ ) , where X ∗ ,j represents the j -th class domain in any task. Of note, Y t = Y t ′ , ∀ t = t ′ for DIL. This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as P ( x ∈ X i |D , θ ) and P ( x ∈ X i,j | x ∈ X i , D , θ ) , respectively. Based on Bayes' theorem, we have

where {∗ , j } represents the j -th class in each domain.

Then we have the following theorems in terms of the sufficient and necessary conditions for improving DIL with pre-training.

Theorem 3 For domain-incremental learning with pre-training, if E x [ H WTP ( x )] ≤ δ , E x [ H TII ( x )] ≤ ϵ , and E x [ H TAP ( x )] ≤ η , we have the loss error L ∈ [0 , max { δ + ϵ +log t, η } ] , regardless whether WTP, TII and TAP are trained together or separately.

Proof of Theorem 3

Likewise, we have

Proof. Let ¯ j ∈ { 1 , ..., |Y t |} and y ∈ Y t be the ground truth of an x w.r.t. the within-task index and class label, and y = Y i, ¯ j for any i ∈ { 1 , ..., t } . Eq. (23) suggests that if we can improve either the WTP performance P ( x ∈ X i, ¯ j | x ∈ X i , D , θ ) , the TII performance P ( x ∈ X i |D , θ ) , or both, then the DIL performance P ( x ∈ X y |D , θ ) would be improved. However, such an improvement is limited since it is upper-bounded by WTP or TII. To further improve the DIL performance, we propose a hierarchical decomposition of the objective. That is, besides the improvement of P ( x ∈ X ∗ , ¯ j |D , θ ) , we also need to directly improve the performance of task-adaptive prediction (TAP), denoted as P ( x ∈ X y |D , θ ) , where y ∈ { 1 , ..., |Y t |} , X y represents the domain of class y in all observed domains, and X y = ⋃ i X i, ¯ j . Then the final goal of DIL is formulated as a multiobjective optimization problem, i.e., max[ P ( x ∈ X ∗ , ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] . Notice that the TII probability is a categorical distribution over all observed domains { 1 : t } , while the TAP probability is over all observed classes ⋃ t k =1 Y k .

As similarly defined in CIL, here

where γ = { γ i } t i =1 represents the possibility of x belonging to each observed domain, γ i ∈ [0 , 1] and ∑ i γ i = 1 .

Then, for any simplex γ , we have

Taking expectations on Eq. (26), we have

Taking expectations on both sides of Eq. (27), we have

Since our objective of DIL with pre-training is max[ P ( x ∈ X ∗ , ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] , then we have the loss error

This finishes the proof.

Theorem 4 For domain-incremental learning with pre-training, if the loss error L ≤ ξ , then there always exist (1) a WTP, s.t. H WTP ≤ ξ ; (2) a TII, s.t. H TII ≤ ξ ; and (3) a TAP, s.t. H TAP ≤ ξ .

Proof of Theorem 4 For DIL with pre-training, its loss error L = max[ L 1 , L 2 ] ≤ ξ . Assume x ∈ X ∗ , ¯ j ⊆ X y . According to the proof of Theorem 3, if we define P ( x ∈ X i, ¯ j |D , θ ) = P ( x ∈ X ∗ , ¯ j |D , θ ) , we have

̸

Likewise, if we define γ i = 1 and γ i ′ = 0 for i ′ = i , we have

In addition, we have formulated the final goal of DIL as a multi-objective optimization problem, i.e., max[ P ( x ∈ X ∗ , ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] . Then, each objective must guarantee the loss error less than ξ , i.e.,

This finishes the proof.

A.3 Task-Incremental Learning (TIL)

̸

For task-incremental learning (TIL), let X t = ⋃ j X t,j and Y t = {Y t,j } , where j ∈ { 1 , ..., |Y t |} indicates the j -th class in task t . Now assume we have a ground event denoted as D = {D 1 , ..., D t } and a pre-trained model f θ . For any sample x ∈ ⋃ t k =1 X k , a general goal of the TIL problem is to learn P ( x ∈ X ¯ i,j | x ∈ X ¯ i , D , θ ) , where ¯ i ∈ { 1 , ..., t } and j ∈ { 1 , ..., |Y ¯ i |} . This can be equivalent to within-task prediction (WTP). Different from CIL, the task identity is provided in TIL. Unlike DIL, Y t ∩ Y t ′ = ∅ , ∀ t = t ′ . Then we have the following theorems in terms of the sufficient and necessary conditions for improving TIL with pre-training.

Theorem 5 For task-incremental learning with pre-training, E x [ H TII ( x )] = 0 , and task-adaptive prediction (TAP) is degraded into within-task prediction (WTP). If E x [ H WTP ( x )] ≤ δ , we have the loss error L ∈ [0 , δ ] .

Proof of Theorem 5

Proof. For task-incremental learning (TIL) with pre-training, assume E x [ H WTP ( x )] ≤ δ . Given an x with the task identity ¯ i ∈ { 1 , ..., t } , let ¯ j ∈ { 1 , ..., |Y i |} be the ground truth of x w.r.t. the within-task index, and y = Y ¯ i, ¯ j be the ground truth label of x .

As similarly defined in CIL, here

Then, we have

Taking expectations on both sides of Eq. (37), we have

Since our objective of TIL with pre-training is P ( x ∈ X ¯ i, ¯ j | x ∈ X ¯ i , D , θ ) , then we have the loss error L ≤ δ . This finishes the proof.

Theorem 6 For task-incremental learning with pre-training, if the loss error L ≤ ξ , then there always exists a WTP, s.t. H WTP ≤ ξ .

Proof of Theorem 6

For TIL with pre-training, its loss error L ≤ ξ . Assume x ∈ X ¯ i, ¯ j ⊆ X ¯ i . According to the proof of Theorem 5, we have

This finishes the proof.

B Impact of Pre-Training on Continual Learning

In this work, we focus on continual learning with pre-training, especially prompt-based continual learning that receives significant attention in this direction. Our theoretical contribution lies in the context of pre-training , where we demonstrate the sufficient and necessary conditions to achieve good continual learning performance. This is clearly different from those previous theorems on continual learning from scratch [13], which is analyzed below.

First, the condition is different. We formulate the hierarchical components as θ -conditional probabilities, i.e., P ( x ∈ X ¯ i,j | x ∈ X ¯ i , D , θ ) , P ( x ∈ X i |D , θ ) and P ( x ∈ X c |D , θ ) for WTP, TII and TAP, respectively, where θ captures the pre-trained knowledge in initialization. When training from scratch, since the randomly-initialized parameter set θ 0 carries no information and is greatly different from the optimal solution, it needs be substantially changed in continual learning and should not be took

into account in objectives. In contrast, the pre-trained parameter set θ carries beneficial knowledge for downstream tasks. If the pre-training is adequately strong, θ is already close to the optimal solution and only requires appropriate fine-tuning. Therefore, θ needs to be stabilized (usually frozen) in continual learning and should be considered in objectives.

̸

Then, due to the incorporation of θ , the sufficient and necessary conditions to achieve good continual learning performance become different. In addition to WTP and TII, TAP is especially required in the problem of continual learning with pre-training. Without considering the impact of pretraining, i.e., training from scratch, TAP is equivalent to TII * WTP, i.e., P ( x ∈ X y |D , θ ) = P ( x ∈ X ¯ i |D , θ ) P ( x ∈ X ¯ i, ¯ j | x ∈ X ¯ i , D , θ ) , corresponding to δ + ϵ = η in our Theorem 1. Therefore, TAP is not required when training from scratch, consistent with the theorems in [13]. On the other hand, with considering the impact of pre-training, TAP and TII * WTP are formulated as two different prediction problems where P ( x ∈ X y |D , θ ) = P ( x ∈ X ¯ i |D , θ ) P ( x ∈ X ¯ i, ¯ j | x ∈ X ¯ i , D , θ ) . Therefore, the final performance of having TAP in continual learning can outperform that without TAP, i.e., max[ P ( x ∈ X ¯ i, ¯ j |D , θ ) , P ( x ∈ X y |D , θ )] > P ( x ∈ X ¯ i, ¯ j |D , θ ) .

In the case of prompt-based continual learning, without considering the impact of pre-training, TAP = TII * WTP, due to classifying the same input in the same semantic space, i.e., exp( h ψ ( f ( x ))[ y ]) ∑ t j =1 ∑ y ′ ∈Y j exp( h ψ ( f ( x ))[ y ′ ]) = exp( ˆ h ω ( f ( x )[ ¯ i ]) ∑ t i =1 exp( ˆ h ω ( f ( x )[ i ]) ∗ exp( h ψ ( f ( x ))[ ¯ j ]) ∑ j ∈Y i exp( h ψ ( f ( x ))[ j ]) . With considering the

the backbone parameters are frozen to stabilize the pre-trained knowledge, with additional prompts p i to fine-tune the semantic space (i.e., representations). TAP is essentially classifying the instructed representations, i.e., h ∈ H i,c in Eq. (12), while TII * WTP is performed on uninstructed representations, i.e., ˆ h ∈ ˆ H i,c in Eq. (11). Targeting on different semantic spaces, TAP and TII * WTP are clearly different, i.e., TAP != TII * WTP. Strong empirical results also demonstrate their respective contributions to continual learning performance (see ablation study in Table 3).

C Implementation Details

In this section, we describe the implementation details of all experiments.

Prompt-Based Approaches : We follow the same implementations of the prompt architectures for all baselines as their original papers [41, 40, 30] (except S-Prompt++), which have been shown to yield strong performance. Specifically, L2P [41] is implemented with M = 30 as the total number of prompts ( M = 10 [41] and M = 30 [40] have similar results), L p = 5 as the prompt length, and N = 5 for the TopN keys. DualPrompt [40] is implemented with L g = 5 as the prompt length of task-sharing prompts g inserted into layers 1-2 and L e = 20 as the prompt length of task-specific prompts e inserted into layers 3-5. S-Prompt++ [39] is implemented similarly to DualPrompt but replaces all task-sharing prompts with task-specific prompts, i.e., the task-specific prompts are inserted into layers 1-5 with prompt length L e = 20 . CODA-Prompt [30] is implemented with M = 100 as the total number of prompts and L p = 8 as the prompt length, inserted into the same layers 1-5 as DualPrompt and S-Prompt++. HiDe-Prompt adopts a similar architecture as S-Prompt++, but replaces the task-specific keys with an auxiliary output layer ˆ h ω to predict the task identity and further preserves statistics of uninstructed and instructed representations. The hyperparameters for HiDe-Prompt are set to α = 0 . 1 , τ = 0 . 8 , and λ = 0 . 1 .

Training Regime : Following the implementations of previous work [41, 40], we employ a pre-trained ViT-B/16 backbone, an Adam optimizer ( β 1 = 0 . 9 , β 2 = 0 . 999 ) and a batch size of 128. The learning rate is set to 0.001 with cosine decay for CODA-Prompt (empirically validated in Table 6), compared to 0.005 for other approaches. We then grid search for an appropriate number of epochs ( E ) under different pre-training paradigms, and observe that the best choice is generally consistent on each benchmark (i.e., E is relatively insensitive to the pre-training paradigms). For Split CIFAR-100 with E ∈ { 5 , 10 , 20 , 40 } , we set E = 10 for L2P and E = 20 for other approaches. For Split ImageNet-R with E ∈ { 25 , 50 , 75 , 150 } , we set E = 50 for all approaches. For 5-Datasets with E ∈ { 5 , 10 , 20 , 40 } , we set E = 40 for supervised pre-training and E = 20 for self-supervised pre-training. For Split CUB-200, we set E = 50 for all approaches.

Evaluation Metric : We focus on three evaluation metrics for continual learning, including the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM). Specifically, we define the accuracy on the i -th task after learning the t -th task as A i,t , and define the average accuracy of all seen tasks as AA t = 1 t ∑ t i =1 A i,t . After learning all T tasks, we report FAA = AA T , CAA = 1 T ∑ T t =1 AA t , and FFM = 1 T -1 ∑ T -1 i =1 max t ∈{ 1 ,...,T -1 } ( A i,t -A i,T ) . FAA is the primary metric to evaluate the final performance of continual learning, CAA further reflects the historical performance, and FFM serves as a measure of catastrophic forgetting.

Compute : We run all experiments of Split CIFAR-100 on eight Tesla P100-SXM2 GPUs, Split ImageNet-R on four NVIDIA A100 GPUs, 5-Datasets on both, and Split CUB-200 on eight NVIDIA GeForce RTX 3090 GPUs.

D Extended Results

In this section, we provide some extended results for the main text.

Reproduction of Baselines : All results of L2P, DualPrompt and CODA-Prompt are reproduced from their official implementations, while S-Prompt++ is implemented with the code of DualPrompt. We compare the reported and reproduced results in Table 5. In particular, we use the same Sup-21K checkpoint as the original papers of L2P [41] and DualPrompt [40], and their reproduced results are comparable to the reported ones. In contrast, the original paper of CODA-Prompt [30] used a different supervised checkpoint, which is first pre-trained on ImageNet-21K in a self-supervised fashion and then fine-tuned on ImageNet-1K in a supervised fashion. Using the same supervised checkpoint as [30], we have justified that the results of CODA-Prompt on Split CIFAR-100 can be faithfully reproduced. As for Split ImageNet-R, we obtain the official code directly from the authors of CODA-Prompt, but it cannot reproduce exactly the reported results on Split ImageNet-R due to some clean-up issues. Consequently, the reproduced results of CODA-Prompt on Split ImageNet-R slightly underperform the reported ones under the same setting [30]. Besides, we observe that [30] used a different data split than originally proposed [40], resulting in an increase of around 3% in FAA. Together with the differences in supervised checkpoints, the reproduced results of CODA-Prompt in Table 1 and Table 5 differ by around 4% in FAA.

Learning Rate of CODA-Prompt : As discussed in the original paper of CODA-Prompt [30], its performance depends heavily on the reduced learning rates, especially on Split ImageNet-R. Here we extensively evaluate the effect of learning rate on the performance of CODA-Prompt (see Table 6), in order to justify the strength of our reproduced results under different pre-training paradigms. It can be clearly seen that using a smaller learning rate with cosine decay (i.e., LR=0.001 Cosine) is generally a good choice for CODA-Prompt, which is consistent with its original paper and therefore employed to reproduce its performance in this work.

Statistical Modeling : Here we evaluate the effect of statistical modeling. As analyzed in Sec. 4.2, preserving a dedicated mean and covariance for each class of representations can faithfully recover their distributions and thus achieves strong continual learning performance (see Table 7). Under Sup21K that is adequately strong for downstream tasks, the performance remains essentially consistent when reducing the covariance to variance (e.g., the FAAs are 91.74% and 73.97% on Split CIFAR-100 and Split ImageNet-R, respectively). As a more generalized form of approximated distributions,

reserving fewer than 10 centroids (around 5 on average) 6 for each class achieves comparably strong performance in all cases (see Table 7), which ensures both efficiency and generality.

5-Datasets : In Table 8, we evaluate the continual learning performance on 5-Datasets for large inter-task differences. We keep the implementation of prompt architecture for all approaches the same as in Split CIFAR-100 and Split ImageNet-R. Then we perform an extensive grid search for learning rate, number of epochs and other applicable hyperparameters with their official or commonlyused codes. Under Sup-21K, we can essentially reproduce the results of L2P [41]. However, the reproduced results of DualPrompt [40] is lower than the reported one. We find that this issue is also discovered by other users and remains open in github. As the original paper of DualPrompt [40] has not described precisely the implementation for 5-Datasets, we speculate that this issue is possibly due to the use of different prompt architectures. A supporting evidence is that the reported results of DualPrompt [40] are similar to those of S-Prompt++, which is equivalent to replacing all task-sharing prompts in DualPrompt with task-specific prompts. Besides, the most recent CODA-Prompt [30] also performs poorly on this challenging benchmark. To ensure the strength of reproduced results, we have extensively searched the learning rate in { 0 . 001 , 0 . 0005 } , the number of epochs in { 5 , 20 , 40 } and the hyperparameter of orthogonality regularization in { 0 , 0 . 1 , 0 . 01 } , and presented the best performance. The inferiority of CODA-Prompt is possibly due to the excessive attentions to the previously-learned prompts, since the task distributions are clearly different and the old knowledge might interfere with the new one. To support this claim, we analyze the attentions of CODA-Prompt and observe a strong preference for previously-learned prompts (around 0 . 6 ∼ 0 . 8 ). Compared to all competitors, HiDe-Prompt achieves the strongest performance under different pre-training paradigms, consistent with the results on Split CIFAR-100 and Split ImageNet-R in Table 1. In particular, the FFM of HiDe-Prompt is almost zero, indicating its great success in overcoming catastrophic forgetting.

Visualization of Representations : We visualize the uninstructed and instructed representations with t-SNE, as shown in Fig. 5 for Sup-21K and Fig. 6 for iBOT-21K. Based on these results, we have the following analysis: (1) The uninstructed representations have shown single-peaked patterns in general, thanks to the use of adequate pre-training. This property allows them to be approximated with Gaussian distributions for preservation and recovery, and allows for correct prediction of task identity from them. (2) The instructed representations tend to be more compact and distinguishable, validating

Task 1

Task 3

Task 4

Task5

Representations

Uninstructed

Representations

Instructed

Task6

Task7

Task 8

Task9

Task 10

Representations

........

Uninstructed

装

州：

Representations

Instructed

Figure 1:

Visualization of uninstructed and instructed representations with t-SNE. Here we present the results of Split CIFAR-100 under Sup-21K. Each color represent a class. the effectiveness of prompt-based continual learning. The degree of compactness and differentiation varies across pre-training paradigms, contributing to their performance differences. (3) The differences in compactness and differentiation between uninstructed and instructed representations suggest that the design of coarse-grained classification by task and fine-grained classification by class is reasonable for prompt-based continual learning, as do our theoretical analysis and the proposed approach.

1

Task 1

Task 2

Task 3

Task 4

Task 5

Representations

......

廉

Uninstructed

蒙

+i

Representations

Instructed

囍

Task6

Task7

Task8

Task9

Task 10

Representations

Uninstructed

Representations

春

Instructed

..

Peer reviews:
comments: SUMMARY:
This paper introduces a CIL approach that specifically addresses the use of pre-trained models. It is intriguing and significant to explore the most suitable CIL method for pre-trained models.

STRENGTHS:
1. CIL and how to do CIL in pre-trained models are very important
2. The writing is smooth and very easy to follow

WEAKNESSES:
Excluding the assumption of a pre-trained model, this paper reduces to the existing research (as mentioned in line 233). Thus, I recommend conducting further analysis on CL for pre-trained models.

1. Considering the extensive exploration of pre-trained models in NLP, it is crucial for this paper to compare its approach with NLP continual learning baselines, e.g., [1,2,3].

2. I see no reason why the theory should be restricted to prompts alone. Other parameter-efficient tuning methods such as adapters, Lora, and Prefix can easily be utilized. The author should include a comparison with these methods (see 1 references).

3. In addition, I suggest comparing this paper's approach with TIL methods as well, which do not require TII. This would serve as an additional evaluation to assess the proposed method's performance in  WTP and TAP.

[1]: Achieving forgetting prevention and knowledge transfer in continual learning, NeurIPS 2021  
[2]: Continual Pre-training of Language Models, ICLR 2023  
[3]: Continual Learning of Natural Language Processing Tasks: A Survey. https://arxiv.org/abs/2211.12701

QUESTIONS:
See above

LIMITATIONS:
I think the discussion of limitations in Sec. 6 makes sense.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides a comprehensive analysis of state-of-the-art prompt-based approaches for continual learning with the use of pre-training. The authors empirically demonstrate a clear performance degradation of current strategies under realistic self-supervised pre-training and extensively analyze the exposed sub-optimality. The authors then provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into three hierarchical components, and propose an innovative approach to optimize them explicitly. Extensive experiments on various pre-training paradigms have demonstrated the clear advantages of the proposed method.\n\nSTRENGTHS:\n1. This paper provides a well-organized formulation of state-of-the-art prompt-based approaches, assessing them as a unified perspective.  \n2. The empirical analysis reveals the degraded performance of prompt-based approaches under self-supervised pre-training, which is an important issue for practical applications.  \n3. The theoretical analysis is very interesting. The hierarchical components are specific to continual learning in the context of pre-training and apply to the settings of task-/domain-/class-incremental learning.  \n4. The proposed method allows for explicit optimization of the continual learning objective through adaptively leveraging pre-training and prompt architectures. Experimental results demonstrate a significant improvement in continual learning performance under different pre-training paradigms.\n\nWEAKNESSES:\n1. Based on proofs in supplementary materials, is the notation $\\bar{c}$ in Eq.(8) equal to $y$? Please check it.  \n2. Could the authors further discuss some concurrent related work such as PromptFusion [1] that exploits prompt and FSA [2] that exploits FiLM adapters for continual learning with pre-training.  \n[1] PromptFusion: Decoupling Stability and Plasticity for Continual Learning. arXiv preprint arXiv:2303.07223.  \n[2] First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning. arXiv preprint arXiv:2303.13199.  \n3. Although experimental results have been sufficient enough, it would be better if some fine-grained datasets can be further analyzed, such as CUB.\n\nQUESTIONS:\nSome minor concerns remain to be addressed. Please refer to the Weakness.\n\nLIMITATIONS:\nThe authors have properly discussed the limitations and potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper studies application of prompts in pre-trained models for continual learning. Building upon the problem of [1], the authors introduce the Task-Adaptive Prediction (TAP) for the CIL problem using pre-trained networks. They demonstrate that a good TAP, WTP, and TII performances are necessary and sufficient for a good CIL model. 
The authors emphasize the significance of these factors, especially in the context of continual learning with pre-trained self-supervised model.

[1] Kim et al., Theoretical study on solving continual learning. NeurIPS 2022

STRENGTHS:
- The paper is easy to follow and the proposed approach is well motivated
- The paper has provided theoretical insights
- The proposed method is much stronger than the existing baselines

WEAKNESSES:
- The method heavily relies on pre-trained models trained with ImageNet. This can be an issue, especially for supervised pre-training, since ImageNet already contains classes similar or identical to classes used for CL and there could be information leak from the pre-training classes to continual learning classes [2].
- The theoretical contribution is somewhat ambiguous since the decomposition and theorem 1 and 2 are similar to [1].

[2] Kim et al., A multi-head model for continual learning via out-of-distribution replay. CoLLAs, 2022.

QUESTIONS:
- I couldn’t fully-understand why it’s necessary to have TAP in the CL problem using pre-trained networks, considering that it wasn’t required when training from scratch [1].
- I am curious about the performance of the proposed method when the feature extractor is pre-trained using only the classes that are dissimilar to CL classes as [2]
- Is the theoretical analysis only relevant to the prompt-based method?

LIMITATIONS:
Refer to Weaknesses and Questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide strong empirical analysis on existing \"prompting for continual learning\" papers. They propose a new hierarchical prompting method that includes several components which take advantage of unstructured data representations. The authors not only propose an interesting approach with SOTA performance, but they have very detailed experiments using several pre-training backbones and datasets.\n\nSTRENGTHS:\nIn general, I feel that for this subject area (which I am sure will have many neurips submissions), this paper is overall high quality.\n1) I appreciate another \"prompting for continual learning\" paper that starts with insights and findings into existing methods, laying an intuitive foundation for the proposed approach.\n2) With good intuitive and theoretical backing that seems to reasonable, the method has great performance gains over SOTA. \n3) Nice, detailed analysis and ablation study. I checked carefully to see if maybe there was a \"hidden\" component that does most of the heavy lifting of the method, but the contribution components seem to all be impactful in some way.\n4) I am incredibly thankful to see the experiments conducted on several different backbones. This code and framework will be a fantastic contribution on its own, along with the analysis, before even considering the method. As someone who has experience in this problem setting, I think open-source code for this project will be greatly appreciated.\n\nWEAKNESSES:\n1) Task-id prediction for class-incremental learning on random splits of classes (i.e., where there is no actual task structure), seems a red flag to me that the pre-trained backbone is too strong for the proposed task datasets, and that it is actually taking advantage of the unfair backbone in order to \"circumvent\" class-incremental learning, instead posing it as the much easier task-incremental learning problem. However, this weakness also exists in the competing methods, and the strengths of the paper out-weigh this weakness imo.\n2) Figure 2 c and d is very confusing. Which method is this for? If you are making strong claims about task specificity, should it not be done for all methods?\n3) This is not relevant to your findings, and I am not requesting any experiments, but Coda-prompt might perform better at a lower constant learning rate if the high, constant learning rate is hurting its performance. Learning rate decay is also not a \"trick\", either - it is a valid hyperparameter choice and important for many representation learning methods.\n\nQUESTIONS:\na) See weakness 2\nb) what is the computation cost of your methods compared to the baselines? I feel that even with a strong training time cost, I am happy to vote for paper acceptance. Just want to ensure transparency.\n\nLIMITATIONS:\nDiscussed reasonable limitations - there is no obvious potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper suggests a new prompt-based continual learning method by leveraging combinatorial objectives with *within task prediction*, *task-identity inference*, and *task-adaptive prediction*. The paper first summarizes recent prompt-based continual learning techniques and demonstrates their unstable performance according to the pre-trained backbones (particularly, initializing with self-supervised representation degrades their performance significantly). And then, proposes a new regularization technique containing core components, including ensemble prompting, contrastive regularization, and TAP loss.

STRENGTHS:
The paper provides a comprehensive analysis and theoretical proof of the motivation. And the suggested idea is reasonable, and the methodological design is also clear. In experiments, the proposed method consistently surpasses strong baselines in terms of conventionally used continual learning metrics under multiple pre-trained backbones, and its improvement and stability are a bit impressive.
Further, they provide the necessary ablation study and analysis in the main paper and appendix.

WEAKNESSES:
The effect of the prompt ensemble in the method is not discussed. And the different behavior and impact of un-/instructed representation for continual learning are partially discussed in Figure 5, but I recommend a more comprehensive and detailed discussion/analysis.

QUESTIONS:
N/A

LIMITATIONS:
Please see the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09