Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality
Authors: Liyuan Wang
Jingyi Xie
Xingxing Zhang
Mingyi Huang
Hang Su
Jun Zhu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Continual Learning
Catastrophic Forgetting
Pre-training
Prompt Tuning
TL;DR: We extensively analyze state-of-the-art prompt-based continual learning from both empirical and theoretical perspectives, and propose an innovative approach to explicitly optimize its objective.

Abstract: Prompt-based continual learning is an emerging direction in leveraging pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical research reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome the exposed sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Following these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively).

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality
Liyuan Wang', Jingyi Xie', Xingxing Zhang'; Mingyi Huang', Hang Su', Jun Zhu'
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab, 'singhua-Bosche Joint Center for ML, Tsinghua University, Beijing, China.
wly19@tsinghha. org. en, jingyi_xiess xie9601633 com, xxzhang1993e huangmingy120028122 com, (suhangss, dcszj) @tsinghua edu..ccducoooooo cn
com
Abstract
Prompt-based continual learning leverages pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under super- vised pre-training. However, our empirical study reveals that the current strategies fall short of their full potential under the more realistic sulf-super pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome such sub-optimality, we conduct a theoret- ical analysis of the continual learning objective in the context of pre-training, and decompose into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Based these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompp) an innovative approach that explicitly optimizes the hierarchical components with an ensemble spec-ip prompts and statistics of both uninstructed and instructed represen- tations. further with the coordination of contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompp and its robustness to pre- pre-training paradigms in continual learning (e.g., up to 15.01% and 9.61 % lead on Split CIFAR-100 100 and Split ImageNet-R, respectively). Our code is available at htttts://gggiiiiee thub..b..tttttttto com com/thu-ne HiDe-iDeePPPee rompt:
Introduction
In the realm of artificial intelligence, continual learning has become an area of significant interest. One of the pivotal techniques that greatly facilitate this domain is pre- training, which only delivers positive knowledge transfer but also enhances resilience to catastrophic forgetting [27 23 37 26 43) recent innovation is the implementation of prompt-base baset methodologies, which freeze a pre- trained transformer backbone and employ a few prompt parameters to steer representation learning. Such approaches typically involve construction of adaptive prompts for each task and inference of appropriate prompts during the test phase. By exploring prompt architectures to accommodate task-sharing and ssecificc knowledge, this emerging direction demonstrates distinct superiority, almost reaching the upper bound of continual learning performance under supervised pre-training.
Nonetheless, given that robust pre-trained models typically necessitate the learning of substantial amounts of unlabeled data in self-supervised manner, the influence of pre-training paradigms
"Cooresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

on the effectiveness of prompt-based continual learning represents significant and unresolved query. To answer this question, we first perform an extensive empirical investigation, and it clearly demonstrates the sub-optimality of recent prompt-based approaches under the more realistic self- supervised pre-training- Since self-supervised representations tend to more general, sesk-specifiic knowledge is difficult to incorporate into instructed representations via prompt parameters, as well as predicted by uninstructed representations at test time. Consequently, the performance of many recent approaches, such as L2P [41]. DualPrompt 40), S-Prompt [39] and CODA- Prompt [30], is seriously compromised. We further disclose the importance of adaptive prediction for all tasks together, which can potentially mitigate the aforementioned shorteomings to some extent.
Motivated by these observations, we provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into hierarchical components such as within-task prediction, task-identity inference and task-adaptive prediction. Thanks to the well-distributed representations resulting from adequate pre-training, the hierarchical components can be optimized explicitly by constructing an ensemble of task-specific prompts and exploiting the preserved statistics of uninstructed and instructed representations. A novel contrastive regularization is further devised to coordinate these hierarchical components. We refer to this approach as Hierarchical Decomposition (Hiiee)Promptt and demonstrate its superiority through extensive continual learning experiments, especially under the more realistic self-supervised pre-training.
Our contributions include: (I) We provide an extensive empirical study under self soff-supeviiee pre- training to demonstrate the sub-optimality of current progress in prompt-based continual learning: (2) To overcome such sub-optimality, we theoretically analyze the objective of continual learning with pre-training, and decompose into hierarchical components for model design; (3) With task-specific prompts and representation statistics, we propose an innovative approach to optimize the hierarchical components explicitly; (4) Across various continual learning benchmarks and pre-training paradigms, our approach achieves clearly state-of-the-art performance in rehearsal-free manner.
2 Related Work
The ability of continual learning is critical for artificial neural networks to accommodate real-world changes [37
Continual Learning:
(5) Numerous efforts in this direction have been devoted to
According to a recent survey [37]. representative strategies include selective stabilization of network parameters, replay of a few old training samples, explicit manipulation of optimization programs, exploitation of well-distribtted representations, construction of task-specific parameters, etc. The performance of such strategies varies with particular settings of continual learning. As one of the most challenging and representative settings, class- incremental learning (CIL) 31 37 requires a continual learning model to perform all old tasks (or classes) without the oracle of task identity. Strong CIL methods generally depend on storage and rehearsal of old training samples (28 8 38), which result in efficiency and privacy issues.
overcoming catastrophic forgetting [22 34 3
The exploitation of well-distributed representations. especially from the success of large-scale pre-training, brings significant benefits for downstream continual learning 337 27 23 Due to the scarcity and expense of explicit labeling in many real- world applications, self-supervised learning is typically involved in the pre- pre- training stage to cope with huge amounts of unlabeled data. In particular, instance discrimination
Self-Supervised Learning and Tre-Training:
with contrastive learning 25 has become the dominant strategy, which aims to maximize representation similarity of the same instance and minimize representation similarity different instances. Besides, self-supervised paradigms have been shown less sensitive to catastrophic forgetting in upstream continual learning 91. providing a practical way to enrich pre-trained knowledge from in-the-wild data.
Prompt-Based Approach:
Inspired by parameter-efficient fine-tuning techniques in NLP [1,10,
recent prompt-based approaches [3 41 40 39 30 are developed to leverage pre-trained knowledge adaptively for downstream continual learning. The basic idea includes construction and inference of adaptive prompts for each task, so as instruct frozen transformer backbone. The former mainly focuses on exploring prompt architectures to instruct representations with task-sharing and task-specific knowledge, closely related to the discussion of model architectures in continual learning [37 36), while the latter attempts to predict appropriate (combinations of) prompts with uninstructed representations. Although such methods have achieved remarkably strong performance under supervised pre-training, whether these advantages consistent under the more realistic self-

---

supervised pre- treinran remains to be explored. A concurrent study [ observed that self supervise pre-training is more challenging for continual learning approaches that require fine-tuning of the backbone, implying a non-trivial impact of pre-training paradigms on downstream continual learning.
3
Preliminary Analysis
In this section, we first introduce the problem formulation of prompt-based continual learning, and then evaluate the impact of pre-training paradigms with an extensive empirical study.
3.1 Formulation of Prompt-Basee Continual Learning
Continual learning aims to learn sequence of tasks on their respective training sets Di,..., Dr and excel on their corresponding test sets. The training set for task typically consists of various data- label pairs D. - { Vt.n) J. J== ) where Øi,n E X2 and Vt., 3 represent the sample and label elements, respectively. We will use |-1 to denote the cardinality of a set and [N]
... NN} as the set of intergers from to N. Consider a neural network model with a backbone fo parameterized by 0, and an output layer he parameterized by 인b This model seeks to learn the projection from X' = X₂ to y' = U Vi. aiming to predict the label y (.,(So(z)) E Y of an unseen test sample æ drawn from previous tasks. The backbone function So is assumed to be pre-trained with a substantial quantity of additional training samples external to each D.. There are commonly three distinct settings for continual learning [31]: task- domain-, and class-increments, learning (TIL, DIL, and CIL). Specifically, V1, .... Pr are identical for DIL while disjoint for TIL and CIL. The task identity is provided for TIL. at test time but is not available for DIL. and CIL. Therefore, CIL considered to be more representative and challenging in general. Of note, the continual learning 330 all elements of DD are available only when learning task t.
process is rehearsal-free
Prompt-based approaches for vision tasks further specify the backbone fo as a pre-trained vision transformer (ViT), where multiple consecutive multi-head self-attention (MSA) layers can transform an input sample into sequence-like like output representation h E R R xD of sequence length L, and embedding dimension D. The backbone parameters 0 are typically frozen to obtain generalizable representations. A few prompt parameters
E RD of sequence length Lp and embedding dimension D are prepended to h to exploit the pre-trained knowledge adaptively. Here we denote the input of the 1-th MSA layer as h' E XDD, which consists of query hig. key h'g and value hiy
and denote the prompt as p' E RL,IDD, For notation clarity, we take one MSA layer as an example and omit the layer label I if not necessary. Then, the output of this MSA layer is given as
MSA(hq, hx, hy) Concat(ha, ..., hm. Wo.
(1)
h Attention(hq WQ. Wx,W hy, E [m].
(2)
where Wo, Wo,,. WK, and Wv,, are projection matrices, is the number of heads, and ho hx = by in ViT. There are two major implementations of prompt-based methodologies [40], i.e. Prompt Tuning (ProT) [8] and Prefix Tuning (Pre'T) [9]. Specifically, ProT prepends an identical to hq. hg and hy:
(p, h) MSA(Pp: ho], [p: hk], [p; hvv),,
(3)
where [:5-] denotes the concatenation operation along the dimension of sequence length, and the output in (R + +ppx D has increased dimensions. In contrast, PreT splits p into Px E /22xD and Pv E IR ////2XD only for hg and hv, respectively:
SPINT (p, ft) = MSA(ko, Px: few,, [Pvikv]).
(4)
where the output dimension remains the same as the input h E RLaxD As the training samples for each task are introduced sequentially, prompt-base based continual learning needs to incorporate task-specific knowledge into prompt parameters while overcoming their catastrophic forgetting. The mainstream idea is to construct adaptive prompts for each task and then infer appropriate (combinations of) prompts at test time. Here we compare state-of-the-art approaches from these two aspects, demonstrated conceptually in [ig.
L2P 41
constructs prompt pool P {Pi,..., pM) potentially shared by all tasks where M is the total number of prompts, and then instructs the last MSA layer in a ProT fashion. Each prompt p; is associated to a learnable key A; E D. optimized by the cosine distance >((()), k,) of the top-N keys to a query function ((x) = fe( to incorporate knowledge. Therefore, the most relevant keys and the corresponding prompts can be selected with uninstructed representations for inference.
3

---

DualPrompt [40] constructs task- sharing prompts g' and task-ipe prompts e:c to instruct different MSA layers in PreTT fashion, All e, belonging to the same task is as- sociated to a task-specific key k; E RD, optimized by (((()), k2), and
intementaal Iiguts
Corstruction Prompito
Lap
DuaiPropt
Pru husned
the index of the best-matched key is selected for inference.
6-Phumpt
80 Inserted inn MBA Layer54
S-Prompt [39] constructs only task- specific prompts e, for each task, and adopts similar ProT strategy as L.2 to instruct the last MSA layer. The inference of task iden- tity is achieved by simple KNN strategy for the nearest task centroid. Unlike other methods, S-Prompt associates an exclusive output head 1/ to each task / 1,..,,
Prediction or Taoos
LearaOo
(1)lrrerecce of Prompt identity
Layer
Figure 1: Illustration of prompt- based continual learning.
exploits the prompt pool P by its weighted summation, i.e., Σ i= OsP,, where a; = (q()), k;) is the weighting factor, and adopts similar PreTT strategy as DualPrompt to instruct multiple MSA layers. The inference of a; enables construction of adaptive prompts.
CODA-Prompt 30
3.2 Empirical Study of Pre-Training Paradigms
Either explicitly implicitly, the above prompt-based approaches all incorporate the knowledge of each task into prompt parameters and predict their identities from uninstructed representations. To evaluate the impact of pre-training paradigms, we perform empirical study with widely-used CIL benchmarks such as Split CIFAR-100 and Split ImageNet-R 41 40]. In addition to supervised pre- training of ImageNet-21K [29] (denoted as Sup-2 (K). consider several powerful self-supervised models that release ViT checkpoints) such as iBOT [44),
DINO 2 and MoCo v3 [5].
30m
75
(S)
CIFAR-100
Final Avarage Rccurac [3)
ImagaNot-R
70-
80-
66-
Duslirrmpt
Schomptrt
CODA-Phropp LR Reduction
LOP
DustPhompp
S-Prompte
COD4-Prompl
Acquisition of Task Spacificty
Prodiction of Task Idanitty
Prediction ofAl Toeks
80-12
Sup-21K
0,901
00.01
0.882
Took Inference Accurscy
S-Prompe+
S-Prompt
IBOT-21K
IBOT-KK
CKA Simberlty
0.8-
0.774
00090
0.021
24.74
17.55
56
0.50
56
14.00
500
DINO-1K
15.71
0.8-
0.088
McCo-1K
0.6
Dm
S0ILCFFRR100 100
Solt ImageNot.R
Solt CIFAR-100
Soll ImagoNosR
Spitt CIFAR110
Spit Inlgeeice
Figure 2: Empirical study of prompt-based continual learning under different pre-training paradigms.
We carefully evaluate the official implementations of all baselines for fair comparison. We follow largely the training regimes of L.2P [TI and DualPrompt [40] which basically consistent. As S- S-Prompt (39] is initially designed for DIL, we slightly modify its implementation by inserting task-specific prompts into the same layers as DualPrompt (i.e., layers 1-5) in a PreT manner, so as to evaluate the impact of prompt architectures. The output layer retains multiple heads associated
*iBOT currently achieves the first first-place performance for self-superviser classification on ImageNet and releases checkpoints both ImageNet-21K and -IK, while others only release checkpoints ImageNet-Ik IK.

---

with the task identity (still denoted as S-Prompt), or single head as with other baselines (denoted as S- s-Prompt++). CODA Prommtt [30] is officially implemented
DualPrompt- like architecture but depends heavily on the use of a smaller learning rate with cosine decay. Here we present its performance with both default and reduced learning rates. Using the same learning rate as [4] (3), we grid search for an appropriate number of epochs (detailed in Appendix D and report the best performance for all baselines.
As shown in Fig. 2 a, b, the above prompt-based approaches achieve outstanding performance under Sup-21K, where the use of task-specific prompts clearly outperforms task-sharing prompts (i.e., S- S-Prompt++ = CODA-Prompp > DualPrompt > L2P) due to explicit avoidance of catastrophic forgetting. However, the four baselines suffer significant performance degradation under the more realistic self-super pre-training, In particular, the performance differences between prompt architectures have become much smaller, suggesting that the task-specific and fask-sharing knowledge are not well differentiated. Besides, CODA-Prompt can generally achieve leading performance as direct result of the learning rate (LR) reduction rather than the prompt architecture.
We perform two additional experiments to demonstrate the obscured sub-optimality First, we evaluate the CKA similarity of uninstructed and instructed representations by learning task-specific prompts (Fig. 2 c). The CKA similarity of self-supervised pre-training is significantly higher, suggesting greater difficulty for prompt parameters to incorporate task specifpciific knowledge. Second, we evaluate the ability to predict task identity from uninstructed representations and task-specific keys, where self-supervised pre-training exhibits much lower accuracy (Fig. 2 d). Interestingly, although only less than 40% task identities are correctly predicted, S- S-Prompt++ can still achieve considerable (albeit sub-optimal) performance, owning to the compensation effects of using a single- head output layer (Fig. 2 e). Together with the results in Fig. _ d, it is conceivable that using an "incorrect" prompt would not severely affect the instructed representations, which can still be correctly predicted in a well-balanced single-head output layer. In contrast, S-Prompt performs much worse than S-Prompt++, as its multi-head output layer undertakes all errors of task-identity inference.
Theoretical Foundation and Our Approach
In this section, first present a theoretical analysis of the sufficient and necessary conditions for improving continual learning in the context of pre-training, and then present an innovative approach for prompt-based continual learning to achieve this objective.
4.1 Hierarchical Decomposition of Continual Learning Objective
For continual learning of sequentially arrived Dr, X₁ and Vd are the domain and label of task t. Here we take CIL as typical scenario for theoretical analysis where 3e n Vi 0, Vi & t' (see Appendix A for DIL and TIL). Let X; = U, and 3% = Dr where j E [Dil] indicates the j-th class in task t. Now assume we have a ground event denoted as D = (D1,, Dr} and a pre-trained model fe. For any sample E U Xx. a general goal of the CIL problem is to learn P(x E &e 0). where E [t] and j E [Di]]. This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as P(x E Xx () and P( E Xi, E xx D. (), respectively. Based on Bayes' theorem, we have
(5) Let ē [t] and J E [[DA]] be the ground truth of an w.r.t. the task identity and within-task index. Eq. (5) shows that if we can improve either the WTP performance P(x E Xx,3 x E X; D,0), 8). the
P(ar E Xx, () = P(# E / E X. D, P(= E X.D,0).
Til performance ((x= E X,(D, 0), or both, then the CIL performance P(* E X3 (D, would be improved. However, such improvement is limited since it is upper- bounded by WTP or TIL. To further improve the CIL performance, we propose a hierarchical decomposition of its objective. That is, besides the improvement of P(æ E X [D, 0), we also need to improve the performance of task-adaptive prediction (TAP), denoted as P(æ E (>) where X represents the domain of
'The experiments in Fig. 2 c, d employ the same prompt architecture and inference methodology as S- +rompt++ to explicitly demonstrate the impact of task specificity. In fact, the objective of continual learning is tantamount to optimizing the probability distribution on the left side of Eq. (5). which can be decomposed into the two probabilities on the right side corresponding to Fig. [ c, d, respectively. Therefore, such empirical analysis is representative for prompt- prompt-based continual learning from theoretical perspective.
5

---

Task- Inference (TII)
Within-Task Prediction (WTP)
TaskoAdappive Prediction (TAP)
P/ze A,
Plae AR A2,
Piz E x
Contrastive Regularization
Unissructed Represertations
Chepur
Instrcttdd Rapressetationa
Output All Tookk
Sampang
Teadso Spaecilii
Samplng
Save
Statialica co
Promge Enicerbll
Figure 3: Illustration of Hierarchical Decomposition (HiDe-) Prompt.
class y in all previous tasks, and y = -V. 3 is the ground truth label of ac. Then the final goal of CII. is formulated multi-objective optimization problem, i.e., max PP(x E (2,lD, 0), P(æ: 2 [D, 0)] Notice that the TII probability a categorical distribution over all observed tasks upto t, while the TAP probability is over all observed classes - Ok.
(D,
To resolve the problems above, we derive the sufficient and necessary conditions in the context of the widely-used cross-entrope loss. Specifically, we define
HwTr (ax = H(1,, (P(# E N, |e E (,D,))),s
(6)
Hrn (x) = H(1,,((( X/D, 0)),)
(7)
(IAP( HH1,, (P(x ID) 0)),),
(8)
where HwTP, HTH, and HTAP are the cross-entropy values of WTP, TII, and TAP, respectively. The operation H(p, 9) _ -Rppplog s) Σ, P, log ds- 1. is one-hot encoding function.
We now present the first theorem under the CIL scenario (see Appendix for a detailed proof):
Theorem
For continual learning with pre-training, T. (PF=(HWTrr (wrr(x)) S 6, Em (Hru())) [x)] S 6, and
E. [H)) S n, we have the loss error L [0, max n,7)), regardless whether WTP, THl and TAP are trained together or separately.
With the use of cross-entropy, the continual learning performance tends to be better as the bounds tightened. In Theorem we have shown that good performances of WTP, THI and TAP are sufficient to guarantee good performance of CIL. For completeness, of a well perfopeef CIL model in Theorem 2
study the necessary conditions
Theorem For continual learning with pre-training, if the loss error L S. then there always exist (1) WTP, s.t. НWTP S &; (2) a THI, s.t. HTH S E; and (3) a TAP, s.t. HTAP S ç.
Theorem 2 suggests that if continual learning model is well trained (i.e., with low loss), then the WTP, TII and TAP for sequential tasks are always implied be small. It is worth noting that without the pre-trained knowledge carried by 0, Theorem conclusion a previous theoretical study [3], suggesting that the presented theorems are particularly directed to the impact of pre-training for continual learning (detailed in Appendix B Besides, the continual learning, because it can
I and Theorem would degrade the main
paradigm of pre-training indeed related to the performance affect the distribution of representations from fe. and further WTP, THI and TAP. Previous work has demonstrated that self-supervieed representations tend to be more robust to parameter changes than supervised ones [ 21 37), which is beneficial for accumulating pre-trained knowledge (if applicable) but challenging for adapting to downstream tasks on continual basis (see Fig. 2 c, d).
4.2 HiDee-rompt for Prompt-Based Continual Learning
Motivated by the above empirical and theoretical insights, we propose to optimize explicitly the hierarchical components (i.e., WTP, TII and TAP) for prompt-based continual learning, as shown in Fig. 3 Our proposal stems from a particular advantage of pre-training, where the distributions of uninstructed and instructed representations can be effectively preserved through their statistical information. In the case of classification, for example, since each class tends to have single-peaked

---

representations (see Appendix D Fig. 5 and Fig. 6 we can naturally approximate them with Gaussian distributions. For generality, here we denote the approximated distributions of uninstructed and instructed representations as G. and ge for each class E V E [t - 1], respectively, and discuss
their specific forms latter.
First, we improve WTP through effective incorporation of task-specific knowledge. We construct an expandable prompt pool with only task-specific prompts e, incorporate the knowledge of Dr, optimized by cross-entropy (CE) loss for HwTP. The previous prompts c1.... ct-1 are frozen to avoid catastrophic forgetting. In order to transfer knowledge for learning each task effectively, we employ a prompt ensemble (PE) strategy, where the current prompt is initialized by the last prompt et 6t-1 and then optimized with weighted combination of all previous prompts
P: - a واا e; (1 er); a is a hyperparameter that controls the strength of inherited old knowledge facilitate p, in learning the current task. Meanwhile, the instructed representations of p, although allowing the new task to be performed well, may overlap with that of the old tasks and thus affect TAP. To overcome this issue, we exploit the old-task statistics of instructed representations (collected by fe and P; for i L,.., 1), 1), where for classification we calculate the mean c of Go for each class E Vr, and design contrastive regularization (CR):
exp(h H./TT
(CR(P.)= = Σ
EED
Σ
Σ log cE>i
(9)
NEW, E
EnEn, exp(h h//T) E exp(h p²/T))
where H₂ is the embedding transformation of Dr with fo and Pt. 7 is the temperature coefficient, which is insensitive and set to 0.8 in practice. Notably, here we use only Me represent each class for efficiency, which can be optionally replaced by sampling from ge for better performance.
Then, the loss function of WTP can be defined as
(10) Therefore, the instructed representations of new classes can be well distinguished for WTP while avoiding overlap with the previous ones. is a hyperparamter to balance the impact of old classes.
LWTP (v: P.) = Lce(V, + AP)
Second, we improve TII and TAP through exploiting the approximated distributions of uninstructed and instructed representations, respectively. For THI, we construct an auxiliary output layer h RRD + TR."" parameterized by w, learning explicitly the projection from uninstructed representations to task identity via cross-entropy (i.e., HTH)::
ΣΣ> Σ Lmn(w) = Σ._ DW 1=1 cE>i keA
exp(hu (h) Σj=1 (h)l))
log
(11)
where Hie is constructed by sampling an equal number of pseudo representations from &. for c E V and E [t]. Unlike other baselines that freeze the projection of old tasks (i.c., the previous keys), our hw is continually adapted for all tasks and thus greatly facilitates TII.
Similarly, the final output layer hys RD - R can be further optimized for TAP (i.c., HTAP):
(h)(c)
(TAP(U)= = Σ. DAI
Σ - log Eyn exp(h>(h))ee))
(12)
DE AeM.
where Hi.e is constructed by sampling equal number of pseudo representations from Go for c E 31 and i = 1....,t. t. As w and tb are usually light-weight, the optimization of TII and TAP is computationally efficient. At test time, HiDe-Prompt predicts the task identity i i= (f(x))) and then the label y h. [o (æ:; p.)). Please refer to Appendix Algorithm [ for more details.
Since the pre-trained representations are usually well-distributed, there are many reasonable strategies to model G. and G.. On default, the distributions of uninstructed and instruted representations can be faithfully recovered by modeling each class as Gaussian with a dedicated mean and covariance. With adequate pre- pre-training, the covariance can be further reduced to variance for efficiency. Alternatively, such statistical modeling can employ multiple centroids obtained from KNN and add Gaussian noise, which is also an efficient choice and is applicable to other task types.
5 Experiment
In this section, we first describe the experimental setups, and then present the experimental results.
7

---

Table 1: Overall performance of continual learning. We present the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM) with + standard deviation under different pre-trained models (PTM), over three runs of different random seeds and task splits.
"The original paper [30] used a different supervised checkpoint and a different data split of ImageNet-
R. We asked the authors for official code and reproduced its results under the same setting as 40
Split CIFAR-100
Split ImageNet-R
PTM
Method
FAA (t)
CA)
FFM (4)
FA)
CAA (t)
(F)
L.P
41
83.06 1.017
88.25 10.01
6.58 1:040
63.65 10.12
67.25 10.02
7.51 10.17
DualPrompl
40
86.60 10.19
90.64 10.01
4.45 10.16
68.79 10.31
71,96
10.04
4.49 10.14
Sup-21K
S-Prompt++
3
88.81 +8.18
=0.03
+0.05
10.12
72.50 #0.04
3.29 +0.05
CODA Prompi
30
86.94 1.0.63
91.57 1:.75
4.04 10.18
70.03 10.47
74.26 10.24
5.17 1:022
HiDe-Prompt (Ours)
92.61 +0.28
94.03 =0.01
3.16 100.10
75.06 10.12
76.60 +0.01
2.17 10.19
L2P 41
79.00 1:0.28
85.13 0.005
5.55 1:0.36
55.35 1:028
58.62 10005
3.73 1:0.53
DualPrompt
30
78.76 +0.23
86.16 =0.02
9.84 +0024
54.55 #0.53
58.69 =0.01
5.38 +0.70
iBOT-21K
S-Prumpt++
39
1:066
10.17
1:1.3
10.83
10.18
4.07 1:016
KODA-Prompp
30
80.83 1:0.27
87.02 1:0000
7.50 1:0.25
61.22 1:033
66.76 0..37
9.66 1:0.20
HiDe-Prompt (Ours)
93,02 15
94,56 0.05
1.33 +:0.24
70.83 =0.17
73,23 =0.08
2.46 #:0.1
L2P [1
75.57 4:0.41
82.69
06006
7.23 +:098
60.97 -0.26
65.95 +0.02
4.07 +:0.66
DualPrompt
40
76.63 +5005
85.08 12.12
8.41 +0040
61.51 #1.05
67.11 00.08
5.02 #:052
iBOT-IK
S- S-Prompt++
39
77.53 1:056
85.66 10.16
8.07 1:0.9
60.82 10.68
66.03 10.91
4.16 1:0.44
30
79.11 #1.02
86.21 +0.49
7.69 11.57
66.56 #0.68
73.14 +0.57
7.22 +0038
PrDmeP (Ours)
93.48 1:0.11
95.02 10.01
1.00 1:024
71.33 1:021
73.62 10.13
2.79 1:0.26
L2P
70.65 +0.57
79.02
=0.01
9.46 1.66
$7.40 20.23
62.56 00.20
3.58 +:0.88
DualPrompt 40
74.90 74.90::0.21 1:021
83.98 10.1
10.26 00.62
58.57 10.45
64.89 10.15
5.80 1:0.1
DINO-IK
S-Prompt++
39
74.97 +0.46
83.82 +0.39
7.78 +0066
57.64 10.16
63,79 #0.05
5.08 +-031
PODA-Prompl
1:.64
84.81 10.30
8.10 10.01
63.15 20.39
20.25
6.86 1:0.11
HiDe-Prompt (Ours)
92.51 1:0111
94.25 10.01
0.99 10.21
68.11 10.18
71.70 1:0001
3.11 1:011
L2P 41
74.85 +:.28
83.14 10.03
6.51 10095
$1.64 10.19
58.87
20.24
2.37 1:0.5
DualPrompt
40
77.77 +0.68
85.31 00.07
+1.08
52.57 +0.82
60.65 10.16
2.73 +:0.49
McCo-IK
S-Prompt++
3
76.30 +:054
83.88 12.12
14.67 00.64
53.15 #1.10
60.03 +0.95
4.11 +1
CODA Prompt
30
76.83
1:034
84.97 10.23
12.60
00.02
55.75 :0.26
65.49 10.36
10.46 20004
HiDe-Prompt (Ours)
91.57 +-0.20
93.70 +0.01
+0.18
63.77 +0.49
#0.01
3.57 +0096
Benchmark: We consider multiple CIL benchmarks that are widely used for prompt-based continual
learning 41 40 30). Specifically, Split CIFAR-100 [14] includes 100-class small-scale images, randomly split into 10 incremental tasks of disjoint classes. Split ImageNet-R [4] includes 200-class large-scale images that are hard examples of ImageNet (29] newly collected examples of different styles, randomly split into 10 incremental tasks of disjoint classes. 5-Datasets [6] includes CIFAR-10 10 [14], MNIST (15), Fashion-MNIST [42]. SVHN 24 and notMNIST (1] datasets, each treated as an incremental task to evaluate the impact of large inter-task differences. Split CUB-200 32 includes 200-class fine-grained images of birds, randomly split into 10 incremental tasks of disjoint classes.
Baseline: We compare four representative nomppt-bsse approaches as discussed in Sec. 3.1 such as L2P [41, DualPrompt 44). S-Prompt++ [39] and CODA-Prompp [30]. To evaluate the perlormance of continual learning, we record the average accuracy of all seen classes after learning each task, presenting the last one the final average accuracy (FAA) and their historical average the cumulative average accuracy (CAA) [37]. We also present the final forgetting measure (FFM) of all tasks (37] We consider variety of pre-training paradigms for ImageNet-21K, and ImageNet- 1K, including Sup-21K, iBOT-21K, IK, DINO-1K K and MoCo-IK, as described in Sec. 3.2
Specifically, we adopt a pre-trained ViT-B/16 backbone and train with an Adam optimizer (3) = 0.9, B3 = 0.999), batch size of 128, and a constant learning rate of 0.005 (except for CODA-Prompt with a cosine- decaying learning rate of 0,001), and grid search for a proper epoch number. The image inputs are resized to 224 224 and normalized to [0, 1J. Please refer to Appendix C for more details.
Implementation: We follow similar implementations as previous work 41
40 30
Overall Performance: Table D presents the main results of all approaches. Consistent with the observations in Sec. 3.2, representative prompt-based approaches achieve outstanding performance under supervised pre-training (i.e., Sup-21K) while perform significantly worse under the more realistic self-supervised pre-training. In particular, the most recent CODA-Prompt (30) outperforms other baselines general (here we take FAA as the primary metric), but sensitive to learning rate (see Fig. 2 a, b and Appendix Table 6). In contrast, HiDe- DiDe-Prompt achieves generally the
8

---

highest FAA, CAA and the lowest FFM, thus greatly superior to all competitors. This advantage more pronounced under sulf-supe pre-training, e.g.. up 15.01% and 9.61% lead on Split CIFAR-101 and Split ImageNet-R, respectively" 4 When considering large inter-task differences and fine-grained classification (see [), the performance lead of our approach under self-supervised pre-training becomes even more significant, e.g., up to 12.63% and 30.44% on 5-Datasets and Split CUB-200, respectively. We further evaluate the impact of pre-trained knowledge that is disjoint from downstream tasks, following the setup of recent work [1].. Specifically, we perform Split CIFAR- 100 with pre-training on subset of ImageNet which 389 similar classes were removed. The FAAs of S-Prompt++, CODA- Prompt and HiDe-Prompt above results demonstrate the strength of our approach, thanks to the explicit optimization of the hierarchical components that overcomes potential uub-optimality.
69.00%, 65.07% and 88. 38,05% respectively. All
Here we explore the characteristics of HiDe- Prompt in more depth. First, HiDe-Prompt requires much smaller GPU memory com- pared to CODA-Prompy (e.g., 1614IMB vs 25325MB on Split -I0),-100), since we optimize the construction and inference of appropriate prompts separately as hier- archical components rather than jointly in an end-to-end fashion. Compared to other baselines (except L2P that requires much smaller epoch number), the computation cost of our approach is comparable in order of magnitude. Using the same single-card A100 GPU, for example, the training times of L2P. DualPrompt. S-Prompt+++ CODA-Prompt and HiDe-Prompt are 0.55h, 2.00h, 2.01h, 2.08h, and 2.80h on Split CIFAR-100, respectively. As for the impact of statistical modeling, reserving around 5 centroids for each class performs comparably to single Gaussian (see Appendix Table 7). which ensures storage efficiency and can potentially be generalized other task types.
Table 2: Extended benchmarks. Here present FAA (t) for all approaches. The results of 5-Datasets are further detailed in Appendix Table
5-Dutasets
Split CUB-200
Method
Sup-2IK
iBOT-21K
Sup-21K iBOT-21K
1.P 41
81.84
82.25
74.48
44.29
DualPrompt
40
77.91
68.03
82.05
41.31
S-Prompt++ 39
86.06
77.20
82.08
42.73
CODA-Prompt
30
64.18
51.65
74.34
47.79
HiDe-Prompp (Ours)
93.83
94.88
86.56
78,23
Table 3: Ablation study of hierarchical components. Here we present FAA (f) for all baselines. "WTP" refers to the use of prompt ensemble within the naive architecture of task-specific prompts.
Split CIFAR 100
Split ImageNNet R
Baseline
Sup-21K
iBOT-21K iBOT-IK DINO-IK McCo-IK
Sup-2IK
iBOT-21K iBOT-IK DINO-IK MoCo-IK
Nuive Architecture
85.11
73.05
72.20
73.74
75.67
60.22
48.00
53.68
54,33
48.77
WTP
87.86
78.86
75.93
75.15
77.15
71.57
55.16
60.86
57.61
53.21
WTP+TII
88.05
80.77
78.90
76.27
77.78
73.76
55.19
61.22
58.41
53.08
WTP+TAP
89.85
84.23
86.04
84.76
85.17
72.57
60.01
67.13
64.26
58.36
WTP+TII-TAP
92.50
90.21
90,52
88.93
89.28
74.89
70.44
70.66
66.78
63.59
WTP+TII-TAP w/ CR
92.61
93.02
93.48
92.51
91.57
75.06
70.83
71.33
68.11
63.77
Table 4: Effect of CR on WTP and the full HiDe-Prompt. Here we present FAA (f) for all baselines.
Split CIFAR-100
Split ImageNet-R
PTM
Baseline
A
0.001
0.01
0.1
0.001
0.01
WTP w/ CR
75.13
72.75
72.53
71.53
61.47
61.17
61.07
59.52
iBOT-IK
WTP+TIITTAP w/ CR
90.15
91.18
91.60
93.35
70.97
71.25
71.57
71.71
WTP w/ CR
74,99
73.23
72.74
70.87
57.71
58.23
57.35
56.37
DINO-IK KK
WTP+TII+TAP w/ CR
88.97
90.06
90.54
92.61
66.88
67.42
68.13
68.21
Ablation Study:
We perform an extensive ablation study in Table validate the effectiveness of hierarchical components in HiDe-Prompt. We first construct naive architecture of task-specific prompts as the baseline and then incorporate progressively the individual designs of Sec. 4.2 In general, the optimization of each component, such as within-task prediction (WTP), task-identity inference (TIl) and task-adeetive prediction (TAP), delivers clear benefits and all contribute to the strong performance of Prommpt Interestingly, the improvement TII only becomes apparent
'A contemporaneous work 200 also reported FAA under DINO- IK on Split CIFAR-100, Split ImageNet-R R and 5- Datasets (83.59%, 61.22% and 89. 10%), which lags far behind ours (92.51%, 68.11% and 93.50%)

---

with WTP+TAP rather than with WTP only, suggesting that these hierarchical components are highly synergistic instead of operating in isolation. The proposed contrastive regularization (CR) helps the instructed representations of individual tasks to be compatible with each other and avoid inter-task overlap, thus further facilitating the performance of TTP+ Moreover, we observe that the improvements of WTP, THI, TAP and CR are generally more significant under self pre-training, due to effectively addressing the obscured sub-optimality.
Now we further an- alyze the contributions of the three com- ponents for continual learning. First, we evaluate the average performance of learn- ing each new task in Fig. + where WTP clearly outperforms the naive architecture, indicating that the knowledge of resolving each task better incorporated into the task-specific prompts. Second, we present the accuracy of predicting task identity from uninstructed representations in Fig. b, which has been substantially improved (up to 67.74%) through optimizing TII explicitly. Third, we evaluate the effects of CR on only WTP and the full model of HiDe- PrDmmt (i.c.. WTP+ in Table 4
Detailed Analysis:
Sup
1BOT21K
BOT-1K
DIND-1K
MoCo 1K
98-
(Ouns)
(Ouns)
New Taak Ausuracy (Ng
Task Anuronc
60-
62-
60-
60-
40-
688
20-
86
9:41 CIFAR-100
Spill Imaghaahe
Sy6i CIFAR-100
Spli IaagehhoRR
Figure 4: Detailed Analysis of WTP and TII.
Increasing the strength of
CR decreases the performance of WTP since the task-specific prompt includes more knowledge about other tasks, but improves the performance of HiDe-Prompt since the inter-task representations become more compatible. These results suggest potential trade-off' between the knowledge for WTP and TAP, which can be modulated explicitly by CR.
6 Discussion and Conclusion
In this work, we analyze extensively the advanced prompt-based continual learning from both empiri- cal and theoretical perspectives. An important finding is that, the continual learning objective (which can be decomposed into WTP, TII and TAP the context of pre- training) is not adequately achieved, and this sub-optimality is clearly exposed under the more realistic self sufervueee pre- training. By leveraging statistics of uninstructed and instructed representations, we present a strong approach to optimize explicitly the hierarchical components, which achieves superior performance across various pre-training paradigms. In particular, our theoretical analysis and the proposed approach can serve as general framework for implementing parameter-efficient fine-tuning techniques (c.g., prompt, adapter, LoRA, FiLM, etc.) in continual learning which differ only in the form of task-specific parameters. Interestingly, our proposal is consistent with recent advances in neuroscience [17 [6]. where the activation of non-memory cells and memory cells (as well as their specific populations) is internally switched. Based on these results, expect subsequent work to further explore the architecture and optimization of continual learning with an effective use of pre-trained knowledge.
This work remains some potential limitations. First, we assume adequate pre-training to provide mean- ingful representations, which may not be available some applications. Second, the prompt-based strategy is mainly applicable to the transformer backbone rather than other backbone architectures. Third, the transformer backbone is frozen and adapted to downstream tasks via prompt parameters, which prevents the pre-trained knowledge from being enriched and updated. As a fundamental research in machine learning, the potential negative societal impact is not obvious at this stage.
Acknowledgements
This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106302), NSFC Projects (Nos. 62061136001 92248303, 62106123, 61972224). BNRist (BNR2022RC01006) Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. L.W. is also supported by Shuimu Tsinghua Scholar, and J.Z. is also supported by the XPlorer Prize.
In fact, our LoRA version is also competitive, e.g., the FAA is 87.53% Split CIFAR-IÓV under iBOT-21K.
10

---

References
[1] Yaroslav Bulatov. Notmnist dataset. 2011.
[2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[3] Haoran Chen, Zuxuan Wu, Xintong Han, Menglin Jia, and Yu-Gang Jiang. Promptfusion: Decoupling stability and plasticity for continual learning. arXiv preprint arXiv:2303.07 072223, 2023.
[4] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR. 2020.
[5] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[6] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In European Conference Computer Vision, pages 386-402. Springer, 2020.
[7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[8] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.
[9] Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? In International Conference Learning Representations, 2021.
[10] Zixuan Ke and Bing Liu. Continual learning of natural language processing tasks: A survey. arXiv preprint 22X.:2211..170000 2022.
[11] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. International Conference on Learning Representations, 2022.
[12] Gyuhak Kim, Bing Liu, and Zixuan Ke. A multi-head model for continual learning via out- of-distribution replay. In Conference on Lifelong Learning Agents, pages 548-563. PMLR, 2022.
[13] Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. A theoretical study on solving continual learning. Advances in Neural Information Processing Systems, 35:5065-5079, 2022.
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[15] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 23244, 1998.
[16] Bo Lei, Bilin Kang. Wantong Lin, Haichao Chen, Yuejun Hao, Jian Ma, Songhai Shi, and Yi Zhong. Adult newborn granule cells confer emotional state-dependent adaptability in memory retrieval. Science Advances, 8(45):cabn2136, 2022.
[17] Bo Lei, Li Lv, Shiqiang Hu, Yikai Tang, and Yi Zhong. Social experiences switch states of memory engrams through regulating hippocampal racl activity. Proceedings of the National Academy of Sciences, 119(15)cc211684411 2022.

---

[18] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[19] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint urXiv:2201.00// 2021.
[20] Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning. arXiv preprint MxXi:23003.09447, 2023.
[21] Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. In International Conference Learning Representations, 2021.
[22] James L McClelland, Bruce L McNaughton, and Randall C O'Reellly Why there are comple- mentary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419, 1995.
[23] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:21 12.0955,, 2021.
[24] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[25] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. 2018.
[26] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. arXiv preprint arXiv:2303. 13199, 2023.
[27] Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
[28] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[29] Tal Ridník, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.
2021.
[30] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023.
[31] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 190400994, 2019.
[32] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[33] Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 5383-5392, 2021.
[34] Liyun Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems, 34:22379-2233 2021.
12

---

[35] Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating cecro-insppied adaptability for continual learning in artificial intelligence. arXiv preprini arXiv:2308. 14991, 2023.
[36] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Cosel: Cooperation of small continual learners is stronger than a big one. European Conference on Computer Vision, pages 254-271. Springer, 2022.
[37] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint trxiiv2302.00487 2023.
[38] Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Langing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations, 2021.
[39] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
[40] Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal frree continual learning. In European Conference Computer Vision, pages 631-648.
Springer, 2022.
[41] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[42] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint orXiv:170.007720 2017.
[43] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier aligmment for continual learning on a pre-trained model. arXiv preprint wrXiv:2303.05118 2023.
[44] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Repre- sentations, 2021.
13

---

Algorithm Training Algorithm of HiDe-Prompt
Input: Pre-trained transformer backbone fo, training sets 11,.., Dr, number of tasks T, number of epochs E, hyperparameters a, 7 and Output: Parameters Pı, Pr. and Initialize e1, w and 2: for trrt- = 1,.,.... do
3: for E 3/2 do
4: Obtain Go from fo and De
Uninstructed Representations
5: if then
6:
Initialize e 6c-1
7:
Construct P: E/=1 e + (1 a)e
8:
else
9:
Construct p: = Et for epoch = I,... E do
10:
11:
Optimize P, and vi with LwTP in Eq. (00
Within-Task Prediction
12:
Optimize w with Crn in Eq. (T)
Task-Identity Inference
13:
Optimize with LTAP in Eq. 12
> Task-Adaptive Prediction
for V% do
15:
Obtain Go from fe. Pt and D. return (P1 ... Pr. W, 16)
Instructed Representations
16:
A Theoretical Foundation
A.1 Class-Incrementss Learning (CIL)
Proof of Theorem [
Proof. For inasemenrrllltll learning (CIL) with pre-training, assume Ez (x) < 8, Eg [x)] S e, and Ege (x)) I 17. Let y Vt,5 be the ground truth of an æ, where i (1,..., t) and j E {1...., [il]] denote the task identity and within-tas index, respectively.
As we defined,
Hwrp(x) H(11, (P(x e Mi x (, --log log P( E 2g E N3 D,0).
(13)
Hru(w) = HH1,, (p( E ()D,))} =- log (( E X 0).
(14)
HπλP(x) -H(la, (P(x x x ())))
- lllog P(x E X' D,0)
(15)
log P(x E x D)
Then, have
H(1,,, (P(x E Xr,
log P(w E X;
Po Xt, x E X; P( e x\D,0))
(16)
- log P(# E X3 |a E X, D,0) 0) - log P( E DD
Hwrr(() + Hru(z).
Taking expectations on Eq. 15) we have
C E. (HTP((()) S
(17)
Taking expectations on both sides of Eq. 16 we have
L2 E, [H j. {P(x E
(x + Em (T)]
(18)
tt..
14

---

Since our objective of CIL with pre-training is max E X1,ID. 0), P(æ 22),, 0)), then we have the loss error
(a [H(,, E &yy E. max L1}
(19)
max + +, n,
This finishes the proof.
Proof of Theorem
Proof. For CIL with pre-training, its loss error C S. Assume Xg N2 According the
proof of Theorem I
we have
HwTP(x) - log P(æ E X1,5 2,0)
P(æ: E 1D,
log
P(x E 2
(20)
- log E (2;(D,0)
-H(15, (P(x X,)lD,
SE.
Likewise, we have
Hrn(x) = - log P(x E x2D
P( (2,
log
P(x E X; x N3 D,0)
(21)
S loog P(æ {b,(D,0)
= =H(1,, (P(x Xrg {(5))
L2 S.
In addition, we have formulated the final goal of CIL as multi-objecctive optimization problem, i... max Pax[P(x E K, (), P(z E (2, 0). Then, each objective must guarantee the loss error less than E, i.e.
HTAP(#) log P(w E DD, 0)
(22)
=L E.
This finishes the proof.
A.2 Domain-Incrementte Learning (DIL)
For Commann-ncremmntal learning (DIL), Let X; = U₁ Rr and V1 = (1t.), where j E {1, ..., Dt denotes the j-th class in task Now assume we have a ground event denoted as = (D D, and pre-traiine treitrained model fe. For any sample x E U Xx. a general goal of the DIL problem is to learn P(x X.,, / 0), where X.,, represents the j-th class domain any task. Of note, 3/2 = Ve.
Vt * t' for DIL. This be decomposed into two probabilities, including task-identity inference (TIf) and within-task prediction (WTP), denoted as P( E X)D, and P(æ E X1,j X, D) respectively, Based on Bayes' theorem, we have
P(x E (., Σ P(a Xig P,0)) & D,0),
(23)
where (+,j) represents the j-th class in each domain.
Then we have the following theorems in terms of the sufficient and necessary conditions for improving DIL with pre-training.
Theorem 3 For demain-incrementee learning with pre-training, if En (m(H)((((+)) S 8, Em [x)) S e, and E,p (TTr((:) S T,, we have the loss error L E [0, max (6 e + +llog mH}. regardless whether WTP, TH and TAP are trained together or separately.
Proof of Theorem 3
15

---

Proof. Let j E [1.. and E 3t be the ground truth of an a w.r.t the within-tase index and class label, and y = V for any E {l....,t, Eq. (23 suggests that we can improve either the WTP performance P( E & E D, the TII performance P(æ X. (2),)) 0), or both, then the DIL performance P(æ E x2)D.,0 would be improved. However, such improvement limited since it upper-bounded by WTP or TII. To further improve the DIL performance, we propose hierarchical decomposition of the objective. That is, besides the improvement of P(a: E X2, 0), we also need to directly improve the performance of task-adaptive prediction (TAP), denoted as P(æ E X in all observed domains, and X U X. Then the final goal of DIL is formulated
[2, 2 0). where E {1.... [.]], A represents the domain of class multi- objective optimization problem, i.e., max[P(x E i 0), P(x E X3ID, O)]- Notice that the TII probability is a categorical distribution over all observed domains {1 tt. while the TAP probability is over all observed classes Ui Vh.
As similarly defined in CIL, here
Hwrr(ar) =
HH(13 (P( E Mij E E. D,0))) log P( X. x; D,0).
(24)
Hru(x) =
H(i,, (P(x 22 0)):) =-% Poz E X2(2,),
(25)
HTAP(æ)) = -H(1e, { P(xx E (D(10))
- log P(x E x² D) =- log P(æ E x2 0),
(26)
where 7 (y represents the possibility of x belonging each observed domain, %/4 E [0,
and E, 7 =
Then, for any simplex y, we have
(11,,1((e E (D,1D,))}))
log P( E (D,)))
log( Σ P( N. X.DD,))P(e E ( x.().))
Σ log( P( E X1 D. &DD,))
(27)
%
Σ ' log P( E X,31æ E X, D, Σ n log P(x E X.1,,0) + Σ 2a log(1:)
Σ HnHWTP + HTU + H(?).
Taking expectations on Eq. 26 we have
L, = Es (T)P S
(28)
Taking expectations both sides of Eq. 27 have
L E. [H (a E X.)(D.
- Σ [HW[(HwPP(z) E. [f)](w)] + H())
(29)
S 5 + logt.
Since our objective of DIL with pre-training is max max[[((x E X..lD, P(a E XM 0)), then we have the loss error
L = max [H((1, {P(æ E E,, = max L2, Li}
(30)
max(6 + log f, n).
This finishes the proof.
16

---

Theorem 4 For comain-inncrmeente learning with pre-training, if the loss ervor L S E, then there always exist (1) a WTP, s.f. HwTP S (2) a TII, s.s. HTU S; and (3) TAP. s.t. HTAP S &.
Proof of Theorem 4 For DIL with pre-training, its loss error L = max L1. L2] < Assume E X,, S X According to the proof of Theorem 3 if we define P( E X D,0) P(æ E
X, [D, 0). we have
(wTP(x)) log P(æ E X2gl 2,)
P( E 2, .)
log
P(æ E X; (D
log (( E X.j(D,0)
(31)
=-log P( E x.,i(D,0)
=H(1,, (a E 2n (D, ())))))
L2 S.
Likewise, if we define " and 7/2 0 for + i, we have
Hru(æ) = Σ ns log P(ar E 2))
= log P(æ E 24 (D,
P(w E X (D,0)
log
P(æ E X x D,0)
(32)
S -oo(æ E N.j D20)
= - -log(x X.,)(D,))
H(13, (P(z [R,]D,,))
L2 E.
In addition, we have formulated the final goal of DIL multi- optimization problem, i.e., max(P(@ (2, 0), P( E ()] Then, each objective must guarantee the loss error less than g, i.e.
HTAP(#) log P(w E x2 0)
(33)
= =1 S
This finishes the proof.
A.3 Insk--ncremennal Learning (TIL)
For lask-incremental learning (TIL), let X, = U, Ke, and D; = -Vig), where (1, (1...., |} indicates the j-th class task Now assume we have a ground event denoted as = (D....., D.} and a pre-trained model fg. For any sample x: E U Xx. general goal of the THL problem is to learn P( E X 2 |æ E X2, D, A), where i (1, 1} and j (1,... DiI]- This can be equivalent to within-task prediction (WTP). Different from CIL, the task identity is provided in TIL. Unlike DIL. 3% n3 = 0, Vt & f'. Then we have the following theorems in terms of the sufficient and necessary conditions for improving TIL with pre-training training.
Theorem For task-incremental learning with pre-training, Eg (x))()))) = O, and task-adaptive prediction (TAP) is degraded into within-task prediction (WTP). E. [Wwre (2)) S 6, we have the
loss error L E [0, 8).
Proof of Theorem 5
Proof. For task-incremental learning (TIL) with pre-training, assume Ez, [HWTTP (x)) S 8. Given an with the task identity E {1,..., (), let j E {1, [) be the ground truth of x w.r.t. the within-task index, and y = -ii be the ground truth label ac.
As similarly defined in CII,, here
Hwrp(x) = H((1g. (P(æ E Mig x E (EX,
(34)
lol P(x Ma D,0).
17

---

Hru(æ)) H(1;, (P(x E (((D,0),)
=- log P( E X/DD,) 0)
(35)
---log 1og1=00
HTΛP(œ)
H(11. (P(x E 1e X; (,)))
log P(æ E x E D.D)
(36)
P(x E X D,0)
HwTP(x).
Then, we have
H(11.3. (P(w E X.,(D,0)).,)
P(x E (2,
log(P(x X2 x E X; P(x E X)),
(37)
-- log P(æ E X la E X, D, 0) - log P( E (D,
HwTr(=) + Hru(æ)
Hw)r
Taking expectations on both sides of Eq. 37
we have
L = Bm [7( (P(x E x.,1D,0)).))
F. (x))
(38)
S 6.
Since our objective TII with pre-training P(æ E Kiglar N; D, 0), then we have the loss error L. < 8. This finishes the proof.
Theorem For laskiincrementll learning with pre-training, if the loss error L S &, then there always exists a WTP, s.t. HWTP S.
Proof of Theorem 6
For TIL with pre-training, its loss error L S &. Assume x E X33 X. According to the proof of Theorem 5 we have
HwTr(æ)
log P(w E Niglæ E 2,0)
P( &,
log
P(x E (D
(39)
<- log (w E (D,(
H(115, (P(x E xDglD,)))a.
s g.
This finishes the proof.
B Impact of Pre- Pre-Training on Continual Learning
In this work, we focus on continual learning with pre-training, especially prompt-basee continual learning that receives significant attention in this direction. Our theoretical contribution lies in the context of pre-training. where we demonstrate the sufficient and necessary conditions to achieve good continual learning performance. This is clearly different from those previous theorems on continual learning from scratch [3] which analyzed below.
First, the condition is different. We formulate the hierarchical components as é-conditional probabili- ties, i.e., P(æ: X2,læ E X; D, 0), P(æ X, (D,0) and P( E x 0) for WTP, THI and TAP, respectively, where 0 captures the pre-trained knowledge in initialization. When training from scratch, since the randomly-initialzzed parameter set O, carries no information and greatly different from the optimal solution, it needs be substantially changed in continual learning and should not be took
18

---

into account in objectives. In contrast, the pre- pre-trained parameter set & carries beneficial knowledge for downstream tasks. If the pre-training is adequately strong, 9 is already close the optimal solution and only requires appropriate fine-tuning. Therefore, 0 needs to be stabilized (usually frozen) in continual learning and should be considered in objectives.
Then, due to the incorporation of 0, the sufficient and necessary conditions to achieve good continual learning performance become different. In addition to WTP and TII, TAP is especially required in the problem of continual learning with pre-training. Without considering the impact of pre- training, i.e., training from scratch, TAP equivalent TII * WTP, P(x E \ 0) P(x & (D, () P( E K, E 2, D) corresponding to 8 + c = in our Theorem TAP is not required when training from scratch, consistent with the theorems in [3 On the other hand, with considering the impact of pre-training. TAP and TII WTP are formulated as two different prediction problems where P(æ= E X [D, 0) * P(x: E Xj (1DD,)PPC P(x: E 22,j l E X; D2, 0). Therefore, the final performance of having TAP in continual learning can outperform that without TAP, i.e., max E 24, 0), P( E Xx)DD,) 0) > P(x E (D,
Therefore,
In the case of prompt-based continual learning, without considering the impact of pre-training, TAP TII * WTP, due to classifying the same input in the same semantic space, i.e., (f(=))
(fp(h.(f(())lv)) 2j-1 Eyy,, (xp() Ei (x1 E,ey, (xp()1 With considering the
oxp(h,(f(am))))
impact of pre-training, TAP TII WTP, because THI WTP is to calculate (((x)_)
exp(ia (f(xp)))) Eis exp(A,
Eicy, (/ )ib)) while TAP is to calculate
exp(h_.(((::p)))1 (IG))
In other words,
Σj=r Eyey,, (f
the backbone parameters are frozen stabilize the pre-trained knowledge, with additional prompts P; to fine-tune the semantic space (i.e.. representations). TAP is essentially classifying the instructed representations, i.e., E His in Eq. (2). while TII * WTP is performed on uninstructed represen- tations, i.e., E Hi,e in Eq- (1). Targeting on different semantic spaces, TAP and TII WTP are clearly different, i.e., TAP != -TI WTP. Strong empirical results also demonstrate their respective contributions to continual learning performance (see ablation study in Table 3
C Implementation Details
In this section, we describe the implementation details of all experiments.
Prompt-Based Approaches: We follow the same implementations of the prompt architectures for all baselines as their original papers [41 40 30 (except S-Prompt++), which have been shown to
yield strong performance. Specifically. L2P is implemented with M = 30 as the total number of prompts (M 10 [41 and M 30 40 have similar results). Lp = 5 as the prompt length, and
N =5 for the Top-N N keys. DualPrompt [40] is implemented with Lg 5 as the prompt length of task-sharing prompts g inserted into layers 1-2 and Le 20 as the prompt length of task-specific prompts e inserted into layers 3-5. S-Prompt++ (39] is implemented similarly to DualPrompt but replaces all task-sharing prompts with task task-specific prompts, i.e., the task-specific prompts are inserted into layers 1-5 with prompt length Le = 20. CODA-Prompl (30) is implemented with M = 100 as the total number of prompts and Lp = 8 as the prompt length, inserted into the same layers 1-5 as DualPrompt and S-Prompt++. HiDe-Prompt adopts similar architecture as S-Prompt+++ but replaces the task-specific keys with an auxiliary output layer hiw to predict the task identity and further preserves statistics of uninstructed and instructed representations. The hyperparameters for HiDe-Prompt are set to a 0.1, - = 0.8, and À = 0.1.
Following the implementations of previous work [4] 40), we employ a pre-trained ViT-B/16 backbone, an Adam optimizer (3) = 0.9, 32 = 0.999) and a batch size of 128. The learning rate is set 0.001 with cosine decay for CODA-Prrmpp (empirically validated in Table compared to 0.005 for other approaches. We then grid search for an appropriate number of epochs (E) under different pre-training paradigms, and observe that the best choice is generally consistent on each benchmark (i.e., E is relatively insensitive to the pre- gre-trannig paradigms). For Split CIFAR-100 10 for L2P and E = 20 for other approaches. For Split ImageNet-R with E E (25, 50, 75, 150]. we set E 50 for all approaches. For 5-Datasets with E. (5, 10, 20, 40). we set = 40 for supervised pre-training and F. = 20 for self-supervised pre-training, For Split CUB-200, we set E 50 for all approaches.
Training Regime:
with E E (5, 10, 20, 40), we set
19

---

We focus on three evaluation metrics for continual learning, including the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM). Specifically, we define the accuracy on the i-th task after learning the 1-th task as A1,,, and define the average accuracy of all seen tasks AA, - A... After learning all T tasks, we report
Evaluation Metric:
FAA- = AAr, CAA +
E AA,, and FFM T i= [ (Ai, A..r).
FAA is the primary metric to evaluate the final performance of continual learning, CAA further reflects the historical performance, and FFM serves measure of catastrophic forgetting.
Compute: We run all experiments of Split CIFAR- 100 on eight Tesla P100-SXM2 GPUs, Split ImageNet-R on four NVIDIA A100 GPUs, 5-Datasets on both, and Split CUB-200 on eight NVIDIA GeForce RTX 3090 GPUs.
D Extended Results
In this section, we provide some extended results for the main text.
Reproduction of Baselines: All results of L2P, DualPrompt and CODA-Prompt are reproduced from their official implementations, while Prrompt++ implemented with the code of DualPrompt. We compare the reported and reproduced results in Table 5 In particular, we use the same Sup-21K
checkpoint as the original papers of L2P 4T and DualPrompt 44), and their reproduced results are comparable to the reported ones. In contrast, the original paper of CODA-Prompt (33) used
different supervised checkpoint, which is first pre-trained on ImageNet-211 in a self-supervised
fashion and then fine-tuned on ImageNet-11 IK in supervised fashion. Using the same supervised checkpoint as [30], we have justified that the results of CODA-Prompt on Split CIFAR-I00 can be faithfully reproduced. As for Split ImageNet-R, we obtain the official code directly from the authors of CODA-Prompt, but it cannot reproduce exactly the reported results on Split ImageNet-R due to some clean-up issues. Consequently, the reproduced results of CODA-Prompp on Split ImageNet-R slightly underperform the reported ones under the same setting 330. Besides, we observe that [30] used different data split than originally proposed [40], resulting in increase of around 3% in FAA. Together with the differences in supervised checkpoints, the reproduced results of CODA-Prompt in Table and Table differ by around 4% in FAA.
Table 5: Comparison of reported and reproduced results. "The forgetting metric for reported results 30
is implemented differently from ours.
Split CIFAR-100
Split ImageNet-R
Baseline
(f)
(L)
(?)
(L)
L2P (Reported)
41
83.86
7.35
61.57
9.73
L2P (Reproduced)
83.06
6,58
63.65
7,51
DualPrompt (Reported)
40
86.51
5.16
68.13
4.68
DualPrompt (Reproduced)
86.60
4.45
68.79
4.49
CODA-Prompt (Reported)
33
85.61
1.82
76.66
1.60
CODA-Prompp Prompt (Reproduced)
86.46
6.67
74.05
6.48
Learning Rate of CODA-Promee
As discussed in the original paper of CODA-Prompt [30], its
performance depends heavily on the reduced learning rates, especially on Split ImageNet-R. Here we extensively evaluate the effect of learning rate on the performance of CODA- in order to justify the strength of our reproduced results under different pre-training paradigms. It can be clearly seen that using smaller learning rate with cosine decay (i.c., LR=0.001 Cosine) is generally a good choice for CODA -rompp, which is consistent with its original paper and therefore employed to reproduce its performance in this work.
(see Table 6).
Statistical Modeling: Here we evaluate the effect of statistical modeling. As analyzed Sec. 4.2 preserving dedicated mean and covariance for each class of representations can faithfully recover their distributions and thus achieves strong continual learning performance (see Table 7 Under Sup- 21K that is adequately strong for downstream tasks, the performance remains essentially consistent when reducing the covariance to variance (e.g., the FA are 91.74% and 73.97% on Split CIFAR-100 and Split ImageNet-R, respectively). As a more generalized form of approximated distributions.
20

---

Table 6: Effect of learning rate the performance of CODA-Prompt. Here present FAA (1) for
all baselines. "The choice in its original paper [30].
Split CIFAR-100
Split ImageNer-R
Learning Rate (LR)
Sup-21K iBOT-2IK iBOT-IK DINO-IK MoCo-IK
Sup-2IK IBOT-21K iBOT-IK DINO-IK MoCo-IK
LR=0,005 Constant
85,92
76.46
72.39
71.43
76.83
62.74
53,51
57.83
54.73
52.27
LR=0.001 Constant
86.78
80.91
79.31
76.86
76.09
67.73
59.80
64.75
61.33
55.75
LR=0.001 Cosine"
86,94
80.83
79.11
77.50
74.55
70.03
61.22
66.56
63.15
55.15
Table 7: Effect of statistical modeling. We present the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM) with + standard deviation under different pre-trained models (PTM), over three runs of different random seeds and task splits.
Split CIFAR 100
Split ImageNet-R
PTM
Method
FAA (*)
CAA (f)
FFM (L)
FAA (t)
CAA (f)
FFM (L)
Covariance
92.61 +0.28
94.03 +0.01
#0.10
75.06 +0012
76.60 =0.01
2.17 #0.19
Sup-21K
Multi- Centroid
92.92 10.12
94.60 +0.06
0.86 10.12
73.55 +0.21
75.93 40.15
0.95 +0.02
Covariance
93.02 10.15
94.56 10.05
1.33 20.24
70.83 10.17
73.23 10.08
2.46 #0.21
IBOT-21K
Multi-eentoodd
92.69 10.10
94.69 10.13
1.57 20.24
70.63 40.07
72.94 10.06
1.31 10.15
Covariance
93.48 +0.11
95.012 +0.01
1.00 0.224
71.33 +0.21
73.62 40.13
2.79 +0.26
iBOT-1K
MultiCeentoid
91.78 10.08
94.10 10.10
1,45 +0.07
71.33 40.29
74.38 40.24
1.73 +0.37
Covariance
92.51 +:011
94.25 +0.01
0,99 +0.21
68.11 +0.18
71.70 +0001
3.11 +0.17
DINO-IK
Multi- Cent-Ce
90.06 +:.14
92.92 +0.19
2.13 #0.10
69.34 +0.16
72.35 +0.11
1.60 +0.16
Covariance
91.57 4:020
93.70 10.01
1.19 10:18
63.77
40.49
68.26 +0.01
3,57 +0.96
McCo-IK
Multi-Centroid
90.70 #0.02
93.23 10.10
1.56 10.12
63.05 10.15
66.90 #0.13
1.78 10.36
reserving fewer than 10 centroids (around 5 on average @ for each class achieves comparably strong performance in all cases (see T7)le which ensures both efficiency and generality.
we evaluate the continual learning performance 5-Datasets for large inter-task differences. We keep the implementation of prompt architecture for all approaches the same in Split 1IFAR-100 and Split ImageNet-R. Then we perform an extensive grid search for learning rate, number of epochs and other applicable hyperparameters with their official or commonly- used codes. Under Sup-21K, we can essentially reproduce the results of L2P [41]. However, the reproduced results of DualPrompt [40] is lower than the reported one. We find that this issue is also discovered by other users and remains open in github. As the original paper of DualPrompt [40] has not described precisely the implementation for 5-Datasets, we speculate that this issue is possibly due to the use of different prompt architectures. A supporting evidence is that the reported results of DualPrompt 40) are similar to those of S-Prompt++, which is equivalent to replacing all task-sharing prompts in DualPrompt with task-specific prompts. Besides, the most recent CODA-Prompp (30) also performs poorly on this challenging benchmark. To ensure the strength of reproduced results, we have extensively searched the learning rate (0.001, 0.0005), the number of epochs {5, 20, 40) and the hyperparameter of orthogonality regularization in {0, 0.1, 0.01}, and presented the best performance. The inferiority of CODA-Prompt possibly due to the excessive attentions to the rrviously-lcaanen prompts, since the task distributions are clearly different and the old knowledge might interfere with the new one. To support this claim, we analyze the attentions of CODA PODA-Prompt and observe a strong preference for previously-learnes prompts (around 0.6 ~ 0.8). Compared to all competitors, HiDe-Prompt achieves the strongest performance under different pre-training paradigms, consistent with the results on Split CIFAR- 100 and Split ImageNet-R in Table | In particular, the FFM of HiDe-Prompt is almost zero, indicating its great success in overcoming catastrophic forgetting.
5-Datasets: In Table g
Visualization of Representations: We visualize the uninstructed and instructed representations with t-SNE, as shown in Fig. (5) for Sup-21K and Fig. for iBOT-21K. Based on these results, we have the following analysis: (I) The uninstructed representations have shown single-peake patterns in general, thanks the use of adequate pre- pre-trannig. This property allows them to be approximated with Gaussian distributions for preservation and recovery, and allows for correct prediction of task identity from them. (2) The instructed representations tend to be more compact and distinguishable, validating
'We use KNN to select multiple centroids (set to maximum of 10) for each class of representations, where the average number of obtained centroids is usually around 5.
21

---

Table 8: Overall performance of continual learning on 5-Datasets.
FAA
FFM
Baseline
S1K
1BOT-21K
iBOT IK DINO-IK MoCo-IK
Sup-21K iBOT-21K iBOT IK DINO-IK IK MoCo-IK
L2P A1
81.84
82.25
80.02
76.26
66.89
4.78
8.23
9.46
8.50
24.22
DualPrompt
40
77.91
68.03
68.92
64.66
59.71
13.11
20.30
24.47
27.24
35.89
S-Prompt++
39
86.06
77.20
73.51
69.51
72.91
4.74
15.83
7.47
17.87
15.86
CODA-
20
64,18
51.65
48.14 14
50.86
39.02
17.23
27:53
22.02
26.69
64.33
HiDe: HiDe-Prompl (Ours)
93.83
94.8%
93.89
93.50
93.28
0.44
0.09
0.07
0.04
0.14
Task
Task 2
Task 3
Task 4
Task
Representations
Uninstructed
Instructed Representations
Task
Task
Task
Task
Task
Uninstructed Representations
Representations
Instructed
Figure 5; Visualization of uninstructed and instructed representations with t-SNE: Part Here
present results of Split CIFAR-100 under Sup-21 Each color represent a class.
the effectiveness of prompt-based continual learning. The degree of compactness and differentiation varies across pre-training paradigms, contributing to their performance differences. (3) The differences in compactness and differentiation between uninstructed and instructed representations suggest that the design of coarse-grained classification by task and fine-graines classification by class is reasonable for prompt-based continual learning, as do our theoretical analysis and the proposed approach.
22

---

Task
Task
Task 3
Task 4
Task
Uninstructed Representations
Instructed Representations
Task 6
Task 7
Task 8
Task 9
Task 10
Uninstructed Representations
Representations
Instructed
Figure 6: Visualization of uninstructed and instructed representations with t-SNE: Part II. Here we present the results of Split CIFAR-I00 under iBOT-21K Each color represent a class.
23


Reviews:
[{"review_id": "jLcIzuO5nq", "reviewer": "Reviewer_pdqM", "summary": "The paper suggests a new prompt-based continual learning method by leveraging combinatorial objectives with *within task prediction*, *task-identity inference*, and *task-adaptive prediction*. The paper first summarizes recent prompt-based continual learning techniques and demonstrates their unstable performance according to the pre-trained backbones (particularly, initializing with self-supervised representation degrades their performance significantly). And then, proposes a new regularization technique containing core components, including ensemble prompting, contrastive regularization, and TAP loss.", "questions": "N/A", "limitations": "Please see the weakness.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a comprehensive analysis and theoretical proof of the motivation. And the suggested idea is reasonable, and the methodological design is also clear. In experiments, the proposed method consistently surpasses strong baselines in terms of conventionally used continual learning metrics under multiple pre-trained backbones, and its improvement and stability are a bit impressive.\nFurther, they provide the necessary ablation study and analysis in the main paper and appendix.", "weaknesses": "The effect of the prompt ensemble in the method is not discussed. And the different behavior and impact of un-/instructed representation for continual learning are partially discussed in Figure 5, but I recommend a more comprehensive and detailed discussion/analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n**Q1: The effect of the prompt ensemble in the method is not discussed.**\n\nA1: We would respectfully point out that the effect of the prompt ensemble has been discussed in experiments, referred to as \"WTP\" in Table 2 and Fig. 4a. As shown in line 248-255, the prompt ensemble is proposed to learn each task more effectively with the architecture of task-specific prompts, so as to improve within-task prediction (WTP). In comparison to the \"naive architecture\" that employs only task-specific prompts, our prompt ensemble strategy can largely improve the performance of learning each new task (Fig. 4a) and thus improve the average performance of all tasks (Table 2), consistent with the motivation of our design. We will add more explanations to make it clearer.\n\n**Q2: The different behavior and impact of un-/instructed representation for continual learning are partially discussed in Figure 5, but I recommend a more comprehensive and detailed discussion/analysis.**\n\nA2: Thank you for your valuable suggestion. We have performed an extensive visualization of un-/instructed representations in terms of continual learning methods and pre-training paradigms. Fig. 5 reports the representations of our approach under iBOT-21K pre-training, and the results of Sup-21K are further included in the rebuttal PDF. From these results, we provide a more comprehensive analysis as below:\n\n(1) The uninstructed representations have shown single-peaked patterns in general, thanks to the use of adequate pre-training. This property allows them to be approximated as Gaussian distributions for preservation and recovery, and allows for correct prediction of task identity from them as shown in Fig. 4b.\n\n(2) The instructed representations tend to be more compact and distinguishable, validating the effectiveness of prompt-based continual learning. The degree of compactness and differentiation varies across continual learning methods and pre-training paradigms, contributing to their performance differences in Table 1. \n\n(3) The differences in compactness and differentiation between un-/instructed representations suggest that the design of coarse-grained classification by task and fine-grained classification by class is reasonable for prompt-based continual learning, as do our theoretical analysis and the proposed method.\n\nWe will add the above discussion with more visualization results in the final version."}, {"review_id": "Vzef1YxtwK", "reviewer": "Reviewer_tgkY", "summary": "The authors provide strong empirical analysis on existing \"prompting for continual learning\" papers. They propose a new hierarchical prompting method that includes several components which take advantage of unstructured data representations. The authors not only propose an interesting approach with SOTA performance, but they have very detailed experiments using several pre-training backbones and datasets.", "questions": "a) See weakness 2\nb) what is the computation cost of your methods compared to the baselines? I feel that even with a strong training time cost, I am happy to vote for paper acceptance. Just want to ensure transparency.", "limitations": "Discussed reasonable limitations - there is no obvious potential negative societal impact", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In general, I feel that for this subject area (which I am sure will have many neurips submissions), this paper is overall high quality.\n1) I appreciate another \"prompting for continual learning\" paper that starts with insights and findings into existing methods, laying an intuitive foundation for the proposed approach.\n2) With good intuitive and theoretical backing that seems to reasonable, the method has great performance gains over SOTA. \n3) Nice, detailed analysis and ablation study. I checked carefully to see if maybe there was a \"hidden\" component that does most of the heavy lifting of the method, but the contribution components seem to all be impactful in some way.\n4) I am incredibly thankful to see the experiments conducted on several different backbones. This code and framework will be a fantastic contribution on its own, along with the analysis, before even considering the method. As someone who has experience in this problem setting, I think open-source code for this project will be greatly appreciated.", "weaknesses": "1) Task-id prediction for class-incremental learning on random splits of classes (i.e., where there is no actual task structure), seems a red flag to me that the pre-trained backbone is too strong for the proposed task datasets, and that it is actually taking advantage of the unfair backbone in order to \"circumvent\" class-incremental learning, instead posing it as the much easier task-incremental learning problem. However, this weakness also exists in the competing methods, and the strengths of the paper out-weigh this weakness imo.\n2) Figure 2 c and d is very confusing. Which method is this for? If you are making strong claims about task specificity, should it not be done for all methods?\n3) This is not relevant to your findings, and I am not requesting any experiments, but Coda-prompt might perform better at a lower constant learning rate if the high, constant learning rate is hurting its performance. Learning rate decay is also not a \"trick\", either - it is a valid hyperparameter choice and important for many representation learning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version. We are pleased that our implementation code is appreciated. It will be published after acceptance.\n\n**Q1: Task-id prediction for class-incremental learning on random splits of classes...it is actually taking advantage of the unfair backbone in order to \"circumvent\" class-incremental learning, instead posing it as the much easier task-incremental learning problem.**\n\nA1: We agree that the pre-trained backbones considered in this work as well as most prompt-based continual learning papers are adequately strong for downstream tasks. In fact, this experimental setup is **complementary** to the commonly-used protocol of class-incremental learning (CIL), which learns incremental classes from scratch or relatively weak pre-training (i.e., learning half of the classes in the first stage). As large-scale pre-training has proven to greatly facilitate a variety of downstream tasks, how to exploit it effectively in continual learning is of increasing interest. Our work demonstrates that CIL has distinctive results with relatively strong pre-training (e.g., more tolerance to the errors in task-identity inference), which extends previous explorations of CIL.\n\nOn the other hand, the CIL in our experiments remains clearly **different** from TIL. As shown in the response to Reviewer UkQY's Q3, the performance of TIL is remarkably better than that of CIL. This is because the access to task identity enables TIL to avoid inter-task interference via multi-head output layers, while CIL usually requires a well-adapted single-head output layer to avoid errors from predicting an incorrect task identity. In fact, the relationship between CIL, TIL and pre-training has been shown in our theoretical analysis, i.e., a cross-talk between within-task prediction (WTP), task-identity inference (TII) and task-adaptive prediction (TAP). Removing the significant impact of pre-training would degenerate our framework into regular CIL, i.e., only WTP and TII remain (see line 231-234). As for TIL, the task identity is provided for multi-head evaluation and thus only WTP remains (see Appendix A.3 and Reviewer UkQY's A3).\n\nBesides, we have evaluated our approach on 5-Dataset, a benchmark with **actual task structure** (see line 287-289), and it can still achieve a significant performance lead (see Appendix Table 6 and line 602-624). \n\nWe will add the above discussion in the final version.\n\n**Q2: Figure 2 c and d is very confusing. Which method is this for? If you are making strong claims about task specificity, should it not be done for all methods?**\n\nA2: In Fig. 2c, we analyze the instructed representations of task-specific prompts, which are identical to the prompt architecture of S-Prompt++ (see line 181-182, 164-168). In Fig. 2d, we analyze the ability of uninstructed representations to predict task identity, corresponding to the same method in Fig. 2c. \nFor all the prompt-based methods considered in Fig. 1, since the training set of each task is provided sequentially, the optimizable parameters for each task inevitably acquire task-specific knowledge and the corresponding knowledge needs to be invoked correctly at test time. However, due to the complexity of their prompt architectures (e.g., L2P, DualPrompt and CODA-Prompt all reuse some prompt parameters optimized for previous tasks), task specificity is difficult to demonstrate explicitly in experiments as it is with S-Prompt++. Therefore, we construct such a demo experiment to analyze task specificity in prompt-based continual learning. \nIn fact, the objective of all methods is tantamount to optimizing the probability distribution on the left side of Eq.(5), which can be decomposed into the two probabilities on the right side corresponding to Fig. 2c and 2d, respectively. Therefore, our demo experiment is representative for prompt-based methods from a theoretical perspective. We will add more descriptions and explanations to make it clearer. \n\n**Q3: Coda-prompt might perform better at a lower constant learning rate if the high constant learning rate is hurting its performance. Learning rate decay is also not a \"trick\", either - it is a valid hyperparameter choice and important for many representation learning methods.**\n\nA3: The main results of CODA-Prompt (i.e., Table 1) are produced from **using a lower learning rate with cosine decay** (see line 300), which is the same as its original paper and indeed performs better than a higher constant learning rate (see Fig. 2a, 2b and Appendix Table 5). We have also performed some experiments to validate that using such a lower learning rate with cosine decay is slightly better or comparable to using a lower constant learning rate (see Appendix Table 5). For example, the final average accuracy (FAA) of CODA-Prompt with a learning rate of 0.005 (constant), 0.001 (constant) and 0.001 (cosine decay) on Split CIFAR-100 under Sup-21K pre-training is 85.92%, 86.78%, and 86.94%, respectively. We agree that the learning rate decay is a valid hyperparameter choice for deep learning methods. We will modify our claim appropriately and add more explanations to avoid potential misunderstanding. \n\n**Q4: What is the computation cost of your methods compared to the baselines?**\n\nA4: Using the same single-card A100 GPU, the training times of L2P, DualPrompt, S-Prompt++, CODA-Prompt and our approach are 0.55h, 2.00h, 2.01h, 2.08h, and 2.80h on Split CIFAR-100, respectively. We observe that L2P requires a much smaller epoch number for convergence but performs the worst in general. Compared to other strong baselines, the computation cost of our approach is comparable in order of magnitude. In practice, we include parallel training in our implementation code, which can largely reduce the training time. We will add the above results in the final version."}, {"review_id": "4UplZzriGL", "reviewer": "Reviewer_Qf6e", "summary": "This paper studies application of prompts in pre-trained models for continual learning. Building upon the problem of [1], the authors introduce the Task-Adaptive Prediction (TAP) for the CIL problem using pre-trained networks. They demonstrate that a good TAP, WTP, and TII performances are necessary and sufficient for a good CIL model. \nThe authors emphasize the significance of these factors, especially in the context of continual learning with pre-trained self-supervised model.\n\n[1] Kim et al., Theoretical study on solving continual learning. NeurIPS 2022", "questions": "- I couldn’t fully-understand why it’s necessary to have TAP in the CL problem using pre-trained networks, considering that it wasn’t required when training from scratch [1].\n- I am curious about the performance of the proposed method when the feature extractor is pre-trained using only the classes that are dissimilar to CL classes as [2]\n- Is the theoretical analysis only relevant to the prompt-based method?", "limitations": "Refer to Weaknesses and Questions", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow and the proposed approach is well motivated\n- The paper has provided theoretical insights\n- The proposed method is much stronger than the existing baselines\n", "weaknesses": "- The method heavily relies on pre-trained models trained with ImageNet. This can be an issue, especially for supervised pre-training, since ImageNet already contains classes similar or identical to classes used for CL and there could be information leak from the pre-training classes to continual learning classes [2].\n- The theoretical contribution is somewhat ambiguous since the decomposition and theorem 1 and 2 are similar to [1].\n\n[2] Kim et al., A multi-head model for continual learning via out-of-distribution replay. CoLLAs, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n**Q1: The method heavily relies on pre-trained models trained with ImageNet. This can be an issue, especially for supervised pre-training, since ImageNet already contains classes similar or identical to classes used for CL and there could be information leak from the pre-training classes to continual learning classes [2].**\n\nA1: In this work, we follow the pre-training dataset from previous work of prompt-based continual learning and focus on the impact of pre-training paradigms (especially *self-supervised pre-training*). Following your suggestion in Q4, we perform an additional experiment of using only the classes that are dissimilar to downstream CL classes for pre-training [2]. In this experimental setup, our approach achieves a more significant lead in performance (19.05\\% on Split CIFAR-100), which is further detailed in our response to Q4. We will add these results in the final version.\n\n**Q2: The theoretical contribution is somewhat ambiguous since the decomposition and theorem 1 and 2 are similar to [1].**\n\nA2: In this work, we focus on continual learning with pre-training, especially prompt-based continual learning that has recently received significant attention in this direction. Our theoretical contribution lies in the context of **pre-training**, where we demonstrate the necessary conditions to achieve good continual learning performance. This is clearly different from the theorem in [1] about continual learning from scratch. Specifically, the major differences between ours and [1] include: (1) The condition is different. We formulate the decomposed components (i.e., TII, WTP and TII) as $\\theta$-conditional probabilities (i.e., $P(\\boldsymbol{x} \\in \\mathcal{X}{i}|\\mathcal{D},\\theta)$,\n$P(\\boldsymbol{x} \\in \\mathcal{X}{i,j}|\\boldsymbol{x} \\in \\mathcal{X}{i},\\mathcal{D},\\theta)$ and $ P(\\boldsymbol{x} \\in \\mathcal{X}^{c}|\\mathcal{D},\\theta)$), where $\\theta$ captures the pre-trained knowledge while not considered in [1]. (2) Due to the additional introduction of $\\theta$, the necessary conditions to achieve good continual learning performance are different from that in [1]. Beside WTP and TII, TAP is especially necessary in the CL problem using pre-training (please refer to A3 for more details). We will add more explanations to make it clearer.\n\n**Q3: Why it’s necessary to have TAP in the CL problem using pre-trained networks, considering that it wasn’t required when training from scratch [1].**\n\nA3: As stated in A2, from the theoretical perspective, TAP is equivalent to TII with WTP when training from scratch. That means $ P(\\boldsymbol{x} \\in \\mathcal{X}^{y}|\\mathcal{D},\\theta) = P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i}}|\\mathcal{D},\\theta)P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i},\\bar{j}}|\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i}},\\mathcal{D},\\theta) $, corresponding to $\\delta +\\epsilon = \\eta$ in our Theorem 1. Therefore, it is unnecessary to have TAP in CIL when training from scratch, which is consistent to [1] as discussed in line 231-233 of our manuscript. On the other hand, when using pre-training with the hierarchical framework as shown in Fig. 3, TAP and TII with WTP are formulated as two different prediction problems where $ P(\\boldsymbol{x} \\in \\mathcal{X}^{y}|\\mathcal{D},\\theta) \\neq P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i}}|\\mathcal{D},\\theta)P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i},\\bar{j}}|\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i}},\\mathcal{D},\\theta) $. Thus, the final performance of having TAP in CIL (i.e., $\\max [P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i},\\bar{j}}|\\mathcal{D},\\theta),P(\\boldsymbol{x} \\in \\mathcal{X}^{y}|\\mathcal{D},\\theta)]$) can outperform that without TAP (i.e., $P(\\boldsymbol{x} \\in \\mathcal{X}{\\bar{i},\\bar{j}}|\\mathcal{D},\\theta)$). These results theoretically demonstrate that the use of pre-training can indeed improve continual learning compared to training from scratch. We will make it clearer.\n\n**Q4: The performance of the proposed method when the feature extractor is pre-trained using only the classes that are dissimilar to CL classes as [2].**\n\nA4: Following the setup in [2], we use a pre-trained checkpoint of the ImageNet subset (with 389 similar classes removed) for continual learning of Split CIFAR-100. The final average accuracy (FAA) of S-Prompt++, CODA-Prompt and our approach is 69.00%, 65.07% and 88.05%, respectively. As can be seen, the performance lead of our approach becomes significantly much larger in this experimental setup, thanks to the explicit optimization of hierarchical components to overcome sub-optimal aspects in prompt-based continual learning. \n\n**Q5: Is the theoretical analysis only relevant to the prompt-based method?**\n\nA5: Indeed, our theoretical analysis could be extended as a general framework of parameter-efficient fine-tuning (PEFT) for continual learning with pre-training. Specifically, based on our theoretical analysis, the objective of continual learning is achieved by three components: (1) optimization of WTP with task-specific parameters, (2) optimization of TII with uninstructed representations, and (3) optimization of TAP with instructed representations. Mainstream PEFT techniques are applicable to this framework in general. The differences lie only in the form of task-specific parameters used in (1), which could be prompt, adapter, LoRA, FiLM, etc. In fact, we have attempted to implement LoRa in our method, which can still achieve considerable performance (e.g., 87.86%/92.24% on Split CIFAR-100 under Sup-21K/iBOT-21K pre-training). Since the major focus of this paper is to improve prompt-based continual learning, which is one of the most active technical routes in this direction, we leave the extension of other PEFT techniques as well as an empirical comparison to further work. We will add it in the final version."}, {"review_id": "49Z9XsPCkb", "reviewer": "Reviewer_6q9Z", "summary": "This work provides a comprehensive analysis of state-of-the-art prompt-based approaches for continual learning with the use of pre-training. The authors empirically demonstrate a clear performance degradation of current strategies under realistic self-supervised pre-training and extensively analyze the exposed sub-optimality. The authors then provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into three hierarchical components, and propose an innovative approach to optimize them explicitly. Extensive experiments on various pre-training paradigms have demonstrated the clear advantages of the proposed method.", "questions": "Some minor concerns remain to be addressed. Please refer to the Weakness.", "limitations": "The authors have properly discussed the limitations and potential negative societal impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides a well-organized formulation of state-of-the-art prompt-based approaches, assessing them as a unified perspective.  \n2. The empirical analysis reveals the degraded performance of prompt-based approaches under self-supervised pre-training, which is an important issue for practical applications.  \n3. The theoretical analysis is very interesting. The hierarchical components are specific to continual learning in the context of pre-training and apply to the settings of task-/domain-/class-incremental learning.  \n4. The proposed method allows for explicit optimization of the continual learning objective through adaptively leveraging pre-training and prompt architectures. Experimental results demonstrate a significant improvement in continual learning performance under different pre-training paradigms.  \n", "weaknesses": "1. Based on proofs in supplementary materials, is the notation $\\bar{c}$ in Eq.(8) equal to $y$? Please check it.  \n2. Could the authors further discuss some concurrent related work such as PromptFusion [1] that exploits prompt and FSA [2] that exploits FiLM adapters for continual learning with pre-training.  \n[1] PromptFusion: Decoupling Stability and Plasticity for Continual Learning. arXiv preprint arXiv:2303.07223.  \n[2] First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning. arXiv preprint arXiv:2303.13199.  \n3. Although experimental results have been sufficient enough, it would be better if some fine-grained datasets can be further analyzed, such as CUB.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n**Q1: Based on proofs in supplementary materials, is the notation $\\bar c$ in Eq.(8) equal to $y$? Please check it.**\n\nA1: Yes, as you understand it, the notation $\\bar c$ in Eq.(8) is equal to $y$, representing the ground truth label of $x$. For clarify, we will replace $\\bar c$ with $y$ and add more explanations in the final version.\n\n**Q2: Could the authors further discuss some concurrent related work such as PromptFusion [1] that exploits prompt and FSA [2] that exploits FiLM adapters for continual learning with pre-training.**\n\nA2: Thank you for pointing out these excellent related work. We will include a discussion on them in the final version. Briefly, PromptFusion [1] employed two prompt-based models to optimize stability and plasticity, respectively, and combined their predictions in a weighted average. The two prompt-based models are constructed from a pre-trained ViT and an additional CLIP, and the combination of their predictions relies on a replay buffer of old training samples. In contrast, our work focuses on a rehearsal-free setting and only requires a pre-trained ViT, which is more resource-efficient and practical in applications.\nFSA [2] adapted a pre-trained backbone with FiLM adapters only in the first learning session and fixed it thereafter. In comparison, our work focuses on prompt-based techniques and adapts the backbone in all learning sessions, so as to accommodate subsequent changes in data distributions.\n\n**Q3: Although experimental results have been sufficient enough, it would be better if some fine-grained datasets can be further analyzed, such as CUB.**\n\nA3: Following your suggestion, we conducted an additional experiment on Split CUB-200-2011, i.e., a random split of its 200 classes into 10 tasks with 20 classes per task. The results are summarized as below: \n\n|   PTM    |     L2P    |  DualPrompt |  S-Prompt++ | CODA-Prompt | HiDe-Prompt |\n| -------- | ---------- | ----------- | ----------- | ----------- | ----------- |\n|  Sup-21K |    74.48   |    82.05    |    82.08    |    74.34    |  **86.56**  |\n| iBOT-21K |    44.29   |    41.31    |    42.73    |    47.79    |  **78.23**  |\n\nFor continual learning of fine-grained classification tasks, the sub-optimality of representative prompt-based approaches is more clearly exposed under self-supervised pre-training (i.e., Sup-21K vs iBOT-21K). In comparison to these baselines, HiDe-Prompt (ours) achieves a substantial lead in performance (more than **30\\%**). Through an extensive ablation study, we observe that this is largely due to the optimization of task-adaptive prediction (TAP) through replaying pseudo representations to adapt the final output layer, as the fine-grained classification generally requires a high degree of precision in classification outputs. Therefore, these results further demonstrate the importance of our empirical and theoretical contributions. We will add it in the final version.\n\n\n[1] PromptFusion: Decoupling Stability and Plasticity for Continual Learning. arXiv preprint arXiv:2303.07223.\n\n[2] First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning. arXiv preprint arXiv:2303.13199."}, {"review_id": "54j3Q704AL", "reviewer": "Reviewer_UkQY", "summary": "This paper introduces a CIL approach that specifically addresses the use of pre-trained models. It is intriguing and significant to explore the most suitable CIL method for pre-trained models.\n\n", "questions": "See above", "limitations": "I think the discussion of limitations in Sec. 6 makes sense.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. CIL and how to do CIL in pre-trained models are very important\n2. The writing is smooth and very easy to follow", "weaknesses": "Excluding the assumption of a pre-trained model, this paper reduces to the existing research (as mentioned in line 233). Thus, I recommend conducting further analysis on CL for pre-trained models.\n\n1. Considering the extensive exploration of pre-trained models in NLP, it is crucial for this paper to compare its approach with NLP continual learning baselines, e.g., [1,2,3].\n\n2. I see no reason why the theory should be restricted to prompts alone. Other parameter-efficient tuning methods such as adapters, Lora, and Prefix can easily be utilized. The author should include a comparison with these methods (see 1 references).\n\n3. In addition, I suggest comparing this paper's approach with TIL methods as well, which do not require TII. This would serve as an additional evaluation to assess the proposed method's performance in  WTP and TAP.\n\n[1]: Achieving forgetting prevention and knowledge transfer in continual learning, NeurIPS 2021  \n[2]: Continual Pre-training of Language Models, ICLR 2023  \n[3]: Continual Learning of Natural Language Processing Tasks: A Survey. https://arxiv.org/abs/2211.12701  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions in final version.\n\n**Q1: Considering the extensive exploration of pre-trained models in NLP, it is crucial for this paper to compare its approach with NLP continual learning baselines, e.g., [1,2,3].**\n\nA1: Thank you for providing these excellent related work. CTR [1] focused on a task-incremental setting, inserting continual learning modules (typically capsule networks) into two locations of a BERT-like pre-trained model. In contrast, our work focuses on a class-incremental setting without the oracle of task identity at test time, and employs prompts as the ``inserted modules''.\nDAS [2] focused on continual pre-training of language models to improve their end-task performance, i.e., upstream continual learning. In comparison, our work focuses on improving the performance for all sequentially arrived tasks ever seen, i.e., downstream continual learning.\nThe third reference [3] is a recent survey on continual learning of NLP tasks. It has summarized representative continual learning strategies, in particular advanced parameter-efficient fine-tuning techniques for continual learning with pre-training. As you mentioned, our work focuses on improving prompt-based methods in this direction. As shown in the responses to your Q2 and Reviewer Qf6e's Q5, our work could potentially be generalized to other PEFT techniques, serving as an important future work.\nWe will add the related work and the above discussion in the final version.\n\n**Q2: I see no reason why the theory should be restricted to prompts alone. Other parameter-efficient tuning methods such as adapters, Lora, and Prefix can easily be utilized. The author should include a comparison with these methods (see 1 references).**\n\nA2: Thank you for your valuable suggestion. We agree that other parameter-efficient fine-tuning (PEFT) techniques could be utilized with our theoretical framework (please refer to the response to Reviewer Qf6e's Q5). All of these PEFT techniques can serve as the task-specific parameters in our HiDe-Prompt, differing mainly in their specific forms. Specifically, prompt-tuning and prefix-tuning both prepended a few parameters to input / hidden vectors, which have been discussed in line 118-124. Adapters inserted adaptive parameters between layers, while LoRa learned an additional low-rank matrix to approximate weight updates and added them to the backbone weights. In fact, we have attempted to implement LoRa in our method, which can still achieve considerable performance (e.g., 87.86%/92.24% on Split CIFAR-100 under Sup-21K/iBOT-21K pre-training). We are now working on implementing other PEFT techniques, which will be released in future work. We will add the above discussion in the final version.\n\n**Q3: In addition, I suggest comparing this paper's approach with TIL methods as well, which do not require TII. This would serve as an additional evaluation to assess the proposed method's performance in WTP and TAP.**\n\nA3: Thank you for your valuable suggestion. As the task identity is provided at test time, TIL can largely reduce the difficulty of continual learning compared to CIL and is usually implemented with task-specific output layers. Therefore, the TIL-version of our framework retains only WTP but removes both TII and TAP (see this conclusion and a theoretical proof in Appendix A.3). In other words, the continual learning problem for our approach becomes evaluating the performance of prompt-tuning itself, i.e., the ability of learning each task with task-specific prompts, without inter-task interference. Since most TIL methods focused on overcoming catastrophic forgetting, they do not constitute a direct comparison with the TIL-version of our approach in terms of motivation. \nBesides, we empirically validate that the TIL-version of our approach achieves sufficiently high performance (e.g., 97.83%/97.83% and 81.55%/80.37% on Split CIFAR-100 and Split ImageNet-R under Sup-21K/iBOT-21K pre-training, respectively), consistent with the above analysis. We will add the above discussion in the final version.\n\n\n[1] Achieving forgetting prevention and knowledge transfer in continual learning, NeurIPS 2021\n\n[2] Continual pre-training of language models, ICLR 2023\n\n[3] Continual learning of natural language processing tasks: A survey. https://arxiv.org/abs/2211.12701\n"}]

Author rebuttal:
We thank all reviewers for their great efforts and constructive comments, which help us to further improve the manuscript. We have tried our best to address these comments with additional experiments, explanations and discussions. Please let us know if you have any further questions.

Meta-review: This paper provides theoretical insight and an empirical method to prompt-based continual learning utilizing self-supervised pre-training. Specifically, a hierarchical decomposition of the continual learning objective is analyzed, yielding within-task prediction, task-identity inference, and task-adaptive prediction as components. Empirically, an ensemble of task-specific prompts and contrastive regularization is employed to optimize the identified hierarchical components. Strong results are shown across a range of architectures and datasets, demonstrating state of art results. 

  All of the reviewers appreciated the work, including the comprehensive insights and analysis leading to the method (pdqM, tgkY), strong methodology (pdqM), and strong empirical performance and ablations (pdqM, tgkY). All of the reviewers had positive scores, with some questions/concerns about the need and success for task-id prediction, lack of fine-grained datasets, and reliance on a pre-trained ImageNet model. The borderline reviewers also asked about the significance of the theoretical contribution, since the role of the pre-training part in the theory is unclear compared to theoretical analysis in prior works.

  The authors provided a thorough response, including new experiments (e.g. fine-grained datasets) and in particular engaged with the reviewers about difference in theoretical analysis with or without pretraining (as analyzed in prior theory). Overall, the rebuttals addressed the substantive parts of the concerns in my view. The setting of utilizing pre-trained self-supervised models is extremely important for continual learning, and in fact some of the discussions about applicability to other PEFT methods could even more significantly increase the impact of this paper. I recommend acceptance and encourage the authors to include a lot of the interesting outcomes of the discussion with reviewers.
Decision: Accept (spotlight)