SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $H_{\mathrm{WTP}}$ ,  $H_{\mathrm{TII}}$ , and  $H_{\mathrm{TAP}}$  are the cross-entropy values of WTP, TII, and TAP, respectively. The operation  $\mathcal{H}(p,q)\triangleq -\mathbb{E}_p[\log q] = -\sum_i p_i\log q_i$ . 1. is a one-hot encoding function.
We now present the first theorem under the CIL scenario (see Appendix A for a detailed proof):
Theorem 1 For continual learning with pre-training, if  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{WTP}}(\boldsymbol{x})] \leq \delta$ ,  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{TII}}(\boldsymbol{x})] \leq \epsilon$ , and  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{TAP}}(\boldsymbol{x})] \leq \eta$ , we have the loss error  $\mathcal{L} \in [0, \max\{\delta + \epsilon, \eta\}]$ , regardless whether WTP, TII and TAP are trained together or separately.
With the use of cross-entropy, the continual learning performance tends to be better as the bounds are tightened. In Theorem 1 we have shown that good performances of WTP, TII and TAP are sufficient to guarantee a good performance of CIL. For completeness, we now study the necessary conditions of a well-performed CIL model in Theorem 2.
Theorem 2 For continual learning with pre-training, if the loss error  $\mathcal{L} \leq \xi$ , then there always exist (1) a WTP, s.t.  $H_{\mathrm{WTP}} \leq \xi$ ; (2) a TII, s.t.  $H_{\mathrm{TII}} \leq \xi$ ; and (3) a TAP, s.t.  $H_{\mathrm{TAP}} \leq \xi$ .
Theorem 2 suggests that if a continual learning model is well trained (i.e., with low loss), then the WTP, TII and TAP for sequential tasks are always implied to be small. It is worth noting that without the pre-trained knowledge carried by  $\theta$ , Theorem 1 and Theorem 2 would degrade to the main conclusion of a previous theoretical study [13], suggesting that the presented theorems are particularly directed to the impact of pre-training for continual learning (detailed in Appendix B). Besides, the paradigm of pre-training is indeed related to the performance of continual learning, because it can affect the distribution of representations from  $f_{\theta}$ , and further WTP, TII and TAP. Previous work has demonstrated that self-supervised representations tend to be more robust to parameter changes than supervised ones [9, 21, 37], which is beneficial for accumulating pre-trained knowledge (if applicable) but challenging for adapting to downstream tasks on a continual basis (see Fig. 2, c, d).
4.2 HiDe-Prompt for Prompt-Based Continual Learning
Motivated by the above empirical and theoretical insights, we propose to optimize explicitly the hierarchical components (i.e., WTP, TII and TAP) for prompt-based continual learning, as shown in Fig. 3. Our proposal stems from a particular advantage of pre-training, where the distributions of uninstructed and instructed representations can be effectively preserved through their statistical information. In the case of classification, for example, since each class tends to have single-peaked
representations (see Appendix D, Fig. 5 and Fig. 6), we can naturally approximate them with Gaussian distributions. For generality, here we denote the approximated distributions of uninstructed and instructed representations as  $\hat{\mathcal{G}}_c$  and  $\mathcal{G}_c$  for each class  $c\in \mathcal{V}_i$ ,  $i\in [t - 1]$ , respectively, and discuss their specific forms latter.
First, we improve WTP through effective incorporation of task-specific knowledge. We construct an expandable prompt pool with only task-specific prompts  $e_t$  to incorporate the knowledge of  $\mathcal{D}_t$ , optimized by a cross-entropy (CE) loss for  $H_{\mathrm{WTP}}$ . The previous prompts  $e_1, \ldots, e_{t-1}$  are frozen to avoid catastrophic forgetting. In order to transfer knowledge for learning each task effectively, we employ a prompt ensemble (PE) strategy, where the current prompt is initialized by the last prompt  $e_t \gets e_{t-1}$  and then optimized with a weighted combination of all previous prompts  $p_t = \alpha \sum_{i=1}^{t-1} e_i + (1 - \alpha) e_t$ .  $\alpha$  is a hyperparameter that controls the strength of inherited old knowledge to facilitate  $p_t$  in learning the current task. Meanwhile, the instructed representations of  $p_t$ , although allowing the new task to be performed well, may overlap with that of the old tasks and thus affect TAP. To overcome this issue, we exploit the old-task statistics of instructed representations (collected by  $f_\theta$  and  $p_i$  for  $i = 1, \ldots, t-1$ ), where for classification we calculate the mean  $\mu_c$  of  $\mathcal{G}_c$  for each class  $c \in \mathcal{Y}_i$ , and design a contrastive regularization (CR):
$$
\mathcal {L} _ {\mathrm {C R}} \left(\boldsymbol {p} _ {t}\right) = \sum_ {\boldsymbol {h} \in \mathcal {H} _ {t}} \frac {1}{\sum_ {i = 1} ^ {t - 1} \left| \mathcal {Y} _ {i} \right|} \sum_ {i = 1} ^ {t - 1} \sum_ {c \in \mathcal {Y} _ {i}} \log \frac {\exp \left(\boldsymbol {h} \cdot \boldsymbol {\mu} _ {c} / \tau\right)}{\sum_ {\boldsymbol {h} ^ {\prime} \in \mathcal {H} _ {t}} \exp \left(\boldsymbol {h} \cdot \boldsymbol {h} ^ {\prime} / \tau\right) + \sum_ {i = 1} ^ {t - 1} \sum_ {c \in \mathcal {Y} _ {i}} \exp \left(\boldsymbol {h} \cdot \boldsymbol {\mu} _ {c} / \tau\right)}, \tag {9}
$$
where  $\mathcal{H}_t$  is the embedding transformation of  $\mathcal{D}_t$  with  $f_{\theta}$  and  $\pmb{p}_t$ .  $\tau$  is the temperature coefficient, which is insensitive and set to 0.8 in practice. Notably, here we use only  $\mu_c$  to represent each class for efficiency, which can be optionally replaced by sampling from  $\mathcal{G}_c$  for better performance.
Then, the loss function of WTP can be defined as
$$
\mathcal {L} _ {\mathrm {W T P}} (\psi , \boldsymbol {p} _ {t}) = \mathcal {L} _ {\mathrm {C E}} (\psi , \boldsymbol {p} _ {t}) + \lambda \mathcal {L} _ {\mathrm {C R}} (\boldsymbol {p} _ {t}). \tag {10}
$$
Therefore, the instructed representations of new classes can be well distinguished for WTP while avoiding overlap with the previous ones.  $\lambda$  is a hyperparameter to balance the impact of old classes.
Second, we improve TII and TAP through exploiting the approximated distributions of uninstructed and instructed representations, respectively. For TII, we construct an auxiliary output layer  $\hat{h}_{\omega}:\mathbb{R}^{D}\to \mathbb{R}^{T}$  parameterized by  $\omega$ , learning explicitly the projection from uninstructed representations to task identity via cross-entropy (i.e.,  $H_{\mathrm{TII}}$ ):
$$
\mathcal {L} _ {\mathrm {T H}} (\omega) = \frac {1}{\sum_ {i = 1} ^ {t} | \mathcal {Y} _ {i} |} \sum_ {i = 1} ^ {t} \sum_ {c \in \mathcal {Y} _ {i}} \sum_ {\hat {\boldsymbol {h}} \in \hat {\mathcal {H}} _ {i, c}} - \log \frac {\exp \left(\hat {h} _ {\omega} (\hat {\boldsymbol {h}}) [ i ]\right)}{\sum_ {j = 1} ^ {t} \exp \left(\hat {h} _ {\omega} (\hat {\boldsymbol {h}}) [ j ]\right)}, \tag {11}
$$
where  $\hat{\mathcal{H}}_{i,c}$  is constructed by sampling an equal number of pseudo representations from  $\hat{\mathcal{G}}_c$  for  $c\in \mathcal{V}_i$  and  $i\in [t]$ . Unlike other baselines that freeze the projection of old tasks (i.e., the previous keys), our  $\hat{h}_{\omega}$  is continually adapted for all tasks and thus greatly facilitates TII.
Similarly, the final output layer  $h_{\psi}:\mathbb{R}^{D}\to \mathbb{R}^{|y|}$  can be further optimized for TAP (i.e.,  $H_{\mathrm{TAP}}$ ):
$$
\mathcal {L} _ {\mathrm {T A P}} (\psi) = \frac {1}{\sum_ {i = 1} ^ {t} | \mathcal {Y} _ {i} |} \sum_ {i = 1} ^ {t} \sum_ {c \in \mathcal {Y} _ {i}} \sum_ {\boldsymbol {h} \in \mathcal {H} _ {i, c}} - \log \frac {\exp \left(h _ {\psi} (\boldsymbol {h}) [ c ]\right)}{\sum_ {j = 1} ^ {t} \sum_ {c ^ {\prime} \in \mathcal {Y} _ {j}} \exp \left(h _ {\psi} (\boldsymbol {h}) [ c ^ {\prime} ]\right)}, \tag {12}
$$
where  $\mathcal{H}_{i,c}$  is constructed by sampling an equal number of pseudo representations from  $\mathcal{G}_c$  for  $c\in \mathcal{Y}_i$  and  $i = 1,\dots,t$ . As  $\omega$  and  $\psi$  are usually light-weight, the optimization of TII and TAP is computationally efficient. At test time, HiDe-Prompt predicts the task identity  $i = \hat{h}_{\omega}(f_{\theta}(\pmb {x}))$  and then the label  $y = h_{\psi}(f_{\theta}(\pmb {x};\pmb {p}_i))$ . Please refer to Appendix Algorithm 1 for more details.
Since the pre-trained representations are usually well-distributed, there are many reasonable strategies to model  $\hat{\mathcal{G}}_c$  and  $\mathcal{G}_c$ . On default, the distributions of uninstructed and instructed representations can be faithfully recovered by modeling each class as a Gaussian with a dedicated mean and covariance. With adequate pre-training, the covariance can be further reduced to variance for efficiency. Alternatively, such statistical modeling can employ multiple centroids obtained from KNN and add Gaussian noise, which is also an efficient choice and is applicable to other task types.
In this section, we first describe the experimental setups, and then present the experimental results.
Table 1: Overall performance of continual learning. We present the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM) with  $\pm$  standard deviation under different pre-trained models (PTM), over three runs of different random seeds and task splits. *The original paper [30] used a different supervised checkpoint and a different data split of ImageNet-R. We asked the authors for official code and reproduced its results under the same setting as [41, 40].
PTM	Method	Split CIFAR-100	Split ImageNet-R
FAA (↑)	CAA (↑)	FFM (↓)	FAA (↑)	CAA (↑)	FFM (↓)
Sup-21K	L2P [41]	83.06 ±0.17	88.25 ±0.01	6.58 ±0.40	63.65 ±0.12	67.25 ±0.02	7.51 ±0.17
DualPrompt [40]	86.60 ±0.19	90.64 ±0.01	4.45 ±0.16	68.79 ±0.31	71.96 ±0.04	4.49 ±0.14
S-Prompt++ [39]	88.81 ±0.18	92.25 ±0.03	3.87 ±0.05	69.68 ±0.12	72.50 ±0.04	3.29 ±0.05
CODA-Prompt [30]*	86.94 ±0.63	91.57 ±0.75	4.04 ±0.18	70.03 ±0.47	74.26 ±0.24	5.17 ±0.22
HiDe-Prompt (Ours)	92.61 ±0.28	94.03 ±0.01	3.16 ±0.10	75.06 ±0.12	76.60 ±0.01	2.17 ±0.19
iBOT-21K	L2P [41]	79.00 ±0.28	85.13 ±0.05	5.55 ±0.36	55.35 ±0.28	58.62 ±0.05	3.73 ±0.53
DualPrompt [40]	78.76 ±0.23	86.16 ±0.02	9.84 ±0.24	54.55 ±0.53	58.69 ±0.01	5.38 ±0.70
S-Prompt++ [39]	79.14 ±0.65	85.85 ±0.17	9.17 ±1.33	55.16 ±0.83	58.48 ±0.18	4.07 ±0.16
CODA-Prompt [30]	80.83 ±0.27	87.02 ±0.20	7.50 ±0.25	61.22 ±0.35	66.76 ±0.37	9.66 ±0.20
HiDe-Prompt (Ours)	93.02 ±0.15	94.56 ±0.05	1.33 ±0.24	70.83 ±0.17	73.23 ±0.08	2.46 ±0.21
iBOT-1K	L2P [41]	75.57 ±0.41	82.69 ±0.06	7.23 ±0.93	60.97 ±0.26	65.95 ±0.02	4.07 ±0.66
DualPrompt [40]	76.63 ±0.05	85.08 ±0.12	8.41 ±0.40	61.51 ±1.05	67.11 ±0.08	5.02 ±0.52
S-Prompt++ [39]	77.53 ±0.56	85.66 ±0.16	8.07 ±0.97	60.82 ±0.68	66.03 ±0.91	4.16 ±0.14
CODA-Prompt [30]	79.11 ±1.02	86.21 ±0.49	7.69 ±1.57	66.56 ±0.68	73.14 ±0.57	7.22 ±0.38
HiDe-Prompt (Ours)	93.48 ±0.11	95.02 ±0.01	1.00 ±0.24	71.33 ±0.21	73.62 ±0.13	2.79 ±0.26
DINO-1K	L2P [41]	70.65 ±0.57	79.02 ±0.01	9.46 ±1.68	57.40 ±0.23	62.56 ±0.20	3.58 ±0.28
DualPrompt [40]	74.90 ±0.21	83.98 ±0.16	10.26 ±0.62	58.57 ±0.45	64.89 ±0.15	5.80 ±0.21
S-Prompt++ [39]	74.97 ±0.46	83.82 ±0.39	7.78 ±0.66	57.64 ±0.16	63.79 ±0.05	5.08 ±0.31
CODA-Prompt [30]	77.50 ±0.64	84.81 ±0.30	8.10 ±0.01	63.15 ±0.39	69.73 ±0.25	6.86 ±0.11
HiDe-Prompt (Ours)	92.51 ±0.11	94.25 ±0.01	0.99 ±0.21	68.11 ±0.18	71.70 ±0.01	3.11 ±0.17
MoCo-1K	L2P [41]	74.85 ±0.28	83.14 ±0.03	6.51 ±0.95	51.64 ±0.19	58.87 ±0.24	2.37 ±0.59
DualPrompt [40]	77.77 ±0.68	85.31 ±0.07	6.61 ±1.08	52.57 ±0.82	60.65 ±0.16	2.73 ±0.49
S-Prompt++ [39]	76.30 ±0.54	83.88 ±0.12	14.67 ±0.64	53.15 ±1.10	60.03 ±0.95	4.11 ±1.84
CODA-Prompt [30]	76.83 ±0.34	84.97 ±0.23	12.60 ±0.02	55.75 ±0.26	65.49 ±0.36	10.46 ±0.04
HiDe-Prompt (Ours)	91.57 ±0.20	93.70 ±0.01	1.19 ±0.18	63.77 ±0.49	68.26 ±0.01	3.57 ±0.96
Benchmark: We consider multiple CIL benchmarks that are widely used for prompt-based continual learning [41, 40, 30]. Specifically, Split CIFAR-100 [14] includes 100-class small-scale images, randomly split into 10 incremental tasks of disjoint classes. Split ImageNet-R [14] includes 200-class large-scale images that are hard examples of ImageNet [29] or newly collected examples of different styles, randomly split into 10 incremental tasks of disjoint classes. 5-Datasets [6] includes CIFAR-10 [14], MNIST [15], Fashion-MNIST [42], SVHN [24] and notMNIST [1] datasets, each treated as an incremental task to evaluate the impact of large inter-task differences. Split CUB-200 [32] includes 200-class fine-grained images of birds, randomly split into 10 incremental tasks of disjoint classes.
Baseline: We compare four representative prompt-based approaches as discussed in Sec. 3.1, such as L2P [41], DualPrompt [40], S-Prompt++ [39] and CODA-Prompt [30]. To evaluate the performance of continual learning, we record the average accuracy of all seen classes after learning each task, presenting the last one as the final average accuracy (FAA) and their historical average as the cumulative average accuracy (CAA) [37]. We also present the final forgetting measure (FFM) of all tasks [37]. We consider a variety of pre-training paradigms for ImageNet-21K and ImageNet-1K, including Sup-21K, iBOT-21K, iBOT-1K, DINO-1K and MoCo-1K, as described in Sec. 3.2.
Implementation: We follow similar implementations as previous work [41, 40, 30]. Specifically, we adopt a pre-trained ViT-B/16 backbone and train with an Adam optimizer  $(\beta_{1} = 0.9, \beta_{2} = 0.999)$ , a batch size of 128, and a constant learning rate of 0.005 (except for CODA-Prompt with a cosine-decaying learning rate of 0.001), and grid search for a proper epoch number. The image inputs are resized to  $224 \times 224$  and normalized to  $[0, 1]$ . Please refer to Appendix C for more details.
Overall Performance: Table 1 presents the main results of all approaches. Consistent with the observations in Sec. 3.2, representative prompt-based approaches achieve outstanding performance under supervised pre-training (i.e., Sup-21K), while perform significantly worse under the more realistic self-supervised pre-training. In particular, the most recent CODA-Prompt [30] outperforms other baselines in general (here we take FAA as the primary metric), but is sensitive to learning rate (see Fig. 2, a, b and Appendix Table 6). In contrast, our HiDe-Prompt achieves generally the
highest FAA, CAA and the lowest FFM, thus greatly superior to all competitors. This advantage is more pronounced under self-supervised pre-training, e.g., up to  $15.01\%$  and  $9.61\%$  lead on Split CIFAR-100 and Split ImageNet-R, respectively. When considering large inter-task differences and fine-grained classification (see Table 2), the performance lead of our approach under self-supervised pre-training becomes even more significant, e.g., up to  $12.63\%$  and  $30.44\%$  on 5-Datasets and Split CUB-200, respectively. We further evaluate the impact of pre-trained knowledge that is disjoint from downstream tasks, following the setup of a recent work [12]. Specifically, we perform Split CIFAR-100 with pre-training on a subset of ImageNet in which 389 similar classes were removed. The FAAs of S-Prompt++, CODA-Prompt and HiDe-Prompt are  $69.00\%$ ,  $65.07\%$  and  $88.05\%$ , respectively. All above results demonstrate the strength of our approach, thanks to the explicit optimization of the hierarchical components that overcomes potential sub-optimality.
Here we explore the characteristics of HiDePrompt in more depth. First, HiDe-Prompt requires much smaller GPU memory compared to CODA-Prompt (e.g., 16141MB vs 25325MB on Split CIFAR-100), since we optimize the construction and inference of appropriate prompts separately as hierarchical components rather than jointly in an end-to-end fashion. Compared to other baselines (except L2P that requires a much smaller epoch number), the computation cost of our approach is comparable in order of magnitude. Using the same single-card
Table 2: Extended benchmarks. Here we present FAA  $(\uparrow)$  for all approaches. The results of 5-Datasets are further detailed in Appendix Table 8.
Method	5-Datasets	Split CUB-200
Sup-21K	iBOT-21K	Sup-21K	iBOT-21K
L2P [41]	81.84	82.25	74.48	44.29
DualPrompt [40]	77.91	68.03	82.05	41.31
S-Prompt++ [39]	86.06	77.20	82.08	42.73
CODA-Prompt [30]	64.18	51.65	74.34	47.79
HiDe-Prompt (Ours)	93.83	94.88	86.56	78.23
A100 GPU, for example, the training times of L2P, DualPrompt, S-Prompt++, CODA-Prompt and HiDe-Prompt are 0.55h, 2.00h, 2.01h, 2.08h, and 2.80h on Split CIFAR-100, respectively. As for the impact of statistical modeling, reserving around 5 centroids for each class performs comparably to a single Gaussian (see Appendix Table 7), which ensures storage efficiency and can potentially be generalized to other task types.
Table 3: Ablation study of hierarchical components. Here we present FAA (↑) for all baselines. "WTP" refers to the use of prompt ensemble within the naive architecture of task-specific prompts.
Baseline	Split CIFAR-100	Split ImageNet-R
Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K	Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K
Naive Architecture	85.11	73.05	72.20	73.74	75.67	60.22	48.00	53.68	54.33	48.77
WTP	87.86	78.86	75.93	75.15	77.15	71.57	55.16	60.86	57.61	53.21
WTP+TII	88.05	80.77	78.90	76.27	77.78	73.76	55.19	61.22	58.41	53.08
WTP+TAP	89.85	84.23	86.04	84.76	85.17	72.57	60.01	67.13	64.26	58.36
WTP+TII+TAP	92.50	90.21	90.52	88.93	89.28	74.89	70.44	70.66	66.78	63.59
WTP+TII+TAP w/ CR	92.61	93.02	93.48	92.51	91.57	75.06	70.83	71.33	68.11	63.77
Table 4: Effect of CR on WTP and the full HiDe-Prompt. Here we present FAA (↑) for all baselines.
PTM	Baseline	Split CIFAR-100	Split ImageNet-R
λ=0	0.001	0.01	0.1	λ=0	0.001	0.01	0.1
iBOT-1K	WTP w/ CR	75.13	72.75	72.53	71.53	61.47	61.17	61.07	59.52
WTP+TII+TAP w/ CR	90.15	91.18	91.60	93.35	70.97	71.25	71.57	71.71
DINO-1K	WTP w/ CR	74.99	73.23	72.74	70.87	57.71	58.23	57.35	56.37
WTP+TII+TAP w/ CR	88.97	90.06	90.54	92.61	66.88	67.42	68.13	68.21
Ablation Study: We perform an extensive ablation study in Table 3 to validate the effectiveness of hierarchical components in HiDe-Prompt. We first construct a naive architecture of task-specific prompts as the baseline and then incorporate progressively the individual designs of Sec. 4.2. In general, the optimization of each component, such as within-task prediction (WTP), task-identity inference (TII) and task-adaptive prediction (TAP), delivers clear benefits and all contribute to the strong performance of HiDe-Prompt. Interestingly, the improvement of TII only becomes apparent
<sup>4</sup>A contemporaneous work [20] also reported FAA under DINO-1K on Split CIFAR-100, Split ImageNet-R and 5-Datasets (83.59%, 61.22% and 89.10%), which lags far behind ours (92.51%, 68.11% and 93.50%).
with WTP+TAP rather than with WTP only, suggesting that these hierarchical components are highly synergistic instead of operating in isolation. The proposed contrastive regularization (CR) helps the instructed representations of individual tasks to be compatible with each other and avoid inter-task overlap, thus further facilitating the performance of WTP+TII+TAP. Moreover, we observe that the improvements of WTP, TII, TAP and CR are generally more significant under self-supervised pre-training, due to effectively addressing the obscured sub-optimality.
Detailed Analysis: Now we further analyze the contributions of the three components for continual learning. First, we evaluate the average performance of learning each new task in Fig. 4, a, where WTP clearly outperforms the naive architecture, indicating that the knowledge of resolving each task is better incorporated into the task-specific prompts. Second, we present the accuracy of predicting task identity from uninstructed representations in Fig. 4, b, which has been substantially improved
Figure 4: Detailed Analysis of WTP and TII.
(up to  $67.74\%$  ) through optimizing TII explicitly. Third, we evaluate the effects of CR on only WTP and the full model of HiDe-Prompt (i.e.,  $\mathrm{WTP + TII + TAP}$ ) in Table 4. Increasing the strength of CR decreases the performance of WTP since the task-specific prompt includes more knowledge about other tasks, but improves the performance of HiDe-Prompt since the inter-task representations become more compatible. These results suggest a potential trade-off between the knowledge for WTP and TAP, which can be modulated explicitly by CR.
In this work, we analyze extensively the advanced prompt-based continual learning from both empirical and theoretical perspectives. An important finding is that, the continual learning objective (which can be decomposed into WTP, TII and TAP in the context of pre-training) is not adequately achieved, and this sub-optimality is clearly exposed under the more realistic self-supervised pre-training. By leveraging statistics of uninstructed and instructed representations, we present a strong approach to optimize explicitly the hierarchical components, which achieves superior performance across various pre-training paradigms. In particular, our theoretical analysis and the proposed approach can serve as a general framework for implementing parameter-efficient fine-tuning techniques (e.g., prompt, adapter, LoRA, FiLM, etc.) in continual learning<sup>5</sup>, which differ only in the form of task-specific parameters. Interestingly, our proposal is consistent with recent advances in neuroscience [17, 16], where the activation of non-memory cells and memory cells (as well as their specific populations) is internally switched. Based on these results, we expect subsequent work to further explore the architecture and optimization of continual learning with an effective use of pre-trained knowledge.
This work remains some potential limitations. First, we assume adequate pre-training to provide meaningful representations, which may not be available in some applications. Second, the prompt-based strategy is mainly applicable to the transformer backbone rather than other backbone architectures. Third, the transformer backbone is frozen and adapted to downstream tasks via prompt parameters, which prevents the pre-trained knowledge from being enriched and updated. As a fundamental research in machine learning, the potential negative societal impact is not obvious at this stage.
This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106302), NSFC Projects (Nos. 62061136001, 92248303, 62106123, 61972224), BNRIST (BNR2022RC01006), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. L.W. is also supported by Shimu Tsinghua Scholar, and J.Z. is also supported by the XPlorer Prize.
In fact, our LoRA version is also competitive, e.g., the FAA is  $87.53\%$  on Split CIFAR-100 under iBOT-21K.