SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since our objective of TIL with pre-training is  $P(\pmb{x} \in \mathcal{X}_{\bar{i},\bar{j}} | \pmb{x} \in \mathcal{X}_{\bar{i}}, \mathcal{D}, \theta)$ , then we have the loss error  $\mathcal{L} \leq \delta$ . This finishes the proof.
Theorem 6 For task-incremental learning with pre-training, if the loss error  $\mathcal{L} \leq \xi$ , then there always exists a WTP, s.t.  $H_{\mathrm{WTP}} \leq \xi$ .
For TIL with pre-training, its loss error  $\mathcal{L} \leq \xi$ . Assume  $\pmb{x} \in \mathcal{X}_{i,\bar{j}} \subseteq \mathcal{X}_{\bar{i}}$ . According to the proof of Theorem 5, we have
$$
\begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = - \log P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) \\ = - \log \frac {P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i} , \bar {j}} \mid \mathcal {D} , \theta\right)}{P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}} \mid \mathcal {D} , \theta\right)} \tag {39} \\ \leq - \log P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, \bar {j}} | \mathcal {D}, \theta) \\ = \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \} _ {i, j}\right) \\ \leq \xi . \\ \end{array}
$$
B Impact of Pre-Training on Continual Learning
In this work, we focus on continual learning with pre-training, especially prompt-based continual learning that receives significant attention in this direction. Our theoretical contribution lies in the context of pre-training, where we demonstrate the sufficient and necessary conditions to achieve good continual learning performance. This is clearly different from those previous theorems on continual learning from scratch [13], which is analyzed below.
First, the condition is different. We formulate the hierarchical components as  $\theta$ -conditional probabilities, i.e.,  $P(\pmb{x} \in \mathcal{X}_{\bar{i},j} | \pmb{x} \in \mathcal{X}_{\bar{i}}, \mathcal{D}, \theta)$ ,  $P(\pmb{x} \in \mathcal{X}_i | \mathcal{D}, \theta)$  and  $P(\pmb{x} \in \mathcal{X}^c | \mathcal{D}, \theta)$  for WTP, TII and TAP, respectively, where  $\theta$  captures the pre-trained knowledge in initialization. When training from scratch, since the randomly-initialized parameter set  $\theta_0$  carries no information and is greatly different from the optimal solution, it needs to be substantially changed in continual learning and should not be too
into account in objectives. In contrast, the pre-trained parameter set  $\theta$  carries beneficial knowledge for downstream tasks. If the pre-training is adequately strong,  $\theta$  is already close to the optimal solution and only requires appropriate fine-tuning. Therefore,  $\theta$  needs to be stabilized (usually frozen) in continual learning and should be considered in objectives.
Then, due to the incorporation of  $\theta$ , the sufficient and necessary conditions to achieve good continual learning performance become different. In addition to WTP and TII, TAP is especially required in the problem of continual learning with pre-training. Without considering the impact of pretraining, i.e., training from scratch, TAP is equivalent to TII * WTP, i.e.,  $P(\pmb{x} \in \mathcal{X}^y | \mathcal{D}, \theta) = P(\pmb{x} \in \mathcal{X}_i | \mathcal{D}, \theta) P(\pmb{x} \in \mathcal{X}_{i,\bar{j}} | \pmb{x} \in \mathcal{X}_i, \mathcal{D}, \theta)$ , corresponding to  $\delta + \epsilon = \eta$  in our Theorem 1. Therefore, TAP is not required when training from scratch, consistent with the theorems in [13]. On the other hand, with considering the impact of pre-training, TAP and TII * WTP are formulated as two different prediction problems where  $P(\pmb{x} \in \mathcal{X}^y | \mathcal{D}, \theta) \neq P(\pmb{x} \in \mathcal{X}_i | \mathcal{D}, \theta) P(\pmb{x} \in \mathcal{X}_{i,\bar{j}} | \pmb{x} \in \mathcal{X}_i, \mathcal{D}, \theta)$ . Therefore, the final performance of having TAP in continual learning can outperform that without TAP, i.e.,  $\max[P(\pmb{x} \in \mathcal{X}_{i,\bar{j}} | \mathcal{D}, \theta), P(\pmb{x} \in \mathcal{X}^y | \mathcal{D}, \theta)] > P(\pmb{x} \in \mathcal{X}_{i,\bar{j}} | \mathcal{D}, \theta)$ .
In the case of prompt-based continual learning, without considering the impact of pre-training, TAP = TII * WTP, due to classifying the same input in the same semantic space, i.e.,  $\frac{\exp(h_{\psi}(f(\pmb{x}))[y])}{\sum_{j=1}^{t}\sum_{y' \in \mathcal{Y}_j}\exp(h_{\psi}(f(\pmb{x}))[y'])} = \frac{\exp(\hat{h}_{\omega}(f(\pmb{x})[\overline{i}]))}{\sum_{i=1}^{t}\exp(\hat{h}_{\omega}(f(\pmb{x})[i]))} * \frac{\exp(h_{\psi}(f(\pmb{x}))[\overline{j}]))}{\sum_{j \in \mathcal{Y}_i}\exp(h_{\psi}(f(\pmb{x}))[j])}$ . With considering the impact of pre-training, TAP != TII * WTP, because TII * WTP is to calculate  $\frac{\exp(\hat{h}_{\omega}(f(\pmb{x})[\overline{i}]))}{\sum_{i=1}^{t}\exp(\hat{h}_{\omega}(f(\pmb{x})[i]))} * \frac{\exp(h_{\psi}(f(\pmb{x};\pmb{p}_i))[\overline{j}]))}{\sum_{j=1}^{t}\exp(h_{\psi}(f(\pmb{x};\pmb{p}_i)[j]))}$ , while TAP is to calculate  $\frac{\exp(h_{\psi}(f(\pmb{x};\pmb{p}_i))[y])}{\sum_{j=1}^{t}\sum_{y' \in \mathcal{Y}_j}\exp(h_{\psi}(f(\pmb{x};\pmb{p}_i)[y'])}$ . In other words, the backbone parameters are frozen to stabilize the pre-trained knowledge, with additional prompts  $\pmb{p}_i$  to fine-tune the semantic space (i.e., representations). TAP is essentially classifying the instructed representations, i.e.,  $\pmb{h} \in \mathcal{H}_{i,c}$  in Eq. (12), while TII * WTP is performed on uninstructed representations, i.e.,  $\hat{\pmb{h}} \in \hat{\mathcal{H}}_{i,c}$  in Eq. (11). Targeting on different semantic spaces, TAP and TII * WTP are clearly different, i.e., TAP != TII * WTP. Strong empirical results also demonstrate their respective contributions to continual learning performance (see ablation study in Table 3).
In this section, we describe the implementation details of all experiments.
Prompt-Based Approaches: We follow the same implementations of the prompt architectures for all baselines as their original papers [41, 40, 30] (except S-Prompt++, which have been shown to yield strong performance. Specifically, L2P [41] is implemented with  $M = 30$  as the total number of prompts ( $M = 10$  [41] and  $M = 30$  [40] have similar results),  $L_{p} = 5$  as the prompt length, and  $N = 5$  for the Top-  $N$  keys. DualPrompt [40] is implemented with  $L_{g} = 5$  as the prompt length of task-sharing prompts  $g$  inserted into layers 1-2 and  $L_{e} = 20$  as the prompt length of task-specific prompts  $e$  inserted into layers 3-5. S-Prompt++ [39] is implemented similarly to DualPrompt but replaces all task-sharing prompts with task-specific prompts, i.e., the task-specific prompts are inserted into layers 1-5 with prompt length  $L_{e} = 20$ . CODA-Prompt [30] is implemented with  $M = 100$  as the total number of prompts and  $L_{p} = 8$  as the prompt length, inserted into the same layers 1-5 as DualPrompt and S-Prompt++. HiDe-Prompt adopts a similar architecture as S-Prompt++, but replaces the task-specific keys with an auxiliary output layer  $\hat{h}_{\omega}$  to predict the task identity and further preserves statistics of uninstructed and instructed representations. The hyperparameters for HiDe-Prompt are set to  $\alpha = 0.1$ ,  $\tau = 0.8$ , and  $\lambda = 0.1$ .
Training Regime: Following the implementations of previous work [41, 40], we employ a pre-trained ViT-B/16 backbone, an Adam optimizer  $(\beta_{1} = 0.9, \beta_{2} = 0.999)$  and a batch size of 128. The learning rate is set to 0.001 with cosine decay for CODA-Prompt (empirically validated in Table 6), compared to 0.005 for other approaches. We then grid search for an appropriate number of epochs  $(E)$  under different pre-training paradigms, and observe that the best choice is generally consistent on each benchmark (i.e.,  $E$  is relatively insensitive to the pre-training paradigms). For Split CIFAR-100 with  $E \in \{5, 10, 20, 40\}$ , we set  $E = 10$  for L2P and  $E = 20$  for other approaches. For Split ImageNet-R with  $E \in \{25, 50, 75, 150\}$ , we set  $E = 50$  for all approaches. For 5-Datasets with  $E \in \{5, 10, 20, 40\}$ , we set  $E = 40$  for supervised pre-training and  $E = 20$  for self-supervised pre-training. For Split CUB-200, we set  $E = 50$  for all approaches.
Evaluation Metric: We focus on three evaluation metrics for continual learning, including the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM). Specifically, we define the accuracy on the  $i$ -th task after learning the  $t$ -th task as  $A_{i,t}$ , and define the average accuracy of all seen tasks as  $AA_{t} = \frac{1}{t}\sum_{i=1}^{t} A_{i,t}$ . After learning all  $T$  tasks, we report  $\mathrm{FAA} = AA_{T}$ ,  $\mathrm{CAA} = \frac{1}{T}\sum_{t=1}^{T} AA_{t}$ , and  $\mathrm{FFM} = \frac{1}{T-1}\sum_{i=1}^{T-1} \max_{t \in \{1, \dots, T-1\}} (A_{i,t} - A_{i,T})$ . FAA is the primary metric to evaluate the final performance of continual learning, CAA further reflects the historical performance, and FFM serves as a measure of catastrophic forgetting.
Compute: We run all experiments of Split CIFAR-100 on eight Tesla P100-SXM2 GPUs, Split ImageNet-R on four NVIDIA A100 GPUs, 5-Datasets on both, and Split CUB-200 on eight NVIDIA GeForce RTX 3090 GPUs.
D Extended Results
In this section, we provide some extended results for the main text.
Reproduction of Baselines: All results of L2P, DualPrompt and CODA-Prompt are reproduced from their official implementations, while S-Prompt++ is implemented with the code of DualPrompt. We compare the reported and reproduced results in Table 5. In particular, we use the same Sup-21K checkpoint as the original papers of L2P [41] and DualPrompt [40], and their reproduced results are comparable to the reported ones. In contrast, the original paper of CODA-Prompt [30] used a different supervised checkpoint, which is first pre-trained on ImageNet-21K in a self-supervised fashion and then fine-tuned on ImageNet-1K in a supervised fashion. Using the same supervised checkpoint as [30], we have justified that the results of CODA-Prompt on Split CIFAR-100 can be faithfully reproduced. As for Split ImageNet-R, we obtain the official code directly from the authors of CODA-Prompt, but it cannot reproduce exactly the reported results on Split ImageNet-R due to some clean-up issues. Consequently, the reproduced results of CODA-Prompt on Split ImageNet-R slightly underperform the reported ones under the same setting [30]. Besides, we observe that [30] used a different data split than originally proposed [40], resulting in an increase of around  $3\%$  in FAA. Together with the differences in supervised checkpoints, the reproduced results of CODA-Prompt in Table 1 and Table 5 differ by around  $4\%$  in FAA.
Table 5: Comparison of reported and reproduced results. *The forgetting metric for reported results [30] is implemented differently from ours.
Baseline	Split CIFAR-100	Split ImageNet-R
FAA (↑)	FFM (↓)	FAA (↑)	FFM (↓)
L2P (Reported) [41]	83.86	7.35	61.57	9.73
L2P (Reproduced)	83.06	6.58	63.65	7.51
DualPrompt (Reported) [40]	86.51	5.16	68.13	4.68
DualPrompt (Reproduced)	86.60	4.45	68.79	4.49
CODA-Prompt (Reported) [30]*	85.61	1.82	76.66	1.60
CODA-Prompt (Reproduced)	86.46	6.67	74.05	6.48
Learning Rate of CODA-Prompt: As discussed in the original paper of CODA-Prompt [30], its performance depends heavily on the reduced learning rates, especially on Split ImageNet-R. Here we extensively evaluate the effect of learning rate on the performance of CODA-Prompt (see Table 6), in order to justify the strength of our reproduced results under different pre-training paradigms. It can be clearly seen that using a smaller learning rate with cosine decay (i.e.,  $\mathrm{LR} = 0.001$  Cosine) is generally a good choice for CODA-Prompt, which is consistent with its original paper and therefore employed to reproduce its performance in this work.
Statistical Modeling: Here we evaluate the effect of statistical modeling. As analyzed in Sec. 4.2, preserving a dedicated mean and covariance for each class of representations can faithfully recover their distributions and thus achieves strong continual learning performance (see Table 7). Under Sup-21K that is adequately strong for downstream tasks, the performance remains essentially consistent when reducing the covariance to variance (e.g., the FAAs are  $91.74\%$  and  $73.97\%$  on Split CIFAR-100 and Split ImageNet-R, respectively). As a more generalized form of approximated distributions,
Table 6: Effect of learning rate on the performance of CODA-Prompt. Here we present FAA (↑) for all baselines. *The choice in its original paper [30].
Learning Rate (LR)	Split CIFAR-100	Split ImageNet-R
Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K	Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K
LR=0.005 Constant	85.92	76.46	72.39	71.43	76.83	62.74	53.51	57.83	54.73	52.27
LR=0.001 Constant	86.78	80.91	79.31	76.86	76.09	67.73	59.80	64.75	61.33	55.75
LR=0.001 Cosine*	86.94	80.83	79.11	77.50	74.55	70.03	61.22	66.56	63.15	55.15
Table 7: Effect of statistical modeling. We present the final average accuracy (FAA), cumulative average accuracy (CAA) and final forgetting measure (FFM) with  $\pm$  standard deviation under different pre-trained models (PTM), over three runs of different random seeds and task splits.
PTM	Method	Split CIFAR-100	Split ImageNet-R
FAA (↑)	CAA (↑)	FFM (↓)	FAA (↑)	CAA (↑)	FFM (↓)
Sup-21K	Covariance	92.61 ±0.28	94.03 ±0.01	3.16 ±0.10	75.06 ±0.12	76.60 ±0.01	2.17 ±0.19
Multi-Centroid	92.92 ±0.12	94.60 ±0.06	0.86 ±0.12	73.55 ±0.21	75.93 ±0.15	0.95 ±0.02
iBOT-21K	Covariance	93.02 ±0.15	94.56 ±0.05	1.33 ±0.24	70.83 ±0.17	73.23 ±0.08	2.46 ±0.21
Multi-Centroid	92.69 ±0.10	94.69 ±0.13	1.57 ±0.24	70.63 ±0.07	72.94 ±0.06	1.31 ±0.15
iBOT-1K	Covariance	93.48 ±0.11	95.02 ±0.01	1.00 ±0.24	71.33 ±0.21	73.62 ±0.13	2.79 ±0.26
Multi-Centroid	91.78 ±0.08	94.10 ±0.10	1.45 ±0.07	71.33 ±0.29	74.38 ±0.24	1.73 ±0.37
DINO-1K	Covariance	92.51 ±0.11	94.25 ±0.01	0.99 ±0.21	68.11 ±0.18	71.70 ±0.01	3.11 ±0.17
Multi-Centroid	90.06 ±0.14	92.92 ±0.19	2.13 ±0.10	69.34 ±0.16	72.35 ±0.11	1.60 ±0.16
MoCo-1K	Covariance	91.57 ±0.20	93.70 ±0.01	1.19 ±0.18	63.77 ±0.49	68.26 ±0.01	3.57 ±0.96
Multi-Centroid	90.70 ±0.02	93.23 ±0.10	1.56 ±0.12	63.05 ±0.15	66.90 ±0.13	1.78 ±0.36
reserving fewer than 10 centroids (around 5 on average)<sup>6</sup> for each class achieves comparably strong performance in all cases (see Table 7), which ensures both efficiency and generality.
5-Datasets: In Table 8, we evaluate the continual learning performance on 5-Datasets for large inter-task differences. We keep the implementation of prompt architecture for all approaches the same as in Split CIFAR-100 and Split ImageNet-R. Then we perform an extensive grid search for learning rate, number of epochs and other applicable hyperparameters with their official or commonly-used codes. Under Sup-21K, we can essentially reproduce the results of L2P [41]. However, the reproduced results of DualPrompt [40] is lower than the reported one. We find that this issue is also discovered by other users and remains open in github. As the original paper of DualPrompt [40] has not described precisely the implementation for 5-Datasets, we speculate that this issue is possibly due to the use of different prompt architectures. A supporting evidence is that the reported results of DualPrompt [40] are similar to those of S-Prompt++, which is equivalent to replacing all task-sharing prompts in DualPrompt with task-specific prompts. Besides, the most recent CODA-Prompt [30] also performs poorly on this challenging benchmark. To ensure the strength of reproduced results, we have extensively searched the learning rate in  $\{0.001, 0.0005\}$ , the number of epochs in  $\{5, 20, 40\}$  and the hyperparameter of orthogonality regularization in  $\{0, 0.1, 0.01\}$ , and presented the best performance. The inferiority of CODA-Prompt is possibly due to the excessive attentions to the previously-learned prompts, since the task distributions are clearly different and the old knowledge might interfere with the new one. To support this claim, we analyze the attentions of CODA-Prompt and observe a strong preference for previously-learned prompts (around  $0.6 \sim 0.8$ ). Compared to all competitors, HiDe-Prompt achieves the strongest performance under different pre-training paradigms, consistent with the results on Split CIFAR-100 and Split ImageNet-R in Table 1. In particular, the FFM of HiDe-Prompt is almost zero, indicating its great success in overcoming catastrophic forgetting.
Visualization of Representations: We visualize the uninstructed and instructed representations with t-SNE, as shown in Fig. 5 for Sup-21K and Fig. 6 for iBOT-21K. Based on these results, we have the following analysis: (1) The uninstructed representations have shown single-peaked patterns in general, thanks to the use of adequate pre-training. This property allows them to be approximated with Gaussian distributions for preservation and recovery, and allows for correct prediction of task identity from them. (2) The instructed representations tend to be more compact and distinguishable, validating
<sup>6</sup>We use KNN to select multiple centroids (set to a maximum of 10) for each class of representations, where the average number of obtained centroids is usually around 5.
Table 8: Overall performance of continual learning on 5-Datasets.
Baseline	FAA (↑)	FFM (↓)
Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K	Sup-21K	iBOT-21K	iBOT-1K	DINO-1K	MoCo-1K
L2P [41]	81.84	82.25	80.02	76.26	66.89	4.78	8.23	9.46	8.50	24.22
DualPrompt [40]	77.91	68.03	68.92	64.66	59.71	13.11	20.30	24.47	27.24	35.89
S-Prompt++ [39]	86.06	77.20	73.51	69.51	72.91	4.74	15.83	7.47	17.87	15.86
CODA-Prompt [30]	64.18	51.65	48.14	50.86	39.02	17.23	27.53	22.02	26.69	64.33
HiDe-Prompt (Ours)	93.83	94.88	93.89	93.50	93.28	0.44	0.09	0.07	0.04	0.14
Figure 5: Visualization of uninstructed and instructed representations with t-SNE: Part I. Here we present the results of Split CIFAR-100 under Sup-21K. Each color represents a class.
the effectiveness of prompt-based continual learning. The degree of compactness and differentiation varies across pre-training paradigms, contributing to their performance differences. (3) The differences in compactness and differentiation between uninstructed and instructed representations suggest that the design of coarse-grained classification by task and fine-grained classification by class is reasonable for prompt-based continual learning, as do our theoretical analysis and the proposed approach.
Figure 6: Visualization of uninstructed and instructed representations with t-SNE: Part II. Here we present the results of Split CIFAR-100 under iBOT-21K. Each color represents a class.