SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liyuan Wang $^{1}$ , Jingyi Xie $^{1}$ , Xingxing Zhang $^{1*}$ , Mingyi Huang $^{1}$ , Hang Su $^{1}$ , Jun Zhu $^{1*}$
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab,
Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing, China.
wly19@tsinghua.org.cn, jingyi_xie96@163.com, xxzhang1993@gmail.com
huangmingyi2002@126.com, {suhangss, dcszj}@tsinghua.edu.cn
Prompt-based continual learning leverages pre-trained knowledge for downstream continual learning, and has almost reached the performance pinnacle under supervised pre-training. However, our empirical study reveals that the current strategies fall short of their full potential under the more realistic self-supervised pre-training, which is essential for handling vast quantities of unlabeled data in practice. This is largely due to the difficulty of task-specific knowledge being incorporated into instructed representations via prompt parameters and predicted by uninstructed representations at test time. To overcome such sub-optimality, we conduct a theoretical analysis of the continual learning objective in the context of pre-training, and decompose it into hierarchical components: within-task prediction, task-identity inference, and task-adaptive prediction. Based on these empirical and theoretical insights, we propose Hierarchical Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes the hierarchical components with an ensemble of task-specific prompts and statistics of both uninstructed and instructed representations, further with the coordination of a contrastive regularization strategy. Our extensive experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning (e.g., up to  $15.01\%$  and  $9.61\%$  lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our code is available at https://github.com/thu-ml/HiDe-Prompt.
In the realm of artificial intelligence, continual learning has become an area of significant interest. One of the pivotal techniques that greatly facilitate this domain is pre-training, which not only delivers positive knowledge transfer but also enhances resilience to catastrophic forgetting [27, 23, 37, 26, 43]. A recent innovation is the implementation of prompt-based methodologies, which freeze a pretrained transformer backbone and employ a few prompt parameters to steer representation learning. Such approaches typically involve construction of adaptive prompts for each task and inference of appropriate prompts during the test phase. By exploring prompt architectures to accommodate task-sharing and task-specific knowledge, this emerging direction demonstrates distinct superiority, almost reaching the upper bound of continual learning performance under supervised pre-training.
Nonetheless, given that robust pre-trained models typically necessitate the learning of substantial amounts of unlabeled data in a self-supervised manner, the influence of pre-training paradigms
on the effectiveness of prompt-based continual learning represents a significant and unresolved query. To answer this question, we first perform an extensive empirical investigation, and it clearly demonstrates the sub-optimality of recent prompt-based approaches under the more realistic self-supervised pre-training. Since self-supervised representations tend to be more general, task-specific knowledge is difficult to incorporate into instructed representations via prompt parameters, as well as predicted by uninstructed representations at test time. Consequently, the performance of many recent approaches, such as L2P [41], DualPrompt [40], S-Prompt [39] and CODA-Prompt [30], is seriously compromised. We further disclose the importance of adaptive prediction for all tasks together, which can potentially mitigate the aforementioned shortcomings to some extent.
Motivated by these observations, we provide an in-depth theoretical analysis of the continual learning objective in the context of pre-training, which can be decomposed into hierarchical components such as within-task prediction, task-identity inference and task-adaptive prediction. Thanks to the well-distributed representations resulting from adequate pre-training, the hierarchical components can be optimized explicitly by constructing an ensemble of task-specific prompts and exploiting the preserved statistics of uninstructed and instructed representations. A novel contrastive regularization is further devised to coordinate these hierarchical components. We refer to this approach as Hierarchical Decomposition (HiDe-) Prompt and demonstrate its superiority through extensive continual learning experiments, especially under the more realistic self-supervised pre-training.
Our contributions include: (1) We provide an extensive empirical study under self-supervised pretraining to demonstrate the sub-optimality of current progress in prompt-based continual learning; (2) To overcome such sub-optimality, we theoretically analyze the objective of continual learning with pre-training, and decompose it into hierarchical components for model design; (3) With task-specific prompts and representation statistics, we propose an innovative approach to optimize the hierarchical components explicitly; (4) Across various continual learning benchmarks and pre-training paradigms, our approach achieves clearly state-of-the-art performance in a rehearsal-free manner.
Continual Learning: The ability of continual learning is critical for artificial neural networks to accommodate real-world changes [37, 35]. Numerous efforts in this direction have been devoted to overcoming catastrophic forgetting [22, 34, 33]. According to a recent survey [37], representative strategies include selective stabilization of network parameters, replay of a few old training samples, explicit manipulation of optimization programs, exploitation of well-distributed representations, construction of task-specific parameters, etc. The performance of such strategies varies with particular settings of continual learning. As one of the most challenging and representative settings, class incremental learning (CIL) [31, 37] requires a continual learning model to perform all old tasks (or classes) without the oracle of task identity. Strong CIL methods generally depend on storage and rehearsal of old training samples [28, 8, 38], which result in efficiency and privacy issues.
Self-Supervised Learning and Pre-Training: The exploitation of well-distributed representations, especially from the success of large-scale pre-training, brings significant benefits for downstream continual learning [37, 27, 23]. Due to the scarcity and expense of explicit labeling in many real-world applications, self-supervised learning is typically involved in the pre-training stage to cope with huge amounts of unlabeled data. In particular, instance discrimination [4, 7] with contrastive learning [25] has become the dominant strategy, which aims to maximize representation similarity of the same instance and minimize representation similarity of different instances. Besides, self-supervised paradigms have been shown less sensitive to catastrophic forgetting in upstream continual learning [9], providing a practical way to enrich pre-trained knowledge from in-the-wild data.
Prompt-Based Approach: Inspired by parameter-efficient fine-tuning techniques in NLP [11, 10], recent prompt-based approaches [3, 41, 40, 39, 30] are developed to leverage pre-trained knowledge adaptively for downstream continual learning. The basic idea includes construction and inference of adaptive prompts for each task, so as to instruct a frozen transformer backbone. The former mainly focuses on exploring prompt architectures to instruct representations with task-sharing and task-specific knowledge, closely related to the discussion of model architectures in continual learning [37, 36], while the latter attempts to predict appropriate (combinations of) prompts with uninstructed representations. Although such methods have achieved remarkably strong performance under supervised pre-training, whether these advantages are consistent under the more realistic self-
supervised pre-training remains to be explored. A concurrent study [43] observed that self-supervised pre-training is more challenging for continual learning approaches that require fine-tuning of the backbone, implying a non-trivial impact of pre-training paradigms on downstream continual learning.
3 Preliminary Analysis
In this section, we first introduce the problem formulation of prompt-based continual learning, and then evaluate the impact of pre-training paradigms with an extensive empirical study.
3.1 Formulation of Prompt-Based Continual Learning
Continual learning aims to learn a sequence of tasks on their respective training sets  $\mathcal{D}_1, \dots, \mathcal{D}_T$  and excel on their corresponding test sets. The training set for task  $t$  typically consists of various data-label pairs  $\mathcal{D}_t = \{(x_{t,n}, y_{t,n})\}_{n=1}^{N_t}$ , where  $x_{t,n} \in \mathcal{X}_t$  and  $y_{t,n} \in \mathcal{Y}_t$  represent the sample and label elements, respectively. We will use  $|\cdot|$  to denote the cardinality of a set and  $[N] = \{1, 2, \dots, N\}$  as the set of intergers from 1 to  $N$ . Consider a neural network model with a backbone  $f_\theta$  parameterized by  $\theta$ , and an output layer  $h_\psi$  parameterized by  $\psi$ . This model seeks to learn the projection from  $\mathcal{X} = \bigcup_{t=1}^T \mathcal{X}_t$  to  $\mathcal{Y} = \bigcup_{t=1}^T \mathcal{Y}_t$ , aiming to predict the label  $y = h_\psi(f_\theta(x)) \in \mathcal{Y}$  of an unseen test sample  $\mathbf{x}$  drawn from previous tasks. The backbone function  $f_\theta$  is assumed to be pre-trained with a substantial quantity of additional training samples external to each  $\mathcal{D}_t$ . There are commonly three distinct settings for continual learning [31]: task-, domain-, and class-incremental learning (TIL, DIL, and CIL). Specifically,  $\mathcal{Y}_1, \dots, \mathcal{Y}_T$  are identical for DIL while disjoint for TIL and CIL. The task identity is provided for TIL at test time but is not available for DIL and CIL. Therefore, CIL is considered to be more representative and challenging in general. Of note, the continual learning process is rehearsal-free [30]—all elements of  $\mathcal{D}_t$  are available only when learning task  $t$ .
Prompt-based approaches for vision tasks further specify the backbone  $f_{\theta}$  as a pre-trained vision transformer (ViT), where multiple consecutive multi-head self-attention (MSA) layers can transform an input sample into a sequence-like output representation  $\pmb{h} \in \mathbb{R}^{L_h \times D}$  of sequence length  $L_h$  and embedding dimension  $D$ . The backbone parameters  $\theta$  are typically frozen to obtain generalizable representations. A few prompt parameters  $\pmb{p} \in \mathbb{R}^{L_p \times D}$  of sequence length  $L_p$  and embedding dimension  $D$  are prepended to  $\pmb{h}$  to exploit the pre-trained knowledge adaptively. Here we denote the input of the  $l$ -th MSA layer as  $h^l \in \mathbb{R}^{L_h^l \times D}$ , which consists of query  $h_Q^l$ , key  $h_K^l$  and value  $h_V^l$ , and denote the prompt as  $p^l \in \mathbb{R}^{L_p^l \times D}$ . For notation clarity, we take one MSA layer as an example and omit the layer label  $l$  if not necessary. Then, the output of this MSA layer is given as
$$
\operatorname {M S A} \left(\boldsymbol {h} _ {Q}, \boldsymbol {h} _ {K}, \boldsymbol {h} _ {V}\right) = \operatorname {C o n c a t} \left(h _ {1}, \dots , h _ {m}\right) W _ {O}, \tag {1}
$$
$$
h _ {i} = \operatorname {A t t e n t i o n} \left(\boldsymbol {h} _ {Q} W _ {Q, i}, \boldsymbol {h} _ {K} W _ {K, i}, \boldsymbol {h} _ {V} W _ {V, i}\right), i \in [ m ], \tag {2}
$$
where  $W_{O}, W_{Q,i}, W_{K,i}$  and  $W_{V,i}$  are projection matrices,  $m$  is the number of heads, and  $h_{Q} = h_{K} = h_{V}$  in ViT. There are two major implementations of prompt-based methodologies [40], i.e., Prompt Tuning (ProT) [18] and Prefix Tuning (PreT) [19]. Specifically, ProT prepends an identical  $p$  to  $h_{Q}, h_{K}$  and  $h_{V}$ :
$$
f _ {\text {P r o T}} (\boldsymbol {p}, \boldsymbol {h}) = \operatorname {M S A} ([ \boldsymbol {p}; \boldsymbol {h} _ {Q} ], [ \boldsymbol {p}; \boldsymbol {h} _ {K} ], [ \boldsymbol {p}; \boldsymbol {h} _ {V} ]), \tag {3}
$$
where  $[\cdot ;\cdot ]$  denotes the concatenation operation along the dimension of sequence length, and the output in  $\mathbb{R}^{(L_h + L_p)\times D}$  has increased dimensions. In contrast, PreT splits  $\pmb{p}$  into  $p_{K}\in \mathbb{R}^{L_{p} / 2\times D}$  and  $p_V\in \mathbb{R}^{L_p / 2\times D}$  only for  $h_K$  and  $h_V$ , respectively:
$$
f _ {\text {P r e T}} (\boldsymbol {p}, \boldsymbol {h}) = \operatorname {M S A} \left(\boldsymbol {h} _ {Q}, \left[ \boldsymbol {p} _ {K}; \boldsymbol {h} _ {K} \right], \left[ \boldsymbol {p} _ {V}; \boldsymbol {h} _ {V} \right]\right), \tag {4}
$$
where the output dimension remains the same as the input  $\pmb{h} \in \mathbb{R}^{L_h \times D}$ . As the training samples for each task are introduced sequentially, prompt-based continual learning needs to incorporate task-specific knowledge into prompt parameters while overcoming their catastrophic forgetting. The mainstream idea is to construct adaptive prompts for each task and then infer appropriate (combinations of) prompts at test time. Here we compare state-of-the-art approaches from these two aspects, demonstrated conceptually in Fig. 1:
L2P [41] constructs a prompt pool  $\pmb{P} = \{\pmb{p}_1, \dots, \pmb{p}_M\}$  potentially shared by all tasks where  $M$  is the total number of prompts, and then instructs the last MSA layer in a ProT fashion. Each prompt  $\pmb{p}_i$  is associated to a learnable key  $\pmb{k}_i \in \mathbb{R}^D$ , optimized by the cosine distance  $\gamma(q(\pmb{x}), \pmb{k}_i)$  of the top- $N$  keys to a query function  $q(\pmb{x}) = f_{\theta}(\pmb{x})[0]$  to incorporate knowledge. Therefore, the most relevant keys and the corresponding prompts can be selected with uninstructed representations for inference.
DualPrompt [40] constructs task-sharing prompts  $\pmb{g}^{l}$  and task-specific prompts  $\pmb{e}_{t}^{l}$  to instruct different MSA layers in a PreT fashion. All  $\pmb{e}_{t}^{l}$  belonging to the same task is associated to a task-specific key  $\pmb{k}_{t} \in \mathbb{R}^{D}$ , optimized by  $\gamma (q(\pmb{x}),\pmb{k}_t)$ , and the index of the best-matched key is selected for inference.
S-Prompt [39] constructs only task-specific prompts  $\boldsymbol{e}_t$  for each task, and adopts a similar ProT strategy as L2P to instruct the last MSA layer. The inference of task identity is achieved by a simple KNN
Figure 1: Illustration of prompt-based continual learning.
strategy for the nearest task centroid. Unlike other methods, S-Prompt associates an exclusive output head  $\psi_t$  to each task  $t = 1,\dots,T$ .
CODA-Prompt [30] exploits the prompt pool  $\pmb{P}$  by its weighted summation, i.e.,  $\pmb{p} = \sum_{i=1}^{M} \alpha_i \pmb{p}_i$ , where  $\alpha_i = \gamma(q(\pmb{x}), \pmb{k}_i)$  is the weighting factor, and adopts a similar PreT strategy as DualPrompt to instruct multiple MSA layers. The inference of  $\alpha_i$  enables construction of adaptive prompts.
3.2 Empirical Study of Pre-Training Paradigms
Either explicitly or implicitly, the above prompt-based approaches all incorporate the knowledge of each task into prompt parameters and predict their identities from uninstructed representations. To evaluate the impact of pre-training paradigms, we perform an empirical study with widely-used CIL benchmarks such as Split CIFAR-100 and Split ImageNet-R [41, 40]. In addition to supervised pretraining of ImageNet-21K [29] (denoted as Sup-21K), we consider several powerful self-supervised models that release ViT checkpoints $^2$ , such as iBOT [44], DINO [2] and MoCo v3 [5].
Figure 2: Empirical study of prompt-based continual learning under different pre-training paradigms.
We carefully evaluate the official implementations of all baselines for fair comparison. We follow largely the training regimes of L2P [41] and DualPrompt [40], which are basically consistent. As S-Prompt [39] is initially designed for DIL, we slightly modify its implementation by inserting task-specific prompts into the same layers as DualPrompt (i.e., layers 1-5) in a PreT manner, so as to evaluate the impact of prompt architectures. The output layer retains multiple heads associated
$^{2}$ iBOT currently achieves the first-place performance for self-supervised classification on ImageNet and releases checkpoints on both ImageNet-21K and -1K, while others only release checkpoints on ImageNet-1K.
with the task identity (still denoted as S-Prompt), or a single head as with other baselines (denoted as S-Prompt++) . CODA-Prompt [30] is officially implemented in a DualPrompt-like architecture but depends heavily on the use of a smaller learning rate with cosine decay. Here we present its performance with both default and reduced learning rates. Using the same learning rate as [41, 39], we grid search for an appropriate number of epochs (detailed in Appendix D) and report the best performance for all baselines.
As shown in Fig. 2, a, b, the above prompt-based approaches achieve outstanding performance under Sup-21K, where the use of task-specific prompts clearly outperforms task-sharing prompts (i.e., S-Prompt++ ≈ CODA-Prompt > DualPrompt > L2P) due to explicit avoidance of catastrophic forgetting. However, the four baselines suffer significant performance degradation under the more realistic self-supervised pre-training. In particular, the performance differences between prompt architectures have become much smaller, suggesting that the task-specific and task-sharing knowledge are not well differentiated. Besides, CODA-Prompt can generally achieve leading performance as a direct result of the learning rate (LR) reduction rather than the prompt architecture.
We perform two additional experiments to demonstrate the obscured sub-optimality<sup>3</sup>. First, we evaluate the CKA similarity of uninstructed and instructed representations by learning task-specific prompts (Fig. 2, c). The CKA similarity of self-supervised pre-training is significantly higher, suggesting a greater difficulty for prompt parameters to incorporate task-specific knowledge. Second, we evaluate the ability to predict task identity from uninstructed representations and task-specific keys, where self-supervised pre-training exhibits much lower accuracy (Fig. 2, d). Interestingly, although only less than  $40\%$  task identities are correctly predicted, S-Prompt++ can still achieve considerable (albeit sub-optimal) performance, owing to the compensation effects of using a single-head output layer (Fig. 2, e). Together with the results in Fig. 2, c, d, it is conceivable that using an "incorrect" prompt would not severely affect the instructed representations, which can still be correctly predicted in a well-balanced single-head output layer. In contrast, S-Prompt performs much worse than S-Prompt++, as its multi-head output layer undertakes all errors of task-identity inference.
4 Theoretical Foundation and Our Approach
In this section, we first present a theoretical analysis of the sufficient and necessary conditions for improving continual learning in the context of pre-training, and then present an innovative approach for prompt-based continual learning to achieve this objective.
4.1 Hierarchical Decomposition of Continual Learning Objective
For continual learning of sequentially arrived  $\mathcal{D}_t$ ,  $\mathcal{X}_t$  and  $\mathcal{Y}_t$  are the domain and label of task  $t$ . Here we take CIL as a typical scenario for theoretical analysis where  $\mathcal{Y}_t \cap \mathcal{Y}_{t'} = \emptyset$ ,  $\forall t \neq t'$  (see Appendix A for DIL and TIL). Let  $\mathcal{X}_t = \bigcup_j \mathcal{X}_{t,j}$  and  $\mathcal{Y}_t = \{\mathcal{Y}_{t,j}\}$ , where  $j \in [| \mathcal{Y}_t|]$  indicates the  $j$ -th class in task  $t$ . Now assume we have a ground event denoted as  $\mathcal{D} = \{\mathcal{D}_1, \dots, \mathcal{D}_t\}$  and a pre-trained model  $f_\theta$ . For any sample  $\pmb{x} \in \bigcup_{k=1}^t \mathcal{X}_k$ , a general goal of the CIL problem is to learn  $P(\pmb{x} \in \mathcal{X}_{i,j} | \mathcal{D}, \theta)$ , where  $i \in [t]$  and  $j \in [| \mathcal{Y}_i|]$ . This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as  $P(\pmb{x} \in \mathcal{X}_i | \mathcal{D}, \theta)$  and  $P(\pmb{x} \in \mathcal{X}_{i,j} | \pmb{x} \in \mathcal{X}_i, \mathcal{D}, \theta)$ , respectively. Based on Bayes' theorem, we have
$$
P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) = P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta) P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta). \tag {5}
$$
Let  $\bar{i} \in [t]$  and  $\bar{j} \in [| \mathcal{Y}_i|]$  be the ground truth of an  $x$  w.r.t. the task identity and within-task index. Eq. (5) shows that if we can improve either the WTP performance  $P(\pmb{x} \in \mathcal{X}_{\bar{i},\bar{j}} | \pmb{x} \in \mathcal{X}_{\bar{i}}, \mathcal{D}, \theta)$ , the TII performance  $P(\pmb{x} \in \mathcal{X}_{\bar{i}} | \mathcal{D}, \theta)$ , or both, then the CIL performance  $P(\pmb{x} \in \mathcal{X}_{\bar{i},\bar{j}} | \mathcal{D}, \theta)$  would be improved. However, such an improvement is limited since it is upper-bounded by WTP or TII. To further improve the CIL performance, we propose a hierarchical decomposition of its objective. That is, besides the improvement of  $P(\pmb{x} \in \mathcal{X}_{\bar{i},\bar{j}} | \mathcal{D}, \theta)$ , we also need to improve the performance of task-adaptive prediction (TAP), denoted as  $P(\pmb{x} \in \mathcal{X}^y | \mathcal{D}, \theta)$ , where  $\mathcal{X}^y$  represents the domain of
3The experiments in Fig. 2, c, d employ the same prompt architecture and inference methodology as S-Prompt++ to explicitly demonstrate the impact of task specificity. In fact, the objective of continual learning is tantamount to optimizing the probability distribution on the left side of Eq. (5), which can be decomposed into the two probabilities on the right side corresponding to Fig. 2, c, d, respectively. Therefore, such empirical analysis is representative for prompt-based continual learning from a theoretical perspective.
Figure 3: Illustration of Hierarchical Decomposition (HiDe-)Prompt.
class  $y$  in all previous tasks, and  $y = \mathcal{Y}_{\bar{i},\bar{j}}$  is the ground truth label of  $\pmb{x}$ . Then the final goal of CIL is formulated as a multi-objective optimization problem, i.e.,  $\max [P(\pmb {x}\in \mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta),P(\pmb {x}\in \mathcal{X}^y |\mathcal{D},\theta)]$ . Notice that the TII probability is a categorical distribution over all observed tasks upto  $t$ , while the TAP probability is over all observed classes  $\bigcup_{k = 1}^{t}\mathcal{Y}_k$ .
To resolve the problems above, we derive the sufficient and necessary conditions in the context of the widely-used cross-entropy loss. Specifically, we define
$$
H _ {\mathrm {W T P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, j} \mid \boldsymbol {x} \in \mathcal {X} _ {\bar {i}}, \mathcal {D}, \theta\right) \right\} _ {j}\right), \tag {6}
$$
$$
H _ {\mathrm {T H}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {i}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta\right) \right\} _ {i}\right), \tag {7}
$$
$$
H _ {\mathrm {T A P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {c}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} ^ {c} \mid \mathcal {D}, \theta\right) \right\} _ {c}\right), \tag {8}
$$