SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[35] Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. arXiv preprint arXiv:2308.14991, 2023.
[36] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In European Conference on Computer Vision, pages 254-271. Springer, 2022.
[37] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
[38] Liuyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations, 2021.
[39] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
[40] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631-648. Springer, 2022.
[41] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[42] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[43] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. arXiv preprint arXiv:2303.05118, 2023.
[44] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2021.
Algorithm 1 Training Algorithm of HiDe-Prompt
Input: Pre-trained transformer backbone  $f_{\theta}$  training sets  $\mathcal{D}_1,\dots,\mathcal{D}_T$  , number of tasks  $T$  , number of epochs  $E$  , hyperparameters  $\alpha$ $\tau$  and  $\lambda$    
Output: Parameters  $p_1,\ldots ,p_T,\omega$  and  $\psi$    
1: Initialize  $e_1,\omega$  and  $\psi$    
2: for  $t = 1,\dots,T$  do   
3: for  $c\in \mathcal{V}_t$  do   
4: Obtain  $\hat{\mathcal{G}}_c$  from  $f_{\theta}$  and  $\mathcal{D}_t$  ▷Uninstructed Representations   
5: if  $t > 1$  then   
6: Initialize  $e_t\gets e_{t - 1}$    
7: Construct  $p_t = \alpha \sum_{i = 1}^{t - 1}e_i + (1 - \alpha)e_t$    
8: else   
9: Construct  $p_t = e_t$    
10: for epoch  $= 1,\dots,E$  do   
11: Optimize  $p_t$  and  $\psi$  with  $\mathcal{L}_{\mathrm{WTP}}$  in Eq. (10) ▷Within-Task Prediction   
12: Optimize  $\omega$  with  $\mathcal{L}_{\mathrm{TII}}$  in Eq. (11) ▷ Task-Identity Inference   
13: Optimize  $\psi$  with  $\mathcal{L}_{\mathrm{TAP}}$  in Eq. (12) ▷ Task-Adaptive Prediction   
14: for  $c\in \mathcal{V}_t$  do   
15: Obtain  $\mathcal{G}_c$  from  $f_{\theta},p_t$  and  $\mathcal{D}_t$  ▷Instructed Representations   
16: return  $(p_1,\dots,p_T,\omega ,\psi)$
A Theoretical Foundation
A.1 Class-Incremental Learning (CIL)
Proof. For class-incremental learning (CIL) with pre-training, assume  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{WTP}}(\boldsymbol{x})] \leq \delta$ ,  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{TII}}(\boldsymbol{x})] \leq \epsilon$ , and  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{TAP}}(\boldsymbol{x})] \leq \eta$ . Let  $y = \mathcal{Y}_{i,\bar{j}}$  be the ground truth of an  $\boldsymbol{x}$ , where  $\bar{i} \in \{1, \dots, t\}$  and  $\bar{j} \in \{1, \dots, |\mathcal{Y}_i|\}$  denote the task identity and within-task index, respectively.
As we defined,
$$
\begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = \mathcal {H} \left(\boldsymbol {1} _ {\bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, j} \mid \boldsymbol {x} \in \mathcal {X} _ {\bar {i}}, \mathcal {D}, \theta\right) \right\} _ {j}\right) \tag {13} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta), \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T H}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {i}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \} _ {i}\right) \tag {14} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta), \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T A P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {c}}, \{P (\boldsymbol {x} \in \mathcal {X} ^ {c} | \mathcal {D}, \theta) \} _ {c}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {\bar {c}} | \mathcal {D}, \theta) \tag {15} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {y} | \mathcal {D}, \theta). \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta\right) \right\} _ {i, j}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \\ = - \log \left(P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, \bar {j}} \mid \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) P \left(\boldsymbol {x} \in \mathcal {X} _ {i} \mid \mathcal {D}, \theta\right)\right) \tag {16} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, \bar {j}} | \boldsymbol {x} \in \mathcal {X} _ {\bar {i}}, \mathcal {D}, \theta) - \log P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i}} | \mathcal {D}, \theta) \\ = H _ {\mathrm {W T P}} (\boldsymbol {x}) + H _ {\mathrm {T I I}} (\boldsymbol {x}). \\ \end{array}
$$
Taking expectations on Eq. (15), we have
$$
\mathcal {L} _ {1} = \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T A P}} (\boldsymbol {x}) \right] \leq \eta . \tag {17}
$$
Taking expectations on both sides of Eq. (16), we have
$$
\begin{array}{l} \mathcal {L} _ {2} = \mathbb {E} _ {\boldsymbol {x}} \left[ \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \left\{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \right\} _ {i, j}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {W T P}} (\boldsymbol {x}) \right] + \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T I I}} (\boldsymbol {x}) \right] \tag {18} \\ \leq \delta + \epsilon . \\ \end{array}
$$
Since our objective of CIL with pre-training is  $\max [P(\pmb {x}\in \mathcal{X}_{\bar{i},\bar{j}}|\mathcal{D},\theta),P(\pmb {x}\in \mathcal{X}^y |\mathcal{D},\theta)]$  , then we have the loss error
$$
\begin{array}{l} \mathcal {L} = \max  \left\{\mathbb {E} _ {\boldsymbol {x}} \left[ \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \} _ {i, j}\right) \right], \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T A P}} (\boldsymbol {x}) \right] \right\} \\ = \max  \left\{\mathcal {L} _ {2}, \mathcal {L} _ {1} \right\} \tag {19} \\ = \max  \{\delta + \epsilon , \eta \}. \\ \end{array}
$$
Proof of Theorem 2
Proof. For CIL with pre-training, its loss error  $\mathcal{L} \leq \xi$ . Assume  $\pmb{x} \in \mathcal{X}_{\bar{i},\bar{j}} \subseteq \mathcal{X}_{\bar{i}}$ . According to the proof of Theorem 1, we have
$$
\begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = - \log P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) \\ = - \log \frac {P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i} , \bar {j}} \mid \mathcal {D} , \theta\right)}{P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}} \mid \mathcal {D} , \theta\right)} \tag {20} \\ \leq - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \\ = \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \} _ {i, j}\right) \\ = \mathcal {L} _ {2} \leq \xi . \\ \end{array}
$$
Likewise, we have
$$
\begin{array}{l} H _ {\mathrm {T I I}} (\boldsymbol {x}) = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \\ = - \log \frac {P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i} , \bar {j}} | \mathcal {D} , \theta)}{P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i} , \bar {j}} | \boldsymbol {x} \in \mathcal {X} _ {\bar {i}} , \mathcal {D} , \theta)} \tag {21} \\ \leq - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \\ = \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta\right) \right\} _ {i, j}\right) \\ = \mathcal {L} _ {2} \leq \xi . \\ \end{array}
$$
In addition, we have formulated the final goal of CIL as a multi-objective optimization problem, i.e.,  $\max [P(\pmb{x} \in \mathcal{X}_{i,\bar{j}}|\mathcal{D},\theta), P(\pmb{x} \in \mathcal{X}^y|\mathcal{D},\theta)]$ . Then, each objective must guarantee the loss error less than  $\xi$ , i.e.,
$$
\begin{array}{l} H _ {\mathrm {T A P}} (\boldsymbol {x}) = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {y} | \mathcal {D}, \theta) \tag {22} \\ = \mathcal {L} _ {1} \leq \xi . \\ \end{array}
$$
A.2 Domain-Incremental Learning (DIL)
For domain-incremental learning (DIL), Let  $\mathcal{X}_t = \bigcup_j\mathcal{X}_{t,j}$  and  $\mathcal{Y}_t = \{\mathcal{Y}_{t,j}\}$ , where  $j\in \{1,\dots,|\mathcal{Y}_t|\}$  denotes the  $j$ -th class in task  $t$ . Now assume we have a ground event denoted as  $\mathcal{D} = \{\mathcal{D}_1,\dots,\mathcal{D}_t\}$  and a pre-trained model  $f_{\theta}$ . For any sample  $\pmb {x}\in \bigcup_{k = 1}^{t}\mathcal{X}_{k}$ , a general goal of the DIL problem is to learn  $P(\pmb {x}\in \mathcal{X}_{*,j}|\mathcal{D},\theta)$ , where  $\mathcal{X}_{*,j}$  represents the  $j$ -th class domain in any task. Of note,  $\mathcal{Y}_t = \mathcal{Y}_{t'}$ $\forall t\neq t'$  for DIL. This can be decomposed into two probabilities, including task-identity inference (TII) and within-task prediction (WTP), denoted as  $P(\pmb {x}\in \mathcal{X}_i|\mathcal{D},\theta)$  and  $P(\pmb {x}\in \mathcal{X}_{i,j}|\pmb {x}\in \mathcal{X}_i,\mathcal{D},\theta)$  respectively. Based on Bayes' theorem, we have
$$
P (\boldsymbol {x} \in \mathcal {X} _ {*, j} | \mathcal {D}, \theta) = \sum_ {i} P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta) P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta), \tag {23}
$$
where  $\{*,j\}$  represents the  $j$  -th class in each domain.
Then we have the following theorems in terms of the sufficient and necessary conditions for improving DIL with pre-training.
Theorem 3 For domain-incremental learning with pre-training, if  $\mathbb{E}_x[H_{\mathrm{WTP}}(\pmb {x})]\leq \delta$ $\mathbb{E}_x[H_{\mathrm{TII}}(\pmb {x})]\leq \epsilon$  , and  $\mathbb{E}_x[H_{\mathrm{TAP}}(\pmb {x})]\leq \eta$  , we have the loss error  $\mathcal{L}\in [0,\max \{\delta +\epsilon +\log t,\eta \} ]$  regardless whether WTP, TII and TAP are trained together or separately.
Proof. Let  $\bar{j} \in \{1, \dots, |\mathcal{Y}_t|\}$  and  $y \in \mathcal{Y}_t$  be the ground truth of an  $x$  w.r.t. the within-task index and class label, and  $y = \mathcal{Y}_{i,\bar{j}}$  for any  $i \in \{1, \dots, t\}$ . Eq. (23) suggests that if we can improve either the WTP performance  $P(\boldsymbol{x} \in \mathcal{X}_{i,\bar{j}} | \boldsymbol{x} \in \mathcal{X}_i, \mathcal{D}, \theta)$ , the TII performance  $P(\boldsymbol{x} \in \mathcal{X}_i | \mathcal{D}, \theta)$ , or both, then the DIL performance  $P(\boldsymbol{x} \in \mathcal{X}^y | \mathcal{D}, \theta)$  would be improved. However, such an improvement is limited since it is upper-bounded by WTP or TII. To further improve the DIL performance, we propose a hierarchical decomposition of the objective. That is, besides the improvement of  $P(\boldsymbol{x} \in \mathcal{X}_{*,\bar{j}} | \mathcal{D}, \theta)$ , we also need to directly improve the performance of task-adaptive prediction (TAP), denoted as  $P(\boldsymbol{x} \in \mathcal{X}^y | \mathcal{D}, \theta)$ , where  $y \in \{1, \dots, |\mathcal{Y}_t|\}$ ,  $\mathcal{X}^y$  represents the domain of class  $y$  in all observed domains, and  $\mathcal{X}^y = \bigcup_i \mathcal{X}_{i,\bar{j}}$ . Then the final goal of DIL is formulated as a multi-objective optimization problem, i.e.,  $\max[P(\boldsymbol{x} \in \mathcal{X}_{*,\bar{j}} | \mathcal{D}, \theta), P(\boldsymbol{x} \in \mathcal{X}^y | \mathcal{D}, \theta)]$ . Notice that the TII probability is a categorical distribution over all observed domains  $\{1:t\}$ , while the TAP probability is over all observed classes  $\bigcup_{k=1}^{t} \mathcal{Y}_k$ .
As similarly defined in CIL, here
$$
\begin{array}{l} \begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} \mid \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) \right\} _ {j}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} \mid \boldsymbol {x} \in \mathcal {X}, \mathcal {D}, \theta) \end{array} \tag {24} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta), \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T I I}} (\boldsymbol {x}) = \mathcal {H} (\gamma , \{P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \} _ {i}) \tag {25} \\ = - \gamma_ {i} \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta), \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T A P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {c}}, \{P (\boldsymbol {x} \in \mathcal {X} ^ {c} | \mathcal {D}, \theta) \} _ {c}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {\bar {c}} | \mathcal {D}, \theta) \tag {26} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {y} | \mathcal {D}, \theta), \\ \end{array}
$$
where  $\gamma = \{\gamma_i\}_{i=1}^t$  represents the possibility of  $\pmb{x}$  belonging to each observed domain,  $\gamma_i \in [0,1]$  and  $\sum_{i} \gamma_i = 1$ .
Then, for any simplex  $\gamma$ , we have
$$
\begin{array}{l} \mathcal {H} \left(\mathbf {1} _ {\bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {*}, j | \mathcal {D}, \theta) \} _ {j}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {\ast , \bar {j}} | \mathcal {D}, \theta) \\ = - \log \left(\sum_ {i} P \left(\boldsymbol {x} \in \mathcal {X} _ {i, \bar {j}} \mid \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) P \left(\boldsymbol {x} \in \mathcal {X} _ {i} \mid \mathcal {D}, \theta\right)\right) \\ \leq - \sum_ {i} \gamma_ {i} \log \left(\frac {P \left(\boldsymbol {x} \in \mathcal {X} _ {i , \bar {j}} \mid \boldsymbol {x} \in \mathcal {X} _ {i} , \mathcal {D} , \theta\right) P \left(\boldsymbol {x} \in \mathcal {X} _ {i} \mid \mathcal {D} , \theta\right)}{\gamma_ {i}}\right) \tag {27} \\ = - \sum_ {i} \gamma_ {i} \log P (\boldsymbol {x} \in \mathcal {X} _ {i, \bar {j}} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta) - \sum_ {i} \gamma_ {i} \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) + \sum_ {i} \gamma_ {i} \log (\gamma_ {i}) \\ = \sum_ {i} \gamma_ {i} H _ {\mathrm {W T P}} + H _ {\mathrm {T I I}} + \mathcal {H} (\gamma). \\ \end{array}
$$
Taking expectations on Eq. (26), we have
$$
\mathcal {L} _ {1} = \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T A P}} (\boldsymbol {x}) \right] \leq \eta . \tag {28}
$$
Taking expectations on both sides of Eq. (27), we have
$$
\begin{array}{l} \mathcal {L} _ {2} = \mathbb {E} _ {\boldsymbol {x}} [ \mathcal {H} (\mathbf {1} _ {\bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {*}, j | \mathcal {D}, \theta) \} _ {j} ] \\ \leq \sum_ {i} \gamma_ {i} \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {W T P}} (\boldsymbol {x}) \right] + \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T I I}} (\boldsymbol {x}) \right] + \mathcal {H} (\gamma) \tag {29} \\ \leq \delta + \epsilon + \log t. \\ \end{array}
$$
Since our objective of DIL with pre-training is  $\max [P(\pmb {x}\in \mathcal{X}_{*,\vec{j}}|\mathcal{D},\theta),P(\pmb {x}\in \mathcal{X}^y |\mathcal{D},\theta)]$  , then we have the loss error
$$
\begin{array}{l} \mathcal {L} = \max  \left\{\mathbb {E} _ {\boldsymbol {x}} \left[ \mathcal {H} \left(\mathbf {1} _ {\bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {*}, j} \mid \mathcal {D}, \theta\right) \right] \right\}, \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {T A P}} (\boldsymbol {x}) \right] \} \\ = \max  \left\{\mathcal {L} _ {2}, \mathcal {L} _ {1} \right\} \tag {30} \\ = \max  \left\{\delta + \epsilon + \log t, \eta \right\}. \\ \end{array}
$$
Theorem 4 For domain-incremental learning with pre-training, if the loss error  $\mathcal{L} \leq \xi$ , then there always exist (1) a WTP, s.t.  $H_{\mathrm{WTP}} \leq \xi$ ; (2) a TII, s.t.  $H_{\mathrm{TII}} \leq \xi$ ; and (3) a TAP, s.t.  $H_{\mathrm{TAP}} \leq \xi$ .
Proof of Theorem 4 For DIL with pre-training, its loss error  $\mathcal{L} = \max[\mathcal{L}_1, \mathcal{L}_2] \leq \xi$ . Assume  $\pmb{x} \in \mathcal{X}_{*,\bar{j}} \subseteq \mathcal{X}^y$ . According to the proof of Theorem 3, if we define  $P(\pmb{x} \in \mathcal{X}_{i,\bar{j}} | \mathcal{D}, \theta) = P(\pmb{x} \in \mathcal{X}_{*,\bar{j}} | \mathcal{D}, \theta)$ , we have
$$
\begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = - \log P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) \\ = - \log \frac {P (\boldsymbol {x} \in \mathcal {X} _ {i , \bar {j}} | \mathcal {D} , \theta)}{P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D} , \theta)} \\ \leq - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, \bar {j}} | \mathcal {D}, \theta) \tag {31} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {\ast , j} | \mathcal {D}, \theta) \\ = \mathcal {H} \left(\mathbf {1} _ {\bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {*}, j | \mathcal {D}, \theta) \} _ {j}\right) \\ = \mathcal {L} _ {2} \leq \xi . \\ \end{array}
$$
Likewise, if we define  $\gamma_{i} = 1$  and  $\gamma_{i^{\prime}} = 0$  for  $i^{\prime}\neq i$ , we have
$$
\begin{array}{l} H _ {\mathrm {T I I}} (\boldsymbol {x}) = - \sum_ {i} \gamma_ {i} \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \\ = - \log \frac {P (\boldsymbol {x} \in \mathcal {X} _ {i , \bar {j}} | \mathcal {D} , \theta)}{P (\boldsymbol {x} \in \mathcal {X} _ {i , \bar {j}} | \boldsymbol {x} \in \mathcal {X} _ {i} , \mathcal {D} , \theta)} \tag {32} \\ \leq - \log (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \\ = - \log (\boldsymbol {x} \in \mathcal {X} _ {\ast , \bar {j}} | \mathcal {D}, \theta) \\ = \mathcal {H} \left(\mathbf {1} _ {\tilde {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {*}, j | \mathcal {D}, \theta) \} _ {j}\right) \\ = \mathcal {L} _ {2} \leq \xi . \\ \end{array}
$$
In addition, we have formulated the final goal of DIL as a multi-objective optimization problem, i.e.,  $\max [P(\pmb{x} \in \mathcal{X}_{*,\bar{j}}|\mathcal{D},\theta), P(\pmb{x} \in \mathcal{X}^y|\mathcal{D},\theta)]$ . Then, each objective must guarantee the loss error less than  $\xi$ , i.e.,
$$
\begin{array}{l} H _ {\mathrm {T A P}} (\boldsymbol {x}) = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {y} | \mathcal {D}, \theta) \tag {33} \\ = \mathcal {L} _ {1} \leq \xi . \\ \end{array}
$$
A.3 Task-Incremental Learning (TIL)
For task-incremental learning (TIL), let  $\mathcal{X}_t = \bigcup_j \mathcal{X}_{t,j}$  and  $\mathcal{Y}_t = \{\mathcal{Y}_{t,j}\}$ , where  $j \in \{1, \dots, |\mathcal{Y}_t|\}$  indicates the  $j$ -th class in task  $t$ . Now assume we have a ground event denoted as  $\mathcal{D} = \{\mathcal{D}_1, \dots, \mathcal{D}_t\}$  and a pre-trained model  $f_\theta$ . For any sample  $\pmb{x} \in \bigcup_{k=1}^t \mathcal{X}_k$ , a general goal of the TIL problem is to learn  $P(\pmb{x} \in \mathcal{X}_{i,j} | \pmb{x} \in \mathcal{X}_{i}, \mathcal{D}, \theta)$ , where  $\bar{i} \in \{1, \dots, t\}$  and  $j \in \{1, \dots, |\mathcal{Y}_{i}|\}$ . This can be equivalent to within-task prediction (WTP). Different from CIL, the task identity is provided in TIL. Unlike DIL,  $\mathcal{Y}_t \cap \mathcal{Y}_{t'} = \emptyset, \forall t \neq t'$ . Then we have the following theorems in terms of the sufficient and necessary conditions for improving TIL with pre-training.
Theorem 5 For task-incremental learning with pre-training,  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{TII}}(\boldsymbol{x})] = 0$ , and task-adaptive prediction (TAP) is degraded into within-task prediction (WTP). If  $\mathbb{E}_{\boldsymbol{x}}[H_{\mathrm{WTP}}(\boldsymbol{x})] \leq \delta$ , we have the loss error  $\mathcal{L} \in [0, \delta]$ .
Proof of Theorem 5
Proof. For task-incremental learning (TIL) with pre-training, assume  $\mathbb{E}_x[H_{\mathrm{WTP}}(x)] \leq \delta$ . Given an  $x$  with the task identity  $\bar{i} \in \{1, \dots, t\}$ , let  $\bar{j} \in \{1, \dots, |\mathcal{V}_i|\}$  be the ground truth of  $x$  w.r.t. the within-task index, and  $y = \mathcal{V}_{\bar{i},\bar{j}}$  be the ground truth label of  $x$ .
$$
\begin{array}{l} H _ {\mathrm {W T P}} (\boldsymbol {x}) = \mathcal {H} \left(\boldsymbol {1} _ {\bar {j}}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, j} \mid \boldsymbol {x} \in \mathcal {X} _ {\bar {i}}, \mathcal {D}, \theta\right) \right\} _ {j}\right) \tag {34} \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta), \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T I I}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {\bar {i}}, \left\{P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \right\} _ {i}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {i} | \mathcal {D}, \theta) \tag {35} \\ = - \log 1 = 0, \\ \end{array}
$$
$$
\begin{array}{l} H _ {\mathrm {T A P}} (\boldsymbol {x}) = \mathcal {H} \left(\mathbf {1} _ {c}, \left\{P \left(\boldsymbol {x} \in \mathcal {X} ^ {c} \mid \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta\right) \right\} _ {c}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {\bar {c}} | \boldsymbol {x} \in \mathcal {X} _ {\bar {i}}, \mathcal {D}, \theta) (36) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} ^ {y} | \boldsymbol {x} \in \mathcal {X} _ {i}, \mathcal {D}, \theta) (50) \\ = H _ {\mathrm {W T P}} (\boldsymbol {x}). \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {H} \left(\mathbf {1} _ {\bar {i}, \bar {j}}, \left\{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \right\} _ {i, j}\right) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {\bar {i}, \bar {j}} | \mathcal {D}, \theta) \\ = - \log \left(P \left(\boldsymbol {x} \in \mathcal {X} _ {i, j} ^ {\prime} \mid \boldsymbol {x} \in \mathcal {X} _ {i} ^ {\prime}, \mathcal {D}, \theta\right) P \left(\boldsymbol {x} \in \mathcal {X} _ {i} ^ {\prime} \mid \mathcal {D}, \theta\right)\right) (37) \\ = - \log P (\boldsymbol {x} \in \mathcal {X} _ {\vec {i}, \vec {j}} | \boldsymbol {x} \in \mathcal {X} _ {\vec {i}}, \mathcal {D}, \theta) - \log P (\boldsymbol {x} \in \mathcal {X} _ {\vec {i}} | \mathcal {D}, \theta) (37) \\ = H _ {\mathrm {W T P}} (\boldsymbol {x}) + H _ {\mathrm {T I I}} (\boldsymbol {x}) \\ = H _ {\mathrm {W T P}} (\boldsymbol {x}). \\ \end{array}
$$
Taking expectations on both sides of Eq. (37), we have
$$
\begin{array}{l} \mathcal {L} = \mathbb {E} _ {\boldsymbol {x}} [ \mathcal {H} (\mathbf {1} _ {\bar {i}, \bar {j}}, \{P (\boldsymbol {x} \in \mathcal {X} _ {i, j} | \mathcal {D}, \theta) \} _ {i, j}) ] \\ = \mathbb {E} _ {\boldsymbol {x}} \left[ H _ {\mathrm {W T P}} (\boldsymbol {x}) \right] \tag {38} \\ \leq \delta . \\ \end{array}
$$