SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Original	CL version	N	# val samples	K
CIFAR100	[26]	[42]	50000	10000	100
Imagenet-R	[13]	[56]	24000	6000	200
Imagenet-A	[14]	[65]	5981	5985	200
CUB	[52]	[65]	9430	2358	200
OmniBenchmark	[64]	[65]	89697	5985	300
VTAB	[62]	[65]	1796	8619	50
Stanford Cars	[25]	[63]	8144	8041	196
Table A2: CIL Datasets. We list references for the original source of each dataset and for split CL versions of them. In the column headers,  $N$  is the total number of training samples,  $K$  is the number of classes following training on all tasks, and # val samples is the number of validation samples in the standard validation sets.
E.2 Domain Incremental Learning (DIL) Datasets
For DIL, we list the domains for each dataset in Table A3. Further details can be found in the cited references in the first column of Table A3. As in previous work, validation data includes samples from each domain for CDDB-Hard, and DomainNet, but three entire domains are reserved for CORe50.
Appendix F Additional results
F.1 Preliminaries
We provide results measured by Average Accuracy and Average Forgetting. Average Accuracy is defined as [30]
$$
A _ {t} = \frac {1}{t} \sum_ {i = 1} ^ {t} R _ {t, i}, \tag {26}
$$
where  $R_{t,i}$  are classification accuracies on the  $i$ -th task, following training on the  $t$ -th task. Average Forgetting is defined as [3]
$$
F _ {t} = \frac {1}{t - 1} \sum_ {i = 1} ^ {t - 1} \max  _ {t ^ {\prime} \in \{1, 2, \dots , t - 1 \}} \left(R _ {t ^ {\prime}, i} - R _ {t, i}\right). \tag {27}
$$
K	T	N	Nt, t = 1,..., T	Nval	val set
CORe50 [29] (Nearly class balanced) (~2400 / class)	50	8	119894	44972	S3, S7, S10
N1 = 14989 (S1)
N2 = 14986 (S2)
N3 = 14995 (S4)
N4 = 14966 (S5)
N5 = 14989 (S6)
N6 = 14984 (S8)
N7 = 14994 (S9)
N8 = 14991 (S11)
CDDB-Hard [27] (Class balanced both train and val)	2	5	16068	5353	Standard
N1 = 6000 (gaugan)	2000
N2 = 2400 (biggan)	800
N3 = 6208 (wild)	2063
N4 = 1200 (whichfaceisreal)	400
N5 = 260 (san)	90
DomainNet [40] (Imbalanced)	345	6	409832	176743	Standard
N1 = 120906 (Real)	52041
N2 = 120750 (Quickdraw)	51750
N3 = 50416 (Painting)	21850
N4 = 48212 (Sketch)	20916
N5 = 36023 (Infograph)	15582
N6 = 33525 (Clipart)	14604
Table A3: DIL Datasets.  $K$  is the number of classes, all of which are included in each task.  $T$  is the total number of tasks,  $N$  is the total number of training samples across all tasks and  ${N}_{\text{val }}$  is the number of validation samples,either overall (first row per dataset) or per task. Also shown are the number of training samples in each task,  ${N}_{t}$  ,and the domain names for the corresponding tasks. Core50 was downloaded from http://bias.csr.unibo.it/maltoni/download/ core50/core50imgs.npz. CDDB was downloaded from https://coral79.github.io/ CDDB_web/. DomainNet was downloaded from http://ai.bu.edu/M3SDA/#dataset - we used the version labelled as "cleaned version,recommended."
Note that for CIL, we calculate the  $R_{t,i}$  as the accuracy for the subset of classes in  $\mathcal{D}_i$ .
For DIL, since all classes are present in each task,  $R_{t,i}$  has a different nature, and is dependent on dataset conventions. For CORe50, the validation set consists of three domains not used during training (S3, S7 and S10, as per Table A3), and in this case, each  $R_{t,i}$  is calculated on the entire validation set. Therefore  $R_{t,i}$  is constant for all  $i$  and  $A_t = R_{t,0}$ . For CDDB-Hard and DomainNet, for the results in Table 3 we treated the validation data in the same way as for CORe50. However, it is also interesting to calculate accuracies for validation subsets in each domain – we provide such results in the following subsection.
F.2 Variability and performance on each task
Fig. A2 shows, for three different random seeds, Average Accuracies and Average Forgetting after each CIL task matching Table 1 in Section 5, without PETL (Phase 1). Due to not using PETL, the only random variability is (i) the choice of which classes are randomly assigned to which task and (ii) the random values sampled for the weights in  $\mathbf{W}$ . We find that after the final task, the Average Accuracy is identical for each random seed when all classes have the same number of samples, and are nearly identical otherwise. Clearly the randomness in  $\mathbf{W}$  has negligible impact. For all datasets except VTAB, the value of RP is clear, i.e. RP (black traces) delivers higher accuracies by the time of the final task than not using RP, or when only using NCM (blue traces). The benefits of second-order statistics even without RP are also evident (magenta traces), with Average Accuracies better than NCM. Note that VTAB always has the same classes assigned to the same tasks, which is why only one repetition is shown. The difference with VTAB is also evident in the average accuracy trend as the number of tasks increases, i.e. the Average Accuracy does not have a clearly decreasing trend as the number of tasks increases. This is possibly due to the difference in domains for each Task making it less likely that confusions occur for classes in one task against those in another task.
Fig. A3 shows comparisons when Phase 1 (PETL) is used. The same trends are apparent, except that due to the SGD training required for PETL parameters, greater variability in Average Accuracy after the final task can be seen. Nevertheless, the benefits of using RP are clearly evident.
Figure A2: Average Accuracies and Forgetting after each Task for CIL datasets (no Phase 1). The left column shows Average Accuracies for three random seeds after each of  $T$  tasks for the seven CIL datasets, without using Phase 1. The right column shows Average Forgetting.
Figure A3: Average Accuracies and Forgetting after each Task for CIL datasets (with Phase 1). The left column shows Average Accuracies for three random seeds after each of  $T$  tasks for the seven CIL datasets, for the best choice of PETL method in Phase 1. The right column shows Average Forgetting.
Fig. A4 shows how accuracy on individual domains changes as tasks are added through training for the DIL dataset, CDDB-Hard. The figure shows that after training data for a particular domain is first used, that accuracy on the corresponding validation data for that domain tends to increase. In some cases forgetting is evident, e.g. for the 'wild' domain, after training on  $T_{4}$  and  $T_{5}$ . The figure also shows that averaging accuracies for individual domains ('mean over domains') is significantly lower than 'Overall accuracy'. For this particular dataset, 'Average accuracy' is potentially a misleading metric as it does not take into account the much lower number of validation samples in some domains, e.g. even though performance on 'san' increases after training on it, it is still under  $60\%$ , which is poor for a binary classification task.
Fig. A5 shows the same accuracies by domain for DomainNet. Interestingly, unlike CDDB-Hard, accuracy generally increases for each Domain as new tasks are learned. This suggests that the pre-trained model is highly capable of forming similar feature representation for the different domains in DomainNet, such that increasing amounts of training data makes it easier to discriminate between classes. The possible difference with CDDB-Hard is that that dataset is a binary classification task in which discriminating between the 'real' and 'fake' classes is inherently more difficult and not reflected in the data used to train the pre-trained model.
Figure A4: Accuracies for DIL dataset CDDB-Hard. Results are shown for the full RanPAC algorithm, using VPT as the PETL method, and  $M = 10000$ . Each task,  $T_{i}$  corresponds to learning on a new domain as shown on the x-axis. The accuracies shown in colors are those for individual domains, following training on the domain shown on the x-axis. 'Mean over domains' is the average of the five domain accuracies after each task.
F.3 Comparison of impact of Phase 1 for different numbers of CIL tasks
As shown in Fig. A2, when Phase 1 is excluded, the final Average Accuracy after the final task has negligible variability despite different random assignments of classes to tasks. This is a consequence primarily of Eqns. (4) being invariant to the order in which a completed set of data from all  $T$  tasks is used in Algorithm 1. As mentioned, the influence of different random values in  $\mathbf{W}$  is negligible. The same effect occurs if the data is split to different numbers of tasks, e.g.  $T = 5$ ,  $T = 10$  and  $T = 20$ , in the event that all classes have equal number of samples, such as CIFAR100.
Therefore, in this section we show in Table A4 results for RanPAC only for the case where variability has greater effect, i.e. when Phase 1 is included, and AdaptMLP chosen. VTAB is excluded from this analysis, since it is a special case where it makes sense to consider only the case of  $T = 5$ . The comparison results for L2P, DualPrompt and ADaM are copied from [65].
Performance when AdaptMLP is used tends to be better for  $T = 5$  and worse for  $T = 20$ . This is consistent with the fact that more classes are used for first-session training with the PETL method when  $T = 5$ . By comparison,  $T = 20$  is often on par with not using the PETL method at all, indicating that the first session strategy may have little value if insufficient data diversity is available within the first task.
Figure A5: Accuracies for DIL dataset DomainNet. Results are shown for the full RanPAC algorithm, using VPT as the PETL method, and  $M = 10000$ . Each task,  $T_{i}$ , corresponds to learning on a new domain as shown on the x-axis. The accuracies shown in colors are those for individual domains, following training on the domain shown on the x-axis. 'Mean over domains' is the average of the six domain accuracies after each task.
T=5	T=10	T=20	RanPAC - No Phase 1
CIFAR100	RanPAC	92.4%	92.2%	90.8%	89.0%
AdaM	88.5%	87.5%	85.2%
DualPrompt	86.9%	84.1%	81.2%
L2P	87.0%	84.6%	79.9%
NCM	88.6%	87.8%	86.6%	83.4%
ImageNet-R	RanPAC	79.9%	77.9%	74.5%	71.8%
AdaM	74.3%	72.9%	70.5%
DualPrompt	72.3%	71.0%	68.6%
L2P	73.6%	72.4%	69.3%
NCM	74.0%	71.2%	64.7%	61.2%
ImageNet-A	RanPAC	63.0%	58.6%	58.9%	58.2%
AdaM	56.1%	54.0%	51.5%
DualPrompt	46.6%	45.4%	42.7%
L2P	45.7%	42.5%	38.5%
NCM	54.8%	49.7%	49.3%	49.3%
CUB	RanPAC	90.6%	90.3%	89.7%	89.9%
AdaM	87.3%	87.1%	86.7%
DualPrompt	73.7%	68.5%	66.5%
L2P	69.7%	65.2%	56.3%
NCM	87.0%	87.0%	86.9%	86.7%
OmniBenchmark	RanPAC	79.6%	79.9%	79.4%	78.2%
AdaM	75.0%*	74.5%	73.5%
DualPrompt	69.4%*	65.5%	64.4%
L2P	67.1%*	64.7%	60.2%
NCM	75.1%	74.2%	73.0%	73.2%
Cars	RanPAC	69.6%	67.4%	67.2%	67.1%
NCM	41.0%	38.0%	37.9%	37.9%
Table A4: Comparison of CIL results for different number of tasks. For this table, the AdaptMLP PETL method was used for RanPAC. The comparison results for L2P, DualPrompt and ADaM are copied from [65]; for ADaM, the best performing PETL method was used. In most cases, Average Accuracy decreases as  $T$  increases, with  $T = 20$  typically not significantly better than the case of no PETL ("No Phase 1"). Values marked with * indicates data is for  $T = 6$  tasks, instead of  $T = 5$ . Note that accuracies for comparison methods for Cars were not available from [65]
F.4 Task Agnostic Continual Learning
Unlike CIL and DIL, 'task agnostic' CL is a scenario where there is no clear concept of a 'task' during training [61]. The concept is also known as 'task-free' [44]. It contrasts with standard CIL where although inference is task agnostic, training is applied to disjoint sets of classes, described as tasks. To illustrate the flexibility of RanPAC, we show here that it is simple to apply it to task agnostic continual learning. We use the Gaussian scheduled CIFAR100 protocol of [57], which was adapted from [44]. We use 200 'micro-tasks' which sample from a gradually shifting subset of classes, with 5 batches of 48 samples in each micro-task. There are different possible choices for how to apply Algorithm 1. For instance, the 'first session' for Phase 1 could be defined as a particular total number of samples trained on, e.g.  $10\%$  of the anticipated total number of samples. Then in Phase 2, the outer for loop over tasks could be replaced by a loop over all batches, or removed entirely. In both cases, the result for  $\mathbf{G}$  and  $\mathbf{C}$  will be unaffected. The greater challenge is in determining  $\lambda$ , but generally for a large number of samples,  $\lambda$  can be small, or zero. For a small number of training samples, a queue of samples could be maintained such that the oldest sample in the queue is used to update  $\mathbf{G}$  and  $\mathbf{C}$ , with all newer samples used to compute  $\lambda$  if inference is required, and then all samples in the buffer added to  $\mathbf{G}$  and  $\mathbf{C}$ .
Here, for simplicity we illustrate application to Gaussian-scheduled CIFAR100 without any Phase 1. Fig. A6 shows how test accuracy changes through training both with and without RP. The green trace illustrates how the number of classes seen in at least one sample increases gradually through training, instead of in a steps like in CIL. The red traces show validation accuracy on the entirety of the validation set. As expected, this increases as training becomes exposed to more classes. The black traces show the accuracy on only the classes seen so far through training. By the end of training, the red and black traces converge, as is expected. Fluctuations in black traces can be partially attributed to not optimizing  $\lambda$ . The final accuracies with and without RP match the values for  $T = 10$  CIL shown in Table 1.
Figure A6: Task agnostic example. Application of Phase 2 of Algorithm 1 to the Gaussian-scheduled CIFAR100 task-agnostic CL protocol.
F.5 Scaling with projection size
Table A5 shows, for the example of split CIFAR100, that it is important to ensure  $M$  is sufficiently large. In order to surpass the accuracy obtained without RP (see 'No RPs or Phase 1' in Table 1),  $M$  needs to be larger than 1250.
F.6 Comparison of PETL Methods and ViT-B/16 backbones
Fig. A7 shows how performance varies with PETL method and by ViT-B/16 backbone. For some datasets, there is a clearly superior PETL method. For example, for CIFAR100, AdaptMLP gives better results than SSF or VPT, for both backbones, for Cars VPT is best, and for ImageNet-A, SSF
M	Accuracy
100	71.6%
200	80.3%
400	83.9%
800	86.2%
1250	86.8%
2500	87.7%
5000	88.4%
10000	88.8%
15000	89.0%
is best. There is also an interesting outlier for VTAB, where VPT and the ImageNet-21K backbone failed badly. This variability by PETL method suggests that the choice of method for first-session CL strategies should be investigated in depth in future work.
Fig. A7 also makes it clear that the same backbone is not optimal for all datasets. For example, the ViT-B/16 model fine-tuned on ImageNet-1K is best for ImageNet-A, but the ViT-B/16 model trained on ImageNet-21K is best for CIFAR100 and OmniBenchmark. For Cars, the best backbone depends on the PETL method.
Fig. A8 summarises how the two ViT networks compare. For all datasets and method variants we plot the Average Accuracy after the final task for one pre-trained ViT network (self-supervised on ImageNet-21K) against the other (fine-tuned on ImageNet-1K). Consistent with Fig. A7, the best choice of backbone is both dataset dependent and method-dependent.
F.7 Experiments with ResNets
Unlike prompting strategies, our approach works with any feature extractor, e.g. both pre-trained transformer networks and pre-trained convolutional neural networks. To illustrate this, Table A6 and Table A7 shows CIL results for ResNet50 and ResNet 152, respectively, pre-trained on ImageNet. We used  $T = 10$  tasks (except for VTAB, which is  $T = 5$  tasks). Although this is different to the  $T = 20$  tasks used for ResNets by [65], the accuracies we report for NCM are very comparable to those in [65]. As with results for pre-trained ViT-B/16 networks, the use of random projections and second-order statistics both provide significant performance boosts compared with NCM alone. We do not use Phase 1 of Algorithm 1 here but as shown by [37, 65], this is feasible for diverse PETL methods for convolutional neural networks. Interestingly, ResNet152 with RP produces reduced accuracies compared to ResNet50 on CUB, Omnibenchmark and VTAB. It is possible that this would be remedied by seeking an optimal value of  $M$ , whereas for simplicity we chose  $M = 10000$ . Note that unlike the pre-trained ViT-B/16 model, the pre-trained ResNets require preprocessing to normalize the input images.
Table A5: Scaling with  $M$  for split CIFAR100. The table shows final average accuracy for  $T = 10$  and no Phase 1, with  $\lambda = 100$  as a constant for all  $M$ , using ViT-B/16 trained on ImageNet21K.
Method	CIFAR100	IN-R	IN-A	CUB	OB	VTAB	Cars
Phase 2	75.5%	56.5%	35.5%	70.9%	68.0%	88.6%	48.5%
Phase 2 (No RPs)	73.3%	54.8%	36.6%	62.7%	60.2%	87.7%	42.6%
NCM	61.5%	43.1%	25.8%	55.4%	57.4%	82.2%	26.3%
Table A6: Results for ResNet50. Results are final Average Accuracies for  $T = 10$  tasks, except VTAB which is  $T = 5$  tasks. Pre-trained weights were IMAGENET1K_V2 variants from PyTorch.
Method	CIFAR100	IN-R	IN-A	CUB	OB	VTAB	Cars
Phase 2	79.7%	58.7%	40.9%	66.7%	65.8%	90.0%	45.1%
Phase 2 (No RPs)	77.8%	56.9%	39.9%	59.4%	58.0%	87.9%	38.8%
NCM	70.1%	47.5%	32.1%	50.6%	56.1%	82.6%	25.2%
Table A7: Results for ResNet152. Other caption information is the same as Table A6.
ResNet results for DIL are provided in Table A8.
Method	Backbone	CORe50	CDDB-Hard	DomainNet
Phase 2	ResNet50	88.5%	75.9%	52.7%
Phase 2 (No RPs)	ResNet50	86.7%	75.6%	41.9%
NCM	ResNet50	73.1%	65.1%	32.9%
Phase 2	ResNet152	87.1%	71.3%	53.8%
Phase 2 (No RPs)	ResNet152	86.8%	71.9%	42.9%
NCM	ResNet152	72.0%	65.6%	35.1%
Table A8: DIL Results for ResNet50 and ResNet152. Pre-trained weights were IMAGENET1K_V2 variants from PyTorch.
F.8 Experiments with CLIP vision model
To further verify the general applicability of our method, we show in Table A9 CIL results from using a CLIP [41] vision model as the backbone pre-trained model. The same general trend as for pre-trained ViT-B/16 models and ResNets can be seen, where use of RPs (Phase 2 in Algorithm 1) produces better accuracies than NCM alone. Interestingly, the results for Cars is substantially better with the CLIP vision backbone than for ViT-B/16 networks. It is possible that data from a very similar domain as the Cars dataset was included in training of the CLIP vision model. The CLIP result for Phase 2 only (see ablations in Table 1) is also better than ViT/B-16 for Imagenet-R, but for all other datasets, ViT/B-16 has higher accuracy. Note that unlike the pre-trained ViT-B/16 model, the pre-trained CLIP vision model requires preprocessing to normalize the input images.
Method	CIFAR100	IN-R	IN-A	CUB	OB	VTAB	Cars
Phase 2	85.0%	76.1%	39.7%	79.4%	75.1%	89.6%	90.4%
Phase 2 (No RPs)	80.9%	67.0%	36.8%	72.8%	66.3%	89.8%	85.8%
NCM	77.1%	68.0%	32.5%	73.7%	67.8%	79.4%	85.6%
Table A9: Results for CLIP-ViT-B/16. Results are final Average Accuracies for  $T = 10$  tasks, except VTAB which is  $T = 5$  tasks. We used pre-trained weights from OpenCLIP, finetuned on ImageNet-1K [19]. The weights were loaded from the timm library using the model file name vit_base_batch32_224 Clip_laion2b.
CLIP results for DIL are provided in Table A10. The CLIP result outperforms that for Phase 2 only for the ViT-B/16 model for CDDB-Hard and DomainNet, but not CORe50 (Table 3).
Method	CORe50	CDDB-Hard	DomainNet
Phase 2	92.9%	84.4%	68.7%
Phase 2 (No RPs)	90.2%	81.7%	59.1%
NCM	83.9%	66.9%	51.0%
Table A10: DIL Results for ResNet50 and ResNet152. Pre-trained weights were IMA-GENET1K_V2 variants from PyTorch.
F.9 Experiments with regression targets using CLIP vision and language models
Until this point, we have defined the matrix  $\mathbf{C}$  as containing Class-Prototypes (CPs), i.e.  $\mathbf{C}$  has  $N$  columns representing averaged feature vectors of length  $M$ . However, with reference to Eqn. (13), the assumed targets for regression,  $\mathbf{Y}_{\mathrm{train}}$ , can be replaced by different targets. Here, we illustrate this using CLIP language model representations as targets, using OpenAI's CLIP ViT-B/16 model.
Using CIFAR100 as our example, we randomly project CLIP's length-512 vision model representations as usual, but also use the length-512 language model's representations of the 100 class names, averaged over templates as in [41]. We create a target matrix of size  $N \times 512$  in which each row is the language model's length 512 representation for the class of each sample. We then solve for  $\mathbf{W}_{\mathrm{o}} \in \mathcal{R}^{M \times 512}$  using this target instead of  $\mathbf{Y}_{\mathrm{train}}$ .
When the resulting  $\mathbf{W}_{\mathrm{o}}$  is applied to a test sample, the result is a length-512 prediction of a language model representation. In order to translate this to a class prediction, we then apply CLIP in a standard zero-shot manner, i.e. we calculate the cosine similarity between the predictions and each of the normalized language model's length 512 representation for the class of each sample.
The resulting final Average Accuracy for  $M = 5000$  and  $T = 10$  is  $77.5\%$ . In comparison, CLIP's zero shot accuracy for the same data is  $68.6\%$ , which highlights there is value in using the training data to modify the vision model's outputs. When RP is not used, the resulting final Average Accuracy is  $71.4\%$ .
In future work, we will investigate whether these preliminary results from applying Algorithm 1 to a combination of pre-trained vision and language models can translate into demonstrable benefits for continual learning.
F.10 Note on reproducibility
Accuracies reported in Tables 1-3 have been updated since the original submission to reflect those published in our code repository at https://github.com/zhoudw-zdw/RevisitingCIL. Note that accuracy results achieved can vary by  $\sim \pm 1\%$  for different random seeds, even for the same class ordering, especially for PETL methods.
Figure A7: Comparison of PETL Methods and two ViT models. For the seven CIL datasets, and  $T = 10$ , the figure shows Average Accuracy and Average Forgetting after each task, for each of AdaptMLP, SSF and VPT. It shows this information for the two ViT-B/16 pre-trained backbones we primarily investigated.
Figure A8: Comparison of backbone ViTs. Scatter plot of results for ViT-B/16 models pre-trained on ImageNet1K vs ImageNet21K.