SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[59] Tz-Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9601–9610, 2022. 2
[60] Qingsen Yan, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. Learning Bayesian sparse networks with full experience replay for continual learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 109-118, 2022. 2
[61] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational Bayes. arXiv:1803.10123, 2019. 27
[62] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. arXiv:1910.04867, 2019. 21
[63] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 19148-19158, October 2023. 2, 3, 8, 9, 10, 16, 21
[64] Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, volume 13667 of Lecture Notes in Computer Science, pages 594-611. Springer, 2022. 21
[65] Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv:2303.07338, Submitted 13 March, 2023. 2, 3, 4, 6, 7, 8, 9, 10, 16, 18, 20, 21, 25, 26, 28
Appendix A Overview of RanPAC and comparison to other strategies
Figure A1 provides a graphical overview of the two phases in Algorithm 1. Table A1 provides a summary of different strategies for leveraging pre-trained models for Continal Learning (CL) and how our own method, RanPAC, compares.
First-session PETL Adaptation (Algorithm 1, Phase 1)
Training data for Phase 1
Figure A1: Overview of RanPAC for CL classification. In Phase 1 of Algorithm 1 we optionally inject parameters for a Parameter-Efficient Transfer Learning (PETL) method into a frozen pretrained model (PTM). The PETL parameters are trained only on Task 1 (the 'first-session') of a set of  $T$  continual learning tasks, to help bridge the domain gap, as in [65, 37]. Then in Phase 2, first,  $L$ -dimensional feature vectors,  $\mathbf{f}$ , are extracted from the network after completion of learning in Phase 1 (now frozen). Then, the extracted feature vectors are randomly projected to dimension  $M$  (typically  $M > L$ ) using frozen weights  $\mathbf{W}$ , followed by nonlinear activation,  $\phi$ , to obtain new feature vectors,  $\mathbf{h} = \phi (\mathbf{f}^{\top}\mathbf{W})$ . Training in Phase 2 is comprised of continual iterative updating of class prototypes and the Gram matrix, followed by a matrix inversion to compute  $M\times N$  matrix  $\mathbf{W}_{\mathrm{o}}$ , following the end of each Task's training. These weights can be thought of as decorrelated class prototypes that linearly weight the features in  $\mathbf{h}$  to obtain class predictions in  $\mathbf{y}$ .
Prompting [57, 56, 46, 55]	Fine-tuning [63, 47]	CP [20, 65, 37]	CP + RP RanPAC
No Rehearsal Buffer	✓	✓	✓	✓
Pre-trained model frozen	✓	✓	✓
Transformers and CNNs	?	✓	✓
Simplicity	✓	✓
Parameter-Efficient	✓	✓	✓
Theoretical support	✓
SOTA Performance	✓
Table A1: Comparison of different strategies. In Section 1, we categorised existing methods for leveraging pre-trained models for CL into three strategies: Prompting, Fine-tuning and Class-Prototypes (CP). Our own method, RanPAC, is a CP strategy in which we introduce a frozen Random Projection (RP) layer with nonlinear activation. Overall, we argue that CP methods provide the best combination of benefits, and within existing CP methods, RanPAC provides the strongest results, complemented by theoretical support for its use of RP and second-order statistics to decorrelate CPs. We note that rehearsal buffers have been used with some of the cited methods to boost performance, and could potentially also do so for methods where they have not yet been used, such as RanPAC.
Appendix B Theoretical support
B.1 Chernoff bound of the norm of the projected vectors
We provide further details for the discussion in Section 4.2. The norm of the vector projected using the Chernoff Bound can be written as:
$$
\mathbb {P} \left( \right.\left| \right.\left\| \mathbf {W} ^ {\top} \mathbf {f} \right\| - \mathbb {E} _ {\mathbf {W}} \left[\left\| \mathbf {W} ^ {\top} \mathbf {f} \right\|\right] \mid > \epsilon \sigma^ {2}\left. \right) \leq 2 \exp \left(- \frac {\epsilon^ {2} \sigma^ {2}}{2 M + \epsilon}\right). \tag {6}
$$
This bound indicates the relation between the dimensionality and the expected variation in the norm of the projected vectors. For fixed  $\sigma$  and  $\epsilon$ , as  $M$  increases, the right-hand side approaches 1, indicating that it is more likely for the norm of the projected vector to be in a desired distance to the expectation. In other words, these projected vectors in higher dimensions almost surely reside on the boundary of the distribution with a similar distance to the mean (the distribution is a better Gaussian fit).
B.2 The effects of increasing the projection dimensions
The Gram matrix of the projected vectors can be obtained by considering the inner product of any two vectors  $\mathbf{f},\mathbf{f}^{\prime}$ . As presented in Eqn. (3), this is derived as:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ (\mathbf {W} ^ {\top} \mathbf {f}) ^ {\top} (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) \right] = \mathbb {E} _ {\mathbf {W}} \left[ (\mathbf {f} ^ {\top} \mathbf {W}) (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) \right] \\ = \mathbb {E} _ {\mathbf {W}} \left[ \sum_ {i} ^ {M} \mathbf {W} _ {(i)} ^ {2} \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime} + \sum_ {i \neq j} ^ {M} \mathbf {W} _ {(i)} ^ {\top} \mathbf {W} _ {(j)} \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime} \right] \\ = \underbrace {\sum_ {i} ^ {M} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(i)} ^ {2} \right] \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime}} _ {= M \sigma^ {2} \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime}} + \underbrace {\sum_ {i \neq j} ^ {M} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(i)} \right] ^ {\top} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(j)} \right] \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime}} _ {= 0}, \tag {7} \\ \end{array}
$$
where the second term is zero for any two zero-mean independently drawn random vectors  $\mathbf{W}_{(i)},\mathbf{W}_{(j)}$ . We can derive the following from this expansion:
1. Using the Chernoff inequality, for any two vectors, we have
$$
\begin{array}{l} \mathbb {P} \left(\left| (\mathbf {W} ^ {\top} \mathbf {f}) ^ {\top} (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) - \mathbb {E} _ {\mathbf {W}} \left[ (\mathbf {W} ^ {\top} \mathbf {f}) ^ {\top} (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) \right] \right| > \epsilon^ {\prime} M \sigma^ {2}\right) \leq 2 \exp \left(- \frac {\epsilon^ {\prime 2} M \sigma^ {2}}{(2 + \epsilon^ {\prime})}\right) \\ \mathbb {P} \left(\left| (\mathbf {W} ^ {\top} \mathbf {f}) ^ {\top} (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) - M \sigma^ {2} \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime} \right| > \epsilon^ {\prime} M \sigma^ {2}\right) \leq 2 \exp \left(- \frac {\epsilon^ {\prime 2} M \sigma^ {2}}{(2 + \epsilon^ {\prime})}\right) \\ \mathbb {P} \left(\left| \frac {\left(\mathbf {W} ^ {\top} \mathbf {f}\right) ^ {\top} \left(\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}\right)}{M \sigma^ {2}} - \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime} \right| > \epsilon^ {\prime}\right) \leq 2 \exp \left(- \frac {\epsilon^ {\prime 2} M \sigma^ {2}}{\left(2 + \epsilon^ {\prime}\right)}\right). \tag {8} \\ \end{array}
$$
This bound indicates that as the dimension of the projections increases, it is more likely for the inner product of any two vectors and their projections to be distinct. In other words, it is increasingly unlikely for the inner product of two vectors and their projections to be equal as  $M$  increases.
2. As  $M$  increases, it is more likely for the inner product of any two randomly projected instances to be distinct (i.e. the inner products in the projected space are more likely to be larger than some constant). That is because, using Markov's inequality, for a larger  $M$  it is easier to choose larger  $\epsilon_{2}$  that satisfies
$$
\mathbb {P} \left(| (\mathbf {W} ^ {\top} \mathbf {f}) ^ {\top} (\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}) | \geq \epsilon_ {2}\right) \leq \frac {M \sigma^ {2}}{\epsilon_ {2}}. \tag {9}
$$
B.3 Connection to least squares
The score of Eqn. (5) can be written in matrix form as
$$
\mathbf {y} _ {\text {t e s t}} = \mathbf {h} _ {\text {t e s t}} (\mathbf {G} + \lambda \mathbf {I}) ^ {- 1} \mathbf {C}, \tag {10}
$$
where  $\mathbf{h}_{\mathrm{test}} := \phi(\mathbf{f}_{\mathrm{test}}^{\top} \mathbf{W})$  are the randomly projected activations following element-wise nonlinear activation  $\phi(\cdot)$ . The form  $\mathbf{W}_0 := (\mathbf{G} + \lambda \mathbf{I})^{-1} \mathbf{C}$  arises in fundamental theory of least squares regression. We can write this as
$$
\mathbf {W} _ {o} = (\mathbf {G} + \lambda \mathbf {I}) ^ {- 1} \mathbf {H} \mathbf {Y} _ {\text {t r a i n}} \tag {11}
$$
$$
= \left(\mathbf {H} \mathbf {H} ^ {\top} + \lambda \mathbf {I}\right) ^ {- 1} \mathbf {H} \mathbf {Y} _ {\text {t r a i n}}, \tag {12}
$$
where  $\mathbf{Y}_{\mathrm{train}}$  is a  $N\times K$  one-hot encoded target matrix, and  $\mathbf{H}$  is as defined in Section 4.3. Eqn. (12) is well known [35, Eqn (7.33), p. 226] to be the minimum mean squared error solution to the  $l_{2}$  regularized set of equations defined by
$$
\mathbf {Y} _ {\text {t r a i n}} ^ {\top} = \mathbf {W} _ {\mathrm {o}} ^ {\top} \mathbf {H}. \tag {13}
$$
For parameter  $\lambda \geq 0$ , this is usually expressed mathematically as
$$
\mathbf {W} _ {\mathrm {o}} = \operatorname {a r g m i n} _ {\mathbf {W}} \left(\left\| \mathbf {Y} _ {\text {t r a i n}} ^ {\top} - \mathbf {W} ^ {\top} \mathbf {H} \right\| _ {2} ^ {2} + \lambda \left\| \mathbf {W} \right\| _ {2} ^ {2}\right). \tag {14}
$$
Consequently, when not using CL, optimizing a linear output head using SGD with mean square error loss and weight decay equal to  $\lambda$  will produce a loss lower bounded by that achieved by directly computing  $\mathbf{W}_{\mathrm{o}}$ .
B.4 Connection to Linear Discriminant Analysis, Mahalanobis distance and ZCA whitening
There are two reasons why we use Eqn. (2) for Algorithm 1 rather than utilize LDA. First and foremost, is the fact that our formulation is mean-square-error optimal, as per Section B.3. Second, use of the inverted Gram matrix results in two convenient simplifications compared with LDA: (i) the simple form of Eqn. (2) can be used for inference instead of a form in which biases calculated from class-prototypes are required, and (ii) accumulation of updates to the Gram matrix and class-prototypes during the CL process as in Eqn. (4) are more efficient than using a covariance matrix.
We now work through the theory that leads to these conclusions. The insight gained is that they show that using Eqn. (2) is equivalent to learning a linear classifier optimized with a mean square error loss function and  $l_{2}$  regularization, applied to the feature vectors of the training set. These derivations apply in both a CL and non-CL context. For CL, the key is to realise that CPs and second-order statistics can be accumulated identically for the same overall set of data, regardless of the sequence in which it arrives for training.
B.4.1 Preliminaries: Class-Prototypes
Class-prototypes for CL after task  $T$  can be defined as
$$
\bar {\mathbf {c}} _ {y} = \frac {1}{n _ {y}} \sum_ {t = 1} ^ {T} \sum_ {n = 1} ^ {N _ {t}} \mathbf {h} _ {t, n} \mathcal {I} _ {t, n} \quad y = 1, \dots , K, \tag {15}
$$
where  $\mathcal{I}_{t,n}$  is an indicator function with value 1 if the  $n$ -th training sample in the  $t$ -th task is in class  $y$  and zero otherwise,  $n_y$  is the number of training samples in each class and  $\mathbf{h}_{t,n}$  is an  $M$ -dimensional projected and activated feature vector. For RanPAC we drop the mean and use
$$
\mathbf {c} _ {y} = \sum_ {t = 1} ^ {T} \sum_ {n = 1} ^ {N _ {t}} \mathbf {h} _ {t, n} \mathcal {I} _ {t, n} \quad y = 1, \dots , K. \tag {16}
$$
For Nearest Class Mean (NCM) classifiers, and a cosine similarity metric as used by [65], the score for argmax classification is
$$
s _ {y} = \frac {\mathbf {f} _ {\text {t e s t}} ^ {\top} \overline {{\mathbf {c}}} _ {y}}{| | \mathbf {f} _ {\text {t e s t}} | | \cdot | | \overline {{\mathbf {c}}} _ {y} | |}, \quad y = 1, \dots , K, \tag {17}
$$
where the  $\overline{\mathbf{c}}_y$  are calculated from feature vectors  $\mathbf{f}_{t,n}$  rather than  $\mathbf{h}_{t,n}$ . These cosine similarities depend only on first order feature statistics, i.e. the mean feature vectors for each class. In contrast, LDA and our approach use second-order statistics i.e. correlations between features within the feature vectors.
B.4.2 Relationship to LDA and Mahalanobis distance
The form of Eqn. (2) resembles Linear Discriminant Analysis (LDA) classification [35, Eqn. (4.38), p. 104]. For LDA, the score from which the predicted class for  $\mathbf{f}_{\mathrm{test}}$  is chosen is commonly expressed in a weighting and bias form
$$
\begin{array}{l} \psi_ {y} = \mathbf {f} _ {\text {t e s t}} \mathbf {a} + \mathbf {b} (18) \\ = \mathbf {f} _ {\text {t e s t}} ^ {\top} \mathbf {S} ^ {- 1} \bar {\mathbf {c}} _ {y} - 0. 5 \bar {\mathbf {c}} _ {y} ^ {\top} \mathbf {S} ^ {- 1} \bar {\mathbf {c}} _ {y} + \log (\pi_ {y}), \quad y = 1, \dots , K, (19) \\ \end{array}
$$
where  $\mathbf{S}$  is the  $M\times M$  covariance matrix for the  $M$  dimensional feature vectors and  $\pi_y$  is the frequency of class  $y$ .
Finding the maximum  $\psi_y$  is equivalent to a minimization involving the Mahalanobis distance,  $d_M := \sqrt{(\mathbf{f}_{\mathrm{test}} - \bar{\mathbf{c}}_y)^\top \mathbf{S}^{-1}(\mathbf{f}_{\mathrm{test}} - \bar{\mathbf{c}}_y)}$ , between a test vector and the CPs, i.e. minimizing
$$
\begin{array}{l} \hat {\psi} _ {y} = d _ {M} ^ {2} - \log \left(\pi_ {y} ^ {2}\right) (20) \\ = \left(\mathbf {f} _ {\text {t e s t}} - \bar {\mathbf {c}} _ {y}\right) ^ {\top} \mathbf {S} ^ {- 1} \left(\mathbf {f} _ {\text {t e s t}} - \bar {\mathbf {c}} _ {y}\right) - \log \left(\pi_ {y} ^ {2}\right) \quad y = 1, \dots . K. (21) \\ \end{array}
$$
This form highlights that if all classes are equiprobable, minimizing the Mahalanobis distance suffices for LDA classification.
B.4.3 Relationship to ZCA whitening
We now consider ZCA whitening [23]. This process linearly transforms data  $\mathbf{H}$  to  $\mathbf{D}_{\mathbf{Z}}\mathbf{H}$  using the Mahalanobis transform,  $\mathbf{D}_{\mathbf{Z}} = \mathbf{S}^{-0.5}$ . Substituting for  $\mathbf{S}$  in Eqn. (21) gives
$$
\begin{array}{l} \hat {\psi} _ {y} = \left(\mathbf {D} _ {\mathrm {Z}} \left(\mathbf {f} _ {\text {t e s t}} - \bar {\mathbf {c}} _ {y}\right)\right) ^ {\top} \left(\mathbf {D} _ {\mathrm {Z}} \left(\mathbf {f} _ {\text {t e s t}} - \bar {\mathbf {c}} _ {y}\right)\right) - \log \left(\pi_ {y} ^ {2}\right) (22) \\ = \left\| \left(\mathbf {D} _ {\mathbf {Z}} \left(\mathbf {f} _ {\text {t e s t}} - \bar {\mathbf {c}} _ {y}\right) \right\| _ {2} ^ {2} - \log \left(\pi_ {y} ^ {2}\right), \quad y = 1, \dots , K. \right. (23) \\ \end{array}
$$
This form implies that if all classes are equiprobable, LDA classification is equivalent to minimizing Euclidean distance following ZCA whitening of both a test sample vector, and all CPs.
B.4.4 Decorating Class-prototypes
The score in Eqn. (2) can be derived in a manner that highlights the similarities and differences to the Mahalanobis transform. Here we use the randomly projected features  $\mathbf{h}$  instead of extracted features  $\mathbf{f}$ . We choose a linear transform matrix  $\mathbf{D} \in \mathbb{R}^{M \times M}$  that converts the Gram matrix,  $\mathbf{G} = \mathbf{H}\mathbf{H}^{\top}$ , to the identity matrix, i.e.  $\mathbf{D}$  must satisfy
$$
(\mathbf {D H}) (\mathbf {D H}) ^ {\top} = \mathbf {I} _ {M \times M}. \tag {24}
$$
It is easy to see that  $\mathbf{D}^{\top}\mathbf{D} = (\mathbf{H}\mathbf{H}^{\top})^{-1} = \mathbf{G}^{-1}$ . Next, treating test samples and class prototypes as originating from the same distribution as  $\mathbf{H}$ , we can consider the dot products
$$
\begin{array}{l} \hat {s} _ {y} = \left(\mathbf {D h} _ {\text {t e s t}}\right) ^ {\top} (\mathbf {D c} _ {y}) \\ = \mathbf {h} _ {\text {t e s t}} ^ {\top} \mathbf {G} ^ {- 1} \mathbf {c} _ {y}, \quad y = 1, \dots , K. \tag {25} \\ \end{array}
$$
Hence, the same similarities arise as those derivable from the minimum mean square error formulation of Eqn. (14). When compared with the Mahalanobis transform,  $\mathbf{D}_{\mathrm{Z}} = \mathbf{S}^{-0.5}$ , the difference here is that the Gram matrix becomes equal to the identity rather than the covariance matrix.
LDA is the same as our method if all classes are equiprobable and  $\mathbf{G} = \mathbf{S}$ , which happens if all  $M$  features have a mean of zero, which is not true in general.
Appendix C Training and implementation
C.1 Optimizing ridge regression parameter in Algorithm 1
With reference to the final step in Algorithm 1, we optimized  $\lambda$  as follows. For each task,  $t$ , in Phase 2, the training data for that task was randomly split in the ratio 80:20. We parameter swept over 17 orders of magnitude, namely  $\lambda \in \{10^{-8}, 10^{-7}, \dots, 10^{8}\}$  and for each value of  $\lambda$  used  $\mathbf{C}$  and  $\mathbf{G}$  updated with only the first  $80\%$  of the training data for task  $t$  to then calculate  $\mathbf{W}_{\mathrm{o}} = (\mathbf{G} + \lambda \mathbf{I})^{-1} \mathbf{C}$ .
We then calculated the mean square error between targets and the set of predictions of the form  $\mathbf{h}^{\top}\mathbf{W}_{\mathrm{o}}$  for the remaining  $20\%$  of the training data. We then chose the value of  $\lambda$  that minimized the mean square error on this  $20\%$  split. Hence,  $\lambda$  is updated after every task, and computed in a manner compatible with CL, i.e. without access to data from previous training tasks. It is worth noting that optimizing  $\lambda$  to a value between orders of magnitude will potentially slightly boost accuracies. Note also that choosing  $\lambda$  only for data from the current task may not be optimal relative to non-CL learning on the same data, in which case the obvious difference would be to optimize  $\lambda$  on a subset of training data from the entire training set.
For Phase 2 in Algorithm 1, the training data was used as follows. For each sample, features were extracted from a frozen pretrained model, in order to update the  $\mathbf{G}$  and  $\mathbf{C}$  matrices. We then computed  $\mathbf{W}_{\mathrm{o}}$  using matrix inversion and multiplication. Hence, no SGD based weight updates are required in Phase 2.
For Phase 1 in Algorithm 1, we used SGD to train the parameters of PETL methods, namely AdaptFormer [6], SSF [28], and VPT [21]. For each of these, we used batch sizes of 48, a learning rate of 0.01, weight decay of 0.0005, momentum of 0.9, and a cosine annealing schedule that finishes with a learning rate of 0. Generally we trained for 20 epochs, but in some experiments reduced to fewer epochs if overfitting was clear. When using these methods, softmax and cross-entropy loss was used. The number of classes was equal to the number in the first task, i.e.  $N_{1}$ . The resulting trained weights and head were discarded prior to commencing Phase 2 of Algorithm 1.
For reported data in Table 1 using linear probes we used batch sizes of 128, a learning rate of 0.01 in the classification head, weight decay of 0.0005, momentum of 0.9 and training for 30 epochs. For full fine-tuning (Table 2), we used the same settings, but additionally used a learning rate in the body (the pre-trained weights of the ViT backbone) of 0.0001. We used a fixed learning rate schedule, with no reductions in learning rate. We found for fine-tuning that the learning rate lower in the body than the head was essential for best performance.
Data augmentation during training for all datasets included random resizing then cropping to  $224 \times 224$  pixels, and random horizontal flips. For inference, images are resized to short side 256 and then center-cropped to  $224 \times 224$  for all datasets except CIFAR100, which are simply resized from the original  $32 \times 32$  to  $224 \times 224$ .
Given our primary comparison is with results from [65], we use the same seed for our main experiments, i.e. a seed value of 1993. This enables us to obtain identical results as [65] in our ablations. However, we note that we use Average Accuracy as our primary metric, whereas [65] in their public repository calculate overall accuracy after each task which can be slightly different to Average Accuracy. For investigation of variability in Section F, we also use seeds 1994 and 1995.
C.3 Training and Inference Speed
The speed for inference with RanPAC is negligibly different to the speed of the original pre-trained network, because both the RP layer and the output linear classification head (comprised from decorrelated class prototypes) are implemented as simple fully-connected layers on top of the underlying network. For training, Phase 1 trains PETL parameters using SGD for 20 epochs, on  $(1 / T)$ 'th of the training set, so is much faster than joint training. Phase 2 is generally only slightly slower than running all training data through the network in inference mode, because the backbone is frozen. The slowest part is the inversions of the Gram matrix, during selection of  $\lambda$ , but even for  $M = 10000$ , this is in the order of 1 minute per task on a CPU, which can be easily optimized further if needed. Our view is that the efficiency and simplicity of our approach compared to the alternatives is very strong.
All experiments were conducted on a single PC running Ubuntu 22.04.2 LTS, with 32 GB of RAM, and Intel Core i9-13900KF x32 processor. Acceleration was provided by a single NVIDIA GeForce 4090 GPU.
Appendix D Parameter-Efficient Transfer Learning (PETL) methods
We experiment with the same three methods as [65], i.e. AdaptFormer [6], SSF [28], and VPT [21]. Details can be found in [65]. For VPT, we use the deep version, with prompt length 5. For AdaptFormer, we use the same settings as in [65], i.e. with projected dimension equal to 64.
Appendix E Datasets
E.1 Class Incremental Learning (CIL) Datasets
The seven CIL datasets we use are summarised in Table A2. For Imagenet-A, CUB, Om-nibenchmark and VTAB, we used specific train-validation splits defined and outlined in detail by [65]. Those four datasets, plus Imagenet-R (created by [56]) were downloaded from links provided at https://github.com/zhoudw-zdw/RevisitingCIL. CIFAR100 was accessed through torchvision. Stanford cars was downloaded from https://ai.stanford.edu/~jkrause/cars/car_dataset.html.
For Stanford Cars and  $T = 10$ , we use 16 classes in  $t = 1$  and 20 in the 9 subsequent tasks. It is interesting to note that VTAB has characteristics of both CIL and DIL. Unlike the three DIL datasets we use, VTAB introduces new disjoint sets of classes in each of 5 tasks that originate in different domains. For this reason we use only  $T = 5$  for VTAB, whereas we explore  $T = 5$ ,  $T = 10$  and  $T = 20$  for the other CIL datasets.