RanPAC: Random Projections and Pre-trained Models for Continual Learning
Authors: Mark McDonnell
Dong Gong
Amin Parvaneh
Ehsan Abbasnejad
Anton van den Hengel
Venue: NeurIPS 2023 poster (2023)
Keywords: continual learning
class incremental learning
domain incremental learning
pre-trained models
parameter-efficient transfer learning
TL;DR: Our algorithm for continual learning leveraging pre-trained models such as ViTs outperforms past methods by large margins by introducing a nonlinear random projection of extracted features prior to the classifier head.

Abstract: Continual learning (CL) aims to incrementally learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. Most CL works focus on tackling catastrophic forgetting under a learning-from-scratch paradigm. However, with the increasing prominence of foundation models, pre-trained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose a concise and effective approach for CL with pre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training-free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject a frozen Random Projection layer with nonlinear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensionality, providing enhanced linear separability for class-prototype-based CL. We also demonstrate the importance of decorrelating the class-prototypes to reduce the distribution disparity when using pre-trained representations. These techniques prove to be effective and circumvent the problem of forgetting for both class- and domain-incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between 20% and 62% on seven class-incremental benchmark datasets, despite not using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast continual learning has not hitherto been fully tapped. Code is available at https://github.com/RanPAC/RanPAC.

RanPAC: Random Projections and Pre-trained Models for Continual Learning
Mark D. McDonnell', Dong Gong?, Amin Parveneh',
Ehsan Abbasnejad and Anton van den Hengel'
1 Australian Institute for Machine Learning, The University of Adelaide 2 Schooo of Computer Science and Engineering, University of New South Wales
mark. markkmccdomellladeelaadd edu.au, dong gongdunsw............ au, amin. www.aaaaaaaaadelaieeeeee.e edu ehsan abbasne addadelaide jabaseejd@@ddlaaiie anton. wandennhengel@dddlll..eee edu. au
edu. au,
Abstract
Continual learning (CL) aims incrementally learn different tasks (such as clas- sification) in a non-stationary data stream without forgetting old ones. Most CL works focus tackling catastrophic forgetting under learning-from-serate paradigm. However, with the increasing prominence of foundation models, pre- trained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose concise and effective approach for CL with gre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training- free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject frozen Random Projection layer with nonlin- ear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensional- ity, providing enhanced linear separability for class-proootypebbsse CL. We also demonstrate the importance of decorrelating the class- protss-pttotppss to reduce the distribution disparity when using pre- pre-trained representations. These techniques prove to be effective and circumvent the problem forgetting for both class- and domain- incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between 20% and 62% seven class-incremental benchmark datasets, despite using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast continual learning has not hitherto been fully tapped. Code is available at htops:///gtthnn com/ com/RanPaa RanPAC/ CRRanAAC
Introduction
Continual Learning (CL) is the subfield of machine learning within which models must Iearn from a distribution of training samples and/or supervision signals that change over time (often divided into distinct set of T episodes//asks//tages) while remaining performant on anything learned previously during training [51 (3). Traditional training methods do not work well for CL because parameter updates become biased to newer samples, overwriting what was learned previously. Moreover, training on sequential disjoint sets of data means there is no opportunity to learn differences between samples from different stages 48 These effects are often characterised as 'catastrophic forgetting' (51).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Although many methods for CL have been proposed [5T 53 2 60), most focus on models that need to be trained from scratch, with resulting performance falling short of that achievable by non-CI. alternatives on the same datasets. Although valid use cases for training from scratch will always exist, the new era of large foundation models has led to growing interest in combining CL with the advantages of powerful pre-trained models, namely the assumption that a strong generic feature-cxtractor can be adapted using fine-tuning to any number of downstream tasks.
Although forgetting still occurs in CL with such transfer-learning 157
(whether using Efficient Transfer Learning (PETL) methods
65 63]
conventional fine-tuning (24] or more-recent
such as 6 28, 21]), commencing CL with a powerful Crature-extractio has opened up new ideas for
avoiding forgetting that are unlikely to work when training from scratch. Such new methods have
been applied successfully to pre-trained transformer networks (39 [0]
[5 [56 46 20
65 63 47 [55]
CNNs
15 33
22
59 37 and multi-moda laioon- models
Three main strategies are evident in these proposals (see Section 2 for details): (i) prompting of
transformer networks; (ii) careful selective fine-tuning of a pre-trained model's parameters; and (iii) Class-Prototype (CP) accumulation. Common to all these strategies is the absence of a need for a buffer of rehearsal samples from past tasks, unlike the best CL methods for training models from scratch. Instead, these strategies leverage pre-trained model's strong feature extraction capabilities.
Each strategy has yielded empirically comparable performance when used with the same benchmarks and pre-trained model (e.g. a ViT B/16 transformer network [46] 63 651). It therefore remains open as to what strategy best leverages pre-trained foundation models for CL., in terms of performance on diverse datasets and CL. scenarios, simplicity, and efficiency. However, we note that the CP-based CL strategy is simple apply to both CNNs and transformer networks, whereas prompting methods rely on a prepending learnable prompts to transformer network inputs, Fine-tuning a pre-trained model's parameters requires more resources for training than the other two strategies, while carrying a greater risk of cumulative forgetting over time, thus requiring use of additional CL methodologies.
In this paper, show that the CP strategy has not come close to exhausting its capacity for accuracy. and can achieve standout performance with carefully tailored strategies to enhance the extracted feature representations from the pre-trained models. CP-based methods use only the prototypes obtained by averaging the extracted features to represent each class, subject to a discrepancy with the data distributions in practice. We propose to handle this via training-free frozen random projections and a decorrelation process, both of which bypass the forgetting issue. Specifically, we introduce a Random-Projeccion (RP) layer of frozen untrained weights, with nonlinear activation, between the pre-trained-model's feature layer, and a CP-based output head.
Nonlinear Random Projection (RP) of Fea- tures: The original RP idea (see Figure T has been proposed [4 4 [8] and applied (e.g. [32]
Random Projection
treaianned Net
L M >
Prototype- paselppe-Baedd Classification
multiple times in the past, as a standalone non- continual Iearner. Our two-fold motivation for using the same method for CL differs from the method's origins (outlined in Section (2). Firstly, we show that past CP strategies for
Extracted Features
Nonlinearly Projected Features
CL with pre-trained models are equivalent to the linear models learned following RP in the past, e.g. 43 4 I8 32) Secondly, the frozen mntrained/trainnn-free free weights do not cause for- getting in CL. These two observations suggest using the RP idea alongside CP strategies for CL.
Figure I: The RP method can lead to a represen- tation space with clear class separation. Colored points are 2D I-SNE visualizations for CIFAR-100 classes with features from pre-trained ViT-B/16 transformer network.
However, why should RPs provide any value in- stead of trying to learn layer of fully-connected weights? It is well known that linear models can benefit from transformations that create non- linear feature interactions. Our hypothesis in proposing to use RP followed by nonlinear-activation for CL. therefore, that the transformed set of features is more linearly separable using a CP method
than features directly extracted from a pre- pre-trained model. The past work of [43] 4 [8] 32 leads us to expect that for this be confirmed, RP will need to increase dimensionality from L M > L.

---

Our contributions are summarised as follows:
1. We examine in detail the CP strategy for CL with pre-trained networks, show that it benefits from injecting a Random Projection layer followed by nonlinear activation, and illustrate why. We also analyse why it is important follow the lead of (37 to linearly transform CPs, via decorrelation using second-order feature statistics.
2. We show that random projections are particularly useful when also using PETL methods with first-session training (see Section 2.I). Accuracy with this combination approaches the CL joint training upper bound on some datasets (Table 2 For ViT-B/16 pre-trained models, report the highest-to-dee rehearsal-free CL accuracies on all class-incremental and domain-incremennal datasets we tested on, with large margins when compared to past CP strategies.
3. We highlight the flexibility of our resulting CL algorithm, RanPAC; it works with arbitrary feature vectors (e.g. ViT. ResNet, CLIP), and is applicable to diverse CL scenarios including class-incremental (Section 5) iamaain-nccrmmentll (Section and task-agnostic CL (Appendix
2 Related Work
2.1 Three strategies for CL with strong pre-trained models
Prompting of transformer networks: Using a ViT-B/16 network 191 Learning To Prompt (L2P) [57] and DualPrompt [56] reported large improvements over the best CL methods that do not leverage pre-trained models, by training small pool of prompts that update through the CL process. CODA- Prompt [46). S S-Prompt (55) and PromptFusion [5] then built on these, showing improvements in performance.
Careful fine-tuning of the backbone: SLCA 63 found superior accuracy to prompt strategies by fine-tuning a ViT backbone with a lower learning rate than in a classifier head. However, it was found that use of softmax necessitated introduction of a 'classifier alignment' method, which incurs high memory cost, in the form of a feature covariance matrix for every class. Another example of this strategy used selected fine-tuning of some ViT attention blocks [47), combined with traditional CL method, L2 parameter regularization. Fine-tuning was also applied to the CLIP vision-language model, combined with well-established CL method, LwF (8).
Mlss-Prototyyp (CP) accumulation: Subsequent to L2P, it was pointed out for CL image classi- fiers (20) that comparable performance can be achieved by appending nearest class mean (NCM) classifier to a ViT model's feature outputs (see also (39]). This strategy can be significantly boosted by combining with Parameter- Effficient Transfer Learning (PETL) methods (originally proposed for 12)) trained only on the first CL stage (" ('first-eesiion training') The three PETL methods considered by (65) for transformer networks, and the FiLM method used by (37] for CNNs have in common with the first strategy (prompting) that they require learning of new parameters, but avoid updating any parameters of the backbone pre-trained network. Importantly, (37 also showed that a simple NCM classifier is easily surpassed in accuracy by also accumulating the covariance matrix of embedding features, and learning linear classifier head based on linear discriminant analysis (LDA) [35]. The simple and computationally lightweight algorithm of (37] enables CL to proceed after the first session in perfect manner relative to the union of all training episodes, with the possibility of catastrophic forgetting avoided entirely.
NLP models in non-CI. context [17
to bridge any domain gap 65 37
CPs are well suited to CL generally 42
31 16 and for application to pre-trained models 20 65
37., because when the model from which feature vectors are extracted is frozen, CPs accumulated across T tasks will be identical regardless of the ordering of the tasks. Moreover, their memory cost is low compared with using rehearsal buffer, the strategy integral to many CL methods (53).
2.2 RP method for creating feature interactions
As mentioned, the original non-CL usage of frozen RP layer followed by nonlinear projection as in [8] had different motivations to us, characterized by the following three properties. First, keeping weights frozen removes the with a línear output layer, the computational cost of training them. Second, when combined output weights can be learned by exact numerical computation using all training data simultaneously (see Appendix (B.3) instead of iteratively. Third, nonlinearly activating random projections of randomly projected features is motivated by the
3

---

assumption that nonlinear random interactions between features may be more linearly separable than the original features. Analysis of the special case of pair-wise interactions induced by nonlinearity can be found in [32. and mathematical properties for general nonlinearities (with higher order interactions) have also been discussed extensively, e.g. 4 18.
3 Background
3.1 Continual learning problem setup
We assume the usual supervised CL setup of sequence of T tasks/stages, D= (D1....D) .rr... In each De, a disjoint set training data paired with their corresponding labels is provided for learning. Subsequent stages cannot access older data. We primarily consider both "Class-Incremental Learning' (CIL.) and Poomain-Increment the class labels for each D, are disjoint. One of the challenging aspects of CII. is that, in contrast to Task- laskemncrementll Learning (TIL), the task identity and, consequently, the class subset of each sample unknown during CIL inference [51). In DIL, while all stages typically share the same of classes, there is distribution shift between samples appearing in each stage. For example, D may include photographs and D: images of paintings.
learning' (DIL) protocols [5I] for classification of images. In CII,,
We introduce as the total number of classes considered within tasks, and the number of training samples in each task as N₂ with N := Σ N,. For the n-th unique training sample within task Dr, we use Yt,z as its length K one-hot encoded label and f.n E RR4 to denote features extracted from a frozen pre-trained model. We denote by frest the encoded features for test instance for which we seek to predict labels.
3.2 Class-Prototype strategies for CL with pre-trained models
For CL., using conventional cross-entropy loss by linear probing or fine-tuning the feature representa- tions of a frozen pre-trained model creates risks of task-recency bias [31] and catastrophic forgetting. Benefiting from the high-quality representations of pre-trained model, the most straightforward
(CP) strategy is to use Nearest Class Mean (NCM) classifiers [58] [34] as applied and
investigated by 665 20). CPs for each class usually constructed by averaging the extracted feature vectors over training samples within classes, which we denote for class y as Cy. In inference, the class of test sample is determined by finding the highest similarity between its representation and the set of CPs. For example, (65] use cosine similarity to find the predicted class for a test sample,
fe Cy
Ytest = arg max Sy', By = || I
E((
However, it is also not difficult to go beyond NCM within the same general CL strategy, by leveraging second-order feature statistics (7] For example, (37 finds consistently better CL results with pre-
trained CNNs than NCM using an incremental version [6 [1 of Linear Discriminant Analysis (LDA) classification [35], in which the covariance matrix of the extracted features is continually updated. Under mild assumptions, LDA is equivalent to comparing feature vectors and class- proso-prtoo using Mahalanobis distance (see Appendix B.4), i.e. different to cosine distance used by 655
3.3 CP-based classification using Gram matrix inversion
We will also incrementally calculated second-order feature statistics, but create simplification compared with LDA (see Appendix B.4. by using the Gram matrix of the features, G, and cy (CPs with averaging dropped), to obtain the predicted label
Utest
arg max Sy', Sy =y==f, -
(2)
y y'E(L.....k)
Like LDA, but unlike cosine similarity, this form makes use of a training set to 'calibrate' similarities. This objective has a basis long established theory for least square error predictions of one-hot encoded class labels [35 (see Appendix [B.3). Similar to incremental LDA [3 M. during CL training, we describe in Section how the Gram matrix and the CPs corresponding to c, can easily be updated progressively with each task. Note that Eqn. (2) is expressed in terms of the maximum number of classes after T tasks, K. However, for CII,, it can calculated after completion of tasks t < T with fewer classes than K. For DIL, all K classes are often available in all tasks.

---

The proposed approach and theoretical insights: RanPAC
4.1 Why second order statistics matter for CPs
We show in Section 5 that Eqn. (2) leads to better results than NCM. We attribute this to the fact that raw CPs are often highly correlated between classes, resulting in poorly calibrated cosine similarities, whereas the use of LDA or Eqn (2) mostly removes correlations between CPs, creating better separability between classes. To illustrate these insights,
the example of ViT-B/16 transformer model [9] pre-trained on ImageNet-21K K with its classifier head removed, and data from the well-established 200-class Split Imagenet-R CIL benchmark [56].
For comparison with CP approaches, we jointly trained, on all 200 classes, a linear probe softmax classifier. We treat the weights of the joint probe class-prototypes for this exercise and then find the Pearson correlation coefficients between each pair of prototypes as shown for the first 10 classes in Fig. [ (right). Compared with the linear probe, very high off-diagonal correlations are clearly observed when using NCM, with a mean value more than twice that of the original I K training data treated in the same way. This illustrates the extent of the domain shift for the downstream dataset. However, these correlations are mostly removed when using Eqn (2). Fig. Fig.2 (left) shows that high correlation coefficients coincide with poorly calibrated cosine similarities between class-prototype and training samples, both for true class comparisons (similarities between sample's 's feature vector and the CP for the class label corresponding that sample.) and inter-class comparisons (similarities between a sample's feature vector and the class prototypes for the set of N-1 classes not equal to the sample's ' class label). However, when using Eqn. 2 the result (third row) is to increase the training-sed set accuracy from 64% to 75%, coinciding with reduced overlap between inter- and true-class similarity distributions, and significantly reduced off-diagonal correlation coefficients between CPs. The net result is linear classification weights that are much closer to those produced by the jointly-trained linear probe. These results are consistent with known mathematical relationships between Eqn 2 and decorrelation, which we outline in Appendix B.4.4
4.2 Detailed overview and intuition for Random Projections
CP methods that use raw by for CL assume Gaussian distribution with isotropic covariance. Empirically, we have found that when used with pre-trained models this assumption is invalid (Figure 2}. One can learn a non-linear function (e.g. using SGD training of a neural network) with
1.0
joint Inear probe, trainin Arc 482%
inter class (10000 bins)
true class (30 bins)
0.5
-0.2
0.0
0.2
0.6
0.8
1.0
0123456789
Class-pretypes and cosine distance, training Accm6456
0.5
0.0
-0.2
0.0
0.2
0.6
0.8
1.0
6123456789
Class-protot ypes and Eqn: (2), training Acc-75%
0.5
0.0
0.2
0.0
0.2
0.6
0.8
1.0
0123456789
1.0
prototypes and (W-2000), training Acc-87%
0.5
0.0
-0.2
8.2
0123456789
Training Sample Class-Prototype Similarities
Classifier Head Pearson CCs
Figure 2: Left: Histograms of similarities between class-prototypes and feature vectors from the ViT-B/16 transformer model pre-trained on ImageNet- 1K, for the training samples of the Imagenet-R dataset. Right: Pearson correlation coefficients (CCs) for 10 pairs of dlass-proottppes. Reduced correlations between CPs of different classes (right), coincides with better class separability (left).
5

---

high capacity and non-convex objective to alleviate this gap (e.g.. learning to adapt on the first task as in (65(37) but that requires additional assumptions and architectural choices. In addition for CL. any learning on all tasks carries high risk of catastrophic forgetting. Instead, we take advantage of the observation that the objectives CP strategies principally require firstly feature projections (that could be possibly adjusted for their distribution by a covariance matrix) and secondary the feature prototypes. For the latter, since using large pre-trained models produce features that capture most prominent patterns and attributes, the average is reasonable expected class representative. However, the feature representations may also benefit from nonlinear transformation to enhance linear separability. In this paper. we consider two points: (i) we show a random projection of the features in the embedding space (or their projection to a random basis) to a higher dimensional space leads to distribution that is more likely to be suitable for Gaussian fit (e.g. using Eqn. (2);; and, (ii) when the distribution of representations is nearly Gaussian, we can simply perform a linear regression (or LDA conditional mean of the joint Gaussian distribution).
This assertion is supported by the bottom row of Fig. 2 which shows that the application of an RP layer of size M = 2000 (chosen arbitrarily, for the purpose of illustration) reduces the overlap between the similarity histograms further than otherwise, and shifts the in-class histogram to the right, coinciding with higher accuracy, surpassing that of the joint linear probe, consistent with our findings in Section 5 We note that these RPs can be generated once and then frozen for use in all continual learning stages, enabling simple and efficient algorithm.
We first analyse the inner products of the features obtained from the pre-trained model projected using some random basis vectors (i.e. random projections). We consider M such random vectors all randomly drawn i.i.d from a Gaussian distribution with variance o². The number M defines the dimension of the space the random projections of these features constitute. Considering random projections W E
R- gx for any two given feature vectors f, f' E ER_
we have (see Appendix B.2).
M
M
Ew [( T)T (W f')]
Σ
EW
W
F
Σ Ew [Wo]
Ew [Wo] fT (3)
where W(a) (z) denotes the ith column. That is, the expected inner products of projections of these two features can be decomposed. Now, we have (()) ~ ~NO, I²)), thus Ew(W (e) = Ew W WG)l = 0
M Ew! [W2 (e) Mo We can make two and the second term in Eqn. (3) vanishes. Further, E;
observations (theoretical details are provided in Appendix
1. As M increases, the likelihood that the norm of any projected feature approaching the variance increases. In other words, the projected vectors in higher dimensions almost surely reside on the boundary of the distribution with almost equal distance to the mean (the distribution approaches isotropic Gaussian).
2. As M increases, more likely for angles between two randomly projected instances to be distinct (i.e. the inner products in the projected space are more likely to be larger than some constant).
This discussion can readily be extended to incorporate nonlinearity. The benefit of incorporating nonlinearity is to (i) incorporate interaction terms (54 50), and, (ii) simulate higher dimensional projections that otherwise could be prohibitively large in number. To see the later point, denoting by é the nonlinearity of interest, we have o( W f) = W where f is obtained from a linear expansion using Taylor series and W is the corresponding projections. The Taylor expansion of the nonlinear function & gives rise to higher order interactions between dimensions. Although vectors of interaction terms can be formed directly, as in the methods of [38] 54), this is computationally prohibitive for non-trivial L. Hence, the use of nonlinear projections of the form hest := of (f
W) is a convenient
alternative, as known to work effectively in non CL context 43 18
4.3 Random projection for continual learning
Using the random projection discussed above, with d(-) as an element- nonlinear activation function, given feature sample f.n, we obtain length M representations for CL training in each task,
h := (f fn W) (Fig. D.. For inference, h := = W) is used in sy in Eqn. ( instead of fusst. We define H as an M x N matrix in which columns are formed from all hgn and for convenience refer to only the final H after all N samples are used. We now have an M x M Gram

---

matrix for the features, G = HH The random projections discussed above are sampled once and left frozen throughout all CL stages.
Like the covariance matrix updates in streaming LDA applied to CL [1 37 variables are updated either for individual samples, or one entire CL. stage, Dr, at time. We introduce matrix C to denote the concatenated column vectors of all the cy. Rather than covariance, S, we update the Gram matrix, and the CPs in C (the concatenation of Cy s) iteratively. This can be achieved either one task at time, or one sample at time, since we can express G and C as summations over outer products as
T
N
T
N2
G=
Σ 1=1n=1
h @ h
C=
hen
@ Vt.n-
1=1
Both C and G will be invariant to the sequence in which the entirety of N training samples are presented, a property ideal for CL algorithms.
The mentioned origins of Eqn. (2) in least squares theory is of practical use: we find it works best to use ridge regression [35, and calculate the l2 regularized inverse, (G AI)- where denotes the identity matrix. This is achieved methodically using poossvvlidationnsse see Appendix [ The revised score for CL can be rewritten as
8y = (CG 17(G-II)-'4
(5)
In matrix form the scores can be expressed predictions for each class label as Ypred hteet W.. Different to streaming LDA, our approach has the benefits of (i) removing the need for bias calcula- tions from the score; (ii) updating the Gram matrix instead of the covariance avoids outer products of means; and (iii) the form W. = =G + -T)= C arises as a closed form solution for mean-square-erro
loss with lz-regularization (see Appendix BB.3). in contrast to NCM, where no such theoretical result exists. Phase 2 in Algorithm 1 summarises the above CL calculations.
Application of Eqn. (5) is superficially similar to AdaptMLP modules (6), but instead of a bottleneck layer, we expand to dimensionality M > L, since past applications found this to be necessary to compensate for W being random rather than learned [43 18 32]. As discussed in the introduction, the use of a random and training- free weights layer is particularly well suited to CL.
The value of transforming the original features to nonlinear random projections is illustrated in Fig. 3 Features for the T = 10 split ImageNet-R CII. dataset were extracted from a pre-trained ViT-B/16 99] network and Eqn. 5 applied after each of the T = 10 tasks. Fig. 3] shows the typical CL average accuracy trend, whereby accuracy falls as more tasks are added. When a nonlinear activation, ((-) is used (e.g. ReLU or squaring), performance improves as M increases, but when the nonlinear activation is omitted, accuracy is no better than not using RP, even with M very large. On the other hand, if dimensionality is reduced without nonlinearity (in this case from 768 to 500), then performance drops below the No RP case, highlighting that if RPs in this application create dimensionality reduction, it leads to poor performance.
Fig. b)) casts light on why nonlinearity is important. We use only the first 100 extracted features per sample and compare application of Eqn. ( to raw feature vectors (black) and to pair-wise interaction terms, formed from the flattened cross- pross-product of each extracted feature vector (blue trace). Use of the former significantly outperforms the latter. However, when Eqn. (5) is used instead (red trace), the drop in performance compared with flattened cross products is relatively small. Although this suggests exhaustively creating products of features instead of RPs, this is computationally infeasible. As an alternative, RP is convenient and computationally cheap means to create nonlinear feature interactions that enhance linear separability, with particular value in CL with pre-trat pre-trained models.
4.4 Combining with parameter-efffcient transfer learning and first-session adaptation
Use of an RP layer has the benefit of being model agnostic, c.g. it can be applied any feature extractor. As we show, it can be applied orthogonally to PETL methods. PETL is very appealing for CL. particularly approaches that do not alter any learned parameters of the original pre-trained model, such as [10, 28]. We combine RP with a PETL method trained using CL-compatible 'first-session' training, as carried out by 665 37 This means training PETL parameters only on the first CL task, D1, and then freezing them thereafter (see Phase of Algorithm 1). The rationale is that the training data and labels in the first task may be more representative of the downstream dataset than that used to train the original pre-trained model. If new dataset drifts considerably, e.g, as in DIL, the
7

---

Average Accuracy A, aatter task (9%
No RP
Flattened cross products
No O. M-500
O~ReLUO, M=5000 No 4.
No 9. M=15000
05
ReLUD) M-15000
M=10000
80
ReLUO), M=5000
M-10000
70-
85
Task ID,
Task ID,
Figure 3: Impact of RP compared to alternatives. (a) Using only Phase 2 of Algorithm 1, show average accuracy (see Results) after each of = 10 tasks in the split ImageNet-R dataset. The data illustrates the value of nonlinearity combined with large numbers of RPs, M. (b) Illustration that interaction terms created from feature vectors extracted from frozen pre-trained models contain important information that can be mostly recovered when RP and nonlinearity used.
benefits of PETI. may be reduced because of this choice. But the assumption is that the domain gap between the pre-training dataset and the new dataset significant enough still provide benefit. First-session training of PETL parameters requires a temporary linear output layer, learned using SGD with crtsspyntropy loss and softmax and only Kj classes, which is discarded prior Phase 2.
For transformer nets, we experiment with the same three methods as [65]. i.c. AdaptFormer (6], SSF 228 and VPT [2I. For details of these methods see the cited references and Appendix D
Unlike 65 we do not concatenate adapted and unadapted features, as we found this added minimal
value when using RP.
4.5
Memory usage of RP layer
The RP layer increases the total memory re- quired by the class prototypes by factor of
Algorithm RanPAC Training
1 1 (M /) L, while an additional LM parameters are frozen and untrainable. For
Input: Sequence of tasks, D (DI,... Dr), pre-trained model, and PETL method Phase 1: 'First-session' PETL adaptation.
typical values of = 768 and K = 200, the injection of an RP layer of size M = 10000 therefore creates ~ times the number of trainable parameters, and adds 10M non- trainable parameters. Although this is sig- nificant number of new parameters, it is still small compared with the overall size of the VIT-B/16 model, with its 84 millions param- eters. Moreover, the weights of W can be bipolar instead of Gaussian and if stored ing a single bit for each element contribute only tiny fraction additional model mem- ory. During training only, we also require updating an M x AM Gram matrix, which is smaller than the K. (L x L) covariance ma- trices the SLCA fine-tuning approach 63
for sample I,..., .1 in 2) do
Extract feature vector, f.... Use in SGD training of PETL parameters end for
Phase 2: Continual Learning with RPs. Create frozen L x M RP weights, W~N((01) for task t 1, ... do for sample n = 1,.... N₂ in Dj do
Extract feature vector, fr..
Apply h o(f,,, W)
Update G and C matrices using Eqn. 44 end for Optimize À (A. and compute (G AI) end for
M < L.L. L2P (57). DualPrompt [56) and ADaM 655 each use -0.3M-0.5M trainable parame-
ters. For K = 200 classes and M = 10000, RanPAC uses a lot more (~222..5M, depending on the PETL method), comparable to CODA -rommpt 46 RanPAC also uses 10M untrained parameters, but we highlight that these are not trainable. Moreover, M need not be as high as 10000 (Table A5).
5 Experiments
We have applied Algorithm to both CII. and DII. benchmarks. For the pre-trained model, we experiment mainly with two ViT-B/16 models 99 per [65]: one self-supervised on ImageNet-2IK,

---

and another with supervised fine-tuning on ImageNet- IK. Comparisons are made using a standard CL metric, Average Accuracy 33), which is defined as A, - Re,,. where Re,; are classification accuracies on the i-th task, following training on the t-th task. We report final accuracy, Ar, in the main paper, with analysis of each A, and R left for Appendix [] along with forgetting metrics and analysis of variability across seeds. Appendix also shows that our method works well with ResNet and CLIP backbones.
We use split datasets previously used for CIL DIL (see citations in Tables and 3
details provided in Appendix E, We use M = 10000 in Algorithm except where stated; investigation of scaling with M is in Appendix F.5] All listed 'Joint' results are non-CL training of comparisons on the entirety of D, using entropy loss and softmax.
Key indicative results for CIL shown in Table for T = 10, with equally sized stages (except VTAB which is 7 = 5, identical to (65)) For T T= 5 and 7 = 20, see Appendix [F] For each dataset, our best method surpasses the accuracy of prompting methods and the CP methods of [20] and [65], by large margins. Ablations of Algorithm 1 listed in Table show that inclusion of our RP layer results in error-rate reductions of between 11% and 28% when PETL is used. The gain is reduced otherwise, but is 2 8%, except for VTAB. Table a highlights the limitations of NCM.
Method
CIFARI00
IN-R
IN-A
CUB
OB
VTAB
Cars
Joint linear probe
87.9%
72.0%
56.6%
88.7%
78.5%
86.7%
51.7%
L2P [57]
84.6%
72.4%
42.5%
65.2%
64.7%
77.1%
38.2%
DualPrompt [6
84.1%
71.0%
45.4%
68.5%
65.5%
81.2%
40.1%*
CODA- 46
86.3%
75.5%
44.5%
79.5%
68.7%
87.4%
43.2%
ADaM [65]
87.6%
72.3%
52.6%
87.1%
74.3%
84.3%
41.4**
Ours (Algorithm 1)
92.2%
77.9%
62.4%
90.3%
79.9%
92.2%
77.5%
Rel. ER vs ADaMi
4 36%
20%
21%
25%
4 22%
4 53%
62%
Ablations
No RPs
90.6%
73.9%
57.7%
86.6%
74.3%
90.0%
73.0%
No Phase
89.0%
71.8%
58.2%
88.7%
78.2%
92.2%
66.0%
No RPs or Phase
87.0%
67.6%
54.4%
86.8%
73.1%
91.9%
62.2%
NCM with Phase 1
87.8%
70.1%
49.7%
85.4%
73.4%
88.2%
40.5%
NCM only
83.4%
61.2%
49.3%
85.1%
73.1%
88.4%
37.7%
Rel. ER (inc. Phase 1)
17%
15%
11%
28%
22%
1
22%
17%
Rel. ER (no Phase 1)
15%
13%
8%
14%
19%
4%
10%
Table 1: Comparison of prompting and CP strategies for CIL: 'Rel. ER' is Relative Error
Rate. No rehearsal buffer is used for any method listed. IN-R is ImageNet-R [66. The versions of
ImageNet-A (IN-A), OmniBenchmark (OB), CUB and VTAB are those defined by [65]. L.PP and
DualPrompt results are those stated within 655 except (Cars) from (63] CODA-Promphl is directly
from [46] for CIFAR 100 and IN-R, and from our own experiments with the PILOT toolbox [49] otherwise. For ADaM, we have used the best reported by (65 across all PETL methods, except for (Cars) which our own. Remaining results are our own implementation. PromptFusion (5) is omitted due to requiring a rehearsal buffer. Using NCM method, (20) achieved 83. 73.7% for CIFAR 100 and 64.3% for Imagenet-R. Ablations of Algorithm 1: NCM uses cosine similarity instcad of Equ. ( RP means Random Projections layer; Phase is defined within Algorithm 1.
Algorithm 1 surpasses the jointly trained linear probe, on all seven datasets; Table I indicates that for all datasets this can be achieved by at least one method that adds additional weights to the pre-trained model. However, NCM alone cannot match the joint linear probe. As shown in Table 2 Algorithm 1 also does better than tine-tuning strategies for CIL [47 633.. This is tabulated separately from Table G because fine-tuning has major downsides as mentioned in the Introduction. Table 2 also shows Algorithm reaches within 2% raw accuracy of the best joint fine-tuning accuracy on three datasets, with larger gap the two ImageNet variants and Cars.
Results for DIL datasets, and corresponding ablations of Algorithm are listed in Table 3 CORe50 is T = 8 stages with 50 classes [29]. HDDB-Hard is T = 5 with 2 classes [27] and DomainNet is = 6 with 345 classes [4) (further details are provided in Appendix D.. The same trends as for CIL can be observed, i.e. better performance than prompting strategies and highlighting of the value of the RP layer. For DomainNet, the value of RP particularly strong, while including PETL adds little value, consistent with the nature of DIL, in which different CL stages originate in different domains.

---

Method
CIFARI00
IN-R
IN-A
CUB
OB
VTAB
Cars
Joint full fine-tuning tuning
93.8%
86.6%
70.8%
90.5%
83.8%
92.0%
86.9%
SLCA [63
91.3%
77.0%
84.7%
67.7%
Ours (Algorithm 1)
92.2%
78.1%
61.8%
90.3%
79.9%
92.6%
77.7%
Table 2: Comparison with fine-tuning strategies for CIL. Results for SLCA are directly from [63]
'Joint' means full fine-tuning of the entire pre-trained ViT network, i.e. a continual learning upper bound. Notes: L2 47) achieved 76. 1% on Imagenet-R, but reported no other ViT results. Cells containing indicates SLCA did not report results, and no codebase is available yet. Superscript * the number of CUB training samples used by [63 is smaller than that defined by [65]. which we use.
Method
CORe50
CDDB-Hard
DomainNet
L2P [37]
78.3%
61.3%
40.2%
S-iPrompts .55
83.
74.5%
50.6%
ADaM
92.0%
70.7%
50.3%
Ours (Algorithm 1)
96.7%
86 2% (M = 5K)
66.6%
S-liPrompts [55] (not ViT B/16)
89.1%
88.7%
67.8%
Ablations
No RPs
94.8%
84.2%
55.2%
No Phase
94.3%
81.6% (M 5K)
64.7%
No RPs or Phase
92.4%
80.7%
54.1%
NCM with Phase 1
91.4%
73.2%
51.3%
NCM only
80.4%
69.8%
46.4%
Rel. ER (inc. Phase I)
37%
4 13%
4 25%
Rel. ER (no Phase 1)
25%
J. 5%
1 23%
Table 3: Comparison with prompting and CP strategies for DIL: No rehearsal buffer is used for any method. The first four rows use VIT B/16 networks. S-liPrompts uses vision-language prompting with pre-trained CLIP model, and cannot be directly compared to ours. For CORe50, 83.2% was reported for simple CP strategy of [20] and 85.4% for FSA-FiLM with pre-trained convnets [37] For
DomainNet we and [55] use 365 classes, different to [46]. ADaM results are based on code from 65 I.2P and S- S-prompts are taken from (55) Ablations: Explanations are the same as in Table 1
Conclusion
We have demonstrated that feature representations extracted from pre-trained foundation models such as ViT-B/16 have not previously achieved their full potential for continual learning. Application of our simple and rehearsal-free class-prototype strategy, RanPAC, results in significantly reduced error rates on diverse CL benchmark datasets, without risk of forgetting in the pre-trained model. These findings highlight the benefits of CP strategies for CL with pre-trained models.
The value of Eqs 4 and (5) are completely reliant on supply of a good generic feature extractor. For this reason, they unlikely be powerful if used in CL methods that train networks from scratch. However, it is possible that existing CL. methods that utilise self-supervised learning, otherwise create good feature extractor backbones could leverage similar approaches to ours for downstream CL tasks. As discussed in Section 4.5) RanPAC uses additional parameters compared to methods like L2P. However, this is arguably worth trading-off for the simplicity of implementation and low-cost training.
Limitations:
Future Work: Examples in Appendix shows that method works well with other CL protocols including: (i) aask-agno,tic, i.c. CL without task boundaries during training (e.g. Gaussian scheduled CIFAR- 100), (ii) use of a non encoode target, e.g. language embeddings in the CLIP model, and (iii) regression targets, which requires extending the conceptualisation of class tlass-prototypes to generic feature prototypes. Each of these has a lot of potential extensions and room for exploration. Other interesting experiments that we leave for future work include investigation of combining our approach with prompting methods, and (particularly for DIL) investigation of whether training PETL parameters beyond the first session is feasible without catastrophic forgetting. Few-shot learning with [1 45) may potentially also benefit from a RanPAC-like algorithm.
pre-trained models
10

---

Acknowledgements
This work was supported by the Centre for Augmented Reasoning at the Australian Institute for Machine Learning, established by a grant from the Australian Govermeett Department of Education. Dong Gong is the recipient of an Australian Research Council Discovery Early Career Award (project number DE230101591) funded by the Australian Government. We would like thank Sebastien Wong of DST Group, Australia, and Lingqiao Liu of the Australian Institute for Machine Learning, The University of Adelaide, for valuable suggestions and discussion.
References
[1] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
10
[2] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. Advances in Neural Information Processing Systems, 33: 13:15520-15930, 2020. 2
[3] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Rie- mannian walk for incremental learning: Understanding forgetting and intransigence. In Proc. European Conference Computer Vision (ECCV), September 2018. 21
[4] C. L. Philip Chen. A rapid supervised learning neural network for funetion interpolation and 1996.
approximation. IEEE Transactions on Neural Networks,
[5] Haoran Chen, Zuxuan Wu, Xintong Han, Menglin Jia, and Yu-G Yu-Gang Jiang. PromptFusion: Decoupling stability and plasticity for continual learning. arXiv:2303.07223, Submitted 13 March 2023, 2023. 3
[6] Shoufa Chen, Chongjian GE, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. AdaptFormer: Adapting vision transformers for scalable visual recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16664-16678. Curran Associates, Inc., 2022.
2
20 21
[7] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proc. IEEE/CVF International Conference Computer Vision (ICCV), pages 8230-8239, 2021.1
[8] Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the CLIP model. arXiv: 2207.09248, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010. 1929 (ICLR 2020), 2020.
[10] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10629-1064 Curran Associates, Inc., 2022. 7
[11] Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. arXiv:1909.01520, 2020. 2
[12] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kerg-Kirkpatrick, and Graham Neubig. To- wards unified view of parfmeter-efficient transfer learning. arXiv: 21 10.04336 (ICLR 2022), 2022. g
11

---

[13] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proc. IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC.
Canada, October 10-17, 2021, pages 8320-8322 8329, IEEE, 2021. 21
[14] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proc. IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pages 15262-152711 June 2021. 21
[15] Guillaume Hoequet, Olivier Bichler, and Damien Querlioz. OvA-INN: Continual learning with invertible neural networks. In Proc. International Joint Conference on Neural Networks
(IJCNN, pages 1-7, 2020. 2
[16] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang. and Dahua Lin. Learning unified classifier incrementally via rebalancing. Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 831 839, 2019.
[17] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. urXiv:2106.0968 (ICLR 2022).
2021. B
[18] Guang-Bin Huang, Qin- Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: Theory and applications. Neurocomputing, 70:489-501, 2006. 3
[19] Gabriel Ilarcoo, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP. July 2021. URL https://dor org/oog.//0. /0. 5281/
zenodo. 5143773 29
[20] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. arXiv: 2210.04428 (2022 NeurIPS Workshop on Distribution Shifts), 2023.
2 3
TO
16
[21] Menglin Jia, Luming Tang, Gor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proc. European Conference on Computer Vision (ECCV), 2022.
21
[22] Shibo Jie, Zhi-Hong Deng, and Ziheng Li. Alleviating representational shift for continual fine-tuning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3810-3819, June 2022. 2
[23] Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72:309-314 2018. 19
[24] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In Proc. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2661-2671. Computer Vision Foundation IEEE, 2019. 2
[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine- grained categorization. In Proc. IECE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[26] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Dept of CS, University of Toronto. See http: // www. cs. www.stooooooooooooo edu/ kriz/cifar. 2009.
21
[27] Chuqiao Li, Zhiwu Huang, Danda Pani Paudel, Yabin Wang, Mohamad Shahbazi, Xiaopeng Hong, and Lue Van Gool. A continual deepfake detection benchmark: Dataset, methods, and essentials. arXiv:2205.0544 05467, 2022, g 22
12

---

[28] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: new baseline for efficient model tuning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 109-123. Curran Associates, Inc., 2022. 2
20
[29] Vincenzo Lomonaco and Davide Maltoni. Core50: A new dataset and benchmark for contin- uous object recognition. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, Proceedings of the st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 17-26. PMLR, 13-15 Nov 2017. 22
[30] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates,
Inc., 2017.
2 21
[31] Zheda Mai, Ruiwen Li. Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-ineremental continual learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pages 3589-3599. Computer Vision Foundation /
IEEE, 2021. 3
[32] Mark D. McDonnell, Robby G. McKilliam, and Philip de Chazal. On the importance of pair- wise feature correlations for image classification. In Proc. 2016 International Joint Conference on Neural Networks (IJCNN), pages 2290-2299 2016.
[33] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv:2112.09153, 2021. 2
[34] Thomas Mensink, Jakob J. Verbeek, Florent Perronnin, and Gabriela Csurka. Metric learning for large scale image classification: Generalizing to new classes at near-zero cost. In Proc. European Conference on Computer Vision, 2012.
[35] Kevin P. Murphy, Machine learning A probabilistic perspective. MIT Press, Cambridge, Mass.
[u.a.], 2013. 3
18
19
[36] Shaoning Pang, S. Ozawa, and N. Kasabov. Incremental linear discriminant analysis for classification of data streams. IEEE Transactions on Systems, Man, and Cybernetics, Part (Cybernetics), 35:905-914, 2005.
[37] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E. Turner. First session adaptation: A strong replay-free baseline for cnase-nna learning. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 18820-18830, 18830, October 2023.
16 28
[38] Yoh-Han Pao and Yoshiyasu Takefuji. Functional-lin net computing: Theory, system architec- ture, and functionalities. Computing Magazine, 3:7679-7682, 1991.
[39] Francesco Pelosin. Simpler is better: Off-the-shelf continual learning through pretrained
backbones. arXiv:2205.01588 2022. 2
[40] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proc. IEFE International Conference on Computer Vision, pages 1406-1415, 2019. 9 22
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 29
[42] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In Proc. IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages
5533-5542. IEEE Computer Society, 2017. 3
21
13

---

[43] Wouter F. Schmidt, Martin A. Kraaijveld, and Robert P. W. Duin. Feedforward neural networks with random weights. In Proc. 11th IAPR International Conference on Pattern Recognition. Vol. Conference B: Pattern Recognition Methodology and Systems, pages 1 1-4, 1992.
[44] Murray Shanahan, Christos Kaplanis, and Jovana Mitrovic. Encoders and ensembles for
task-free continual learning. arXiv:2105.1333 2021. 27
[45] Aliaksandra Shysheya, John Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E Turner. FiT: Parameter efficient few-shot transfer learning for personalized and federated image classification, arXiv:2206.0866 2023. TO
[46] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonile Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and 7solt Kira. CODA-Prompt: COntinual De- composed Attention-based prompting for rehearsal-free continual learning. In Proc. IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11909-11919, June 2023.
10 16
[47] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. closer look at rehearsal- continual learning. MxXiv:2003.17266 (Revised 3 April, 2023. 2
33 10
I6
[48] Albin Soutif- Cuntif-Cormerais, Mare Masana, Joost van de Weijer, and Bartlomiej Twardowski. On the importance of cross-task features for class-incremental learning. arXiv:2106. 1930, 2021.
[49] Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. PILOT: A pre-trained model- based continual learning toolbox. arXiv:2309.0711 2023. 9
[50] Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In Proc. Intl Conference on Learning Representations (ICLR), 2018.
[51] Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, pages 1185-1197 2022.
[52] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie, The Caltech- UCSD Birds-200-2011 Dataset. Technical report, California Institute of Technology, 2011. Technical Report 1SS-TR20011-01
21
[53] Liyunn Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv: 2302.00487, 2023.
[54] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. arXiv: 1708.051222 (ADKDD' 17), 2017.
[55] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-Prompts learning with pre-trained transform- ers: An Occam's 's razor for domain incremental learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 5682-66660 5695. Curran Associates, Inc., 2022.
16
[56] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. DualPrompt: Complementary prompting for rehearsal- -ree continual learning. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, page 631-648, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 978-3-031-19888-11
16
21
[57] Zifeng Wang, Zizhao Zhang, Chen- Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proc. IEFF/CV Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 139-149, IEEE, 2022.
16
27
[58] Andrew R. Webb and Keith D. Copsey. Statistical Pattern Recognition. John Wiley Sons, Ltd, 2011. 4
14

---

[59] Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vascon- celos, Rahul Bhotika, and Stefano Soatto. class-incremental learning with strong pre-trained models. In Proc. IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9601-9610 2022. 2
[60] Qingsen Yan, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. Learning Bayesian sparse networks with full experience replay for continual learning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 109-118, 2022. 2
[61] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational Bayes.
2019. 27
[62] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. arXiv: 191.04991 2019,,
21
[63] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SLCA: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 19148-19158, October 2023.
3
8
21
[64] Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision repre- sentation through the lens of visual realms. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, volume
13667 of Lecture Notes in Computer Science, pages 594-611. Springer, 2022.
2T
[65] Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class- classinccremenal learning with pre-traines me-trained models: Generalizability and adaptivity are all you need, arXiv:2303.07338, Submitted 13 March, 2023. 2
26
28
15

---

Appendices
Appendix A Overview of RanPAC and comparison to other strategies
Figure Al provides graphical overview of the two phases in Algorithm 1. Table Al provides summary of different strategies for leveraging pre-trained models for Continal Learning (CL) and how our own method, RanPAC, compares.
First-session PETL Adaptation (Algorithm 1, Phase 1)
Continual Learning with Random Projections (Algorithm 1, Phase 2)
Frozen
PTM
PETL
Aduptted FTM
Adapted PTTA
W
W
y
Extrasted Fasturee
Fandimm Activatedd Projection Features
Class
Prototypes
Task I(Pi)
Task 1(D,)
Tisk 2(Dg)
Task T(Dr)
Training data for Phase
Sequence of training tasks for Phase 2
Figure A1: Overview of RanPAC for CL classification. In Phase 1 of Algorithm 1 we optionally inject parameters for Parameter-Efficient Transfer Learning (PETL) method into frozen pre- trained model (PTM). The PETL parameters are trained only on Task (the 'first-session') T continual learning tasks, help bridge the domain gap, as in [65 337. Then in Phase first, I-dimenniioal feature vectors, f, are extracted from the network after completion of learning in Phase 1 (now frozen). Then, the extracted feature vectors randomly projected dimension M (typically M > L) using frozen weights W. followed by nonlinear activation, é, to obtain new feature vectors,
a
o(f W). Training in Phase 2 is comprised of continual iterative updating of class prototypes and the Gram matrix, followed by a matrix inversion to compute M x N matrix W following the end of each Task's training. These weights can be thought of decorrelated class prototypes that linearly weight the features in h to obtain class predictions in
Prompting
Fine-tuning
CP RP
57
56
46
55
63
47
20
65
37
RanPAC
No Rehearsal Buffer Pre-trained model frozen Transformers and CNNs
Simplicity Carameter-Efficien
Theoretical support
SOTA Performance
Table A1: Comparison of different strategies. In Section I we categorised existing methods for leveraging pre-trained models for CI. into three strategies: Prompting, Fine-tuning and Class- Prototypes (CP). Our own method, RanPAC, a CP strategy in which we introduce frozen Random argue that CP methods provide the best
Projection (RP) layer with nonlinear activation. Overall, combination of benefits, and within existing CP methods, RanPAC provides the strongest results, complemented by theoretical support for its use of RP and second-order statistics to decorrelate CPs. We note that rehearsal buffers have been used with some of the cited methods to boost performance, and could potentially also do so for methods where they have not yet been used, such as RanPAC.
16

---

Appendix B Theoretical support
B.1 Chernoff bound of the norm of the projected vectors
We provide further details for the discussion in Section 4.2 The norm of the vector projected using the Chernoff Bound can be written as:
6²02
P (III W T [l| - Ew [IW ri]] > co²) S 2xxp
(6)
2M +
This bound indicates the relation between the dimensionality and the expected variation in the norm of the projected vectors. For fixed and E, as M increases, the right-hand side approaches 1, indicating that it more likely for the norm the projected vector to be in a desired distance to the expectation. In other words, these projected vectors in higher dimensions almost surely reside the boundary of the distribution with a similar distance to the mean (the distribution is better Gaussian fit).
B.2 The effects of increasing the projection dimensions
The Gram matrix of the projected vectors can be obtained by considering the inner product of any two vectors f, f'. As presented in Eqn. (3), this derived as:
Ew [WW r) (WTT)) = Ew [f (WTTP))
M
M WOF
Ew
W
F
(
i
M
M
Ew
f F
Ew
W
[w)] Ew [ (7)
=
where the second term is zero for any two zero-mean independently drawn random vec- tors WG, Way- We can derive the following from this expansion:
1. Using the Chernoff inequality, for any two vectors, we have
M22
P (((WTF) (w f') EEw [(W r) (w r)]] > e'Mo²) 2exp
(2+e')
e2M
P P(((WFF) (W - MM > c'Mo²) - 2
(2+
P (WTT
(WF PT )) s 2 exp
e2
(8)
Mo²
(2+
This bound indicates that as the dimension of the projections increases, it is more likely for the inner product of any two vectors and their projections to be distinct. other words, increasingly unlikely for the inner product of two vectors and their projections to be equal as M increases.
2. As M increases, it is more likely for the inner product of any two randomly projected instances to be distinet (i.e. the inner products in the projected space are more likely be larger than some constant). That is because, using Markov's 's inequality, for a larger M it is easier to choose larger eg that satisfies
M
P (((WTT (IWTE)T(CE f`) E2) <
(9)
62
17

---

B.3
Connection to least squares
The score of Eqn. can be written in matrix form
Ytest hteet (G + CC,
(10)
where Prest == @f test W) are the randomly projected activations following element-wise nonlinear activation (6-). The form W. = (G + AI) 10--C CC arises in fundamental theory of least squares
regression. We can write this as
W. (G + XI) train
(11)
(HH + A-I)-HHY train
(12)
where train is x K one-hot encoded target matrix, and H is as defined in Section 4.3
Eqn. (22
is well known [35 Eqn (7.33), p. 226] to be the minimum mean squared error solution to the (2
regularized set equations defined by
W H.
(13)
train
For parameter 2 0, this is usually expressed mathematically
W. argminw (I train w T wI
(14)
Consequently, when not using CL. optimizing a linear output head using SGD with mean square error loss and weight decay equal to À will produce a loss lower bounded by that achieved by directly computing W..
B.4 Connection to Linear Discriminant Analysis, Mahalanobis distance and ZCA whitening
There are two reasons why we use Eqn. ( for Algorithm rather than utilize LDA. First and foremost, is the fact that our formulation is wrean-squaee-rror optimal, as per Section B.3 Second, use of the inverted Gram matrix results in two convenient simplifications compared with LDA: (i) the simple form of Eqn. ( can be used for inference instead of a form in which biases calculated from plass-prototypes are required, and (ii) accumulation of updates the Gram matrix and class- prototypes during the CL process as in Eqn. are more efficient than using covariance matrix.
We now work through the theory that leads to these conclusions. The insight gained is that they show that using Eqn. (2) is equivalent to learning linear classifier optimized with mean square error loss function and l2 regularization, applied to the feature vectors of the training set. These derivations apply in both CL and non-CL context. For CL, the key is to realise that CPs and second-ord -oddrr statistics can be accumulated identically for the same overall set of data, regardless of the sequence in which it arrives for training.
B.4.1 Preliminaries: Class-Prototypes
Class-prototypes for CL after task T be defined as
cy ny t=1n=1
ΣΣ
h I
y y= K,
(15)
where Ie, is an indicator function with value 1 if the n-th training sample in the 1-th task is in class and zero otherwise, ny is the number of training samples in each class and h,,, is an M-dimensional projected and activated feature vector. For RanPAC we drop the mean and use
Cy
Te.T y 1. K.
(16)
For Nearest Class Mean (NCM) classifiers, and a cosine similarity metric as used by 65 the score for argmax classification is
Gest Cy
= 1
(17)
y
Frealll ||,
where the C, are calculated from feature vectors fr, rather than h These cosine similarities depend only on first order feature statistics, i.e. the mean feature vectors for each class. In contrast, I.DA and our approach use second-order statistics i.e. correlations between features within the feature vectors.
18

---

B.4.2 Relationship to LDA and Mahalanobis distance
The form of Eqn. 2 resembles Linear Discriminant Analysis (LDA) classification [35 Eqn. (4.38),
p. 104]. For LDA, the score from which the predicted class for frest chosen is commonly expressed in weighting and bias form
by first a +
(18)
E S - 0.55,, + log (Ty),
y y= 1, y..,..., K,
(19)
where S is the M M covariance matrix for the M dimensional feature vectors and Fy is the frequency of class y.
Finding the maximum By is equivalent to minimization involving the Mahalanobis distance, dı (frest Byy) S S Gyy, between test vector and the CPs, i.e. minimizing
Vy = dh log (m
(20)
(frest Cyy) S S -,,) log (m
T
y I. K.
(21)
This form highlights that all classes are equiprobable, minimizing the Mahalanobis distance suffices for LDA classification.
B.4.3 Relationship to ZCA whitening
We now consider ZCA whitening Mahalanobis transform, Dz S 0.5 Substituting for S in Eqn. (21) gives
[23 This process linearly transforms data H to DzH using the
Sy (Dz(frest -,,) (Dz c2)) - log (m
(22)
I1(Dz(frest 2, log (m))) y= l.,....,K
(23)
This form implies that if all classes are equiprobable, LDA classification is equivalent to minimizing Euclidian distance following ZCA whitening of both a test sample vector, and all CPs.
B.4.4 Decorrelating Class-prototypes
The score in Eqn. (2) can be derived in a manner that highlights the similarities and differences the Mahalanobis transform. Here we use the randomly projected features h instead of extracted features f. We choose linear transform matrix D E M XM that converts the Gram matrix, G HH to the identity matrix, i.e. D must satisfy
(H) IMXM
(24)
is easy to see that D (HH - G-1, Next, treating test samples and class prototypes as originating from the same distribution H, we can consider the dot products
8y (Dh T T(Dcy)
hT G -
Cy. y
K.
(25)
Hence, the same similarities arise as those derivable from the minimum mean square error formulation of Eqn. (14). When compared with the Mahalanobis transform, Dz = S- 0.5 the difference here is that the Gram matrix becomes equal to the identity rather than the covariance matrix.
LDA is the same as our method if all classes are equiprobable and G = S, which happens if all M features have a mean of zero, which not true general.
Appendix C Training and implementation
C.1 Optimizing ridge regression parameter in Algorithm 1
With reference to the final step in Algorithm 1, we optimized as follows. For each task, 4, in Phase
2, the training data for that task was randomly split in the ratio 80:20. We parameter swept over
17 orders of magnitude, namely E {10-8, 10-7 .o., 10y) and for each value of À used C and G updated with only the first 80% of the training data for task t then calculate W. (G -CC CC.
19

---

We then calculated the mean square enror between targets and the set of predictions of the form h" W. for the remaining 20% of the training data. We then chose the value of À that minimized the mean square error on this 20% split. Hence, À is updated after every task, and computed in manner compatible with CL., i.e. without access to data from previous training tasks. It is worth noting that optimizing À to value between orders of magnitude will potentially slightly boost accuracies. Note also that choosing only for data from the current task may not be optimal relative to non-CL learning on the same data, in which case the obvious difference would be to optimize À on subset of training data from the entire training set.
C.2 Training details
For Phase 2 in Algorithm 1, the training data was used as follows. For each sample, features were extracted from a frozen pretrained model, in order to update the G and C matrices. We then computed W. using matrix inversion and multiplication. Hence, no SGD based weight updates are required in Phase 2.
For Phase 1 in Algorithm we used SGD to train the parameters of PETL methods, namely AdaptFormer [6. SSF (28), and VPT 220. For each of these, we used batch sizes of 48, a learning rate of 0.01, weight decay of 0.0005, momentum of 0.9, and cosine annealing schedule that finishes with learning rate of O. Generally we trained for 20 epochs. but in some experiments reduced to fewer epochs if overfitting was clear. When using these methods, softmax and cross-entropy loss was used. The number of classes was equal to the number in the first task, i.c. NN. The resulting trained weights and head were discarded prior to commencing Phase of Algorithm 1.
For reported data in Table ! Jsing linear probes we used batch sizes of 128, a learning rate of 0.01 in the classification head, weight decay of 0.0005, momentum of 0.9 and training for 30 epochs. For full fine-tuning (Table 2 we used the same settings, but additionally used a learning rate in the body (the pre-trained weights of the ViT backbone) of 0.0001. We used a fixed learning rate schedule, with no reductions in learning rate. We found for fine-tuning that the learning rate lower in the body than the head was essential for best performance.
Data augmentation during training for all datasets included random resizing then cropping to 224x224 pixels, and random horizontal flips. For inference, images are resized to short side 256 and then center-cropped to 224 x 224 for all datasets except CIFAR100, which are simply resized from the original 32 32 to 224 224.
Given our primary comparison is with results from 665 we use the same seed for our main experi- ments, i.e. seed value of 1993, This enables us to obtain identical results (65) in our ablations.
However, we note that we use Average Accuracy as our primary metric, whereas [65] in their public repository calculate overall accuracy after each task which can be slightly different to Average Accuracy. For investigation of variability in Section F we also use seeds 1994 and 1995.
C.3 Training and Inference Speed
The speed for inference with RanPAC is negligibly different to the speed of the original pre-trained network, because both the RP layer and the output linear classification head (comprised from decorrelated class prototypes) are implemented as simple fully-connected layers on top of the underlying network. For training, Phase trains PETL parameters using SGD for 20 epochs, on (1/T)'th of the training set, so is much faster than joint training. Phase is generally only slightly slower than running all training data through the network in inference mode, because the backbone is frozen. The slowest part is the inversions of the Gram matrix, during selection of X, but even for M 10000, this is in the order of minute per task CPU, which can be easily optimized further needed. Our view is that the efficiency and simplicity of our approach compared to the alternatives is very strong.
C.4 Compute
All experiments were conducted on single PC running Ubuntu 22.04.2 LTS, with 32 GB of RAM, and Intel Core i9-13900KF x32 processor, Acceleration was provided by a single NVIDIA GeForce 4090 GPU.
20

---

Appendix D Parameter-Efficiet Transfer Learning (PETL) methods
We experiment with the same three methods as [65], i.e. AdaptFormer 6 SSF 28 and VPT 21 Details can be found in [65] For VPT, we use the deep version, with prompt length 5. For AdaptFormer, we use the same settings as in [65] i.e. with projected dimension equal to 64.
Appendix E Datasets
E.1 Class Incremental Learning (CIL) Datasets
The seven CIL datasets we use are summmarised in Table A2 For Imagenet-A, CUB, Om- nibenchmark and VTAB, used specific train-validation splits defined and outlined in detail by [65]. Those four datasets, plus Imagenet-R (created by [56] were downloaded from links pro- vided htttt:t:://// thub.
CIFARI00 was accessed through torchvision. Stanford cars was downloaded from htttss:/aa.ssseeeeeeeespe stanfordd edu/" ikrause/cars car car_dataset.html
For Stanford Cars and T = 10, we use 16 classes in f = and 20 the 9 subsequent tasks. is interesting to note that VTAB has characteristics of both CIL and DIL.. Unlike the three DIL datasets we use, VTAB introduces new disjoint sets of classes in each of 5 tasks that originate in different domains. For this reason we use only T for VTAB, whereas we explore T T 10 and T = 20 for the other CIL datasets.
Original
CL version
N
# val samples
CIFAR 1OO Imagenet-R Imagenet-A
26
42
50000
10000
100
[66
24000
6000
200
14
655
5981
5985
200
CUB
[52
[65
9430
2358
200
OmniBenchmark
664
65
89697
5985
300
VTAB Stanford Cars
62
665
1796
8619
50
25
63
8144
8041
196
Table A2: CIL Datasets. We list references for the original source of each dataset and for split CL versions of them. In the column headers, N is the total number of training samples, K is the number of classes following training on all tasks, and # val samples is the number validation samples in the standard validation sets.
E.2 Domain Incremental Learning (DIL) Datasets
For DIL, list the domains for each dataset in Table A3 Further details can be found in the cited references in the first column of Table A3 As previous work, validation data includes samples from each domain for CDDB-Hard and DomainNet, but three entire domains reserved for CORe50.
Appendix F Additional results
F.1 Preliminaries
We provide results measured by Average Accuracy and Average Forgetting. Average Accuracy is defined as [30]
A, = Σ R
(26)
1=1
where Rs,; are classification accuracies on the i-th task, following training on the f-th task. Average Forgetting defined as (3)
max "E(1,,...
(Re Re,s)-
(27)
21

---

K
T
N
N1, T
Nval
val set
CORe50 20
50
8
119894
44972
S3, S7, S10
(Nearly class balanced)
N1 14989 (S1)
N2 14986 (S2)
(~ 2400 class)
Ns 14995 (S4)
N1 14966 (S5)
N₂ = 14989 (S6)
No = 14984 (S8)
N7 14994 (S9)
Ng 14991 (S11)
CDDB-Hard [27
5
16068
5353
Standard
(Class balanced both train and val)
N1 6000 (gaugan)
2000
N2 2400 (biggan)
800
N₃ 6208 (wild)
2063
N4 1200 (whichfaceisreal)
100
N₂ 260 (san)
90
DomainNet 40 (Imbalanced)
3-55
6
409832
176743
Standard
N1 120906 (Real)
52041
N2 120750 (Quickdraw)
51750
N₃ 50416 (Painting)
21850
N4 48212 (Sketch)
20916
N₂ 36023 (Infograph)
15582
N₂ 33525 (Clipart)
14604
Table A3: DIL Datasets. is the number of classes, all of which are included in each task. T is the total number of tasks, N is the total number of training samples across all tasks and Nyal is the number of validation samples, either overall (first row per dataset) or per task. Also shown the number of training samples in each task, N,, and the domain names for the correspond- csr unibo. /mm.bb..t//mmltooo////ooooooo core50/ ctre/oore50___0 1ar000_mms____ .pp CDDB was downloaded from htttss:/corra/99 github. io/ CDDB_ web/ DomainNet was downloaded from http://at //ai: bu ecuu/K3SDA/Hdatasees we used the version labelled as "cleaned version, recommended."
ing tasks. Core50 was downloaded from http::/biis
Note that for CIL,, we calculate the Re, as the accuracy for the subset of classes in Di
For DIL, since all classes are present in each task, Re.; has different nature, and is dependent on dataset conventions. For CORe50, the validation set consists of three domains not used during training (S3, S7 and S10, as per Table A3). and in this case, each Re calculated on the entire validation set. Therefore R. is constant for all i and A, Re For CDDB-Hard and DomainNet, for the results in Table 3 we treated the validation data in the same way as for CORe50. However, is also interesting to calculate accuracies for validation subsets in each domain we provide such results in the following subsection.
F.2 Variability and performance on each task
Fig. A2 shows, for three different random seeds, Average Accuracies and Average Forgetting after each CIL task matching Table I Section without PETL (Phase 1). Due to not using PETL, the only random variability is (i) the choice of which classes are randomly assigned to which task and (ii) the random values sampled for the weights in W. We find that after the final task, the Average Accuracy identical for each random seed when all classes have the same number of samples, and are nearly identical otherwise. Clearly the randomness in W has negligible impact. For all datasets except VTAB, the value of RP clear, i.e. RP (black traces) delivers higher accuracies by the time of the final task than not using RP, or when only using NCM (blue traces). The benefits of second-order statistics even without RP are also evident (magenta traces), with Average Accuracies better than NCM. Note that VTAB always has the same classes assigned to the same tasks, which is why only one repetition is shown. The difference with VTAB is also evident the average accuracy trend as the number of tasks increases, i.e. the Average Accuracy does not have a clearly decreasing trend as the number of tasks increases. This is possibly due the difference in domains for each Task making it less likely that confusions occur for classes in one task against those in another task.
22

---

Fig. A3 shows comparisons when Phase 1 (PETL) is used. The same trends are apparent, except that due to the SGD training required for PETL parameters, greater variability Average Accuracy after the final task can be seen. Nevertheless, the benefits of using RP are clearly evident.
NCH Only
8.0
6.0
100
4.0
2.0
85.0
0.0
91
01
D1
04
0g
Di
D2
0g
D4
Ds
D₂
Dv
Dio
97.5
ICH Only
90.0
87.5
1
01
94
04
06
0.
D1o
02
0g
04
0g
9,
0g
0g
Di
85.0
ICC Only
RP
RP. M-10000
10.0
75.0
70.0
60.0
D4
D5
25.0
NCM Only
20.0
RP. 1=10000
70.0
15.0
60.0
0.0
0,
D1
D.
0x
D.o
D2
01
Da
D4
Do
Dio
05.0
15.0
90.0
85.0
00.0
75.0
0.
0,
D,
D1
D1
D2
Dio
D2
01
D4
0g
D2
Da
Dv
Dw
3.0
NCM Oily
02.0
2.0
14-10 1000
88.0
D,
Dg
0,
NCM Only
No AP
00.0
20.0
RF: 10000000
60.0
0.0
02
0,
04
04
1200
Figure A2: Average Accuracies and Forgetting after each Task CIL datasets (no Phase 1). The left column shows Average Accuracies for three random seeds after each of T tasks for the seven CIL datasets, without using Phase The right column shows Average Forgetting.
23

---

6.0
PETI +MEM
47.5
PETL-NA RP
RonPKC
05.0
92.5
00.0
87.5
030
D4
Dto
97.5
92.5
90.0
09
02
D1
94
14
06
D1
Dx
D1
D70
D:
D1
04
0x
94
D4
On
Din
PETL -MEM
90.0
PLTL-ND
Ran/lic
85.0
6.0-
75.0
2.0
02
12
D1e
0g
Dip
PETL+ARE
80.0
PETL-ND Hoo RP
20.0
RanPIC
70.0
15.0
60.0
0.01
4
0.
10.0
5.0
0.0
0,
04
9
01
D,
D,
D1e
D2
D4
D4
D)
0g
Di
05.0
PETL
3.0
42.5
Ran
2.0
90.0
87.5
85.0
02
05
91
Dv
90.0
25.0
PETI +NZM PETL.ND RP
80.0
Ran
15.0
60.0
50.0
5.0
Figure A3: Average Accuracies and Forgetting after each Task for CIL datasets (with Phase 1). The left column shows Average Accuracies for three random seeds after each of T tasks for the seven CIL datasets, for the best choice of PETL method in Phase The right column shows Average Forgetting.
24

---

Fig. A4 shows how accuracy on individual domains changes tasks are added through training for the DIL dataset, CDDB- The figure shows that after training data for a particular domain first used, that accuracy on the corresponding validation data for that domain tends to increase. In some cases forgetting is evident, e-g. for the 'wild' domain, after training on T4 and Ts. The figure also shows that averaging accuracies for individual domains ("mean over domains') )) is significancly lower than 'Overall accuracy'. For this particular dataset, 'Average accuracy" is potentially misleading metric it does not take into account the much lower number of validation samples in some domains, e.g. even though performance on 'san" increases after training on it, it is still under 60%, which is poor for binary classification task.
Fig. A5 shows the same accuracies by domain for DomainNet. Interestingly. unlike CDDB-Hard, accuracy generally increases for each Domain as new tasks are learned. This suggests that the pre-trained model highly capable of forming similar feature representation for the different domains in DomainNet, such that increasing amounts of training data makes it easier to discriminate between classes. The possible difference with CDDB-Hard is that that dataset is a binary classification task in which discriminating between the 'real' and 'fake' classes is inherently more difficult and not reflected the data used to train the pre-trained model.
gaugan
biggan
90
wild
whichfaceisreeal
Accuracy (%)
san
Mean over domains
Average Accuracy
70
(gaugan)
To (biggan)
(wild)
To (whichlaceissrea)
(san)
Figure A4: Accuracies for DIL dataset CDDB-Hard. Results are shown for the full RanPAC algorithm, using VPT the PETL method, and M 10000. Each task, Ti corresponds to learning on a new domain as shown on the x-axis. The accuracies shown in colors are those for individual domains, following training on the domain shown the x-axis. "Mean over domains" is the average of the five domain accuracies after each task.
F.3 Comparison of impact of Phase for different numbers of CIL tasks
As shown in Fig. A2 when Phase I excluded, the final Average Accuracy after the final task has negligible variability despite different random assignments of classes tasks. This is consequence primarily of Eqns. (4) being invariant to the order in which a completed set of data from all T tasks is used in Algorithm I. As mentioned, the influence of different random values in W is negligible. The same effect occurs if the data is split to different numbers of tasks, e.g. T = 5, T = 10 and T = 20, in the event that all classes have equal number of samples, such CIFAR100.
Therefore, this section we show in Table A4 results for RanPAC only for the case where variability has greater effect, i.c. when Phase 1 is included, and AdaptMLP chosen. VTAB is excluded from this analysis, since it is a special case where it makes sense to consider only the case of = 5. The comparison results for L2P, DualPrompt and ADaM are copied from (65).
Performance when AdaptMLP is used tends be better for T 5 and worse for T 20. This is consistent with the fact that more classes are used for first-session training with the PETL method when T By comparison, T 20 is often on par with not using the PETL method all, indicating that the first session strategy may have little value if insufficient data diversity is available within the first task.
25

---

*- real
Accuracy
quickdraw
painting
sketch
50
infograph
clipart
Mean over domains
Average Accuracy
T: (real)
(quickdraw)
T3 (painting)
Ta (Sketch)
Ts (infograph)
Tx (clipart)
Figure A5: Accuracies for DIL dataset DomainNet. Results are shown for the full RanPAC algorithm, using VPT as the PETL method, and M = 10000. Each task, T, corresponds to learning on new domain as shown on the x-axis. The accuracies shown in colors are those for individual domains, following training on the domain shown on the x-axis. 'Mean over domains' is the average of the six domain accuracies after each task.
T= 5
T 10
T 20
RanPAC- No Phase
CIFARI00
RanPAC
92.4%
92.2%
90.8%
89.0%
AdaM
88.5%
87.5%
85.2%
DualPrompt
86.9%
84.1%
81.2%
L2P
87.0%
84.6%
79.9%
NCM
88.6%
87.8%
86.6%
83.4%
ImageNet-R
RanPAC
79.9%
77.9%
74.5%
71.8%
AdaM
74.3%
72.9%
70.5%
DramPtompt
72.3%
71.0%
68.6%
L2P
73.6%
72.4%
69.3%
NCM
74.0%
71.2%
64.7%
61.2%
ImageNet-A
RanPAC
63.0%
58.6%
58.9%
58.2%
AdaM
56.1%
54.0%
51.5%
DualPrompt
46.6%
45.4%
42.7%
L2P
45.7%
42.5%
38.5%
NCM
54.8%
49.7%
49.3%
49.3%
CUB
RanPAC
90.6%
90.3%
89.7%
89.9%
AdaM
87.3%
87.1%
86.7%
DualPrompt
73.7%
68.5%
66.5%
I.2P
69.7%
65.2%
56.3%
NCM
87.0%
87.0%
86.9%
86.7%
OmniBenchmark
RanPAC
79.6%
79.9%
79.4%
78.2%
AdaM
75.0%
74.5%
73.5%
DualPrompt
69.4%*
65.5%
64.4%
L2P 67.1%
64.7%
60.2%
NCM
75.1%
74.2%
73.0%
73.2%
Cars
RanPAC 69.6%
67.4%
67.2%
67.1%
NCM 41.0%
38.0%
37.9%
37.9%
Table A4: Comparison of CIL results for different number of tasks. For this table, the AdaptMLP PETL method was used for RanPAC. The comparison results for L2P, DualPrompt and ADaM are copied from oS for ADaM, the best performing PETL method was used. In most cases, Average Accuracy decreases as T increases, with T = 20 typically not significantly better than the case of no PETL ("No Phase 1"). Values marked with indicates data is for Note that accuracies for comparison methods for Cars were not available from 66
tasks, instead of T
26

---

F.4
Task Agnostic Continual Learning
Unlike CIL and DIL, 'task agnostic' CL is a scenario where there no clear concept of a 'task' during training [61. The concept is also known as 'task-free" [44]. It contrasts with standard CIL where although inference is task agnostic, training is applied to disjoint sets of classes, described as tasks. To illustrate the flexibility of RanPAC, we show here that it is simple to apply it to task agnostic continual learning. We use the Gaussian scheduled CIFAR 100 protocol of [57]. which was adapted from [44]. We use 200 'micro-tasks' which sample from gradually shifting subset of classes, with batches of 48 samples in each micro-task. There are different possible choices for how to apply Algorithm 1. For instance, the 'first session' for Phase 1 could be defined particular total number of samples trained on, e.g. 10% of the anticipated total number of samples. Then in Phase 2, the outer for loop tasks could be replaced by loop over all batches, removed entirely. In both cases, the result for G and C will be unaffected. The greater challenge is in determining X, but generally for large number of samples, can be small, zero. For small number of training samples, queue of samples could be maintained such that the oldest sample in the queue is used to update G and C, with all newer samples used to compute if inference is required, and then all samples in the buffer added to G and C.
Here, for simplicity we illustrate application to Gaussian- Fig. A6 shows how test accuracy changes through training both with and without RP. The green trace illustrates how the number of classes training. instead of in steps like in CIL. The red traces show validation accuracy on the entirety of the validation
CIFARI00 without any Phase
in at least one sample increases gradually through
As expected, this increases training becomes exposed traces show the accuracy on only the classes seen so far through training. By the end of training, the red and black traces converge, expected. Fluctuations in black traces be partially attributed to not optimizing X. The final accuracies with and without RP match the values for T = 10 CIL. shown in Table
classes. The black
100
Accuracy on entire test set (no RP)
Accuracy on classes seen so far (no RP)
Accuracy on entire test set (M=10000)
Accuracy on classes seen so far (0=00000)
% of classes seen
125
150
Micro-task ID
Figure A6: Task agnostic example. Application of Phase 2 of Algorithm 1 to the Gaussian-
scheduled CIFARIO0 task-agnostic CL protocol.
F.5 Scaling with projection size
Table A5 shows, for the example of split CIFAR100, that it is important ensure M is sufficiently large. In order to surpass the accuracy obtained without RP (see 'No RPs or Phase 1 in Table T). M needs to be larger than 1250.
F.6 Comparison of PETL Methods and ViT-B/16 backbones
Fig. A7 shows how performance varies with PETL method and by ViT-B/16 backbone. For some datasets, there is a clearly superior PETL method. For example, for CIFAR 100, AdaptMI.P gives better results than SSF or VPT. for both backbones, for Cars VPT is best, and for ImageNet-A, SSF
27

---

M
Accuracy
100
71.6%
200
80.3%
400
83.9%
800
86.2%
1250
86.8%
2500
87.7%
5000
88.4%
10000
88.8%
15000
89.0%
Table A5: Scaling with AM for split CIEARI00. The table shows final average accuracy for T = 10 and no Phase with À = 100 as a constant for all M, using ViT-B/16 trained on ImageNer21K.
best. There is also interesting outlier for VTAB, where VPT and the ImageNet-21K backbone failed badly. This variability by PETL method suggests that the choice of method for first-session CL strategies should be investigated in depth in future work.
Fig. A7 also makes it clear that the same backbone is not optimal for all datasets. For example, the VIT-B/T6 model fine-tuned on ImageNet- K is best for ImageNet-A, but the ViT-B/16 model trained on ImageNet-21K is best for CIFAR100 and OmniBenchmark. For Cars, the best backbone depends on the PETL method.
Fig. summarises how the two VIT networks compare. For datasets and method variants we plot the Average Accuracy after the final task for one pre-trained VIT network (self-supervied on ImageNet-21K3 against the other (fine-tuned on ImageNet- IK). Consistent with Fig. A7 the best choice of backbone is both dataset dependent and method nethod-dependent.
F.7 Experiments with ResNets
Unlike prompting strategies, approach works with any feature extractor, e.g. both pre-trained transformer networks and pre-trained convolutional neural networks. To illustrate this, Table | and Table A7 shows CIL results for ResNet50 and ResNet 152, respectively, pre-trained We used = 10 tasks (except for VTAB, which is T = tasks). Although this is different to the T = 20 tasks used for ResNets by [65]. the accuracies those in 65 As with results for pre-trained ViT-B/16 networks, the use of random projections and second-order statistics both provide significant performance boosts compared with NCM alone. We do not use Phase of Algorithm here but as shown by [37 (65) this is feasible for diverse PETL methods for convolutional neural networks. Interestingly, ResNet152 with RP produces reduced accuracies compared to ResNet50 on CUB, Omnibenchmark and VTAB. It is possible that this would be remedied by seeking optimal value of M, whereas for simplicity chose M 10000. Note that unlike the pre-trained ViT-B/16 model, the pre-trained ResNets require preprocessing to normalize the input images.
ImageNet.
report for NCM very comparable to
Method
CIFAR100
IN-R
IN-A
CUB
OB
VTAB
Cars
Phase
75.5%
56.5%
35.5%
70.9%
68.0%
88.6%
48.5%
Phase (No RPs)
73.3%
54.8%
36.6%
62.7%
60.2%
87.7%
42.6%
NCM
61.5%
43.1%
25.8%
55.4%
57.4%
82.2%
26.3%
Table A6: Results for ResNet50. Results are final Average Accuracies for 10 tasks, except VTAB which is T = 5 tasks. Pre-trained weights were IMAGENETIK V2 variants from PyTorch.
Method
CIFAR100
IN-R
IN-A
CUB
OB
VTAB
Cars
Phase
79. 79.7%
58.7%
40.9%
66.7%
65.8%
90.0%
45.1%
Phase (No RPs)
77.8%
56.9%
39.9%
59.4%
58.0%
87.9%
38.8%
NCM
70.1%
47.5%
32.1%
50.6%
56.1%
82.6%
25.2%
Table A7: Results for ResNet152. Other caption information is the same as Table A6
28

---

ResNet results for DIL are provided in Table A8
Method
Backhone
CORe50
CDDB-Hard
Domain Net
Phase 2
ResNet50
88.5%
75.9%
52.7%
Phase (No RPs)
ResNet50
86.7%
75.6%
41.9%
NCM
ResNet50
73.1%
65.1%
32.9%
Phase
ResNet152
87.1%
71.3%
53.8%
Phase 2 (No RPs)
ResNet152
86.8%
71.9%
42.9%
NCM
ResNet152
72.0%
65.6%
35.1%
Table A8: DIL Results for ResNet50 and ResNet152. Pre-trained weights were IMAGENETIK, variants from PyTorch.
F88 Experiments with CLIP vision model
To further verify the general applicability of our method, we show in Table A9 CIL results from using a CLIP 44 vision model as the backbone pre-trained model. The same general trend as for pre-trained VIT-B/16 models and ResNets can be seen, where use of RPs (Phase 2 in Algorithm 1) produces better accuracies than NCM alone. Interestingly, the results for Cars is substantially better with the CLIP vision backbone than for ViT-B/16 networks. It is possible that data from a very similar domain as the Cars dataset was included in training of the CLIP vision model. The CLIP result for Phase 2 only (sec ablations in Table is also better than VIT/B-16 16 for Imagenet-R, but for all other datasets, ViT/B- 16 has higher accuracy. Note that unlike the pre-traine trained ViT-B/16 model, the pre-trained CLIP vision model requires preprocessing normalize the input images.
Method
CIFAR100
IN-R
IN-A
CUB
OB
VTAB
Cars
Phase
85.0%
76.1%
39.7%
79.4%
75.1%
89.6%
90.4%
Phase (No RPs)
80.9%
67.0%
36.8%
72.8%
66.3%
89.8%
85.8%
NCM
77.1%
68.0%
32.5%
73.7%
67.8%
79.4%
85.6%
Table A9: Results for CLIP-ViT-B/16. Results are final Average Accuracies for T = 10 tasks, except VTAB which T tasks. We used pre-trained weights from OpenCLIP, finetuned on ImageNet-IK [1]. The weights were loaded from the timm library using the model file name vit_ utt_base_patchh2.2 22rpp2224______ _tt.
CLIP results for DII. are provided in Table AT0, The CLIP result outperforms that for Phase 2 only for the ViT-B/16 model for CDDB-Hard and DomainNet, but not CORe50 (Table
Method
CORe50
CDDB-Hard
DomainNet
Phase
92.9%
84.4%
68.7%
Phase (No RPs)
90.2%
81.7%
59.1%
NCM
83.9%
66.9%
51.0%
Table A10: DIL Results for ResNet50 and ResNet152. GENETIK V2 variants from Porch.
Pre-trained weights were IMA-
F9 Experiments with regression targets using CLIP vision and language models
Until this point, we have defined the matrix C as containing Class-Prototypes (CPs), i.c. C has N columns representing averaged feature vectors of length M. However, with reference to Eqn. (33). the assumed targets for regression, Ytrain can be replaced by different targets. Here, we illustrate this using CLIP language model representations as targets, using OpenAl's CLIP ViT-B/16 model.
Using CIFARI00 as our example, we randomly project CLIP's length-511 512 vision model representa- tions as usual, but also use the length-512 language model's representations of the 100 class names, averaged over templates
in [41. We create target matrix of size N 512 in which each row the language model's length 512 representation for the class of each sample. We then solve for W. R Mx612 using this target instead of Y train-
29

---

When the resulting W. is applied to test sample, the result is length-512 prediction of language model representation. In order translate this class prediction, we then apply CLIP in standard zero-shot manner, i.e. we calculate the cosine similarity between the predictions and each of the normalized language model's length 512 representation for the class of each sample.
The resulting final Average Accuracy for M = 5000 and 7' = 10 is 77.5%. In comparison, CLIP's zero shot accuracy for the same data is 68.6%, which highlights there value using the training data to modify the vision model's outputs. When RP is not used, the resulting final Average Accuracy 71.4%.
In future work, we will investigate whether these preliminary results from applying Algorithm 1 to a combination of pre-traing pre-trained vision and language models can translate into demonstrable benefits for continual learning.
F.10
Note on reproducibility
Accuracies reported in Tables 1-3 have been updated since the original submission to reflect those published our code repository at httts://giithtt com/ zhoudw wwt//e/iiiiiinnII. Note that accuracy results achieved can vary by =1% for different random seeds, even for the same class ordering, especially for PETL methods.
30

---

97.5
CT 100
AdsyOLL IBLI
F:
2.0
92.5
(FTHR22311
0.01
0
Gx
Dmn
97.5
6.0-
95.0
VFTH492311
ADSEPILE 18120
VETIBCC
0.0
8.0
90.0
6.0
85.0
AdapPLL 1823K
4.0
-a- AGayOLLF 18100
2.0
0.0
0,
20.0
80.0
15.0
70.0
10.0
60.0
NOYTTPFI 1BL00
5.0
10.0
F. (80
AdkySALF HR2IK
83.0
NOyTTE (BUTIIFIBUO
VIAB-A 251
75.0
60.0
30.0
HRIZI
40.0
NoyTT 181100
0.0
15.0
90.0
(.55)
10.0
80.0
Va (10
5.0
AdEESILI 18120
198 (90.0
12
12
D10
Figure A7: Comparison of PETL Methods and two VIT models. For the seven CIL datasets, and T 10. the figure shows Average Accuracy and Average Forgetting after each task, for each of AdaptMLP, SSF and VPT. It shows this information for the two ViT-B/16 pre-traines backbones we primarily investigated.
31

---

100
NCM
VIT-B/16 finetuned on ImageNet-1K
RP (M=10000)
80
60
30
20
30
40
50
60
70
80
90
100
VIT-B/16 pretrained on ImageNet21K
Figure A8: Comparison backbone ViTs. Scatter plot of results for ViT-B/16 models pre-trained on ImageNetIK vs ImageNet2IK.
32


Reviews:
[{"review_id": "dHRt9iavU7", "reviewer": "Reviewer_cPUm", "summary": "The paper investigates the issue of continual learning using frozen pretrained vision transformers. The authors conduct a thorough analysis of potential limitations and strengths of continual learning methods that utilize pretrained models, supported by theoretical studies and derivations. Additionally, they introduce a novel algorithm called RANPAC, which incorporates a random projection layer with a nonlinear activation function, combined with a Parameter-Efficient Transfer Learning method. To assess the efficacy of their proposed solution, the authors conduct extensive experiments on various datasets such as Cifar and ImageNet, and across different scenarios including Class-Incremental Learning and Domain Incremental Learning.", "questions": "- The lack of an analysis of different PETL methods is a notable gap in the paper. Since Algorithm 1 is designed to work with any PETL method, it would have been interesting to observe and compare the performances of different PETL solutions.\n\n- In section 3.2, the authors discuss the impact of RP, M. However, they fail to demonstrate the impact on performances adequately. It would have been beneficial to compare models with the same phi but varying M from 1 to a significantly larger dimensionality (ideally infinite from the claim of the authors).\n\n- The authors claim that M=2000 is a good choice in section 3.2, but during experimentation, they raised it to 10000. The reason for this discrepancy should be addressed to provide clarity on the choice of M and its impact on the results.\n\n- To further assess the effectiveness of the proposed solution, it would be valuable to study larger first task scenarios, such as 50-10, or more challenging scenarios like 50-5/50-2. ", "limitations": "The authors analyzed the limitations of their method in few lines. before concluding the paper. This section should be definitely expanded.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors have successfully addressed the problem at hand and conducted a comprehensive set of experiments to assess the effectiveness of their proposed solution. The motivation behind their research is evident, and the current trend of expanding foundation models to the continual learning (CL) field makes their work particularly relevant.\n\nBoth the introduction and related works sections are clear and well-organized, providing a solid foundation for the study. The method description is well-supported by theoretical derivations, which are also included in the supplementary material for further clarity.\n\nThe presence of an ablation study to support the obtained results is highly commendable, as it enhances the reliability of their findings. Additionally, the authors' submission of code for transparency and reproducibility is greatly appreciated, further validating the credibility of their research.", "weaknesses": "The paper suffers from issues with clarity and organization. The current structure makes it challenging to follow as the authors have mixed background information with the method, resulting in an unclear narration. To improve the paper's flow, the authors should consider moving the \"Overview and Intuition\" subsection to the beginning of the method section. This would provide readers with a better understanding of the approach before delving into the technical details.\n\nWhile the topic covered is of interest to the CL community, the contribution of the paper appears to be limited. Although the proposed algorithm is supported by derivations and theoretical foundations, it seems like a combination of existing tricks in CL to mitigate the shift towards new classes. As a result, the novelty of the approach is limited.\n\nRegarding Equation 1, the usage of the Gram matrix updated over time, while effective, does not appear to be a strong innovation compared to techniques like LDA.\n\nTable 1 raises concerns about the upper bound, particularly the performances of the joint linear probe. The results are surprising and not entirely convincing.\n\nThe message of figure 2, is not clear and easy to get. The left part of the figure is not clarified in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. We address the points individually and kindly ask the reviewer to increase their score based on our response.\n\n> “...the authors have mixed background information with the method ... should consider moving the \"Overview and Intuition\" subsection to the beginning of the method section.”\n\n**Response:**  We agree with the recommendation and in our final version will introduce an explicit separate Background section, and move material into it.\n\n> “... it seems like a combination of existing tricks in CL to mitigate the shift towards new classes. As a result, the novelty of the approach is limited.” \n\n**Response:** We respectfully disagree. The novelty and primary contribution is the introduction to pretrained models of frozen untrained RP (random projection layer to dimension $M\\gg L$) with nonlinear activation *which has never been done in the continual learning context*. This is *not* an existing trick in CL with deep neural networks.  For the CL community, frozen RP weights is a fresh strategy that may inspire its use within other CL methodologies, because forgetting cannot happen in those weights.\n\n> “…the usage of the Gram matrix updated over time, while effective, does not appear to be a strong innovation...”\n\n**Response:** We agree, which is why Pages 1-2 (lines 91--103) do not list this as a methodological contribution. Three reasons for using the Gram matrix are explained in the Appendix, Sec.B.4. The choice  helps our conceptual contributions illustrated in Fig.2. Moreover, it enables the links to theory in Sec B.3 and B.4.4, that lead to our interpretation of the method as decorrelating classprototypes (which is absent in NCM) and our use of ridge regression (important for our empirical results).\n\n\n> Surprising upper bound results in Table 1.\n\n**Response:**  The \"upper bound\" for continual learning performance is *joint fine-tuning*. Our results for joint fine tuning are in Table 2. In Table 1, we report only joint linear probe results (with frozen backbone), and these are not an \"upper bound.\" We will clarify this when we mention upper bound on line 97, and within the Table captions.\n\n\n>  “The message of figure 2, is not clear and easy to get. The left part of the figure is not clarified in the text.”\n\n**Response:** Fig.2 shows that our method reduces correlations between the class-prototypes of different classes (right side), thus creating better class separability (left side). We will add this to the caption. The left side of Fig.2 is explained on Page 4, lines 175-182, which is the next paragraph after the right side was explained. We will join these paras together.\n\n> **Question 1:** “...compare the performances of different PETL solutions...\"\n\n**Response:** Analysis of the 3 PETL methods is provided for 7 datasets in the Appendix, Sec.F.6 (lines 766-771 for analysis) and Fig.SM7 shows comprehensive comparisions. We followed ref [51] for PETL. Although our overall methods are different to [51], like them, we found that the best performing PETL method is very dependent on the dataset (and pretrained weights). E.g. we found for CIFAR100 and $T=10$ that AdaptMLP gives better results than SSF or VPT, for Cars VPT is best, and for ImageNet-A, SSF is best (lines 766-771).\n\n> **Question 2:** “It would have been beneficial to compare models with the same phi but varying M from 1 to a significantly larger dimensionality....”\n\n**Response:** We explore the scaling with $M$ in the Appendix, Sec.F.5. Table 5 shows results for split CIFAR100 for $M$ increasing from $100$ to $15000$ for the same $\\phi$. The table shows diminishing returns as M goes past ~$5000$. Also, Fig 3 (left) shows data for $M=5000$, $10000$, and $15000$ with the same $\\phi$.\n\n> **Question 3:** “The authors claim that M=2000 is a good choice in section 3.2, but during experimentation, they raised it to 10000. The reason for this discrepancy should be addressed to provide clarity on the choice of M and its impact on the results”\n\n**Response:** In Sec.3.2, we arbitrarily chose $M=2000$ solely for the purpose of illustration in Fig.2. We made no claim about this choice being “good”. For consistency, we will update Fig.2 to use $M=10000$, but there are no perceptible differences in this visualization.\n\n> **Question 4:** “...it would be valuable to study larger first task scenarios, such as 50-10, or more challenging scenarios like 50-5/50-2.”\n\n**Response:** In the Appendix, Sec.F.3, we compare $T=5,~T=10$ and $T=20$. The $T=5$ scenario produces a larger first task. Table 4 shows that performance is better when the first task is larger. The reason is that Phase 1 adapts the pretrained model to a larger chunk of the overall dataset. Also note the column “No PETL” in Table 4 that stands alone without a choice of $T$. This is because Phase 2 has no dependency on the size of task increments. We mention this on lines 242--243, and with further details in Sec.F.3, i.e. in Phase 2, final accuracy is invariant to the order in which a completed set of data from all $T$ tasks is used. This property follows from Eqn (3), the use of frozen untrained RP weights, and the use of class-prototypes. This combination enables one class at a time to be added, with prediction outputs for that class unaffected by the subsequent addition of more classes.\n\n> “The authors analyzed the limitations of their method in few lines. before concluding the paper. This section should be definitely expanded.”\n\n**Author Response:** Thanks for the comments. We will expand the discussion of limitations in the camera-ready version."}, {"review_id": "ARpCR2YsK6", "reviewer": "Reviewer_2MfY", "summary": "The authors propose a method for replay-free continual learning from a pre-trained model based on random projections and prototypes. The method has a high parameter cost compared to SOTA prompting-based methods, but also is a unique method which strongly outperforms these SOTA methods. Overall, the experiments and analysis on class-incremental and domain-incremental learning provide a strong motivation for the proposed approach.", "questions": "a) How does the method perform on longer task sequences? For example, 20-tasks of ImageNet-R? There is one table in the SM with 20 tasks, but I would rather see something comparing to the other methods from Table 1.\nb) Thank you for being transparent in your experiment section about the parameter costs of your method. How does the additional parameter costs compare to other methods?\nc) How does computation costs compare (both training time costs and inference time costs) to other methods?", "limitations": "Reasonable discussion on limitations is included.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) I enjoyed the writing style of this paper. It was clear and scientific.\n2) I appreciate the authors thinking outside of the box. Rather than another prompting method, they propose a clever new direction with high motivation and experimental justification. The method has good intuition and motivation for the problem setting.\n3) The experiment section is very comprehensive and mostly satisfying.\n4) I explored the SM and am very happy with all of the additional details and experiments provided by the authors.", "weaknesses": "1) There seems to be large number of additional trainable parameters. While for 10 tasks, this is only 10/84 of the model size, it could double the necessary parameters for a longer task sequence.\n2) Speaking of task sequences, I would like to see how the method performs for longer task sequences. A simple and easy experiment would be 20 task ImageNet-R.\n3) Computation time analysis (training and inference) seems to be missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. We address the points individually and kindly ask the reviewer to increase their score based on our response.\n\n> “There seems to be large number of additional trainable parameters. While for 10 tasks, this is only 10/84 of the model size, it could double the necessary parameters for a longer task sequence.”\n\n**Response:** The mentioned 10/84 relates to the $LM$ untrained RP weights. This does not change as the number of tasks or classes grows. For DIL (Table 3), there is no parameter growth with tasks. For CIL, the number of parameters in the output head will double for any method (not just ours), if the number of classes doubles. Our weights per class is larger due to expansion from $L$ to $M$, but when using RP with $M=10000$, we would need ~4200 classes to reach half the ViT-B/16 size. Smaller $M$ can suffice, however, as indicated in Table SM5.\n\n\n\n> **Question a):** “How does the method perform on longer task sequences? For example, 20-tasks of ImageNet-R? There is one table in the SM with 20 tasks, but I would rather see something comparing to the other methods from Table 1”\n\n**Response:** We agree a comparison of $T=20$ for more methods will be valuable. The following table compares our approach to results extracted from ref [51] and will be added into Table SM4:\n\n| Method     | CIFAR100 | IN-R  | IN-A  | CUB   | OB    |\n| ---------- | -------- | ----- | ----- | ----- | ----- |\n| Ours       | **90.8**     | **75.4**  | **58.9**  | **89.7**  | **79.4**  |\n| L2P        | 70.96    | 56.25 | 40.71 | 58.23 | 60.19 |\n| DualPrompt | 72.04    | 69.25 | 40.95 | 67.46 | 64.39 |\n| ADaM       | 89.67    | 70.47 | 51.48 | 86.7  | 73.53 |\n\n\n\n> **Question b):** “How does the additional parameter costs compare to other methods?”\n\n**Response:** In comparison methods, L2P, DualPrompt and ADaM each use ~0.3M-0.5M parameters, while CodaPrompt uses ~4M parameters. SLCA trains all 84M ViT parameters. In comparison, for $K=200$ classes and $M=10000$ we use between 2M and 2.5M trainable parameters (depending on the PETL method), and 10M untrained parameters. We need to highlight that the random projections are not trainable. Therefore, the training overhead compared to other approaches is the difference in the size of the projection space from $L$ to $M$. We believe this increase in dimensionality is a trade-off for simplicity of implementation and low-cost training (e.g. the extracted features can be cached). \n\n> **Question c):** “How does computation costs compare (both training time costs and inference time costs) to other methods?\"\n\n**Response:**  Inference speed is negligibly different to the speed of the original pretrained network, because both the RP layer and the final linear head are implemented as simple fully-connected layers on top of the underlying network. For training, Phase 1 trains PETL parameters using SGD for 20 epochs, on ($1/T$)'th of the training set, so is much faster than joint training. Phase 2 is generally only slightly slower than running all training data through the network in inference mode, because the backbone is frozen. The slowest part is the inversions of the Gram matrix, during selection of $\\lambda$, but even for $M=10000$ this is in the order of 1 minute per task on a CPU, which can be easily sped up. In general, we believe the efficiency and simplicity of our approach compared to the alternatives is very strong. We will add remarks on this to the final version."}, {"review_id": "mzBKcNeXkx", "reviewer": "Reviewer_Q6EM", "summary": "The manuscript proposes a Continual Learning method called RanPAC, which belongs to the category of class prototype methods. They use a frozen pretrained model to extract feature vectors from the input images and non-linear random projections to project them to a higher dimensional space. During the training on the task sequence, they continuously update the gram matrix obtained from the projected representations and use that matrix to make predictions. Through mathematical proofs, they show that the distribution of the projected features in the higher dimensional space approaches an isotropic Gaussian. The experimental results in the paper show that their method is superior in accuracy to other state-of-the-art Continual Learning techniques and approaches to the upper bound performance. ", "questions": "The contribution of the work and the results show are valid. However, as I said in the previous sections, I believe that some parts of the paper could be improved and explained better.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea to investigate the role of random projections of feature vectors obtained from large pretrained models is new in Continual Learning research. The experimental evidence that random projections help in obtaining low correlation coefficients between class prototypes is surely a useful contribution.\n", "weaknesses": "I believe the paper has the following minor weaknesses:\n- Page 4, section 2.2, rows 145-161: The paper presents the LDA formula and subsequently mentions that the authors employed the Gram matrix. Nevertheless, it was not clear to me the motivation behind this choice.\n- Page 5, figure 2: I didn't quite understand the plots in Figure 2, left. They are introduced in Section 2, and it is explained that they represent the similarities of class prototypes. However, the meaning of \"true class\" and \"inter-class\" was not very clear to me.\n- Page 5-6, sections 3.2 and 3.3: I found the mathematical parts hard to understand. I think it would be more beneficial to provide simplified explanations of the equations, using simpler language to clarify their meaning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. We address the points individually and kindly ask the reviewer to increase their score based on our response.\n\n> \"The paper presents the LDA formula and subsequently mentions that the authors employed the Gram matrix. Nevertheless, it was not clear to me the motivation behind this choice.\"\n\n**Response:** This is explained in detail in the Appendix, Sec.B.4, where 3 reasons are listed. In the final version we will clarify by inserting a summary in the Approach section.\n\n> \"...the meaning of \"true class\" and \"inter-class\" was not very clear to me.\"\n\n**Response:** “True class” refers to the histogram of cosine similarities between a sample’s feature vector and the class prototype for the class label corresponding to that sample. “Inter-class” refers to the histogram of cosine similarities between a sample’s feature vector and the class prototypes for the set of N-1 classes not equal to the sample’s class label. We will clarify in the camera-ready.\n\n> \"I found the mathematical parts hard to understand. I think it would be more beneficial to provide simplified explanations of the equations, using simpler language to clarify their meaning.\"\n\n**Response:** Appendix B (Sections B1-B4), expand on the details of all the mathematical concepts.  We will add clarifying remarks to enhance readability in the camera-ready version."}, {"review_id": "l7Qg0xTrO5", "reviewer": "Reviewer_b1mf", "summary": "This paper proposed a frozen random projection layer with nonlinear activation to exploit pre-trained representations for continual learning. Combining PETL techniques and class prototypes, the proposed method achieves strong performance in class- and domain-incremental learning.", "questions": "My major concerns include the novelty and technical contributions of the proposed method, as well as the parameter usage in experiments. Please refer to Weakness.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized and easy to follow.\n2. The introduction clearly summarizes the major strategies of leveraging pre-training for continual learning.\n3. The proposed method seems to be easy to implement and achieves strong performance.\n4. Many recent methods on arxiv have been compared in experiments.", "weaknesses": "1. The idea of the proposed method is somewhat incremental. The authors claim that they borrow some ideas from PETL methods. Actually, the Phase 1 seems to (almost) inherit the previous works (line 273-284) and constitute half of the performance improvement (Table 1, 3). \n2. Meanwhile, the proposed random projection with nonlinear activation seems equivalent to a randomly-initialized MLP layer, which is a widely-used training trick for transformer backbone in NLP as well as in CV.\n3. The authors claim that the extra parameter usage is small compared to the ViT-B/16 backbone (10M vs 84M). However, the compared baselines in Table 1, 3 generally require much smaller parameter usage, especially for prompt-based approaches (e.g., L2P and DualPrompt usually require ~0.03M/Task and 0.3M in total). Therefore, the comparison might be somewhat unfair. Also, how is ''10 times the number of trainable parameters'' (line 302) being calculated?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. We address the points individually and kindly ask the reviewer to increase their score based on our response.\n\n> “The authors claim that they borrow some ideas from PETL methods. Actually, the Phase 1 seems to (almost) inherit the previous works (line 273-284) and constitute half of the performance improvement (Table 1, 3).”\n\n**Response:** We have been explicit in the use of PETL as methods that our approach can benefit from. Our primary contribution is introducing random projections (RP) for continual learning (see lines 92-103, pages 1-2). Therefore, we can either choose to use PETL or not. The second-last row of Tables 1 and 3 list the performance gains due to RP added on top of PETL, and shows at least 10\\% improvement in 9 out of 10 datasets, of which 4 cases are over 20\\% improved. Also, the final row of Tables 1 and 3 shows the improvement when PETL is absent, and this is also over 10\\% in 6 out of 10 datasets, illustrating that the method does not need PETL to be of value.\n\n> “… the proposed random projection with nonlinear activation seems equivalent to a randomly-initialized MLP layer, which is a widely-used training trick for transformer backbone in NLP as well as in CV.\"\n\n**Response:** Our RP layer's weights are randomly sampled from the specified distribution and *never trained*, which is very important for CL. This is different to random initialization of *trained* MLP layers. For instance, Gani *et al.* (2022) randomly initialised but then trained MLP weights appended to a pretrained NLP transformer (https://arxiv.org/abs/2210.07240). Please refer us to any specific papers that, like us, do leave random weights frozen. \n\n> \"The authors claim that the extra parameter usage is small compared to the ViT-B/16 backbone (10M vs 84M). However, the compared baselines in Table 1, 3 generally require much smaller parameter usage...\"\n\n**Response:** Our 10M *non-trainable* parameters are not used in Phase 1 and only ever require forward pass and are never back-propagated through during Phase 2 training, so our training is very fast.\n\n>  \"how is ''10 times the number of trainable parameters'' (line 302) being calculated?\"\n\n**Response:** Without RP, the number of trainable parameters after $K$ classes is $LK$. When using RP, this increases to $MK$. The increase factor is $(M-L)/L$, i.e. ~10 when $M=10000$ and $L=784$ (but note that M is often effective for smaller values)."}]

Author rebuttal:

Meta-review: After discussion and rebuttal, the paper had mainly positive scores (4,5,7,7). The AC agrees with the positive reviewers and recommends acceptance. The authors should include the answers in preparing for the final version, especially the longer task sequence results and improved discussion on #Params and computational costs.
Decision: Accept (poster)