SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mark D. McDonnell<sup>1</sup>, Dong Gong<sup>2</sup>, Amin Parveneh<sup>1</sup>, Ehsan Abbasnejad<sup>1</sup> and Anton van den Hengel<sup>1</sup>
$^{1}$ Australian Institute for Machine Learning, The University of Adelaide  $^{2}$ School of Computer Science and Engineering, University of New South Wales
mark.mcdonnell@adelaide.edu.au, dong.gong@unsw.edu.au, amin.parvaneh@adelaide.edu.au, ehsan.abbasnejad@adelaide.edu.au, anton.vandenhengel@adelaide.edu.au
Continual learning (CL) aims to incrementally learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. Most CL works focus on tackling catastrophic forgetting under a learning-from-scratch paradigm. However, with the increasing prominence of foundation models, pretrained models equipped with informative representations have become available for various downstream requirements. Several CL methods based on pre-trained models have been explored, either utilizing pre-extracted features directly (which makes bridging distribution gaps challenging) or incorporating adaptors (which may be subject to forgetting). In this paper, we propose a concise and effective approach for CL with pre-trained models. Given that forgetting occurs during parameter updating, we contemplate an alternative approach that exploits training-free random projectors and class-prototype accumulation, which thus bypasses the issue. Specifically, we inject a frozen Random Projection layer with nonlinear activation between the pre-trained model's feature representations and output head, which captures interactions between features with expanded dimensionality, providing enhanced linear separability for class-prototype-based CL. We also demonstrate the importance of decorrelating the class-prototypes to reduce the distribution disparity when using pre-trained representations. These techniques prove to be effective and circumvent the problem of forgetting for both class- and domain-incremental continual learning. Compared to previous methods applied to pre-trained ViT-B/16 models, we reduce final error rates by between  $20\%$  and  $62\%$  on seven class-incremental benchmark datasets, despite not using any rehearsal memory. We conclude that the full potential of pre-trained models for simple, effective, and fast continual learning has not hitherto been fully tapped. Code is available at https://github.com/RanPAC/RanPAC.
Continual Learning (CL) is the subfield of machine learning within which models must learn from a distribution of training samples and/or supervision signals that change over time (often divided into a distinct set of  $T$  episodes/tasks/stages) while remaining performant on anything learned previously during training [51, 53]. Traditional training methods do not work well for CL because parameter updates become biased to newer samples, overwriting what was learned previously. Moreover, training on sequential disjoint sets of data means there is no opportunity to learn differences between samples from different stages [48]. These effects are often characterised as 'catastrophic forgetting' [51].
Although many methods for CL have been proposed [51, 53, 2, 60], most focus on models that need to be trained from scratch, with resulting performance falling short of that achievable by non-CL alternatives on the same datasets. Although valid use cases for training from scratch will always exist, the new era of large foundation models has led to growing interest in combining CL with the advantages of powerful pre-trained models, namely the assumption that a strong generic feature-extractor can be adapted using fine-tuning to any number of downstream tasks.
Although forgetting still occurs in CL with such transfer-learning [57, 65, 63] (whether using conventional fine-tuning [24] or more-recent Parameter-Efficient Transfer Learning (PETL) methods such as [6, 28, 21]), commencing CL with a powerful feature-extractor has opened up new ideas for avoiding forgetting that are unlikely to work when training from scratch. Such new methods have been applied successfully to pre-trained transformer networks [39, 10, 57, 56, 46, 20, 65, 63, 47, 55], CNNs [11, 15, 33, 22, 59, 37] and multi-modal vision-language models [8].
Three main strategies are evident in these proposals (see Section 2) for details): (i) prompting of transformer networks; (ii) careful selective fine-tuning of a pre-trained model's parameters; and (iii) Class-Prototype (CP) accumulation. Common to all these strategies is the absence of a need for a buffer of rehearsal samples from past tasks, unlike the best CL methods for training models from scratch. Instead, these strategies leverage a pre-trained model's strong feature extraction capabilities.
Each strategy has yielded empirically comparable performance when used with the same benchmarks and pre-trained model (e.g. a ViT B/16 transformer network [46, 63, 65]). It therefore remains open as to what strategy best leverages pre-trained foundation models for CL, in terms of performance on diverse datasets and CL scenarios, simplicity, and efficiency. However, we note that the CP-based CL strategy is simple to apply to both CNNs and transformer networks, whereas prompting methods rely on a prepending learnable prompts to transformer network inputs. Fine-tuning a pre-trained model's parameters requires more resources for training than the other two strategies, while carrying a greater risk of cumulative forgetting over time, thus requiring use of additional CL methodologies.
In this paper, we show that the CP strategy has not come close to exhausting its capacity for accuracy, and can achieve standout performance with carefully tailored strategies to enhance the extracted feature representations from the pre-trained models. CP-based methods use only the prototypes obtained by averaging the extracted features to represent each class, subject to a discrepancy with the data distributions in practice. We propose to handle this via training-free frozen random projections and a decorrelation process, both of which bypass the forgetting issue. Specifically, we introduce a Random-Projection (RP) layer of frozen untrained weights, with nonlinear activation, between the pre-trained-model's feature layer, and a CP-based output head.
Nonlinear Random Projection (RP) of Features: The original RP idea (see Figure 1) has been proposed [43, 4, 18] and applied (e.g. [32]) multiple times in the past, as a standalone non-continual learner. Our two-fold motivation for using the same method for CL differs from the method's origins (outlined in Section 2). Firstly, we show that past CP strategies for CL with pre-trained models are equivalent to the linear models learned following RP in the past, e.g. [43, 4, 18, 32]. Secondly, the frozen untrained/training-free weights do not cause forgetting in CL. These two observations suggest using the RP idea alongside CP strategies for CL.
However, why should RPs provide any value instead of trying to learn a layer of fully-connected weights? It is well known that linear models can benefit from transformations that create non
Figure 1: The RP method can lead to a representation space with clear class separation. Colored points are 2D t-SNE visualizations for CIFAR-100 classes with features from a pre-trained ViT-B/16 transformer network.
linear feature interactions. Our hypothesis in proposing to use RP followed by nonlinear-activation for CL, therefore, is that the transformed set of features is more linearly separable using a CP method than features directly extracted from a pre-trained model. The past work of [43, 4, 18, 32] leads us to expect that for this to be confirmed, RP will need to increase dimensionality from  $L$  to  $M > L$ .
Our contributions are summarised as follows:
1. We examine in detail the CP strategy for CL with pre-trained networks, show that it benefits from injecting a Random Projection layer followed by nonlinear activation, and illustrate why. We also analyse why it is important to follow the lead of [37] to linearly transform CPs, via decorrelation using second-order feature statistics.
2. We show that random projections are particularly useful when also using PETL methods with first-session training (see Section 2.1). Accuracy with this combination approaches the CL joint training upper bound on some datasets (Table 2). For ViT-B/16 pre-trained models, we report the highest-to-date rehearsal-free CL accuracies on all class-incremental and domain-incremental datasets we tested on, with large margins when compared to past CP strategies.
3. We highlight the flexibility of our resulting CL algorithm, RanPAC; it works with arbitrary feature vectors (e.g. ViT, ResNet, CLIP), and is applicable to diverse CL scenarios including class-incremental (Section 5), domain-incremental (Section 5) and task-agnostic CL (Appendix F).
2.1 Three strategies for CL with strong pre-trained models
Prompting of transformer networks: Using a ViT-B/16 network [9], Learning To Prompt (L2P) [57], and DualPrompt [56] reported large improvements over the best CL methods that do not leverage pre-trained models, by training a small pool of prompts that update through the CL process. CODA-Prompt [46], S-Prompt [55] and PromptFusion [5] then built on these, showing improvements in performance.
Careful fine-tuning of the backbone: SLCA [63] found superior accuracy to prompt strategies by fine-tuning a ViT backbone with a lower learning rate than in a classifier head. However, it was found that use of softmax necessitated introduction of a 'classifier alignment' method, which incurs a high memory cost, in the form of a feature covariance matrix for every class. Another example of this strategy used selected fine-tuning of some ViT attention blocks [47], combined with traditional CL method, L2 parameter regularization. Fine-tuning was also applied to the CLIP vision-language model, combined with well-established CL method, LwF [8].
Class-Prototype (CP) accumulation: Subsequent to L2P, it was pointed out for CL image classifiers [20] that comparable performance can be achieved by appending a nearest class mean (NCM) classifier to a ViT model's feature outputs (see also [39]). This strategy can be significantly boosted by combining with Parameter-Efficient Transfer Learning (PETL) methods (originally proposed for NLP models in a non-CL context [17, 12]) trained only on the first CL stage ('first-session training') to bridge any domain gap [65, 37]. The three PETL methods considered by [65] for transformer networks, and the FiLM method used by [37] for CNNs have in common with the first strategy (prompting) that they require learning of new parameters, but avoid updating any parameters of the backbone pre-trained network. Importantly, [37] also showed that a simple NCM classifier is easily surpassed in accuracy by also accumulating the covariance matrix of embedding features, and learning a linear classifier head based on linear discriminant analysis (LDA) [35]. The simple and computationally lightweight algorithm of [37] enables CL to proceed after the first session in a perfect manner relative to the union of all training episodes, with the possibility of catastrophic forgetting avoided entirely.
CPs are well suited to CL generally [42, 7, 31, 16] and for application to pre-trained models [20, 65, 37], because when the model from which feature vectors are extracted is frozen, CPs accumulated across  $T$  tasks will be identical regardless of the ordering of the tasks. Moreover, their memory cost is low compared with using a rehearsal buffer, the strategy integral to many CL methods [53].
2.2 RP method for creating feature interactions
As mentioned, the original non-CL usage of a frozen RP layer followed by nonlinear projection as in [43, 4, 18] had different motivations to us, characterized by the following three properties. First, keeping weights frozen removes the computational cost of training them. Second, when combined with a linear output layer, the mean-square-error-optimal output weights can be learned by exact numerical computation using all training data simultaneously (see Appendix B.3) instead of iteratively. Third, nonlinearly activating random projections of randomly projected features is motivated by the
assumption that nonlinear random interactions between features may be more linearly separable than the original features. Analysis of the special case of pair-wise interactions induced by nonlinearity can be found in [32], and mathematical properties for general nonlinearities (with higher order interactions) have also been discussed extensively, e.g. [4, 18].
3.1 Continual learning problem setup
We assume the usual supervised CL setup of a sequence of  $T$  tasks/stages,  $\mathcal{D} = \{\mathcal{D}_1,\dots \mathcal{D}_T\}$ . In each  $\mathcal{D}_t$ , a disjoint set of training data paired with their corresponding labels is provided for learning. Subsequent stages cannot access older data. We primarily consider both 'Class-Incremental Learning' (CIL) and 'Domain-Incremental learning' (DIL) protocols [51] for classification of images. In CIL, the class labels for each  $\mathcal{D}_t$  are disjoint. One of the challenging aspects of CIL is that, in contrast to Task-Incremental Learning (TIL), the task identity and, consequently, the class subset of each sample is unknown during CIL inference [51]. In DIL, while all stages typically share the same set of classes, there is a distribution shift between samples appearing in each stage. For example,  $\mathcal{D}_1$  may include photographs and  $\mathcal{D}_2$  images of paintings.
We introduce  $K$  as the total number of classes considered within  $T$  tasks, and the number of training samples in each task as  $N_{t}$  with  $N\coloneqq \sum_{t = 1}^{T}N_{t}$ . For the  $n$ -th unique training sample within task  $\mathcal{D}_t$ , we use  $\mathbf{y}_{t,n}$  as its length  $K$  one-hot encoded label and  $\mathbf{f}_{t,n}\in \mathbb{R}^L$  to denote features extracted from a frozen pre-trained model. We denote by  $\mathbf{f}_{\mathrm{test}}$  the encoded features for a test instance for which we seek to predict labels.
3.2 Class-Prototype strategies for CL with pre-trained models
For CL, using conventional cross-entropy loss by linear probing or fine-tuning the feature representations of a frozen pre-trained model creates risks of task-recency bias [31] and catastrophic forgetting. Benefiting from the high-quality representations of a pre-trained model, the most straightforward Class-Prototype (CP) strategy is to use Nearest Class Mean (NCM) classifiers [58, 34], as applied and investigated by [65, 20]. CPs for each class are usually constructed by averaging the extracted feature vectors over training samples within classes, which we denote for class  $y$  as  $\overline{\mathbf{c}}_y$ . In inference, the class of a test sample is determined by finding the highest similarity between its representation and the set of CPs. For example, [65] use cosine similarity to find the predicted class for a test sample,
$$
y _ {\text {t e s t}} = \underset {y ^ {\prime} \in \{1, \dots , K \}} {\arg \max } s _ {y ^ {\prime}}, \quad s _ {y} := \frac {\mathbf {f} _ {\text {t e s t}} ^ {\top} \overline {{\mathbf {c}}} _ {y}}{| | \mathbf {f} _ {\text {t e s t}} | | \cdot | | \overline {{\mathbf {c}}} _ {y} | |}. \tag {1}
$$
However, it is also not difficult to go beyond NCM within the same general CL strategy, by leveraging second-order feature statistics [11, 37]. For example, [37] finds consistently better CL results with pretrained CNNs than NCM using an incremental version [36, 11] of Linear Discriminant Analysis (LDA) classification [35], in which the covariance matrix of the extracted features is continually updated. Under mild assumptions, LDA is equivalent to comparing feature vectors and class-prototypes using Mahalanobis distance (see Appendix B.4), i.e. different to cosine distance used by [65].
3.3 CP-based classification using Gram matrix inversion
We will also use incrementally calculated second-order feature statistics, but create a simplification compared with LDA (see Appendix B.4), by using the Gram matrix of the features,  $\mathbf{G}$ , and  $\mathbf{c}_y$  (CPs with averaging dropped), to obtain the predicted label
$$
y _ {\text {t e s t}} = \underset {y ^ {\prime} \in \{1, \dots , K \}} {\arg \max } s _ {y ^ {\prime}}, \quad s _ {y} := \mathbf {f} _ {\text {t e s t}} ^ {\top} \mathbf {G} ^ {- 1} \mathbf {c} _ {y}. \tag {2}
$$
Like LDA, but unlike cosine similarity, this form makes use of a training set to 'calibrate' similarities. This objective has a basis in long established theory for least square error predictions of one-hot encoded class labels [35] (see Appendix B.3). Similar to incremental LDA [36, 11], during CL training, we describe in Section 4.3 how the Gram matrix and the CPs corresponding to  $\mathbf{c}_y$  can easily be updated progressively with each task. Note that Eqn. (2) is expressed in terms of the maximum number of classes after  $T$  tasks,  $K$ . However, for CIL, it can be calculated after completion of tasks  $t < T$  with fewer classes than  $K$ . For DIL, all  $K$  classes are often available in all tasks.
4 The proposed approach and theoretical insights: RanPAC
4.1 Why second order statistics matter for CPs
We show in Section 5 that Eqn. (2) leads to better results than NCM. We attribute this to the fact that raw CPs are often highly correlated between classes, resulting in poorly calibrated cosine similarities, whereas the use of LDA or Eqn (2) mostly removes correlations between CPs, creating better separability between classes. To illustrate these insights, we use the example of a ViT-B/16 transformer model [9] pre-trained on ImageNet-21K with its classifier head removed, and data from the well-established 200-class Split Imagenet-R CIL benchmark [56].
For comparison with CP approaches, we jointly trained, on all 200 classes, a linear probe softmax classifier. We treat the weights of the joint probe as class-prototypes for this exercise and then find the Pearson correlation coefficients between each pair of prototypes as shown for the first 10 classes in Fig. 2 (right). Compared with the linear probe, very high off-diagonal correlations are clearly observed when using NCM, with a mean value more than twice that of the original ImageNet-1K training data treated in the same way. This illustrates the extent of the domain shift for the downstream dataset. However, these correlations are mostly removed when using Eqn (2). Fig. 2 (left) shows that high correlation coefficients coincide with poorly calibrated cosine similarities between class-prototypes and training samples, both for true class comparisons (similarities between a sample's feature vector and the CP for the class label corresponding to that sample.) and inter-class comparisons (similarities between a sample's feature vector and the class prototypes for the set of N-1 classes not equal to the sample's class label). However, when using Eqn. (2), the result (third row) is to increase the training-set accuracy from  $64\%$  to  $75\%$ , coinciding with reduced overlap between inter- and true-class similarity distributions, and significantly reduced off-diagonal correlation coefficients between CPs. The net result is linear classification weights that are much closer to those produced by the jointly-trained linear probe. These results are consistent with known mathematical relationships between Eqn (2) and decorrelation, which we outline in Appendix B.4.4.
4.2 Detailed overview and intuition for Random Projections
CP methods that use raw  $\bar{c}_y$  for CL assume a Gaussian distribution with isotropic covariance. Empirically, we have found that when used with pre-trained models this assumption is invalid (Figure 2). One can learn a non-linear function (e.g. using SGD training of a neural network) with
Figure 2: Left: Histograms of similarities between class-prototypes and feature vectors from the ViT-B/16 transformer model pre-trained on ImageNet-1K, for the training samples of theImagenet-R dataset. Right: Pearson correlation coefficients (CCs) for 10 pairs of class-prototypes. Reduced correlations between CPs of different classes (right), coincides with better class separability (left).
high capacity and non-convex objective to alleviate this gap (e.g., learning to adapt on the first task as in [65, 37]), but that requires additional assumptions and architectural choices. In addition for CL, any learning on all tasks carries a high risk of catastrophic forgetting. Instead, we take advantage of the observation that the objectives in CP strategies principally require firstly feature projections (that could be possibly adjusted for their distribution by a covariance matrix) and secondary the feature prototypes. For the latter, since using large pre-trained models produce features that capture most prominent patterns and attributes, the average is a reasonable expected class representative. However, the feature representations may also benefit from nonlinear transformation to enhance linear separability. In this paper, we consider two points: (i) we show a random projection of the features in the embedding space (or their projection to a random basis) to a higher dimensional space leads to a distribution that is more likely to be suitable for a Gaussian fit (e.g. using Eqn. (2)); and, (ii) when the distribution of representations is nearly Gaussian, we can simply perform a linear regression (or LDA or a conditional mean of the joint Gaussian distribution).
This assertion is supported by the bottom row of Fig. 2, which shows that the application of an RP layer of size  $M = 2000$  (chosen arbitrarily, for the purpose of illustration) reduces the overlap between the similarity histograms further than otherwise, and shifts the in-class histogram to the right, coinciding with higher accuracy, surpassing that of the joint linear probe, consistent with our findings in Section 5. We note that these RPs can be generated once and then frozen for use in all continual learning stages, enabling a simple and efficient algorithm.
We first analyse the inner products of the features obtained from the pre-trained model projected using some random basis vectors (i.e. random projections). We consider  $M$  such random vectors all randomly drawn i.i.d from a Gaussian distribution with variance  $\sigma^2$ . The number  $M$  defines the dimension of the space the random projections of these features constitute. Considering random projections  $\mathbf{W} \in \mathbb{R}^{L \times M}$ , for any two given feature vectors  $\mathbf{f}, \mathbf{f}' \in \mathbb{R}^L$  we have (see Appendix B.2),
$$
\mathbb {E} _ {\mathbf {W}} \left[ \left(\mathbf {W} ^ {\top} \mathbf {f}\right) ^ {\top} \left(\mathbf {W} ^ {\top} \mathbf {f} ^ {\prime}\right) \right] = \sum_ {i} ^ {M} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(i)} ^ {2} \right] \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime} + \sum_ {i \neq j} ^ {M} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(i)} \right] ^ {\top} \mathbb {E} _ {\mathbf {W}} \left[ \mathbf {W} _ {(j)} \right] \mathbf {f} ^ {\top} \mathbf {f} ^ {\prime}, \tag {3}
$$
where  $\mathbf{W}_{(i)}$  denotes the  $i$ th column. That is, the expected inner products of projections of these two features can be decomposed. Now, we have  $\mathbf{W}_{(i)} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$ , thus  $\mathbb{E}_{\mathbf{W}}[\mathbf{W}_{(i)}] = \mathbb{E}_{\mathbf{W}}[\mathbf{W}_{(j)}] = 0$  and the second term in Eqn. (3) vanishes. Further,  $\sum_{i}^{M}\mathbb{E}_{\mathbf{W}}[\mathbf{W}_{(i)}^2] = M\sigma^2$ . We can make two observations (theoretical details are provided in Appendix B.2):
1. As  $M$  increases, the likelihood that the norm of any projected feature approaching the variance increases. In other words, the projected vectors in higher dimensions almost surely reside on the boundary of the distribution with almost equal distance to the mean (the distribution approaches isotropic Gaussian).
2. As  $M$  increases, it is more likely for angles between two randomly projected instances to be distinct (i.e. the inner products in the projected space are more likely to be larger than some constant).