SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This discussion can readily be extended to incorporate nonlinearity. The benefit of incorporating nonlinearity is to (i) incorporate interaction terms [54, 50], and, (ii) simulate higher dimensional projections that otherwise could be prohibitively large in number. To see the later point, denoting by  $\phi$  the nonlinearity of interest, we have  $\phi (\mathbf{W}^{\top}\mathbf{f})\approx \hat{\mathbf{W}}^{\top}\hat{\mathbf{f}}$  where  $\hat{\mathbf{f}}$  is obtained from a linear expansion using Taylor series and  $\hat{\mathbf{W}}$  is the corresponding projections. The Taylor expansion of the nonlinear function  $\phi$  gives rise to higher order interactions between dimensions. Although vectors of interaction terms can be formed directly, as in the methods of [38, 54], this is computationally prohibitive for non-trivial  $L$ . Hence, the use of nonlinear projections of the form  $\mathbf{h}_{\mathrm{test}}\coloneqq \phi (\mathbf{f}_{\mathrm{test}}^{\top}\mathbf{W})$  is a convenient alternative, as known to work effectively in a non-CL context [43, 4, 18, 32].
4.3 Random projection for continual learning
Using the random projection discussed above, with  $\phi(\cdot)$  as an element-wise nonlinear activation function, given feature sample  $\mathbf{f}_{t,n}$  we obtain length  $M$  representations for CL training in each task,  $\mathbf{h}_{t,n} \coloneqq \phi(\mathbf{f}_{t,n}^{\top} \mathbf{W})$  (Fig. 1). For inference,  $\mathbf{h}_{\mathrm{test}} \coloneqq \phi(\mathbf{f}_{\mathrm{test}}^{\top} \mathbf{W})$  is used in  $s_y$  in Eqn. (2) instead of  $\mathbf{f}_{\mathrm{test}}$ . We define  $\mathbf{H}$  as an  $M \times N$  matrix in which columns are formed from all  $\mathbf{h}_{k,n}$  and for convenience refer to only the final  $\mathbf{H}$  after all  $N$  samples are used. We now have an  $M \times M$  Gram
matrix for the features,  $\mathbf{G} = \mathbf{H}\mathbf{H}^{\top}$ . The random projections discussed above are sampled once and left frozen throughout all CL stages.
Like the covariance matrix updates in streaming LDA applied to CL [11, 37], variables are updated either for individual samples, or one entire CL stage,  $\mathcal{D}_t$ , at a time. We introduce matrix  $\mathbf{C}$  to denote the concatenated column vectors of all the  $\mathbf{c}_y$ . Rather than covariance,  $\mathbf{S}$ , we update the Gram matrix, and the CPs in  $\mathbf{C}$  (the concatenation of  $\mathbf{c}_y$ 's) iteratively. This can be achieved either one task at a time, or one sample at a time, since we can express  $\mathbf{G}$  and  $\mathbf{C}$  as summations over outer products as
$$
\mathbf {G} = \sum_ {t = 1} ^ {T} \sum_ {n = 1} ^ {N _ {t}} \mathbf {h} _ {t, n} \otimes \mathbf {h} _ {t, n}, \quad \mathbf {C} = \sum_ {t = 1} ^ {T} \sum_ {n = 1} ^ {N _ {t}} \mathbf {h} _ {t, n} \otimes \mathbf {y} _ {t, n}. \tag {4}
$$
Both  $\mathbf{C}$  and  $\mathbf{G}$  will be invariant to the sequence in which the entirety of  $N$  training samples are presented, a property ideal for CL algorithms.
The mentioned origins of Eqn. (2) in least squares theory is of practical use; we find it works best to use ridge regression [35], and calculate the  $l_{2}$  regularized inverse,  $(\mathbf{G} + \lambda \mathbf{I})^{-1}$ , where  $\mathbf{I}$  denotes the identity matrix. This is achieved methodically using cross-validation—see Appendix C. The revised score for CL can be rewritten as
$$
s _ {y} = \phi (\mathbf {f} _ {\mathrm {t e s t}} ^ {\top} \mathbf {W}) (\mathbf {G} + \lambda \mathbf {I}) ^ {- 1} \mathbf {c} _ {y}. \tag {5}
$$
In matrix form the scores can be expressed as predictions for each class label as  $\mathbf{y}_{\mathrm{pred}} = \mathbf{h}_{\mathrm{test}}\mathbf{W}_{\mathrm{o}}$ . Different to streaming LDA, our approach has the benefits of (i) removing the need for bias calculations from the score; (ii) updating the Gram matrix instead of the covariance avoids outer products of means; and (iii) the form  $\mathbf{W}_{\mathrm{o}} = (\mathbf{G} + \lambda \mathbf{I})^{-1}\mathbf{C}$  arises as a closed form solution for mean-square-error loss with  $l_{2}$ -regularization (see Appendix B.3), in contrast to NCM, where no such theoretical result exists. Phase 2 in Algorithm 1 summarises the above CL calculations.
Application of Eqn. (5) is superficially similar to AdaptMLP modules [6], but instead of a bottleneck layer, we expand to dimensionality  $M > L$ , since past applications found this to be necessary to compensate for  $\mathbf{W}$  being random rather than learned [43, 4, 18, 32]. As discussed in the introduction, the use of a random and training-free weights layer is particularly well suited to CL.
The value of transforming the original features to nonlinear random projections is illustrated in Fig. 3. Features for the  $T = 10$  split ImageNet-R CIL dataset were extracted from a pre-trained ViT-B/16 [9] network and Eqn. (5) applied after each of the  $T = 10$  tasks. Fig. 3(a) shows the typical CL average accuracy trend, whereby accuracy falls as more tasks are added. When a nonlinear activation,  $\phi(\cdot)$  is used (e.g. ReLU or squaring), performance improves as  $M$  increases, but when the nonlinear activation is omitted, accuracy is no better than not using RP, even with  $M$  very large. On the other hand, if dimensionality is reduced without nonlinearity (in this case from 768 to 500), then performance drops below the No RP case, highlighting that if RPs in this application create dimensionality reduction, it leads to poor performance.
Fig. 3(b) casts light on why nonlinearity is important. We use only the first 100 extracted features per sample and compare application of Eqn. (2) to raw feature vectors (black) and to pair-wise interaction terms, formed from the flattened cross-product of each extracted feature vector (blue trace). Use of the former significantly outperforms the latter. However, when Eqn. (5) is used instead (red trace), the drop in performance compared with flattened cross products is relatively small. Although this suggests exhaustively creating products of features instead of RPs, this is computationally infeasible. As an alternative, RP is a convenient and computationally cheap means to create nonlinear feature interactions that enhance linear separability, with particular value in CL with pre-trained models.
4.4 Combining with parameter-efficient transfer learning and first-session adaptation
Use of an RP layer has the benefit of being model agnostic, e.g. it can be applied to any feature extractor. As we show, it can be applied orthogonally to PETL methods. PETL is very appealing for CL, particularly approaches that do not alter any learned parameters of the original pre-trained model, such as [10, 28]. We combine RP with a PETL method trained using CL-compatible 'first-session' training, as carried out by [65, 37]. This means training PETL parameters only on the first CL task,  $\mathcal{D}_1$ , and then freezing them thereafter (see Phase 1 of Algorithm 1). The rationale is that the training data and labels in the first task may be more representative of the downstream dataset than that used to train the original pre-trained model. If a new dataset drifts considerably, e.g. as in DIL, the
Figure 3: Impact of RP compared to alternatives. (a) Using only Phase 2 of Algorithm 1, we show average accuracy (see Results) after each of  $T = 10$  tasks in the split ImageNet-R dataset. The data illustrates the value of nonlinearity combined with large numbers of RPs,  $M$ . (b) Illustration that interaction terms created from feature vectors extracted from frozen pre-trained models contain important information that can be mostly recovered when RP and nonlinearity are used.
benefits of PETL may be reduced because of this choice. But the assumption is that the domain gap between the pre-training dataset and the new dataset is significant enough to still provide a benefit. First-session training of PETL parameters requires a temporary linear output layer, learned using SGD with cross-entropy loss and softmax and only  $K_{1}$  classes, which is discarded prior to Phase 2.
For transformer nets, we experiment with the same three methods as [65], i.e. AdaptFormer [6], SSF [28], and VPT [21]. For details of these methods see the cited references and Appendix D. Unlike [65] we do not concatenate adapted and unadapted features, as we found this added minimal value when using RP.
4.5 Memory usage of RP layer
The RP layer increases the total memory required by the class prototypes by a factor of  $1 + (M - L) / L$ , while an additional  $LM$  parameters are frozen and untrainable. For typical values of  $L = 768$  and  $K = 200$ , the injection of an RP layer of size  $M = 10000$  therefore creates  $\sim 10$  times the number of trainable parameters, and adds  $\sim 10\mathrm{M}$  non-trainable parameters. Although this is a significant number of new parameters, it is still small compared with the overall size of the ViT-B/16 model, with its 84 millions parameters. Moreover, the weights of  $\mathbf{W}$  can be bipolar instead of Gaussian and if stored using a single bit for each element contribute only a tiny fraction to additional model memory. During training only, we also require updating an  $M\times M$  Gram matrix, which is smaller than the  $K(L\times L)$  covariance matrices of the SLCA fine-tuning approach [63]
Algorithm 1 RanPAC Training
Input: Sequence of  $T$  tasks,  $\mathcal{D} = \{\mathcal{D}_1,\dots \mathcal{D}_T\}$  pre-trained model, and PETL method
Phase 1: 'First-session' PETL adaptation.
for sample  $n = 1,\dots ,N_{1}$  in  $\mathcal{D}_1$  do Extract feature vector,  $\mathbf{f}_{1,n}$
Use in SGD training of PETL parameters for
Phase 2: Continual Learning with RPs.
Create frozen  $L\times M$  RP weights,  $\mathbf{W}\sim \mathcal{N}(0,1)$
for task  $t = 1,\dots ,T$  do
for sample  $n = 1,\dots ,N_{t}$  in  $\mathcal{D}_t$  do
Extract feature vector,  $\mathbf{f}_{t,n}$
Apply  $\mathbf{h}_{t,n} = \phi (\mathbf{f}_{t,n}^{\top}\mathbf{W})$
Update G and C matrices using Eqn. (4) for
Optimize  $\lambda$  (A. C) and compute  $(\mathbf{G} + \lambda \mathbf{I})^{-1}$
if  $M < \sqrt{K} L$ . L2P [57], DualPrompt [56] and ADaM [65] each use  $\sim 0.3\mathrm{M} - 0.5\mathrm{M}$  trainable parameters. For  $K = 200$  classes and  $M = 10000$ , RanPAC uses a lot more ( $\sim 2 - 2.5\mathrm{M}$ , depending on the PETL method), comparable to CODA-Prompt [46]. RanPAC also uses 10M untrained parameters, but we highlight that these are not trainable. Moreover,  $M$  need not be as high as 10000 (Table A5).
We have applied Algorithm 1 to both CIL and DIL benchmarks. For the pre-trained model, we experiment mainly with two ViT-B/16 models [9] as per [65]: one self-supervised on ImageNet-21K,
and another with supervised fine-tuning on ImageNet-1K. Comparisons are made using a standard CL metric, Average Accuracy [30], which is defined as  $A_{t} = \frac{1}{t}\sum_{i = 1}^{t}R_{t,i}$ , where  $R_{t,i}$  are classification accuracies on the  $i$ -th task, following training on the  $t$ -th task. We report final accuracy,  $A_{T}$ , in the main paper, with analysis of each  $A_{t}$  and  $R_{t,i}$  left for Appendix F, along with forgetting metrics and analysis of variability across seeds. Appendix F also shows that our method works well with ResNet and CLIP backbones.
We use split datasets previously used for CIL or DIL (see citations in Tables 1 and 3); details are provided in Appendix E. We use  $M = 10000$  in Algorithm 1 except where stated; investigation of scaling with  $M$  is in Appendix F.5. All listed 'Joint' results are non-CL training of comparisons on the entirety of  $\mathcal{D}$ , using cross-entropy loss and softmax.
Key indicative results for CIL are shown in Table 1, for  $T = 10$ , with equally sized stages (except VTAB which is  $T = 5$ , identical to [65]). For  $T = 5$  and  $T = 20$ , see Appendix F. For each dataset, our best method surpasses the accuracy of prompting methods and the CP methods of [20] and [65], by large margins. Ablations of Algorithm 1 listed in Table 1 show that inclusion of our RP layer results in error-rate reductions of between  $11\%$  and  $28\%$  when PETL is used. The gain is reduced otherwise, but is  $\geq 8\%$ , except for VTAB. Table 1 also highlights the limitations of NCM.
Method	CIFAR100	IN-R	IN-A	CUB	OB	VTAB	Cars
Joint linear probe	87.9%	72.0%	56.6%	88.7%	78.5%	86.7%	51.7%
L2P [57]	84.6%	72.4%	42.5%	65.2%	64.7%	77.1%	38.2%*
DualPrompt [56]	84.1%	71.0%	45.4%	68.5%	65.5%	81.2%	40.1%*
CODA-Prompt [46]	86.3%	75.5%	44.5%	79.5%	68.7%	87.4%	43.2%
ADaM [65]	87.6%	72.3%	52.6%	87.1%	74.3%	84.3%	41.4**
Ours (Algorithm 1)	92.2%	77.9%	62.4%	90.3%	79.9%	92.2%	77.5%
Rel. ER vs ADaM†	↓ 36%	↓ 20%	↓ 21%	↓ 25%	↓ 22%	↓ 53%	↓ 62%
Ablations
No RPs	90.6%	73.9%	57.7%	86.6%	74.3%	90.0%	73.0%
No Phase 1	89.0%	71.8%	58.2%	88.7%	78.2%	92.2%	66.0%
No RPs or Phase 1	87.0%	67.6%	54.4%	86.8%	73.1%	91.9%	62.2%
NCM with Phase 1	87.8%	70.1%	49.7%	85.4%	73.4%	88.2%	40.5%
NCM only	83.4%	61.2%	49.3%	85.1%	73.1%	88.4%	37.7%
Rel. ER (inc. Phase 1)	↓ 17%	↓ 15%	↓ 11%	↓ 28%	↓ 22%	↓ 22%	↓ 17%
Rel. ER (no Phase 1)	↓ 15%	↓ 13%	↓ 8%	↓ 14%	↓ 19%	↓ 4%	↓ 10%
Table 1: Comparison of prompting and CP strategies for CIL: 'Rel. ER'  $(^{\dagger})$  is Relative Error Rate. No rehearsal buffer is used for any method listed. IN-R is ImageNet-R [56]. The versions of ImageNet-A (IN-A), OmniBenchmark (OB), CUB and VTAB are those defined by [65]. L2P and DualPrompt results are those stated within [65], except * (Cars) from [63]. CODA-Prompt is directly from [46] for CIFAR100 and IN-R, and from our own experiments with the PILOT toolbox [49] otherwise. For ADaM, we have used the best reported by [65] across all PETL methods, except for ** (Cars) which is our own. Remaining results are our own implementation. PromptFusion [5] is omitted due to requiring a rehearsal buffer. Using a NCM method, [20] achieved  $83.7\%$  for CIFAR100 and  $64.3\%$  for Imagenet-R. Ablations of Algorithm 1: NCM uses cosine similarity instead of Eqn. (2); RP means Random Projections layer; Phase 1 is defined within Algorithm 1.
Algorithm 1 surpasses the jointly trained linear probe, on all seven datasets; Table 1 indicates that for all datasets this can be achieved by at least one method that adds additional weights to the pre-trained model. However, NCM alone cannot match the joint linear probe. As shown in Table 2, Algorithm 1 also does better than fine-tuning strategies for CIL [47, 63]. This is tabulated separately from Table 1 because fine-tuning has major downsides as mentioned in the Introduction. Table 2 also shows Algorithm 1 reaches within  $2\%$  raw accuracy of the best joint fine-tuning accuracy on three datasets, with a larger gap on the two ImageNet variants and Cars.
Results for DIL datasets, and corresponding ablations of Algorithm 1 are listed in Table 3. CORe50 is  $T = 8$  stages with 50 classes [29], CDDB-Hard is  $T = 5$  with 2 classes [27] and DomainNet is  $T = 6$  with 345 classes [40] (further details are provided in Appendix E). The same trends as for CIL can be observed, i.e. better performance than prompting strategies and highlighting of the value of the RP layer. For DomainNet, the value of RP is particularly strong, while including PETL adds little value, consistent with the nature of DIL, in which different CL stages originate in different domains.
Method	CIFAR100	IN-R	IN-A	CUB	OB	VTAB	Cars
Joint full fine-tuning	93.8%	86.6%	70.8%	90.5%	83.8%	92.0%	86.9%
SLCA [63]	91.5%	77.0%	-	84.7%*	-	-	67.7%
Ours (Algorithm 1)	92.2%	78.1%	61.8%	90.3%	79.9%	92.6%	77.7%
Table 2: Comparison with fine-tuning strategies for CIL. Results for SLCA are directly from [63]. 'Joint' means full fine-tuning of the entire pre-trained ViT network, i.e. a continual learning upper bound. Notes: L2 [47] achieved  $76.1\%$  on Imagenet-R, but reported no other ViT results. Cells containing - indicates SLCA did not report results, and no codebase is available yet. Superscript*: the number of CUB training samples used by [63] is smaller than that defined by [65], which we use.
Method	CORe50	CDDB-Hard	DomainNet
L2P [37]	78.3%	61.3%	40.2%
S-iPrompts [55]	83.1%	74.5%	50.6%
ADaM	92.0%	70.7%	50.3%
Ours (Algorithm 1)	96.7%	86.2% (M=5K)	66.6%
S-liPrompts [55] (not ViT B/16)	89.1%	88.7%	67.8%
Ablations
No RPs	94.8%	84.2%	55.2%
No Phase 1	94.3%	81.6% (M=5K)	64.7%
No RPs or Phase 1	92.4%	80.7%	54.1%
NCM with Phase 1	91.4%	73.2%	51.3%
NCM only	80.4%	69.8%	46.4%
Rel. ER (inc. Phase 1)	↓37%	↓13%	↓25%
Rel. ER (no Phase 1)	↓25%	↓5%	↓23%
Table 3: Comparison with prompting and CP strategies for DIL: No rehearsal buffer is used for any method. The first four rows use ViT B/16 networks. S-liPrompts uses vision-language prompting with a pre-trained CLIP model, and cannot be directly compared to ours. For CORe50,  $83.2\%$  was reported for simple CP strategy of [20] and  $85.4\%$  for FSA-FiLM with pre-trained convnets [37]. For DomainNet we and [55] use 365 classes, different to [46]. ADaM results are based on code from [65]. L2P and S-prompts are taken from [55]. Ablations: Explanations are the same as in Table 1.
We have demonstrated that feature representations extracted from pre-trained foundation models such as ViT-B/16 have not previously achieved their full potential for continual learning. Application of our simple and rehearsal-free class-prototype strategy, RanPAC, results in significantly reduced error rates on diverse CL benchmark datasets, without risk of forgetting in the pre-trained model. These findings highlight the benefits of CP strategies for CL with pre-trained models.
Limitations: The value of Eqs (4) and (5) are completely reliant on supply of a good generic feature extractor. For this reason, they are unlikely to be as powerful if used in CL methods that train networks from scratch. However, it is possible that existing CL methods that utilise self-supervised learning, or otherwise create good feature extractor backbones could leverage similar approaches to ours for downstream CL tasks. As discussed in Section 4.5, RanPAC uses additional parameters compared to methods like L2P. However, this is arguably worth trading-off for the simplicity of implementation and low-cost training.
Future Work: Examples in Appendix F shows that our method works well with other CL protocols including: (i) task-agnostic, i.e. CL without task boundaries during training (e.g. Gaussian scheduled CIFAR-100), (ii) use of a non one-hot-encoded target, e.g. language embeddings in the CLIP model, and (iii) regression targets, which requires extending the conceptualisation of class-prototypes to generic feature prototypes. Each of these has a lot of potential extensions and room for exploration. Other interesting experiments that we leave for future work include investigation of combining our approach with prompting methods, and (particularly for DIL) investigation of whether training PETL parameters beyond the first session is feasible without catastrophic forgetting. Few-shot learning with pre-trained models [1, 45] may potentially also benefit from a RanPAC-like algorithm.
This work was supported by the Centre for Augmented Reasoning at the Australian Institute for Machine Learning, established by a grant from the Australian Government Department of Education. Dong Gong is the recipient of an Australian Research Council Discovery Early Career Award (project number DE230101591) funded by the Australian Government. We would like to thank Sebastien Wong of DST Group, Australia, and Lingqiao Liu of the Australian Institute for Machine Learning, The University of Adelaide, for valuable suggestions and discussion.
[1] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 10
[2] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920-15930, 2020. 2
[3] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proc. European Conference on Computer Vision (ECCV), September 2018. 21
[4] C. L. Philip Chen. A rapid supervised learning neural network for function interpolation and approximation. IEEE Transactions on Neural Networks, 7:1220-1230, 1996. 2, 3, 4, 6, 7
[5] Haoran Chen, Zuxuan Wu, Xintong Han, Menglin Jia, and Yu-Gang Jiang. PromptFusion: Decoupling stability and plasticity for continual learning. arXiv:2303.07223, Submitted 13 March 2023, 2023. 3, 9
[6] Shoufa Chen, Chongjian GE, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. AdaptFormer: Adapting vision transformers for scalable visual recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16664-16678. Curran Associates, Inc., 2022. 2, 7, 8, 20, 21
[7] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), pages 8230-8239, 2021. 3
[8] Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the CLIP model. arXiv: 2207.09248, 2022. 2, 3
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929 (ICLR 2020), 2020. 3, 5, 7, 8
[10] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 10629-10642. Curran Associates, Inc., 2022. 2, 7
[11] Tyler L. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. arXiv:1909.01520, 2020. 2, 4, 7
[12] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv: 2110.04366 (ICLR 2022), 2022. 3