RanDumb: Random Representations Outperform Online Continually Learned Representations
Authors: Ameya Prabhu
Shiven Sinha
Ponnurangam Kumaraguru
Philip Torr
Ozan Sener
Puneet K. Dokania
Venue: NeurIPS 2024 poster (2024)
Keywords: online continual learning
exemplar-free
baseline
analysis
Primary area: evaluation
TL;DR: We shed light that the performance of state-of-the-art methods, including ICLR '24 best paper runner-up, does not surpass a random baseline (RanDumb), demonstrating the poor performance of representation learning

Abstract: Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach embeds raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb,  significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in the online continual learning.

RanDumb:
Random Representations Outperform
Online Continually Learned Representations
Ameya Prabhu Shiven Sinha
Ponnurangam Kumaraguru
Philip H.S. Torr
Ozan Sener
Puncet K. Dokania
University of Oxford
2IIT Hyderabad
3 Apple
https://github. http:///gtthuu..co//rrmmmpppeee com//rrimpossibe//fannnii
Abstract
Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability- plassiiity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contri- bution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared dom transforms. Our approach projects raw pixels using a fixed random transform, approximating RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample time in an online continual learning setting. This method, called RanDumb, significantly outperforms state-of-the-art continually learned represen- tations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending investigation popular exemplar-fre scenarios with pretrained models, we find that training only a linear classifier top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in online continual learning.
simple pre-defined ran-
RanDumb
Decorrolate
Approximate RBF-Kernel
NCM
Random Transform
Linear
Traditional
Figure 1: Our primary analysis in this work is ablating the deep feature extractor (bottom center) by replacing with a random projection (top center) to isolate the effect of online continual representation learning in the deep network. We demonstrate that random projections not only match but consistently outperform continually learned representations, highlighting the poor quality of the continually learned representations. RanDumb (top) maps pixels high- dihh-diieniin space using random Fourier projections, then decorrelates the features using the Mahalanobis distance [43] and classifies based on the nearest class mean.
authorr contributed equally, equal advising
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Introduction
Continual learning aims to develop models capable of learning from non-stationary data streams, inspired by the lifelong learning abilities exhibited by humans and the prevalence of such real- world applications (see Verwimp al. [65] for a survey). It is characterized by sequentially arriving tasks, coupled with additional computational and memory constraints [33, 38, 54, 63, 49].
Building on the foundations of supervised deep learning, the prevalent approach in continual learning has been to jointly train representations alongside classifiers. This approach simply follows from the assumption that learned representations are expected to outperform fixed representation functions such as kernel classifiers, as demonstrated in supervised deep learning [34, 23, 57]. However, this assumption is never validated in continual learning, with scenarios having limited updates where networks might not be trained until convergence, such as online continual learning (OCL).
In this paper, we study the efficacy of representations derived from continual learning algorithms. Surprisingly, our findings suggest that these representations might not be as beneficial as presumed. To test this, we introduce simple baseline method named RanDumb, which combines a random representation function with straightforward linear classifier, illustrated in detail in Figure 1. Our empirical evaluations, summarized in Table 1 (left, top), reveal that despite replacing the representation learning with a pre-defined random representation, RanDumb surpasses current state- of-the-an art methods in latest online continual learning benchmarks [76].
We further expand evaluations scenarios incorporating methods that use pre-traine pre-trained feature extractors [68]. By substituting our random projections with these feature extractors and retaining the linear classifier, RanDumb again outperforms leading methods as shown in Table (right, top).
1.1 Technical Summary: Construction of RanDumb and Empirical Findings
Design. RanDumb first projects input pixels into a high-dimensional space using a fixed kernel based random Fourier basis, which is low-rank data-independent approximation of the RBF Kernel [52]. Then, we use a simple linear classifier which first normalizes distances across different feature dimensions (anisotropy) with Mahalanobis distance [43] and then
nearest class means for classification [44]. In scenarios with pretrained feature extractors, we use the fixed pretrained model embedder and learn linear classifier described above, similar Hayes and Kanan [27].
Key Properties. RanDumb needs no storage of exemplars and requires only one pass over the data in a one-sample- ceeipppee-e---tmeeee perstiimsttep fashion. Furthermore, it only requires online estimation of the sample covariance matrix and nearest class mean.
Key Finding 1: Poor Representation Learning. We compare RanDumb with leading methods: VAE- GC [64] in Table 1 (left, middle) and SLCA [79] in Table 1 (right, middle). The primary distinction between them is their representation: RanDumb uses a fixed function crandom/pretraingg network),
Table (Left) Online Continual Learning. Performance comparison of RanDumb on the setup [76] and VAB-GC [64). Setup and numbers borrowed from PEC [76]. RanDumb outperforms the best OCL method. (Right) Offline Continual Learning. Performance comparison with ImageNet21 K VIT-B16 model using initial classes and new class per task. RanPAC-imp is an improved version of the RanPAC code which mitigates the instability issues in RanPAC. RanDumb nearly matches performance of joint for both online and offline, demonstrating the inefficacy of current benchmarks.
Method
MNIST CIFARIO CIFARI00 m-IMN Method
CIFAR IN-A IN-R CUB OB VTAB Cars
Comparison with Best Method
Comparison with Best Method
Best (PEC)
92.3
58.9
26.5
14.9
Best (RanPAC-imp)
89.4
33.8
69.4
89.6
75.3
91.9
$7.3
RanDumb (Ours)
98.3
55.6
28.6
17.7
RunDumb (Ours)
86.8
42.2
64.9
88.5
75.3
92.4
67.1
Improvement
+6.0
-3.3
+2.1
+2.8
Improvement
-2.6
+8.4
-.5
-1.1
+0.0
+0.5
+9.8
Random vs. Learned Representations
Random vs. Finetuned Representations
VAE-GC
84.0 42.7
19.7
12.1
SLCA
86.8
54.2
82.1
18.2
RanDumb (Ours)
98.3
55.6
28.6
17.7
RanDumb (Ours)
86.8
42.2
64.9
88.5
75.3
92.4
67.1
Improvement
+14.3
+12.9
+8.9
+5.6
Improvement
+0.0
+10.7
46.4
+48.9
Scope Improvement
Scope of Improvement
Joint (One Pass)
98.3
74.2
33.0
25.3
Joint
93.8
70.8
86.6
91.1
83.8
95.5
86.9
RunDumb (Ours)
08.3
55.6
28.6
17.7
RunDumb (Ours)
86.8
42,2
64.9
88.5
75.3
92,4
67.1
Gap Covered. (%)
100%
75%
87%
70%
Gap Covered. (%)
93%
60%
75%
97%
92%
97%
77%

---

Inpui
SD View
Projection (Honzoonaa ddway)
Oupul
Embed
Decorrelate
Figure 2: RanDumb projects the datapoints high-dimensional space create clearer separation between classes. Subsequently, corrects the anisotropy across feature dimensions, scaling them to be unit variance each. This allows cosine similarity accurately separates classes. The figure is adapted from [48].
whereas VAE-GC and SLCA further continually trained deep networks. RanDumb consistently surpasses VAE-GC and SLCA by wide margins of 5-15%. This shows that state-of-the-re arrt online continual learning algorithms fail learn effective representations across standard exemplar-fie free continual learning benchmarks.
Finding 2: Over-Constrainee Benchmarks. Given the demonstrated limitations of existing continual representation learning methods, an important question arises: Can better methods learn more effective representations? To explore this, we evaluated the performance of RanDumb against joint training, models trained without continual learning constraints, in both online and offline settings, as shown in Table (left, bottom) and Table (right, bottom). Our straightforward baseline, RanDumb, bridges 70-90% of the performance gap relative to the respective joint classifiers in both scenarios. This significant recovery of performance by such simple method suggests that if our goal is to advance the study of representation learning, current benchmarks may be overly restrictive and conducive to truly effective representation learning.
We highlight that the goal in our work is not to introduce state-of- -arte continual learning method, challenge prevailing assumptions and open a discussion on the efficacy of representation learning in continual learning algorithms, especially in online and low-exemplar scenarios.
RanDumb: Mechanism & Intuitions
RanDumb has two main elements: random projection and the dumb learner. We illustrate the mechanism of RanDumb using three toy examples in Figure 2. To classify test sample Xtest we start with simple classifier, the nearest class mean (NCM). predicts the class among C classes by highest value of the similarity function f among class means jaz
Ypred arg max f(Xisss (s), where (E((....//C)
f(xust (exsst:H): == Xeest
M4s
(1)
and are the meass in the pixel space: C.I. Exec, x. RanDumb adds two additional components to this classifier: 1) Kemnelizttion and 2) Decorrelation.
1
Kernelization: Classes are typically not linearly separable in the pixel space, unlike in the feature space of deep models. Hence, we randomly project the pixels into high-dimennsioal representation space, computing all distances between the data and class-means in this embedding space. This phenomena is illustrated on three toy examples to build intuitions in Figure 2 (Embed). We use an RBF-Kernel, which for two points and y is defined as: (x, y) exp (->lIx y where is a scaling parameter. However, calculating the RBF kernel is not possible due to the online continual learning constraints preventing computation of pairwise-distance between all points. We use a data-indeepndent approximation, random Fourier projection oi (). as given in [52]:
KRBE (x, y) o(x)² Tol))

---

where the random Fourier features 6xx) are defined by first sampling D vectors (wi..... wp} from Gaussian distribution with mean zero and covariance matrix 2yI, where I is the identity matrix. Then (xx) is 2D-dimensional feature, defined as:
x(x)
[cos(w] x), sin(w] x)..., cos(w))x), sin(wh[)]
VD
We keep these bases fixed throughout online learning. Thus, obtain modified similarity function from Equation as:
(Xtest, (a) == Φ(xrest) ) P4
(2)
where Tt are the class-means in the kernel space:
P4
((x)
xEC;
Decorrelation: Projected raw pixels have feature dimensions with different variances (anisotropic). Hence, instead naively computing ((x.est) N4 we further decorrelate the feature dimensions using Mahalonobis distance with the shrinked covariance matrix S using OAS shrinkage [15], inverse obtained by least squares minimization (S + XI). We illustrate this phenomena as well on three toy examples in Figure 2 (Decorrelate) to build intuitions. Our similarity function finally is:
[Xtesst (i) = (xtes))) ( jic))
Online Computation. Our random projection is fixed before seeing any data. During continual learning, we only perform online update on the running class mean and empirical covariance matrix?.
3 Experiments
We compare RanDumb with algorithms across online continual learning benchmarks with emphasis on exemplar-fre and low-exemplar storage regime. All numbers in tables with the caption (Ref: table and citation) except our method are taken from the aforementioned table in the cited paper.
Benchmarks. The benchmarks which we used in our experiments are sum- mized in Table the right. We aim for a comprehensive coverage and show results four standard online tinual learning benchmarks (A, B, D. E) which reflect the latest trends ('22- "24) across exemplar free, contrastive- training', meta-continual learning, and network-expansion based approaches re-
Setup
Num #Classes #Samples eStered Contrastive Passes Per Tisk Per Stop
Exemplars
Augment
Method:
RanDumb
(Zajac et (76])
1
10
Na
B1 [Guo et al. (25))
1
10
100-2000
No
B2 (Guo al. (25)
100-1000
Yes
C (Smith et al. (60)
Many
10
No
(Wu et al. [71])
2-10
10
1000
No
and (75))
1000-5000
(Wang et al. [68), modified)
Many
All
No
spectively. We also evaluate a rehearsal-free offline continual learning benchmark C. These benchmarks are ordered by increasingly relaxed constraints, moving further away from the training scenario of RanDumb..eenhhaart A closely matches RanDumb with one class per timestep and no stored exemplars. Benchmark B, D, E progressively relax the constraints on exemplars and classes timestep. Benchmark C and E remove the online constraint by allowing unrestricted training and sample access within a task without exemplar-storage of past tasks. Benchmark allows using large pretrained models, modified by us with one class per task, i.e. testing learning over longer timespans.
We further test exemplar-free scenarios in offline continual learning using Benchmark F [68] with the challenging one-class per task constraint borrowed from [76]. This benchmark allows using pretrained models along with unrestricted training time and access to all class samples each timestep. However, RanDumb is restricted to learning from a single pass seeing only one sample at a given pretrained model in Benchmark F.
time. RanDumb only leams linear classifier
inverse of the covariance matrix is possible using Sherman Morrison formula. BBnnhharkk B is split into two sections: (B) methods that do not rely on contrastive learning and heavy augmentation, and (B2) approaches that incorporate contrastive learning and extra augmentations.
Online update

---

Table 2: Benchmark A (Ref: Table 1 from PEC (76]). We compare RanDumb in a 1-class per task setting referred 'Dataset coum_tasks/l) We observe that RanDumb outperforms all approaches across all datasets by 2-6% margins, with an exception of latest work PEC [76] on CIFARIO.
Method
Memory
MNIST (10/1)
CIFAR-10 CIFAR-100 minilmageNet
(101)
(100/1)
(100/1)
Fine-tuning
all
10.1 0.0
10.0 0.0
1.0 0.0
1.0 0.0
Joint, 1 epoch
all
98.3=
0.0
74.24 0.1
33.04
25.3 0.2
[13]
500
84.4=
0.3
40.6+ 1.1
12.54 0.3
5.7+
A-GEM [12]
500
59.8+
0.8
10.2+ 0.1
1.0
1.1
iCaRL [54]
500
83.1
0.3
37.84
5.74
7.11
Rehearsal
BiC [72]
500
86.0 0.4
0.4
0.4
1.11.11
Based
ER-ACE [10]
500
87.8+00.2
39.9=0.5
8.2:
5.740.2
Methods
DER [9]
500
91.7= 0.1
40.0+ 1.5
1.0
1.0
DER++ [9]
500
91.9 0.2
35.64
6.2+
1.4
X-DER [8]
500
83.0 0.1
4.5
0.
8.2 0.4
GDumb [49]
500
91.0++022
50.7=0.7
8.2+002
EWC [33]
10.1 0.0
10.6
1.0
1.0
[1
12.7
1.0
0.11
1.100.0
1.010.1
LwF [37]
0
0.6
0.11
0,9+0.0
1.0+
Rehearsal
LTT [78]
0
10.9
0.9
10.0 0.2
1.1
1.0
Free
Gen-NCM |31|
0
82.0 0.0
27.7+
0.0
7.5+
Methods
Gen-SLDA [27]
0
88.0 0.0
41.4+
0.0
088
12.90000
VAE-GC [64]
0
84.0 0.5
42.7 1.3
0.
12.1
PEC [76]
92.3
0.1
58.91
0.1
2615
14,9
RanDumb (Ours)
0
98.3 (+5.9)
55.6
(-3.3)
28.6
(+2.1)
17.7 (+2.8)
Table 3: Benchmark B.1 (Ref: Table adopted from OnPro [69]. OCM[25]) We compare RamDumb in
many-classes per task setting referred as 'Dataset (num tasks/num classes per "ask)' We categorize memory buffer sizes with 'M'. RanDumb outperforms the competing approaches without heavy- augmentations by 3-20% margins despite being exemplar free. Only one case, it is second best.
Method
MNIST
CIFARIO
CIFARIO0
CIFAR100
TinyImageNe (100/2)
(5/2)
(5/2)
(10/10)
(50/2)
M 0.1k
0.1k
0.2k
M 0.5k
M
M
1k
M 2k
AGEM [12]
56,945.2
17.7+0.3
22.7+1.8
5.8+0.2
5.9+0.1
1.840.2
0.8a0.1
0.9+0.1
GSS [4]
70.4+1.5
18.4+0.2
26.9+1.2
8.1+0.2
11.1+0.2
4.3+0.2
1.1+0.1
3.3+0.5
ER [13]
78.7+0.4
19.4+0.6
29.7+1.0
8.7+0.3
15.7+0.3
8.340.3
1.2+0.1
5.640.5
ASER [58]
61.642.1
20.0+1.0
27.8+1.0
11.020.3
16.4+0.3
9.6211.
2.240.1
5.340.3
MIR [3]
79.0+0.5
20.720.7
37.3+0.3
9.720.3
15.7+0.2
12.720.3
1.440.1
6.120.5
ER-AML [10]
76.5+0.1
40.5+0.7
16.120.4
5.440.2
iCaRL [54]
31,0+1.2
33.940.9
12.8+0.4
16.5+0.4
5.040.3
6.640.4
DER+ [9]
74.4+1.1
31,542.9
44.2+1.1
16.040.6
21.400.9
9,340.3
3.7+0.4
5.140.8
GDumb [49]
81,2+0.5
23.3+1.3
35.9+1.1
8.240.2
18.1
1
4.6+0.3
12.620.1
CoPE [19]
33.543.2
37.3+2.2
11.640.4
14.6&1.
2.1.0.3
2.340.4
DVC [25]
35,2+1.7
41.6+2.7
15.4+0.3
20.3+1.0
4.9+0.6
7.5+0.5
CoaL. [11]
83.1+0.1
42.1+1.2
17.1+0.4
10.1
R-RT [6]
89.1+0.3
45.240.4
15.4+0.3
6.640.3
CCIL [46]
86.4+0.1
50,5+0.2
18.5+0.3
5.640.9
IL2A [82]
90.2+0.1
54.740.5
18.2+1.2
5.5+0.7
BiC [72]
90.420.1
48.240.7
21.2+0.3
10.2+0.9
SSIL [1]
88.2+0.1
59,5
26.0+0.1
9.6+0.7
Rehearsal Free
PASS [82]
33.7+2.2
33.7+2.2
7.5w0.7
7.5+0.7
0.5+0.1
0.5+0.1
RanDumb (Ours) 98.3 (+7.8) 55.6 (+20.4) 55.6 (+5.9) 28.6 (+12.6) 28.6 (+2.6) (816.5) 11.6 (+6.6) 11.6 (-1.0)
We use LAMDA-PILOT [61] codebase for all methods, except RanPAC and SLDA for which use their codebases. We use the original hyperparameters. We only change initial classes to 2 and number of classes per task to and test using both ImageNet21K and ImageNetIK ViT-B/I6 models.
Implementation Details (RanDumb). We evaluate RanDumb using five datasets: MNIST, CIFARIO, CIFAR100, TinylmageNet200, and minilmageNet100.. For the latter two, downscale all images to 32x32. We augment each datapoint with flipped version, hence two images are seen by the classifier at each timestep (except for MNIST and Benchmark F). We normalize all images and flatten them into vectors, obtaining 784-dim input vectors for MNIST and 3072-dim input vectors for all the other. For Benchmark F, compare RanDumb on seven datasets used in LAMDA-PILE replacing ObjectNet
5

---

from Table 4: (Left) Benchmark B.2 (Ref: Table from OnPro [69]) We compare RanDumb with contrastive representation learning based approaches which additionally use sophisticated augmentations. We observe that RanDumb often outperforms these sophisticated methods despite all of these factors on small-exemplar settings. (Right) Benchmark C (Ref: Table 2 from [60]). We compare RanDumb with latest rehearsal-free methods. RanDumb outperforms them by 4% margin.
Method
CIFAR100 (10/10)
Method
MNIST (5/2) CIFARIO (5/2) CIFAR100 (10/10) Tiny TinyImageNee (100/2)
0.1k
0.1k
M 0.5k
M 1k
Rehearsal Free
SCR [40]
86.2+0.5
40.2+1.3
19.3+0.6
8.9+0.3
OCM [25]
90.740.1
47.5+1.7
19.7+0.5
10.8+0.4
PredKD [37]
24.6
OnPro [69]
57.8+1.1
22.7+0.7
11.9+0.3
PredKD) FeatKD
12.4
Rehearsal Free
PredKD EWC
23.3
RanDumb
98.3 (+7.5)
(-2.2)
28.6 (+5.9)
11.6 (-0.3)
PredKD +
21.5
RanDumb (Ours)
28.6 (+4.0)
with Stanford Cars as ObjectNet license prohibits training models. We use the 768 -o8-dimensional features from the same pretrained ViT-B models used in this benchmark. We measure accuracy on the test set of all past seen classes after completing the full one-pass. We take the average accuracy after the last task all past tasks [76, 25, 68J. In Benchmark A and F, since we have one class per task, the average accuracy across past tasks is the same regardless of the task ordering. In Benchmarks A-E, all datasets have the same number of samples, hence similarly the average accuracy across past tasks the same regardless the task ordering. We used the Scikit-Learn implementation of Random Fourier Features [52] with 25K embedding size, 7 1.0. We use progressively increasing ridge regression parameter (A) with dataset complexity, 10-6 for MNIST, 10-5 for CIFAR and À = 10-1 for Tinyymmaeee220mmiimmageee
3.1 Results
Benchmark A (single-class per task). We assess continual learning models in the challenging setup of one class per timestep, closely mirroring our training assumptions, and present our results in Table 2. Comparing across rows, and see that RanDumb improves over prior state-of-the-art across all datasets with 2-6% margins. The only exception is PEC on CIFARIO, where RanDumb underperforms by 3.3%. Nonetheless, it outperforms the second-best model, GDumb with a 500 memory size, by 4.9%.
Benchmark B.1 (many-classes per task). We present our results comparing with non-contrastive methods in Table 3. We notice that scenario allows two classes per task and relaxes the memory constraints for online continual learning methods, allowing for higher accuracies compared to Benchmark A. Despite that, RanDumb outperforms latest OCL algorithms on MNIST, CIFAR10 and CIFAR100- by margins exceeding 10%. The lone exception is GDumb achieving a higher performance with 2K memory samples on TinylmageNet, indicating that this already is in the high-memory regime.
Benchmark B.2 (many-classes per task, with contrastive losses and data augmentations). We additionally compare performance with the latest OCL approaches using contrastive losses with sophisticated data augmentations. As shown in in Table 4 (Left), these advancements provide large performance improvements over methods from Benchmark B. To compensate, exemplar budgets. The best approach, OnPro [69], outperforms RanDumb on CIFARIO by 2.2% and TinylmageNe by 0.3%, but falls significantly short on CIFAR 100 by 5.9%. Overall, RanDumb achieves strong results compared to representation learning using state-of-the contrastive learning approaches customized to continual learning, despite storing exemplars.
compare on lower
Benchmark C (rehearsal-free). We compare against offline rehearsal-free continual learning approaches in Table 4 (Right) on CIFAR100. Despite online training, RanDumb outperforms PredKD by over 4% margins.
Benchmark D (meta-continual learning). We compare performance of RanDumb against meta- continual learning methods, which require large exemplars with buffer sizes of IK in Table 5 (left). RanDumb achieves strong performance under these conditions, exceeding all prior work by a large margin of 9.1% on CIFAR100 and outperforms all but VR-MCL approach on the TinyImageNet dataset. GDumb performs the best on CIFARIO, indicating this is already in a large-exemplar regime uniquely unsuited for RunDum..

---

Table 5: (Left) Benchmark D (Ref: Table 2 from VR-MCL We compare RanDumb with meta- continual learning approaches operating in high memory setting, allowing buffer sizes up 1K exemplars. RanDumb outperforms all methods except VR-MCL on TinylmageNet. RanDumb also surpasses all prior work by a substantial 9.1% CIFAR 100. Allowing generous replay buffers shifts scenarios to a high exemplar regime where GDumb performs the hest on CIFAR10. Yet RanDumb competes favorably even under these conditions. (Right) Benchmark E (Ref: Table I from SEDEM [75]) We compare RanDumb with network expansion based approaches. Despite allowing access to much larger memory buffers, RanDumb matches the performance of best method SEDEM on MNIST, while exceeding by 0.3% on CIFAR1O and 3.8% on CIFAR100.
Method
CIFARIO CIFARIOO TinylmageNet
(5/2)
(10/10)
(20/10) M 1k
Method
MNIST
CIFARIO
CIEARIO0
M -11 M
1k
(5/2)
(5/2)
(20/5)
Finetune
0.60
0.3
3.9 0.2
M = 2k
M =111k
M = 5k
LWF [37]
18.8
5.6
4.0 0.3
Finetune
19.8 18.5
+
3.5
A-GEM [12]
1..4
6.2
4.0 0.2
MIR [3]
93.2 42.8
4.2
20.0 20.0
IS [77]
1.2
5.2
0.3
GEM [12]
93.2
24.1 24.1 11.1
MER [55]
3..9
iCARL [54]
8.2
37.3 0.2
10.8
La-MAML [26]
33.4
1.66
6.74
G-MED [32]
82 +7.5+32
19.6
GDumb [49]
61.2 1.0 18.1 0.3
4.3
GSS [4]
92.5+ 38.5
+ 1.4
13.1 +.9
ER [13]
4.8 16.1
0.9
1..
CoPE [19]
9329 48.9
= 1.3
21.6
DER [9]
29.9 6.1
4.1
CURL [53]
0.7
DER+ [9]
1.9
0.7
0.3
CNDPM [36]
95.4 + 0.2 48.8
+ 0.3 22.5
1.3
CLSER [5]
52.8 + 1.7
17.9
+1.1
OCM [25]
53.4
= 1.0
14.4
0.8
0.5
Dynamic- OCM [74]
94.0 + 0.2 49.2 + 1.5
21.8
+ 0.7
ER-OBC [18]
54,8
1
2.2
17.2 0.9
1.2
SÉDEM [75]
98.3 0.2 55.3 + 1.3 24.8 -
VR-MCL. [71]
56.5
+ 1.8 19.5 0.7
13.3
Rehearsal-Fiee Free
Rehearsal-Fiee Free
RanDumb (Ours)
98.3 (0.0)
55.6 (40.3) 28.6 (13.8)
RanDumb (Curs) 55.6 (-5.6) 28.6 (+9.1)
(110
90.0
05.5
050
9.00
1300
1000
3030
25030
5.00
109n
10000
30000
35030
5400
1038
3800
2003
536
1034
1990
3000
2000
Enrekking
Exmekkng Sker
Figure 3: Accuracy of RanDumb with respect embedding dimensionality across datasets.
Benchmark E (network-expansion), We compare RanDumb against network expansion bbssd online continual learning methods in Table 5 (right). These approaches grow model capacity to mitigate forgetting while dealing with shifts in the data distribution, and are allowed larger memory buffers. RanDumb matches the performance of the state-of-the-art method SEDEM [75] on MNIST, while exceeding it by 0.3% on CIFARIO and 3.8% on CIFARI00.
3.2 Analysis of RanDumb
Ablating Components of RanDumb. We ablate the contribution of only using Random Fourier features for embedding and decorrelation to the overall performance of RanDumb in Table 6 (left, top). Ablating the decorrelation and relying solely on random Fourier features, colloquially dubbed Kemne--NCM, has performance drops ranging from 6-25% across the datasets. Replacing random Fourier features with raw features, ie. the SLDA baseline, leads to pronounced drop in performance ranging from 3- 5-14% across the datasets. Morcover, ablating both components results in the base nearest class mean classifier, and exhibits the poorest performance with an average reduction of 17%. Therefore, both decorrelation and random embedding are crucial for RanDumb.
Impact of Embedding Dimensions. We vary the dimensions of the random Fourier features ranging from compressing 3K input dimensions to 1K to projecting it to 25K dimensions and evaluate its impact on performance in Figure 3. Surprisingly. the random projection to a 3x compressed 1K dimensional space allows for significant performance improvement over not using embedding, given in Table 6 (left, top). Furthermore, increasing the dimension from 1K to 25K results in improvements of 3.6%, 10.4%. 7.0%. and 2.5% on MNIST, CIFARIO, CIFAR 100, and TinylmageNet respectively. Increasing the embedding sizes beyond 15K. however, only results in modest improvements of 0. 16, 1.4%, 1.1% and 0.2% on the same datasets, indicating 15K dimensions would be a good point for a perrormance-computttioonn cost tradeoff.
7

---

Table 6: (Left) Analysis of RanDumb:
We study contributions of decorrelation, random embedding, and data augmentation. We further vary the embedding sizes and regularisation parameter. Finally, compare with alternate embeddings. (Right) Architectures (Ref: Table from Mirzadeh et al. [45]) RanDumb surpasses continual representation learning across a wide range of architectures, achieving close to 94% of the joint performance.
Method
MNIST
CIFARIO CIFAR100 T-ImNet (200/1)
m-ImNet (100/1)
(10/1)
(101))
(10/1)
Model
CIFARI00
Ablating Components of RanDumb
Joint
79.58
RanDumb
98.3 55.6
28.6
11.1
17.7
CNN
62.2 11.35
Decorrelate
83.8 (-14.5) 30.0 (-25.6) 12.0 (-16.6)
4.7 (-6.4)
8.9 (-8.8)
CNN
66.3 +1.12
Embed
88.0 (-10.3) 41.6 (-14.0)
19.0 (-9.6)
8,0 (-3.I)
12.9 (-4.8)
CNN
68.1 +0.5
CNN x8
69.9
-Both
82.1 (-16.2) 28.5 (-27.1) 10.4 (-18.2)
4.1 (-7.0)
7.28 (-10.4)
CNN x16
76.8 10.76
Effect of Adding Flip Augmentation
ResNet-18
45.0 +5.03
ResNet-34
44,8 +2.34
With Without
55.6 52.5 (-3.1) 26.9 (-1.7) 10.7 (-0.4)
28.6
11.1
17.7
ResNet-50
56.2 +0.88
98.3
16.6 (-1.))
ResNet-101
56.8 11.62
Variation with Ridge Parameter X
WRN-10-2
50.5 12.65
WRN-10-10
56.8 12003
A 10
98.3
53.9
27.8
10.3
15.8
WRN-16-2
44.6
A 10
55.6
28.6
11.1
15.9
WRN-16-10
51.3 +1.47
A 10-4
96.6
52.6
26.1
11.6
17.7
WRN-28-2
42.27
WRN-28-10
49.3 12.02
Variation Across Embedding Projections
VIT-512/1024
$1.7 11.
No-Embed
88.0
41.6
19.0
8.0
12.9
VTT-1024/1546
60.4 +1.5
RP+ReLU (RanPAC)
95.2
48.8
23.1
9.7
15.7
RanDumb (Ours) 74.8 (-2.0)
RanDumb (Ours)
98.3 (+3.1) 55.6 (+6.8)
28.6 (+5.5)
11.1 (+1.4)
17.7 (42.0)
Impact of Flip Augmentation. We evaluate the impact of adding the flip augmentation on the perfor- mance of RanDumb in Table (left, middle). Note that MNIST was not augmented. Augmentation provided large gains of 3.1% 1% on CIFARIO, 1.7% on CIFARI00, and 0.4% on TinyImageNet. We did not augment the data further with RandomCrop transform as done with standard augmentations.
Impact of Varying Ridge Parameter. All prior experiments use a ridge parameter (A) that increases with dataset complexity: À = 10-6 for MNIST, 10 for CIFAR 10 and CIFAR100, and 10 for TinylmageNet and minilmageNNt. Table 6 (left, middle) shows the effect varying À on RanDumb's performance. With a smaller À 10-6 CIFARIO, CIFARIO0 100, TinylmageNet and minilmageNet all exhibit minor drops of 1..%% 0.8%, 0.8%. Increasing shrinkage to and CIFAR 100 performance more substantially by 3% and 2.5% versus their optimal X = 10-3 the other hand, this larger leads improvements of 0.5% and 1.8% on TinylmageNet and minilmageNet. This aligns with the trend that datasets with greater complexity benefit from more regularisation, with the optimal balaneing under- and over-regularisation effects.
10-4 reduces CIFARIO
Comparison with Extreme Learning Machines. We compared our random Fourier features with random projections based extreme learning machines, recently adapted to continual learning by RP+ReLU [41] in Table 6 (left, bottom) with their best embedding size. Our method performs significantly better on each dataset, averaging gain of 3.4%.
Comparisons across Architectures. In table 6 (right), we compare whether using random Fourier features as embeddings outperforms models across various architectures for continual representation learning. We use experience replay (ER) baseline in the task-incremental CIFAR setup (for details, see Mirzadeh et al. [45] as it differs significantly from earlier setups). Our comparison spanned various architectures. The findings revealed that RanDumb surpassed the performance of nearly all considered architectures, and achieved close to 94% of the joint multi-task performance. This suggests that RanDumb outperforms continual representation learning across architectures.
Conclusion. Overall, both random embedding and decorrelation are critical components in the performance of RanDumb. Using random Fourier features is substantially better than Ran- PAC. Lastly, one can substantially reduce the embedding dimension without a large drop in performance for large gains in computational cost, additional augmentation may further signif- icantly help performance and optimal shrinkage parameter increases with dataset complexity. RanDumb outperforms continual representation learning across wide range of architectures.

---

RanDumb
Decarrelate
# Teie
NCM
Linear
Traditional
30 Monte
Figure 4: In previous experiments, models were trained from scratch, so we used random projec- tion. Here, with a pretrained backbone, RanDumb starts with the frozen pretrained backbone to explore whether continual representation learning is necessary. By comparing this frozen backbone (RanDumb) with a continually trained one, show that using the pretrained features consistently matches the best continually learned representations, similarly challenging the value of continual representation learning.
Table 7: Benchmark F We compare RanDumb with prompt-tuning approaches using ViT-B/16 ImageNet- 21K/IK pretrained models using 2 init classes and class per task setting. Most prompt-tuning based methods col- lapse and RanDumb achieves either state-of-the-an or second-best performance. RanPAC-imp is improved version of the RanPAC mitigating the instability issues identified in a previous version of this work.
3.3 Should
we learn representations when strong pre-trained features are available?
Say for a specific application (e.g., where the test data distribution is more or less known during training), practitioners should use strong pretrained models as starting point as they are likely to per- form better. However, we still ask the key question of whether representation learning is necessary by fixing the pre- trained backbone and only training the linear classifier, as illustrated in Figure 4 in the benchmark.
Method
CIFAR IN-A IN-R CUB VTAB
VIT-B/16 (IN- IK Pretrained)
Finetune
1.0
1.2
1.1
1.0
2.1
L2P [68]
2.4
0.3
0.8
1.3
DualPrompt [67]
2.3
0.3
0.8
0.9
4.2
CODA-Prompi [59]
2.6
0.3
0.8
1.9
6.3
Adam-Adapt [8
76.7
49.3
62.0
85.2
83.6
Benchmark F. We compare perfor- mance of approaches which do not fur- ther train the deep network like Ran- Dumb against popular continual finetun- ing and prompt-tuning approaches in Ta- ble 7. We discover that prompt-tuning approaches completely collapse under large timesteps and approaches which do not finetune their pretrained model achieve strong performance, even under challenging one class per timestep con- straint. Note that RanPAC [41] adds a RP+ReLU and finetunes in firs-sesssinn adaptation fashion over RanDumb, yet fails to achieve higher accuracies.
Adam-SSF [81]
76.0
47.3
64.2
85.6
84.2
Adam-VPT [81]
79.3
35.8
61.2
83.8
86.9
Adam-FFT [81]
72.6
49.3
85.2 61.0
83.8
Memo [80]
69.8
81.4
iCARL [54]
72.4
35.2
72.4
Foster [66]
52.2
76.8
86.6
NCM [31]
78.3
44.3
62.5
84.8
88.2
SLCA [79]
86.3
52.8
84.7
RanPAC [41]
88.2
39.0
72.8
77.7
93.0
RanPAC-imp [41]
87.8
43.5
72.6
89.6
93.0
RanDumb (Ours)
84.5
49.5
66.9
88.0
93.6
VIT-B/16 (IN-2 IK Pretrained)
Finetune
2.8
0.5
1.2
1.2
0.5
Adam-Adapt [81]
82.4
48.8
55.4
86.7
84.4
Adam-SSF [81]
82.7
46.0
59.7
86.2
84.9
Why is the performance of prompt- ing methods low? We argue that the true test of continual learning lies in the ability to leam over prolonged periods. Our observations indicate that prompt- ing methods collapse early across tasks because the learned prompts do not gen- eralize effectively. Supporting this, paral- lel work [62] suggests that most current prompt-tuning methods lack prompt di-
Adam-VPP [81]
70.8
34.8
53.9
84.0
81.1
Adam-FT [81]
65.7
48.5
56.1
86.5
84.4
Foster [66]
87.3
5.1
86.9
iCARL [54]
71.6
35.1
71.6
NCM [31]
83.5
41.4
54.8
86.5
88.5
SLCA [79]
86.8
54.2
82.1
RanPAC [41]
89.6
26.8
67.3
87.2
88.2
RanPAC-imp [41]
89.4
33.8
69.4
89.6
91.9
RanDumb (Ours)
86.8
42.2
64.9
88.5
92.4

---

versity and can be characterized by a single prompt, making classification performance heavily reliant on the quality of that prompt. We hypothesize when prompts are designed for large number of classes (c.g., 20 or 50), they produce generally discriminative representations that extend to future tasks. However. prompts designed for only two classes. as in our case, have limited discriminative power, leading to the collapse of pommpttuunig methods across tasks.
Overall, despite RanDumb being exemplar- freee, it outperforms nearly all online continual learning methods across various tasks when exemplar storage is limited. We specifically benchmark on lower exemplar sizes to complement settings in which GDumb does not perform well.
4
Related Works
There have been extensive theoretical and empirical investigations into random representations in machine learning, compressed sensing, and other fields, often utilizing extreme learning machines [56, 14, 21J (see [30, 29] for a survey). Other investigations include efficient kernel methods using Fourier features and Nyström approximations [52, 70J, and extensions to efficiently parameterize linear classifiers [2]. They are also embedded into deep networks [17, 35, 73, 16]. We tailored the already successful random fourier representations [52] to the problem at hand and applied to the online continual learning problem for the first time.
Random Representations.
Continual Representation Learning. There are various works focusing continual representation learning itself [53, 20, 39, 28), but they address the problem of alleviating the stability- stability-plasticiy dilemma in high-exemplar and offline continual learning scenarios where models are trained until convergence. comparison, we focus on online and low-examplar regime.
Representation Learning Free Methods in CL. Several works have developed the idea of using
fixed pretrained networks after adapting on the first task across various settings [50, 41, 24]. Our work contributes to this growing evidence, however, we do not perform first-task adaptation [47]. and propose
without any extra SLDA structurally simplest but highly accurate continual linear classifier Moreover, we are the first work to introduce a representation learning free method with random features for continually learning from scratch.
Equivalent formulations to RanDumb. If the classes are equiprobable, which the case for most datasets here, nearest class mean classifier with the Mahalanobis distance metric is equivalent to linear discriminant analysis (LDA) classifier [42]. Hence, one could say RanDumb is exactly equivalent to a Streaming LDA classifier with an approximate RBF Kernel. Alternatively, one could think of the decorrelation operation explicitly decorrelating the features with ZCA whitening [7].
5 Discussion and Concluding Remarks
Our investigation reveals surprising result simply using random embedding (RanDumb) consis- tently outperforms learned representations from methods specifically designed for online continual training. Furthermore, using random/pretrained features also recovers 70-90% of the gap to joint learning, leaving limited room for improvement in representation learning techniques on standard benchmarks. Overall, our investigation questions our understanding of how to effectively design and train models that require efficient continual representation learning, and necessitates a re- re-investigation of the widely explored problem formulation itself. We believe adoption of computationally bounded scenarios without memory constraints and corresponding benchmarks [51, 50, 22] could be a promis- ing way forward.
Limitations & Future Directions. We currently do not provide theory or justification for why training dynamics of continual learning algorithms fails to effectively learn good representations; doing so would provide deeper insights into continual learning algorithms. Moreover, our proposed method, RanDumb with random Fourier features is limited in scope towards low-exemplar scenarios and online-continual learning. Extending studies on representation learning high-exemplar and offline continual learning scenarios might be exciting directions to investigate.
Social Impact. RanDumb is an algorithm solely designed to perform scientific study and we do not recommend use of Dumb for any application in real-world production systems, hence no direct socictal impact or explicit limitations use in production systems is discussed.
10

---

Acknowledgements
AP funded by Meta AT Grant No. DFR05540. PT thanks the Royal Academy of Engineering. PT and PD thank FiveAl for their support. This work is supported in part by UKRI grant: Turing AI Fellowship EP/W002981// 1/1 and an EPSRC/MURI grant: EP/N019474/1. The authors would like to thank Arvindh Arun, Kalyan Ramakrishnan and Shashwat Goel for helpful feedback.
References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim. Hyeonsu Bang. Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In ICCV, 2021.
[2] Nir Ailon and Bernard Chazelle. The fast johnson--lnnenstraans transform and approximate nearest neighbors. SIAM Journal on computing. 2009.
[3] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NewrlPS, 2019.
[4] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, 2019.
[5] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In ICLR, 2022.
[6] Jihwan Bang. Heesu Kim, YoungJoon Yoo, Jung-Wop Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
[7] Anthony Bell and Terrence Sejnowski. Edges are the independent components' of natural scenes. In NewrIPS, 1996.
Mattco Boschini, Lorenzo Bonicelli, Pictro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. TPAMI, 2022.
[9] Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. NeurIPS, 2020.
[10] Lucas Caccia, Rabaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights reducing abrupt representation change in online continual learning. In ICLR, 2022.
[11] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV, 2021.
[12] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In JCLR, 2019.
[13] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. In ICML-W, 2019.
[14] CL Philip Chen. A rapid supervised learning neural network for function interpolation and approximation. IEEE Transactions on Neural Networks, 1996.
[15] Yilun Chen, Ami Wiesel, Yonina Eldar, and Alfred O Hero. Shrinkage algorithms for mmsee covariance estimation. IEEE transactions on signal processing, 2010.
[16] Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV, 2015.
[17] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. NewrIPS, 2009.
[18] Aristotelis Chrysakis and Maric-Francine Moens. Online bias correction for task-free continual learning. In ICLR, 2023.
[19] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. ICCV, 2021.
[20] Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pines Pineda. Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-super models are continual learners. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[21] Dmitriy Fradkin and David Madigan. Experiments with random projections for machine learning. In KDD.
2003.
[22] Saurabh Garg, Mchrdad Farajtabar, Hadi Pouramsari, Raviteja Vemulapalli, Sachin Mchta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. ArXiv, 2023.
[23] Ross Girshick. Jeff Donabue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.

---

[24] Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NewrlPS, 2023.
[25] Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maxi- mization. In JCML, 2022.
[26] Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. NeurlPS, 2020.
[27] Tyler L Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In CVPR-W, 2020.
[28] Timm Hess, Eli Verwimp. Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting. arXiv preprint arXiv:2304.000 000933 2023.
[29] Guang-Bi Bin Huang, Qin-Yu Yu Zhu, and Shee-KKeong Siew. Extreme learning machine: theory and applica- tions. Neurocomputing, 2006.
[30] Guang-Bin Bin Huang, Dian Hui Wang, and Yuan Lan. Extreme learning machines: survey. International journal of machine learning and cybernetics, 2011.
[31] Paul Janson, Wenxuan Zhang. Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. In NeurlPS-W. W, 2022.
[32] Xisen Jin, Junyi Du, and Xiang Ren. Gradient based memory editing for task-free continual learning. In NeurlPS, 2021.
[33] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnsikk,, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
[34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017.
[35] Quoc Le, Tamás Sarlós, Alex Smola, et al. sestfood-approximatii kernel expansions in loglinear time. In ICMI,, 2013.
[36] Soochan Lec, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free contimual learning. In ICLR, 2020.
[37] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 2017.
[38] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurlPS,
2017.
[39] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. arXiv preprint arXiv:2/ 10.06976, 2021.
[40] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In CVPR, 2021.
[41] Mark D McDonnell, Dong Gong, Amin Parvench, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. In NeurIPS, 2023.
[42]
Geoffrey McLachlan. Discriminant analysis and statistical pattern recognition. John Wiley & Sons, 2005.
[43] Goeffrey McLachlan. Mahalanobis distance. Resonance, 4(6):20 26. 1999.
[44] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-basee based image classifi- cation: Generalizing to new classes at near-zero cost. TPAMI, 2013.
[45] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXiv preprint arXiv:2202.0 00275, 2022,
[46] Sudhanshu Mittal, Silvio Galesso. and Thomas Brox, Essentials for class incremental learning. In CVPR, 2021.
[47] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class- incremental leamingg arXiv preprint arXiv:2303.13199 2023.
[48] Karl Ezra Pilario, Mahmood Shafice, Yi Cao, Liyun Lao, and Shuang-Hua Yang. A review of kernel methods for feature extraction in nonlinear process monitoring. Processes, 2020. doi: 10.3390/pr8010024
[49] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions progress continual learning. ECCV, 2020.
[50] Ameya Prabhu, Zhipeng Cai, Puncet Dokania, Philip Torr. Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253 2023.
12

---

[51] Ameya Prabhu. Hasan Abed AI Kader Hammoud, Puneet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR, 2023.
[52] Ali Rahimi and Benjamin Recht. Random features for large-scale kemnel machines. NewrlPS, 2007.
[53] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell, Continual unsupervised representation reppeettiiserrpreteetttttoooo learning. NewrlPS, 2019.
[54] Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: classifier and representation learning. In CVPR, 2017.
Incremental
[55] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. ICLR, 2019.
[56] Wouter F Schmidt, Martin A Kraaijveld, Robert PW Duin, et al. Feed forward neural networks with random weights. In ICPR. 1992.
[57] Pierre Sermanet, David Eigen, Xiang Zhang. Michaël Mathieu, Rob Fergus, and Yann LeCun. Over- feat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint 17Xiv:1312.622 2013.
[58] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class inusemnncrmental continual learning with adversarial shapley value. In AAAI, 2021.
[59] James Seale Smith, Iconid Karlinsky, Vyshnavi Gutta, Paola Cascante-Boninn Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehcarsal-free continual learning. In CVPR, 2023.
[60] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning. In CVPR-W W, 2023.
[61] Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.071 2023.
[62] Lukas Thede, Karsten Roth, Olivier J Hénaff, Matthias Bethge, and Zeynep Akata. Reflecting on the state of rehearsal-free continual learning with pretrained models. arXiv preprint 2Xix:24006093844 2024.
[63] Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. In NeurlPS-W W, 2018.
[64] Gido Van Ven, Zhe Li, and Andreas S Tolias. Class-incremental learning with generative classifiers In CVPR- W, 2021.
[65] Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler I. Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph Lampert, et al. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311. 1.:::2/11.1111 1908, 2023.
[66] Fu- Yu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class inssemnncrmmettll learning. ECCV, 2022.
[67] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot. Jennifer Dy, et al. Dualprompt: Complementary prompting for fehearsal-free continual learning. In European Conference on Computer Vision (ECCV), 2022.
[68] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot,Jennife Dy, and Tomas Pfister. Learning prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[69] Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In ICCV, 2023.
[70] Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. NewrIPS, 2000.
[71] Yichen Wu, Long-Ko Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In ICLR, 2024. URL https://openreviee
net/forumt iitrrmmmdddTDD2achoo
[72] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu, Large scale incremental learning. In CVPR, 2019.
[73] Zichao Yang, Marein Moczulski, Misha Denil, Nando De Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In ICCV, 2015.
[74] Fei Ye and Adrian G Bors. Continual variational autoencoder Icarning via online cooperative memorization. In ECCV, 2022.
[75] Fei Ye and Adrian G Bors. Self-evolved dynamic expansion model for fase-free continual learning. In ICCV, 2023.
13

---

[76] Michal Zajac. Tinne Tuytelaars. and Gido M van de Ven. Prediction error-based classification for class- incremental learning. ICLR, 2024.
[77] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In JCML, 2017.
[78] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprini arXiv: 10123, 2018.
[79] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow leamnr with classifier alignment for continual learning pre-trained model. In ICCV, 2023.
[80] Da- Wei Zhou, Qi-we Wei Wang. Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. arXiv preprint arXiv:2205. 13218, 2022.
[81] DeiWei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338
2023.
[82] Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng- lin Liu. Class incremental learning via dual augmentation. NeurIPS, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and/or introduction should clearly state the claims made, including the contributions made the paper along with important assumptions.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitation Section is provided in the supplementary material.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Rahimi and Recht [52] from our references details the theory for why random fourier representations perform well quite beautifully. The random representations do not change (no continual aspect), hence the theory can applied as-is in our case with no changes. We do not claim any novel theoretical contributions.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: RanDumb is fairly simple to implement. We dedicated half page towards explaining hyperparameters and other information needed to reproduce all of our results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code in the supplementary material to reproduce our results.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We use standard datasets and splits, we provide hyperparameters in experimen- tal details along with ablations in experiment sections to understand the contribution of each component in our algorithm.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We state here that the only random component being the random fourier features kernel across, otherwise our method is simple and exactly reproducible. We conducted experiments with three different initialisations corresponding to seeds of the random kernel in sklearn to investigate this and different kernel initialisations lead to around +0.2 variation in the reported results.
15

---

8. Experiments Compute Resourees
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed Section 3 in Implementation Details. For easy access, we restate: All experiments were conducted CPU server with a153 48-core Intel Xeon Platinum 8268 CPU and 392GB of RAM, requiring less than minutes per experiment.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tttpp:://eeuree neurips....../. cc/ Beppalic/Ethissaaaiiiieeeeeeaeee
Answer: [Yes]
Justification: We have read the ethics guidelines and confirm that we do not use human subjects, use existing datasets, explicitly discuss social impacts.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The primary contribution in RanDumb was not to introduce a novel state- of-the-art continual learning method, but challenge prevailing assumptions and open discussion on the efficacy of representation learning in continual learning algorithms. As such, we do not recommend use of RanDumb for deployment in real-world production systems, hence no direct societal impact or explicit limitations on use in production systems is discussed.
Justification:
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not have any high-risk model or dataset introduced.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: MNIST, CIFAR-10, CIFAR-100, tinylmageNet and minilmagenet. are cited appropriately. The licenses for these datasets is not explicitly released, hence we do not include that information.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we provide RanDumb with proper documentation under GPL3 license.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing research with human subjects was performed.
16

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing research with human subjects was performed.
17

---

A Conceptual and Methodological Differences from GDumb [49]
Our core claim that random representations from raw image pixels consistently outperform deep- learning-basee representations designed for online continual learning. In contrast, GDumb's 's central claim is that continual learning methods need not actually minimize forgetting of previous online samples as their performance can be entirely recovered simply by a baseline using the latest memory.
The key differences between thee two works is as follows:
Forgetting of Online Samples (GDumb): GDumb argues that continual learning methods suffer from forgetting online samples and addresses this by relying entirely on memory. This dismisses the value of directly learning from online data, as GDumb does not retain or utilize information from online samples. RanDumb, in contrast, exclusively learns from online samples without using memory, emphasizing the significance of ongoing data streams for performance.
Inadequate Representation Learning (RanDumb): GDumb's s approach to representation learning mirrors the experience replay (ER) baseline but does not address the quality of the learned representations. RanDumb explicitly focuses the inadequacy of these representations and ablates their role to highlight their impact on continual learning. This reveals a key distinction in how each method evaluates representation quality in online learning settings.
Furthermore, the experimental setups for RanDumb and GDumb highlight their complementary nature:
RanDumb primarily targets low- ow-memory environments, whereas GDumb excels in high-memory scenarios. For example, in rehearsal -reeeersl-feeeeeeeeeeeee settings without exemplar storage, a common trend in continual learning, GDumb would inherently produce random performance due its dependence on memory. RanDumb, on the other hand, thrives in these low-memory contexts, providing an alternative solution when memory is constrained.
Memory Settings:
Complementary Nature: RanDumb and GDumb occupy complementary spaces within the continual learning landscape. RanDumb performs well in benchmarks where GDumb falters, and vice versa. As GDumb has been acknowledged as a valuable baseline, we argue that RanDumb similarly deserves recognition in the continual learning literature for its distinct strengths.
In summary, the only commonality between RanDumb and GDumb is that they serve as simple baselines. The points above underscore the fundamental distinctions between the two methods and the specific aims behind their development, as emphasized here and in the title of this work.
18

---

B Online Continual Learning: Our Setting
Current Problem formulation. We formally define the online continual learning (OCL) problem as follows. In classification settings, we aim to continually learn a function f: X + Y parameterized by O2 at time t. OCL is an iterative process where each step consists of learner receiving information and updating its model. For RanDumb, at each step & of the interaction,
1. One data point (Ie, (1) ~ Te sampled from a non-stationary distribution n, is revealed.
2. Learner updates the model 0s+1 using a compute budget, Blearn and discards the datapoint.
Simplifications by Compared Approaches. Traditional online continual learning literature makes several concessions over this which makes the problem casier by allowing the datapoint to be saved for more timesteps. Training deep networks requires those simplifications as more data per batch helps stabilize the gradient updates. Typically, compared approaches store samples across for 10 timesteps, and performs an update with that batch of samples before discarding it. Most works further relax this by storing memory buffer of samples indefinitely.
Drawbacks. Traditional online continual learning setups cannot effectively test for rapid adaptation because they use a inseemeactttt setup. Online learning is generally intended to enable quick adaptation to changing data and label distributions in a data stream. We believe better formulation for online continual learning is described in [50].
19


Reviews:
[{"review_id": "PNG4MgE5cC", "reviewer": "Reviewer_k3sw", "summary": "The authors show that a model with random Fourier features as a representation, followed by a normalisation then nearest class means for classification, outperforms most Continual Learning methods in online CL benchmarks. The representation is not learned as opposed to other online CL methods the model is compared against. Based on this result, the authors conclude that, in the online learning setting, continual learning algorithms fail to learn effective representations. The second main contribution is that this naive learning method bridges 70% to 90% of the performance of joint learning, leaving a small improvement gap to develop an effective continual representation learning method.", "questions": "One point I wanted to discuss was the following : One property of RanDumb is that training is not gradient based. This contrasts of all other training methods. Given that the other methods train the final layer with gradient based optimisation methods, it could be that the representations are still decent for previous tasks but that the forgetting occurs mostly in the final layer. I was thinking that, in order to validate that the other continual learning methods don't learn effective representations, one experiment could be : Run the full CL sequence on a given dataset with a given model, training the full model. then freeze the learned representation and check the joint accuracy with the frozen representation vs the joint accuracy with a fully trainable model.\nI am curious about your thoughts on this point.", "limitations": "There are no potential negative societal impacts of the work. The authors adequately highlighted various limitations of their work. These limitations are out of scope of the current work and can be addressed in follow-up investigations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality : \n   - Even though the observation is simple, it highlights a blind spot in the field. Also, the naive model contrasts with most methods in the field and raises interesting questions.\n- Quality : \n   - The authors present extensive benchmarks and each benchmark is clearly motivated wrt the literature and the scope of the paper.\n   - An ablation study on the components of the naive model is presented, highlighting that both the normalisation and Fourier features are critical for the performance.\n- Clarity : \n   - The motivation, method and results are very clear and easy to follow.\n   - The paper is well structured as well, it's relatively easy to browse the results and sections.\n- Significance : \n   - This study will likely motivate subsequent works to investigate online representation learning", "weaknesses": "I don't see any major weaknesses, except the question I raise in the section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful question. To compare the effect, we will use two baselines: ER  [13] and iCARL [54].\n\n**Effect of gradient-free classifiers:** The primary difference between these methods is that ER trains the last layer along with the previous network, while iCARL learns a non-gradient based Nearest Class Mean (NCM) classifier. Similar to iCARL, RanDumb also employs an NCM classifier, but differs as it does not train a deep network.\n\n\n| Method          | Embedder      | Classifier | CIFAR10 (5/2) M = 0.1k | CIFAR10 (5/2) M = 0.2k | CIFAR100 (10/10) M = 0.5k | CIFAR100 (10/10) M = 1k | TinyImageNet  (100/2) M = 1k | TinyImageNet (100/2) M = 2k |\n|-----------------|---------------|------------|------------------------|------------------------|---------------------------|-------------------------|------------------------------|-----------------------------|\n| ER [13]         | Deep network  | Linear     | 19.4                            | 29.7                            | 8.7                                | 15.7                        | 2.5                             | 5.6                                |\n| iCARL [54]      | Deep network  | NCM        | 31.0                            | 33.9                            | 12.8                               | 16.5                        | 5.0                             | 6.6                                |\n| RanDumb (Ours)  | Random        | NCM        | 55.6                            | 55.6                            | 28.6                               |  28.6                       | 11.6                           | 11.6                                |     \n\n\nTraining the last layer in a gradient-based manner, as noted by the reviewer, can lead to an additional degree of forgetting. However, even after addressing this issue, there remains a significant gap  in performance that needs to be addressed, as highlighted by our comparison with RanDumb.\n\nThe experiment proposed by the reviewer requires two passes over the data, not permissible in the OCL setting. We suggest that our experiment above might provide a valid comparison to clarify the concern while maintaining the online continual learning scenario.\n\nWe hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZHUnoeRM8", "reviewer": "Reviewer_8BCQ", "summary": "To obtain powerful representation in an online continual learning setting, the authors propose a new learning method referred to as RanDumb, that embeds raw pixels using a fixed random transform, approximating an RBF kernel initialized before any data is seen.\n\nThe proposed model trains a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting.\n\nExtensive experiments demonstrate its effectiveness and power with several ablations.", "questions": "Please refer to the weakness.", "limitations": "The embedding (RBF-kernel) is unclear; the authors could better explain how the random Fourier projection affects continual learning input representation with simple math.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) Extending the investigation to popular exemplar-free scenarios with pre-trained models, this work shows that training only a linear classifier on top of pre-trained representations surpasses most continual fine-tuning and prompt-tuning strategies.\n\n(+) The author's investigation challenges the prevailing assumptions about effective representation learning in online continual learning.\n\n(+) The authors explain how the random Fourier basis (a low-rank data-independent approximation of the RBF Kernel) affects the representation of online continual learning along with the accuracy of RanDumb (embedding dimensions).", "weaknesses": "(-)  The structure of RamDumb (Fig. 1) is too simple to follow the structure. Could the authors depict the architecture of input, RBF-Kernel, pre-trained model (ViT), Decorrelate(D), and NCM(C)? or include an additional figure to illustrate the RamDumb.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in reviewing our work and for providing their concerns and suggestions. We appreciate the recognition of the work's contribution in *\"challenging the prevailing assumptions about effective representation learning in online continual learning.\"* The acknowledgment of our *\"extensive experiments\"* and results *\"extending the investigation to popular exemplar-free scenarios with pre-trained models\"* is highly valued.\n\nBelow, we address the reviewer’s request for clarification on RanDumb’s structure.\n\n> P1 [Structure of RanDumb needs Clarity]: (i) Could the authors depict the architecture of input, RBF-Kernel, pre-trained model (ViT), Decorrelate(D), and NCM(C)? Or include an additional figure to illustrate the RanDumb structure.\n> \n> P2 [Embedding]: The embedding (RBF-kernel) is unclear; the authors could better explain how the random Fourier projection affects continual learning input representation with simple math.\n\nWe would like to clarify the structure of RanDumb across benchmarks (and will add a figure in the revised draft to clarify this).\n\n- **For training from scratch**:  \n  Raw flattened pixels are inputted, followed by random projection, decorrelation, and then classification using the NCM (as described in Equation 3). No input representations are learned; we use raw pixels as inputs, as illustrated in Figure 1.\n\n- **For pretrained models**:  \n  The input image is passed through the pretrained model to obtain features, followed by decorrelation, and then NCM classification. Mathematically, this would be the same as Equation 3 if we overload $\\phi$ as the pretrained model instead of random projection.\n\nWe acknowledge our initial description was inadequate and appreciate you bringing this to our attention. We will clarify this in the paper with a separate figure for pretrained models and additional equations to avoid overloading $\\phi$.\n\nWe hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qQvLsisYX", "reviewer": "Reviewer_sAWs", "summary": "This paper proposes RanDumb, a representation learning-free method for online continual learning (OCL). It uses data-independent random Fourier transform to project the data to a high-dimensional space (embed), scales the features to have unit variance (decorrelation), and finally performs classification with a nearest mean classifier (NCM) in the feature space. \n\nThe authors compares RanDumb with existing methods (when they are trained from scratch) on various OCL benchmarks. The authors additionally substitute the \"embed\" part of RanDumb with frozen pretrained representations and find that it still performs decently compared to existing methods that finetune the representations, on one of the benchmarks.", "questions": "I listed my main concerns in the weaknesses section above and I only have some minor questions or comments here.\n\n1. I wonder if it's worth providing the definition of OCL clearly so that readers not familiar with OCL can grasp the ideas more easily. Also, I recommend adding details on the method explicitly (even if it's just a few equations in the appendix) rather than only providing references. For example, I'd appreciate seeing how the inverse of the covariance matrix is computed online.\n\n2. What part of the results are the reported numbers in prior work? Is it all the numbers except for the proposed method? \n\n3. I thought Co$^2$L was a contrastive learning-based method. Why is it included in benchmark B.1 and not B.2?\n\n4. Could the authors provide some hypothesis on why fine-tuning/prompt-tuning based methods collapse in Table 6?", "limitations": "The authors have addressed the limitations clearly in Sec. 5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and I did not find any major technical flaws.\n\n2. I find that random representations outperform continually trained representations (from scratch) under the data/memory-constraint continual learning regime interesting and novel.\n\n3. The benchmarks used are extensive, involving multiple datasets (MNIST, CIFAR, subsets of ImageNet, etc.), different levels of OCL constraints (e.g., number of classes per task), and different model architectures (ViT, ResNet, etc.).", "weaknesses": "1. I think the main novelty lies in the usage of random Fourier features because, without it, the proposed method is very similar to SLDA. Therefore, I'm not entirely convinced by the motivation. It makes sense that representation learning from scratch performs poorly in the low-data regime. In what cases do we not want to use a frozen pretrained representation directly? \n\n\n2. The method involves using a very large embedding space (25K in most experiments), which raises concerns on efficiency. Kernel trick is efficient because you can compute the dot product in a high-dimensional space without explicitly projecting the data to that space, but I'm not sure how the dot product $\\phi(x)^\\top \\bar{\\mu}_i$ can be computed efficiently here. Could the authors elaborate on that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time reviewing our work and providing encouraging remarks such as *\"no major technical flaws,\"* *\"interesting and novel,\"* and *\"extensive benchmarks.\"* We also appreciate giving the presentation of our work a remark of being *\"excellent.\"*\n\nBelow, we seek to address the potential weaknesses/questions raised by the reviewer in the same order as noted by the reviewer.\n\n---\n\n> \"I think the main novelty lies in the usage of random Fourier features because, without it, the proposed method is very similar to SLDA. Therefore, I'm not entirely convinced by the motivation. It makes sense that representation learning from scratch performs poorly in the low-data regime. In what cases do we not want to use a frozen pretrained representation directly?\"\n\nWhen strong pre-trained features are available for a specific application (e.g., where the test data distribution is more or less known during training), practitioners should use them as they are likely to perform better. However, in the absence of such pre-trained representations, as is the case with online continual learning where the future datapoints/classes are unknown, we argue that a baseline, such as ours in this work that simply uses random features, should be first evaluated thoroughly before relying too much on the online representation learning methods that exist in the literature.\n\nWe have made the above point empirically as well via extensive experiments where we show that several of the existing continual learning methods (including VR-MCL—the ICLR'24 best paper honorable mention [70]) fail to outperform our simple non-deep-learning based random feature baseline! We strongly believe that our random transform as a baseline will help in guiding future research directions. We are thankful to the reviewer for raising this question as we believe that adding a small discussion and related implications around this in our revised draft would strengthen our arguments further.\n\n---\n\n> \"The method involves using a very large embedding space (25K in most experiments), which raises concerns on efficiency. Kernel trick is efficient because you can compute the dot product in a high-dimensional space without explicitly projecting the data to that space, but I'm not sure how the dot product can be computed efficiently here.\"\n\nWe will expand our discussion on implementation details of our method versus SLDA [27]. Briefly, computing the dot product was extremely fast. For our use case, the runtime (on an Alienware x17 R2 laptop for 25K dimensions) was **0.3±0.02 seconds**.\n\nHowever, as you asked for details in Q1, the major bottleneck is not the inner product, it is computing $S^{-1}$. We use a standard approach of solving linear systems $S \\cdot x = \\mu$ for this using a heavily optimized function in PyTorch called `torch.linalg.lstsq`. The solution $S^{-1} \\mu$ is then used to compute $x^T (S^{-1} \\mu) - \\mu^T (S^{-1} \\mu)$.\n\nCertainly, one could just invert the matrix $S$ at the beginning and then use the Sherman-Morrison formula for online updates as each update in our case is rank-1.\n\n---\n\n> \"I wonder if it's worth providing the definition of OCL clearly so that readers not familiar with OCL can grasp the ideas more easily. Also, I recommend adding details on the method explicitly (even if it's just a few equations in the appendix) rather than only providing references. For example, I'd appreciate seeing how the inverse of the covariance matrix is computed online.\"\n\nThank you for this question. We agree that providing a more formal definition of OCL will improve the readability of the paper. We will provide that in our revised draft. For details about matrix inversion, please check the answer above.\n\n---\n\n> \"What part of the results are the reported numbers in prior work? Is it all the numbers except for the proposed method?\"\n\nWe clarify—all numbers in tables where the caption says (Ref: table and citation) except our method are taken from the aforementioned table in the cited paper. We ensure that the experimental settings are exactly the same. Please note that reporting numbers directly from prior works makes the baselines stronger as these are already heavily optimized by the authors of their corresponding papers, which most of the time is difficult to reproduce as the specific design choices (hyper-parameters, etc.) to achieve such numbers are not always well explained in the paper.\n\n---\n\n> \"I thought Co2L was a contrastive learning-based method. Why is it included in benchmark B.1 and not B.2?\"\n\nThank you for bringing this to our notice. We will correct this right away!\n\n---\n\n> \"Could the authors provide some hypothesis on why fine-tuning/prompt-tuning based methods collapse in Table 6?\"\n\nThank you for the great question. Based on recent research, and our observation that the collapse is early across tasks:\n\nParallel work {1} (will be cited and included in the revised draft) suggests that most current prompt-tuning methods often suffer from a lack of prompt diversity and can be characterized with a single prompt. As a result, the effectiveness of classification depends heavily on the quality of that prompt.\n\nWhen designing a prompt for a large number of classes (e.g., 50 or 20), the prompt leads to generally discriminative representations (even for future tasks), whereas a prompt for just 2 classes, in our case, is limited in its discriminative power. This causes all prompt-tuning methods to collapse across tasks.\n\n{1} Thede et. al., Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models, CoLLAs 2024\n\n---\n\nWe hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UqS4CMoydt", "reviewer": "Reviewer_5ZCt", "summary": "The paper shows that a fixed random feature space of the data, followed by some linear classifier, outperforms continually learned feature spaces across multiple benchmarks. The paper then shows an ablation study of the suggested method.", "questions": "Can you please elaborate on the problems in CL that this paper introduces, and the difference between them and the points raised in g-dumb?", "limitations": "not relevant", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Clarity:** The suggested idea is simple, and it is easy to understand how it works and why.\n\n**Quality:** The random features are tested across multiple benchmarks, showing improvements in many of them. The ablation study is comprehensive. Overall, I find the results very convincing: I believe that RanDumb can outperform many continual learning algorithms, despite it being learned using a single pass over the data in a non-deep learning way.\n\n**Significance:** This kind of meta-study, showing weak points of the entire field rather than weak points of a specific work is important for the community. The paper shows that continual learning methods are not the optimal solutions for many continual learning benchmarks.", "weaknesses": "I'll review this paper from 2 perspectives: the first is from a \"technical\" point of view, judging the actual suggested method RanDumb. The second is the \"meta\" point of view, judging the \"meta\" claim of the paper, that we should re-think CL methods, as they are not optimal for many of the existing CL benchmarks.\n\n**Originality:**\nFrom the technical point of view, the suggested method is simply an implementation of known methods on several different benchmarks, and as such has very limited originality. From the meta point of view, I think that the additional benefits of this paper over g-dumb [1] are somehow limited, as it has been shown before that CL methods do not optimally solve the CL benchmarks in many cases.\n\n**Quality and significance:**\nWhile the quality of the technical point is high, I'm not so convinced about the meta point. The discussion in the paper about the meta point is rather minimalistic and is left for the reader. A more thorough discussion will be useful, especially a more in-depth comparison with previous works that suggested similar things, such as g-dumb.\nAs I don't find the \"technical\" point of view of the paper novel significant enough to merit a top-tier publication in its own right, I think the paper should focus more on the meta point of view. Specifically, I find that there is a missing discussion on why the existence of simple methods that outperform CL methods on CL benchmarks should be so worrisome. The counter-argument is that the benchmarks themselves are more of \"toy problems\", and that achieving high performance on them is not the major goal of the works, but instead learning how to deal with problems that deep models encounter when facing changing data, including catastrophic forgetting and distribution shifts. \n\n**Clarity:**\nWhile the main idea is clear, the paper is very confusing to read. The different benchmarks are marked by letters, which makes it hard to follow which is which. Moreover, I often had to go to other papers to understand the experiments that were done in a better way, as this formulation was very confusing. This is a major problem in the current version, as a substantial part of the paper revolves around this formulation.\n\n\n\n\n**Summary:**\nI find the technical point of view good and convincing, but not significant and novel enough for to merit a publication in a top-tier conference. The meta point of view in my opinion is important and relevant for the community, but is not supported enough by the paper. Therefore, I am leaning toward rejection. \n\n\n\n\n\n\n\n\n\n\n------\n\n[1] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. \"Gdumb: A simple approach that questions our progress in continual learning.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback. We appreciate the recognition that our idea is simple and easy to understand, as well as the recognition of the comprehensive nature of our ablation study and the improvements demonstrated across multiple benchmarks. The reviewer also pointed out that our meta-study, which highlights weak points of the entire field rather than specific works, is important for the community. We seek to address the concerns raised by the reviewer below. Below we have merged different parts of the review that come under the reviewer’s “meta-claim” and “technical claims” perspectives of the paper to provide a thorough reply in these aspects.\n\n---\n\n## Meta Claim\n\n> \"... judging the 'meta' claim of the paper, that we should re-think CL methods, as they are not optimal for many of the existing CL benchmarks. I think that the additional benefits of this paper over g-dumb [1] are somehow limited, as it has been shown before that CL methods do not optimally solve the CL benchmarks in many cases. Can you please elaborate on the problems in CL that this paper introduces, and the difference between them and the points raised in g-dumb?\"\n\n- Our meta-claim, as consistently stated across the work, is that *random representations of the raw pixels of the images consistently outperform the deep-learning based learned representations from methods specifically designed for online continual training*.\n\n- In contrast, the meta-claim made in GDumb is that continual learning methods forget all previous online samples, and their performance can be recovered by using the latest memory. Additionally, GDumb still uses deep models to justify their claims.\n\n### Main differences between GDumb and RanDumb meta-claims\n\n- **GDumb primarily argues about forgetting of online samples:** GDumb relies entirely on memory and learns nothing from the online samples themselves. This ablates the effect of online samples on performance. In contrast, we only learn from online samples and use no memory.\n\n- **RanDumb primarily argues about inadequate representation learning:** GDumb learns representations similar to the experience replay (ER) baseline and does not make any claims regarding the quality of the representations learned via online representation learning. We specifically ablate representations and discuss the effects.\n\nOverall, there are critical fundamental differences between RanDumb and GDumb in their meta-points.\n\n### Differences between the experimental settings of GDumb and RanDumb\n\n- **RanDumb focuses mainly on low-memory settings,** whereas GDumb primarily shines in high-memory settings. For example, it is trivial to note that in our primary rehearsal-free setting with no exemplars stored (currently popular in continual learning), GDumb would, by design, effectively provide random performance.\n\n- **RanDumb is entirely complementary to GDumb,** i.e., RanDumb shines in benchmarks where GDumb performs poorly and vice-versa. Therefore, just like GDumb, we believe that RanDumb would be a critical addition to the continual learning literature.\n\nOverall, we request the reviewer not to simply dismiss the meta-point because of a broad claim that both RanDumb and GDumb are simple baselines (with no other points in common), and we re-iterate that our meta-point is emphasized above and in the title.\n\n---\n\n## Technical Claim\n\n> \"From the technical point of view, the suggested method is simply an implementation of known methods on several different benchmarks, and as such has very limited originality. I find the technical point of view good and convincing, but not significant and novel enough to merit a publication in a top-tier conference.\"\n\nWe agree that a **part** of our work heavily relies on an existing and very well-cited work of Rahimi and Recht [52] to support our arguments. However, we believe that it is absolutely fine to do so for various reasons:\n\n- It is well accepted in the research community to use fundamental insights and findings from a related field. The above work is a fundamental work with thousands of citations of top-tier conference papers, which have been built on top of this work.\n- We are the first to show the effectiveness of the above approach in continual learning literature.\n- We had to combine various insights from the continual learning literature to make the above work effective across a variety of experiments we presented in this work.\n\nJust the fact that an approach seems simple should never be the reason to undermine its effectiveness, and we hope that the reviewer agrees with us.\n\n---\n\n## Presentation\n\n> \"While the main idea is clear, the paper is very confusing to read. The different benchmarks are marked by letters, which makes it hard to follow which is which...\"\n\nWe would like to highlight that other reviewers found the presentation of the paper to be *\"good\"* or *“excellent.”* They found that *\"This paper is well-written,\" \"The motivation, method, and results are very clear and easy to follow,\"* and even specifically, *\"The paper is well-structured; it is relatively easy to browse the results and sections.\"*\n\nHowever, we do understand that it can sometimes be difficult to follow notations and references. Hence, we had summarized the important details about the different benchmarks relevant to our paper in a table alongside the “Benchmarks” paragraph in Section 3. We also describe critical aspects of the benchmarks in detail there.\n\nHaving said the above, we do acknowledge that labeling experiments by alphabets might affect readability, and we will **improve this aspect by replacing these labels with short, explanatory phrases in our revised draft.**\n\nCould the reviewer suggest further ways to improve the structure of experiments? We are happy to make necessary changes.\n\n---\n\nWe hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The meta-message of this work is actually quite valuable to the community. Inline with the reviewers, I find this work to be ready for acceptance. I urge the authors to incorporate all the feedback they received in the camera ready version of the paper.
Decision: Accept (poster)