Title: RanDumb: Random Representations Outperform Online Continually Learned Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: online continual learning

exemplar-free

baseline

analysis
Abstract: Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach embeds raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb,  significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in the online continual learning.

RanDumb: Random Representations Outperform Online Continually Learned Representations

Abstract

Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach projects raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb, significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in online continual learning.

1 Introduction

Continual learning aims to develop models capable of learning from non-stationary data streams, inspired by the lifelong learning abilities exhibited by humans and the prevalence of such real-world applications (see Verwimp et al. [65] for a survey). It is characterized by sequentially arriving tasks, coupled with additional computational and memory constraints [33, 38, 54, 63, 49].

Building on the foundations of supervised deep learning, the prevalent approach in continual learning has been to jointly train representations alongside classifiers. This approach simply follows from the assumption that learned representations are expected to outperform fixed representation functions such as kernel classifiers, as demonstrated in supervised deep learning [34, 23, 57]. However, this assumption is never validated in continual learning, with scenarios having limited updates where networks might not be trained until convergence, such as online continual learning (OCL).

In this paper, we study the efficacy of representations derived from continual learning algorithms. Surprisingly, our findings suggest that these representations might not be as beneficial as presumed. To test this, we introduce a simple baseline method named RanDumb, which combines a random representation function with a straightforward linear classifier, illustrated in detail in Figure 1. Our empirical evaluations, summarized in Table 1 (left, top), reveal that despite replacing the representation learning with a pre-defined random representation, RanDumb surpasses current stateof-the-art methods in latest online continual learning benchmarks [76].

We further expand our evaluations to scenarios incorporating methods that use pre-trained feature extractors [68]. By substituting our random projections with these feature extractors and retaining the linear classifier, RanDumb again outperforms leading methods as shown in Table 1 (right, top).

1.1 Technical Summary: Construction of RanDumb and Empirical Findings

Design. RanDumb first projects input pixels into a high-dimensional space using a fixed kernel based on random Fourier basis, which is a low-rank data-independent approximation of the RBF Kernel [52]. Then, we use a simple linear classifier which first normalizes distances across different feature dimensions (anisotropy) with Mahalanobis distance [43] and then uses nearest class means for classification [44]. In scenarios with pretrained feature extractors, we use the fixed pretrained model as embedder and learn a linear classifier as described above, similar to Hayes and Kanan [27].

Key Properties. RanDumb needs no storage of exemplars and requires only one pass over the data in a one-sample-per-timestep fashion. Furthermore, it only requires online estimation of the sample covariance matrix and nearest class mean.

Key Finding 1: Poor Representation Learning. We compare RanDumb with leading methods: VAEGC [64] in Table 1 (left, middle) and SLCA [79] in Table 1 (right, middle). The primary distinction between them is their representation: RanDumb uses a fixed function (random/pretrained network),

Table 1: (Left) Online Continual Learning. Performance comparison of RanDumb on the PEC setup [76] and VAE-GC [64]. Setup and numbers borrowed from PEC [76]. RanDumb outperforms the best OCL method. (Right) Offline Continual Learning. Performance comparison with ImageNet21K ViT-B16 model using 2 initial classes and 1 new class per task. RanPAC-imp is an improved version of the RanPAC code which mitigates the instability issues in RanPAC. RanDumb nearly matches performance of joint for both online and offline, demonstrating the inefficacy of current benchmarks.

Embed

(Horizontal midway)

Input

Output

Decorrelate

whereas VAE-GC and SLCA further continually trained deep networks. RanDumb consistently surpasses VAE-GC and SLCA by wide margins of 5-15%. This shows that state-of-the-art online continual learning algorithms fail to learn effective representations across standard exemplar-free continual learning benchmarks.

Finding 2: Over-Constrained Benchmarks. Given the demonstrated limitations of existing continual representation learning methods, an important question arises: Can better methods learn more effective representations? To explore this, we evaluated the performance of RanDumb against joint training, models trained without continual learning constraints, in both online and offline settings, as shown in Table 1 (left, bottom) and Table 1 (right, bottom). Our straightforward baseline, RanDumb, bridges 70-90% of the performance gap relative to the respective joint classifiers in both scenarios. This significant recovery of performance by such a simple method suggests that if our goal is to advance the study of representation learning, current benchmarks may be overly restrictive and not conducive to truly effective representation learning.

We highlight that the goal in our work is not to introduce a state-of-the-art continual learning method, but challenge prevailing assumptions and open a discussion on the efficacy of representation learning in continual learning algorithms, especially in online and low-exemplar scenarios.

2 RanDumb: Mechanism & Intuitions

RanDumb has two main elements: random projection and the dumb learner. We illustrate the mechanism of RanDumb using three toy examples in Figure 2. To classify a test sample x test , we start with a simple classifier, the nearest class mean (NCM). It predicts the class among C classes by highest value of the similarity function f among class means µ i :

and µ i are the class-means in the pixel space: µ i = 1 | C i | ∑ x ∈ C i x . RanDumb adds two additional components to this classifier: 1) Kernelization and 2) Decorrelation.

Kernelization: Classes are typically not linearly separable in the pixel space, unlike in the feature space of deep models. Hence, we randomly project the pixels into a high-dimensional representation space, computing all distances between the data and class-means in this embedding space. This phenomena is illustrated on three toy examples to build intuitions in Figure 2 (Embed). We use an RBF-Kernel, which for two points x and y is defined as: K RBF ( x , y ) = exp ( -γ ∥ x -y ∥ 2 ) where γ is a scaling parameter. However, calculating the RBF kernel is not possible due to the online continual learning constraints preventing computation of pairwise-distance between all points. We use a data-independent approximation, random Fourier projection ϕ ( x ) , as given in [52]:

where the random Fourier features ϕ ( x ) are defined by first sampling D vectors { ω 1 , . . . , ω D } from a Gaussian distribution with mean zero and covariance matrix 2 γ I , where I is the identity matrix. Then ϕ ( x ) is a 2 D -dimensional feature, defined as:

We keep these ω bases fixed throughout online learning. Thus, we obtain our modified similarity function from Equation 1 as:

where ¯ µ i are the class-means in the kernel space:

Decorrelation: Projected raw pixels have feature dimensions with different variances (anisotropic). Hence, instead of naively computing ϕ ( x test ) ⊤ ¯ µ i , we further decorrelate the feature dimensions using a Mahalonobis distance with the shrinked covariance matrix S using OAS shrinkage [15], inverse obtained by least squares minimization ( S + λ I ). We illustrate this phenomena as well on three toy examples in Figure 2 (Decorrelate) to build intuitions. Our similarity function finally is:

Online Computation. Our random projection is fixed before seeing any data. During continual learning, we only perform online update on the running class mean and empirical covariance matrix 2 .

3 Experiments

Wecompare RanDumb with algorithms across online continual learning benchmarks with an emphasis on exemplar-free and low-exemplar storage regime. All numbers in tables with the caption (Ref: table and citation) except our method are taken from the aforementioned table in the cited paper.

Benchmarks. The benchmarks which we used in our experiments are summized in Table on the right. We aim for a comprehensive coverage and show results on four standard online continual learning benchmarks (A, B, D, E) which reflect the latest trends ('22'24) across exemplar-free, contrastivetraining 3 , meta-continual learning, and network-expansion based approaches re-

spectively. We also evaluate on a rehearsal-free offline continual learning benchmark C. These benchmarks are ordered by increasingly relaxed constraints, moving further away from the training scenario of RanDumb.Benchmark A closely matches RanDumb with one class per timestep and no stored exemplars. Benchmark B, D, E progressively relax the constraints on exemplars and classes per timestep. Benchmark C and E remove the online constraint by allowing unrestricted training and sample access within a task without exemplar-storage of past tasks. Benchmark F allows using large pretrained models, modified by us with one class per task, i.e. testing learning over longer timespans.

We further test on exemplar-free scenarios in offline continual learning using Benchmark F [68] with the challenging one-class per task constraint borrowed from [76]. This benchmark allows using pretrained models along with unrestricted training time and access to all class samples at each timestep. However, RanDumb is restricted to learning from a single pass seeing only one sample at a time. RanDumb only learns a linear classifier over a given pretrained model in Benchmark F.

We use LAMDA-PILOT [61] codebase for all methods, except RanPAC and SLDA for which use their codebases. We use the original hyperparameters. We only change initial classes to 2 and number of classes per task to 1 and test using both ImageNet21K and ImageNet1K ViT-B/16 models.

Implementation Details (RanDumb). We evaluate RanDumb using five datasets: MNIST, CIFAR10, CIFAR100, TinyImageNet200, and miniImageNet100. For the latter two, we downscale all images to 32x32. We augment each datapoint with flipped version, hence two images are seen by the classifier at each timestep (except for MNIST and Benchmark F). We normalize all images and flatten them into vectors, obtaining 784-dim input vectors for MNIST and 3072-dim input vectors for all the other. For Benchmark F, we compare RanDumb on seven datasets used in LAMDA-PILOT, replacing ObjectNet

with Stanford Cars as ObjectNet license prohibits training models. We use the 768-dimensional features from the same pretrained ViT-B models used in this benchmark. We measure accuracy on the test set of all past seen classes after completing the full one-pass. We take the average accuracy after the last task on all past tasks [76, 25, 68]. In Benchmark A and F, since we have one class per task, the average accuracy across past tasks is the same regardless of the task ordering. In Benchmarks A-E, all datasets have the same number of samples, hence similarly the average accuracy across past tasks is the same regardless of the task ordering. We used the Scikit-Learn implementation of Random Fourier Features [52] with 25K embedding size, γ = 1 . 0 . We use progressively increasing ridge regression parameter ( λ ) with dataset complexity, λ = 10 -6 for MNIST, λ = 10 -5 for CIFAR10/100 and λ = 10 -4 for TinyImageNet200/miniImageNet100.

3.1 Results

Benchmark A (single-class per task). We assess continual learning models in the challenging setup of one class per timestep, closely mirroring our training assumptions, and present our results in Table 2. Comparing across rows, and see that RanDumb improves over prior state-of-the-art across all datasets with 2-6% margins. The only exception is PEC on CIFAR10, where RanDumb underperforms by 3.3%. Nonetheless, it outperforms the second-best model, GDumb with a 500 memory size, by 4.9%.

Benchmark B.1 (many-classes per task). We present our results comparing with non-contrastive methods in Table 3. We notice that scenario allows two classes per task and relaxes the memory constraints for online continual learning methods, allowing for higher accuracies compared to Benchmark A. Despite that, RanDumb outperforms latest OCL algorithms on MNIST, CIFAR10 and CIFAR100-often by margins exceeding 10%. The lone exception is GDumb achieving a higher performance with 2K memory samples on TinyImageNet, indicating that this already is in the high-memory regime.

Benchmark B.2 (many-classes per task, with contrastive losses and data augmentations). We additionally compare our performance with the latest OCL approaches using contrastive losses with sophisticated data augmentations. As shown in in Table 4 (Left), these advancements provide large performance improvements over methods from Benchmark B.1. To compensate, we compare on lower exemplar budgets. The best approach, OnPro [69], outperforms RanDumb on CIFAR10 by 2.2% and TinyImageNet by 0.3%, but falls significantly short on CIFAR100 by 5.9%. Overall, RanDumb achieves strong results compared to representation learning using state-of-the-art contrastive learning approaches customized to continual learning, despite storing no exemplars.

Benchmark C (rehearsal-free). We compare against offline rehearsal-free continual learning approaches in Table 4 (Right) on CIFAR100. Despite online training, RanDumb outperforms PredKD by over 4% margins.

Benchmark D (meta-continual learning). We compare performance of RanDumb against metacontinual learning methods, which require large exemplars with buffer sizes of 1K in Table 5 (left). RanDumb achieves strong performance under these conditions, exceeding all prior work by a large margin of 9.1% on CIFAR100 and outperforms all but VR-MCL approach on the TinyImageNet dataset. GDumb performs the best on CIFAR10, indicating this is already in a large-exemplar regime uniquely unsuited for RanDumb.

Table 5: (Left) Benchmark D (Ref: Table 2 from VR-MCL [71]) We compare RanDumb with metacontinual learning approaches operating in a high memory setting, allowing buffer sizes up to 1K exemplars. RanDumb outperforms all methods except VR-MCL on TinyImageNet. RanDumb also surpasses all prior work by a substantial 9.1% on CIFAR100. Allowing generous replay buffers shifts scenarios to a high exemplar regime where GDumb performs the best on CIFAR10. Yet RanDumb competes favorably even under these conditions. (Right) Benchmark E (Ref: Table 1 from SEDEM [75]) We compare RanDumb with network expansion based approaches. Despite allowing access to much larger memory buffers, RanDumb matches the performance of best method SEDEM on MNIST, while exceeding it by 0.3% on CIFAR10 and 3.8% on CIFAR100.

Method

CIFAR10 CIFAR100 TinyImageNet

(5/2)

(10/10)

(20/10)

M

= 1

k

M

= 1

k

M

= 1

k

Finetune

17.0

±

0.6

5.3

±

0.3

3.9

±

0.2

LWF [37]

18.8

±

0.1

5.6

±

0.4

4.0

±

0.3

A-GEM [12]

18.4

±

0.2

6.0

±

0.2

4.0

±

0.2

IS [77]

17.4

±

0.2

5.2

±

0.2

3.3

±

0.3

MER [55]

36.9

±

2.4

-

-

La-MAML [26]

33.4

±

1.2 11.8

±

0.6

6.74

±

0.4

GDumb [49]

61.2

±

1.0

18.1

±

0.3

4.6

±

0.3

ER [13]

43.8

±

4.8 16.1

±

0.9

11.1

±

0.4

DER [9]

29.9

±

2.9

6.1

±

0.1

4.1

±

0.1

DER++ [9]

52.3

±

1.9 11.8

±

0.7

8.3

±

0.3

CLSER [5]

52.8

±

1.7 17.9

±

0.7

11.1

±

0.2

OCM[25]

53.4

±

1.0 14.4

±

0.8

4.5

±

0.5

ER-OBC [18]

54.8

±

2.2 17.2

±

0.9

11.5

±

0.2

VR-MCL [71]

56.5

±

1.8 19.5

±

0.7

13.3

±

0.4

Rehearsal-Free

RanDumb (Ours) 55.6 (-5.6)

28.6 (+9.1)

11.6 (-1.7)

Method

MNIST

CIFAR10 CIFAR100

(5/2)

(5/2)

(20/5)

M

= 2

k

M

= 1

k

M

= 5

k

Finetune

19.8

±

0.1 18.5

±

0.3

3.5

±

0.1

MIR [3]

93.2

±

0.4 42.8

±

2.2

20.0

±

0.6

GEM [12]

93.2

±

0.4 24.1

±

2.5

11.1

±

2.4

iCARL [54]

83.9

±

0.2 37.3

±

2.7

10.8

±

0.4

G-MED [32]

82.2

±

2.9 47.5

±

3.2

19.6

±

1.5

GSS [4]

92.5

±

0.9 38.5

±

1.4

13.1

±

0.9

CoPE [19]

93.9

±

0.2 48.9

±

1.3

21.6

±

0.7

CURL [53]

92.6

±

0.7

-

-

CNDPM [36]

95.4

±

0.2 48.8

±

0.3

22.5

±

1.3

Dynamic-OCM [74] 94.0

±

0.2 49.2

±

1.5

21.8

±

0.7

SEDEM [75]

98.3

±

0.2 55.3

±

1.3

24.8

±

1.2

Rehearsal-Free

RanDumb (Ours)

98.3

(0.0)

55.6 (+0.3) 28.6 (+3.8)

98.5

MNIST

CIFART0

CIFAR100

ImageNet

27

10.5

97.0

52

26

10.0

96.5

50

96.0

24

9.5

95.5

48

23

9.0

95.0

46

22

94.5

0

5000

10000

15000

20000

25000

0

5000

00001

15000

20000

25000

0

5000

10000

15000

20000

25000

8.5

0

5000

10000

15000

20000

25000

Benchmark E (network-expansion). We compare RanDumb against network expansion-based online continual learning methods in Table 5 (right). These approaches grow model capacity to mitigate forgetting while dealing with shifts in the data distribution, and are allowed larger memory buffers. RanDumb matches the performance of the state-of-the-art method SEDEM [75] on MNIST, while exceeding it by 0.3% on CIFAR10 and 3.8% on CIFAR100.

3.2 Analysis of RanDumb

Ablating Components of RanDumb. We ablate the contribution of only using Random Fourier features for embedding and decorrelation to the overall performance of RanDumb in Table 6 (left, top). Ablating the decorrelation and relying solely on random Fourier features, colloquially dubbed Kernel-NCM, has performance drops ranging from 6-25% across the datasets. Replacing random Fourier features with raw features, ie. the SLDA baseline, leads to pronounced drop in performance ranging from 3-14% across the datasets. Moreover, ablating both components results in the base nearest class mean classifier, and exhibits the poorest performance with an average reduction of 17%. Therefore, both decorrelation and random embedding are crucial for RanDumb.

Impact of Embedding Dimensions. We vary the dimensions of the random Fourier features ranging from compressing 3K input dimensions to 1K to projecting it to 25K dimensions and evaluate its impact on performance in Figure 3. Surprisingly, the random projection to a 3x compressed 1K dimensional space allows for significant performance improvement over not using embedding, given in Table 6 (left, top). Furthermore, increasing the dimension from 1K to 25K results in improvements of 3.6%, 10.4%, 7.0%, and 2.5% on MNIST, CIFAR10, CIFAR100, and TinyImageNet respectively. Increasing the embedding sizes beyond 15K, however, only results in modest improvements of 0.1%, 1.4%, 1.1% and 0.2% on the same datasets, indicating 15K dimensions would be a good point for a performance-computational cost tradeoff.

Impact of Flip Augmentation. We evaluate the impact of adding the flip augmentation on the performance of RanDumb in Table 6 (left, middle). Note that MNIST was not augmented. Augmentation provided large gains of 3.1% on CIFAR10, 1.7% on CIFAR100, and 0.4% on TinyImageNet. We did not augment the data further with RandomCrop transform as done with standard augmentations.

Impact of Varying Ridge Parameter. All prior experiments use a ridge parameter ( λ ) that increases with dataset complexity: λ = 10 -6 for MNIST, 10 -5 for CIFAR10 and CIFAR100, and 10 -4 for TinyImageNet and miniImageNet. Table 6 (left, middle) shows the effect of varying λ on RanDumb's performance. With a smaller λ = 10 -6 , CIFAR10, CIFAR100, TinyImageNet and miniImageNet all exhibit minor drops of 0.1%-1.7%, 0.8%, 0.8%. Increasing shrinkage to a λ = 10 -4 reduces CIFAR10 and CIFAR100 performance more substantially by 3% and 2.5% versus their optimal λ = 10 -5 . On the other hand, this larger λ leads to improvements of 0.5% and 1.8% on TinyImageNet and miniImageNet. This aligns with the trend that datasets with greater complexity benefit from more regularisation, with the optimal λ balancing under- and over-regularisation effects.

Comparison with Extreme Learning Machines. We compared our random Fourier features with random projections based extreme learning machines, as recently adapted to continual learning by RP+ReLU [41] in Table 6 (left, bottom) with their best embedding size. Our method performs significantly better on each dataset, averaging a gain of 3.4%.

Comparisons across Architectures. In table 6 (right), we compare whether using random Fourier features as embeddings outperforms models across various architectures for continual representation learning. We use experience replay (ER) baseline in the task-incremental CIFAR100 setup (for details, see Mirzadeh et al. [45] as it differs significantly from earlier setups). Our comparison spanned various architectures. The findings revealed that RanDumb surpassed the performance of nearly all considered architectures, and achieved close to 94% of the joint multi-task performance. This suggests that RanDumb outperforms continual representation learning across architectures.

Conclusion. Overall, both random embedding and decorrelation are critical components in the performance of RanDumb. Using random Fourier features is substantially better than RanPAC. Lastly, one can substantially reduce the embedding dimension without a large drop in performance for large gains in computational cost, additional augmentation may further significantly help performance and optimal shrinkage parameter increases with dataset complexity. RanDumb outperforms continual representation learning across a wide range of architectures.

7x7 conv,64,/2

pool, /2

★

3x3conv，64

3x3conv,64

3×3conv,64

3x3conv,64

3x3conv,64

→

3x3conv,64

3x3 conv,128,/2

3x3conv,128

3x3 conv,512,/2

↑

3x3 conv,512

3x3conv,512

3x3conv，512

3x3conv,512

3x3conv,512

—

avgpool

K

=

I

.....

7x7 conv,64,/2

pool, /2

3x3conv,64

3x3conv,64

3x3conv，64

3x3conv,64

3x3conv,64

3x3 conv,64

3x3conv,128,/2

3x3conv，128

3x3conv，512,/2

3x3 conv,512

3x3conv,512

3x3conv，512

3x3conv，512

3x3conv,512

avgpool

I

3.3 Should

we learn representations when strong pre-trained features are available?

Say for a specific application (e.g., where the test data distribution is more or less known during training), practitioners should use strong pretrained models as a starting point as they are likely to perform better. However, we still ask the key question of whether representation learning is necessary by fixing the pretrained backbone and only training the linear classifier, as illustrated in Figure 4 in the next benchmark.

Benchmark F. We compare performance of approaches which do not further train the deep network like RanDumb against popular continual finetuning and prompt-tuning approaches in Table 7. We discover that prompt-tuning approaches completely collapse under large timesteps and approaches which do not finetune their pretrained model achieve strong performance, even under challenging one class per timestep constraint. Note that RanPAC [41] adds a RP+ReLU and finetunes in a first-session adaptation fashion over RanDumb, yet fails to achieve higher accuracies.

Why is the performance of prompting methods so low? We argue that the true test of continual learning lies in the ability to learn over prolonged periods. Our observations indicate that prompting methods collapse early across tasks because the learned prompts do not generalize effectively. Supporting this, parallel work [62] suggests that most current prompt-tuning methods lack prompt di-

versity and can be characterized by a single prompt, making classification performance heavily reliant on the quality of that prompt. We hypothesize when prompts are designed for a large number of classes (e.g., 20 or 50), they produce generally discriminative representations that extend to future tasks. However, prompts designed for only two classes, as in our case, have limited discriminative power, leading to the collapse of prompt-tuning methods across tasks.

Overall, despite RanDumb being exemplar-free, it outperforms nearly all online continual learning methods across various tasks when exemplar storage is limited. We specifically benchmark on lower exemplar sizes to complement settings in which GDumb does not perform well.

4 Related Works

Random Representations. There have been extensive theoretical and empirical investigations into random representations in machine learning, compressed sensing, and other fields, often utilizing extreme learning machines [56, 14, 21] (see [30, 29] for a survey). Other investigations include efficient kernel methods using Fourier features and Nyström approximations [52, 70], and extensions to efficiently parameterize linear classifiers [2]. They are also embedded into deep networks [17, 35, 73, 16]. We tailored the already successful random fourier representations [52] to the problem at hand and applied to the online continual learning problem for the first time.

Continual Representation Learning. There are various works focusing on continual representation learning itself [53, 20, 39, 28], but they address the problem of alleviating the stability-plasticity dilemma in high-exemplar and offline continual learning scenarios where models are trained until convergence. In comparison, we focus on online and low-examplar regime.

Representation Learning Free Methods in CL. Several works have developed the idea of using fixed pretrained networks after adapting on the first task across various settings [50, 41, 24]. Our work contributes to this growing evidence, however, we do not perform first-task adaptation [47], and propose OAS-shrinked SLDA as structurally simplest but highly accurate continual linear classifier without any extra bells-and-whistles. Moreover, we are the first work to introduce a representation learning free method with random features for continually learning from scratch.

Equivalent formulations to RanDumb. If the classes are equiprobable, which is the case for most datasets here, nearest class mean classifier with the Mahalanobis distance metric is equivalent to linear discriminant analysis (LDA) classifier [42]. Hence, one could say RanDumb is exactly equivalent to a Streaming LDA classifier with an approximate RBF Kernel. Alternatively, one could think of the decorrelation operation as explicitly decorrelating the features with ZCA whitening [7].

5 Discussion and Concluding Remarks

Our investigation reveals a surprising result - simply using random embedding (RanDumb) consistently outperforms learned representations from methods specifically designed for online continual training. Furthermore, using random/pretrained features also recovers 70-90% of the gap to joint learning, leaving limited room for improvement in representation learning techniques on standard benchmarks. Overall, our investigation questions our understanding of how to effectively design and train models that require efficient continual representation learning, and necessitates a re-investigation of the widely explored problem formulation itself. We believe adoption of computationally bounded scenarios without memory constraints and corresponding benchmarks [51, 50, 22] could be a promising way forward.

Limitations & Future Directions. We currently do not provide theory or justification for why training dynamics of continual learning algorithms fails to effectively learn good representations; doing so would provide deeper insights into continual learning algorithms. Moreover, our proposed method, RanDumb with random Fourier features is limited in scope towards low-exemplar scenarios and online-continual learning. Extending studies on representation learning to high-exemplar and offline continual learning scenarios might be exciting directions to investigate.

Social Impact. RanDumb is an algorithm solely designed to perform a scientific study and we do not recommend use of RanDumb for any application in real-world production systems, hence no direct societal impact or explicit limitations on use in production systems is discussed.

References

Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In ICCV , 2021.

Nir Ailon and Bernard Chazelle. The fast johnson-lindenstrauss transform and approximate nearest neighbors. SIAM Journal on computing , 2009.

Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NeurIPS , 2019.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS , 2019.

Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In ICLR , 2022.

Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR , 2021.

Anthony Bell and Terrence J Sejnowski. Edges are the'independent components' of natural scenes. In NeurIPS , 1996.

Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Classincremental continual learning into the extended der-verse. TPAMI , 2022.

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS , 2020.

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In ICLR , 2022.

Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV , 2021.

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR , 2019.

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. Continual learning with tiny episodic memories. In ICML-W , 2019.

CL Philip Chen. A rapid supervised learning neural network for function interpolation and approximation. IEEE Transactions on Neural Networks , 1996.

Yilun Chen, Ami Wiesel, Yonina C Eldar, and Alfred O Hero. Shrinkage algorithms for mmse covariance estimation. IEEE transactions on signal processing , 2010.

Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV , 2015.

Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. NeurIPS , 2009.

Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. In ICLR , 2023.

Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from nonstationary data streams. In ICCV , 2021.

Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Dmitriy Fradkin and David Madigan. Experiments with random projections for machine learning. In KDD , 2003.

Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. ArXiv , 2023.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.

Dipam Goswami, Yuyang Liu, Bartłomiej Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS , 2023.

Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In ICML , 2022.

Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. NeurIPS , 2020.

Tyler L Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In CVPR-W , 2020.

Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting. arXiv preprint arXiv:2304.00933 , 2023.

Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing , 2006.

Guang-Bin Huang, Dian Hui Wang, and Yuan Lan. Extreme learning machines: a survey. International journal of machine learning and cybernetics , 2011.

Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. In NeurIPS-W , 2022.

Xisen Jin, Junyi Du, and Xiang Ren. Gradient based memory editing for task-free continual learning. In NeurIPS , 2021.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS , 2017.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 2017.

Quoc Le, Tamás Sarlós, Alex Smola, et al. Fastfood-approximating kernel expansions in loglinear time. In ICML , 2013.

Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In ICLR , 2020.

Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI , 2017.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS , 2017.

Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. arXiv preprint arXiv:2110.06976 , 2021.

Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In CVPR , 2021.

Mark D McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. In NeurIPS , 2023.

Geoffrey J McLachlan. Discriminant analysis and statistical pattern recognition . John Wiley & Sons, 2005.

Goeffrey J McLachlan. Mahalanobis distance. Resonance , 4(6):20-26, 1999.

Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near-zero cost. TPAMI , 2013.

Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXiv preprint arXiv:2202.00275 , 2022.

Sudhanshu Mittal, Silvio Galesso, and Thomas Brox. Essentials for class incremental learning. In CVPR , 2021.

Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. arXiv preprint arXiv:2303.13199 , 2023.

Karl Ezra Pilario, Mahmood Shafiee, Yi Cao, Liyun Lao, and Shuang-Hua Yang. A review of kernel methods for feature extraction in nonlinear process monitoring. Processes , 2020. doi: 10.3390/pr8010024.

Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV , 2020.

Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253 , 2023.

Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR , 2023.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. NeurIPS , 2007.

Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. NeurIPS , 2019.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR , 2017.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR , 2019.

Wouter F Schmidt, Martin A Kraaijveld, Robert PW Duin, et al. Feed forward neural networks with random weights. In ICPR , 1992.

Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 , 2013.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In AAAI , 2021.

James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attentionbased prompting for rehearsal-free continual learning. In CVPR , 2023.

James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning. In CVPR-W , 2023.

Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117 , 2023.

Lukas Thede, Karsten Roth, Olivier J Hénaff, Matthias Bethge, and Zeynep Akata. Reflecting on the state of rehearsal-free continual learning with pretrained models. arXiv preprint arXiv:2406.09384 , 2024.

Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. In NeurIPS-W , 2018.

Gido M Van De Ven, Zhe Li, and Andreas S Tolias. Class-incremental learning with generative classifiers. In CVPR-W , 2021.

Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H Lampert, et al. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311.11908 , 2023.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In ECCV , 2022.

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision (ECCV) , 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In ICCV , 2023.

Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. NeurIPS , 2000.

Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In ICLR , 2024. URL https://openreview.net/forum?id=TpD2aG1h0D .

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR , 2019.

Zichao Yang, Marcin Moczulski, Misha Denil, Nando De Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In ICCV , 2015.

Fei Ye and Adrian G Bors. Continual variational autoencoder learning via online cooperative memorization. In ECCV , 2022.

Fei Ye and Adrian G Bors. Self-evolved dynamic expansion model for task-free continual learning. In ICCV , 2023.

Michał Zaj ˛ ac, Tinne Tuytelaars, and Gido M van de Ven. Prediction error-based classification for classincremental learning. ICLR , 2024.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML , 2017.

Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123 , 2018.

Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In ICCV , 2023.

Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. arXiv preprint arXiv:2205.13218 , 2022.

Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338 , 2023.

Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. NeurIPS , 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper along with important assumptions.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitation Section is provided in the supplementary material.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Rahimi and Recht [52] from our references details the theory for why random fourier representations perform so well quite beautifully. The random representations do not change (no continual aspect), hence the theory can be applied as-is in our case with no changes. We do not claim any novel theoretical contributions.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: RanDumb is fairly simple to implement. We dedicated half a page towards explaining hyperparameters and other information needed to reproduce all of our results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code in the supplementary material to reproduce our results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We use standard datasets and splits, we provide hyperparameters in experimental details along with ablations in experiment sections to understand the contribution of each component in our algorithm.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We state here that the only random component being the random fourier features kernel across, otherwise our method is simple and exactly reproducible. We conducted experiments with three different initialisations corresponding to seeds of the random kernel in sklearn to investigate this and different kernel initialisations lead to around ± 0.2 variation in the reported results.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed in Section 3 in Implementation Details. For easy access, we restate: All experiments were conducted on a CPU server with a153 48-core Intel Xeon Platinum 8268 CPU and 392GB of RAM, requiring less than 30 minutes per experiment.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the ethics guidelines and confirm that we do not use human subjects, use existing datasets, explicitly discuss social impacts.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The primary contribution in RanDumb was not to introduce a novel stateof-the-art continual learning method, but challenge prevailing assumptions and open a discussion on the efficacy of representation learning in continual learning algorithms. As such, we do not recommend use of RanDumb for deployment in real-world production systems, hence no direct societal impact or explicit limitations on use in production systems is discussed.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have any high-risk model or dataset introduced.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: MNIST, CIFAR-10, CIFAR-100, tinyImageNet and miniImagenet are cited appropriately. The licenses for these datasets is not explicitly released, hence we do not include that information.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Yes, we provide RanDumb with proper documentation under a GPL3 license.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects was performed.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects was performed.

A Conceptual and Methodological Differences from GDumb [49]

Our core claim is that random representations from raw image pixels consistently outperform deeplearning-based representations designed for online continual learning. In contrast, GDumb's central claim is that continual learning methods need not actually minimize forgetting of previous online samples as their performance can be entirely recovered simply by a baseline using the latest memory.

The key differences between thee two works is as follows:

Forgetting of Online Samples (GDumb) : GDumb argues that continual learning methods suffer from forgetting online samples and addresses this by relying entirely on memory. This dismisses the value of directly learning from online data, as GDumb does not retain or utilize information from online samples. RanDumb, in contrast, exclusively learns from online samples without using memory, emphasizing the significance of ongoing data streams for performance.

Inadequate Representation Learning (RanDumb) : GDumb's approach to representation learning mirrors the experience replay (ER) baseline but does not address the quality of the learned representations. RanDumb explicitly focuses on the inadequacy of these representations and ablates their role to highlight their impact on continual learning. This reveals a key distinction in how each method evaluates representation quality in online learning settings.

Furthermore, the experimental setups for RanDumb and GDumb highlight their complementary nature:

Memory Settings : RanDumb primarily targets low-memory environments, whereas GDumb excels in high-memory scenarios. For example, in rehearsal-free settings without exemplar storage, a common trend in continual learning, GDumb would inherently produce random performance due to its dependence on memory. RanDumb, on the other hand, thrives in these low-memory contexts, providing an alternative solution when memory is constrained.

Complementary Nature : RanDumb and GDumb occupy complementary spaces within the continual learning landscape. RanDumb performs well in benchmarks where GDumb falters, and vice versa. As GDumb has been acknowledged as a valuable baseline, we argue that RanDumb similarly deserves recognition in the continual learning literature for its distinct strengths.

In summary, the only commonality between RanDumb and GDumb is that they serve as simple baselines. The points above underscore the fundamental distinctions between the two methods and the specific aims behind their development, as emphasized here and in the title of this work.

B Online Continual Learning: Our Setting

Current Problem formulation. We formally define the online continual learning (OCL) problem as follows. In classification settings, we aim to continually learn a function f : X → Y , parameterized by θ t at time t . OCL is an iterative process where each step consists of a learner receiving information and updating its model. For RanDumb, at each step t of the interaction,

One data point ( x t , y t ) ∼ π t sampled from a non-stationary distribution π t is revealed.

Learner updates the model θ t +1 using a compute budget, B learn t and discards the datapoint.

Simplifications by Compared Approaches. Traditional online continual learning literature makes several concessions over this which makes the problem easier by allowing the datapoint to be saved for more timesteps. Training deep networks requires those simplifications as more data per batch helps stabilize the gradient updates. Typically, compared approaches store samples across for 10 timesteps, and performs an update with that batch of samples before discarding it. Most works further relax this by storing a memory buffer of samples indefinitely.

Drawbacks. Traditional online continual learning setups cannot effectively test for rapid adaptation because they use a class-incremental setup. Online learning is generally intended to enable quick adaptation to changing data and label distributions in a data stream. We believe a better formulation for online continual learning is described in [50].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that a fixed random feature space of the data, followed by some linear classifier, outperforms continually learned feature spaces across multiple benchmarks. The paper then shows an ablation study of the suggested method.\n\nSTRENGTHS:\n**Clarity:** The suggested idea is simple, and it is easy to understand how it works and why.\n\n**Quality:** The random features are tested across multiple benchmarks, showing improvements in many of them. The ablation study is comprehensive. Overall, I find the results very convincing: I believe that RanDumb can outperform many continual learning algorithms, despite it being learned using a single pass over the data in a non-deep learning way.\n\n**Significance:** This kind of meta-study, showing weak points of the entire field rather than weak points of a specific work is important for the community. The paper shows that continual learning methods are not the optimal solutions for many continual learning benchmarks.\n\nWEAKNESSES:\nI'll review this paper from 2 perspectives: the first is from a \"technical\" point of view, judging the actual suggested method RanDumb. The second is the \"meta\" point of view, judging the \"meta\" claim of the paper, that we should re-think CL methods, as they are not optimal for many of the existing CL benchmarks.\n\n**Originality:**\nFrom the technical point of view, the suggested method is simply an implementation of known methods on several different benchmarks, and as such has very limited originality. From the meta point of view, I think that the additional benefits of this paper over g-dumb [1] are somehow limited, as it has been shown before that CL methods do not optimally solve the CL benchmarks in many cases.\n\n**Quality and significance:**\nWhile the quality of the technical point is high, I'm not so convinced about the meta point. The discussion in the paper about the meta point is rather minimalistic and is left for the reader. A more thorough discussion will be useful, especially a more in-depth comparison with previous works that suggested similar things, such as g-dumb.\nAs I don't find the \"technical\" point of view of the paper novel significant enough to merit a top-tier publication in its own right, I think the paper should focus more on the meta point of view. Specifically, I find that there is a missing discussion on why the existence of simple methods that outperform CL methods on CL benchmarks should be so worrisome. The counter-argument is that the benchmarks themselves are more of \"toy problems\", and that achieving high performance on them is not the major goal of the works, but instead learning how to deal with problems that deep models encounter when facing changing data, including catastrophic forgetting and distribution shifts. \n\n**Clarity:**\nWhile the main idea is clear, the paper is very confusing to read. The different benchmarks are marked by letters, which makes it hard to follow which is which. Moreover, I often had to go to other papers to understand the experiments that were done in a better way, as this formulation was very confusing. This is a major problem in the current version, as a substantial part of the paper revolves around this formulation.\n\n\n\n\n**Summary:**\nI find the technical point of view good and convincing, but not significant and novel enough for to merit a publication in a top-tier conference. The meta point of view in my opinion is important and relevant for the community, but is not supported enough by the paper. Therefore, I am leaning toward rejection. \n\n\n\n\n\n\n\n\n\n\n------\n\n[1] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. \"Gdumb: A simple approach that questions our progress in continual learning.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.\n\nQUESTIONS:\nCan you please elaborate on the problems in CL that this paper introduces, and the difference between them and the points raised in g-dumb?\n\nLIMITATIONS:\nnot relevant", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RanDumb, a representation learning-free method for online continual learning (OCL). It uses data-independent random Fourier transform to project the data to a high-dimensional space (embed), scales the features to have unit variance (decorrelation), and finally performs classification with a nearest mean classifier (NCM) in the feature space. \n\nThe authors compares RanDumb with existing methods (when they are trained from scratch) on various OCL benchmarks. The authors additionally substitute the \"embed\" part of RanDumb with frozen pretrained representations and find that it still performs decently compared to existing methods that finetune the representations, on one of the benchmarks.\n\nSTRENGTHS:\n1. This paper is well-written and I did not find any major technical flaws.\n\n2. I find that random representations outperform continually trained representations (from scratch) under the data/memory-constraint continual learning regime interesting and novel.\n\n3. The benchmarks used are extensive, involving multiple datasets (MNIST, CIFAR, subsets of ImageNet, etc.), different levels of OCL constraints (e.g., number of classes per task), and different model architectures (ViT, ResNet, etc.).\n\nWEAKNESSES:\n1. I think the main novelty lies in the usage of random Fourier features because, without it, the proposed method is very similar to SLDA. Therefore, I'm not entirely convinced by the motivation. It makes sense that representation learning from scratch performs poorly in the low-data regime. In what cases do we not want to use a frozen pretrained representation directly? \n\n\n2. The method involves using a very large embedding space (25K in most experiments), which raises concerns on efficiency. Kernel trick is efficient because you can compute the dot product in a high-dimensional space without explicitly projecting the data to that space, but I'm not sure how the dot product $\\phi(x)^\\top \\bar{\\mu}_i$ can be computed efficiently here. Could the authors elaborate on that?\n\nQUESTIONS:\nI listed my main concerns in the weaknesses section above and I only have some minor questions or comments here.\n\n1. I wonder if it's worth providing the definition of OCL clearly so that readers not familiar with OCL can grasp the ideas more easily. Also, I recommend adding details on the method explicitly (even if it's just a few equations in the appendix) rather than only providing references. For example, I'd appreciate seeing how the inverse of the covariance matrix is computed online.\n\n2. What part of the results are the reported numbers in prior work? Is it all the numbers except for the proposed method? \n\n3. I thought Co$^2$L was a contrastive learning-based method. Why is it included in benchmark B.1 and not B.2?\n\n4. Could the authors provide some hypothesis on why fine-tuning/prompt-tuning based methods collapse in Table 6?\n\nLIMITATIONS:\nThe authors have addressed the limitations clearly in Sec. 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
To obtain powerful representation in an online continual learning setting, the authors propose a new learning method referred to as RanDumb, that embeds raw pixels using a fixed random transform, approximating an RBF kernel initialized before any data is seen.

The proposed model trains a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting.

Extensive experiments demonstrate its effectiveness and power with several ablations.

STRENGTHS:
(+) Extending the investigation to popular exemplar-free scenarios with pre-trained models, this work shows that training only a linear classifier on top of pre-trained representations surpasses most continual fine-tuning and prompt-tuning strategies.

(+) The author's investigation challenges the prevailing assumptions about effective representation learning in online continual learning.

(+) The authors explain how the random Fourier basis (a low-rank data-independent approximation of the RBF Kernel) affects the representation of online continual learning along with the accuracy of RanDumb (embedding dimensions).

WEAKNESSES:
(-)  The structure of RamDumb (Fig. 1) is too simple to follow the structure. Could the authors depict the architecture of input, RBF-Kernel, pre-trained model (ViT), Decorrelate(D), and NCM(C)? or include an additional figure to illustrate the RamDumb.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
The embedding (RBF-kernel) is unclear; the authors could better explain how the random Fourier projection affects continual learning input representation with simple math.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that a model with random Fourier features as a representation, followed by a normalisation then nearest class means for classification, outperforms most Continual Learning methods in online CL benchmarks. The representation is not learned as opposed to other online CL methods the model is compared against. Based on this result, the authors conclude that, in the online learning setting, continual learning algorithms fail to learn effective representations. The second main contribution is that this naive learning method bridges 70% to 90% of the performance of joint learning, leaving a small improvement gap to develop an effective continual representation learning method.\n\nSTRENGTHS:\n- Originality : \n   - Even though the observation is simple, it highlights a blind spot in the field. Also, the naive model contrasts with most methods in the field and raises interesting questions.\n- Quality : \n   - The authors present extensive benchmarks and each benchmark is clearly motivated wrt the literature and the scope of the paper.\n   - An ablation study on the components of the naive model is presented, highlighting that both the normalisation and Fourier features are critical for the performance.\n- Clarity : \n   - The motivation, method and results are very clear and easy to follow.\n   - The paper is well structured as well, it's relatively easy to browse the results and sections.\n- Significance : \n   - This study will likely motivate subsequent works to investigate online representation learning\n\nWEAKNESSES:\nI don't see any major weaknesses, except the question I raise in the section below.\n\nQUESTIONS:\nOne point I wanted to discuss was the following : One property of RanDumb is that training is not gradient based. This contrasts of all other training methods. Given that the other methods train the final layer with gradient based optimisation methods, it could be that the representations are still decent for previous tasks but that the forgetting occurs mostly in the final layer. I was thinking that, in order to validate that the other continual learning methods don't learn effective representations, one experiment could be : Run the full CL sequence on a given dataset with a given model, training the full model. then freeze the learned representation and check the joint accuracy with the frozen representation vs the joint accuracy with a fully trainable model.\nI am curious about your thoughts on this point.\n\nLIMITATIONS:\nThere are no potential negative societal impacts of the work. The authors adequately highlighted various limitations of their work. These limitations are out of scope of the current work and can be addressed in follow-up investigations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}