Schedule-Robust Online Continual Learning
Authors: Ruohan Wang
Marco Ciccone
Giulia Luise
Andrew Yapp
Massimiliano Pontil
Carlo Ciliberto
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual Learning
Online Class-incremental Learning
Meta-Learning

Abstract: A continual learning (CL) algorithm learns from a non-stationary data stream. The non-stationarity is modeled by some schedule that determines how data is presented over time. Most current methods make strong assumptions on the schedule and have unpredictable performance when such requirements are not met. A key challenge in CL is thus to design methods robust against arbitrary schedules over the same underlying data, since in real-world scenarios schedules are often unknown and dynamic. In this work, we introduce the notion of schedule-robustness for CL and a novel approach satisfying this desirable property in the challenging online class-incremental setting. We also present a new perspective on CL, as the process of learning a schedule-robust predictor, followed by adapting the predictor using only replay data. Empirically, we demonstrate that our approach outperforms existing methods on CL benchmarks for image classification by a large margin.

Under review as a conference paper at ICLR 2023
SCHEDULE-ROBUL ONLINE CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A continual learning (CL) algorithm learns from a non-stationary data stream. The non-stationarity is modeled by some schedule that determines how data is presented over time. Most current methods make strong assumptions on the schedule and have unpredictable performance when such requirements are met. A key challenge in CL is thus to design methods robust against arbitrary schedules over the same underlying data, since in real-world scenarios schedules are often unknown and for CL
dynamic. In this work, we introduce the notion of and novel approach satisfying this desirable property the challenging online class-incremental setting. We also present a new perspective on CL., as the process of learning a schedule-robust predictor, followed by adapting the predictor using only replay data. Empirically, we demonstrate that approach outperforms existing methods on CL benchmarks for image classification by a large margin.
INTRODUCTION
hallmark of natural intelligence is its ability continually absorb new knowledge while retaining and updating existing one. Achieving this objective in machines is the goal of continual learning (CL.. Ideally, CL algorithms learn online from never-ending and non-stationary stream of data, without catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990; French, 1999).
The non-stationarity of the data stream is modeled by some schedule that defines what data arrives and how its distribution evolves over time. Two family of schedules commonly investigated are task-based (De Lange et al., 2021) and task-free (Aljundi assumes that new data arrives one task at a time and data distribution is stationary for each task. Many CL algorithms (e.g., Buzzega et al., 2020; Kirkpatrick et al., 2017; Hou et al., 2019) thus train offline, with multiple passes and shuffles over task data. The frsk-free setting does not assume the existence of separate tasks but instead expects CL algorithms to learn online from streaming data, with evolving sample distribution (Caccia al., 2022; Shanahan et al., 2021). In this work, tackle the task-free setting with focus on class-incremental learning, where novel classes are observed incrementally and single predictor is trained discriminate all of them (Rebuffi et al., 2017).
al., 2019b). The task-based setting
Existing works are typically designed for specific schedules, since explicitly modeling and evaluating across all possible data schedules is intractable. Consequently, methods have often unpredictable performance when scheduling assumptions fail to hold (Farquhar & Gal, 2018; Mundt et al., 2022; Yoon et al., 2020). This is considerable issue for practical applications, where the actual schedule is either unknown or may differ from what these methods were designed for. This challenge calls for ideal notion of different schedules over the same underlying data.
CL methods should behave consistently when trained on
To achieve schedule-robusee robusteess,,,, we introduce a new strategy based on a two-stage approach: 1) learning online a schedule-robust predictor, followed by 2) adapting the predictor using only data from experience replay (ER) (Chaudhry al., 2019b). We will show that both stages are robust to diverse data schedules, making the whole algorithm schedule-robust. We refer to it as SChedule-Robust Online continuaL Learning (SCROLL). Specifically, we propose two online predictors that by design are robust against arbitrary data schedules and catastrophic forgetting. To learn appropriate priors for these predictors, we present a meta- learnigg perspective (Finn et al., 2017; Wang et al., 2021) and connect it to the pre-training strategies in CL (Mehta et al., 2021). We show that pre-training offers an alternative and efficient procedure for learning predictor priors instead of directly solving the meta-learning formulation. This makes our method computationally competitive and at the same

---

Under review as a conference paper at ICLR 2023
Dataset D
Pre-training Ju
Online Learning Je
Adapting fi 5"
Large buffer Madel Tuaning
Small butter Rosidal Mapters
Prefmsining Detsset
Data Streem
Schooukc keo
M, Reckay Butfer
My Peplay Buffer
Figure 1: Left. Illustration of classification dataset streamed according to different schedules (dashed vertical lines identify separate batches). Right. Pre-training + the two stages of SCROLL: 1) online learning and store replay samples from data stream, 2) adapting the predictor using the replay buffer (green indicates whether the representation to is being updated).
time offers a clear justification for adopting pre-training in CL. Finally, we present effective routines for adapting the predictors from the first stage. We show that using only ER data for this step is key to preserving
and discuss how to mitigate overfitting when ER data is limited.
We introduce the novel concept of schedule- robustness for CL, an important property that lacks in existing methods. 2) We propose novel online strategy that satisfies schedule- robustness, along with practical algorithms. 3) Theoretically, we connect CI. to standard meta-learning methods via schedule-robustness. This justifies the use of pre-trained models as a enowledge-prioe for CL. We further show that multi-class classification an efficient and principled procedure for learning such priors in CL. 4) Empirically, we show that SCROLL outperforms state-of-the- methods by large margin (over a 20% improvement in accuracy in many settings). This further supports the focus of this work on schedule-robustnes and our strategy to achieve it in practice.
Contributions. 1)
2 PRELIMINARIES AND RELATED WORKS
We formalize CL as learning from non-stationary data sequences. A data sequence consists of dataset D = {(,,, 1 | regulated by a schedule S = (o, 3). Applying the schedule S to D is denoted by SDD 2 B(a(D)). where (D) is specific ordering of D, and ((((D) = {B.)T= splits the sequence (D) into T batches of samples B, (Z)r()) Vort) Vort)) F with k the batch boundaries. Intuitively, determines the order in which (z, 3) E D are observed, while 3 determines how many samples are observed at a time. Fig. (Left) illustrates how the same dataset D could be streamed according different schedules. For example, S1 (D) in Fig. I (Left) depicts the standard schedule to split and stream D in batches of C' classes at the time (C = 2).
2.1 CONTINUAL LEARNING
CL algorithm learns from S(D) one batch B, at a time, iteratively training a predictor ft -> y to fit the observed data. Some formulations assume access fixed-size replay buffer M, which mitigates forgetting by storing and reusing samples for future training. Given an initial predictor fo and an initial buffer Mg. we define the update rule of a CL algorithm Alg(-) at step
(fe, M,) = Alg ft-1
(I)
where the algorithm learns from the current batch B, and updates both the replay buffer M,-1 and predictor fi-1 1 from the previous iteration.
At test time, the performance of the algorithm is evaluated on a distribution aD that samples (xx sharing the same labels with the samples in D. The generalization error denoted by
L(S(D), for Alg) B(z,y)~ {ff(x).
(2)
2

---

Under review as a conference paper at ICLR 2023
where the final predictor fr is recursively obtained from (1) and fo. ! In the following, we review existing approaches for updating fo and M,, and strategies for initializing fo-
Predictor Update (fr). Most CL methods learn fi by solving an optimization problem of the form:
Jt = arg min on current barch loss
Σ (((+)),)++ 02
Σ [((()),)+ 0z
(i-) regularization loss
(z.0)EB,
(1:))EM,1
roplay loss
where 1,,, are prescribed non-negative weights, l loss function, and R regularizer. This general formulation for updating ft recovers replay-based methods such as iCarl (Rebuffi et al., 2017) and DER (Buzzega et al., 2020) for specific choices of l and R. Moreover, if M,=0 't' (or set 02 =0), the replay loss is omitted and we recover regularization- based methods (e.g Kirkpatrick et al., 2017; Li & Hoiem, 2017; Yu al., 2020) that update selective parameters of f-1 to prevent forgetring.
Replay Buffer Update (M,). In replay-based methods, Alg(.) must also define a buffering strategy that decides what samples to store in M, for future reuse, and which ones to discard from a full buffer. Common strategies include exemplar selection (Rebuffi et al., 2017) and random reservoir sam- pling (Vitter, 1985), with more sophisticated methods like gradient-based sample selection (Aljundi et al., 2019c) and meta-ER (Riemer et al., 2019). As in (3), replay- based methods typically mix the current data with replay data for predictor update (e.g Aljundi et al., 2019c; Riemer et al., 2019; Caccia et al., 2022). In contrast, Prabhu et al. (2020) learns the predictor using only replay data. We will adopt a similar strategy and diseuss how it crucial for achieving chhedule-robustnes.
Predictor Initialization (fo). The initial predictor fo in (2) represents the prior knowledge available to CL algorithms, before learning from sequence S(D). Most methods are designed for randomly initialized fo with no prior knowledge (e.g.. Rebuffi et al., 2017; Gupta et al., 2020; Prabhu et al., 2020; Kirkpatrick et al., 2017). However, this assumption may be overly restrictive for several applications (e.g.. vision-based tasks like image classification), where available domain knowledge and data can endow CL algorithms with more informative priors than random initialization. We review two strategies for predictor initialization relevant to this work.
Initialization by Pre-training. One way for initializing fo is to pre-train a representation on data related to the CL task (e.g.. ImageNet for vision- tasks) via either sulf--uue learning (Shanahan et al., 2021) or multi-class classification (Mehta et al., 2021; Wang et al., 2022; Wu et al., 2022). Boschini et al. (2022) observed that while pre-training mitigates forgetting, model updates quickly drift the current Ji away from fo. diminishing the benefits of prior knowledge as CL algorithms continuously learn from more data. To mitigate this, Shanahan et al. (2021); Wang al. (2022) keep the pre-trained representation fixed while introducing additional parameters for learning the sequence. In contrast, we will offer effective routines for updating the pre-trained representation and significantly improve test performance.
Initialization by Meta- LeatnnLe Another approach for initializing fo is meta-learning (Hospedales et al., 2021). Given the CL generalization error in (2), we may learn fo by solving the meta-CL problem below,
fo = arg min B(D,S)~T ((S(D),
(4)
where T is a meta-distribution over datasets D and schedules S. For instance, Javed & White (2019) set Alg(-) be MAML (Finn et al., 2017) and observed that the learned fo encodes sparse representation to mitigate forgetting. However, directly optimizing (4) is computationally expensive since the cost of gradient computation scales with the size of D. To overcome this, we will leverage Wang et al. (2021) to show that meta-learnigg fo is analogous to pre-training for certain predictors, which provides a much more efficient procedure to learn fo without directly solving (4).
2.2 SCHEDULE- BHHEDUULE-INVARIANCE INVARIANCE AND ROBUSSTNES
The majority of CL methods depend significantly on the specific data schedule. This typically leads to unpredictable behaviors when considering different schedules (See e.g. Farquhar 2018; Yoon et al., 2020; Mundt al., 2022, or experiments in Sec. 4.1). To address this issue,
& Gal,
We omitted the initial memory buffer Mo since is typically empty.
3

---

Under review as a conference paper at ICLR 2023
first introduce the ideal notion of schedule-invariane for CL. We say that CL algorithm is schedule-ivaariant if
(D), fo, Alg) = L(S: Alg), VS1. S2 schedules, vD dataset.
(5)
Eq. (5) captures the idea that CL algorithms should perform identically for all data schedules. In practice, however, inherent randomness of learning algorithms often leads to similar but not identical performance. We thus further introduce schedule-robusteess as a relaxation of schedule-invarinne
((S, Alg) s (D), fo. Alg), 12 schedules, VD dataset,
(6)
where the Rs symbol loosely captures potentially small differences in performance". We argue that achieving schedule-robustnes: is key challenge in real-world scenarios, where data schedules are often unknown and possibly dynamic. CL algorithms should carefully satisfy (5) for safe deployment.
We note that schedule-robustness is a more general and stronger notion than order-robustness from Yoon et al. (2020). Our definition applies to online task-free CL while order-robustness only considers offline task- setting. We also allow arbitrary ordering for individual samples instead of task-level ordering. In the following, we will present our method and show how it achieves schedule-robustnes
METHOD
We present SChedule- (Chedule-Robust Online continuaL Learning (SCROLL) as a two-stage process: I) learning online schedule-robust predictor for CL. followed by 2) adapting the predictor using only replay data. In the first stage, we consider two schedule-robust predictors and discuss how to initialize them, motivated by the meta-CL perspective introduced in (4). In the second stage, tackle how to adapt the predictors from the first stage with ER and the buffering strategy. We will show that SCROLL satisfies schedule-robustnes by construction, given that optimizing a CL algorithm against all possible schedules as formulated in (5) is clearly intractable.
3.1 SCHEDULE- INNIULEE--IVAIANN ONLINE PREDICTOR
We model the predictors introduced in (1) as the composition / = d o vi of a feature extractor d X + R" and a classifier RT - V. In line with recent meta-learning strategies (e.g. Bertinetto et al., 2019; Raghu et al., 2020), we keep & fixed during our method's s first stage while only adapting the classifier @ to learn from data streams. We will discuss how learn & in Sec. 3.2.
key observation is that some choices of σ, such as Nearest Centroid Classifier (NCC) (Salakhutdinov & Hinton, 2007) and Ridge Regression (Kailath et al., 2000), are schedule-invane invariant by design.
Nearest Centroid Classifier (NCC), NCC classifies a sample x by comparing it to the learned "prototypes" Cy for each class X A {-](,, y) e D} in the dataset,
f(x) = arg min [b)()) cg.
where
Cy 6 n xEXv
Σ (Z))
The prototypes cy can be learned online: given new (r. v). we update only the corresponding as
Ty Cigh + ((r)
ny +1
where ny the number of observed samples for class y so far. We note that the prototype for each class is invariant to any ordering of D once the dataset has been fully observed. Since v is fixed, the resulting predictor in (7) mntedull-invarian and online. Further, f is unaffected by catastrophic forgetting by construction, as keeping & fixed prevents forgetting while learning each prototype independently mitigates cross-class interference (Hou et al., 2019).
Ridge Regression. Ridge Regression enjoys the same desirable properties of NCC, being schedule- invariant and unaffected by forgetting (see Appendix A for a full derivation). Let OneHot (y) denote
A formal definition is possible but outside the scope of the current work (see Appendix F. for more details).
4

---

Under review as a conference paper at ICLR 2023
the one-hot encoding of class y. obtain the predictor f as the one-vs-all classifier
(zx = arg max w/y (x))
where
y
(9)
W W1 kK arg min W
(x)) OneHot(y)]] >|I
(x.D))
Ridge Regression admits online updates via recursive least squares (Kailath et al., 2000); given (z, y).
W² g0w + A1) new
Vz E
where
[ dight + (x() Caul
=
(10)
A A = Aold (())(())
and
new
otherwise.
Here, denotes the covariance matrix of all the samples observed so far during schedule, while c₂ is the sum of the embeddings (xxx) of samples in class 2.
3.2 PREDICTOR INITIALIZATION
Within our formulation, initializing fo consists of learning a suitable feature representation dh As discussed in Sec. 2, we could learn So = 2 o 1b by solving the meta-CL problem in (4) (with do classifier for the meta weta-training data to be discarded after initialization, see Fig. 1 (Right)). As shown in Sec. 3.1, if we set Alg(-) ) in (4) be the online versions of NCC or Ridge Regression, the resulting predictor fr becomes invariant to any schedule over D. This implies that the meta- CL problem is equivalent to a standard meta-learning one where all data D are observed at once, namely
fo = arg min En-T ((
where
fn Al)
(11)
For example, setting Alg(-) to NCC recovers exactly the well-known ProtoNet (Snell et al., 2017).
Meta-learning and pre-training. Meta-learning methods for solving (11) are mostly designed for low data regimes with small while in contrast, CL. has to tackle long sequences (i.e. is large). In these settings, directly optimizing (11) quickly becomes prohibitive since the cost of computing gradients scales linearly with the size of D. Alternatively, recent works in meta-learning strongly support learning fo via standard multi-class classification (Tian et al., 2020; Wang al., 2021). In particular, Wang et al. (2021) showed that the cross-entropy loss in multi-class classification is an upper bound meta-learning fu in (11). Classification thus offers a theoretically sound estimator for meta-learning- As direct consequence, our meta-learning perspective on initializing So provides principled justification for exploiting pre-training CL. Pre-training is computationally efficient, since gradients can be computed small batches rather than the entire D in (11). Extensive evidence also suggests that pre- vee-trining often yiclds more robust fo than meta- learning (El Baz et al., 2022).
Learning fo in SCROLL. Given the reasoning above, we propose to not tackle (11) directly but rather learn fu by multi-class classification. Following the standard practices from meta-learning, we ensure the meta-training data to be disjoint from the sequences Alg(-) is indeed learning novel data during CL. For instance, in Sec. 4.1 we will use fo trained on Meta-Dataset (Triantafillou al., 2019) to perform CL CIFAR datasets, which do not overlap.
run the CL algorithm on, such that
3.3 PREDICTOR ADAPTATION WITH EXPERIENCE REPLAY
The predictor fr obtained via online learning is schedule-invariant and readily available for classifi- cation. However, shares the same feature extractor with fo and may therefore suffer from the distribution shift between the pre-training data and D. We thus propose to further adapt Jr into using Experience Replay (ER). Our key insight to consider buffering strategies that yield similar replay buffers, regardless of the schedule over D, hence keeping f* schedule-robust.
Buffering strategy. We adopt the exemplar buffering from Rebuffi et al. (2017). The method uses greedy moment matching to select the same number of samples for each class independently, according to how well the mean embedding (via v/) of the selected samples approximates the mean embedding of all points in the class. The greedy selection provides total ordering for all points in each class, which is used to determine which samples to store or discard. This step filters out outliers and ensures that Mr contains representative samples from each class.
5

---

Under review as a conference paper at ICLR 2023
This process aims to keep each M, balanced class wise, hence the final Mf attains similar class distribution after D is fully observed, irrespective of the specific schedule. In particular, it is easy to show that My is X is split across multiple batches), since the replay exemplars selected for each class is deterministic for such schedules. More generally, adapting fr with Mr yields schedule-robust f We will further discuss schedule-robustness with respect to smaller batch sizes in Appendix B.4. Below we consider two adaptation strategies.
if the data schedule present one entire class a time (namely no
Small buffer size: residual adapters. When M has limited capacity (e.g., 200 or 500 in our experiments), there may be insufficient data for adapting all parameters in fr. To avoid overfitting we introduce residual adapters inspired by task-specific adapters (Li et al., 2022). Concretely, we insert residual layers (e.g. residual convolutions) to each level of tb. Then, while keeping 1b fixed, we jointly optimize the new layers and the classifier using a cross-entropy loss (see Fig. 1 (Right)).
Large buffer size: model tuning. For larger buffers, we empirically observed that updating all parameters of fr is more effective. This is not surprising given the added representational power from the full model. We will also demonstrate that the threshold for switching from residual adapters to full-model tuning appears consistent and may be determined a priori.
3.4 PRACTICAL ALGORITHM
We present the complete algorithm Alg. 1 (see also Fig. 1): We first pre-train & via multi-class classification using the cross-entropy loss (Sec. 3.2) and initialize fo with the pre-trained vb. As we receive data online from the sequence S(D), we update the replay buffer M, (Sec. 3.3) and the nec- essary statistics for computing the classifier de (via e.g. NCC or Ridge). At each step we can output the intermediate model ft = di o v (Sec. 3.1). The final model f* is obtained by adapting fr using Mr via cross-entropy loss (Sec. 3.3).
Algorithm 1 SCROLL
Input: Pre-trained vi, CL sequence S(D),
Buffer Mo = 2, buffer max size m.
ST. My ONLINE-CL(() (L))) Mo)
If MT
f ResidualAdapt((rr, Mr)
Else:
I FullAdap(((r...k Mr)
return l"
def ONLINE- CLEE-CLLL (v, S(D),,
For B, in S(D):
We highlight several properties of SCROLL: 1) it is schedule-robus since we have shown in Sec. 3.1 that Jr is schedule-invariant, and Sec. 3.3 that adapting it to f is schedule-robust. Therefore our overall algorithm combining the two components is schedule- robuus.. 2) it is class-incremental with single classifier for all observed classes; 3) it meets the definition of online CL (Chaudhry et al., 2019a; Mai et al., 2022), where the data stream is observed once, and only the replay data is reused.
Update M, (exemplar strategy, See. 3.3)
Update statistics cy. A using (8) or (10).
Compute the classifier dee via (7) or (10).
return fr dr 1o and Mr
Memory-free SCROLL. We note that the fr returned by the Online -LL routine in Alg. 1, is already valid and deterministic predictor for D. Hence, if we have constraints (e.g., privacy concerns Shokri & Shmatikov (2015)) or memory limitations, SCROLL can be memory-fres by not using ER. We empirically investigate such strategy in our ablation studies Sec. 4.3 and Appendix B.2.
4 EXPERIMENTS
We consider class- caas-inncremetal learning and compare SCROLL primarily to online CL methods, including GDumb (Prabhu et al., 2020), MIR (Aljundi al., 2019a), ER-ACE and ER-AML (Caccia et al., 2022), and SSIL. (Ahn et al., 2021). As shown by (Buzzega et al., 2020), online methods often perform poorly due to the limited number of model updates. To disentangle forgetting from underfitting in the online setting, we further compare to recent offline methods for completeness, including ER (Riemer et al., 2019), BIC (Wu et al., 2019), DER and DER++ (Buzzega et al., 2020). We recall that the conventional offline setting allows multiple passes and shuffles over data for each task (De Lange et al., 2021). We differentiate our method by the classifier used, namely SCROLL (NCC) and SCROLL (Ridge).
Setup. We first consider sequential CIFAR-10 10 and sequential CIFAR- 100 benchmarks. We use a ResNet18 initial predictor trained Meta-Dataset using multi-class classification (Li et al., 2021).

---

Under review as a conference paper at ICLR 2023
CIEAR-10 5-split
CIFAR-100 10-spii
Joint Training (Li.d.)
91.22 1:1.6
79.3 10.4
CL Algorithm
|M] 200
M 500
[M] 2000
[M] 200
[M] 500
[M] 2000
OIFLINE
67.9
77.3 #0.4
=0.4
22.9 #0.6
33.2 +0.6
53.1 #0.3
BIC
77.4 +2.2
88.6 +5.33
90.4 +0.5
34.9 =1.8
45 +1.9
57.5 +007
DER
69.7 11.7
78.2 +0.8
86.6 =0.7
26.2 00.8
38.1 #2.1
55.3 #0.6
DER+
76.8 1.77
12.2
86.8 20.8
28.5 20.7
42.7
59.1 10.5
GDumb
73.6 11.9
81.2 +0.9
87.3
21.9 #1.3
59.8 +0.2
ONLINE
ER-ACE
77.8 20.9
#0.6
#0.8
40.3 1..
49.0 +0.0
56.5 #1.0
ER-AML
69.9 12.9
76.3 1:077
80.6 20.6
32.4 20.5
41.9 1:10
51.0 1:12
SSIL
68.9 1144
70.8 1:144
73.2 #1.7
46.6 #1.0
53.9 1..6
59.2 4:0.5
MIR
55.2 1.5
08.7 +1.3
82.0 +1.0
23.2 20.8
31.9 10.5
47.5 #1.1
SCROLL
(Ouss)
SCROLL (NCC) fr + f-
81.5
81.6 1:01
88.1
48.3 10.1
56.4 1:0.2
66.6 1:0.2
SCROLL (Ridge) fr
81.7
57.1 4:0
SCROLL (Ridge) fr f.
84.0
#0.2
86.3 +0.2
89.4
#0.2
59.9
61.5 #0.1
68.0 +0.2
Table 1:
Class-incremental classification accuracy sequential CIFAR-10/ 100. Joint training accuracy obtained by training on classes with standard supervised learning. Best online/offline methods in bold.
Following the convention of meta-learning, Meta-Dataset has no class overlap with CIFAR datasets. This ensures that CL algorithms must learn about the new classes during CL. We also evaluate SCROLL on miniIMAGENET as another sequential learning task to highlight that even a weakly initialized fo is very beneficial. Following Javed & White (2019), all methods are allowed learn an initial predictor using the meta- veta-training set of miEiIMAGENET (38400 samples over 64 classes), and perform CL on the meta-testing set (20 novel classes). A ResNet12 is used on miniIMAGENET.
Baseline methods. For faimess, all baseline methods use the same pre-trained feature extractor. We also performed extensive hyper-parameter tuning for all baseline methods such that they perform well with pre-training (see Appendix D). Lastly, we standardize that buffer size reported in all experiments as the total buffer for all classes, not buffer per class.
4.1 CLASS-INCREMENTAL LEARNING ON CIFAR DATASETS
For CIFAR datasets, we first follow a standard schedule from Buzzega al. (2020); Wu et al. (2019): CIFAR-10 is divided into 5 splits of 2 classes each while CIFAR- 100 into 10 splits of 10 classes each. At each split, this schedule abruptly changes from one data distribution to another.
Tab. shows that SCROLL (Ridge) outperforms all baselines by large margins on CIFAR- 100 for all buffer sizes. For CIFAR-10, SCROLL only lags behind BIC for buffer sizes 500 and 2000 while outperforming the rest. However, we emphasize that BIC is an offline method and not directly comparable ours. addition, we will demonstrate next that the baseline methods including BIC have unreliable performance when the schedule changes.
Varying Schedules. We now consider the effect of using schedules for CIFAR-100 with splits presenting 20/ 10/4/2 classes at a time (respectively 5/ 25/02///22 25/50-splits). Fig. 2, evaluates the best performing baselines from Tab. under these schedules. We observe that SCROLL performs consistently across all settings and outperforms all baselines by large margin, with BIC performing competitively only under the 5-split. Additionally, we clearly see that the accuracy of both online and offline methods drops drastically the dataset split increases. This further validates Yoon et al. (2020); Mundt et al. (2022) who observed that existing methods are brittle to schedule variations. The only exception is GDumb, which only uses replay data and is thus (mostly) robust against schedule changes, according to our discussion in Sec. 3.3. We note, however, that GDumb performs poorly in some "worst-case" regimes (see Appendix B.3). Additional results are reported in Appendices B. and B.2. including comparison with (Shanahan et al., 2021) who introduces an ensemble CL method to tackle schedules variations, and a challenging Gaussian schedule for empirical evaluation.
7

---

Under review as a conference paper at ICLR 2023
AM
200
M/ 500
M 2000
Accuracy (%)
40
SCROLL
BIC
G-Dumb
ER-ACE
DER+
ER-AMI.
10
25
50
25
Data schedule (# splits of D)
Figure 2: Robustness SCROLLI (Ridge) vs offlinc/online baselines different schedules of CIFAR-100.
4.2 CLASS-INCREMENTAL LEARNING ON minilMAGENET
minÍTMAGENE 10-split
In Tab. 2, qe evaluate SCROLL on minIMAGENET
Joint Training (i..d.)
89.2 00.4
10-split following the setup described earlier. We stress that all methods perform pre-training on the small meta-train vetaatraiiing set and then carry out CL on the meta-test set. We note that SCROLL (Ridge) outperforms all baseline methods with noticeable improvements. This suggests that our method does not require pre-training on massive datasets and could work well with limited data. Further, Ridge Regression classifier clearly outperforms NCC clas- sifier. We further investigate this phenomenon in the ablation studies.
CL Algorithm
[M] = 200
[M] 500
OPE INE
BIC
04.6
73.4 11.6
63.7
73.6 10.6
DER
61.
75.2 1:0.8
DER++
68.8
78.0 40.66
GDumb
65.6
71.5 1:0.5
ONLINE
ER-ACB
+0.6
ER-AML
52.1
58.4 1:17
SSIL
37.0
63.3 +1.6
MIR
48.4 #2.88
67.3 #2299
(ours)
SCROLL (NCC)
72.8 +00.2
76.0 +0.1
SCROLL (Ridge)
80.9 10.1
#0.1
Table Classification accuracy on sequential MinilMAGENET 10-split.
4.3 ABLATION STUDIES
Effects of classifier initialization. In our formulation of ER, the role of NCC or Ridge Regression is to initialize the classifier & at fr before adapting fr towards f" In principle, one could skip this online learning step (i.e. Online-CL routine in Alg. 1), initialize á in Jr randomly, and adapt fr. Tab. 3 compares these options for initializing the classifier".
The results suggest that how classifier á is initial- ized at fr drastically affect the performance of final predictor f*. In particular, the performance of fr cor- relates strongly with the performance of f": Ridge Regression has the best predictor both before and after adaptation. In contrast, randomly initialized ø does not learn from data streams, and the resulting l falls short of the other two initializers. Crucially, the results suggest that online learning described in Sec. 3.1 vital for fully exploiting the pre-training. We also remark that while randomly initialized ó resembles GDumb, there are still crucial differences between them, including buffering strategy, model architecture and training routine. In particular, we study in depth the role of the buffering strategy in Appendix B.3.
CIFAR-100 10-plit
minTMAGENET 10
Init
fr
f
fr
f"
Random
20.3 49.0
1..3
75.3
NCC Ridge
40.4 40 56.4
+0.2
65.3 40
76.0
57.1 61.5 #0.1
80.6
20.2
82.0
Table 3: Classification accuracy with different
classifier initialization for |M| 500.
Effects of replay buffer size. We study the impact of changing replay buffer capacity for SCROLL. We focus on SCROLL (Ridge), given its better performance over the NCC variant. Tab. reports how buffer capacity affect model performance using either residual adapters or full-model tuning.
We first observe that ER contributes significantly final performance. Using Ridge Regression, the memory-free fr only obtains 57% for CIFAR-100 100 using Ridge Regression. Even with limited buffer of 200 (i.e. 2 samples per class), ER adds about 3% the final accuracy. When A 2000, ER adds more than 10%. The results suggest that updating the pre-trained representation and the associated classifier is important. When combined with the effects from classifier initialization, the results suggest that all components in our method are important for learning robust predictors.
We will discuss in Appendix C. how implement (7) as standard linear layer
8

---

Under review as a conference paper at ICLR 2023
CIFAR-10
CIFAR-100
fr Accuracy
81.7 +
57.1 +
Adaptation Routine
|M] 200
[M] 500
[M] 2000
|M] 200
[M] 500
[M] 2000
Residual Adapter
84 1:
86.3 0.2
89.0 0.1
59.9 0.1
0.1
64.7 1: 0.2
Full-model tuning
81.5 + 0.03
84.1 + 0.4
89.4 00.40022
36.5
0.3
61.1 0.2
68.0 0.2
Table 4: Classification accuracy of different adaptation routines for SCROLL (Ridge).
Tab. 4 also suggests that full-model tuning causes overfitting when replay data is limited. At JM 200. the final predictor f consistently underperforms fr. As the buffer size increases, full-model tuning becomes more effective compared to residual adapters. For CIFAR-100, full-model tuning is over 3% higher |M] = 2000. We also stress that the residual adapters are still usable with high buffer capacity, improving nearly 8% from fr. For both datasets, we set the threshold m = 500 for the practical algorithm to switch from residual adapters full-model tuning.
5 DISCUSSION
In this work, we introduced the notion of schedule-robustness in CL, which requires algorithms to perform consistently across different data streams. We argued that such property is key to CL applications, where the schedule is often unknown a priori. However, most existing methods disregard this aspect and are designed around specific schedules. We empirically demonstrated that this severely degrades their performance when different schedule is used. To tackle these issues, we proposed SCROLL and formulated CL as learning online a schedule-robus classifier, followed by predictor adaptation using only replay data. SCROLL is schedule-robut by design and outperforms existing methods across all evaluated schedules. We now focus on two key aspects that emerged from this work, which we believe provide useful insights for CL:
Pre-training and CL. The results from our experiments show that pre- pre-training is broadly beneficial for all evaluated methods (see our Tab. 1 vs Tab. 2 in Buzzega al. (2020)). We believe that this observation should call for greater focus on how to exploit pre-training to develop robust CL methods: our choice pre-train the feature extractor & in SCROLL and combining with linear online methods such as incremental Ridge Regression, is a principled decision derived from our meta-lcarning-basee perspective, which also helps mitigate forgetting. In contrast, most previous CL methods adopting pre-training see it as a preliminary phase rather than an integral component of their methods (Wang al., 2022; Shanahan al., 2021; Wu et al., 2022). This approach often leads to sub-optimal design decisions and potentially less effective techniques for leveraging pre-training as previously observed in Mehta et al. (2021) and further highlighted by our experiments (see also Appendix B.2 where SCROLL (Ridge) outperforms the more sophisticated ensemble model Shanahan et al. (2021)).
Pre-training is also realistic assumption. We have argued that application domains (e.g., computer vision and natural language understanding) have vast amounts of curated data and increasingly available pre-trained models that CL algorithms could readily exploit. Additionally, we demonstrated in Sec. 4.2 that pre-training with even modest dataset is very beneficial. which is promising result for applying this strategy to other domains. In particular, we note these settings are analogous to initializing a CL. algorithm with random model but then having large first batch (i.e., containing samples from several classes), a schedule that is gaining increasing traction in the literature (Hou al., 2019; Mittal al., 2021). Our strategy extends naturally to these settings if we allow offline training on the first batch to replace pre-training.
Class-independed Online Learning and On-demand Adaptation. We believe that SCROLL provides a new general practice to approach CL problems. In particular, the strategy of first learning online a coarse-yet-robue model for each individual class and then adapting it on-demand on data of interest grants flexibility as well as consistency to CL algorithms. The online step focuses on learning each class independently, which prevents inter-class interference (Hou et al., 2019) by construction. This single-class learning process is open-ended (i.e. one-vs-all for each class) with any new classes easily assimilated. We have demonstrated that the predictor fr resulting from this first step, is already competitive with most previous CL algorithms. Then, at any moment T in time (not necessarily at the end of the sequence), we can adapt fo to a f specializing to all the observed classes so far. This step is performed with cross-entropy loss, which enforces interaction among classes and ultimately leads to improved performance. SCROLL's 's robust performance offer strong support to adopting this strategy more broadly in CI.

---

Under review as a conference paper at ICLR 2023
Limitation and Future Works. We close by discussing some limitations of our work and future research directions. Firstly, while model adaptation works well in practice, most previously observed data omitted from directly updating the feature extractor. One key direction plan to investigate how to perform the periodic update of the feature extractor during Cl., to improve data efficiency while preserving schedule-robustness. Secondly,
note that the residual adapters in model specific to convolutional networks. We will extend our method to more powerful architectures like transformers.
Reproducibility Statement. As per ICLR 2023 guidelines", we will make sample code available to AC and reviewers via private message on OpenReview.net once the discussion forum will be opened. The code reproduces our results on CIFAR-10U for [M| = 500 and |M = 2000, and on MiTTMAGENET for M 500. See README in the code for the setup guide.
REFERENCES
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss- il: Separated softmax for incremental learning. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp.
2021.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. Advances in Neural Information Processing Systems, 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Procedd- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019b.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, 32, 2019c.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. International Conference on Learning Representations, 2019.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone
Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European Conference on Computer Vision, 2022.
Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-522 5266, 2002.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920-15930, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky, New insights reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022. URL https: http///pppreeee..ooo//ooe net /fouum? UU.B......
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019a.
URL htt.h:t:///// / /pwnrevieeeeeeeeeeeeeeeeeeeeeeee n forum? 1d=
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint 10ti:1992210488 2019, 2019b.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366-3388 2021.
20we wwwee//mmmkee//0oorr200000oee0/. oi
C2 A0
10

---

Under review as a conference paper at ICLR 2023
Adrian El Baz, Ihsan Ullah, Edesio Alcobaça, André CPLF Carvalho, Hong Chen, Fabio Ferreira, Henry Gouk, Chaoyu Guan, Isabelle Guyon, Timothy Hospedales, et al. Lessons learned from the neurips 2021 metadl challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification. In NeurlPS /021 Competitions and Demonstrations Track, pp. 80-96. PMLR, 2022.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70. JMLR. org, 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 33: 1588-11598, 2020.
Tyler Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 220-221, 2020.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):
2021.
Saihui Hou, Xinyu Pan. Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 331-839, 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear Estimation. Prentice Hall, 2000.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnskaa-awiisaaa,.ma et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526. 2017.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains few-shot classification. In Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 9526-995 90666995,,222 2021.
Weihong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task-specific adapters. In JEEE/CVF Conference on Computer Vision and Pattern Recognition 2022, 2022.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2221)))99352947,2222 2017.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 669:28-51, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
11

---

Under review as a conference paper at ICLR 2023
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153, 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understand- ing the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33:7308-7323 2020.
Sudhanshu Mittal, Silvio Galesso, and Thomas Brox. Essentials for class incremental learning- In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
3513-3522, 2021.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. CLEVA-compass: A continual learning evaluation assessment compass to promote research transparency and comparability, In International Conference on Learning Representations, 2022.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp- 721-731, 2018.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 524-540, 2020.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Representations, 2020. URL https: http://ppeeeeeeeeeeeeeeeeeeeeeee net/ www.orrummii---ggoooo--o---
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 331-339, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019. URL https: /ttpss:////eeeeieeeeeee net/ net/Eorum?
Eorum? (wwdda11ghhhhhhh
Ruslan Salakhutdinov and Geoff Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In Artificial Intelligence and Statistics, pp. 412-419. PMLR, 2007.
Murray Shanahan, Christos Kaplanis, and Jovana Mitrovié. Encoders and ensembles for task-free continual learning. arXiv preprint arXiv:2105.13327 2021.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference Computer and Communications Security, pp. 1310-1321, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision. pp. 266-282. Springer. 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pitrre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples. International Conference Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2023
Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1)377-57, 1985.
Ruohan Wang, Massimiliano Pontil, and Carlo Ciliberto. The role of global labels in few-shot classification and how to infer them. Advances in Neural Information Processing Systems, 34, 2021.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9601-9610, 2022.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust contin- ual learning with additive parameter decomposition. International Conference on Learning Representations, 2020.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang. Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1982-6991) 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint arXiv: arXiv:1212.5700 1212.5701 2012.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
This appendix organized in the following manner:
In Appendix A we comment on the scheduue-robusttees property of Ridge Regression;
In Appendix we present additional empirical investigation on the behavior of SCROLL and its main components;
In Appendix C we report the implementation details of SCROLL used in our experiments;
In Appendix D we provide more information on model tuning and baseline methods.
In Appendix E we discuss in more detail the definition of schedule-robustnen introduced in Sec. 2.2 providing a more formal definition of this notion.
A RIDGE REGRESSION AND SHHEDULE-ROBUSTNERR
We show that Ridge Regression is schedule-robust and mitigates forgetting. From Sec. 3.1, the Ridge Regression classifier is defined as f(z) = arg max, wy (xr), where
W [w wk] = arg min W
Σ
[ OneHot(y)) ||]I
(12)
[
(E.
with de as a feature extractor. The closed-form solution for W is
W (X XX
where
(13)
= [v((1)) ... (xN)) and Y = [OneHot(y1).... OnneHo((yy)]] Substituting (14) into (13), we have
(14)
x
- (a))(a)) and x le1...ck] d=1
(15)
where = ExEx. (xx) is the sum of embeddings from class X and K is the total number of classes in the dataset D.
From (15), it is clear that both X X and X Y, summations, invariant to the ordering of samples in D. Therefore computing W is also invariant to sample ordering in D and may be performed online using (10). We thus conclude that Ridge Regression is schedule-invariant. Since Ridge Regression learns each w; independently. it also mitigates catastrophic forgetting caused by class interference.
We note that while Ridge regression with fixed feature representation resembles streaming linear discriminant analysis with deep models Hayes & Kanan (2020), key difference is that the former guarantees
while the latter is sensitive to sample ordering..
B ADDITIONAL EXPERIMENTS
B. ADDITIONAL RESULTS ON DIFFERENT SPLITS FOR CIFAR DATASETS
We compare SCROLL with the baseline methods on CIFAR-10 10-split and CIFAR-100 50-split respectively. The classification accuracies are reported Tab. 5.
SCROLL (Ridge) outperforms all baselines in Tab. 5. We observe that most baseline methods performance drop drastically compared to Tab. I except GDumb, suggesting worse forgetting with more dynamic schedules, even though the underlying data remains unchanged. In contrast, SCROLL able to perform consistently due to its schedule-robus design. The drop in baselines performance also consistent with Boschini et al. (2022): most existing methods not designed to effectively exploit pre-training. When the same dataset split into more tasks (or batches), the later ones benefit less from the prior knowledge encoded in the initial predictor fo-
14

---

Under review as a conference paper at ICLR 2023
CIEAR-10 10-splät
CIFAR-100 50ssplit
Joint Training (i..d.)
91.2 11.6
79.3 10.4
CT. Algorithm
[M] 200
500
[M] 2000
[M] 200
[M] 500
[M] 2000
OFFLINE
54.8
71.5
15.1
+0.4
BIC
64.9 +2.0
78.8 +0.5
87.5 +0.6
16.1
25.7 4.066
36.8 4:2.7
DER
54.4 #6.2
63.2 #2.1
75.4 #7.4
15.7 #1.7
29.6 2.00
48.2 11.4
DFR+
62.9
68.8
77.4
18.2
42.0
53.1 10.9
87.1
22.8
#1.1
ONLINE
FR-ACE
11.
77.3 20.8
81.7 11.2
26.8 =0.9
35.1 +0.9
47.2 20.5
ER-AML
11.7
80.0 1:0.3
86.2 10.6
13.2 #1111
24.4 11.8
41.1 11.1
SSIL
65.1 #1.6
70.7 1:0.4
74.2
32.9
40.9
52.2 1:0.9
MIR
22.6
63.8 11.5
72.5 11.7
4..7
17.1
41.1 2:05
(OUI))
SCROLL (NCC)
+0.1
SCROLL (Ridge)
84 1..2
86.3 1.0.2
89.4 1:0.2
59.9 =0.1
61.5 1:0.1
68.0 1:0.2
Table 5: Class-incremental classification accuracy on CIFAR- 100 50-split and CIFAR-10 10-split,
B.2 COMPARISON WITH ENSEMBLE METHOD
We also compare SCROLL ENS (Shanahan et al., 2021), which is designed tackle different schedules, including Gaussian schedule introduced in the same paper (see Fig. 3). Specifically, ENS keeps the pre-trained feature extractor fixed and learns an ensemble classifiers for the novel classes. For ENS, we keep its original design for pre-training: using ResNet50 via self-supervised learning on ImageNet We also include the memory- free variant of SCROLL (Ridge) the comparison, since ENS does not use ER.
CIEAR-10
CIFAR-100
CL. Algorithms
5-split
10-spii
Gaussian schd
20-split
100-split
Gaussian Schd
ENS (ReaNe150)
79.0 10.4
78.3 100.4
50.1 19.5
55.3 20.4
54.1 20.5
39.0 114
SCROLL (ResNet50, M
67.2
www.
www............................. w...............................
SCROLL (ResNet18, M
81.7 +:0
57.1 +0
SCROLL (Resmet18, MM = 500)
86.3 #0.2
61.5 20.1
Table 6: Classification accuracy comparison SCROLL (Ridge) vs. Ensemble.
In Tab. 6, observe that the ResNet50 provided by ENS is substantially more powerful than the ResNet18 used in our main experiments. In addition, memory-free SCROLL returns deterministic predictors fr for all possible schedules, while the performance of ENS degrades on the Gaussian schedule, despite its robustness in handling different splits. The results further validates that is intractable optimize CL algorithms against all schedules, and robbustees should be considered as part of algorithm design. Lastly, we note that SCROLL has the additional advantage of using ER further improve performance, compared to the lack of ER in ENS.
Precabiit of cocs Incuston
3.8
0.2
3.0
20
300
Banch x10
Figure 3: Example of Gaussian schedule. Figure originally from Shanahan et al. (2021). The schedule proposes sample class with Gaussian probability, where each class peaks different time. There class boundaries, the probability sampling each class evolves smoothly over time.
SResNet50 checkpoint available at httpp: / thub. com/ coo//eeee deepmi nd deepmi nd nd-researchh cree/master//e
15

---

Under review as a conference paper at ICLR 2023
B.3 THE EFFECTS OF BUFFERING STRATEGY
In this section, we study different buffering strategies used in ER. We consider both exemplar selection (Rebuffi al., 2017) and random reservoir sampling (Vitter, 1985), which are commonly used in existing works. We also introduce two simple variants to exemplar selection, including selecting samples nearest the mean representation of each class (Nearest Selection), or selecting samples furthest to the mean represeniation of each class (Outlier Selection).
CIEAR-10
CIFAR-I00
Buffering strategy
M] = 500
[M] = 2000
M 500
[ 2000
Exemplar
86.3 20.2
89.4 #0.2
61.5 #0.1
68.0 #0.2
Random
84.5 #0.2
88.8 #0.1
61.1 #0.1
65.4 #0.1
Nearest
72.3 40.1
82.2 #0.1
55.6 +0.1
57.0 +0.2
Outlier
72.6
82.1
55.5 +0.1
57.0
Table 7: Classification accuracy with different buffering strategy
We report the effects of different buffering strategies in Tab. 7. The exemplar strategy clearly performs the best, followed by random reservoir sampling. Both nearest and outlier selection perform drastically worse. From the results, we hypothesize that an ideal buffering strategy should store representative samples of each class (e.g., still tracking the mean representation of each class as in exemplar selection), while maintaining sufficient sample diversity (e-g.. avoid nearest selection). We remark that the results in Tab. 7 are obtained using fixed feature extractor vb. We leave it to future work to investigate the setting where continuously updated during CL.
The results have several implications for CL algorithms. While random reservoir sampling performs well on average, it can perform poorly in the worst cases, such as when the random sampling happens to coincide with either nearest outlier selection. Therefore, it appears helpful to rank the samples observed during CL and maintain representative yet diverse samples. On the other hand, the results show that GDumb (Prabhu al., 2020) is not fully schedule-robust, since random reservoir sampling would perform poorly in worst cases.
B.4 BUFFERING STRATEGY AND SCHEDULE ROBUSSTESS
As discussed in Sec. 3.3, exemplar selection yields a deterministic replay buffer when one or more classes presented in single batch. In this section, further investigate its behaviors under schedules with smaller batches, where data from a class is spread out in sequence. We use random reservoir sampling baseline for comparison.
In this experiment, we consider each class separately, since both exemplar strategy and random sampling select samples from each class independently. We thus use buffer size by and batch size b2 on a per-class basis here. Batch size refers to the number of samples in a batch for single class, while buffer size refers to the allowed buffer capacity per class.
We use the & distance |1 w as the metric to measure how well the stored samples in the last replay buffer Mr approximates the population mean of each class, where f, the mean of the points in Mr belonging to class y E y and μy the mean of X namely all points belonging to class y in the dataset D.
We consider 4 scenarios: 1) b 20, b, = 20; 2) b, = 20, by = 80; 3) bj = 90, by = 10; 4) b = 50, b2 50. We also note that dj 20 represents one of the worst possible scenario for CIFAR-10, since it implies a total buffer size of 200 and no class is allowed to temporarily borrow capacity from other classes. All scenarios are run with 100 random shuffling of the class data to simulate various schedules.
Fig. shows that the exemplar strategy is much better at approximating the population mean of each class for all scenarios, compared to random sampling. This corroborates our results from Appendix B.3 that exemplar selection selects more representative samples and thus performs better during ER. In addition, note that exemplar selection achieves drastically lower variance for all settings, suggesting that the selected samples at My form similar distributions and make SCROLL schedule-robust.
16

---

Under review as a conference paper at ICLR 2023
0.18
Buffering Strategy
Random Exemplar
0.16
0.14
0.12
Distance
0.10
0.08
0.06
0.04
0.02
b1 by
20
50,
90, b>
Buffer size Batch
Figure Effects buffer and batch size on selected samples Mr. Exemplar selection vs. Random Sampling. Lower distance and variance are better. Distance denotes how well the selected samples approximates the population mean. Lower variance suggests similar distribution at My.
The experiment also suggests practical ways to ensure schedule-robustness: robustness.. We first observe that the exemplar strategy decide on what samples to keep based on the current batch and the current buffer M,. Therefore its behavior is predominantly determined by the combined size of b and b2. Consequently, simply use a large buffer (e.g. bi 90 in scenario 4) will yield My closely approximating desired population mean, even when the batch size b2 10 is very small and generally considered challenging for CL.
B.5 GENERALIZATION PERFORMANCE OF INTERMEDIATE PREDICTORS
Existing CL methods can produce intermediate predictors for partial sequences, such as after observ- ing each class. SCROLL can similarly do on demand: whenever predictor required, SCROLL could compute the current predictor ft Φt zb with the stored statistics, followed by adapting ft on the current buffer M,. The key difference from existing methods is that we will always begin the adaptation from the initial 1a to preserve schedule-robustncd
In Fig. 5, we compare SCROLL (Ridge) to existing baselines with respect the generalization performance of intermediate predictors. We note that the improved generalization performance directly implies reduced forgetting. The results show that SCROLL achieves performance similar to the baselines when the number of observed classes are low. However, SCROLL clearly outperforms the basclines with increasing number of observed classes. This clearly indicates that the proposed method is more robust in mitigating forgetting and achieves better generalization performance.
C MODEL DETAILS
For CIFAR datasets, use ResNet18 backbone pre-trained on the Meta-Dataset (Triantafillou et al., 2019). The model definition and pre-trained representation can be found here". For MGENIM ResNet- 12 backbone as commonly adopted in existing meta-learning works (Lee et al., 2019; Oreshkin et al., 2018; Ravichandran et al., 2019; Tian et al., 2020), and use the default architecture from the official implementation of Tian et al. (2020). Please refer to the accompanying source code for more details.
Following Tian al. (2020), we find beneficial to normalize the embedding (xx) to unit length before classification. The normalization is implemented in predictor ft as non-linearity layer.
'https:/ http::///gghuuuuueeeee thub com VICOOUOR/URI,
17

---

Under review as a conference paper at ICLR 2023
M 500
[M] 2000
SCROLL
BIC
(-DDmmb
BR-ACE
90
DER
ER-AML
Accuracy (%6
70
60
40
30
70
100
10
40
50
80
90
Number of observed classes
Figure S: Intermediate predictors' performance CIFAR-100 10-split schedule: SCROLL (Ridge) vs baselines.
Adapting NCC with Experience Replay. Since we normalize sample embedding, NCC can be implemented standard linear layer. Recall that NCC is
f(z) = arg min [[x(x) cy
s.t.
Cy = n
Σ
((x).
(16)
ZEXV
When (xx is unit-length,
arg min [[b(()) - cyl12 = arg min vxx) (xx) - +xxx) T co + c cv
-
(17)
7/
arg min (x)) T cy + c.y cv
(18)
arg max (z -c
(19)
We thus use cy and -c, Cy to initialize the weights and bias for class y respectively, Adapting NCC with ER therefore treats NCC as standard linear layer as described above.
Optimizer. Following Li et al. (2022), we AdaDelta (Zeiler, 2012) for model adaptation with ER.
ER Hyper-parameters, We list the hyper-parameters for ER in Tab. and state the values used in our grid search.
Hyper- paprm-prr
Description
Possible values
4
learning rate for classifier & learning rate for feature extractor &b temperature for cross-entropy loss batch size from ER
[0.5, 0. 0.01, 0.001] [0.02, 0.01, 0.002, 0.001] [1, 2, 4. 5] [50, 100]
Table 8: ER Hyper-paramettrs in the proposed method and their values used in the grid search.
D MODEL TUNING FOR BASELINE METHODS
D.I CIFAR-10/ 100
Dataset. We resize the input images 84 x 84, both training and test time, to be compatible with the pre-trained representation from Li et al. (2021). During training, we apply random cropping and horizontal flipping standard data augmentations.
Hyperparameters. Starting from the exhaustive search in Buzzega et al. (2020), we tuned all the baselines by testing different learning rates and method-specific hyperparameters using the official
18

---

Under review as a conference paper at ICLR 2023
repository". We used vanilla SGD optimizer without momentum and batch size of 32, training for 50 epochs per task for all offline algorithms. We found that good trade-off between preserving and updating the knowledge in the pre-trained representation is given by learning rates of 0.01 and 0.03. For small buffer sizes, typically /r = 0.01 achieves a lower standard deviation across different random seeds at the cost of slightly lower average performance, while i 0.03 works well with larger memory. Other empirical works also recommend updating the model parameters with lower learning rates by studying the training dynamies of CL algorithms (Mirzadeh et al., 2020). For ER and BIC Wu et al. (2019), we used Ir = 0.01 (no other hyperparameters are present). For DER/DER++ (Buzzega al., 2020), we tested different values of replay loss coefficients a. {0.1, 0.3, 0.5,1.0), 1.0). defined in Eq.6 of the original paper. While for small buffer sizes (200, 500), no clear winner emerges, we found that larger buffer sizes (2000) benefit from larger coefficients. Assuming that the actual data schedule is not known in advance, safe choices for are 0.5 or 0.3, confirming the original paper suggestion. DER++ usually requires 3 2 0.5 and is less sensitive to the choice a. For CIFAR- 100 10-split we used Ir 0.01, 0.5, 3 0.5, while for the 50-split (r 0.03 works best.
All online baselines use single epoch for training. For GDumb (Prabhu et al., 2020), we decay Ir to 5 10-4 with Cosine annealing and modify the original training recipe by removing CutMix regularization (Yun et al., 2019) as we found it not to be compatible with the pre-trained representation. For other online methods, we follow the official implementation of ER-AML and ER-ACE (Caccia et al., 2022) and tunes the learning rate of each method. We observe similarly that i = 0.01 works well for all. Loss coefficients are set using the recommended values the original implementation.
Moreover, our experiments confirm that, if an algorithm is not schedule-robust, its hyper- parameersssssss should be tuned independently for each schedule. However, this is infeasible as the actual schedule is generally unknown, further motivating the necessity of designing schedule-robust CL algorithms.
D.2 minilMAGENET
Dataset. For miniMM we used the 64 classes of the meta-training split for pre-training the representation. Then, we train and test all CL algorithms on the 20 meta-test classes, by splitting the dataset in 10 sequential binary tasks. During training, we apply random cropping and horizontal flipping standard data augmentations. Input images have shape 84 x 84 x 3.
Hyperparameters. Similarly as for EIFAR-10/100 we search for the best hyperparameters of the baseline methods. We train all algorithms with SGD optimizer without momentum and batch size of 32. We found that for buffer size MM = 200, training for 10 epochs enough, while for [M] = 500, better performance are obtained with 20 epochs. Best hyperparameters reported below.
Buffer size |M| 200 (10 epochs)
ER: Ir 0.01
BIC: Ir = 0.01
DER: ir = 0.03, a = 1.0
DER++ Ir 0.03, a 1, B 1.0
Buffer size [M| = 500 (20 epochs)
ER: Ir 0.01
BIC: Ir 0.01
DER: Ir 0.05, a 1.0
DER++ (r 0.05, 0.5, , = 1.0
Buffer size |M| = 200 and |M| = 500 (1 epochs, online methods)
GDumb: Iy 0.03 decayed to 5 x 1 with Cosine annealing, ER epoch E = 10, No Cutmix.
htttpp://ggthhub... com/a imagel / Pttpps: //gi thub. com http///llucaae 1 4/am1
19

---

Under review as a conference paper at ICLR 2023
ER-ACE: Ir 0.01
ER-AML: Ir 0.01, supercon temperature 0.2
SSIL: Ir = 0.01, distillation coefficient a =
MIR: = 0.01, sub-sampling rate r 50.
Buffer size |M| 500 (1 epochs, online methods)
GDumb: i 0.03 decayed to 5 10 with Cosine annealing, ER epoch E 20. No Cutmix.
ER-ACE: Ir 0.01
ER-AML: Ir 0.01, supercon temperature 0.2
SSIL: Ir 0.01, distillation coefficient
MIR: /r = 0.01, sub-sampling rate r 50.
E FORMAL DEFINITION OF
In Sec. 2.2, introduced the notions of schedule invariance and robustness. In particular, said that a CL algorithm is schedule-robust if its performance across different schedules does not vary dramatically. We captured this intuition in (6)
((S, for alg) (() Algg)
VS S2 schedules, vD dataset,
where the symbol loosely implies that small discrepancies between generalization errors are allowed. This definition is however not formal, but rather conveying the general intuition and goal of this work, which to design CL algorithms that are as robust as possible to changes in the schedule.
We can however introduce algorithm is c-schedule-robust (where c >> 0 a given constant), if
formal definition of echedule-robustness
say that CL
C(S) Alg) C(S2(D). Alg) Sc.
(20)
holds for any dataset D and any two schedules S1 and S2 over it. This definition can be weakened to the form
En-D Es ((S) Jo. Alg) L(S, Alg)
Se.
(21)
namely, considering schedule-robustness expectation with respect to a family of datasets D sampled from cutasst--geneaatie distribution D, and of sequences S1, S2 sampled (independently) from a sequence-genee
distribution S. The latter would capture settings where not all datasets and sequences possible, but have knowlede of what kind of sequences and data the CL. algorithm might encounter.
While the two above more rigorous definitions of rrobuttee,,,,sssess, in the present work, we preferred focusing on the more intuitive perspective on the problem, loosely captured by the approximation in (6) with the symbol ss, This is because precisely quantifying the the error incurred by SCROLL would require substantial theoretical analysis of the effects of experience replay (a problem akin to algorithmic stability, see e.g. Bousquet & Elisseeff (2002)), which is outside the scope of this work and would not add to our overall saee-home-msssaae however, that our experiments in Sec. show that SCROLL is schedule-robuutt with small e, according to the above definition (at least on k-split schedules and Gaussian schedules). This is particularly evident from Fig. Our method is also provably invariant to the set of schedules when all samples from one more classes observed together (see discussion in Sec. 3.3), SCROLL's performance does not change as we change the schedule, in contrast to previous methods.
We care to point out,
20


Reviews:
[{"review_id": "j6s8Qht_WGN", "reviewer": "Reviewer_5vq1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper concerns itself with continual learning and observes that current methods are making assumptions on the schedule that can be unrealistic. In order to ameliorate this, the notion of \\emph{schedule-robustness} is introduced. Intuitively schedule-robustness for CL means that the performance is \"independent\" of the schedule. Empirical evaluations are also provided.", "summary_of_the_review": "Given the comments above, I believe that the paper needs improvements in mathematical rigorousness, and some discussion on why SCROLL outperforms other algorithms even on the schedules that they are designed for, as (at least for me) this does not check out and I am looking forward for the author's response in the rebuttal.", "strength_and_weaknesses": "The contributions of the paper are threefold:\n\n1. The notion of schedule-robustness is introduced (see Equation 5). Although I do intuitively understand the definition some things are unclear. First, what is the exact meaning of the $\\approx$-sign? Second, should Equation (5) hold on every dataset D, or only on some datasets? \n\n2. It is observed that some specific pre-existing classifiers are already schedule-robust. This gives a specific algorithm called SCROLL\n\n3. Scroll is evaluated experimentally. Not surprisingly, SCROLL outperforms other approaches when they are applied to different schedules than \"expected\". However it also outperforms them on the schedules for which they were designed, which I don't fully understand. Do you have some intuition on why that is? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MdbkS4OLDvI", "reviewer": "Reviewer_kayx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work raises the issue of schedule robustness in continual learning, and proposes an approach that is schedule-robust by construction. The proposed approach assumes access to a pre-trained feature extractor, then trains a one-layer classifier online, and adapts the feature extractor after the end of the stream using only memory data.", "summary_of_the_review": "Despite the fact that the paper raises an important issue in continual learning (i.e., schedule robustness), I am not convinced that its contribution is significant enough for a top conference like ICLR.", "strength_and_weaknesses": "STRENGTHS\n- The paper raises the issue of schedule robustness, which is very interesting in my opinion.\n\n\nWEAKNESSES\n- I think the contribution of the paper is very limited. In my opinion, the most difficult aspect of continual learning is representation learning, but the proposed approach assummes a pre-trained feature extractor and only deals with the schedule robustness of the final layer of the model. Moreover, training a classifier online while keeping a pre-trained feature extractor frozen has already been explored in past work [1].\n- The use of residual adapters does not really fit into the paper since they do not contribute to schedule robustness. Moreover, we can see from the results in Table 4 that they are only beneficial when the memory size is very very small. Also, it is not possible to say whether they would work well for other kinds of architectures, or even other kinds of convolutional architectures (only ResNet variants are used in the paper).\n- Regarding the results in Figure 2, I think that the main factor that allows scroll to outperform the other methods is that the feature extractor is kept frozen.\n- A minor comment: there are some formatting issues with the pdf (for example, see top of page 9, Table 3 appearing after Table 4, etc.). \n\n\nQUESTIONS:\n- It is unclear to me how a continual learning setting can be both task-free and class-incremental at the same time. Could you explain further?\n- For the results in Tables 1 and 2, do the data arrive in small mini-batches online, or does the model have access to all the data from the present task at the same time?\n- I quickly skimmed over the proof that ridge regression is schedule robust and it seems to me that the proof assumes that all data are available at the same time. This assumption does not hold in online continual learning though. Could you please explain further? Could you also give an intuitive explanation of how ridge regression can be schedule robust?\n- Why do you think NCC performs worse than ridge regression?\n- Is SCROLL applicable to streams with tasks that are not class-disjoint?\n\n\nREFERENCES\n[1] Hayes, T. L., & Kanan, C. (2020). Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 220-221).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G3PMN3SQ7F", "reviewer": "Reviewer_wm7u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method for continual learning from non-stationary data under changing schedules (presentation/observation order of data over time). The general idea is to make the continual learning algorithm robust to different changes of schedules, that is, under exchangeability. For that purpose, the method includes a predictor based on composition. While the first one (or feature extractor) is invariant to permutations (schedules) and trained in an online manner, the second uses Experience Replay (Chaudry 2019).", "summary_of_the_review": "An interesting paper with two important flaws that make it somehow weak for acceptance.", "strength_and_weaknesses": "**Strengths:** The problem considered is relevant for the community, as learning from streaming data might be affected by the permutations of the data, that in the paper are denoted as schedules. This problem is well introduced, and the proposed predictor based on combination is meaninful. I particulary liked the idea of the feature extractor that is invariant to permutations for later training the classifier on a different way. Even simpler, the results were understandable and clear to me, with several related baselines that produce a broad picture of the performance.\n\n**Weaknesses:** From my perspective, there are two main points of weakness that make the paper borderline to me. The first one is around the presentation of the schedule and the treatment of such concepts. In the end, when having some i.i.d data, there are already significant contributions and studies of exchangeability and permutation invariant models. For instance, in Bayesian inference, the log-marginal likelihood is invariant to the permutations of the data, and there are multiple ways of building this loss as a combination of log-predictive conditionals where the order and size of batches might change. In the same direction, there are multiple statistical operators that already consider this problem and is somehow a well-studied problem (independent of the continual learning scenario). Therefore, I see important connections missing in section 3.1 when making references to schedules and permutation invariant feature extractors without any reference to well-known concepts in statistics. For example, Eq. (7) seems just a correction of the expectation equation in Eq. (6).\n\nThe second point is around the novelty. Even if the schedule robust predictor with the new feature extractor would be well connected to statistics and the mentioned perspectives, the use of Experience replay for the classifier does not seem particularly new w.r.t. (Chaudry 2019). In that sense, I would like to know extra details on which are the exact differences between SCROLL and the previous methods based on Experience replay. Otherwise, the technical contribution is perhaps a bit reduced in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3vwvjC1J3c", "reviewer": "Reviewer_dgx2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work builds on the prior work of curriculum learning to make the curriculum learning process neutral to the training schedule. They first form initial pre-representation-training of the network with the help of NCC/ridge regression. The network is divided into 2 components: ϕ (classifier) and ψ (backbone). The ψ from the pre-training is used for the further training of the network (ϕ is discarded). ψ stays fixed after this (This combined with NCC or the ridge regression are the components that help make the entire training process neutral to the training schedule).  \n\nThis is followed by the part where they perform online curriculum learning. They employ memory buffers and episode replay techniques for this purpose. For a given input batch from a data stream, they update the memory buffer and the NCC/Ridge-regression clusters accordingly. Here they also update the ϕ/classifier parameters. Additionally, if the memory buffer is small, they also have some residual connections from ψ to ϕ which also get updated. ", "summary_of_the_review": "A good attempt at solving a not-so-well-studied problem of curriculum learning. As stated before, marginal technical novelty. I personally couldn’t find a comparable schedule-invariant curriculum learning approach. However, they could have done a better job at making the paper’s theory simpler, easier, and most importantly, intuitive to read. Their work improves over SOTA or was very comparable and delivered what they promised. Therefore, at the moment I am leaning toward a marginal rejection. ", "strength_and_weaknesses": "Strengths: \n- Thorough and clearly explained experiments where they improve upon the SOTA. \n- Marginally novel technical contribution. \n- Increasing the memory buffer size increases the curriculum learning accuracy. However, Residual Adapters helped get better performance at lower buffer sizes than the full model training. And the ridge regression initialization did the best job. Another way to think about this work is to aim toward making the memory buffer robust to the schedule.\n\nWeaknesses: \n- Only used on CIFAR10,100 with the help of resnet 18. It would help to see the results on other datasets and networks as well. We need to make sure that the results are data/model agnostic. \n- The transition from F_0 -> F_t -> F_T-> f* is very confusing. \n- They need to do a better job at stating what is being trained and what isn’t and what losses are being used at each stage. Maybe having a better diagram or a few more diagrams or just a table explaining the different stages and loss combinations of training will help. \n- It wasn’t very clear if the memory buffer is big, does the whole ψ get trained or not?? \n- Additionally, they need more experiments like Fig.2 (The paper is about preventing catastrophic forgetting, and I only see one experiment where they are analyzing the model’s performance across time). \n- It also isn’t clear whether Fig2. is using NCC or Ridge regression. \n- They need to clearly state as bullet points or a list all the factors which are contributing towards making their method schedule robust. It felt like these factors were all over the place.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject