Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
Authors: Hyunseo Koh
Dahyun Kim
Jung-Woo Ha
Jonghyun Choi
Venue: ICLR 2022 Poster (2022)
Keywords: online
continual learning
task-free continual learning
any-time inference

Abstract: Despite rapid advances in continual learning, a large body of research is devoted to improving performance in the existing setups.
While a handful of work do propose new continual learning setups, they still lack practicality in certain aspects.
For better practicality, we first propose a novel continual learning setup that is online, task-free, class-incremental, of blurry task boundaries and subject to inference queries at any moment.
We additionally propose a new metric to better measure the performance of the continual learning methods subject to inference queries at any moment.
To address the challenging setup and evaluation protocol, we propose an effective method that employs a new memory management scheme and novel learning techniques.
Our empirical validation demonstrates that the proposed method outperforms prior arts by large margins. Code and data splits are available at https://github.com/naver-ai/i-Blurry.

Published as a conference paper at ICLR 2022
ONLINE CONTINUAL LEARNING ON CLASS INCRE- MENTAL BLURRY TASK CONFIGURATION WITH ANY- TIME INFERENCE
Hyunsco Koh
Dahyun Kim
Jung-Woo Ha² NAVER AI Lab.
Jonghyun Choi341 *Yonsei University
GIST, South Korea
UUpstage Al Research
hyunsec8157 76 gm. gist kc.kr, kdabyun&upstr upt.ayg...uptaag........ jungwee. ha@navercorp.com jceyonsei kr
ABSTRACT
Despite rapid advances in continual learning, a large body of research is devoted to improving performance in the existing setups. While handful of work do propose new continual learning setups, they still lack practicality in certain aspects. For better practicality, we first propose novel continual learning setup that is online, task-free, class-incremental, of blurry task boundaries and subject to inference queries any moment. We additionally propose new metric better measure the performance of the continual learning methods subject to inference queries at any moment. To address the challenging setup and evaluation protocol, we propose an effective method that employs new memory management scheme and novel learning techniques. Our empirical validation demonstrates that the proposed method outperforms prior arts by large margins. Code and data splits are available at https: http//gttttttt www.tttub.ommmeeee wwwooom//avvraaiaeeeeea aii-BBuurr//
INTRODUCTION
Continual learning (CL) is learning scenario where a model learns from a continuous and online stream of data and is regarded
more realistic and practical learning setup than offline learning on a fixed dataset (He et al., 2020). However, many CL methods still focus on the offline setup (Kirk- patrick al., 2017; Rebuffi et al., 2017; Saha et al., 2021) instead of the more realistic online setup. These methods assume access to a large storage, storing the entire data of the current task and it- erating on it multiple times. On the other hand, online setup where only a small memory is allowed as storage. Meanwhile, even for the online CL methods, we argue they have room for more practical and realistic improvements concerning mul- tiple crucial aspects. The aspects include the class distributions such as the disjoint (Rebuffi et al., 2017) or the blurry (Aljundi et al., 2019c) splits and the evaluation metric that focuses only on the task accuracy such as average task accuracy (Aaug).
interested extensively in the more realistic
The two main assumptions on the class distributions in existing CL setups, i.e.. the disjoint and blurry splits, are less realistic for the following reasons. The disjoint split assumes no classes overlap over different tasks; already observed classes will never appear again.
The above is not plausible because already observed classes can still appear later on in real-world scenarios (see Fig. 2 of (Bang et al., 2021)). On the other hand, in the blurry split (Aljundi al., 2019c) no new classes appear after the first task even though the split assumes overlapping classes over tasks. This is also not plausible as observing new classes is common in real-world scenarios.
The typical evaluation metric such as Aang in which the accuracy is measured only at the task transition also less realistic. It implicitly assumes that no inference queries occur in the middle of a task. However, in real-world scenarios, inference queries can occur any-time. Moreover, there is no explicit task transition boundary in most real-world scenarios. Thus, is desirable for CL
indicates equal contribution. indicates corresponding author. This work was done while HK, DK and JC were interns and an AT technical advisor NAVER AI Lab.

---

Published as a conference paper at ICLR 2022
models to provide good inference results at any time. To accurately evaluate whether a CL model is effective at such 'any-time" inference, we need a new metric for CL models.
In order to address the issues of the current CL setups, propose new CL setup that is more realistic and practical by considering the following criteria: First, the class distribution is comprised of the advantages from both blurry and disjoint. That is, assume that the model continuously encounters new classes as tasks continue, i.e., class-incremental and that classes overlap across tasks, i.e., blurry task boundaries, while not suffering from the restrictions of blurry and disjoint. Second, the model is evaluated throughout training and inference such that it can be evaluated for any-time inference. We call this new continual learning setup 'i-Blurry'.
For the i-Blurry setup, we first propose plausible baseline using experience replay (ER) with reser- voir sampling and a tuned learning rate scheduling. While existing online CL methods are applicable to the i- i-Blurry setup, they perform only marginally better than our baseline or often worse.
To better handle the i-Blurry setup, we propose a novel continual learning method, which improves the baseline in three aspects. We design a new memory management scheme to discard samples using per-sample importance score that reflects how useful a sample is for training. We then propose to draw training samples only from the memory instead of drawing them from both memory and the online stream as is done in ER. Finally, we propose a new learning rate scheduling to adaptively decide whether increase or decrease the learning rate based on the loss trajectory, i.e. a data-driven manner. To evaluate the algorithms in the new setup, we evaluate methods by conventional metries, and further define new metric called 'area under the curve of accuracy' (AAuc) which measures the model's 's accuracy throughout training.
We summarize contributions follows:
Proposing a new CL setup called i- i-Blurry, which addresses a more realistic setting that is online, task-free, glass-incremental, of blurry task boundaries, and subject to any-time inference. Proposing novel online and fask-free CL method by new memory management, memory usage, and learning rate scheduling strategy.
Outperforming existing CL. models by large margins on multiple datasets and settings. Proposing new metric to better measure CL model's capability for the desirable any-time inference.
2 RELATED WORK
Continual learning setups. There are many CL setups that have been proposed to reflect the real- world scenario of training learning model from a stream of data (Prabhu et al., 2020). We catego- rize them in the following aspects for brevity.
First, we categorize them into (I) task-incremental (task-IL) and (2) class-incremental learning (class-IL), depending whether the task-ID is given at test time. Task-IL, also called multi-head setup, assumes that task-ID is given at test time (Lopez-Pat Paz & Ranzato, 2017; Aljundi et al., 2018; Chaudhry al., 2019). In contrast, in class-IL, or single-head setup, task-II is given test time and has to be inferred (Rebuffi et al., 2017; Wu et al., 2019; Aljundi et af., 201 20))... Class-II. more challenging than task-IL, but is also more realistic since task-ID will not likely be given in the real-world scenario (Prabhu et al., 2020). Most CL works assume that task ID is provided at training time, allowing CL methods to utilize the task ID to save model parameters at task bound- aries (Kirkpatrick et al., 2017; Chaudhry et al., 2018b) for later use. However, this assumption is impractical (Lee et al., 2019) since real-world data usually do not have clear task boundaries. To address this issue, a task-free setup (Aljundi et al., 2019b), where task-ID at training is not avail- able, has been proposed. We focus extensively on the task-free setup as it is challenging and being actively investigated recently (Kim et al., 2020; Lee et al., 2019; Aljundi et al., 2019c).
We now categorize CL setups into disjoint and blurry setup by how the data split is configured. In the disjoint task setup, each task consists of a set of classes disjoint from all other tasks. But the disjoint setup is less realistic as the classes in the real-world can appear at any time not only in a disjoint manner. Recently, to make the setup more realistic, a blurry task setup has been proposed and investigated (Aljundi et al., 2019c; Prabhu et al., 2020; Bang et al., 2021), where 100 M% of the sampels are from the dominant class of the task and M% of the samples are from all classes,
2

---

Published as a conference paper at ICLR 2022
where M is the bluryy level (Aljundi et al., 2019c). However, the blurry setup assumes class is added in new tasks, i.., not dtaseeincremental, which makes the setup still not quite realistic.
time, we categorize CL setups into online (Rolnick et al., 2018; Aljundi et al., 2019a; Chaudhry ct al., 2019) and offline (Wu et al,, 2019; Rebuffi et al., 2017; Chaudhry et al., 2018b; Castro et al., 2018). In the offline setup, all data from the current task can be used an unlimited number of times. This is impractical since it requires additional memory of size equal to the current task's data. For the online setup, there are many notions of 'online" that differs in each literature. Prabhu et al. (2020); Bang et al. (2021) refer online to setup using each streamed sample only once to train a model while Aljundi al. (2019c;a) refer online to setup where only one or a few samples are streamed at a time. We follow the latter as the former allows storing the whole task's s data, which is similar to offline and less realistic.
Finally, depending on how many samples are streamed
In this work, we propose a novel CL setup that is online, task-free, class-incremenntl, of blurry task boundaries, and subject to any-time inference as the most realistic setup for continual learning. Note that task-frree and class-incremental are compatible (Jin et al., 2020; van de Ven et al., 2021).
Continual learning methods. Given neural networks would suffer from catastrophic forget- ting (McCloskey & Neal, 1989; Ratcliff, 1990), the online nature of streaming data in continual learning generally aggravates the issue. To alleviate the forgetting, there are various proposals to store the previous task information; (I) regularization, (2) replay, and (3) parameter isolation.
(I) Regularization methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017b; Ebrahimi et al., 2020) store previous task information in the form of model priors and use it for regularizing the neural network currently being trained. (2) Replay methods store a subset of the samples from the previous tasks in an episodic memory (Rebuffi et al., 2017: Castro et al., 2018; Chaudhry et al., 2019; Wu et al., 2019; Kim et al., 2019) or keep a generative model that is trained to generate previous task samples (Shin al., 2017; Wu et al., 2018; Hu al., 2019; Cong al., 2020). The sampled or generated examplars are replayed on future tasks and used for distillation, constrained training, or joint training. (3) Parameter isolation methods augment the networks (Rusu al., 2016; Lee al.,
2017a; Aljundi et al., 2017) or decompose the network into subnetworks for each task (Mallya & Lazebnik, 2018; Cheung et al., 2019; Yoon et al., 2020).
Since (1), (2), and (3) all utilize different ways of storing information that incurs parameter storage costs, episodic memory requirement and increase in network size respectively, a fair comparison among the methods is not straighforward, We mostly compare our method with episodic memory- based methods (Aljundi et al., 2019a; 99; Prabhu et al., 2020; Bang et al., 2021) due to performance, but also with methods that use regularization well (Chaudhry et al., 2018b; Wu et al., 2019).
Online continual learning. Despite being more realistic (Losing et al., 2018; He et al., 2020), online CL have not been popular (Prabhu et al., 2020) due to the difficulty and subtle differences in the setups in the literature. ER (Rolnick et al., 2018) is a simple yet strong episodic memory-based online CL method using reservoir sampling for memory management and jointly trains model with half of the batch sampled from memory. Many online CL. methods are based on ER (A (Aljundi al., 20;)c;)). GSS (Aljundi et al., 2019c) selects samples based on cosine similarity of gradients. MIR (Aljundi et .l., 201 9a) retrieves maximally interfering samples from memory to use for training.
Different from ER, A-GEM (Chaudhry al., 2019) uses the memory to enforce constraints on the loss trajectory of the stored samples. GDumb (Prabhu et al., 2020) only updates the memory during training phase and trains from scratch at the test time only using the memory.
The recently proposed RM (Bang et al., 2021) uses an uncertainty-based memory sampling and two-stage training scheme where the model trains for one epoch on the streamed samples and trains extensively only using the memory at the end of each task, delaying most of the learning to the end of each task. The uncertainty-bent two-stage training leads to poor any-time inference. Our method outperforms all online CL methods introduced in this section while strictly adhering to the online and task-free restrictions.
memory sampling is not well-suited for online CL and the
3 PROPOSED CONTINUAL LEARNING SETUP: I-BLURRY
For a more realistic and practical CL setup, considering real-world scenarios, we strictly adhere to the online and fask-free CL. setup (Lee et al., 2019; Losing et al., 2018). Specifically, we propose
3

---

Published as a conference paper at ICLR 2022
Class partitioning in i-Blurry-N-M
Class distribution in time
ssample
# saappess
Class 5 Class
N% Disjoint
Only appears fixed
Class 504
Cless Class 245
BlurryM Always appoars across tasks
class
class
Clss
Task @ f
Task @ 1+1
Figure 1: i-Blurry-N-A split. N% of classes are partitioned into the disjoint set and the rest into the BlurryM M set where M denotes the blurry level (Aljundi et al., 2019c). To form the i-Blurry- N-M task splits, we draw training samples from uniform distribution from the 'disjoint' or the "BlurryM"' set (Aljundi et al., 2019c). The 'blurry' classes always appear over the tasks while disjoint classes gradually appear.
Asyg: 61.52%
Aavg: 73.90%
(Ours)
0.8
Accuracy
0.6
0.4
Auuc 70.26%
0.2
A 23.35%
0.0
10k
20k
30k
40k
50k
0
10k
20k
30k
40k
50k
samples
samples
(a) Rainbow Memory (RM) (Bang al., 2021)
(b) CLIB
Figure 2: Comparison of AAUC with Aang- (a) online version of RM (Bang et al., 2021) (b) proposed
CLIB. The two-stage method delays most of the training to the end of the task. The accuracy-to-f of samples) plot shows that our method is more effective at any time inference than the two-stage method. The difference in Aavg for the two methods is much smaller than that of in AAUC. implying that AsuC captures the effectiveness at any-time inference better.
novel CL setup (named i-Blurry), with two characteristics: 1) class distribution being class incremental and having blurry task boundaries and 2) allowing for any-time inference.
i-Blurry-N-M Split. We partition the classes into groups where N% of the classes are for disjoint and the rest of 100 100-N% N% of the classes are used for Blurry sampling (Aljundi et al., 2019c), where M is the blurry level. Once determine the partition, draw samples from the partitioned groups. We call the resulting sequence of tasks as i-Blurty-N-M split. The i-Blurry-N-M splits feature both class-incremental and blurry task boundaries. Note that the i-Blurry- previous CL setups. For instance, N = 100 is the disjoint split as there are no blurry classes. N = 0 the Blurry M split (Aljundi et al., 2019c) as there are no disjoint classes. M 0 is the disjoint split as the blurry level is M = 0 (Bang et al., 2021). We use multiple i- i-Blurry-N- reliable empirical validations and share the splits. Fig. illustrates the B-Blurry- N -M split.
splits generalize
splits for
A New Metric Area Under the Curve of Accuracy (AAuc). Average accuracy (i.e., Aaeg = + +E=, A; where A; is the accuracy at the end of the 1th task) is one of the widely used measures in continual learning. But Auvg only tells us how good CL model is at the few discrete moments of task transitions Thus, a CL method could be poor at any-time inference but the Aang may be insufficient to deduce that conclusion due to its temporal sparsity of measurement. For example, Fig. 2 compares the online version of RM (Bang et al., 2021), which conducts most of the training by iterating over the memory at the end of a task, with our method. RM is shown as it is a very recent method that performs well on Aong but particularly poor at any-time inference. Only evaluating with Aaeg might give the false sense that the difference between the two methods is not severe. However, the accuracy-to- (# of samples} curve reveals that our method shows much more consistently high accuracy during training, implying that our method is more suitable for any-time inference than RM. To alleviate the limitations of Aaeg. we shorten the accuracy measuring frequency after every An
10 times for most CL setups) when the model could be queried any time.
4

---

Published as a conference paper at ICLR 2022
Sample-wise Importance Sampling
Memory
Sample-wise Importance M
Model
Inference Query
Random Samping
Train
User
Inference Resull
Online Stream
Adaptive
Figure 3: Overview of the proposed CLIB. We compute sample-wist wise importance during training to manage our memory. Note that we only draw training samples from the memory whereas ER based methods draw them from both the memory and the online stream.
samples observed instead of at discrete task transitions. The new metric is equivalent to the area under the curve (AUC) of the accuracy-to- (# of samples) curve for CL methods when An = 1. We call it area under the curve of accuracy (AAuc)):
& AAUC = Σ f(i An) An,
(1)
where the step size An is the number of samples observed between inference queries and f()) is the curve in the accuracy-to- [# samples} plot. High AAUC corresponds to CL method that consistently maintains high accuracy throughout training.
The large difference AAUC (see Fig. 4) implies that delaying strategies like the two-stage training scheme not effective for any-time inference, conclusion harder to deduce with just Aaeg-
METHOD
4.1 BASELINE FOR I-BLURRY SETUP
To address the realistic B-Blurry setup, we establish a baseline for the challenging online and task- i-Blurry setup. For the memory management policy, we use reservoir sampling (Vitter, 1985) and for memory usage, we we use experience replay (ER). For the LR scheduling, we use an exponential LR schedule but reset the LR when new class is encountered. Please see Sec. A. for details.
Note that the above baseline still has room to improve. The reservoir sampling does not consider whether one sample could be more useful for training than the others. ER uses samples from the stream directly, which can skew the training the exponential with reset does increase the LR periodically, the sudden changes may disrupt the CL model. Thus, we discuss how we can improve the baseline in the following sections. The final method with all the improvements is illustrated in Fig. 3.
CL. models to recently observed samples. While
4.2 SAMPLE- WISE IMPORTANCE BASED MEMORY MANAGEMENT
In reservoir sampling, the samples are removed from the memory at random. Inspired by works on sample importance (Klock & Van Dijk, 1978; LeCun et al., 1990; Chang et al., 2017; Katharopoulos & Fleuret, 2018; Csiba & Richtárik, 2018), we propose a novel sampling strategy specific for CL that removes samples from the memory based on sample-wise importance as following Theorem 1.
Theorem 1. Les C = M U {[Znew )new)}, ), M be memory from the previous time step, (Inew: Vnew) be the newly encountered sample, I(-) be the loss and be the model parameters. Assuming that the model trained with the optimal memory, M will induce maximal loss decrease on C, the optimal memory given by M = =C) ((5, 7)) with
(7. σ) = arg min Eg
Σ (2,0)EC
1(, y: 0) - [ ;: 0 Vol 94 0))
(2)
(E)
Proof. Please see Sec. A.2 for the proof.
5

---

Published as a conference paper at ICLR 2022
Theorem 1 states that when new sample is appended to the memory and sample has to be dis- carded, we should discard the sample that incurs the least loss decrease i.e., least useful for training. We solve Eq. 2 by keeping track of the sample-wise importance H4:
H4s = Eg
1(, y: 8) - (x, y: e - Vol(z,, Ye: 8)) (a.N)EE
Σ
(3)
Intuitively, H. is the expected loss decrease when the associated sample is used for training. We update H. associated with the samples used for training after every training iteration. Specifically, the model is trained with random batch from the memory. Then, for all samples in C., we measure the loss difference before and after training with the batch. If the loss decreases, the importance scores of the samples in the batch increase and vice versa. Note that because the importance score of a sample is relative to that of other samples in the memory, when the importance scores of the samples in the batch decrease, the scores of the samples not in the batch increase comparatively. Thus, two types of samples have high importance scores; 1) samples that were in the batch when the loss decreased and 2) samples that were not in the batch when the loss increased.
The expectation in Eq. 3 is taken over 0's optimization trajectory i.e., over time. For computational efficiency, we use the exponential moving average empirical estimates instead. The empirical estimates are calculated by the discounted sum of the differences between the actual loss decrease and the predicted loss decrease (see Alg. 1). The memory is updated whenever a new sample is encountered, and the full process is given in Alg. 2. The memory management strategy significantly outperforms reservoir sampling, especially with memory only training.
4.3 MEMORY ONLY TRAINING
ER uses joint training where half of the training batch is obtained from the online stream and the other half from memory. However, we argue that using the streamed samples directly will skew the training favor the recent samples more. Thus, propose use samples only from the memory for training, without using the streamed samples. The memory works as a distribution stabilizer for streamed samples through the memory update process (see Sec. 4.2), and samples are used for training only with the memory. We observe that the memory only training improves the performance despite its simple nature. Note that this is different from Prabhu et al. (2020) as we train with the memory during the online stream but Prabhu et al. (2020) does not.
4.4 ADAPTIVE LEARNING RATE SCHEDULING
The exponential with reset scheduler resets the LR to the initial value when a new class is encoun- tered. As the reset occurs regardless of the current LR value, it could result in a large change in LR value. We argue that such abrupt changes may harm the knowledge learned from previous samples.
Instead, propose
data-driven LR scheduling scheme that adaptively changes the LR in data-driven manner based on how good the LR is for optimizing over the memory.
Specifically, from the current base LR and step size 7 < 1, we try both high LR î/ and a low LR rj y for training. For each LR, we keep a history of length m that tracks the loss decrease for each LR. When both histories are full, perform Student's t-test with significance level 0.05 to compare the LRs. If one LR is better, the base LR is set to the better LR, i.e., 212 or n-
We depict this scheme in Alg. 3 in Sec. A.4. The adaptive LR is important for CL methods because the data distribution changes over time and adapting the current training data more useful.
With all the proposed components, we call our method Continual Learning for i-Blurry or (CLIB).
5 EXPERIMENTS
Experimental Setup. We use the CIFARIO, CIFARIO0, TinylmageNet, and ImageNet datasets for empirical validations. We the i-Blurry setup with N 50 and M 10 (i- (i-Bluryy-0-10) for our experiments unless otherwise stated. All results are averaged over 3 independent runs except ImageNet (Wu al., 2019; Bang al., 2021; Prabhu al., 2020). For metrics, we use the average

---

Published as a conference paper at ICLR 2022
Methods
CIFARIO
CIFARI00
TmyllnggeNet
ImageNet
AwIc
Ang
Augy
Avg
Aure
Aus
Aaue
Aws
Joint Training' (Sof: Upper Bound)
96.01
79.89
$3.05
69.26
EWCC+ (Kirkpiärick et ol, 2017)
57,34:22.10 10
60.33::2.73
35.35::1 166
28.78:2.33
22.26::1.15
24.39:1.118
24.81
26.21
(Wo al, 2019)
58.38+0.554
61.49+0.6
33.51-3.04
5761-3.0
22 02.0000044
34.90+1.00
27.41
28.18
ER MIR (Aljundi al, 2019g)
57.2812.43
61,9513.35
35.3511441
38.28.1.15
22 12.101114
24.5111.26
20.18
20.68
GDumb (Prablu
2020)
53.20111.3
55.27:1.60
32.84:0045
34.03-0.899
18.17.0119
18.9100.5
14.41
RM' (Bang al. 2021)
23.00111,43
61.5243.69
8.6310.19
33.27-1.599
5.74
17.0410.77
6.22
28.30
Resellne-FR ER (Sec. A.1)
37,46++2.25
60.17+2.96
55.61=2.06 08
59. 2.000-2202
22:45-1.15
24.54+1.26
25.16
26.50
CLIB (Ours)
70.26.11.28
73.90.10.23
46.67_00.79
49.22.0..79
23.77.0.68
25.0510.52
28.16
28.88
Table I: Comparison of online CL methods on the i-Blurry setup for CIFARIO, CIFAR100, Tiny-
ImageNet and ImageNet. 'Joint Training shows the final accuracy of non-CL joint training a soft upper bound where all relevant syper-parameter, were kept consistent with other compared CL methods. CLIB outperforms all other CL methods by large margins on both the AAUC and the Ageg-
(Ours)
(baseline)
BIC
ER-MIR
GDumb
0.6
0.30
0.35
0.5
0.25
0.30
Accuracy
0.6
0.25
0.20
0.4g
0.3
0.15
0.2
0.10
0.10
0.1
0.05
0.05
0.0
10k 20k 30k
40k
50k
0.0g
10k 20k
30K
40k
50k
0.00
0
20k
40k
60%
80k 1ook
0.00
25 512K 766K 1.0m 1.3m
samples
# samples
samples
samples
(a) CIFARIO
(b) CIFARI00
(c) TinylmageNet
(d) ImageNet
Figure 4: Accuracy-to-(mumber of samples} for various CL methods on CIFARIO, CIFARIOO, TinyImageNet and ImageNet Our CLIB is consistent at maintaining high accuracy throughout in- ference while other CL methods are not as consistent.
accuracy (Aavg) and the proposed AAUC (see Sec. 3). Additional discussion with other metrics such as the forgetting measure (Fims) can be found in Sec. A.10.
Implementation Details. For all methods, we fix the batch size and the number of updates per streamed samples observed when possible. For CIFAR1O, use batch size of 16 and updates per streamed sample. When using ER, this translates to 8 updates using the same streamed batch, since each batch contains 8 streamed samples. For CIFARI00. we use a batch size of 16 and 3 updates per streamed sample. For TinyImageNet, we use a batch size of 32 and 3 updates per streamed sample. For ImageNet, we use a batch size of 256 and I update per every streamed samples. We use ResNet-18 as the model for CIFARIO and ResNet-34 for CIFAR 100 and Tiny TinylmageNet. For all methods, we apply AutoAugment (Cubuk et al., 2019) and CutMix (Yun et al., 2019) following RM (Bang et al., 2021). For memory size, we use 500, 2000, 4000, 20000 for CIFAR10. CIFAR 100, TinylmageNet, ImageNet, respectively. We use 5 tasks for CIFARIO. 10, CIFAR 100, and Tiny ImageNet and 10 tasks for ImageNet. We follow prior works (Bang et al., 2021; Prabhu et al., 2020) for choosing number of updates per sample, memory size, batch size, and number of tasks. Additional analysis on the sample memory size and number of tasks can be found in Sec. A.6. Adam optimizer with initial LR of 0.0003 is used. Exponential with reset LR schedule is applied for all methods except ours and GDumb, with 7 0.9999 for CIFAR datasets and y 0.99995 for TinyImageNet and ImageNet. Ours use adaptive LR with ? = 0.95, m = 10 for all datasets, GDumb (Prabhu al., 2020) and RM (Bang al., 2021) follow original settings in their respective paper. All experiments were performed based on NAVER Smart Machine Learning (NSML) platform (Kim al., 2018; Sung et al., 2017). All code and i-Blurry-N-M splits (see Supp.) is at https: //github.com///// htt.://..coo...vveeeeeeoo aaavve--aii-Biene
Baselines. We compare our method with both online CL methods and ones that can be extended to the online setting; EWC++ (Chaudhry et al., 2018b), BiC (Wu et al., 2019), GDumb (Prabhu et al., 2020), A-GEM (Chaudhry et al., 2019), MIR (Aljundi et al., 2019a) and RM (Bang et al., 2021). .f indicates that the two-stage training scheme (Bang et al., 2021) is used. For details of the online versions of these methods, see Sec. A.5. Note that A-GEM performs particularly worse (also observed in (Prabhu et al., 2020; Mai et al., 2021)) as A-GEM was designed for the task-incremental setup and our setting is task-free. We discuss the comparisons to A-GEM in Sec. A.7.
5.1 RESULTS ON THE I-BLURRY SETUP
In all our experiments, we denote the best result for each of the metrics in bold.
7

---

Published as a conference paper at ICLR 2022
N = 0 (Blurry)
N 50 (i-Blurry)
N = 100 (Disjoint)
Varying
AauC
Awg
AAuC
Awe
AAUC
Aag
EWC+
53.2440..6
57.04=0..94
57.34422.10
60.33:42.73
77.64+1.81
77.80=+1.92
BiC
51.51:0.112
54.8810.30
58.3810..54
61.4910..8
78.78.1..2
80.12.12.20
ER-MIR
52.21+0.85
56.33+0.47
57.28+-2.43
61.93+3.335
76.49+1.97
78.20+22.01
GDumb
45.86=0.80
46.37=2.09
53.20:11.3
55.27=2.69
65.27:11.54
66.74:2..99
RM
1.11
5..07
23.00::
61.52:13.69
33.17:13.71
66.67
Baseline-Bi ER
53.28
1..1
60.17:12.96 57.46.12.25
77.82.12.06
77.47
CLIB (Ours)
68.87+00.8
72.79-0.96
70.26+ 1.28
73.90+0.22
78.58+2.09
77.96+3.28
M = 10
M = 30
M = 50
Varying M
AauC
Awg
AAuC
Awg.
AAUC:
Awg
EWC++
57.34112.10
2..3
65.71
69.94:13.01
68.01
73.36.1.2.33
BiC
58.38=0.54
61.49=0.68
65.88+33.24
70.31+4.88
68.08+3.72
1..3
ER-MIR
57.28:12.43
61.93
65.99.12.28
70.4711..41
68.13-10.65
73.33-11.21
GDumb
53,20++1.93
55.27+2.69
54.73+1.54
54.63+2.39
53.86+00.99
52.82+1.24
RM
23.00=11443
61.52=1..9
2..60::1.74
61.52:10.88
28.32::5.08
59.32:14.57
Baseline-ER
57.46=22.25
60.17=22.96
65.92++2.25
70.04+2.87
68.26+00.4
13.16=1149
CLIB (Ours)
70.26-11.28
73.90
75.04
77.8712.57
75.1411.27
78.30
Table 2: Analysis on various values of N (top) and M (bottom) in the i-Blurry-N-M setup using CIFARIO dataset. For varying N, we use M = 10. For varying M, we use N = 50. Note that N 0 corresponds to the blurry split and N = 100 corresponds to the disjoint split. For N 100, CLIB outperforms or performs on par with other CL methods. For = 0, the gap between CLIB and other methods widens. For N = 50, CLIB again outperforms all comparisons by large margins. For varying M, CLIB outperforms all comparisons excepting only the Aavg when M = 0.
We first compare various online CL methods in the i-Blurry setup on CIFARIO, CIFARIOO I00, Tinylm- ageNet and ImageNet in Table 1. On CIFARIO, proposed CLIB outperforms all other CL methods by large margins; at least + 12.67% in AAUC and +1..5% in Aaeg- On CIFAR 100, CLIB also out- performs all other methods by large margins; least 11147% in AAUC and +10.188 18% in Aeeg- On TinylmageNet, all methods score very low. Nonetheless, CLIB outperforms other CL methods by at least +2.71% in ALc and +1.90% in Aurg- Surprisingly on ImageNet, most methods perform slightly better than on Imayent We believe the reason is that the samples per class is the same with TinylmageNet (e.g., 20) but ImageNet has higher resolution images, making learning from those images easier than Tiny Imagenaeeeet. On ImageNet, CLIB still outperforms other CL methods by least +2.60% in AAUC and +2.06% in Aaug. Since, CLIB uses memory only training scheme, the distribution of the training samples are stabilized through the memory (see Sec. 4.3) which is helpful in the i-Blurry setup where the model encounters samples from more varied classes
Note that Rainbow Memory (RM) (Bang et al., 2021) exhibits a very different trend than other compared methods. On TinylmageNet, it performs poorly. We conjecture that the delayed learning from the two-stage training is particularly detrimental in larger datasets with longer training duration and tasks such as TinyImageNer
On CIFARIO I0 and CIFARI00, RM performs reasonably well in Aaeg but poorly in AaUc- This verifies that its two-stage training method delays most of the learning to the end of each task, resulting in a poor any-time inference performance measured by AAUc- Note that Aaeg fails to capture this; on CIFARIO. 10, the difference in Auvg for CLIB and RM is +13.27% which is similar to other methods but the difference for AAUC is +47.55% which is noticeably larger than other methods. Similar trends can be found on other datasets as well.
We also show the accuracy-to-(ft of samples) curve for the CL methods on CIFAR1O. CIFAR100, TinyImageNet and ImageNet for comprehensive analysis throughout training in Fig 4. Interestingly, RM shows a surged accuracy only at the task transitions due to its two-stage training method and the accuracy is overall low. Additionally, GDumb shows severe decreasing trend in accuracy as tasks progress. is because the 'Dumb Learner' trains from scratch at every inference query leading to accuracy degradation. In contrast, CLIB not only outperforms other methods but also shows the most consistent accuracy at all times. More discussion of other methods are in Sec. A.9.
5.2 ANALYSIS ON DISIOINT CLASS PERCENTAGES (N) AND BLURRR LEVELS (M).
We further investigate the effect of different N and M values in the i- Blurry splits with various CL. methods and summarize the results for varying values of the disjoint class percentages such as
8

---

Published as a conference paper at ICLR 2022
CIFARIO
CIFARI00
Methods
AAUC
Aoeg
AAUC
Aavg
CLIB
10.26:11.28
73.20-10.22
46.67.10.99
49.22.10.79
w/o Sample Importance Mem. (Sec.4.2)
53.75+2.11
56.31
36.59+1.22
1.25
w/o Memory-only training (Sec.4.3)
67.06.11.51
7..65:1.877
44.63:10.22
48.66:10.39
w/o Adaptive LR scheduling (Sec.4.4)
69.70.1.34
73.06::1.35
45.01
48.97:10.12
Table 3: Ablations for proposed components of our method using CIFAR1O and CIFAR100 dataset.
All proposed components improve the performance, with sample-wise importance based memory management providing the biggest gains. While adaptive LR scheduling provides small gains in CIFARIO, the gains increase in the more challenging CIFARI00.
N 0, 50, 100 in Table 2 (top). For N = 0, CLIB outperforms other methods by at least + +16.59% in Aatr and + +6.36% methods, with CLIB being the best in AAUC. For N 50, CLIB outperforms all comparisons by least +12.55% in AAUC and +13.31% in Aavg. Even though CLIB was designed with the i-Blurry setup in mind, it also outperforms other CL methods in conventional setups such the N = 100 (disjoint) or the N = 0 (blurry) setups. It implies that CLIB is generally applicable to online CL setups and not restricted just the i-Blurry setup. Meanwhile, except GDumb, all methods show similar performance on the N = 100 (disjoint) setup. The results imply that the i-Blurry setup differentiates CL methods better than the more traditionally used disjoint setup.
Aang. For N = 100, the performance is similar for the majority of the
Following Bang et al. (2021), 1), we additionally summarize the results for varying values of the blurry level such as M 10, 30, 50 Table 2 (bottom). We observe that CLIB again outperforms or performs on par with other CL methods on various blurry levels.
5.3 ABLATION STUDIES
We show the ablation studies (on CIFARI0/100) for each of the proposed components in Table 3.
Sample-wise Importance Based Memory Management. We replace the 'sample-wise importance memory management' module with the reservoir sampling. As shown in the table, the removal of our memory management strategy degrades the performance in both Axuc and Aarg on both CIFAR10 and CIFAR100, As explained in Sec. 4.2, reservoir sampling removes samples at random, hence samples are discarded without considering if some samples are more important than others. Thus, using sample-wise importance to select which sample to discard greatly contributes to performance.
Memory Only Training. We replace the memory usage strategy from our memory only training with ER. Training with ER means that samples from both the online stream and the memory are used. Without the proposed memory only training scheme, the performance degrades across the board by fair margins. As the streamed samples are being used directly without the sample memory acting as distribution regularizer (see Sec. 4.3), the CL model is more influenced by the recently observed samples, skewing the training and resulting in worse performance.
Adaptive Learning Rate Scheduling. We change the LR scheduling from adaptive LR to expo- nential with reset. The performance drop is larger in the more challenging CIFAR100 where more training iterations make the ablated model suffer more from a lack of good adaptive LR scheduling.
6 CONCLUSION
We question the practicality of existing continual learning setups for real-world application and pro- pose a novel CL setup named i-Blurry. It online, task-free, blass-nccremental, has blurry task boundaries, and is subject to any-time inference. Additionally, evaluate the effectiveness method which uses per-sample memory management, memory only training, and adaptive LR scheduling, named Continual Learning for i-Blurry (CLIB). Our proposed CLIB consistently out- performs existing CL methods in multiple datasets and setting combinations by large margins.
propose new metric to better any-time inference. To address this realistic CL setup, we propose

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
All continual learning (CL) methods including the proposed one would adapt and extend the already trained AI model recognize better with the streamed data. The CL methods will expedite the deployment of AI systems to help humans by its versatility of adapting to new environment out of the factory or research labs. As all CL methods, however, would suffer from adversarial streamed data as well as data bias, which may cause ethnic, gender or biased gender issues, the proposed CLIB would not be an exception. Although the proposed CLIB has no intention allow such problematic cases, the method may be exposed to such threats. Relentless efforts should be made to develop mechanisms to prevent such usage cases in order to make the continuously updating machine learning models safer and enjoyable to be used by humans.
REPRODUCIBILITY STATEMENT
We take the reproducibility of the research very seriously and release all codes, data splits and containers (e.g., Docker) that include the general framework, learned models, and downstream tasks hhttp:://githubee cwtp:/aerreo//eeeooo///eooeee naver- aaa--aiiii--yyy
ACKNOWLEDGEMENT
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2022RIA4 A2CC400300)) and Institute for Information & communications Technology Promotion (ITTP) grants funded by the Korea government (MSIT) (No.2020-0-01361-000 and 2019-0-01842, Artificial Intelligence Graduate School Program (Yonsei University, GIST), and No.2021-0-0206 Artificial Intelligence Innovation Hub)).
REFERENCES
R. Aljundi, P. Chakravarty, and Tuytelaars. Expert Gate: Lifelong Learning with a Network of Experts. In CVPR, 2017. 3
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018. 2
Rahaf Aljundi, Fugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:1 02:11849-- 1849-1189, 2019a. 2, 15
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11254 11263, 2019b. 2
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurlPS, pp. 11816-11825, 2019c. 1, 2, 3, 4, 15, 17
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-War Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In Proceedings of the IEFF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8218-8227, 2021. 1, 3, 4, 6, 7, 8, 14, 15
Francisco M. Castro, Manuel Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteck Ala- hari. End-to-end incremental learning. In ECCV, 2018.
Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30:1002-1012 2017.
Arslan Chaudhry, Puncet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018a. 17
10

---

Published as a conference paper at ICLR 2022
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. ECCV, 2018b. 2, 3, 7, 15, 18, 20
Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR, 2019. 2, 3, 7, 15
B. Cheung, A. Terekhov, Y. Chen, P. Agrawal, and B. Olshausen. Superposition of Many Models into One. In NeurlPS, 2019. 3
Yulai Cong, Miaoyun Zhao, J. Li, Sijia Wang, and L. Carin. GAN memory with no forgetting. In NeurIPS, 2020.
Dominik Csiba and Peter Richtárik. Importance sampling for minibatches. The Journal of Machine Learning Research, 19(1):962-982, 2018.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. AutoAugment: Learning augmentation strategies from data. In CVPR, June 2019.
S. Ebrahimi, M. Elhoseiny, T. Darrell, and M. Rohrbach. Uncertainty-Guie Guided Continual Learning with Bayesian Neural Networks. In ICLR, 2020. 3
J. He, R. Mao, Z. Shao, and F. Zhu. Incremental Learning In Online Scenario. In CVPR, 2020. 1,
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting via model adaptation. In ICLR, 2019. 3
Xisen Jin, Junyi Du, and Xiang Ren. Gradient based memory editing for task-free continual learning. arXiv preprint arXiv:2006.1 15294, 2020. 3
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-25344 PMLR, 2018. 5
C. Kim, J. Jeong, and G. Kim. Imbalanced Continual Learning with Partitioning Reservoir Sam- pling. In ECCV, 2020. 2
Dahyun Kim, Jihwan Bae, Yeonsik Jo, and Jonghyun Choi. Incremental learning with maximum entropy regularization: Rethinking forgetting and intransigence. arXiv preprint https//arrivarr/ddb////000000000 2019.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the mlaas platform with real-world case study. arXiv preprint arXiv: 181009991 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barskkikka,, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-35 -3266, 2017. 2, 3, 7, 14
Tuen Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an ap- plication of integration by monte carlo. Econometrica: Journal of the Econometric Society, pp- 1-19, 1978. 5
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. Advances neural information processing systems, pp. 598-659. 1990. 5
Jeongtae Lee, Jachong Yoon, Eunho Yang. and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. ICLR, 2017a.
Sang-Wo Woo Lee. Jin-Hwa Kim, Woo Ha, and Byoung- -ak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In NeurlPS, 2017b.
11

---

Published as a conference paper at ICLR 2022
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. In International Conference on Learning Representations, 2019. 2,
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020. 14
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In NIPS, 2017. 2
Viktor Losing, Barbara Hammer, and Heiko Wersing. Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing, 275:1 275:1261 1274, 2018. ISSN 0925-
2312. 3
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey, arXiv preprint arXiv:2101.10422) 2021. 7, 14, 17, 18
A. Mallya and S. Lazebnik. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. In CVPR, 2018.
M. McCloskey and Neal. Catastrophic interference in connectionist networks: The sequential learn- ing problem. Psychology of Learning and Motivation, 24:109-165, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. 14
Ameya Prabhu, P. Torr, and Puneet K. Dokania. GDumb: A simple approach that questions our progress in continual learning. In ECCV, 2020. 2, 3, 6, 7, 15, 17, 18
R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97(2):285-308, 1990.
Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert, iCaRL: Incremental classifier and representation learning. In CVPR, 2017. 1, 2, 3, 14, 15
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap. and Greg Wayne. Experience replay for continual learning. arXiv preprint arXiv: 1682, 2018.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv, 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021. 1
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NeurlPS, 2017.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang. Jingwoong Kim, Leonard Lausen, Youngk-
wan Kim, Gayoung Lee, Donghyun Kwak, Jung- Woo Ha, et al. Nsml: A machine learning platform that enables you focus on your models. arXiv preprint 17Xiv: 2017. 7
Gido M de Ven, Zhe Li, and Andreas S Tolias. Class-incremental learning with generative classifiers. In CVPR, 2021. 3
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang. Joost van de Weijer, and Bogdan Raducanu. Memory Replay GANs: learning to generate images from new categories without forgetting. In NeurIPS, 2018.
12

---

Published as a conference paper at ICLR 2022
Yue Wu, Yan-Jia Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019. 2, 3, 6, 7. 15
Jaehong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. ICLR, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp- 6023 6032, 2019. 7
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 ADDITIONAL DISCUSSION FOR OUR BASELINE
In constructing our baseline, we used the reservoir sampling for memory management and ER for memory usage. We used reservoir sampling it is widely used in the online and task-free setups with good performance. Note that in online CL, memory management policies that use the entire task's samples once, such as herding selection (Rebuffi et el., 2017), mnemonics (Liu al., 2020), and rainbow memory (Bang et al., 2021) are inapplicable.
For the memory usage, we use ER which draws half of the training batch from the stream and the other half from the memory. following a large number of online CL. methods based on ER with good performance (Mai et al., 2021).
first note that other CL methods use either (1) exponential decay (Rebuffi et al., 2017; Kirkpatrick et al., 2017; Mirzadeh et al., 2020) or (2) constant LR. We do not use (1) as it is nyper-parameter sensitive; the decay rate that works for CIFAR 10 decayed the LR too quickly for larger datasets such as CIFARIO0 If the LR is decayed too fast, the LR becomes too small to learn about new classes that are introduced in the future. Thus, with the modification that the LR is reset when a new class is observed. Comparing with the constant LR, obtain slightly better performance for EWC++ and our baseline on CIFARIO, 10, shown in Table 9. Thus, we denote this LR schedule as the exponential with reset and use it in our baseline.
For the LR scheduling,
exponential LR scheduler
A.2 PROOF OF THEOREM 1
We give the proof of Theorem below. Our assumption is that when selecting memory M from set of candidates C, we should select M that optimizing on M maximizes the loss decrease on C. In equations, optimal memory M is
M = arg max
Σ MCC, (M|Sm )MM (R.N)EC
Σ
Es [1x,, 0) - 1(x, v: 0 Volla', v/: 0))
(4)
arg max MMIEm
Σ
Σ
E. [[x, 0) - (x, 0 Volla', f))
(e' (r,y)Ee
Σ
Σ
Es [1x y; 0) - (x, Vol(z' 0))]
(5)
(w' W'EEEM M (a.y)EC
arg min MIC. (a' ( M (R.))EC
Σ
Σ
Es (1 y: 0) - -155, Vol(o' y/: 0)),
(6)
where 0 is the model parameter, I is the loss function, and m is the memory size. Since we perform memory update after every streamed sample, the problem reduces to selecting one sample (7. 7) to
remove when the memory is full. Thus, C M ()). The optimal removed sample (x would be
"8' (x'.8") arg min Eg (Z.
Σ (x.0)EC
1(x y: 6) - (x, y: Vol(z, Ø: A))
A.3 DETAILS ON SAMPLE- -ISE IMPORTANCE BASED MEMORY MANAGEMENT
We describe the details of our sample-wist importance based memory management here. We update H. the estimate of sample-wist wise importance for episodic memory, after every model update. The details are in Alg. I. The runtime complexity of updating the sample-wise importance (M)) where M is the fixed memory size. Because the memory size is kept constant regardless of the data size, the algorithm can scale to larger datasets. Note that for practical efficiency. Alg. can be used after every k model updates instead of every model update described above. With the sample-we wise importance scores, we update the memory everytime a new sample is encountered. The details are in Alg. 2.
14

---

Published as a conference paper at ICLR 2022
Algorithm Uppdaee Sample-wise Importance
Input model fe. memory M, sample-wise importance H. previous loss (prev. indices used for training I. update coefficient X
2: leur [M Σ L(z.)em (z, y; 8)
Obtain memory loss
3: A = 4prev - Leur
D. Obtain memory loss decrease
4: Alpeed = [ ΣiEI Hi
b Memory loss decrease prediction using current H.
5: for ii E EI do
6: Update Hi t- His + A(LI Alpra)
b Update H. for samples used for training
7: end for
8: Update 1prov f leur
9: Output H. 1prev
Algorithm 2 Sample-wise Importance Based Memory Update
1: Input model fo. memory M, memory size m, sample (6. f). per-sample criterion H. previous
loss lprev
2: if [M| < m then
If the memory is not full Append the sample the memory
3: Update M + MU MU(($,0) ({)
4:
MM
5:
else
If the memory is already full Ymax = max, ({ (Fr, vo) EM (MU yr = yHl & Find the most frequent label
6:
7:
Iy = {i](z,, yii E M, Bi = ymax
8:
= arg min:e1van His
Find the sample with the lowest importance
9:
Update lprev + m-
m
m- 12
10:
Update Mi x- (f,
b Replace that sample with the new sample
11: end if
Update 1prev < LMI-1
[ paov
1 (0,0,0)
[.
Zy = {i]( yi) E M, y/1 = Ú + i}
Update Hi < Hgl EiET, Hi
Initialize the importance for the new sample
15: Output M, H. prev
A.4 ADAPTIVE LEARNING RATE SCHEDULER
We describe the adaptive LR schedule from 4.4 in Alg. A.4. We fix the significance level to the commonly used a = 0.05. Our adaptive LR scheduling decreases or increases the LR based on its current value. Thus, the rate in which the LR can change is bounded and sudden changes in LR do not happen.
A.5 DETAILS ON THE ONLINE VERSIONS OF COMPARED CL. METHODS
We implemented online versions of RM (Bang al., 2021), EWC++ (Chaudhry et al., 2018b), BiC (Wu et al., 2019), GDumb (Prabhu et al., 2020), A-GEM (Chaudhry et al., 2019), GSS (Aljundi al., 2019c) and MIR (Aljundi et al., 2019a) by incorporating ER and exponential decay with reset to the methods whenever possible. There is no specified memory management strategy for EWC++ and BiC uses herding selection from iCeRL(Rebuffi et al., 2017). However, herding selection not possible in online since it requires whole task data for calculating class mean, so we attach reservoir memory to both methods instead. EWC++ does not require any other modification.
Additional modification should be applied to bias correction stage of BiC. BiC originally performs bias correction at end of each task, but since evaluation also performed at the middle of task in our setup, we modified the method to perform bias correction whenever the model receives inference query.
In RM, their memory management strategy based on uncertainty is not applicable in online setup, since it requires uncertainty rankings of the whole task samples. Thus, we replace their sampling strategy with balanced random sampling, while keeping their two-stage training scheme. Methods
15

---

Published as a conference paper at ICLR 2022
Algorithm 3 Adaptive Learning Rate Scheduler
Input current LR 17, current base LR 17, loss before applying current LR beffor. current loss leur- LR performance history liggh and H!om)) LR step 7 < 1, history length m, significance level
a
2: lai Inefore leur
D Obtain loss decrease
3: Update Inefore +- laur
4: if >> then
If is higher than base LR Di Append loss decrease in high LR history
5: Update (1igggh) t H(igggh) L {lair
6: if [Hitiggh)|
> then Update H H(thigh) {71thigh1g
7:
8: end if
9: else
is lower than base Append loss decrease in low LR history
10:
Update H!llow) &- H(llow U (aair)
11: if' (H(low)) > m then
Update H(llow) -- Hllow) (M
12:
end if
14:
end if
if [7 and [7/(0w)| m then
If both histories are full
16:
OneSiddddSudenssTTlss (H H(iggh)
b. Perforrm one-sided Student's 's f-test with alternative hypothesis flew >> Mligh pvalue significantly low
if p then
18:
Update 17 8- 2- Update H!loww (hggggh) 0
P Decrease base LR b Reset histories
20:
else if > 1 1 a then
D If pvalue is significantly high
Update 17 + 72
b Increase base
22:
Update (low) (high)) 0,
23:
end 24: end if
25: if > then
> Alternately apply high and low LR (note that y < 1)
26:
Update 17 1=-1-1
27: else
28:
Update 77 -
end if
30:
Output 17, 77, loctore: H(low (ighh
that were converted from offline to online, namely EWC++, BiC, and RM, may have suffered some performance drop due to deviation from their original methods.
A.6 ANALYSIS ON SAMPLE MEMORY SIZR AND NUMBER OF TASKS
We conduct analysis various sample memory sizes (K) and summarize results in Table 4. We observe that CLIB outperforms other CL methods both AxUC and Aaeg no matter the memory size. It interesting note that CLIB with memory size of K 200 outperforms other CL methods with memory size of K 1000 in the AuUC and performs on par in Ageg- Thus, CLIB the only method using memory only training scheme but is the least sensitive memory size. implies that our memory management policy is the most effective, which shows the superiority of our per-sample memory management method.
We additionally conduct analysis on the number of tasks used for the CIFARI00 dataset and summarize the results in Table Note that we use CIFAR100 as CIFARIO has too few classes to divide into longer task sequences. We observe that CLIB outperforms other CL methods in both AAUC and Ageg even when the number tasks increase from 5 to 10 or 25, Moreover, CLIB shows only minor performance drops with increasing number of tasks, indicating that CLIB is capable of achieving high performance the long-run.
Interestingly, we note that BiC shows a slight performance increase when the number of tasks is increased to 10 or 25. We believe this is because BiC separate bias parameters for
16

---

Published as a conference paper at ICLR 2022
K=200
K=500
K=1000
Methods
AAUC
Awe
AAUC
Awg
AAUC
Aay
EWC+
52.06=22.24
54.09:13.57
57.34=2.10
60.33-2.73
60.93=1.02
65.86=22005
BiC
53.00=11.03
14.36111.64
58.38-10.54
61.49-0.68
61.52:12.24
64.8
ER-MIR
41.63+22.43
54.40-13.50
57.28
6..93-3.35
61.18+1.088
66.05-12.29
GDumb
42.54-12.01
43.99-42.28
13.20-+1.93
25.27
66.55-11.10
69.21=#1229
RM
1..24
46.79-13.78
23.00-+1.43
61.52-3.69
1..1311161
72.29-2.117
Baseline-ER
52.11=2.32
34.34:13.34
47.46
60.17=2.96
61.18 1.08
66.05-12.29
CLIB (Ours)
64.67=1.886
66.0611..78
70.26:11.28
73.90=0.22
73.00=1.30
77.87=1115
Table 4: Analysis on various sample memory sizes (K) using CIFARIO. The i-Blurry-500 splits are used. The results are averaged over 3 runs. CLIB outperforms all other CL methods by large margins for all the memory sizes. CLIB uses the given memory budget most effectively, showing the superiority of our per-sample memory management method.
5 Tasks
10 Tasks
25 Tasks
Methods
AAuC
Awe
AAuC
Awg
AAUC
A
EWC++
35.3
38.78:12.3
32.25=11.56
34.85=1.71
27.08
29.46
BiC
33.51+3.04
37.61+3..00
35.29+11007
37.40-1.04
14.6
35.55+1.35
ER-MIR
35.35+1.41
38.28+1.15
33.65+1.51
35.29+1.56
28.16+1.229
30.00-1.17
GDumh
32.84-0.45
34.03++0.89
31,22+00511
32.35+1.73
30.79+0.90
31.87=1.088
RM
8.63+-0.19
33.27+1.59
6.56+0.43
34.93-3.94
1.66+0.11
36.85+1.17
Baseline-FR
35.61+2.08
39.10+22.0
32.46+1.335
34.76+1.34
28.35+1.56
30.45-1.43
CLIB (Ours)
46.6710.79
49.22.10.79
44.61
1.015
43.09
43.6411.08
Table 5: Analysis on the number of tasks on CIFAR100. The i-Blurry-5--00 splits are used. The results are averaged over 3 runs. CLIB outperforms all other CL methods by large margins, even when the task sequences becomes longer. Additionally, CLIB does not suffer severe performance drops as the number of tasks increase, indicating that CLIB is well-suited for long-run CL problem setups as well.
each task, longer task sequences allows more fine-grained bias correction and may actually be favorable for BiC. We also point out that the absolute performance of BiC is still lacking. Thus, it may imply that the bias correction of BiC contributes to strong stability in the stability-llasiiitty trade-off (Chaudhry et al., 201 where forgetting would be more severe. However, as stability is enforced strongly, BiC might lack plasticity sufficiently learn from new data, resulting in low performance overall.
2018a), 8a), which help maintain performance for longer task sequences
We also that EWC+ and Baseline-ER have the same AAUC and Auvg values up to the second decimal places in the 10 tasks setting. This may imply that EWC++ is seldom beneficial over Baseline-ER ER as the added episodic memory negates the need for the regularization in EWC++.
A.7 ADDITIONAL COMPARISONS WITH A-GEM
We present additional comparisons to A-GEM. Note that as A-GEM was designed for the task- incremental setting, it performs very poorly in our i-Blurry setup which is task-free. Notably, it achieves only 4.62 AAUC and 6.94 Aavg on CIFAR100, but other works (Prabhu et al., 2020; Mai et al., 2021) have also reported very poor performance for A-GEM in their studies as well.
A.8 COMPARISONS TO OTHER MEMORY MANAGEMENT CL METHODS
We present additional comparison to CL methods that use different memory management strategy in Table 7. GSS (Aljundi et al., 2019c) is added as an additional comparison while Baseline-ER is used represent the reservoir sampling. CLIB outperforms both methods by large margins in both AMUC and Anug. implying that the sample-wise importance memory management method is better than reservoir GrS-greedy.
17

---

Published as a conference paper at ICLR 2022
CIFARIO
CIFARI00
Methods
AAUC
Aavg
AAUC
Aavg
EWC+
27.3412.10
60.33+2.73
35.35=
38.78+2.32
BiC
08.3
61.49++0.6
33.51+3.04
37.61+3.00
ER-MIR
57.28.12443
61.9313.35
35.35
38.28
A-GEM
39.29-2.88
44.85+t4.70
4.62=#0.3
6.94
GDumb
53.20
55.27+12.69
32.84:10.45
34.03++0.89
RM
23.00+1143
61.52:+3.69
8.63+0.19
33.27=1.599
Baseline-ER
57.46:12.25
60.17:2.99
35.61
39.1012.002
CLIB (Ours)
70.26+1.28
73.20::0.22
46.67:10.79
49.22:10.79
Table 6: Additional comparisons to A-GEM with various online CL methods on the i-Blurry setup for CIFARIO and CIFARI00 are shown. The i-Blurry-50-10 splits used for all the datasets and the results are averaged over runs. A-GEM performs very poorly, especially on CIFAR 100 as was designed for the task-incremental setting whereas i-Blurry setup is task-free. CLIB outperforms all other CL methods by large margins on both the AAUC and the Aaug.
CIFARIO
CIFARI00
Methods
Mem. Management
AAUC
Aavg
AAUC
Aayg
GSS Baseline-ER
GSS-Greedy Reservoir
55.51+3.33
49.27=4.36
30.09+1.38
35.06++1.43
57.46:12..25
60.1711.99
35.61
39.10:12.02
CLIB (Ours)
Sample-wise Importance
7..26:11.28
73.90:10.22
46.67
49.22 100.9
Table 7: Comparisons to other CL methods with different memory management strategies in the i-Blurry setup for CIFAR10 and CIFAR100 are shown. The i-Blurry-50-N splits are used for all the datasets and the results are averaged over runs. CLIB outperforms all other CL methods by large margins both the AaUC and the Aeng implying that the sample-wise importance memory management method is effective.
A.9 ADDITIONAL DISCUSSION OF OTHER METHODS
present additional discussion on compared methods in Table 1.
A.9.1 BASELINE-ER
ER has been reported to be a simple yet strong method for online CL. (Prabhu et al., 2020; Mai al., 2021) We also observed strong performance for Baseline-ER, as other methods such EWC++. BiC, and MIR only show marginal to no improvements. However, by using different memory man- agement and memory usage strategy, CLIB was able outperform Baseline-ER by large margins
A.9.2 EWC++
EWC regularization-batt -based method, which regularizes parameters based on the parameter's 's im- portance measured by the accumulated Fisher Information. Note that EWC calculates importance of parameters to regularize them, while CLIB calculates importance of samples to manage episodic memory, We use online version of EWC, called EWC++ proposed in (Chaudhry et al., 201 8b). EWC++ shows almost no improvement over Baseline-ERs possibly because it was developed as method to prevent forgetting without using episodic memory. Since episodic memory with ER alleviates most the forgetting, EWC+ have reduced effect in preventing forgetting, while its side-effect of reducing intransigence still exists (Chaudhry et al., 2018b).
18

---

Published as a conference paper at ICLR 2022
A.9.3 BIC
BiC uses episodic memory and a distillation loss for training and has an additional bias-correction step to correct the model's bias towards current task's classes. BiC shows strong performance on the disjoint task split. However, it performs poorly for the blurry and the i-Blurry splits, possibly because the bias correction using only two parameters for previous and current tasks cannot correctly capture the bias in the presence of blurry classes.
A.9.4 ER-MIR
Note that ER-MIR performs surprisingly similar to the Baseline-ER for CIFARIO, possibly because they are both based on ER. However, ER-MIR does not scale well to larger datasets potentially because its memory usage scheme is not effective for larger datasets.
To extensively highlight the differences between MIR and CLIB, we first have to point out that sample memory methods can be discussed two aspects: memory management and memory usage. MIR uses reservoir sampling for memory management and improved version of experi- ence replay (ER) for memory usage. MIR improves the memory usage by selecting samples from memory that would suffer the largest loss increase if streamed data used to train the model (called "maximally interfered retrieval' or MIR).
MIR can be viewed as using some form of 'importance weighting', and has some similari- ties with our method in the sense that they both utilize the change loss determine which samples are used for training. However, these two are different as MIR uses importance scores for selecting samples from a constructed memory whereas CLIB uses for constructing the memory. Also, MIR assigns high importance scores to samples that show large increases in individual loss when current streamed data are used for training. In contrast, CLIB assigns high importance scores to samples that cause large decreases in memory's total loss when that sample is used for training. Lastly, MIR's s importance score temporary, as new importance scores are calculated every iteration, while CLIB accumulates the measured importance scores over time to calculate the overall importance scores.
Our method performs better than MIR possibly because sample-wise importance memory considers each samples' importance whereas reservoir sampling does not and memory-only training is not skewed by recently seen samples whereas joint training is. Note that since we are selecting random samples in the memory usage step, a possible future direction would be improving the memory usage done in MIR.
A.9.5 GDUMB
GDumb utilizes a greedy sampler, which stores samples greedily, and dumb learner, which trains new model from scratch using only the memory whenever an inference query is made. While we also train only using the memory, we have completely different memory management method that updates both the model and the memory during training. Because GDumb trains from scratch, its performance degrades in the later phases as it cannot accunulate past knowl- edge. Not only that, because GDumb has to train from scratch for every inference query, it takes more computation than other methods when there are many inference queries for any-time inference.
A.9.6 RAINBOW MEMORY
Rainbow memory uses uncertainty based memory management with a two-stage training method that prolongs most of the training until the end of task. As such, RM performs very poorly AAuC metric as it measures any-time inference performance. Nevertheless, RM performs strongly with respect to the Auog metric which shows the usefulness of proposed A AUC-
19

---

Published as a conference paper at ICLR 2022
CIRART
CIFARIO0
Tinymaggeee
ImageNet
Astr
Fiee
Awe
Fle
Ave
Fiee
Fin
ERC++
57.34=2.10
60.3342.55
26.44:2.55
35.35=1 196
38.78-2.32
32.01:55.28
22 1.15
34.39=1.18
31.65+1.50
24.81
36.21
44.06
BiC
58.36::3.44
61.4920.88
22.50=:4338
33.51=304
37.61=3.00
23 20:5872
22 12,00:0.94
3490:1.07
22.01
27.41
28.38
36.12
ER-MIR
57.26:2.43
61.9383.35
27.10:2.17
35.35:141
38.28=1.15
31.76:4.34
22.00:1.14
24.441.266
34.57&1.39
20.48
20.68
66.96
GDumb
53,20s193
35.2722..9
183023340
32.8:2045
34.03:0.89
22.4223.52
18.720019
18,922045
17.3421.13
14.41
14.21
31.18
RMU 23,00=143
61,52+369
7.11-2.44
863-0.19
33.27-1.59
7.76+1.21
574+0.3
17,04-0.77
9,40:8.40
6.22
26.3
10.71
Basciin-ER
57,46+2.25
60.17:2.90
20.00=2966
35.1=2008
39.10=2.02
31.31=1.96
22.45=1.15
34.54=1.26
33.02+1.00
25.16
26.50
44.22
CLIB (Ous)
7.236:1228
73.90.0.22
23.882268
46.67.0.79
49.23:0.79
23.93.00.66
23.57.0.68
25.05:0.52
23.3421.96
28.16
28.88
38.66
Table 8: Additional comparisons including the Flat measure of various online CL methods on the i-Blurry setup for CIFARIO, CIFARIO0, IinyImageNet and ImageNet are shown. The i- i-Blurry-50- 10 splits are used for all the datasets and the results are averaged over runs except ImageNet. Ours outperforms all other CL methods by large margins on both the Auc and the Aang- While CLIB does not explicitly handle forgetting, shows similar Fiss with most methods. The best result for each of the metrics is shown in bold.
1.0
CLIB(Ours)
MIR
ER
GDumb
EWC
RM
BIC
0.8
Accuracy
0.6
0.4
0.0
10000
20000
30000
40000
50000
# samples
Figure 5: Accuracy-to-f of samples curve for the blurry setup (no new classes are encountered after the first task). We can see that there is no performance drop for all methods because no new classes are encountered after the first task.
A.10 ADDITIONAL RESULTS WITH THE FLAST MEASURE
We report the result of the forgetting measure (Chaudhry et al., 2018b) here. As in the i-Blurry setup it is not clear which classes belong to each task, we calculate the forgetting class-wise. Note that while forgetting is a useful metric for analyzing stability-plasstiiiy of the method, lower forgetting does not necessarily mean that a CL method is better. For example, if method do not train with the new task at all, its forgetting will be
Also, we do not propose measured with the best accuracy of each class, and best accuracy usually occur at the end of each task. Thus, measuring the best accuracy among all inferences would not be much different from best accuracy among the inferences at the end of each task.
forgetting measure for anytime inference. It is because forgetting is
Surprisingly, CLIB shows similar Fast values to most other methods despite not explicitly designed to handle catastrophic forgetting. We believe that the episodic memory used in our memory only training reduces catastrophic forgetting of our method to be similar to other methods.
20

---

Published as a conference paper at ICLR 2022
Methods
LR Schedule
AAuC
Aavg
Constant Exp w/ Reset
56.73=11.73
58.18113.41
Baseline-ER
57.46=22.25
60.17422.96
Constant Exp w/ Reset
56.54:1..57
57.92.13.26
EWC++
57.34=2.10
60.33+2.73
Table 9: Comparison between exponential with reset schedule and constant LR on CIFARIO are shown. It shows that our baseline LR schedule, exponential with reset, is reasonable. It shows better performance than constant LR, especially in Aavy metric.
Interestingly, RM shows the best Fast results all datasets. We believe because of RM's two-stage training scheme, which greatly reduces the effect of the streamed data during training. This prevents dominant classes in the task from achieving very high accuracy. Since forgetting is calculated as the difference between the best accuracy and the last accuracy for each class, the forgetting is reduced as the best accuracy itself has been lowered.
Note that GDumb is even less affected by streamed data as it does not train with streamed data at all. However, GDumb does not accumulate learned knowledge, leading to decreasing accuracy as number of samples per class in the memory decreases. Thus, while GDumb scores relatively good forgetting considering the decreasing accuracy trend, ultimately scores worse forgetting than RMM
A.11 ADDITIONAL DISCUSSION ON POTENTIAL PERFORMANCE DROP LATER PHASES
We discuss the potential cause for the 'performance drop' of CLIB in the i-Blurry setup as is shown Fig. 4. We argue that the 'performance drop' is attributed the number of encountered classes increasing in i-Blurry setups at later phases. To see what would happen when the number of en- countered classes do not increase at later phases, we show the accuracy-to-f of samples curve for the blurry setup results in Table 2 for N = O in Fig. 5, where no new classes are encountered after the first task. There is no apparent performance drop at later phases for all methods, which supports our argument that the drop due to more and more classes being encountered at later phases.
A. 12 PERFORMANCE OF EXPONENTIAL WITH RESET LR SCHEDULE
We show brief results for the LR schedule used in our baseline in Table 9. We compare the constant LR with the exponential with reset used in our baseline. The exponential with reset is better than the constant LR, which is why we used it in our baseline.
21


Reviews:
[{"review_id": "faXxUIbrMZ0", "reviewer": "Reviewer_QQWa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has excellent plots for the new problem setup. However, without further clarification of the following concepts/comments, the significance of the new setup could be weak.\n* **The paper claimed the new setup is both “task-free” and “class-incremental.”** In my opinion, the two setups are not compatible with each other. Specifically, they are different at the output layer in supervised learning. \n* **“Any-time inference” may require more rigorous justifications; so as to the new metric.** Only one baseline demonstrated the incapability of making any-time inference in Table 1. The significance of any-time inference needs more evidence. \n* **Thm. 1 may suffer from catastrophic forgetting.** Thm. 1 can be seen as a performance-driven memory management strategy. However, it has nothing to do with catastrophic forgetting. In Fig. 4, the proposed method has a projection of performance drop in the later phase. In the long run, it is hard to tell whether the proposed method will outperform the baselines. This performance drop could be attributed to the catastrophic forgetting of Thm. 1.\n* **The paper doesn’t discuss the algorithm complexity**, which could be O(M) in general, where M is the memory size.\n", "summary_of_the_paper": "The paper proposes a new problem setup in continual learning. As the title suggests, the paper focuses on online, task-free, class incremental, task blurry learning with any-time inference. The authors also came up with new baselines and importance-based memory management. They empirically tested their methods in the proposed problem setup.", "summary_of_the_review": "There are unsolved concerns about the concepts/comments in the current version. Without addressing them, the proposed problem setup and method could not be that significant for continual learning. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WSDsev-yKpv", "reviewer": "Reviewer_Vfw2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper, I believe, is the memory update method, which (as stated above) updates the memory when new samples arrive in a way that preserves the most \"useful\" samples (wrt to their effect on the loss) while promoting class balance. The authors also train exclusively on samples drawn from the memory, rather than on batches split between memory and newly arrived samples, which seems to be advantageous. And the authors use a learning-rate scheduling method that also gives some advantages. The memory management method is novel, as far as I'm aware, and the overall method performs very well. Evaluation is carried out in the class incremental, online, task-free setting, which is a good decision in my opinion. So many other methods fall short under these conditions.  So I think the paper deserves to be accepted.\n\nThe authors make a big deal out of their \"blurry\" sampling method for evaluation. I think it's a good approach, but isn't really a big contribution in its own right.\n\nPresumably, following memory update, the batch that is trained on is sampled uniformly from the memory. Is that right?\n\nSpace permitting, it would be nice to see a more detailed comparison with related experience replay methods. In particular, Aljundi et al's MIR method also uses a kind of importance weighting. It would be good to see the differences spelled out, and to see some sort of explanation for why the present method performs better.", "summary_of_the_paper": "The paper presents an experience replay method for continual learning (CL) whose main innovation is a memory-management algorithm that updates the memory when new samples arrive in a way that preserves the most \"useful\" samples (wrt to their effect on the loss) while promoting class balance. The method is evaluated on the trickiest CL setting - class incremental, online, and task-free - where it outperforms baselines on CIFAR-10 and -100 datasets.", "summary_of_the_review": "A nice paper describing an effective and somewhat novel replay-based CL method, evauated in a demanding setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GvO_gSxFg1n", "reviewer": "Reviewer_F4rS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\n- The proposed benchmark protocol, “i-Blurry”, is reasonable and interesting. It is important to analyse the results when some of the classes are not disjoint in continual learning tasks. \n\n- The authors provided extensive experimental results.\n\n- The proposed method is technically sound and proven to be effective by the empirical results.\n\n- The paper is well-organized and easy to follow.\n\n&nbsp;\n\n### Weaknesses\n\n- ***The technical contributions of this paper are somewhat weak.*** In this paper, the authors mainly propose three components in the method section: “sample importance memory”, “memory-only training”, and “adaptive LR scheduling”. For the first component, the main idea is to find the optimal exemplars that can minimise the total loss so that it could be regarded as a simplified version of “mnemonics” (Liu et al., 2020). For the second component, it can be summarised as “sample importance memory”+GDumb (Prabhu et al. 2020). For the third component, adaptive LR can benefit all classification models. The authors don’t explain why it is important in continual learning tasks. \n\n-  ***This paper is not self-contained.*** Alg.3 and Section A.3 are not some additional information and results. I cannot understand the proposed method without reading Alg.3 and Section A.3. Thus the authors should definitely include these parts in the main paper.\n\n- ***The authors only provide experiment results on small-scale datasets.*** Most continual learning papers, such as iCaRL (Rebuffi et al., 2017) and BiC (Wu et al., 2019), provide the results on large-scale datasets (e.g., ImageNet-1k). As the authors trying to establish a new benchmark protocol, it is important to provide the results on large-scale datasets.\n\n- ***An ablation study on the number of tasks should be provided.*** In many continual learning papers (iCaRL (Rebuffi et al., 2017), BiC (Wu et al., 2019), etc.), the number of tasks will significantly influence the continual learning performance. However, I don’t find an ablation study on the number of tasks in this paper. I don’t even see an explanation about how the authors choose that hyperparameter. ", "summary_of_the_paper": "In this paper, the authors proposed a new benchmark protocol (i-Blurry) for continual learning. In this benchmark protocol, the class distribution is class incremental and has blurry task boundaries, and the training is online. They also propose a new method, CILB. This method contains three important components: “sample importance memory”, “memory-only training”, and “adaptive LR scheduling.” Extensive experimental results are provided to show the effectiveness of the proposed method. ", "summary_of_the_review": "Overall, I think this is an interesting paper. The authors design a new benchmark for continual learning and propose a simple yet effective method. My primary concern is that “how likely is it for following researchers to refer to this benchmark?” As there are already many different benchmark protocols in continual learning, I think the following researchers will tend to choose the benchmark protocol already used in many popular papers. Nevertheless, I still think the paper might be useful for the continual learning community. \n\nMy rating is “borderline accept”, and I will consider upgrading my rating if the authors successfully address my questions.\n\n&nbsp;\n\n### Post-rebuttal update\nThe authors addressed most of my concerns in the rebuttal. They also provided the results I asked for in the revision (e.g., the results on ImageNet-1k). I think I tend to accept this submission. So I upgrade my rating to eight.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5f_09rmohap", "reviewer": "Reviewer_6BAu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The proposed method works well on a various instantiations of the i-Blurry-N-M setup. The proposed algorithms is better in the blurry (N=0) setting and performs comparably well in disjoint setting (N=100) as well. \n\n2. The ablation study highlights the main component in the proposed approach, the memory management scheme that relies on sample importance, that results in the highest benefits. \n\n3. Lots of supporting experiments are provided in the appendix that further corroborate most claims. \n\n4. Good organization of related work and great figures.  \n\n### Weaknesses:\n\n1. I am unsure if the proposed approach of obtaining sample importance is entirely novel. A discussion regarding this is necessary as it is likely that similar ideas present (beyond CL for instance) in literature. \n\n2. A natural language description of Theorem 1 is missing. \n\n3. I felt that more insights into the calculation of importance are required in the main text. A note about the efficiency/runtime needs to be specified. \n \n**Personally**, I think dedicating a portion of the main-text (Section 4) in discussing these would probably be more beneficial than describing memory only training and adaptive lr, as they seemingly play only a minor role (as indicated by the ablations).\n\n4. No information as to why TinyImagenet is challenging for all methods is provided. There is also no justification provided for choosing TinyImagenet over Imagenet (RM, BiC and GDumb all show their results on Imagenet). \n\n5. Since the i-Blurry setting is an **online** one, the choice of hyperparameters are likely to affect the results greatly. As such hyperparameter choices (updates per sample, memory size and batchsize) need to justified. Specifically, were the hyperparameters chosen consistent with previous work? \n\n6. For a fixed $M$, is it reasonable to expect that performance of all algorithms would increase (in general) with increasing N? This can be used to justify choosing $N$ to be from $0, 50, 100$ (would also be helpful reference for future works). \n\n**Suggestion** (not a weakness): The paper can also report the performance on some (soft) upper bounds. For instance, what if the task was not restricted to be online. What is the performance when the entire training data is available at once (non CL scenario)?\n\n\n##### There are some typos:\n\n* Section 4.2 (page 5): Arguing that considering ... lead to training efficacy **decrease**.\n\n* Section 4.1 (page 5): For the LR scheduling, other **CL methods use either (1) exponential decay** or (2) constant LR ...\n\n* Last line on page 4 is missing a period (full-stop)\n    \n* Also, Table 3 $A_{avg}$ on CIFAR-100 has the same numbers for the last two rows, which might be a typo.", "summary_of_the_paper": "The paper proposes a more realistic setting (called *i-Blurry*) for continual learning (CL) that generalizes the *blurry* and *disjoint*  settings proposed in prior work. The disjoint setting assumes that there is no class that appears in multiple tasks and the blurry setting assumes that no new classes are seen after the first task. In the *i-Blurry* setting, one can have overlapping classes across tasks as well as new classes appearing in each task. This setting is also **online** and thus the paper is interested in continuous model evaluation i.e., any-time inference too. It proposes a metric which calculates the area under the accuracy curve during training for the same. \n\nFor the new configuration, the paper proposes a strong baseline and a new algorithm called CLIB. CLIB is a memory-based CL method that refines its memory by throwing out *least important* samples and updating it with more important ones. The sample importance is calculated as the expected decrease in training loss when the sample is used for training. CLIB is also equipped with a data-driven adaptive learning rate scheduling scheme which provides some additional performance benefits. \n\nThe results showcase that CLIB is able to outperform other online CL methods and the baseline by large margins for various instantiations of their i-Blurry setting, *including the disjoint and Blurry setting*, on the CIFAR-10, CIFAR-100 and TinyImagenet datasets. An ablation study shows the main benefits of the proposed method come from the sample importance based memory scheme. ", "summary_of_the_review": "### Basis for the scores\n\n1. The experiments justify most of the information present in the paper. Some additional information regarding the sample importance based memory management scheme is required. Also, some decisions regarding hyperparameters need to justified, given the *i-Blurry* setting is online and the main motivation of the paper is to propose and solve a more practical CL setting. \n\n2. I'm not sure about the novelty of the sample-importance based memory management scheme and a discussion regarding prior work related to sample-importance is required. The proposed setting on *i-Blurry* is marginally novel as it is a generalization of the disjoint and blurry configurations. Any-time inference and $A_{AUC}$ are  likely novel.\n\n3. Empirical results shown cannot be easily (and extensively) compared to some prior work as there isn't a strict adherence to prior dataset and hyperparameter choices. Some justification for the decisions is required. \n\n4. Insights into the proposed CLIB method along with a nuanced discussion regarding the differences between the algorithms is lacking. There is no description regarding efficiency and scalability.\n\n## Update\nThe authors clarify almost all of my questions pertaining to the paper. The paper now contains experiments on Imagenet-1k with additional insights into the algorithm and its efficiency. As such, I've improved my scores. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new continual-learning setting with a few distinguishing features: 1) the task boundaries are blurry (in other words, past task samples can reappear); 2) training is online; and 3) evaluation using online accuracy (instead of average accuracy). The authors also propose a useful method for this scenario and benchmark it using four different datasets.

The first round of review pointed to two main limitations of the manuscript. 
+ The authors only provided small-scale experiments. The reviewers argued that for the setup and method to have an impact having good results using larger-scale data would go a long way. 
+ Whether “task-free” and “class-incremental” were compatible. 

For the former, the authors were very reactive and provided results using a standard "ImageNet for CL" dataset. 

For the latter, I must thank the authors and also the reviewers for discussing this thoroughly. In the end, my understanding is that there was a reconciliation that both were in fact compatible, but the reviewer suggested that this be discussed very clearly by the authors. I second this suggestion. The CL field given its many slightly different settings might be partly to blame here (reviewer Vfw2 made a similar comment, and I also thank them for playing a role in resolving the issue).


A few additional thoughts: 
+ I believe that more general setups in CL are worthwhile even in the absence of any immediate applications. This is especially true since some of the standard CL assumptions do not seem to be well motivated. However, I find that claiming that something is more realistic requires grounding (e.g. a set of examples from the "real world" or a specific domain/setting). I know the authors backed some of their claims with references, but different real-world problems will come with different limitations and I would be hesitant to use phrases such as "most real-world" settings without thorough justification.
+ While different from the core of your work, I believe the framework proposed in this other recent paper has similar goals (although the setup allows pre-training and is not online). Might be worth knowing about it in case you do not: 
Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning, NeurIPS 2020
https://papers.nips.cc/paper/2020/file/c0a271bc0ecb776a094786474322cb82-Paper.pdf

All in all, this is a good contribution that proposes an interesting and rich setting along with a good baseline method for it. I strongly encourage the authors to follow through on their promise to provide the community with code, dataset splits, Kaggle leaderboard, etc., as a way to maximize the impact of their work.
Decision: Accept (Poster)