Year: 2021

Abstract:
Despite rapid advances in continual learning, a large body of research is devoted to improving performance in the existing setups. While a handful of work do propose new continual learning setups, they still lack practicality in certain aspects. For better practicality, we first propose a novel continual learning setup that is online, task-free, class-incremental, of blurry task boundaries and subject to inference queries at any moment. We additionally propose a new metric to better measure the performance of the continual learning methods subject to inference queries at any moment. To address the challenging setup and evaluation protocol, we propose an effective method that employs a new memory management scheme and novel learning techniques. Our empirical validation demonstrates that the proposed method outperforms prior arts by large margins.

heading: 1 INTRODUCTION
text: Continual learning (CL) is a learning scenario where a model learns from a continuous and online stream of data and is regarded as a more realistic and practical learning setup than offline learning on a fixed dataset (He et al., 2020). However, many CL methods still focus on the offline setup (Kirkpatrick et al., 2017; Rebuffi et al., 2017; Saha et al., 2021) instead of the more realistic online setup. These methods assume access to a large storage, storing the entire data of the current task and iterating on it multiple times. On the other hand, we are interested extensively in the more realistic online setup where only a small memory is allowed as storage. Meanwhile, even for the online CL methods, we argue they have room for more practical and realistic improvements concerning multiple crucial aspects. The aspects include the class distributions such as the disjoint (Rebuffi et al., 2017) or the blurry (Aljundi et al., 2019c) splits and the evaluation metric that focuses only on the task accuracy such as average task accuracy (Aavg).
The two main assumptions on the class distributions in existing CL setups, i.e., the disjoint and blurry splits, are less realistic for the following reasons. The disjoint split assumes no classes overlap over different tasks; already observed classes will never appear again. This assumption is not plausible because already observed classes can still appear later on in real-world scenarios (see Fig. 2 of (Bang et al., 2021)). On the other hand, in the blurry split (Aljundi et al., 2019c) no new classes appear after the first task even though the split assumes overlapping classes over tasks. This is also not plausible as observing new classes is common in real-world scenarios.
The typical evaluation metric such as Aavg in which the accuracy is measured only at the task transition is also less realistic. It implicitly assumes that no inference queries occur in the middle of a task. However, in real-world scenarios, inference queries can occur at any-time. Moreover, there is no explicit task transition boundary in most real-world scenarios. Thus, it is desirable for CL models to provide good inference results at any time. To accurately evaluate whether a CL model is effective at such ‘any-time’ inference, we need a new metric for CL models.
In order to address the issues of the current CL setups, we propose a new CL setup that is more realistic and practical by considering the following criteria: First, the class distribution is comprised of the advantages from both blurry and disjoint. That is, we assume that the model continuously encounters new classes as tasks continue, i.e., class-incremental and that classes overlap across tasks, i.e., blurry task boundaries, while not suffering from the restrictions of blurry and disjoint.
Second, the model is evaluated throughout training and inference such that it can be evaluated for any-time inference. We call this new continual learning setup ‘i-Blurry’.
For the i-Blurry setup, we first propose a plausible baseline that employs experience replay (ER) with reservoir sampling and a learning rate scheduling tuned for the online and task-free CL setting. While existing online CL methods are applicable to the i-Blurry setup, they perform only marginally better than our baseline or often worse.
To better handle the i-Blurry setup, we propose a novel continual learning method, which improves the baseline in three aspects. We design a new memory management scheme to discard samples using a per-sample importance score that reflects how useful a sample is for training. We then propose to draw training samples only from the memory instead of drawing them from both memory and the online stream as is done in ER. Finally, we propose a new learning rate scheduling to adaptively decide whether to increase or decrease the learning rate based on the loss trajectory, i.e. a data-driven manner. To evaluate the algorithms in the new setup, we evaluate methods by conventional metrics, and further define a new metric called ‘area under the curve of accuracy’ (AAUC) which measures the model’s accuracy throughout training.
We summarize our contributions as follows: • Proposing a new CL setup called i-Blurry, which addresses a more realistic setting that is online,
task-free, class-incremental, of blurry task boundaries, and subject to any-time inference. • Proposing a novel online and task-free CL method by a new memory management, memory usage,
and learning rate scheduling strategy. • Outperforming existing CL models by large margins on multiple datasets and settings. • Proposing a new metric to better measure a CL model’s capability for the desirable any-time
inference.

heading: 2 RELATED WORK
text: Continual learning setups. There are many CL setups that have been proposed to reflect the realworld scenario of training a learning model from a stream of data (Prabhu et al., 2020). We categorize them in the following aspects for brevity.
First, we categorize them into (1) task-incremental (task-IL) and (2) class-incremental learning (class-IL), depending on whether the task-ID is given at test time. Task-IL, also called multi-head setup, assumes that task-ID is given at test time (Lopez-Paz & Ranzato, 2017; Aljundi et al., 2018; Chaudhry et al., 2019). In contrast, in class-IL, or single-head setup, task-ID is not given at test time and has to be inferred (Rebuffi et al., 2017; Wu et al., 2019; Aljundi et al., 2019a). Class-IL is more challenging than task-IL, but is also more realistic since task-ID will not likely be given in the real-world scenario (Prabhu et al., 2020). Most CL works assume that task ID is provided at training time, allowing CL methods to utilize the task ID to save model parameters at task boundaries (Kirkpatrick et al., 2017; Chaudhry et al., 2018b) for later use. However, this assumption is impractical (Lee et al., 2019) since real-world data usually do not have clear task boundaries. To address this issue, a task-free setup (Aljundi et al., 2019b), where task-ID at training is not available, has been proposed. We focus extensively on the task-free setup as it is challenging and being actively investigated recently (Kim et al., 2020; Lee et al., 2019; Aljundi et al., 2019c).
We now categorize CL setups into disjoint and blurry setup by how the data split is configured. In the disjoint task setup, each task consists of a set of classes disjoint from all other tasks. But the disjoint setup is less realistic as the classes in the real-world can appear at any time not only in a disjoint manner. Recently, to make the setup more realistic, a blurry task setup has been proposed and investigated (Aljundi et al., 2019c; Prabhu et al., 2020; Bang et al., 2021), where 100 −M% of the sampels are from the dominant class of the task and M% of the samples are from all classes, where M is the blurry level (Aljundi et al., 2019c). However, the blurry setup assumes no class is added in new tasks, i.e., not class-incremental, which makes the setup still not quite realistic.
Finally, depending on how many samples are streamed at a time, we categorize CL setups into online (Rolnick et al., 2018; Aljundi et al., 2019a; Chaudhry et al., 2019) and offline (Wu et al., 2019; Rebuffi et al., 2017; Chaudhry et al., 2018b; Castro et al., 2018). In the offline setup, all data from the current task can be used an unlimited number of times. This is impractical since it requires additional memory of size equal to the current task’s data. For the online setup, there are many
notions of ‘online’ that differs in each literature. Prabhu et al. (2020); Bang et al. (2021) refer online to a setup using each streamed sample only once to train a model while Aljundi et al. (2019c;a) refer online to a setup where only one or a few samples are streamed at a time. We follow the latter as the former allows storing the whole task’s data, which is similar to offline and more unrealistic than using the sample more than a few times.
Incorporating the pros and cons of existing setups, we propose a novel CL setup that is online, task-free, class-incremental, of blurry task boundaries, and subject to any-time inference as the most realistic setup for continual learning.
Continual learning methods. Given neural networks would suffer from catastrophic forgetting (McCloskey & Neal, 1989; Ratcliff, 1990), the online nature of streaming data in continual learning generally aggravates the issue. To alleviate the forgetting, there are various proposals to store the previous task information; (1) regularization, (2) replay, and (3) parameter isolation.
(1) Regularization methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017b; Ebrahimi et al., 2020) store previous task information in the form of model priors and use it for regularizing the neural network currently being trained. (2) Replay methods store a subset of the samples from the previous tasks in an episodic memory (Rebuffi et al., 2017; Castro et al., 2018; Chaudhry et al., 2019; Wu et al., 2019) or keep a generative model that is trained to generate previous task samples (Shin et al., 2017; Wu et al., 2018; Hu et al., 2019; Cong et al., 2020). The sampled or generated examplars are replayed on future tasks and used for distillation, constrained training, or joint training. (3) Parameter isolation methods augment the networks (Rusu et al., 2016; Lee et al., 2017a; Aljundi et al., 2017) or decompose the network into subnetworks for each task (Mallya & Lazebnik, 2018; Cheung et al., 2019; Yoon et al., 2020).
Since (1), (2), and (3) all utilize different ways of storing information that incurs parameter storage costs, episodic memory requirement and increase in network size respectively, a fair comparison among the methods is not straighforward. We mostly compare our method with episodic memorybased methods (Wu et al., 2019; Aljundi et al., 2019a; Bang et al., 2021), as they perform the best in various CL setups but also with methods that use both regularization and episodic memory (Chaudhry et al., 2018b; Wu et al., 2019).
Online continual learning. Despite being more realistic (Losing et al., 2018; He et al., 2020), online CL setups have not been popular (Prabhu et al., 2020) due to the difficulty and subtle differences in the setups in the published literature. ER (Rolnick et al., 2018) is a simple yet strong episodic memory-based online CL method. It employs reservoir sampling for memory management and jointly trains a model with half of the batch sampled from memory. Many online CL methods are based on ER (Aljundi et al., 2019c;a). GSS (Aljundi et al., 2019c) selects samples using a score based on cosine similarity of gradients. MIR (Aljundi et al., 2019a) retrieves maximally interfering samples from memory to use for training.
Different from ER, A-GEM (Chaudhry et al., 2019) uses the memory to enforce constraints on the loss trajectory of the stored samples. GDumb (Prabhu et al., 2020) only updates the memory during training phase and trains from scratch at the test time only using the memory.
Unlike these methods, recently proposed RM (Bang et al., 2021) uses an uncertainty-based memory sampling and two-stage training scheme where the model is trained for one epoch on the streamed samples and trains extensively only using the memory at the end of each task, effectively delaying most of the learning to the end of the tasks. Note that the uncertainty-based memory sampling cannot be implemented in the online CL setup and the two-stage training performs particularly worse in our i-Blurry setup. Our method outperforms all other online CL method introduced in this section while strictly adhering to the online and task-free restrictions.

heading: 3 PROPOSED CONTINUAL LEARNING SETUP: I-BLURRY
text: For a more realistic and practical CL setup, considering real-world scenarios, we strictly adhere to the online and task-free CL setup (Lee et al., 2019; Losing et al., 2018). Specifically, we propose a novel CL setup (named as i-Blurry), with two characteristics: 1) class distribution being class incremental and having blurry task boundaries and 2) allowing for any-time inference.
i-Blurry-N-M Split. We partition the classes into groups where N% of the classes are for disjoint and the rest of 100−N% of the classes are used for BlurryM sampling (Aljundi et al., 2019c), where M is the blurry level. Once we determine the partition, we draw samples from the partitioned groups. We call the resulting sequence of tasks as i-Blurry-N -M split. The i-Blurry-N -M splits feature both class-incremental and blurry task boundaries. Note that the i-Blurry-N -M splits generalize previous CL setups. For instance, N = 100 is the disjoint split as there are no blurry classes. N = 0 is the BlurryM split (Aljundi et al., 2019c) as there are no disjoint classes. M = 0 is the disjoint split as the blurry level is M = 0 (Bang et al., 2021). We use multiple i-Blurry-N -M splits for reliable empirical validations and share the splits for reproducible research. Fig. 1 illustrates the i-Blurry-N -M split.
A New Metric: Area Under the Curve of Accuracy (AAUC). Average accuracy (i.e., Aavg = 1 T ∑T i=1Ai where Ai is the accuracy at the end of the i
th task) is one of the widely used measures in continual learning. But Aavg only tells us how good a CL model is at the few discrete moments of task transitions (5− 10 times for most CL setups) when the model could be queried at any time. Thus, a CL method could be poor at any-time inference but the Aavg may be insufficient to deduce that conclusion due to its temporal sparsity of measurement. For example, Fig. 2 compares the online version of RM (Bang et al., 2021), which conducts most of the training by iterating over the memory at the end of a task, with our method. RM is shown as it is a very recent method that performs well on Aavg but particularly poor at any-time inference. Only evaluating with Aavg might give the false sense that the difference between the two methods is not severe. However, the accuracy-to-{# of samples} curve reveals that our method shows much more consistently high accuracy during training, implying that our method is more suitable for any-time inference than RM. To alleviate the limitations of Aavg, we shorten the accuracy measuring frequency to after every ∆n samples are observed instead of at discrete task transitions. The new metric is equivalent to the area under the curve (AUC) of the accuracy-to-{# of samples} curve for CL methods when ∆n = 1. We
?
call it area under the curve of accuracy (AAUC):
AAUC = k∑ i=1 f(i ·∆n) ·∆n, (1)
where the step size ∆n is the number of samples observed between inference queries and f(·) is the curve in the accuracy-to-{# of samples} plot. High AAUC corresponds to a CL method that consistently maintains high accuracy throughout training.
The large difference in AAUC between the two methods in Fig. 4 implies that delaying strategies like the two-stage training scheme are not effective for any-time inference, a conclusion which is harder to deduce with just Aavg .

heading: 4 METHOD

heading: 4.1 A BASELINE FOR I-BLURRY SETUP
text: To address the realistic i-Blurry setup, we establish a baseline for the challenging online and task-free i-Blurry setup. For the memory management policy, we use reservoir sampling (Vitter, 1985) and for memory usage, we we use experience replay (ER). For the LR scheduling, we use an exponential LR schedule but reset the LR when a new class is encountered. Please see Sec. A.1 for details.
Note that the above baseline still has room to improve. The reservoir sampling does not consider whether one sample could be more useful for training than the others. ER uses samples from the stream directly, which can skew the training of CL models to recently observed samples. While the exponential with reset does increase the LR periodically, the sudden changes may disrupt the CL model. Thus, we discuss how we can improve the baseline in the following sections. The final method with all the improvements is illustrated in Fig. 3.

heading: 4.2 SAMPLE-WISE IMPORTANCE BASED MEMORY MANAGEMENT
text: In reservoir sampling, the samples are removed from the memory at random. Inspired by works on sample importance (Kloek & Van Dijk, 1978; LeCun et al., 1990; Chang et al., 2017; Katharopoulos & Fleuret, 2018; Csiba & Richtárik, 2018), we propose a novel sampling strategy specific for CL that removes samples from the memory based on sample-wise importance as following Theorem 1.
Theorem 1. Let C = M ∪ {(xnew, ynew)}, M be a memory from the previous time step, (xnew, ynew) be the newly encountered sample, l(·) be the loss and θ be the model parameters. Assuming that the model trained with the optimal memory,M∗ will induce maximal loss decrease on C, the optimal memory is given byM∗ = C \ {(x̄, ȳ)} with
(x̄, ȳ) = arg min (xi,yi)∈C
Eθ  ∑ (x,y)∈C l(x, y; θ)− l (x, y; θ −∇θl(xi, yi; θ))  . (2) Proof. Please see Sec. A.2 for the proof.
Theorem 1 states that when a new sample is appended to the memory and a sample has to be discarded, we should discard the sample that incurs the least loss decrease i.e., least useful for training. We solve Eq. 2 by keeping track of the sample-wise importanceHi
Hi = Eθ  ∑ (x,y)∈C l(x, y; θ)− l (x, y; θ −∇θl(xi, yi; θ))  . (3) Intuitively, H is the expected loss decrease when the associated sample is used for training. We update H associated with the samples used for training after every training iteration. Specifically, the model is trained with a random batch from the memory. Then, for all samples in C, we measure the loss difference before and after training with the batch. If the loss decreases, the importance scores of the samples in the batch increase and vice versa. Note that because the importance score of a sample is relative to that of other samples in the memory, when the importance scores of the samples in the batch decrease, the scores of the samples not in the batch increase comparatively. Thus, two types of samples have high importance scores; 1) samples that were in the batch when the loss decreased and 2) samples that were not in the batch when the loss increased.
The expectation in Eq. 3 is taken over θ’s optimization trajectory i.e., over time. For computational efficiency, we use the exponential moving average as empirical estimates instead. The empirical estimates are calculated by the discounted sum of the differences between the actual loss decrease and the predicted loss decrease (see Alg. 1). The memory is updated whenever a new sample is encountered, and the full process is given in Alg. 2. The memory management strategy significantly outperforms reservoir sampling, especially with memory only training.

heading: 4.3 MEMORY ONLY TRAINING
text: ER uses joint training where half of the training batch is obtained from the online stream and the other half from memory. However, we argue that using the streamed samples directly will skew the training to favor the recent samples more. Thus, we propose to use samples only from the memory for training, without using the streamed samples. The memory works as a distribution stabilizer for streamed samples through the memory update process (see Sec. 4.2), and samples are used for training only with the memory. We observe that the memory only training improves the performance despite its simple nature. Note that this is different from Prabhu et al. (2020) as we train with the memory during the online stream but Prabhu et al. (2020) does not.

heading: 4.4 ADAPTIVE LEARNING RATE SCHEDULING
text: The exponential with reset scheduler resets the LR to the initial value when a new class is encountered. As the reset occurs regardless of the current LR value, it could result in a large change in LR value. We argue that such abrupt changes may harm the knowledge learned from previous samples.
Instead, we propose a new data-driven LR scheduling scheme that adaptively changes the LR in a data-driven manner based on how good the LR is for optimizing over the memory.
Specifically, from the current base LR η̄ and step size γ < 1, we try both a high LR η̄/γ and a low LR η̄ · γ for training. For each LR, we keep a history of length m that tracks the loss decrease when each LR is used. When both histories are full, we perform a Student’s t-test to decide which LR is better. If one LR is better than the other, up to some significance level α = 0.05, the base LR is set to the better LR, i.e., η̄γ or η̄ · γ.
We depict this scheme in Alg. 3 in Sec. A.4. The adaptive LR is important for CL methods because the data distribution changes over time and adapting to the current training data is more useful.
With all the proposed components, we call our method Continual Learning for i-Blurry or (CLIB).

heading: 5 EXPERIMENTS
text: Experimental Setup. We use the CIFAR10, CIFAR100, TinyImageNet, and ImageNet datasets for empirical validations. We use the i-Blurry setup with N = 50 and M = 10 (i-Blurry-50-10) for our experiments unless otherwise stated. All results are averaged over 3 independent runs except
ImageNet (Wu et al., 2019; Bang et al., 2021; Prabhu et al., 2020). For metrics, we use the average accuracy (Aavg) and the proposed AAUC (see Sec. 3). Additional discussion with other metrics such as the forgetting measure (Flast) can be found in Sec. A.10.
Implementation Details. For all methods, we fix the batch size and the number of updates per streamed samples observed when possible. For CIFAR10, we use a batch size of 16 and 1 updates per streamed sample. When using ER, this translates to 8 updates using the same streamed batch, since each batch contains 8 streamed samples. For CIFAR100, we use a batch size of 16 and 3 updates per streamed sample. For TinyImageNet, we use a batch size of 32 and 3 updates per streamed sample. For ImageNet, we use a batch size of 256 and 1 update per every 4 streamed samples. We use ResNet-18 as the model for CIFAR10 and ResNet-34 for CIFAR100 and TinyImageNet. For all methods, we apply AutoAugment (Cubuk et al., 2019) and CutMix (Yun et al., 2019) following RM (Bang et al., 2021). For memory size, we use 500, 2000, 4000, 20000 for CIFAR10, CIFAR100, TinyImageNet, ImageNet, respectively. We use 5 tasks for CIFAR10, CIFAR100, and TinyImageNet and 10 tasks for ImageNet. We follow prior works (Bang et al., 2021; Prabhu et al., 2020) for choosing number of updates per sample, memory size, batch size, and number of tasks. Additional analysis on the sample memory size and number of tasks can be found in Sec. A.6. Adam optimizer with initial LR of 0.0003 is used. Exponential with reset LR schedule is applied for all methods except ours and GDumb, with γ = 0.9999 for CIFAR datasets and γ = 0.99995 for TinyImageNet and ImageNet. Ours use adaptive LR with γ = 0.95,m = 10 for all datasets, GDumb (Prabhu et al., 2020) and RM (Bang et al., 2021) follow original settings in their respective paper. All code and i-Blurry-N -M splits (see Supp.) will be publicly released.
Baselines. We compare our method with both online CL methods and ones that can be extended to the online setting; EWC++ (Chaudhry et al., 2018b), BiC (Wu et al., 2019), GDumb (Prabhu et al., 2020), A-GEM (Chaudhry et al., 2019), MIR (Aljundi et al., 2019a) and RM (Bang et al., 2021). ·† indicates that the two-stage training scheme (Bang et al., 2021) is used. For details of the online versions of these methods, see Sec. A.5. Note that A-GEM performs particularly worse (also observed in (Prabhu et al., 2020; Mai et al., 2021)) as A-GEM was designed for the task-incremental setup and our setting is task-free. We discuss the comparisons to A-GEM in Sec. A.7.

heading: 5.1 RESULTS ON THE I-BLURRY SETUP
text: In all our experiments, we denote the best result for each of the metrics in bold.
We first compare various online CL methods in the i-Blurry setup on CIFAR10, CIFAR100, TinyImageNet and ImageNet in Table 1. On CIFAR10, proposed CLIB outperforms all other CL methods by large margins; at least +12.67% in AAUC and +12.85% in Aavg . On CIFAR100, CLIB also outperforms all other methods by large margins; at least +11.47% in AAUC and +10.18% in Aavg . On TinyImageNet, all methods score very low. Nonetheless, CLIB outperforms other CL methods by at least +2.71% in AAUC and +1.90% in Aavg . Surprisingly on ImageNet, most methods perform slightly better than on TinyImagenet. We believe the reason is that the samples per class is the same with TinyImageNet (e.g., 20) but ImageNet has higher resolution images, making learning from those images easier than TinyImageNet. On ImageNet, CLIB still outperforms other CL methods by at least +2.60% in AAUC and +2.06% in Aavg. Since, CLIB uses memory only training scheme, the distribution of the training samples are stabilized through the memory (see Sec. 4.3) which is helpful in the i-Blurry setup where the model encounters samples from more varied classes.
Note that Rainbow Memory (RM) (Bang et al., 2021) exhibits a very different trend than other compared methods. On TinyImageNet, it performs poorly. We conjecture that the delayed learning from the two-stage training is particularly detrimental in larger datasets with longer training duration and tasks such as TinyImageNet. On CIFAR10 and CIFAR100, RM performs reasonably well in Aavg but poorly inAAUC. This verifies that its two-stage training method delays most of the learning to the end of each task, resulting in a poor any-time inference performance measured byAAUC. Note that Aavg fails to capture this; on CIFAR10, the difference in Aavg for CLIB and RM is +13.27% which is similar to other methods but the difference forAAUC is +47.55% which is noticeably larger than other methods. Similar trends can be found on other datasets as well.
We also show the accuracy-to-{# of samples} curve for the CL methods on CIFAR10, CIFAR100, TinyImageNet and ImageNet for comprehensive analysis throughout training in Fig 4. Interestingly, RM shows a surged accuracy only at the task transitions due to its two-stage training method and the accuracy is overall low. Additionally, GDumb shows a severe decreasing trend in accuracy as tasks progress. It is because the ‘Dumb Learner’ trains from scratch at every inference query leading to accuracy degradation. In contrast, CLIB not only outperforms other methods but also shows the most consistent accuracy at all times. More discussion of other methods are in Sec. A.9.

heading: 5.2 ANALYSIS ON DISJOINT CLASS PERCENTAGES (N ) AND BLURRY LEVELS (M ).
text: We further investigate the effect of differentN andM values in the i-Blurry-N-M splits with various CL methods and summarize the results for varying values of the disjoint class percentages such as
N = 0, 50, 100 in Table 2 (top). For N = 0, CLIB outperforms other methods by at least +16.59% in AAUC and +16.36% in Aavg . For N = 100, the performance is similar for the majority of the methods, with CLIB being the best in AAUC. For N = 50, CLIB outperforms all comparisons by at least +12.55% in AAUC and +13.31% in Aavg . Even though CLIB was designed with the i-Blurry setup in mind, it also outperforms other CL methods in conventional setups such as the N = 100 (disjoint) or the N = 0 (blurry) setups. It implies that CLIB is generally applicable to online CL setups and not restricted to just the i-Blurry setup. Meanwhile, except GDumb, all methods show similar performance on the N = 100 (disjoint) setup. The results imply that the i-Blurry setup differentiates CL methods better than the more traditionally used disjoint setup.
Following Bang et al. (2021), we additionally summarize the results for varying values of the blurry level such as M = 10, 30, 50 in Table 2 (bottom). We observe that CLIB again outperforms or performs on par with other CL methods on various blurry levels.

heading: 5.3 ABLATION STUDIES
text: We investigate the benefit of each proposed component by ablation studies in Table 3. We use both CIFAR10 and CIFAR100.
Sample-wise Importance Based Memory Management. We replace the ‘sample-wise importance memory management’ module with the reservoir sampling. As shown in the table, the removal of our memory management strategy degrades the performance in both AAUC and Aavg on both CIFAR10 and CIFAR100. As explained in Sec. 4.2, reservoir sampling removes samples at random, hence samples are discarded without considering if some samples are more important than others. Thus, using sample-wise importance to select which sample to discard greatly contributes to performance.
Memory Only Training. We replace the memory usage strategy from our memory only training with ER. Training with ER means that samples from both the online stream and the memory are used. Without the proposed memory only training scheme, the performance degrades across the board by fair margins. As the streamed samples are being used directly without the sample memory acting as a distribution regularizer (see Sec. 4.3), the CL model is more influenced by the recently observed samples, skewing the training and resulting in worse performance.
Adaptive Learning Rate Scheduling. We change the LR scheduling from adaptive LR to exponential with reset. Although performance drop is small on CIFAR10 but gets bigger in the more challenging CIFAR100. CIFAR100 induces more training iterations which makes the ablated model suffer more from a lack of good adaptive LR scheduling.

heading: 6 CONCLUSION
text: We question the practicality of existing continual learning setups for real-world application and propose a novel CL setup named i-Blurry. It is online, task-free, class-incremental, has blurry task boundaries, and is subject to any-time inference. To address this realistic CL setup, we propose a method which uses per-sample memory management, memory only training, and adaptive LR scheduling, named Continual Learning for i-Blurry (CLIB). Additionally, we propose a new metric to better evaluate the effectiveness of any-time inference. Our proposed CLIB consistently outperforms existing CL methods in multiple datasets and setting combinations by large margins.

heading: ETHICS STATEMENT
text: All continual learning (CL) methods including the proposed one would adapt and extend the already trained AI model to recognize better with the streamed data. The CL methods will expedite the deployment of AI systems to help humans by its versatility of adapting to a new environment out of the factory or research labs. As all CL methods, however, would suffer from adversarial streamed data as well as data bias, which may cause ethnic, gender or biased gender issues, the proposed CLIB would not be an exception. Although the proposed CLIB has no intention to allow such problematic cases, the method may be exposed to such threats. Relentless efforts should be made to develop mechanisms to prevent such usage cases in order to make the continuously updating machine learning models safer and enjoyable to be used by humans.

heading: REPRODUCIBILITY STATEMENT
text: We take the reproducibility of the research very seriously and solemnly promise to release all codes, containers (e.g., Docker) that includes running environments and learned models of pretraining and downstream tasks in a public repository.

heading: A APPENDIX

heading: A.1 ADDITIONAL DISCUSSION FOR OUR BASELINE
text: In constructing our baseline, we used the reservoir sampling for memory management and ER for memory usage. We used reservoir sampling as it is widely used in the online and task-free setups with good performance. Note that in online CL, memory management policies that use the entire task’s samples at once, such as herding selection (Rebuffi et al., 2017), mnemonics (Liu et al., 2020), and rainbow memory (Bang et al., 2021) are inapplicable. For the memory usage, we use ER which draws half of the training batch from the stream and the other half from the memory, following a large number of online CL methods based on ER with good performance (Mai et al., 2021). For the LR scheduling, we first note that other CL methods use either (1) exponential decay (Rebuffi et al., 2017; Kirkpatrick et al., 2017; Mirzadeh et al., 2020) or (2) constant LR. We do not use (1) as it is hyper-parameter sensitive; the decay rate that works for CIFAR10 decayed the LR too quickly for larger datasets such as CIFAR100. If the LR is decayed too fast, the LR becomes too small to learn about new classes that are introduced in the future. Thus, we use exponential LR scheduler with the modification that the LR is reset when a new class is observed. Comparing with the constant LR, we obtain slightly better performance for EWC++ and our baseline on CIFAR10, as shown in Table 9. Thus, we denote this LR schedule as the exponential with reset and use it in our baseline.

heading: A.2 PROOF OF THEOREM 1
text: We give the proof of Theorem 1 below. Our assumption is that when selecting memoryM from a set of candidates C, we should select M so that optimizing on M maximizes the loss decrease on C. In equations, optimal memoryM∗ is
M∗ = arg max M⊂C, |M|≤m ∑ (x′,y′)∈M ∑ (x,y)∈C Eθ [l(x, y; θ)− l(x, y; θ −∇θl(x′, y′; θ))] (4)
= arg max M⊂C, |M|≤m  ∑ (x′,y′)∈C ∑ (x,y)∈C Eθ [l(x, y; θ)− l (x, y; θ −∇θl(x′, y′; θ))]
− ∑
(x′,y′)∈C\M ∑ (x,y)∈C Eθ [l(x, y; θ)− l (x, y; θ −∇θl(x′, y′; θ))]  (5) = arg min M⊂C, |M|≤m ∑ (x′,y′)∈C\M ∑ (x,y)∈C Eθ [l(x, y; θ)− l (x, y; θ −∇θl(x′, y′; θ))] , (6)
where θ is the model parameter, l is the loss function, and m is the memory size. Since we perform memory update after every streamed sample, the problem reduces to selecting one sample (x̄, ȳ) to remove when the memory is full. Thus, C \M = {(x̄, ȳ)}. The optimal removed sample (x̄∗, ȳ∗) would be
(x̄∗, ȳ∗) = arg min (x̄,ȳ)∈C Eθ  ∑ (x,y)∈C l(x, y; θ)− l (x, y; θ −∇θl(x̄, ȳ; θ))  . (7)

heading: A.3 DETAILS ON SAMPLE-WISE IMPORTANCE BASED MEMORY MANAGEMENT
text: We describe the details of our sample-wise importance based memory management here. We update H, the estimate of sample-wise importance for episodic memory, after every model update. The details are in Alg. 1. The runtime complexity of updating the sample-wise importance is O(M) where M is the fixed memory size. Because the memory size is kept constant regardless of the data size, the algorithm can scale to larger datasets. Note that for practical efficiency, Alg. 1 can be used after every k model updates instead of every model update as described above. With the sample-wise
Algorithm 1 Update Sample-wise Importance 1: Input model fθ, memoryM, sample-wise importance H, previous loss lprev, indices used for
training I, update coefficient λ 2: lcur = 1 |M| ∑ (x,y)∈M l(x, y; θ) . Obtain memory loss 3: ∆l = lprev − lcur . Obtain memory loss decrease 4: ∆lpred = 1 |I| ∑ i∈I Hi . Memory loss decrease prediction using current H 5: for i ∈ I do 6: UpdateHi ← Hi + λ(∆l −∆lpred) . Update H for samples used for training 7: end for 8: Update lprev ← lcur 9: OutputH, lprev
Algorithm 2 Sample-wise Importance Based Memory Update 1: Input model fθ, memoryM, memory size m, sample (x̂, ŷ), per-sample criterion H, previous
loss lprev 2: if |M| < m then . If the memory is not full 3: UpdateM←M∪ {(x̂, ŷ)} . Append the sample to the memory 4: ī = |M| 5: else . If the memory is already full 6: ymax = arg maxy |{(xi, yi)|(xi, yi) ∈M∪ {(x̂, ŷ)}, yi = y}| . Find the most frequent label 7: Iymax = {i|(xi, yi) ∈M, yi = ymax} 8: î = arg mini∈Iymax Hi . Find the sample with the lowest importance 9: Update lprev ← mm−1 lprev − 1 m−1 l(xî, yî; θ) 10: UpdateMî ← (x̂, ŷ) . Replace that sample with the new sample 11: end if 12: Update lprev ← |M|−1|M| lprev + 1 |M| l (θ, x̂, ŷ) 13: Iŷ = {i|(xi, yi) ∈M, yi = ŷ, i 6= î} 14: UpdateHî ← 1 |Iŷ| ∑ i∈Iŷ Hi . Initialize the importance for the new sample 15: OutputM,H, lprev
importance scores, we update the memory everytime a new sample is encountered. The details are in Alg. 2.

heading: A.4 ADAPTIVE LEARNING RATE SCHEDULER
text: We describe the adaptive LR schedule from 4.4 in Alg. A.4. We fix the significance level to the commonly used α = 0.05. Our adaptive LR scheduling decreases or increases the LR based on its current value. Thus, the rate in which the LR can change is bounded and sudden changes in LR do not happen.

heading: A.5 DETAILS ON THE ONLINE VERSIONS OF COMPARED CL METHODS
text: We implemented online versions of RM (Bang et al., 2021), EWC++ (Chaudhry et al., 2018b), BiC (Wu et al., 2019), GDumb (Prabhu et al., 2020), A-GEM (Chaudhry et al., 2019), GSS (Aljundi et al., 2019c) and MIR (Aljundi et al., 2019a) by incorporating ER and exponential decay with reset to the methods whenever possible. There is no specified memory management strategy for EWC++, and BiC uses herding selection from iCaRL(Rebuffi et al., 2017). However, herding selection is not possible in online since it requires whole task data for calculating class mean, so we attach reservoir memory to both methods instead. EWC++ does not require any other modification.
Additional modification should be applied to bias correction stage of BiC. BiC originally performs bias correction at end of each task, but since evaluation is also performed at the middle of task in our setup, we modified the method to perform bias correction whenever the model receives inference query.
Algorithm 3 Adaptive Learning Rate Scheduler 1: Input current LR η, current base LR η̄, loss before applying current LR lbefore, current loss lcur,
LR performance history H(high) and H(low), LR step γ < 1, history length m, significance level α 2: ldiff = lbefore − lcur . Obtain loss decrease 3: Update lbefore ← lcur 4: if η > η̄ then . If LR is higher than base LR 5: UpdateH(high) ← H(high) ∪ {ldiff} . Append loss decrease in high LR history 6: if |H(high)| > m then 7: UpdateH(high) ← H(high) \ {H(high)1 } 8: end if 9: else . If LR is lower than base LR 10: UpdateH(low) ← H(low) ∪ {ldiff} . Append loss decrease in low LR history 11: if |H(low)| > m then 12: UpdateH(low) ← H(low) \ {H(low)1 } 13: end if 14: end if 15: if |H(high)| = m and |H(low)| = m then . If both histories are full 16: p = OneSidedStudentsTTest ( H(low),H(high)
) . Perform one-sided Student’s t-test with alternative hypothesis µlow > µhigh
17: if p < α then . If pvalue is significantly low 18: Update η̄ ← γ2 · η̄ . Decrease base LR 19: UpdateH(low),H(high) = ∅, ∅ . Reset histories 20: else if p > 1− α then . If pvalue is significantly high 21: Update η̄ ← 1γ2 · η̄ . Increase base LR 22: UpdateH(low),H(high) = ∅, ∅ 23: end if 24: end if 25: if η > η̄ then . Alternately apply high and low LR (note that γ < 1) 26: Update η = γ · η̄ 27: else 28: Update η = 1γ · η̄ 29: end if 30: Output η, η̄, lbefore,H(low),H(high)
In RM, their memory management strategy based on uncertainty is not applicable in an online setup, since it requires uncertainty rankings of the whole task samples. Thus, we replace their sampling strategy with balanced random sampling, while keeping their two-stage training scheme. Methods that were converted from offline to online, namely EWC++, BiC, and RM, may have suffered some performance drop due to deviation from their original methods.

heading: A.6 ANALYSIS ON SAMPLE MEMORY SIZE AND NUMBER OF TASKS
text: We conduct analysis over various sample memory sizes (K) and summarize results in Table 4. We observe that CLIB outperforms other CL methods in both AAUC and Aavg no matter the memory size. It is interesting to note that CLIB with a memory size of K = 200 outperforms other CL methods with a memory size of K = 1000 in the AAUC and performs on par in Aavg . Thus, CLIB is the only method using memory only training scheme but is the least sensitive to memory size. It implies that our memory management policy is the most effective, which shows the superiority of our per-sample memory management method.
We additionally conduct analysis on the number of tasks used for the CIFAR100 dataset and summarize the results in Table 5. Note that we use CIFAR100 as CIFAR10 has too few classes to divide into longer task sequences. We observe that CLIB outperforms other CL methods in both AAUC and Aavg even when the number of tasks increase from 5 to 10 or 25. Moreover, CLIB shows
only minor performance drops with increasing number of tasks, indicating that CLIB is capable of achieving high performance in the long-run.
Interestingly, we note that BiC shows a slight performance increase when the number of tasks is increased to 10 or 25. We believe this is because BiC uses separate bias parameters for each task, longer task sequences allows more fine-grained bias correction and may actually be favorable for BiC. We also point out that the absolute performance of BiC is still lacking. Thus, it may imply that the bias correction of BiC contributes to strong stability in the stability-plasticity trade-off (Chaudhry et al., 2018a), which help maintain performance for longer task sequences where forgetting would be more severe. However, as stability is enforced strongly, BiC might lack plasticity to sufficiently learn from new data, resulting in low performance overall.
We also note that EWC++ and Baseline-ER have the same AAUC and Aavg values up to the second decimal places in the 10 tasks setting. This may imply that EWC++ is seldom beneficial over Baseline-ER as the additional episodic memory negates the need for the regularization done in EWC++.

heading: A.7 ADDITIONAL COMPARISONS WITH A-GEM
text: We present additional comparisons to A-GEM. Note that as A-GEM was designed for the taskincremental setting, it performs very poorly in our i-Blurry setup which is task-free. Notably, it achieves only 4.62 AAUC and 6.94 Aavg on CIFAR100, but other works (Prabhu et al., 2020; Mai et al., 2021) have also reported very poor performance for A-GEM in their studies as well.

heading: A.8 COMPARISONS TO OTHER MEMORY MANAGEMENT CL METHODS
text: We present additional comparison to CL methods that use a different memory management strategy in Table 7. GSS (Aljundi et al., 2019c) is added as an additional comparison while Baseline-ER is used to represent the reservoir sampling. CLIB outperforms both methods by large margins in both AAUC and Aavg, implying that the sample-wise importance memory management method is better than reservoir or GSS-greedy.

heading: A.9 ADDITIONAL DISCUSSION OF OTHER METHODS
text: We present additional discussion on compared methods in Table 1.

heading: A.9.1 BASELINE-ER
text: ER has been reported to be a simple yet strong method for online CL. (Prabhu et al., 2020; Mai et al., 2021) We also observed strong performance for Baseline-ER, as other methods such as EWC++, BiC, and MIR only show marginal to no improvements. However, by using different memory management and memory usage strategy, CLIB was able to outperform Baseline-ER by large margins
A.9.2 EWC++
EWC is a regularization-based method, which regularizes parameters based on the parameter’s importance measured by the accumulated Fisher Information. Note that EWC calculates importance of parameters to regularize them, while CLIB calculates importance of samples to manage episodic memory. We use an online version of EWC, called EWC++, proposed in (Chaudhry et al., 2018b). EWC++ shows almost no improvement over Baseline-ER, possibly because it was developed as a method to prevent forgetting without using episodic memory. Since episodic memory with ER alleviates most of the forgetting, EWC++ have reduced effect in preventing forgetting, while its side-effect of reducing intransigence still exists (Chaudhry et al., 2018b).
A.9.3 BIC
BiC uses episodic memory and a distillation loss for training and has an additional bias-correction step to correct the model’s bias towards current task’s classes. BiC shows strong performance on the disjoint task split. However, it performs poorly for the blurry and the i-Blurry splits, possibly because the bias correction using only two parameters for previous and current tasks cannot correctly capture the bias in the presence of blurry classes.
A.9.4 ER-MIR
Note that ER-MIR performs surprisingly similar to the Baseline-ER for CIFAR10, possibly because they are both based on ER. However, ER-MIR does not scale well to larger datasets potentially because its memory usage scheme is not effective for larger datasets.
To extensively highlight the differences between MIR and CLIB, we first have to point out that sample memory methods can be discussed in two aspects: memory management and memory usage. MIR uses reservoir sampling for memory management and an improved version of experience replay (ER) for memory usage. MIR improves the memory usage by selecting samples from memory that would suffer the largest loss increase if streamed data were used to train the model (called ‘maximally interfered retrieval’ or MIR).
MIR can be viewed as using some form of ‘importance weighting’, and has some similarities with our method in the sense that they both utilize the change in loss to determine which samples are used for training. However, these two are different as MIR uses importance scores for selecting samples from a constructed memory whereas CLIB uses it for constructing the memory. Also, MIR assigns high importance scores to samples that show large increases in individual loss when current streamed data are used for training. In contrast, CLIB assigns high importance scores to samples that cause large decreases in memory’s total loss when that sample is used for training. Lastly, MIR’s importance score is temporary, as new importance scores are calculated every iteration, while CLIB accumulates the measured importance scores over time to calculate the overall importance scores.
Our method performs better than MIR possibly because sample-wise importance memory considers each samples’ importance whereas reservoir sampling does not and memory-only training is not skewed by recently seen samples whereas joint training is. Note that since we are selecting random samples in the memory usage step, a possible future direction would be improving the memory usage as done in MIR.

heading: A.9.5 GDUMB
text: GDumb utilizes a greedy sampler, which stores samples greedily, and a dumb learner, which trains a new model from scratch using only the memory whenever an inference query is made. While we also train only using the memory, we have a completely different memory management method that updates both the model and the memory during training. Because GDumb trains from scratch, its performance degrades in the later phases as it cannot accumulate past knowledge. Not only that, because GDumb has to train from scratch for every inference query, it takes more computation than other methods when there are many inference queries for any-time inference.

heading: A.9.6 RAINBOW MEMORY
text: Rainbow memory uses uncertainty based memory management with a two-stage training method that prolongs most of the training until the end of a task. As such, RM performs very poorly on our AAUC metric as it measures any-time inference performance. Nevertheless, RM performs strongly with respect to the Aavg metric which shows the usefulness of our proposed AAUC .
A.10 ADDITIONAL RESULTS WITH THE FLAST MEASURE
We report the result of the forgetting measure (Chaudhry et al., 2018b) here. Note that while forgetting is a useful metric for analyzing stability-plasticity of the method, lower forgetting does not necessarily mean that a CL method is better. For example, if a method do not train with the new task at all, its forgetting will be 0.
Also, we do not propose a new forgetting measure for anytime inference. It is because forgetting is measured with the best accuracy of each class, and best accuracy usually occur at the end of each task. Thus, measuring the best accuracy among all inferences would not be much different from best accuracy among the inferences at the end of each task.
Surprisingly, CLIB shows similar Flast values to most other methods despite not explicitly designed to handle catastrophic forgetting. We believe that the episodic memory used in our memory only training reduces catastrophic forgetting of our method to be similar to other methods.
Interestingly, GDumb shows the best Flast value on ImageNet, a different trend than on the other three datasets. We believe the reason is that as GDumb learns little knowledge in the first place possibly due the ‘dumb learner’ not being effective for ImageNet ( indicated by its low accuracy measure), there is not much to forget which results in good Flast values.

heading: A.11 ADDITIONAL DISCUSSION ON POTENTIAL PERFORMANCE DROP AT LATER PHASES
text: We discuss the potential cause for the ‘performance drop’ of CLIB in the i-Blurry setup as is shown in Fig. 4. We argue that the ‘performance drop’ is attributed to the number of encountered classes increasing in i-Blurry setups at later phases. To see what would happen when the number of encountered classes do not increase at later phases, we show the accuracy-to-# of samples curve for the blurry setup results in Table 2 for N = 0 in Fig. 5, where no new classes are encountered after the first task. There is no apparent performance drop at later phases for all methods, which supports our argument that the drop is due to more and more classes being encountered at later phases.

heading: A.12 PERFORMANCE OF EXPONENTIAL WITH RESET LR SCHEDULE
text: We show brief results for the LR schedule used in our baseline in Table 9. We compare the constant LR with the exponential with reset used in our baseline. The exponential with reset is better than the constant LR, which is why we used it in our baseline.

Review summaries:
The paper proposes a more realistic setting (called *i-Blurry*) for continual learning (CL) that generalizes the *blurry* and *disjoint*  settings proposed in prior work. The disjoint setting assumes that there is no class that appears in multiple tasks and the blurry setting assumes that no new classes are seen after the first task. In the *i-Blurry* setting, one can have overlapping classes across tasks as well as new classes appearing in each task. This setting is also **online** and thus the paper is interested in continuous model evaluation i.e., any-time inference too. It proposes a metric which calculates the area under the accuracy curve during training for the same.   For the new configuration, the paper proposes a strong baseline and a new algorithm called CLIB. CLIB is a memory-based CL method that refines its memory by throwing out *least important* samples and updating it with more important ones. The sample importance is calculated as the expected decrease in training loss when the sample is used for training. CLIB is also equipped with a data-driven adaptive learning rate scheduling scheme which provides some additional performance benefits.   The results showcase that CLIB is able to outperform other online CL methods and the baseline by large margins for various instantiations of their i-Blurry setting, *including the disjoint and Blurry setting*, on the CIFAR-10, CIFAR-100 and TinyImagenet datasets. An ablation study shows the main benefits of the proposed method come from the sample importance based memory scheme. 

In this paper, the authors proposed a new benchmark protocol (i-Blurry) for continual learning. In this benchmark protocol, the class distribution is class incremental and has blurry task boundaries, and the training is online. They also propose a new method, CILB. This method contains three important components: “sample importance memory”, “memory-only training”, and “adaptive LR scheduling.” Extensive experimental results are provided to show the effectiveness of the proposed method. 

The paper proposes a new problem setup in continual learning. As the title suggests, the paper focuses on online, task-free, class incremental, task blurry learning with any-time inference. The authors also came up with new baselines and importance-based memory management. They empirically tested their methods in the proposed problem setup.