Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference

1 INTRODUCTION . Continual learning ( CL ) is a learning scenario where a model learns from a continuous and online stream of data and is regarded as a more realistic and practical learning setup than offline learning on a fixed dataset ( He et al. , 2020 ) . However , many CL methods still focus on the offline setup ( Kirkpatrick et al. , 2017 ; Rebuffi et al. , 2017 ; Saha et al. , 2021 ) instead of the more realistic online setup . These methods assume access to a large storage , storing the entire data of the current task and iterating on it multiple times . On the other hand , we are interested extensively in the more realistic online setup where only a small memory is allowed as storage . Meanwhile , even for the online CL methods , we argue they have room for more practical and realistic improvements concerning multiple crucial aspects . The aspects include the class distributions such as the disjoint ( Rebuffi et al. , 2017 ) or the blurry ( Aljundi et al. , 2019c ) splits and the evaluation metric that focuses only on the task accuracy such as average task accuracy ( Aavg ) . The two main assumptions on the class distributions in existing CL setups , i.e. , the disjoint and blurry splits , are less realistic for the following reasons . The disjoint split assumes no classes overlap over different tasks ; already observed classes will never appear again . This assumption is not plausible because already observed classes can still appear later on in real-world scenarios ( see Fig . 2 of ( Bang et al. , 2021 ) ) . On the other hand , in the blurry split ( Aljundi et al. , 2019c ) no new classes appear after the first task even though the split assumes overlapping classes over tasks . This is also not plausible as observing new classes is common in real-world scenarios . The typical evaluation metric such as Aavg in which the accuracy is measured only at the task transition is also less realistic . It implicitly assumes that no inference queries occur in the middle of a task . However , in real-world scenarios , inference queries can occur at any-time . Moreover , there is no explicit task transition boundary in most real-world scenarios . Thus , it is desirable for CL models to provide good inference results at any time . To accurately evaluate whether a CL model is effective at such ‘ any-time ’ inference , we need a new metric for CL models . In order to address the issues of the current CL setups , we propose a new CL setup that is more realistic and practical by considering the following criteria : First , the class distribution is comprised of the advantages from both blurry and disjoint . That is , we assume that the model continuously encounters new classes as tasks continue , i.e. , class-incremental and that classes overlap across tasks , i.e. , blurry task boundaries , while not suffering from the restrictions of blurry and disjoint . Second , the model is evaluated throughout training and inference such that it can be evaluated for any-time inference . We call this new continual learning setup ‘ i-Blurry ’ . For the i-Blurry setup , we first propose a plausible baseline that employs experience replay ( ER ) with reservoir sampling and a learning rate scheduling tuned for the online and task-free CL setting . While existing online CL methods are applicable to the i-Blurry setup , they perform only marginally better than our baseline or often worse . To better handle the i-Blurry setup , we propose a novel continual learning method , which improves the baseline in three aspects . We design a new memory management scheme to discard samples using a per-sample importance score that reflects how useful a sample is for training . We then propose to draw training samples only from the memory instead of drawing them from both memory and the online stream as is done in ER . Finally , we propose a new learning rate scheduling to adaptively decide whether to increase or decrease the learning rate based on the loss trajectory , i.e . a data-driven manner . To evaluate the algorithms in the new setup , we evaluate methods by conventional metrics , and further define a new metric called ‘ area under the curve of accuracy ’ ( AAUC ) which measures the model ’ s accuracy throughout training . We summarize our contributions as follows : • Proposing a new CL setup called i-Blurry , which addresses a more realistic setting that is online , task-free , class-incremental , of blurry task boundaries , and subject to any-time inference . • Proposing a novel online and task-free CL method by a new memory management , memory usage , and learning rate scheduling strategy . • Outperforming existing CL models by large margins on multiple datasets and settings . • Proposing a new metric to better measure a CL model ’ s capability for the desirable any-time inference . 2 RELATED WORK . Continual learning setups . There are many CL setups that have been proposed to reflect the realworld scenario of training a learning model from a stream of data ( Prabhu et al. , 2020 ) . We categorize them in the following aspects for brevity . First , we categorize them into ( 1 ) task-incremental ( task-IL ) and ( 2 ) class-incremental learning ( class-IL ) , depending on whether the task-ID is given at test time . Task-IL , also called multi-head setup , assumes that task-ID is given at test time ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2018 ; Chaudhry et al. , 2019 ) . In contrast , in class-IL , or single-head setup , task-ID is not given at test time and has to be inferred ( Rebuffi et al. , 2017 ; Wu et al. , 2019 ; Aljundi et al. , 2019a ) . Class-IL is more challenging than task-IL , but is also more realistic since task-ID will not likely be given in the real-world scenario ( Prabhu et al. , 2020 ) . Most CL works assume that task ID is provided at training time , allowing CL methods to utilize the task ID to save model parameters at task boundaries ( Kirkpatrick et al. , 2017 ; Chaudhry et al. , 2018b ) for later use . However , this assumption is impractical ( Lee et al. , 2019 ) since real-world data usually do not have clear task boundaries . To address this issue , a task-free setup ( Aljundi et al. , 2019b ) , where task-ID at training is not available , has been proposed . We focus extensively on the task-free setup as it is challenging and being actively investigated recently ( Kim et al. , 2020 ; Lee et al. , 2019 ; Aljundi et al. , 2019c ) . We now categorize CL setups into disjoint and blurry setup by how the data split is configured . In the disjoint task setup , each task consists of a set of classes disjoint from all other tasks . But the disjoint setup is less realistic as the classes in the real-world can appear at any time not only in a disjoint manner . Recently , to make the setup more realistic , a blurry task setup has been proposed and investigated ( Aljundi et al. , 2019c ; Prabhu et al. , 2020 ; Bang et al. , 2021 ) , where 100 −M % of the sampels are from the dominant class of the task and M % of the samples are from all classes , where M is the blurry level ( Aljundi et al. , 2019c ) . However , the blurry setup assumes no class is added in new tasks , i.e. , not class-incremental , which makes the setup still not quite realistic . Finally , depending on how many samples are streamed at a time , we categorize CL setups into online ( Rolnick et al. , 2018 ; Aljundi et al. , 2019a ; Chaudhry et al. , 2019 ) and offline ( Wu et al. , 2019 ; Rebuffi et al. , 2017 ; Chaudhry et al. , 2018b ; Castro et al. , 2018 ) . In the offline setup , all data from the current task can be used an unlimited number of times . This is impractical since it requires additional memory of size equal to the current task ’ s data . For the online setup , there are many notions of ‘ online ’ that differs in each literature . Prabhu et al . ( 2020 ) ; Bang et al . ( 2021 ) refer online to a setup using each streamed sample only once to train a model while Aljundi et al . ( 2019c ; a ) refer online to a setup where only one or a few samples are streamed at a time . We follow the latter as the former allows storing the whole task ’ s data , which is similar to offline and more unrealistic than using the sample more than a few times . Incorporating the pros and cons of existing setups , we propose a novel CL setup that is online , task-free , class-incremental , of blurry task boundaries , and subject to any-time inference as the most realistic setup for continual learning . Continual learning methods . Given neural networks would suffer from catastrophic forgetting ( McCloskey & Neal , 1989 ; Ratcliff , 1990 ) , the online nature of streaming data in continual learning generally aggravates the issue . To alleviate the forgetting , there are various proposals to store the previous task information ; ( 1 ) regularization , ( 2 ) replay , and ( 3 ) parameter isolation . ( 1 ) Regularization methods ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Lee et al. , 2017b ; Ebrahimi et al. , 2020 ) store previous task information in the form of model priors and use it for regularizing the neural network currently being trained . ( 2 ) Replay methods store a subset of the samples from the previous tasks in an episodic memory ( Rebuffi et al. , 2017 ; Castro et al. , 2018 ; Chaudhry et al. , 2019 ; Wu et al. , 2019 ) or keep a generative model that is trained to generate previous task samples ( Shin et al. , 2017 ; Wu et al. , 2018 ; Hu et al. , 2019 ; Cong et al. , 2020 ) . The sampled or generated examplars are replayed on future tasks and used for distillation , constrained training , or joint training . ( 3 ) Parameter isolation methods augment the networks ( Rusu et al. , 2016 ; Lee et al. , 2017a ; Aljundi et al. , 2017 ) or decompose the network into subnetworks for each task ( Mallya & Lazebnik , 2018 ; Cheung et al. , 2019 ; Yoon et al. , 2020 ) . Since ( 1 ) , ( 2 ) , and ( 3 ) all utilize different ways of storing information that incurs parameter storage costs , episodic memory requirement and increase in network size respectively , a fair comparison among the methods is not straighforward . We mostly compare our method with episodic memorybased methods ( Wu et al. , 2019 ; Aljundi et al. , 2019a ; Bang et al. , 2021 ) , as they perform the best in various CL setups but also with methods that use both regularization and episodic memory ( Chaudhry et al. , 2018b ; Wu et al. , 2019 ) . Online continual learning . Despite being more realistic ( Losing et al. , 2018 ; He et al. , 2020 ) , online CL setups have not been popular ( Prabhu et al. , 2020 ) due to the difficulty and subtle differences in the setups in the published literature . ER ( Rolnick et al. , 2018 ) is a simple yet strong episodic memory-based online CL method . It employs reservoir sampling for memory management and jointly trains a model with half of the batch sampled from memory . Many online CL methods are based on ER ( Aljundi et al. , 2019c ; a ) . GSS ( Aljundi et al. , 2019c ) selects samples using a score based on cosine similarity of gradients . MIR ( Aljundi et al. , 2019a ) retrieves maximally interfering samples from memory to use for training . Different from ER , A-GEM ( Chaudhry et al. , 2019 ) uses the memory to enforce constraints on the loss trajectory of the stored samples . GDumb ( Prabhu et al. , 2020 ) only updates the memory during training phase and trains from scratch at the test time only using the memory . Unlike these methods , recently proposed RM ( Bang et al. , 2021 ) uses an uncertainty-based memory sampling and two-stage training scheme where the model is trained for one epoch on the streamed samples and trains extensively only using the memory at the end of each task , effectively delaying most of the learning to the end of the tasks . Note that the uncertainty-based memory sampling can not be implemented in the online CL setup and the two-stage training performs particularly worse in our i-Blurry setup . Our method outperforms all other online CL method introduced in this section while strictly adhering to the online and task-free restrictions .

Summary: The paper proposes a more realistic setting (called *i-Blurry*) for continual learning (CL) that generalizes the *blurry* and *disjoint*  settings proposed in prior work. The disjoint setting assumes that there is no class that appears in multiple tasks and the blurry setting assumes that no new classes are seen after the first task. In the *i-Blurry* setting, one can have overlapping classes across tasks as well as new classes appearing in each task. This setting is also **online** and thus the paper is interested in continuous model evaluation i.e., any-time inference too. It proposes a metric which calculates the area under the accuracy curve during training for the same.   For the new configuration, the paper proposes a strong baseline and a new algorithm called CLIB. CLIB is a memory-based CL method that refines its memory by throwing out *least important* samples and updating it with more important ones. The sample importance is calculated as the expected decrease in training loss when the sample is used for training. CLIB is also equipped with a data-driven adaptive learning rate scheduling scheme which provides some additional performance benefits.   The results showcase that CLIB is able to outperform other online CL methods and the baseline by large margins for various instantiations of their i-Blurry setting, *including the disjoint and Blurry setting*, on the CIFAR-10, CIFAR-100 and TinyImagenet datasets. An ablation study shows the main benefits of the proposed method come from the sample importance based memory scheme.