Online Coreset Selection for Rehearsal-based Continual Learning

1 INTRODUCTION . Humans possess the ability to learn a large number of tasks by accumulating knowledge and skills over time . Building a system resembling human learning abilities is a deep-rooted desire since sustainable learning over a long-term period is essential for general artificial intelligence . In light of this need , continual learning ( CL ) ( Thrun , 1995 ) , or lifelong learning , tackles a learning scenario where a model continuously learns over a sequence of tasks ( Kumar & Daume III , 2012 ; Li & Hoiem , 2016 ) within a broad research area , such as classification ( Kirkpatrick et al. , 2017 ; Chaudhry et al. , 2019a ) , image generation ( Zhai et al. , 2019 ) , language learning ( Li et al. , 2019b ; Biesialska et al. , 2020 ) , clinical application ( Lee & Lee , 2020 ; Lenga et al. , 2020 ) , speech recognition ( Sadhu & Hermansky , 2020 ) , and federated learning ( Yoon et al. , 2021 ) . A well-known challenge for continual learning is catastrophic forgetting ( McCloskey & Cohen , 1989 ) , where the continual learner loses the fidelity for past tasks after adapting the previously learned knowledge to future tasks . Recent rehearsal-based continual learning methods adapt the continual model to the previous tasks by maintaining and revisiting a small replay buffer ( Titsias et al. , 2020 ; Mirzadeh et al. , 2020 ) . However , the majority of these methods store random-sampled instances as a proxy set to mitigate catastrophic forgetting , limiting their practicality to real-world applications ( see Figure 1a ) when all the training instances are not equally useful , as some of them can be more representative or informative for the current task , and others can lead to performance degeneration for previous tasks . Furthermore , these unequal potentials could be more severe under practical scenarios containing imbalanced , streaming , or noisy instances ( see Figure 2 ) . This leads to an essential question in continual learning : How can we obtain a coreset to promote task adaptation for the current task while minimizing catastrophic forgetting on previously seen tasks ? To address this question , we propose Online Coreset Selection ( OCS ) , a novel method for continual learning that selects representative training instances for the current and previous tasks from arriving streaming data in an online fashion based on our following three selection strategies : ( 1 ) Minibatch similarity selects samples that are representative to the current task Tt . ( 2 ) sample diversity encourages minimal redundancy among the samples of current task Tt . ( 3 ) Coreset affinity promotes minimum interference between the selected samples and knowledge of the previous tasks Tk , ∀k < t. To this end , OCS minimizes the catastrophic forgetting on the previous tasks by utilizing the obtained coreset for future training , and also encourages the current task adaptation by updating the model parameters on the top-κ selected data instances . The overall concept is illustrated in Figure 1b . Our method is simple , intuitive , and is generally applicable to any rehearsal-based continual learning method . We evaluate the performance of OCS on various continual learning scenarios and show that it outperforms state-of-the-art rehearsal-based techniques on balanced , imbalanced , and noisy continual learning benchmarks of varying complexity . We also show that OCS is general and exhibits collaborative learning with the existing rehearsal-based methods , leading to increased task adaptation and inhibiting catastrophic forgetting . To summarize , our contributions are threefold : • We address the problem of coreset selection for realistic and challenging continual learning scenarios , where the data continuum is composed of class-imbalanced or noisy instances that deteriorate the performance of the continual learner during training . • We propose Online Coreset Selection ( OCS ) , a simple yet effective online coreset selection method to obtain a representative and diverse subset that has a high affinity to the previous tasks from each minibatch during continual learning . Specifically , we present three gradient-based selection criteria to select the coreset for current task adaptation while mitigating catastrophic forgetting . • We demonstrate that OCS is applicable to any rehearsal-based continual learning method and experimentally validate it on multiple benchmark scenarios , where it largely improves the performance of the base algorithms across various performance metrics . 2 RELATED WORK . Continual learning . In the past few years , there has been significant progress in continual learning to alleviate catastrophic forgetting ( McCloskey & Cohen , 1989 ) . The regularization approaches ( Kirkpatrick et al. , 2017 ; Lee et al. , 2017 ; Serrà et al. , 2018 ) modify the model parameters with additional regularization constraints to prevent catastrophic forgetting . The architecture approaches ( Rusu et al. , 2016 ; Yoon et al. , 2018 ; Xu & Zhu , 2018 ; Li et al. , 2019a ; Yoon et al. , 2020 ) utilize network isolation or expansion during continual learning to improve network performance . Another line of research uses rehearsal approaches , which memorize or generate a small fraction of data points for previous tasks and utilizes them to retain the task knowledge ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; Aljundi et al. , 2019b ; Borsos et al. , 2020 ) . For example , Gradient-based Sample Selection ( GSS ) ( Aljundi et al. , 2019b ) formulates the selection of the replay buffer as a constraint selection problem to maximize the variance of gradient direction . ER-MIR ( Aljundi et al. , 2019a ) iteratively constructs the replay buffer using a loss-based criterion , where the model selects the top-κ instances that increase the loss between the current and previous iteration . However , the existing rehearsalbased methods ( Rebuffi et al. , 2017 ; Aljundi et al. , 2019b ; a ; Chaudhry et al. , 2019a ; b ) do not select the coreset before the current task adaptation and update the model on all the arriving data streams , which makes them susceptible to real-world applications that include noisy and imbalanced data distributions . In contrast , OCS selects the instances before updating the model using our proposed selection criteria , which makes it robust to past and current task training across various CL scenarios . Coreset selection . There exist various directions to obtain a coreset from a large dataset . Importance sampling ( Johnson & Guestrin , 2018 ; Katharopoulos & Fleuret , 2018 ; Sinha et al. , 2020 ) strengthens the loss/gradients of important samples based on influence functions . Kool et al . ( 2019 ) connect stochastic Gumbel-top-k trick and beam search to hierarchically sample sequences without replacement . Rebuffi et al . ( 2017 ) propose a herding based strategy for coreset selection . Nguyen et al . ( 2018 ) formulate the coreset summarization in continual learning using online variational inference ( Sato , 2001 ; Broderick et al. , 2013 ) . Aljundi et al . ( 2019b ) select the replay buffer to maximize the variance in the gradient-space . Contrary to these methods , OCS considers the diversity , task informativity and relevancy to the past tasks . Recently , Borsos et al . ( 2020 ) propose a bilevel optimization framework with cardinality constraints for coreset selection . However , their method is extremely limited in practice and inapplicable in large-scale settings due to the excessive computational cost incurred during training . In contrast , our method is simple , and scalable since it can construct the coreset in the online streaming data continuum without additional optimization constraints . 3 REHEARSAL-BASED CONTINUAL LEARNING . We consider learning a model over a sequence of tasks { T1 , . . . , TT } = T , where each task is composed of independently and identically distributed datapoints and their labels , such that task Tt includes Dt = { xt , n , yt , n } Ntn=1 ∼ Xt × Yt , where Nt is the total number of data instances , and Xt × Yt is an unknown data generating distribution . We assume that an arbitrary set of labels for task Tt , yt = { yt , n } Ntn=1 has unique classes , yt ∩ yk = ∅ , ∀t 6= k. In a standard continual learning scenario , the model learns a corresponding task at each step and t-th task is accessible at step t only . Let neural network fΘ : X1 : T → Y1 : T be parameterized by a set of weights Θ = { θl } Ll=1 , where L is the number of layers in the neural network . We define the training objective at step t as follows : minimize Θ Nt∑ n=1 ` ( fΘ ( xt , n ) , yt , n ) , ( 1 ) where ` ( · ) is any standard loss function ( e.g. , cross-entropy loss ) . The naive CL design where a simple sequential training on multiple tasks without any means for tackling catastrophic forgetting can not retain the knowledge of previous tasks and thus results in catastrophic forgetting . To tackle this problem , rehearsal-based methods ( Nguyen et al. , 2018 ; Chaudhry et al. , 2019a ; Titsias et al. , 2020 ) update the model on a randomly sampled replay buffer Ck constructed from the previously observed tasks , where Ck = { xk , j , yk , j } Jkj=1 ∼ Dk , ∀k < t and Jk Nk . Consequently , the quality of selected instances is essential for rehearsal-based continual learning . For example , some data instances can be more informative and representative than others to describe a task and improve model performance . In contrast , some data instances can degrade the model ’ s memorization of past tasks ’ knowledge . Therefore , obtaining the most beneficial examples for the current task is crucial for the success of rehearsal-based CL methods. ) Classwise acc . of T2 Average Fgt . of T1 To validate our hypothesis , we design a learning scenario with a sequence of two tasks , MNIST ( T1 ) → CIFAR-10 ( T2 ) using ResNet-18 . After the standard single epoch training on T1 , we update the model weights through a single backpropagation step using a randomly selected data point from T2 , and measure test accuracy of its corresponding class c and forgetting of the entire dataset of a past task T1 . Results for individual impacts on 1000 data points from T2 are described in Section 3 . The influence of each data point from T2 has a large disparity not only on the corresponding class accuracy but also on past task ’ s forgetting that results in a very high standard deviation . We emphasize that each data point has a different potential impact in terms of forgetting past tasks . Few data points are much more robust to catastrophic forgetting than others , and this can be severe when the influences are accumulated during training . Based on this motivation , our objective is to select the data instances that can promote current task adaptation while minimizing catastrophic forgetting on the previous tasks . We propose a selection criterion that selects the subset that maximizes the gradient similarity between the representative instances and the current task dataset . More formally : u∗ = maximize u∈Nκ S ( 1 Nt ∇fΘ ( Dt ) , 1 κ ∑ n∈u ∇fΘ ( xt , n , yt , n ) ) , where u = { n : n ∈ N < Nt } , ( 2 ) where S is any arbitrary similarity function and u∗ is an index set that selects top-κ informative samples without replacement . However , obtaining a representative subset from the entire dataset is computationally expensive and intractable for online continual learning ; therefore , we consider a minibatch as an approximation of the dataset and select few representative data instances at each minibatch iteration . We empirically validate that our approximation generally holds across various datasets , network structures , and minibatch sizes in Appendix B and Figure B.9 . Consequently , the model iteratively updates the parameters to find the optimal local minima of the loss using informative data points , which obtain similar gradient directions with the averaged gradients of the dataset . In the next section , we propose OCS which consists of a simple similarity criterion to achieve this objective . However , similarity criterion is not sufficient to select the representative coreset for online continual learning ; hence , we propose diversity and coreset affinity criteria to mitigate catastrophic forgetting .

Summary: The author propose a novel approach for online coreset selection, i.e exemplars used in the rehearsal process of past tasks in a continual learning framework. The proposed method is based on the observation that not all the samples in a dataset are equally valuable, but their quality affects model's effectiveness and efficiency. The method selects the most representative and informative samples at each iteration and trains them in an online manner. The approach has been conviently compared with state-of-the art methods and demonstrated its superiority.