Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer

1 INTRODUCTION . Computer vision models in the real-world are often frozen and not updated after deployment , yet they may encounter novel data in the environment . Unlike the typical supervised learning setting , class-incremental continual learning challenges the learner to incorporate new information as it sequentially encounters new object classes without forgetting previously-acquired knowledge ( catastrophic forgetting ) . Research has shown that rehearsal of prior classes is a critical component for class-incremental continual learning ( Hsu et al. , 2018 ; van de Ven & Tolias , 2019 ) . Unfortunately , rehearsal requires a substantial memory budget , either in the form of a coreset of stored experiences or a separate learned model to generate samples from past experiences . This is not acceptable for memory-constrained applications which can not afford to increase the size of their memory as they encounter new classes . Instead , we consider a novel real-world setting where an incremental learner ’ s labeled task data is a product of its environment and the learner encounters a vast stream of unlabeled data in addition to the labeled task data . In such a setting ( visualized in Figure 1 ) , the unlabeled datastream is intrinsically correlated to each learning tasks due to the underlying structure of the environment . We explore many ways in which this correlation may exist . For example , when an incremental learner is tasked to learn samples of the previously-unseen class ci at time i in the real world , examples of ci may be encountered in the environment ( in unlabeled form ) during some future task . In such a setting , an incremental learner could use the unlabeled data in its environment as a source of memory-free rehearsal , though it would need a method to determine which unlabeled data is relevant to the incremental task ( i.e . detecting in-distribution data ) . We formalize this realistic paradigm in the semi-supervised continual learning ( SSCL ) setting , wherein unlabeled and labeled data are not i.i.d . as they are correlated through the underlying structure of the environment . We propose and conduct experiments over a realistic setting in which this correlation may exist , in the form of label super-class structure ( e.g . unlabeled examples of household furniture such as chairs , couches , and tables will appear while learning the labeled examples of household electrical devices such as lamp , keyboard , and television ( Krizhevsky et al. , 2009 ; Zhu & Bain , 2017 ) ) . We measure the final-task accuracy A , the accuracy over all tasks ⌦ , and the coreset memory required to attain a specific level of ⌦ accuracy over several realistic SSCL settings . Our experiments demonstrate that state-of-the-art continual learning methods ( Lee et al. , 2019 ) perform inconsistently in the novel SSCL paradigm with no prior method performing “ best '' . across all settings . This leads us to ask `` How can an approach to catastrophic forgetting be robust to several realistic , memory-constrained continual learning scenarios ? '' To answer the above question , we propose a novel learning approach that works well in both the simple ( i.e. , no correlations ) and realistic SSCL settings : DistillMatch . We leverage unlabeled data not only for knowledge distillation ( in which the distilling model is fixed ) , but also for a semi-supervised loss ( in which the supervisory signal can adapt during training on the new task ) . Key to our approach is that we address the distribution mismatch between the labeled and unlabeled data ( Oliver et al. , 2018 ) with out-of-distribution ( OoD ) detection ( and are the first to do so in the continual learning setting ) . Compared to nearest prior state-of-the-art ( all methods from ( Lee et al. , 2019 ) ) configured to work as well as possible in the novel SSCL setting , we outperform the state-of-the-art in all of our experiment scenarios by as much as a 54.5 % increase in ⌦ and no less than an 8.7 % increase . Furthermore , we find that our method can save up to 0.23 stored images per processed unlabeled image over naive rehearsal ( compared to ( Lee et al. , 2019 ) which only saved 0.08 stored images per processed unlabeled image ) . In summary , we make the following contributions : 1 . We propose the realistic semi-supervised continual learning ( SSCL ) setting , where object-object correlations between labeled and unlabeled sets are maintained through a label super-class structure . We show that state-of-the-art continual learning methods perform inconsistently in the SSCL setting ( i.e . no baseline method is “ best '' across all settings ) . 2 . We propose a novel continual learning method DistillMatch for the SSCL setting leveraging pseudo-labeling , strong data augmentations , and out-of-distribution detection . Compared to the baselines , DistillMatch achieves superior performance on a majority of metrics for 8/8 experiments and results in substantial memory budget savings . 2 BACKGROUND AND RELATED WORK . Knowledge Distillation in Continual Learning : Several related methods leverage distillation losses on past tasks to mitigate catastrophic forgetting using soft labels from a frozen copy of the previous task ’ s model ( Castro et al. , 2018 ; Hou et al. , 2018 ; Li & Hoiem , 2017 ; Rebuffi et al. , 2017 ) . For example , learning using two teachers , with one teacher distilling knowledge from previous tasks and another distilling knowledge from the current task , has been found to increase adaptability to a new task while preserving knowledge on the previous tasks ( Hou et al. , 2018 ; Lee et al. , 2019 ) . Classbalancing and fine-tuning have been used to encourage the model ’ s final predicted class distribution to be balanced across all tasks ( Castro et al. , 2018 ; Lee et al. , 2019 ) . These methods are related in that they rely on distillation losses to mitigate catastrophic forgetting , but the losses are designed to distill knowledge about specific local tasks and can not discriminate between classes from different tasks ( crucial for class incremental learning ) . More context on where our work fits into the greater body of continual learning research is provided in Appendix H .. Global distillation ( GD ) introduces a global teacher which provides a knowledge ensemble from both the past tasks and current task ( Lee et al. , 2019 ) . This addresses a crucial shortcoming of common knowledge distillation methods which do not reconcile information from the local tasks ( i.e . the groups of object classes presented sequentially to the learner ) with the global task ( i.e . all object classes seen at any time ) . GD leverages a large stream of uncorrelated unlabeled data from sources such as data mining social media or web data ( Lee et al. , 2019 ) to boost its distillation performance . Similar to GD , we leverage an unlabeled datastream to mitigate forgetting , but we take the perspective that this datastream is from the agent ’ s environment and reflects object-object correlation structures imposed by the world ( i.e . correlations between the task data and the unlabeled data ) . Out of Distribution Detection : Leveraging unlabeled data for rehearsal is key to our work , but it can contain a mix of classes not in the distribution of the data seen by the learner thus far . Therefore , we include Out-of-Distribution ( OoD ) Detection ( Hsu et al. , 2020 ; Lee et al. , 2018 ; Liang et al. , 2017 ) to select unlabeled data corresponding to the classes our learner has seen so far with high confidence . Semantic OoD detection is a difficult challenge ( Hsu et al. , 2020 ) and we do not have access to any known OoD data to calibrate our confidence . We therefore build on a recent method , Decomposed Confidence ( DeConf ) ( Hsu et al. , 2020 ) , which can be calibrated using only in-distribution training data . The method consists of decomposed confidence scoring with a learned temperature scaling in addition to input pre-processing . For further details , the reader is referred to ( Hsu et al. , 2020 ) . Semi-Supervised Learning : In semi-supervised learning ( which motivates the SSCL setting ) , models are given a ( typically small ) amount of labeled data and leverage unlabeled data to boost performance . This is an active area of research given that large , labeled datasets are expensive , but most applications have access to plentiful , cheap unlabeled data . There are several approaches to semi-supervised learning ( Berthelot et al. , 2019 ; Lee , 2013 ; Kingma et al. , 2014 ; Kuo et al. , 2019 ; Miyato et al. , 2018 ; Oliver et al. , 2018 ; Sohn et al. , 2020 ; Springenberg , 2015 ; Tarvainen & Valpola , 2017 ) which involve balancing a supervised loss ` s applied to the labeled data with an unsupervised loss ` ul applied to unlabeled data . Additional details on these methods are provided in Appendix H . 3 SSCL SETTING . In class-incremental continual learning , a model is gradually introduced to labeled data corresponding to M semantic object classes c1 , c2 , . . . , cM over a series of N tasks , where tasks are non-overlapping subsets of classes . We use the notation Tn to denote the set of classes introduced in task n , with |Tn| denoting the number of object classes in task n. Each class appears in only a single task , and the goal is to incrementally learn to classify new object classes as they are introduced while retaining performance on previously learned classes . The class-incremental learning setting ( Hsu et al . ( 2018 ) ) is a challenging continual learning settings because no task indexes are provided to the learner during inference and the learner must support classification across all classes seen up to task n. We extend the class-incremental continual learning setting in the realistic semi-supervised continual learning ( SSCL ) setting , where data distributions reflect existing object class correlations between , and among , the labeled and unlabeled data distributions . The amount of labeled data in this setting is drastically reduced as is common in semi-supervised learning . For example , our experiments reduce the number of labeled examples per class by 80 % compared to a prior setting ( Lee et al. , 2019 ) . At task n , we denote batches of labeled data as Xn = { ( xb , yb ) : b 2 ( 1 , · · · , B ) | yb 2 Tn } and batches of unlabeled data as Un = { ub : b 2 ( 1 , · · · , µB ) . Here , B refers to batch-size and µ is a hyperparameter describing the relative size of Xn to Un . The goal in task n is to learn a model ✓n which predicts object class labels for any query input over all classes seen in the current and previous tasks ( T1 [ T2 [ · · · [ Tn ) . The index n on ✓n indicates that our model is updated each task ; i.e . ✓n 1 refers to model from the previous task and ✓n refers to the model from the current task . To simulate an environment where unlabeled and labeled data are naturally correlated , we leverage well-defined relationships between objects derived from a super-class structure ( i.e . various animals within one super-class ) . We use the CIFAR-100 dataset ( Krizhevsky et al. , 2009 ) because object correlations among classes and parent classes , crucial to our experiments , are well defined and explored ( Zhu & Bain , 2017 ) . This dataset contains eight unbalanced super-classes , which we use to simulate realistic data environments . Each super-class contains a number of parent classes ( e.g . one super-class contains the parent classes flowers , fruits/vegetables , and trees ) . There are 20 parent classes in total which form the 20 continual learning tasks , with each parent class consisting of five object classes ( e.g . flowers parent class consists of orchids , poppies , roses , sunflowers , tulips ) . For a single task , when our learner is being shown labeled training data from one of the parent classes ( e.g . flowers , fruit/vegetables , or trees ) , the unlabeled data for this task will contain examples from the entire super-class ( e.g . flowers , fruits/vegetables , and trees ) . SSCL with this realistic super-class “ environment '' structure is our main setting , but we also explore several other correlation combinations , including the simple SSCL setting without any super-class structure . We use the following terminology to describe the correlations of the tasks ( i.e . labeled data ) : RandomClass Tasks , where no correlations exist in task classes , and ParentClass Tasks , where tasks are introduced by CIFAR-100 parent classes ( i.e . each task is to learn the five classes of a single CIFAR-100 parent class ) . For the unlabeled data distribution we have : Uniform Unlabeled , where all classes are uniformly dsitributed in unlabeled data for all tasks , PositiveSuperclass Unlabeled , where the unlabeled data of each tasks consists of the parent classes in the same super-class as the current task , NegativeSuperclass Unlabeled , where the unlabeled data of each tasks consists of parent classes from different super-class as the current task , and RandomUnlabeled , where the unlabeled data of each task consists of 20 randomly sampled classes ( roughly equal to the average class size in a super-class ) . Further discussion and details , including figures depicting example streams for each task sequence , are provided in Appendix F .

Summary: This paper investigates a semi-supervised continual learning (SSCL) setting and proposes a new method called DistillMatch for this setting. The major contributions are: (1) The authors carefully design a realistic SSCL setting where object-object correlations between labeled and unlabeled sets are maintained through a label super-class structure. And then, they develop the DistillMatch method combining knowledge distillation, pseudo-labels, out of distribution detection, and consistency regularization. (2) They show that  DistillMatch outperforms other existing methods on CIFAR-100 dataset, and ablation study results are shown also.