New Insights on Reducing Abrupt Representation Change in Online Continual Learning

1 INTRODUCTION . Continual learning is concerned with building models that can learn and accumulate knowledge and skills over time . A continual learner receives training data sequentially , from a potentially changing distribution , over the course of its learning process . The distribution change might be either a shift in the input domain or new categories being learned . The main challenge is to design models that can learn how to use the new data and acquire new knowledge , while preserving or improving the performance on previously learned data . While different settings have been investigated of how new data are being received and learned , we focus on the challenging scenario of learning from an online stream of data with new classes being introduced at unknown points in time and where memory and compute constraints are applied on the learner . Additionally , we assume a shared output layer among all the learned classes ( Aljundi et al. , 2019b ) . This setting is different and harder than the conventional multi-head setting ( Farquhar & Gal , 2018 ) where each new group of classes is considered as a new task with a dedicated head ( classification layer ) , requiring a task oracle at test time to activate the correct head . The axes of our setting ( online learning , no task boundary , no test time oracle , constant memory , and bounded compute ) align with the main desiderata of continual learning as described in De Lange et al . ( 2019 ) . Catastrophic forgetting ( McCloskey & Cohen , 1989 ) , where previous knowledge is overwritten as new concepts are learned , remains a key challenge in the online continual learning setting . To prevent forgetting , methods usually rely on storing a small buffer of previous training data and replaying samples from it as new data is learned . This can partially counteract catastrophic forgetting , but ∗Corresponding Author lucas.page-caccia @ mail.mcgill.ca 1Code to reproduce experiments is available at www.github.com/pclucas14/AML still tends to lead to large disruptions in accuracy , particularly at the initial task boundary or shift in distribution . Various works focus on studying which samples to store ( Borsos et al. , 2020 ; Aljundi et al. , 2019b ) or which samples to replay when receiving new data ( Aljundi et al. , 2019a ) . In this work , we direct our attention to the representations being learned and investigate how the features of previously learned classes change and drift over time . Consider the time point in a stream when a new class is introduced after previous classes have been well learned . If we consider the representation being learned , incoming samples from new classes are likely to be dispersed , potentially near and between representations of previous classes , while the representations of previous classes will typically cluster according to their class . Indeed , one might expect minimal changes to the learned representation of the previous classes , while the new classes samples are pushed away from the clusters of old class data . However , with a standard Experience Replay ( ER ) algorithm ( Chaudhry et al. , 2019 ) , we observe that it is the representations of older classes that is heavily perturbed after just a few update steps when training on the new class samples . We hypothesize that the fundamental issue arises from the combination of : new class samples representations lying close to older classes and the loss structure of the standard cross entropy applied on a mix of seen and unseen classes . We illustrate the observed effect in Fig . 1 ( left ) . This behavior is exacerbated especially in the regime of low buffer size . With larger replay buffers , the learner can recover knowledge about the prior classes over time , while with smaller buffers the initial disruptive changes in representations are challenging to correct . Indeed we illustrate this effect in Fig . 1 ( right ) , we see that ER only recovers from the initial displacement given a much larger buffer size . In standard continual learning with replay ( Aljundi et al. , 2019a ; Chaudhry et al. , 2019 ) the same loss function is usually employed on both the newly received samples and the replayed samples . In contrast , we propose a simple and efficient solution to mitigate this representation drift by using separate losses on the incoming stream and buffered data . The key idea is to allow the representations of samples from new classes to be learned in isolation of the older ones first , by excluding the previously learned classes from the incoming data loss . The discrimination between the new classes and the older ones is learned through the replayed batches , but only after incoming data has been learned , added to the buffer , and made available for replay . To allow more direct control of the structure in representations we first consider a metric learning based loss for the incoming data , proposed in Khosla et al . ( 2020 ) , where we propose to exclude samples of previously learned classes from the negative samples . We show that this type of negative selection is critical , and in contrast issues arise when negative examples are sampled uniformly from the buffer . These issues mimic those seen with standard losses in experience replay ( ER ) ( Aljundi et al. , 2019a ) . On the other hand we use a different loss on replay buffer data that is allowed to consider new and old classes , thereby consolidating knowledge across current and previous tasks . We call this overall approach ER with asymmetric metric learning ( ER-AML ) . Since cross entropy losses can be more efficient in training for classification than metric learning and contrastive losses ( avoiding positive and negative selection ) and it is widely used in incremental and continual learning , we also propose an alternative cross entropy solution that similarly applies an asymmetric loss between incoming and replay data . Notably , the cross entropy applied to the incoming data only considers logits of classes of the incoming data . This variant , named ER with asymmetric cross-entropy ( ER-ACE ) , along with ER-AML show strong performance , with little disruption at task boundaries Fig . 1 ( right ) . We achieve state of the art results in existing benchmarks while beating different existing methods including the traditional ER solution with an average relative gain of 36 % in accuracy . Our improvements are especially high in the small buffer regime . We also show that the mitigation of the old representation drift does not hinder the ability to learn and discriminate the new classes from the old ones . This property emerges from only learning the incoming data in isolation ; as we will see , also isolating the rehearsal step ( as in Ahn et al . ( 2020 ) ) leads to poor knowledge acquisition on the current task . Furthermore we show our ER-ACE objective can be combined with existing methods , leading to additional gains . Finally , we take a closer look at the computation cost of various existing methods . We show that some methods , while obtaining good performance under standard evaluation protocols , fail to meet the computational constraints required in online CL . We provide an extensive evaluation of computational and memory costs across several baselines and metrics . To summarize , our contributions are as follows . We first highlight the problem of representation drift in the online continual learning setting . We identify a root cause of this issue through an extensive empirical analysis ( Sec . 4.2 ) . Second , we propose a new family of methods addressing this issue by treating incoming and past data asymmetrically ( Sec . 4.1 , 4.3 ) . Finally , we show strong gains over replay baselines in a new evaluation framework designed to monitor real world constraints ( Sec . 5 ) . To the best or our knowledge , we are the first to report the computation costs of different methods in our setting , revealing new insights . 2 RELATED WORK . Research on continual learning can be divided based on the sequential setting being targeted ( see Zeno et al . ( 2018 ) ; van de Ven & Tolias ( 2019 ) ; Normandin et al . ( 2021 ) ; Lesort et al . ( 2021 ) for categorizations of the settings and De Lange et al . ( 2019 ) for a broad survey on continual learning ) . Earlier works consider the relaxed setting of task incremental learning ( Aljundi et al. , 2017 ; Serrà et al. , 2018 ; Li & Hoiem , 2016 ) where the data stream is divided into chunks of tasks and each task is learned offline with multiple iterations over the data of this task . While this setting is easier to handle as one task can be learned entirely , it limits the applicability of the solution . In this work , we consider the challenging setting of an online stream of non-i.i.d . data where changes can anytime occur in the input domain or in the output space . This more realistic setting has attracted increasing interest lately ( Lopez-Paz et al. , 2017 ; Aljundi et al. , 2019a ) . Specifically , we study the single-head ( or shared head ) setting , where when queried , the learner is not told which task the sample belongs to ( as opposed to the multi-head setting ) . The single-head assumption is further studied in task-agnostic continual learning settings ( He et al. , 2019 ; Caccia et al. , 2020 ; Ostapenko et al. , 2021 ; Von Oswald et al. , 2021 ) in which the task-boundary assumption , amongst others , is also relaxed . Many of the solutions to the online continual learning problem rely on the use of a buffer formed of previous memories which are replayed alongside new data during the learning process . Several works ( Borsos et al. , 2020 ; Chaudhry et al. , 2019 ; Aljundi et al. , 2019b ) propose solutions to select which samples should be stored , or retrieved for replay ( Aljundi et al. , 2019a ) , or both ( Shim et al. , 2021 ) . Lopez-Paz et al . ( 2017 ) ; Chaudhry et al . use replay to perform constrained optimization , limiting interference with previous tasks as new ones are learned . Our work , on the other hand , focuses on the appropriate loss function in this context . Tang & Matteson ( 2020 ) propose a graph-based approach that capture pairwise similarities between samples . Dark Experience Replay ( DER ) ( Buzzega et al. , 2020 ) suggests an alternative replay loss . Samples are stored along with their predicted logits and once replayed the current model is asked to keep its output close to the previously recorded logits . While the method is simple and effective it is worth noting that it relies heavily on data augmentation . Our work is orthogonal and can be combined with DER as we show in Sec . D. Finally , concurrent work ( Mai et al. , 2021 ) also use a contrastive loss for online continual learning , but not in an asymmetric fashion . In our work we also investigate the underlying causes for performance degradation in replay-based methods . Related to this study are works in the class incremental setting , where similar to our case a shared output layer is used , but classes are learned offline . Works in this area address the implicit class imbalance issue occurring when new classes are learned alongside replayed data . Zhao et al . ( 2019 ) proposes to correct last layer weights after a group of classes is learned via adjusting the weights norm . Wu et al . ( 2019 ) suggests to deploy extra additional parameters in order to linearly correct the “ bias ” in the shared output layer . Those parameters are learned at the end of each training phase . Hou et al . ( 2019 ) considers addressing this imbalance through applying cosine similarity based loss as opposed to the typical cross entropy loss along with a distillation loss and a margin based loss with negatives mining to preserve the feature of previous classes . Recently , Ahn et al . ( 2020 ) propose to learn the incoming tasks and the previous tasks separately . They use a masked softmax loss for the incoming and rehearsal data , to counter the class imbalance . All the methods highlighted above operate in the offline setting , where data from the current task can be revisited as needed making the disruptive issues emphasized at the task boundary less critical . In this paper , we focus on the online setting , with potentially overlapping tasks . As we will see , work by Ahn et al . ( 2020 ) developed to counter class imbalance , can inhibit learning of the current task in the online setting ( see Appendix B ) . Lastly , Zeno et al . ( 2018 ) uses a logit masking related to our method but their context is based on the multi-head setting , and does not consider replay based methods , where learning across tasks occurs . Their goal is to activate only the head of which the samples within the new batch belong to . However , our approach is more general and it applies to the single head setting ( where we have a single output layer for all classes , and no task oracle . )

Summary: In the online continual learning setting, representation of previous classes will change over time, especially at the task boundary. The authors present a new explanation for the abrupt representation change when applying experience replay (ER) algorithm. They claim that without sufficient training, the representation of new samples is initially mixed with that of previous classes, which causes a sharp increase of gradient resulting shift. To reduce the negative impact of negative new samples, they modify the ER by leveraging asymmetric losses on current data and replay data separately.