Gradient-based Editing of Memory Examples for Online Task-free Continual Learning

1 Introduction . Learning from a continuous stream of data – referred to as continual learning ( CL ) or lifelong learning – has recently seen a surge in interest , and many works have proposed ways to mitigate CL models ’ catastrophic forgetting of previously learned knowledge [ 20 , 32 , 33 ] . Here , we study online task-free CL [ 3 ] , where task identifiers and boundaries are absent from the data stream . This setting reflects many real-world data streams [ 6 , 25 ] and offers a challenging testbed for online CL research . Memory-based methods , a prominent class of approaches used for task-free continual learning , store a small number of training examples ( from the data stream ) in a memory and replay them at the later training iterations [ 33 , 34 ] . Existing methods operate over the original examples in the data-stream and focus on identifying samples to populate the memory [ 4 , 9 ] and finding samples in the memory to be replayed [ 2 ] . However , for continually updating models , using stored-seen examples in their original form , may lead to diminishing utility over time — i.e. , model may gradually memorize the stored examples after runs of replay , as the memory refreshes slowly . An alternate approach is to use generative models to create samples that suffers more from forgetting such as in GEN-MIR in [ 2 ] . In practice , training the generator network with limited data is challenging and leads to low-quality generated examples . Further , in the online learning setup , the generative model itself suffers from forgetting . As such , generative models perform worse than their memory counter-parts . In this paper , we present a novel memory-based CL framework , Gradient based Memory EDiting ( GMED ) , which looks to directly “ edit ” ( via a small gradient update ) examples stored in the replay memory . These edited examples are stored ( replacing their unedited counterparts ) , replayed , and further edited , thereby making the future replays more effective in overcoming catastrophic forgetting . 1Code can be found at https : //github.com/INK-USC/GMED . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Since no explicit generative model is involved , GMED approach retains the advantages of memorybased methods and is straightforward to train only inducing a small computation overhead . The main consideration in allowing “ editing ” via a gradient update is the choice of the optimization objective . In light of recent work on designing alternative replay strategies [ 2 , 7 , 40 ] , we hypothesize that “ interfering ” examples ( i.e. , past examples that suffer from increased loss ) should be prioritized for replay . For a particular stored example , GMED finds a small update over the example ( “ edit ” ) such that the resulting edited example yields the most increase in loss when replayed . GMED additionally penalizes the loss increase in the edited example to enforce the proximity of the edited example to the original sample , so that the edited examples stay in-distribution . As a result , replaying these edited examples is more effective in overcoming catastrophic forgetting . Since GMED focuses only on editing the stored examples , by construction , GMED is modular , i.e. , it can be seamlessly integrated with other state-of-the-art memory-based replay methods [ 2 , 5 , 26 ] . We demonstrate the effectiveness of GMED with a comprehensive set of experiments over six benchmark datasets . In general , combining GMED with existing memory-based approaches results in consistent and statistically significant improvements with our single best method establishing a new state-of-art performance on five datasets . Our ablative investigations reveal that the gains realized by GMED are significantly larger than those obtained from regularization effects in random perturbation , and can be accumulated upon data augmentation to further improve performance . To summarize , our contributions are two-fold : ( i ) we introduce GMED , a modular framework for taskfree online continual learning , to edit stored examples and make them more effective in alleviating catastrophic forgetting ( ii ) we perform intensive set of experiments to test the performance of GMED under various datasets , parameter setups ( e.g. , memory size ) and competiting baseline objectives . 2 Related Work . Continual Learning studies the problem of learning from a data stream with changing data distributions over time [ 20 , 23 ] . A major bottleneck towards this goal is the phenomenon of catastrophic forgetting [ 33 ] where the model “ forgets ” knowledge learned from past examples when exposed to new ones . To mitigate this effect , a wide variety of approaches have been investigated such as adding regularization [ 1 , 19 , 30 , 43 ] , separating parameters for previous and new data [ 24 , 36 , 37 ] , replaying examples from memory or a generative model [ 26 , 33 , 38 ] , meta-learning [ 18 ] . In this work , we build on memory-based approaches which have been more successful in the online task-free continual learning setting that we study . Online Task-free Continual Learning [ 3 ] is a specific formulation of the continual learning where the task boundaries and identities are not available to the model . Due to its broader applicability to realworld data-streams , a number of algorithms have been adapted to the task-free setup [ 2 , 14 , 15 , 22 , 44 ] . In particular , memory-based CL algorithms which store a subset of examples and later replay them during training , have seen elevated success in the task-free setting . Improvements in this space have focused on : storing diverse examples as in Gradient-based Sample Selection ( GSS ) [ 4 ] , and replaying examples with larger estimated “ interference ” as in Maximally Interfered Retrieval ( MIR ) with experience replay [ 2 ] . In contrast , GMED is used in conjunction with memory-based approaches and explicitly searches for an edited example which is optimized to be more “ effective ” for replay . Replay Example Construction . Moving away from replaying real examples , a line of works on deep generative replay [ 17 , 35 , 38 ] generates synthetic examples to replay with a generative model trained online . GEN-MIR [ 2 ] is further trained to generate examples that would suffer more from interference for replay . However , training a generative network is challenging , even more so in the online continual learning setup where the streaming examples are encountered only once leading to poorly generated examples . Moreover , the forgetting of generative networks themselves can not be perfectly mitigated . As a result , these methods generally perform worse than their memory-based counter-parts . Instead of generating a new example via a generator network , GMED uses gradient updates to directly edit the stored example thereby retaining advantages of memory-based techniques while creating new samples . Novel examples can also be constructed via data augmentation ( e.g . random crop , horizontal flip ) to help reduce over-fitting over the small replay memory [ 5 ] . Unlike GMED , these data-augmentations are usually pre-defined and can not adapt to the learning pattern of the model . Constructing edited examples has also been used for adversarial robustness [ 31 , 39 ] . The key difference lies in the optimization objective : while adversarial construction focuses on finding mis-classified examples [ 13 , 28 ] , GMED aims at constructing “ interfered ” examples that would suffer from loss increase in future training steps . 3 Method . We introduce the formulation of online task-free continual learning ( Sec . 3.1 ) and then present our gradient-based memory editing method GMED ( Sec . 3.2 ) and detail how GMED can be integrated with experience replay ( Sec 3.3 ) and other memory-based CL algorithms ( Sec . 3.4 ) . 3.1 Problem Formulation . In continual learning ( CL ) , we consider a ( potentially infinite ) stream D of labeled examples ( x , y ) , having a non-stationary data distribution – i.e. , the data distribution p ( x , y ) evolves over time . Let ( xt , yt ) denote the labeled example ( or a mini-batch of examples ) received by the model at time-step t in the data-stream D. We assume , for simplicity , that ( xt , yt ) is generated by first sampling a latent “ task ” z ∼ p ( z ; t ) , followed by sampling a data example from a conditional data distribution p ( x , y|z ) , – i.e. , ( xt , yt ) ∼ p ( x , y|z ) . Here p ( z ; t ) is non-i.i.d and time-dependent . In task-free CL setting , the latent task z is not revealed to the model . Finally , we emphasize that while models in task-aware setup proceed to the next task only after convergence on the current task [ 19 , 43 ] , this work focuses on online task-free CL setup [ 8 , 9 ] . It simulates a practical setting where models must perform online update on every incoming example , without accumulating examples within a task . Following above definitions , our goal is to learn a classification model f ( x ; θ ) on the data stream D that can preserve its performance over all the tasks inD . At time step t during the training process , the model is updated to minimize a predefined empirical loss ` ( xt , yt ; θ ) on the newly received example ( xt , yt ) , without increasing the loss on the previously visited examples ( before t ) . Specifically , let pc ( x , y ; T ) denotes the distribution of the examples visited until the time step T . We look to minimize the expected loss Epc ( x , y ; T ) ` ( x , y ; θ ) – i.e. , retaining performance on tasks encountered before T . 3.2 Gradient based Memory Editing ( GMED ) . In online task-free continual learning , examples visited earlier can not be accessed ( revisited ) and thus computing the loss over all the visited examples ( in D ) is not possible . To deal with this challenge , memory-based CL algorithms store and ( continuously ) maintain a set of visited examples in a fixed-size memory and use them for replay or regularization in the future training steps [ 26 , 33 ] . For instance , in Experience Replay ( ER ) [ 33 ] the examples are randomly sampled from the memory for replay ; whereas recent works explore more sophisticated replay strategies , such as Maximally Interfered Retrieval with experience replay ( ER-MIR ) [ 2 ] where memory examples which interfere the most with the newly visited example are selected for replay . However , these algorithms only train over samples drawn from a small replay memory in their original form . As such , the utility of the stored examples could diminish over time as the model could potentially memorize these examples if the memory examples are refreshed in a slow pace – which is found often the case [ 5 ] . We address the above limitation in our proposed approach Gradient based Memory Editing ( GMED ) by allowing examples stored in the memory to be edited in the continuous input space , illustrated in Figure 1 . The editing step is guided by an optimization objective instead of being pre-defined and involves drawing examples from the memory , editing the drawn examples , replaying the edited examples and at the same time writing the edited examples back to the memory . We now state our optimization objectives of example editing followed by algorithmic details of GMED . Algorithm 1 : Gradient Memory EDiting with ER ( ER+GMED ) 1 : Input : learning rate τ , edit stride α , regularization strength β , decay rate γ , model parameters θ 2 : Receives : stream example ( xD , yD ) 3 : Initialize : replay memory M 4 : for t = 1 to T do 5 : //when γ = 1 , k is not required 6 : ( xm , ym ) ∼M ; k ← replayed_time ( xm , ym ) 7 : ` before ← loss ( xm , ym , θt ) ; 8 : ` stream ← loss ( xD , yD , θt ) 9 : //update model with stream examples 10 : θ′t ← SGD ( ` stream , θt , τ ) 11 : //evaluate forgetting of memory examples 12 : ` after ← loss ( xm , ym , θ′t ) 13 : d← ` after − ` before 14 : //edit memory examples 15 : x′m ← xm + γkα∇x ( d− β ` before ) 16 : ` = loss ( { ( x′m , ym ) , ( xD , yD ) } , θt ) 17 : θt+1 ← SGD ( ` , θt , τ ) 18 : replace ( xm , ym ) with ( x′m , ym ) in M 19 : reservoir_update ( xD , yD , M ) 20 : end for Editing Objective . Clearly , the most crucial step involved in GMED is identifying “ how ” should the stored examples in the memory be edited to reduce catastrophic forgetting of early examples . If ( xm , ym ) denotes an example drawn from the memory M , the goal is to design a suitable editing function φ to generate the edited example x̂m where x̂m=φ ( xm ) . Such “ editing ” process can also be found in white-box adversarial example construction for robustness literature [ 13 ] where the task is to find an “ adversarial ” example ( x′ , y ) that is close to an original example ( x , y ) but is mis-classified by the model . Typically , adversarial example construction utilizes the gradients obtained from the classifier and edits the examples to move towards a different target class . While , in the context of continual learning , the editing objective should be different . We employ a similar hypothesis as previous works [ 2 , 7 , 40 ] that examples that are likely forgotten by models should be prioritized for replay . Accordingly , we edit examples so that they are more likely to be forgotten in future updates . With this objective of editing in place , we detail the process of incorporating GMED with Experience Replay ( ER ) .

Summary: This paper proposes a replay-based method (GMED) for tackling the challenging and important problem of task-free continual learning (CL), which is concerned with mitigating catastrophic forgetting when learning from a non-i.i.d. stream of data without knowledge of task boundaries. The key idea of the method is to *edit* samples from the replay buffer during training in a way that maximises the interference that would be caused to the performance on these examples by a gradient update on the minibatch from the current task. The method is motivated by previous work that has shown that prioritising replay of samples that would be most interfered with mitigates catastrophic forgetting (e.g.[1]).  Experiments are run demonstrating that GMED yields a statistically significant performance increase on a number of standard image recognition CL tasks (with discrete task boundaries and also with fuzzy task boundaries, which is more challenging and less common) when combined with a handful of existing replay-based methods ((namely experience replay (ER), ER with augmentations and Maximally Interfered Retrieval (MIR)). Further experiments are run to show that the method is robust to different memory sizes and to a range of values for the hyperparameters that the method uses to balance terms in the editing update. Additionally, experiments are also run to show that the type of editing used by GMED results in a better performance than random edits or adversarial edits to replay samples.  [1] Aljundi, Rahaf, et al. "Online Continual Learning with Maximal Interfered Retrieval." Advances in Neural Information Processing Systems 32 (2019): 11849-11860.