Gradient-based Editing of Memory Examples for Online Task-free Continual Learning

1 Introduction . Learning from a continuous stream of data – referred to as continual learning ( CL ) or lifelong learning – has recently seen a surge in interest , and many works have proposed ways to mitigate CL models ’ catastrophic forgetting of previously learned knowledge [ 20 , 32 , 33 ] . Here , we study online task-free CL [ 3 ] , where task identifiers and boundaries are absent from the data stream . This setting reflects many real-world data streams [ 6 , 25 ] and offers a challenging testbed for online CL research . Memory-based methods , a prominent class of approaches used for task-free continual learning , store a small number of training examples ( from the data stream ) in a memory and replay them at the later training iterations [ 33 , 34 ] . Existing methods operate over the original examples in the data-stream and focus on identifying samples to populate the memory [ 4 , 9 ] and finding samples in the memory to be replayed [ 2 ] . However , for continually updating models , using stored-seen examples in their original form , may lead to diminishing utility over time — i.e. , model may gradually memorize the stored examples after runs of replay , as the memory refreshes slowly . An alternate approach is to use generative models to create samples that suffers more from forgetting such as in GEN-MIR in [ 2 ] . In practice , training the generator network with limited data is challenging and leads to low-quality generated examples . Further , in the online learning setup , the generative model itself suffers from forgetting . As such , generative models perform worse than their memory counter-parts . In this paper , we present a novel memory-based CL framework , Gradient based Memory EDiting ( GMED ) , which looks to directly “ edit ” ( via a small gradient update ) examples stored in the replay memory . These edited examples are stored ( replacing their unedited counterparts ) , replayed , and further edited , thereby making the future replays more effective in overcoming catastrophic forgetting . 1Code can be found at https : //github.com/INK-USC/GMED . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Since no explicit generative model is involved , GMED approach retains the advantages of memorybased methods and is straightforward to train only inducing a small computation overhead . The main consideration in allowing “ editing ” via a gradient update is the choice of the optimization objective . In light of recent work on designing alternative replay strategies [ 2 , 7 , 40 ] , we hypothesize that “ interfering ” examples ( i.e. , past examples that suffer from increased loss ) should be prioritized for replay . For a particular stored example , GMED finds a small update over the example ( “ edit ” ) such that the resulting edited example yields the most increase in loss when replayed . GMED additionally penalizes the loss increase in the edited example to enforce the proximity of the edited example to the original sample , so that the edited examples stay in-distribution . As a result , replaying these edited examples is more effective in overcoming catastrophic forgetting . Since GMED focuses only on editing the stored examples , by construction , GMED is modular , i.e. , it can be seamlessly integrated with other state-of-the-art memory-based replay methods [ 2 , 5 , 26 ] . We demonstrate the effectiveness of GMED with a comprehensive set of experiments over six benchmark datasets . In general , combining GMED with existing memory-based approaches results in consistent and statistically significant improvements with our single best method establishing a new state-of-art performance on five datasets . Our ablative investigations reveal that the gains realized by GMED are significantly larger than those obtained from regularization effects in random perturbation , and can be accumulated upon data augmentation to further improve performance . To summarize , our contributions are two-fold : ( i ) we introduce GMED , a modular framework for taskfree online continual learning , to edit stored examples and make them more effective in alleviating catastrophic forgetting ( ii ) we perform intensive set of experiments to test the performance of GMED under various datasets , parameter setups ( e.g. , memory size ) and competiting baseline objectives . 2 Related Work . Continual Learning studies the problem of learning from a data stream with changing data distributions over time [ 20 , 23 ] . A major bottleneck towards this goal is the phenomenon of catastrophic forgetting [ 33 ] where the model “ forgets ” knowledge learned from past examples when exposed to new ones . To mitigate this effect , a wide variety of approaches have been investigated such as adding regularization [ 1 , 19 , 30 , 43 ] , separating parameters for previous and new data [ 24 , 36 , 37 ] , replaying examples from memory or a generative model [ 26 , 33 , 38 ] , meta-learning [ 18 ] . In this work , we build on memory-based approaches which have been more successful in the online task-free continual learning setting that we study . Online Task-free Continual Learning [ 3 ] is a specific formulation of the continual learning where the task boundaries and identities are not available to the model . Due to its broader applicability to realworld data-streams , a number of algorithms have been adapted to the task-free setup [ 2 , 14 , 15 , 22 , 44 ] . In particular , memory-based CL algorithms which store a subset of examples and later replay them during training , have seen elevated success in the task-free setting . Improvements in this space have focused on : storing diverse examples as in Gradient-based Sample Selection ( GSS ) [ 4 ] , and replaying examples with larger estimated “ interference ” as in Maximally Interfered Retrieval ( MIR ) with experience replay [ 2 ] . In contrast , GMED is used in conjunction with memory-based approaches and explicitly searches for an edited example which is optimized to be more “ effective ” for replay . Replay Example Construction . Moving away from replaying real examples , a line of works on deep generative replay [ 17 , 35 , 38 ] generates synthetic examples to replay with a generative model trained online . GEN-MIR [ 2 ] is further trained to generate examples that would suffer more from interference for replay . However , training a generative network is challenging , even more so in the online continual learning setup where the streaming examples are encountered only once leading to poorly generated examples . Moreover , the forgetting of generative networks themselves can not be perfectly mitigated . As a result , these methods generally perform worse than their memory-based counter-parts . Instead of generating a new example via a generator network , GMED uses gradient updates to directly edit the stored example thereby retaining advantages of memory-based techniques while creating new samples . Novel examples can also be constructed via data augmentation ( e.g . random crop , horizontal flip ) to help reduce over-fitting over the small replay memory [ 5 ] . Unlike GMED , these data-augmentations are usually pre-defined and can not adapt to the learning pattern of the model . Constructing edited examples has also been used for adversarial robustness [ 31 , 39 ] . The key difference lies in the optimization objective : while adversarial construction focuses on finding mis-classified examples [ 13 , 28 ] , GMED aims at constructing “ interfered ” examples that would suffer from loss increase in future training steps . 3 Method . We introduce the formulation of online task-free continual learning ( Sec . 3.1 ) and then present our gradient-based memory editing method GMED ( Sec . 3.2 ) and detail how GMED can be integrated with experience replay ( Sec 3.3 ) and other memory-based CL algorithms ( Sec . 3.4 ) . 3.1 Problem Formulation . In continual learning ( CL ) , we consider a ( potentially infinite ) stream D of labeled examples ( x , y ) , having a non-stationary data distribution – i.e. , the data distribution p ( x , y ) evolves over time . Let ( xt , yt ) denote the labeled example ( or a mini-batch of examples ) received by the model at time-step t in the data-stream D. We assume , for simplicity , that ( xt , yt ) is generated by first sampling a latent “ task ” z ∼ p ( z ; t ) , followed by sampling a data example from a conditional data distribution p ( x , y|z ) , – i.e. , ( xt , yt ) ∼ p ( x , y|z ) . Here p ( z ; t ) is non-i.i.d and time-dependent . In task-free CL setting , the latent task z is not revealed to the model . Finally , we emphasize that while models in task-aware setup proceed to the next task only after convergence on the current task [ 19 , 43 ] , this work focuses on online task-free CL setup [ 8 , 9 ] . It simulates a practical setting where models must perform online update on every incoming example , without accumulating examples within a task . Following above definitions , our goal is to learn a classification model f ( x ; θ ) on the data stream D that can preserve its performance over all the tasks inD . At time step t during the training process , the model is updated to minimize a predefined empirical loss ` ( xt , yt ; θ ) on the newly received example ( xt , yt ) , without increasing the loss on the previously visited examples ( before t ) . Specifically , let pc ( x , y ; T ) denotes the distribution of the examples visited until the time step T . We look to minimize the expected loss Epc ( x , y ; T ) ` ( x , y ; θ ) – i.e. , retaining performance on tasks encountered before T . 3.2 Gradient based Memory Editing ( GMED ) . In online task-free continual learning , examples visited earlier can not be accessed ( revisited ) and thus computing the loss over all the visited examples ( in D ) is not possible . To deal with this challenge , memory-based CL algorithms store and ( continuously ) maintain a set of visited examples in a fixed-size memory and use them for replay or regularization in the future training steps [ 26 , 33 ] . For instance , in Experience Replay ( ER ) [ 33 ] the examples are randomly sampled from the memory for replay ; whereas recent works explore more sophisticated replay strategies , such as Maximally Interfered Retrieval with experience replay ( ER-MIR ) [ 2 ] where memory examples which interfere the most with the newly visited example are selected for replay . However , these algorithms only train over samples drawn from a small replay memory in their original form . As such , the utility of the stored examples could diminish over time as the model could potentially memorize these examples if the memory examples are refreshed in a slow pace – which is found often the case [ 5 ] . We address the above limitation in our proposed approach Gradient based Memory Editing ( GMED ) by allowing examples stored in the memory to be edited in the continuous input space , illustrated in Figure 1 . The editing step is guided by an optimization objective instead of being pre-defined and involves drawing examples from the memory , editing the drawn examples , replaying the edited examples and at the same time writing the edited examples back to the memory . We now state our optimization objectives of example editing followed by algorithmic details of GMED . Algorithm 1 : Gradient Memory EDiting with ER ( ER+GMED ) 1 : Input : learning rate τ , edit stride α , regularization strength β , decay rate γ , model parameters θ 2 : Receives : stream example ( xD , yD ) 3 : Initialize : replay memory M 4 : for t = 1 to T do 5 : //when γ = 1 , k is not required 6 : ( xm , ym ) ∼M ; k ← replayed_time ( xm , ym ) 7 : ` before ← loss ( xm , ym , θt ) ; 8 : ` stream ← loss ( xD , yD , θt ) 9 : //update model with stream examples 10 : θ′t ← SGD ( ` stream , θt , τ ) 11 : //evaluate forgetting of memory examples 12 : ` after ← loss ( xm , ym , θ′t ) 13 : d← ` after − ` before 14 : //edit memory examples 15 : x′m ← xm + γkα∇x ( d− β ` before ) 16 : ` = loss ( { ( x′m , ym ) , ( xD , yD ) } , θt ) 17 : θt+1 ← SGD ( ` , θt , τ ) 18 : replace ( xm , ym ) with ( x′m , ym ) in M 19 : reservoir_update ( xD , yD , M ) 20 : end for Editing Objective . Clearly , the most crucial step involved in GMED is identifying “ how ” should the stored examples in the memory be edited to reduce catastrophic forgetting of early examples . If ( xm , ym ) denotes an example drawn from the memory M , the goal is to design a suitable editing function φ to generate the edited example x̂m where x̂m=φ ( xm ) . Such “ editing ” process can also be found in white-box adversarial example construction for robustness literature [ 13 ] where the task is to find an “ adversarial ” example ( x′ , y ) that is close to an original example ( x , y ) but is mis-classified by the model . Typically , adversarial example construction utilizes the gradients obtained from the classifier and edits the examples to move towards a different target class . While , in the context of continual learning , the editing objective should be different . We employ a similar hypothesis as previous works [ 2 , 7 , 40 ] that examples that are likely forgotten by models should be prioritized for replay . Accordingly , we edit examples so that they are more likely to be forgotten in future updates . With this objective of editing in place , we detail the process of incorporating GMED with Experience Replay ( ER ) .

Summary: This paper introduces the Gradient based Memory Editing (GMED) framework for task-free online continual learning that edits stored examples using gradient updates. GMED can be easily applied with other memory-based continual learning algorithms to improve their performance. The paper conducts several experiments to validate their method.