Gradient Based Memory Editing for Task-Free Continual Learning

1 INTRODUCTION . Accumulating past knowledge and adapting to evolving environments are one of the key traits in human intelligence ( McClelland et al. , 1995 ) . While contemporary deep neural networks have achieved impressive results in a range of machine learning tasks Goodfellow et al . ( 2015 ) , they haven ’ t yet manifested the ability of continually learning over evolving data streams ( Ratcliff , 1990 ) . These models suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ; Robins , 1995 ) when trained in an online fashion—i.e. , performance drops over previously seen examples during the sequential learning process . To this end , continual learning ( CL ) methods are developed to alleviate catastrophic forgetting issue when models are trained on non-stationary data streams ( Goodfellow et al. , 2013 ) . Most existing work on continual learning assume that , when models are trained on a stream of tasks sequentially , the task specifications such as task boundaries or identities are exposed to the models . These task-aware CL methods make explicit use of task specifications to avoid catastrophic forgetting issue , including consolidating important parameters on previous tasks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Nguyen et al. , 2018 ) , distilling knowledge from previous tasks ( Li & Hoiem , 2017 ; Rannen et al. , 2017 ) , or separating task-specific model parameters ( Rusu et al. , 2016 ; Serrà et al. , 2018 ) . However , in practice , it is more likely that the data instances comes in a sequential , non-stationary fashion without task identity or boundary—a setting that is commonly termed as task-free continual learning ( Aljundi et al. , 2018 ) . To tackle this setting , recent attempts on task-free CL methods have been made ( Aljundi et al. , 2018 ; Zeno et al. , 2018 ; Lee et al. , 2020 ) . These efforts revolve around regularization and model expansion based approaches , which rely on inferring task boundaries or identities ( Aljundi et al. , 2018 ; Lee et al. , 2020 ) and perform online paramater importance estimation ( Zeno et al. , 2018 ) , to consolidate or separate model parameters . In another line of efforts , memory-based CL methods have achieved strong results in task-free setting Aljundi et al . ( 2019b ) . These methods store a small set of previously seen instances in a fix-sized memory , and utilize them for replay ( Robins , 1995 ; Rolnick et al. , 2019 ) or regularization ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ) . The core problem in memory-based CL methods is how to manage the memory instances ( e.g. , which to replace with new instances ) and replay them given a restricted computation budget , so that the model performance can be maximally preserved or 1Code has been uploaded in the supplementary materials and will be published . enhanced . Prior work developing these methods have tried to either identify : 1 ) what instances to include in memory from a data stream ( Aljundi et al. , 2019b ; Rebuffi et al. , 2017 ; Chaudhry et al. , 2019b ) ; and 2 ) which instances in memory need to be replayed at what training step ( Aljundi et al. , 2019a ) . In this paper , we provide a new approach to solving the memory management problem in task-free continual learning by studying how to make gradient updates on stored memory examples . We develop a novel memory editing algorithm which complements existing memory-replay methods and data-sampling strategies for memory management ( updates ) . The challenge is to propose a plausible and sound optimization objective of editing . We employ the same intuition as previous study ( Toneva et al. , 2019 ; Chaudhry et al. , 2020 ; Aljundi et al. , 2019a ) : examples that are likely to be forgotten should be prioritized . Our proposed method , named Gradient-based Memory EDiting ( GMED ) , edits examples stored in the memory with gradient-based updates so that they are more likely to be forgotten . Specifically , we estimate the “ forgetting ” of a stored example by its loss increase in the upcoming one online model update . Finally , we perform gradient ascent on stored examples so that they are more likely to be forgotten . Experiments show that our algorithm consistently outperforms baselines on five benchmark datasets under various memory sizes . Our ablation study shows the proposed editing mechanism outperforms alternative editing strategies such as random editing . We demonstrate that the proposed algorithm is general enough to be used with other strong ( more recent ) memory-based CL methods to further enhance performance , thus allowing for improvements in many benchmark datasets . 2 RELATED WORKS . Task-aware Continual Learning . Most of continual learning algorithms are studied under “ taskaware ” settings , where the model visits a sequence of clearly separated “ tasks ” . A great portion of algorithms make explicit use of task boundaries ( Kirkpatrick et al. , 2017 ; Rusu et al. , 2016 ; Lopez-Paz & Ranzato , 2017 ) , by learning separate parameters for each task , or discourage changes of parameters that are important to old tasks . Existing continual learning algorithms can be summarized into three categories : regularization-based , architecture-based and data-based approaches . Regularization based approaches ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Nguyen et al. , 2018 ; Adel et al. , 2020 ) discourage the change of parameters that are important to previous data . Model expansionbased approaches ( Rusu et al. , 2016 ; Serrà et al. , 2018 ; Li et al. , 2019 ) allows expansion of model architecture to separate parameters for previous and current data . Data-based approaches ( Robins , 1995 ; Shin et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) replay or constrain model updates with real or synthetic examples . Task-free Continual Learning . Recently , task-free continual learning ( Aljundi et al. , 2018 ) have drawn increasing interest , where we do not assume knowledge about task boundaries . To the best of our knowledge , only a handful number of regularization based ( Zeno et al. , 2018 ; Aljundi et al. , 2018 ) , model-expansion based ( Lee et al. , 2020 ) , generative replay based ( Rao et al. , 2019 ) , continual meta-learning and meta-continual learning ( He et al. , 2019 ; Caccia et al. , 2020 ; Harrison et al. , 2020 ) approaches are applicable in the task-free CL setting . Meanwhile , most memory based continual learning algorithms are applicable to the task-free setting ( Aljundi et al. , 2019a ; b ) . Memory-based CL algorithms such as Experience Replay ( ER ) ( Robins , 1995 ) store a subset of examples in a fix-sized replay memory and utilize them later at training to alleviate forgetting . Recent research has studied online strategies to improve the performance gain when examples get replayed from two dimensions : in terms of which examples to store , and which examples to replay . For example , in terms of deciding which examples to store , Gradient based Sample Selection ( GSS ) ( Aljundi et al. , 2019b ) proposes to store most diverse examples . In terms of deciding which examples to replay , maximally Interfering Retrieval ( MIR ) ( Aljundi et al. , 2019a ) select examples with the largest estimated forgetting . In particular , a task-aware approach , Hindsight Anchor Learning ( HAL ) ( Chaudhry et al. , 2020 ) , shares the same assumption that forgettable examples should be prioritized more . However , HAL only applies to task-aware settings and requires extra memory storage to keep track of the learned anchors . Figure 1 shows a categorization of memory-based task-free continual learning . 3 PRELIMINARIES . In this section we first present the problem formulation of task-free continual learning and then introduce preliminaries on memory-based continual learning methods . 3.1 PROBLEM FORMULATION . In task-free continual learning , we consider a ( potentially infinite ) stream of data examples D , which have a non-stationary data distribution , i.e. , the data distribution P ( x , y ) over time.At each time step t , the model receives a single or a mini batch of labeled examples ( xt , yt ) from the data stream D. For simplicity , here we assume that example ( xt , yt ) from D is generated by : first sampling a latent “ task ” z ∼ P ( z ; t ) , followed by sampling a data example from a joint data distribution P ( x , y|z ; t ) that is conditioned on task z , i.e. , ( xt , yt ) ∼ P ( x , y|z ; t ) . Here P ( z ; t ) is non-i.i.d and time-dependent . Similarly , P ( x , y|z ; t ) also changes over time . The goal of task-free online continual learning is to seek a classification model f ( x ; θ ) , parameterized by θ , over new example ( s ) ( x , y ) from the data stream D that minimizes a predefined loss ` ( x , y ; θ ) while not increasing the loss on previously seen examples . This capability is evaluated by testing the model over a test set of all visited tasks . 3.2 MEMORY-BASED CL METHODS . Briefly , memory-based CL algorithms maintain a fix-sized replay memory M which is used to store ( subset of ) previously seen examples ( xt , yt ) from the stream D. When the memory is full , the algorithm needs to either identify a memory example ( x , y ) to be replaced by new example , or to discard the new example it just received . Following the same setup in previous memory-based CL methods , our experiments use reservoir sampling ( Vitter , 1985 ) to determine how the memory will be updated with new examples received from stream D. Every time the model receives a new example , it draws an integer j between 0 and N randomly , where N is the number of examples visited so far . If j < |M | ( i.e. , the memory size or budget ) , it replace the example at the j-th position in the memory with the new example ; otherwise , this newly received example will be discarded . Reservoir sampling ensures at each time step each visited example is kept with an equal probability |M |/N . At each time step t , the algorithm also needs to determine the memory examples to be used for replay . Similar to previous methods , we randomly sample one or a mini-batch of examples ( x , y ) from the memory M . As an alternative replay strategy , MIR ( Aljundi et al. , 2019a ) identifies a subset of memory examples based on a predefined optimization objective ( i.e , perform one step of training on ( x , y ) ) , and then replays the selected examples . 4 GRADIENT BASED MEMORY EDITING . We propose Gradient based Memory Editing ( GMED ) , a novel algorithm for updating stored memory examples in an online fashion . We state our hypothesis about which examples should be stored in Sec . 4.1 . We then formulate an online optimization objective for example editing in Sec . 4.2 . In Sec . 4.3 , we introduce algorithmic details of GMED and its integration with MIR .

Summary: This paper deals with continual learning. Specifically, given a stream of tasks we want to maximise performance across all tasks. Typically neural networks suffer from catastrophic forgetting which results in worse performance on tasks seen earlier in training. There are many proposed solutions to this problem. One specific set of approaches are "memory based" algorithms. Here we store some training examples in memory from the tasks seen thus far. These are then mixed in with new training data so as to encourage the model to not forget past tasks.