Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System

1 INTRODUCTION . Continual learning ( CL ) refers to the ability of a learning agent to continuously interact with a dynamic environment and process a stream of information to acquire new knowledge while consolidating and retaining previously obtained knowledge ( Parisi et al. , 2019 ) . This ability to continuously learn from a changing environment is a hallmark of intelligence and a critical missing component in our quest towards making our models truly intelligent . The major challenge towards enabling CL in deep neural networks ( DNNs ) is that the continual acquisition of incrementally available information from non-stationary data distributions generally leads to catastrophic forgetting whereby the performance of the model on previously learned tasks drops drastically as it learns new tasks ( McCloskey & Cohen , 1989 ) . Several approaches have been proposed to address the issue of catastrophic forgetting in CL . These can be broadly categorized into regularization-based methods ( Farajtabar et al. , 2020 ; Kirkpatrick et al. , 2017 ; Ritter et al. , 2018 ; Zenke et al. , 2017 ) which penalizes changes in the network weights , network expansion-based methods ( Rusu et al. , 2016 ; Yoon et al. , 2017 ) which dedicate a distinct set of network parameters to distinct tasks , and rehearsal-based methods ( Chaudhry et al. , 2018 ; Lopez-Paz & Ranzato , 2017 ) which maintains a memory buffer and replays samples from previous tasks . Amongst these , rehearsal-based methods have proven to be more effective in challenging CL tasks ( Farquhar & Gal , 2018 ) . However , an optimal approach for replaying memory samples and constraining the model update to efficiently consolidate knowledge remains an open question . In the brain , the ability to continually acquire , consolidate , and transfer knowledge over time is mediated by a rich set of neurophysiological processing principles ( Parisi et al. , 2019 ; Zenke et al. , 2017 ) and multiple memory systems ( Hassabis et al. , 2017 ) . In particular , the CLS theory ( Kumaran et al. , 2016 ) posits that efficient learning requires two complementary learning systems : the hippocampus exhibits short-term adaptation and rapid learning of episodic information which is then gradually consolidated to the neocortex for slow learning of structured information . Furthermore , a recent study by Hayes et al . ( 2021 ) identified the missing elements of biological reply in the replay ∗Contributed equally . mechanisms employed in DNNs for CL . They highlight that many existing approaches only focus on modeling the prefrontal cortex directly and do not have a fast learning network which plays a critical role in enabling efficient CL in the brain . Inspired by these studies , we hypothesize that mimicking the slow and rapid adaptation of information and having an efficient mechanism for incorporating them into the working memory can enable better CL in DNNs . To this end , we propose a novel dual memory experience replay method based on the complementary learning systems theory in the brain , dubbed as CLS-ER . In addition to a small episodic memory , our method builds long-term and short-term semantic memories which mimic the rapid and slow adaptation of information ( Figure 1 ) . As the network weights encode the learned representations of the tasks ( Krishnan et al. , 2019 ) , the semantic memories are maintained by taking the exponential moving average of the working model ’ s weights to consolidate information across the tasks with varying time windows and frequencies . The semantic memories interact with the episodic memory to extract consolidated replay activation patterns and enforce a consistency loss on the update of the working model so that new knowledge is acquired while aligning the decision boundary of the working model with the decision boundaries of semantic memories . This maintains a balance between the plasticity and stability of the model for effective knowledge consolidation . CLS-ER provides a general CL method that does not utilize the task boundaries or make any strong assumption regarding the distribution of the data and tasks . We demonstrate the versatility and effectiveness of our method on a wide range of CL benchmark tasks as well as more challenging scenarios which simulate the complexities of CL in the real world . 2 RELATED WORK . The base method for the rehearsal-based approach , Experience Replay ( ER ) ( Riemer et al. , 2018 ) combines the memory samples with the task samples into the training batch . Several techniques have since been employed on top of ER . Meta Experience Replay ( MER ) ( Riemer et al. , 2018 ) considers replay as a meta-learning problem for maximizing the transfer from previous tasks and minimizing the interference . iCARL ( Rebuffi et al. , 2017 ) uses the nearest average representation of past exemplars to classify in an incrementally learned representation space . Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) formulates optimization constraints on the exemplars in memory . Gradient-based Sample Selection ( GSS ) ( Aljundi et al. , 2019 ) aims for memory sample diversity in the gradient space and provides a greedy selection approach . Function Distance Regularization ( FDR ) ( Benjamin et al. , 2018 ) saves the network response at the task boundaries and adds a consistency loss on top of ER . Dark Experience Replay ( DER++ ) relaxes this restriction and samples logits during the entire optimization trajectory . CLS has been used as a source of inspiration for dual memory learning systems in earlier works ( French , 1999 ; Robins , 1993 ) but they have not been shown to scale to current computer vision tasks ( Parisi et al. , 2019 ) . Recently , Rostami et al . ( 2019 ) utilizes a generative model to couple sequential tasks in a latent embedding space . Kamra et al . ( 2017 ) utilizes two generative models in a dual memory architecture . However , they utilize the task boundaries and generative replay has its own set of challenges as it is difficult to learn a faithful distribution and performs sub-par in comparison to instance-based replay methods on challenging CL settings . Generally , the inspiration from CLS theory in DNNs has been mostly limited to episodic memory and mimicking the rapid and slow learning mechanism is majorly ignored ( Hayes et al. , 2021 ) which we aim to address . 3 METHOD . We first provide an overview of the CLS theory for the brain and how we aim to mimic it for DNNs before introducing the main components of our method and the overall formulation . 3.1 COMPLEMENTARY LEARNING SYSTEM THEORY . The CLS theory posits that effective lifelong learning in the brain requires two complementary learning systems . The hippocampus rapidly encodes novel information as a short-term memory which is subsequently used to transfer and consolidate knowledge in the neocortex which gradually acquires structured knowledge representation as long-term memory through experience replay . The interplay between the functionality of the hippocampus and neocortex is crucial for concurrently learning efficient representations ( for better generalization ) and the specifics of instance-based episodic memory . 3.2 COMPLEMENTARY LEARNING SYSTEM BASED EXPERIENCED REPLAY . Inspired by the CLS theory , we propose a dual memory experience replay method , CLS-ER , which aims to mimic the interplay between fast learning and slow learning mechanisms for enabling effective CL in DNNs . Our method maintains short-term and long-term semantic memories of the encountered tasks which interact with the episodic memory for replaying the associated neural activities . The working model is updated so that it acquires new knowledge while aligning its decision boundary with the semantic memories to enable the consolidation of structured knowledge across the tasks . Figure 1 highlights the parallels between CLS theory and our method . Semantic Memories : Central to our method is the maintenance of two semantic memories which accumulate and consolidate information over long-term and short-term periods . As the acquired knowledge of the learned tasks is encoded in the weights of DNNs ( Krishnan et al. , 2019 ) , we aim to form our semantic memories by accumulating the knowledge encoded in the corresponding weights of the model as it sequentially learns different tasks . An efficient method for aggregating the weights of a model is provided by Mean Teacher ( Tarvainen & Valpola , 2017 ) which is a knowledge distillation approach that uses an exponential moving average ( EMA ) of the student ’ s weights during training as a teacher for semi-supervised learning . It can also be considered as forming a self-ensemble of the intermediate model states that leads to better internal representations . We adapt the Mean Teacher approach to build our semantic memories as it provides a computational and memory-efficient method for accumulating knowledge over the tasks . As CL involves learning tasks sequentially , the model weights at each training step can be considered as a student model specialized for a particular task . Therefore , averaging the weights during training can be considered as forming an ensemble of task-specific student models which effectively aggregates information across the tasks and leads to smoother decision boundaries . CLS-ER builds long-term ( stable model ) and short-term ( plastic model ) semantic memories by maintaining two EMA-weighted models over the working model ’ s weights . The stable model is updated less frequently with a larger window size so that it retains more information from the earlier tasks while the plastic model is updated more frequently with a smaller window size so that it adapts faster to information from new tasks ( Figure 2 ) . Section D further demonstrates the benefits of employing two semantic memories instead of a single semantic memory . Episodic Memory : Replay of samples from the previous tasks stored in a small episodic memory is a common approach in CL that has proven to be effective in mitigating catastrophic forgetting . As we aim to position CLS-ER as a versatile general incremental learning method , we do not utilize the task boundaries or make any strong assumptions about the distribution of the tasks or samples . Therefore , to maintain a fixed episodic memory buffer , we employ reservoir sampling ( Vitter , 1985 ) which assigns equal probability to each sample in the stream for being represented in the buffer and randomly replaces the existing memory samples ( Algorithm 2 ) . It is a global distribution matching strategy that ensures that at any given time the distribution of samples in the buffer will approximately match the distribution of all the samples seen so far ( Isele & Cosgun , 2018 ) . Consolidation of Information : The key challenge in CL is the consolidation of new information with the previously acquired information . This requires an effective balance between the stability and plasticity of the model . Furthermore , the sharp change in decision boundary as a new task is learned makes the consolidation of information over tasks more challenging . CLS-ER tackles these challenges through a novel dual memory experience replay mechanism . The long-term and shortterm semantic memories interact with the episodic memory to extract the consolidated activations for the memory samples which are then utilized to constrain the update of the working model so that new knowledge is obtained whilst the decision boundary is aligned with the semantic memories . This prevents rapid changes in the parameter space as new tasks are learned . Furthermore , aligning the working model ’ s decision boundary with the semantic memories serves two goals : ( i ) helps in retaining and consolidating information and ( ii ) leads to a smoother adaptation of decision boundary . 3.3 FORMULATION . CLS-ER involves training a working model f ( . ; θw ) on a data stream D sampled from a non-iid distribution . Two additional EMA-weighted models are maintained as semantic memories : plastic model f ( . ; θP ) and the stable model f ( . ; θS ) . Finally , reservoir sampling ( Vitter , 1985 ) is employed to maintain a small episodic memoryM . At each training step , the working model receives the training batch Xb from the data stream and retrieves a random batch of exemplars Xm from the episodic memory . This is then followed by the retrieval of optimal semantic information , i.e . the structural knowledge encoded in the semantic memories which account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks . The semantic memories are designed so that the plastic model has higher performance on recent tasks whereas the stable model prioritizes retaining information on the older tasks . Therefore , we would prefer to use the logits from the stable model ZS for older exemplars and the plastic model ZP for recent exemplars . As CLS-ER is a general incremental learning method , instead of using a hard threshold or task information , we opt for a simple task-agnostic approach of using the performance of the semantic memories on the exemplars as a selection criterion that empirically works well . For each exemplar , we select the replay logits Z based on which model has the highest softmax score for the ground-truth class ( line 5-6 in Algorithm 1 ) . The selected replay logits from the semantic memories are then used to enforce a consistency loss on the working model so that it does not deviate from the already learned experiences . Hence , the working model is updated with a combination of the cross-entropy loss on the union of the data stream and episodic memory samples , X , and the consistency loss on the exemplars Xm , L = LCE ( σ ( f ( X ; θW ) ) , Y ) + λLMSE ( f ( Xm ; θW ) , Z ) ( 1 ) Algorithm 1 Complementary Learning System-Experience Replay Algorithm Input : Data stream D , Learning rate η , Consistency weight λ , Update rates rP and rS , Decay parameters αP and αS Initialize : θW = θP = θS M←− { } 1 : while Training do 2 : ( Xb , Yb ) ∼ D 3 : ( Xm , Ym ) ∼M 4 : ( X , Y ) = { ( Xb , Yb ) , ( Xm , Ym ) } 5 : ZP , ZS ←− f ( Xm ; θP ) , f ( Xm ; θS ) . Select optimal semantic memory 6 : Z ←− ZP if σ ( ZP ) ( Ym ) > σ ( ZS ) ( Ym ) else ZS 7 : L = LCE ( σ ( f ( X ; θW ) ) , Y ) + λLMSE ( f ( Xm ; θW ) , Z ) . Update working model 8 : θW ←− θW − η∇θWL 9 : a , b ∼ U ( 0 , 1 ) . Update semantic memories 10 : θP ←− αpθP + ( 1− αP ) θW if a < rP else θP 11 : θS ←− αSθS + ( 1− αS ) θW if b < rS else θS 12 : M←− Reservoir ( M , ( Xb , Yb ) ) . Update episodic memory ( Algorithm 2 ) return θW , θP , θS where σ is the softmax function , λ the regularization parameter , and LMSE the mean squared error loss used as consistency term . After updating the working model , we stochastically update the plastic and stable models with rates rP and rS ( note that rP ≥ rS so that the plastic model is updated more frequently ) . Using a stochastic rather than a deterministic approach is more biologically plausible ( Maass , 2014 ) which reduces the overlap in the snapshots of the working model and leads to more diversity in semantic memories . The semantic memories are updated by taking an exponential moving average of the working model ’ s weights ( Tarvainen & Valpola , 2017 ) with decay parameters αP and αS , respectively . θi = αiθi + ( 1− αi ) θW , i ∈ { P , S } ( 2 ) Note that αP ≤ αS so that the plastic model mimics the rapid adaptation of information while the stable model mimics slow acquisition of structured knowledge . For inference , we use the stable model as it retains long-term memory across the tasks , consolidates structural knowledge , and learns efficient representations for generalization ( Figure 1 ) . Details of our proposed method are provided in Algorithm 1 .

Summary: The paper presents CLS-ER, a dual memory mechanism for the continual learning setting. During training the model receives data from a non-i.i.d. source as well as random samples from a sample buffer (the episodic memory). The stable and plastic models act as teacher models. Their logits are the target of the student model. Which one is used as a target depends on the highest softmax score w.r.t. a ground trough class. The teacher networks are both exponential moving averages of the previous student models. Thus, the only difference between the teacher models "plastic" and "stable" is their EMA rates which in the case of the plastic model is larger such that it focuses on more recent weight updates.  The slower learning of the stable model is the mechanism which ought to mirror the consolidation of structural knowledge. It is used from inference Samples from the datastream are transferred to the sample buffer using reservoir sampling.   The authors experiment on image datasets (variants of MNIST, CIFAR10, Tiny-ImageNet, see the Mammoth framework) with data augmentation (random flips and crops). They compare with several prio works and the evaluation is thorough and convincing.