Differentiable Hebbian Consolidation for Continual Learning

1 INTRODUCTION . A key aspect of human intelligence is the ability to continually adapt and learn in dynamic environments , a characteristic which is challenging to embed into artificial intelligence . Recent advances in machine learning ( ML ) have shown tremendous improvements in various problems , by learning to solve one complex task very well , through extensive training on large datasets with millions of training examples or more . However , most of the ML models that are used during deployment in the real-world are exposed to non-stationarity where the distributions of acquired data changes over time . Therefore , after learning is complete , and these models are further trained with new data , responding to distributional changes , performance degrades with respect to the original data . This phenomenon known as catastrophic forgetting or catastrophic interference ( McCloskey & Cohen , 1989 ; French , 1999 ) presents a crucial problem for deep neural networks ( DNNs ) that are tasked with continual learning ( Ring , 1994 ) , also called lifelong learning ( Thrun & Mitchell , 1995 ; Thrun , 1998 ) . In continual learning , the goal is to adapt and learn consecutive tasks without forgetting how to perform well on previously learned tasks , enabling models that are scalable and efficient over long timescales . In most supervised learning methods , DNN architectures require independent and identically distributed ( iid ) samples from a stationary training distribution . However , for ML systems in realworld applications that require continual learning , the iid assumption is easily violated when : ( 1 ) There is concept drift in the training data distribution . ( 2 ) There are imbalanced class distributions and concept drift occuring simultaneously . ( 3 ) Data representing all scenarios in which the learner is expected to perform are not initially available . In such situations , learning systems face the “ stability-plasticity dilemma ” which is a well-known problem for artificial and biological neural networks ( Carpenter & Grossberg , 1987 ; Abraham & Robins , 2005 ) . This presents a continual learning challenge for an ML system where the model needs to provide a balance between its plasticity ( to integrate new knowledge ) and stability ( to preserve existing knowledge ) . In biological neural networks , synaptic plasticity has been argued to play an important role in learning and memory ( Howland & Wang , 2008 ; Takeuchi et al. , 2013 ; Bailey et al. , 2015 ) and two major theories have been proposed to explain a human ’ s ability to perform continual learning . The first theory is inspired by synaptic consolidation in the mammalian neocortex ( Benna & Fusi , 2016 ) where a subset of synapses are rendered less plastic and therefore preserved for a longer timescale . The general idea for this approach is to consolidate and preserve synaptic parameters that are considered important for the previously learned tasks . This is normally achieved through task-specific updates of synaptic weights in a neural network . The second is the complementary learning system ( CLS ) theory ( McClelland et al. , 1995 ; Kumaran et al. , 2016 ) , which suggests that humans extract highlevel structural information and store it in different brain areas while retaining episodic memories . Recent work on differentiable plasticity has shown that neural networks with “ fast weights ” that leverage Hebbian learning rules ( Hebb , 1949 ) can be trained end-to-end through backpropagation and stochastic gradient descent ( SGD ) to optimize the standard “ slow weights ” , as well as also the amount of plasticity in each synaptic connection ( Miconi , 2016 ; Miconi et al. , 2018 ) . These works use slow weights to refer to the weights normally used to train vanilla neural networks , which are updated slowly and are often associated with long-term memory . The fast weights represent the weights that are superimposed on the slow weights and change quickly from one time step to the next based on input representations . These fast weights behave as a form of short-term memory that enable “ reactivation ” of long-term memory traces in the slow weights . Miconi et al . ( 2018 ) showed that simple plastic networks with learned plasticity outperform networks with uniform plasticity on various problems . Moreover , there have been several approaches proposed recently for overcoming the catastrophic forgetting problem in fixed-capacity models by dynamically adjusting the plasticity of each synapse based on its importance for retaining past memories ( Parisi et al. , 2019 ) . Here , we extend the work on differentiable plasticity to the task-incremental continual learning setting ( van de Ven & Tolias , 2019 ) , where tasks arrive in a batch-like fashion , and have clear boundaries . We develop a Differentiable Hebbian Consolidation1 model that is capable of adapting quickly to changing environments as well as consolidating previous knowledge by selectively adjusting the plasticity of synapses . We modify the traditional softmax layer and propose to augment the slow weights in the final fully-connected ( FC ) layer ( softmax output layer ) with a set of plastic weights implemented using Differentiable Hebbian Plasticity ( DHP ) . Furthermore , we demonstrate the flexibility of our model by combining it with recent task-specific synaptic consolidation based approaches to overcoming catastrophic forgetting such as elastic weight consolidation ( Kirkpatrick et al. , 2017 ; Schwarz et al. , 2018 ) , synaptic intelligence ( Zenke et al. , 2017b ) and memory aware synapses ( Aljundi et al. , 2018 ) . Our model unifies core concepts from Hebbian plasticity , synaptic consolidation and CLS theory to enable rapid adaptation to new unseen data , while consolidating synapses and leveraging compressed episodic memories in the softmax layer to remember previous knowledge and mitigate catastrophic forgetting . We test our proposed method on established benchmark problems including the Permuted MNIST ( Goodfellow et al. , 2013 ) , Split MNIST ( Zenke et al. , 2017b ) and Vision Datasets Mixture ( Ritter et al. , 2018 ) benchmarks . We also introduce the Imbalanced Permuted MNIST problem and show that plastic networks with task-specific synaptic consolidation methods outperform networks with uniform plasticity . 2 RELEVANT WORK . Neural Networks with Non-Uniform Plasticity : One of the major theories that have been proposed to explain a human ’ s ability to learn continually is Hebbian learning ( Hebb , 1949 ) , which suggests that learning and memory are attributed to weight plasticity , that is , the modification of the strength of existing synapses according to variants of Hebb ’ s rule ( Paulsen & Sejnowski , 2000 ; Song et al. , 2000 ; Oja , 2008 ) . It is a form of activity-dependent synaptic plasticity where correlated activation of pre- and post-synaptic neurons leads to the strengthening of the connection between the two neurons . According to the Hebbian learning theory , after learning , the related synaptic strength are enhanced while the degree of plasticity decreases to protect the learned knowledge ( Zenke et al. , 2017a ) . Recent approaches in the meta-learning literature have shown that we can incorporate fast weights into a neural network to perform one-shot and few-shot learning ( Munkhdalai & Trischler , 2018 ; Rae et al. , 2018 ) . Munkhdalai & Trischler ( 2018 ) proposed a model that augments FC layers preceding the softmax with a matrix of fast weights to bind labels to representations . Here , the fast weights were implemented with non-trainable Hebbian learning-based associative memory . Rae et al . ( 2018 ) 1Code is available at : proposed a Hebbian Softmax layer that can improve learning of rare classes by interpolating between Hebbian learning and SGD updates on the output layer using an engineered scheduling scheme . Miconi et al . ( 2018 ) proposed differentiable plasticity , which uses SGD to optimize the plasticity of each synaptic connection , in addition to the standard fixed ( slow ) weights . Here , each synapse is composed of a slow weight and a plastic ( fast ) weight that automatically increases or decreases based on the activity over time . Although this approach served to be a powerful new method for training neural networks , it was mainly demonstrated on recurrent neural networks ( RNNs ) for solving pattern memorization tasks and maze exploration with reinforcement learning . Also , these approaches were only demonstrated on meta-learning problems and not the continual learning challenge of overcoming catastrophic forgetting . Our work also augments the slow weights in the FC layer with a set of plastic ( fast ) weights , but implements these using DHP . We only update the parameters of the softmax output layer in order to achieve fast learning and preserve knowledge over time . Overcoming Catastrophic Forgetting : This work leverages two strategies to overcome the catastrophic forgetting problem : 1 ) Task-specific Synaptic Consolidation — Protecting previously learned knowledge by dynamically adjusting the synaptic strengths to consolidate and retain memories . 2 ) CLS Theory — A dual memory system where , the neocortex ( neural network ) gradually learns to extract structured representations from the data while , the hippocampus ( augmented episodic memory ) performs rapid learning and individuated storage to memorize new instances or experiences . There have been several notable works inspired by task-specific synaptic consolidation for overcoming catastrophic forgetting ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017b ; Aljundi et al. , 2018 ) and they are often categorized as regularization strategies in the continual learning literature ( Parisi et al. , 2019 ) . All of these regularization approaches estimate the importance of each parameter or synapse , Ωk , where least plastic synapses can retain memories for long timescales and more plastic synapses are considered less important . The parameter importance and network parameters θk are updated in either an online manner or after learning task Tn . Therefore , when learning new task Tn+1 , a regularizer is added to the original loss function Ln ( θ ) , so that we dynamically adjust the plasticity w.r.t Ωk and prevent any changes to important parameters of previously learned tasks : L̃n ( θ ) = Ln ( θ ) + λ ∑ k Ωk ( θ n k − θn−1k ) 2 ︸ ︷︷ ︸ regularizer ( 1 ) where θn−1k are the learned network parameters after training on the previous n− 1 tasks and λ is a hyperparameter for the regularizer to control the amount of forgetting ( old versus new memories ) . The main difference in these regularization strategies is on the method used to compute the importance of each parameter , Ωk . In Elastic Weight Consolidation ( EWC ) , Kirkpatrick et al . ( 2017 ) used the values given by the diagonal of an approximated Fisher information matrix for Ωk , and this was computed offline after training on a task was completed . An online variant of EWC was proposed by Schwarz et al . ( 2018 ) to improve EWC ’ s scalability by ensuring the computational cost of the regularization term does not grow with the number of tasks . Zenke et al . ( 2017b ) proposed an online method called Synaptic Intelligence ( SI ) for computing the parameter importance where , Ωk is the cumulative change in individual synapses over the entire training trajectory on a particular task . Memory Aware Synapses ( MAS ) from Aljundi et al . ( 2018 ) is an online method that measures Ωk by the sensitivity of the learned function to a perturbation in the parameters , instead of measuring the change in parameters to the loss as seen in SI and EWC . Our work draws inspiration from CLS theory which is a powerful computational framework for representing memories with a dual memory system via the neocortex and hippocampus . There have been numerous approaches based on CLS principles involving pseudo-rehersal ( Robins , 1995 ; Ans et al. , 2004 ; Atkinson et al. , 2018 ) , exact or episodic replay ( Lopez-Paz & Ranzato , 2017 ; Li & Hoiem , 2018 ) and generative replay ( Shin et al. , 2017 ; Wu et al. , 2018 ) . Exact replay methods require storage of the data from previous tasks which are later replayed . Generative replay methods train a separate generative model to generate images to be replayed . iCaRL ( Rebuffi et al. , 2017 ) performs rehearsal and regularization , where an external memory is used to store exemplar patterns from old task data and rehearse the model via distillation . However , in our work , we are primarily interested in neuroplasticity techniques inspired from CLS theory for alleviating catastrophic forgetting . Earlier work from Hinton & Plaut ( 1987 ) ; Gardner-Medwin ( 1989 ) showed how each synaptic connection can be composed of a fixed weight where slow learning stores long-term knowledge and a fast-changing weight for temporary associative memory . This approach involving slow and fast weights is analogous to properties of CLS theory to overcome catastrophic forgetting during continual learning . Recent research in this vein has included replacing soft attention mechanism with fast weights in RNNs ( Ba et al. , 2016 ) , the Hebbian Softmax layer ( Rae et al. , 2018 ) , augmenting slow weights in the FC layer with a fast weights matrix ( Munkhdalai & Trischler , 2018 ) , differentiable plasticity ( Miconi , 2016 ; Miconi et al. , 2018 ) and neuromodulated differentiable plasticity ( Miconi et al. , 2019 ) . We did not evaluate and compare against neuroplasticity-inpired CLS methods as baselines because they were designed for meta-learning problems and would be unfair to evaluate their performance on continual learning benchmark problems given some of their limitations . All of these methods were designed for rapid learning on simple tasks or meta-learning over a distribution of tasks or datasets , where a few number of examples from a class are seen by the network when training on different tasks to perform one-shot and few-shot learning . For instance , the Hebbian Softmax layer modifies its parameters by annealing between Hebbian and SGD updates based on an engineered scheduling scheme which achieves fast binding for rarer classes . However , when a large number of examples are observed frequently from the same class , the annealing function switches completely to SGD updates . Thus , when evaluating this model in continual learning setups , the effect of the fast weights memory storage becomes non-existent as the network learns from a large number of examples per class on each task . With a focus on continual learning , the goal of our work is to metalearn a local learning rule for the fast weights via the fixed ( slow ) weights and an SGD optimizer .

Summary: The authors introduce DIFFERENTIABLE HEBBIAN CONSOLIDATION,a new framework for continual learning that can be implemented in the usual differentiable programming setups. This framework is motivated in terms of complementary learning system (CLS) theory which features an episodic memory module. The method is shown to be easily implemented as seen in their pytorch pseudocode (authors also suggest code will be released). Additionally, authors show the method leads to significant  improvements over simple baselines, and can complement other task-specific hebbian-based learning paradigms