Learning to remember: Dynamic Generative Memory for Continual Learning

1 INTRODUCTION . Continual lifelong learning is unquestionably one of the most important cognitive skills of humans and artificial neural networks alike . This can be largely attributed to the fact that every time something new is learned , a little of bit of old knowledge gets overwritten . Therefore , addressing this is of particular relevance in a continual learning setting where forgetting is largely unavoidable . In this regard , this phenomenon in context of artificial neural network algorithms is commonly referred to as catastrophic forgetting ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ; French , 1999 ) . Catastrophic forgetting is believed to occur due to a lack of a plastic component in the neuron connections ( Kirkpatrick et al. , 2016 ; Zenke et al. , 2017 ) . In analogy with biological systems , neural or synaptic plasticity is the capability of a neuron or synapse to “ lock ” itself to a state or a connection , thus retaining previously attained knowledge . Indeed , it has been shown that this plasticity is responsible for maintaining previously acquired structure in the neo-cortical circuits of brains ( Cichon & Gan , 2015 ; Hayashi-Takagi et al. , 2015 ) . Several recent approaches try to mitigate forgetting by simulating neuroplasticity in artificial neural networks . To that end , one possibility is to identify critically important parameters of a network w.r.t . a given task , and imposing a penalty for changing them for learning another task ( Kirkpatrick et al. , 2016 ; Chaudhry et al. , 2018 ; Zenke et al. , 2017 ; Aljundi et al. , 2017 ) . Common to all these methods is that they seek to find a point in the model parameter space that minimizes the loss jointly for all previously learned tasks . However , the existence of such point is not always guaranteed , hence the hard attention to the task ( HAT ) mechanism was proposed Serrà et al . ( 2018 ) . HAT finds a parameter subspace for each task that can overlap with other tasks ’ parameter subspaces . The optimal solution is then found in the corresponding parameter subspace of each task . These solutions can be seen as reserving sub-spaces of the neural network parameter space for each task at hand and driving the subsequent network updates into the free-capacity space . Another important factor in the continual learning setting is the ability to scale , i.e . to maintain sufficient capacity to accommodate for a continuously growing number of tasks . Given equitable resource constraints , it is inevitable that with a growing number of tasks to learn , the model capacity is depleted at one point in time . Without a smart and adaptive capacity strategy network expansion can swiftly go beyond existing hardware limits , or unnecessarily waste resources by reservation . In order to address catastrophic forgetting in a continual learning setting some approaches rely on storing raw samples of previously seen data and making use of replay strategies during the training of subsequent tasks . However , this starkly contrasts with the natural learning mechanisms of the brain , which does not feature the retrieval of raw information identical to originally exposed impressions Mayford et al . ( 2012 ) . What is more , storing raw samples of previous data may violate data privacy and memory restrictions in the real world applications . Instead of storing raw samples , the proposed approach relies upon on the idea of training a deep generative model to memorize previously seen data distributions . Simultaneous with the adversarial training on each incoming data batch , we learn a sparse mask for the layer activations of the generator network . This serves the purpose of encouraging parameter re-usability across tasks . The values of the learned mask correspond to the plasticity of the connections between the layers . Thus units with low plasticity are reserved for previous tasks and can be reused but not changed during the subsequent training . Our contribution is twofold : ( a ) we address the catastrophic forgetting problem in continual learning ; we introduce Dynamic Generative Memory ( DGM ) - an adversarially trainable generative network endowed with neuronal plasticity through efficient learning of a sparse parameter attention mask.1 for layer activations ; a single generator is able to incrementally learn new information during normal adversarial training without the need to replay previous knowledge ; ( b ) W e address the scalability problem in continual learning proposing an adaptive network expansion mechanism , facilitating resource efficient continual learning . We compare the proposed method to state-of-theart approaches for continual learning . 2 RELATED WORK . Among the first works dealing with catastrophic forgetting in the context of lifelong learning are ( French , 1999 ; McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) . In contrast to the proposed approach , these methods tackle this problem by employing shallow neural networks , whereas our method makes use of modern deep architectures . In this regard , lately a wealth of works dealing with catastrophic forgetting in context of deep neural networks have appeared in the literature , see e.g. , ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Li & Hoiem , 2016 ; Serrà et al. , 2018 ; Aljundi et al. , 2017 ) . However , all of these methods have been proposed for a “ task-incremental learning ” setup , where a sequence of disjoint tasks should to be learned one after the other by a single network . In our work , however , a completely different incremental methodology is used . We specifically propose a method to overcome catastrophic forgetting within the “ class-incremental learning ” setup . The key difference in the task-incremental setup is that the model learns a separate classifier for each task ( i.e. , multi-head ) , whereas in the latter the model learns only a single classifier for all of the observed classes of all tasks ( i.e. , single-head ) . Several continuous learning approaches ( Rebuffi et al. , 2016 ; Nguyen et al. , 2017 ; Kemker & Kanan , 2017 ) , address catastrophic forgetting in the class-incremental setting , i.e . by storing raw samples of previously seen data and making use of them during the training of subsequent tasks . There is a growing interest in employing deep generative models for memorizing previously seen data distributions instead of storing old samples . Thus Shin et al . ( 2017 ) ; Wu et al . ( 2018 ) rely on the idea of generative replay , which requires retraining the generator from scratch at each time step on a mixture of synthesized images of previous classes and new real samples . Apart from being inefficient for training , it is severely prone to “ semantic drifting ” . Namely , the quality of images generated at every memory replay point highly depends on the images generated during previous replays , which can result in loss of quality over time . In contrast to the methods described above , we propose to utilize a single generator that is able to incrementally learn new information during the normal adversarial training without the need to replay previous knowledge . We achieve it though efficiently learning a sparse mask for the layer activations of the generator network . Similar to our method , Seff et al . ( 2017 ) proposed to avoid retraining the generator at every timestep on the previous classes , by applying per-parameter regularization selectively in the generative network . We pursue a similar goal with the key difference that we expand the capacity of generator dynamically ( with increasing amount of attained knowledge ) instead of regularizing parameters of previous tasks . A similar strategy has been proposed in Yoon et al . ( 2018 ) . However , they propose to keep track of the semantic drift in every neuron and then expand the network by duplicating neurons that are subject to sharp changes . In contrast , we compute weights importance concurrently during the course of network training by modeling the neuron behavior using an explicit binary mask . As 1Represented as a learnable binary mask applied to the network ’ s activations . a result , our method explicitly does not require any further network retraining after adding new capacity . Another approach relying on dynamic network extension was proposed by Rusu et al . ( 2016 ) and improved by Schwarz et al . ( 2018 ) . In Progressive Neural Networks Rusu et al . ( 2016 ) authors propose to assign a separate column ( network ) to every new task , while insure positive knowledge transfer by maintaining trainable connections from the pretrained columns of previous task to the current task ’ s column . Schwarz et al . ( 2018 ) address the scalability problem of Progressive Networks by only keeping two columns : a knowledge base and an active column while distilling the knowledge continuously from the later to the former one . In our approach we only keep a single column for all tasks in which knowledge transfer happens naturally , since parameters of old tasks can be freely reused by the new ones . Other approaches like ( Kemker & Kanan , 2017 ; Kamra et al. , 2017 ) try to explicitly model short and long term memory with separate networks while transferring the knowledge from the former one to the later in a separate “ sleeping ” phase . In contrast to these methods our approach does not explicitly keep two separate memory locations , but models it implicitly in a single memory network . The memory transfer occurs during the binary mask learning from non-binary ( short term ) to completely binary ( long term ) values . Most related to our paper is the work by Serrà et al . ( 2018 ) , which considers a task-based hard attention mechanism that preserves previous tasks ’ information without affecting the current task ’ s learning . In this regard , they propose to learn a binary ask for the parameters of a base network similar to Courbariaux et al . ( 2015 ) and Mallya & Lazebnik ( 2018 ) and Mancini et al . ( 2018 ) . In this fashion , while the parameters of the network are kept constant , the binary mask is used to find a task specific perturbation of the base network that maximizes performance on incrementally introduced new tasks . We build upon this idea and extend it by a mask for the neurons to be jointly learned with the network parameters directly within the generator , thus allowing it to work in classincremental setup without storing samples of previous tasks . Such binary mask also provides us with an efficient growth mechanism for network capacity by explicit modeling of plasticity and abandoning of over-capacity . 3 PRELIMINARIES . Adopting the notation of Chaudhry et al . ( 2018 ) , let Dk = { ( xki , yki ) } n k i=1 denote a collection of data belonging to the task k ∈ K , where xki ∈ X is the input data and yki ∈ yk are the ground truth labels . While in the ( standard ) non-incremental multi-task setup the entire data D = ∪|K|k=0Dk is available at once , in a task-incremental setup the incoming dataset Dk becomes available to the model continuously and specifically only during the learning of task k. Thereby , Dk can be composed of a collection of class items from the same task , or more generally only from a single class . We address the latter class-incremental setup , in which the model is presented data from a single class at a time . Furthermore , during test time the output space covers all the labels observed , independently from the task they belong to : Yk = ∪kj=1yj . Such an evaluation scenario is often referred to as a single-head evaluation ( Chaudhry et al. , 2018 ; Rebuffi et al. , 2016 ) and is predominantly used in class-incremental learning .

Summary: The proposed method tackles class-incremental continual learning, where new categories are incrementally exposed to the network but a classifier across all categories must be learned. The proposed method seems to be essentially a combination of generative replay (e.g. Deep Generative Replay) with AC-GAN as the model and attention (HAT), along with a growing mechanism to support saturating capacity. Quantitative results are shown on MNIST and SVHN while some analysis is provided on CIFAR.