Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference

1 SOLVING THE CONTINUAL LEARNING PROBLEM . A long-held goal of AI is to build agents capable of operating autonomously for long periods . Such agents must incrementally learn and adapt to a changing environment while maintaining memories of what they have learned before , a setting known as lifelong learning ( Thrun , 1994 ; 1996 ) . In this paper we explore a variant called continual learning ( Ring , 1994 ) . In continual learning we assume that the learner is exposed to a sequence of tasks , where each task is a sequence of experiences from the same distribution ( see Appendix A for details ) . We would like to develop a solution in this setting by discovering notions of tasks without supervision while learning incrementally after every experience . This is challenging because in standard offline single task and multi-task learning ( Caruana , 1997 ) it is implicitly assumed that the data is drawn from an i.i.d . stationary distribution . Unfortunately , neural networks tend to struggle whenever this is not the case ( Goodrich , 2015 ) . Over the years , solutions to the continual learning problem have been largely driven by prominent conceptualizations of the issues faced by neural networks . One popular view is catastrophic forgetting ( interference ) ( McCloskey & Cohen , 1989 ) , in which the primary concern is the lack of stability in neural networks , and the main solution is to limit the extent of weight sharing across experiences by focusing on preserving past knowledge ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Lee et al. , 2017 ) . Another popular and more complex conceptualization is the stability-plasticity dilemma ( Carpenter & Grossberg , 1987 ) . In this view , the primary concern is the balance between network 1We consider task agnostic future gradients , referring to gradients of the model parameters with respect to unseen data points . These can be drawn from tasks that have already been partially learned or unseen tasks . stability ( to preserve past knowledge ) and plasticity ( to rapidly learn the current experience ) . For example , these techniques focus on balancing limited weight sharing with some mechanism to ensure fast learning ( Li & Hoiem , 2016 ; Riemer et al. , 2016a ; Lopez-Paz & Ranzato , 2017 ; Rosenbaum et al. , 2018 ; Lee et al. , 2018 ; Serrà et al. , 2018 ) . In this paper , we extend this view by noting that for continual learning over an unbounded number of distributions , we need to consider weight sharing and the stability-plasticity trade-off in both the forward and backward directions in time ( Figure 1A ) . The transfer-interference trade-off proposed in this paper ( section 2 ) presents a novel perspective on the goal of gradient alignment for the continual learning problem . This is right at the heart of the problem as these gradients are the update steps for SGD based optimizers during learning and there is a clear connection between gradients angles and managing the extent of weight sharing . The key difference in perspective with past conceptualizations of continual learning is that we are not just concerned with current transfer and interference with respect to past examples , but also with the dynamics of transfer and interference moving forward as we learn . Other approaches have certainly explored operational notions of transfer and interference in forward and backward directions ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ) , the link to weight sharing ( French , 1991 ; Ajemian et al. , 2013 ) , and the idea of influencing gradient alignment for continual learning before ( Lopez-Paz & Ranzato , 2017 ) . However , in past work , ad hoc changes have been made to the dynamics of weight sharing based on current learning and past learning without formulating a consistent theory about the optimal weight sharing dynamics . This new view of the problem leads to a natural meta-learning ( Schmidhuber , 1987 ) perspective on continual learning : we would like to learn to modify our learning to affect the dynamics of transfer and interference in a general sense . To the extent that our meta-learning into the future generalizes , this should make it easier for our model to perform continual learning in non-stationary settings . We achieve this by building off past work on experience replay ( Murre , 1992 ; Lin , 1992 ; Robins , 1995 ) that has been a mainstay for solving non-stationary problems with neural networks . We propose a novel meta-experience replay ( MER ) algorithm that combines experience replay with optimization based meta-learning ( section 3 ) as a first step towards modeling this perspective . Moreover , our experiments ( sections 4 , 5 , and 6 ) , confirm our theory . MER shows great promise across a variety of supervised continual learning and continual reinforcement learning settings . Critically , our approach is not reliant on any provided notion of tasks and in most of the settings we explore we must detect the concept of tasks without supervision . See Appendix B for a more detailed positioning with respect to related research . 2 THE TRANSFER-INTERFERENCE TRADE-OFF FOR CONTINUAL LEARNING . At an instant in time with parameters θ and loss L , we can define2 operational measures of transfer and interference between two arbitrary distinct examples ( xi , yi ) and ( xj , yj ) while training with 2Throughout the paper we discuss ideas in terms of the supervised learning problem formulation . Extensions to the reinforcement learning formulation are straightforward . We provide more details in Appendix N. SGD . Transfer occurs when : ∂L ( xi , yi ) ∂θ · ∂L ( xj , yj ) ∂θ > 0 , ( 1 ) where · is the dot product operator . This implies that learning example i will without repetition improve performance on example j and vice versa ( Figure 1B ) . Interference occurs when : ∂L ( xi , yi ) ∂θ · ∂L ( xj , yj ) ∂θ < 0 . ( 2 ) Here , in contrast , learning example i will lead to unlearning ( i.e . forgetting ) of example j and vice versa ( Figure 1C ) . 3 There is weight sharing between i and j when they are learned using an overlapping set of parameters . So , potential for transfer is maximized when weight sharing is maximized while potential for interference is minimized when weight sharing is minimized ( Appendix C ) . Past solutions for the stability-plasticity dilemma in continual learning operate in a simplified temporal context where learning is divided into two phases : all past experiences are lumped together as old memories and the data currently being learned qualifies as new learning . In this setting , the goal is to simply minimize the interference projecting backward in time , which is generally achieved by reducing the degree of weight sharing explicitly or implicitly . In Appendix D we explain how our baseline approaches ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) fit within this paradigm . The important issue with this perspective , however , is that the system still has learning to do and what the future may bring is largely unknown . This makes it incumbent upon us to do nothing to potentially undermine the networks ability to effectively learn in an uncertain future . This consideration makes us extend the temporal horizon of the stability-plasticity problem forward , turning it , more generally , into a continual learning problem that we label as solving the Transfer-Interference Trade-off ( Figure 1A ) . Specifically , it is important not only to reduce backward interference from our current point in time , but we must do so in a manner that does not limit our ability to learn in the future . This more general perspective acknowledges a subtlety in the problem : the issue of gradient alignment and thus weight sharing across examples arises both backward and forward in time . With this temporally symmetric perspective , the transfer-interference trade-off becomes clear . Here we propose a potential solution where we learn to learn in a way that promotes gradient alignment at each point in time . The weight sharing across examples that enables transfer to improve future performance must not disrupt performance on what has come previously . As such , our work adopts a meta-learning perspective on the continual learning problem . We would like to learn to learn each example in a way that generalizes to other examples from the overall distribution . 3 A SYSTEM FOR LEARNING TO LEARN WITHOUT FORGETTING . In typical offline supervised learning , we can express our optimization objective over the stationary distribution of x , y pairs within the dataset D : θ = argmin θ E ( x , y ) ∼D [ L ( x , y ) ] , ( 3 ) where L is the loss function , which can be selected to fit the problem . If we would like to maximize transfer and minimize interference , we can imagine it would be useful to add an auxiliary loss to the objective to bias the learning process in that direction . Considering equations 1 and 2 , one obviously beneficial choice would be to also directly consider the gradients with respect to the loss function evaluated at randomly chosen datapoints . If we could maximize the dot products between gradients at these different points , it would directly encourage the network to share parameters where gradient directions align and keep parameters separate where interference is caused by gradients in opposite directions . So , ideally we would like to optimize for the following objective 4 : θ = argmin θ E [ ( xi , yi ) , ( xj , yj ) ] ∼D [ L ( xi , yi ) + L ( xj , yj ) − α ∂L ( xi , yi ) ∂θ · ∂L ( xj , yj ) ∂θ ] , ( 4 ) 3We borrow our terminology from operational measures of forward transfer and backward transfer in LopezPaz & Ranzato ( 2017 ) , but adopt a temporally symmetric view of the phenomenon by dropping the specification of direction . Interference commonly refers to negative transfer in either direction in the literature . 4The inclusion of L ( xj , yj ) is largely an arbitrary notation choice as the relative prioritization of the two types of terms can be absorbed in α . We use this notation as it is most consistant with our implementation . Algorithm 1 Meta-Experience Replay ( MER ) procedure TRAIN ( D , θ , α , β , γ , s , k ) M ← { } for t = 1 , ... , T do for ( x , y ) in Dt do // Draw batches from buffer : B1 , ... , Bs ← sample ( x , y , s , k , M ) θA0 ← θ for i = 1 , ... , s do θWi,0 ← θ for j = 1 , ... , k do xc , yc ← Bi [ j ] θWi , j ← SGD ( xc , yc , θWi , j−1 , α ) end for // Within batch Reptile meta-update : θ ← θWi,0 + β ( θWi , k − θWi,0 ) θAi ← θ end for // Across batch Reptile meta-update : θ ← θA0 + γ ( θAs − θA0 ) // Reservoir sampling memory update : M ←M ∪ { ( x , y ) } ( algorithm 3 ) end for end for return θ , M end procedure where ( xi , yi ) and ( xj , yj ) are randomly sampled unique data points . We will attempt to design a continual learning system that optimizes for this objective . However , there are multiple problems that must be addressed to implement this kind of learning process in practice . The first problem is that continual learning deals with learning over a non-stationary stream of data . We address this by implementing an experience replay module that augments online learning so that we can approximately optimize over the stationary distribution of all examples seen so far . Another practical problem is that the gradients of this loss depend on the second derivative of the loss function , which is expensive to compute . We address this by indirectly approximating the objective to a first order Taylor expansion using a meta-learning algorithm with minimal computational overhead . 3.1 EXPERIENCE REPLAY Learning objective : The continual lifelong learning setting poses a challenge for the optimization of neural networks as examples come one by one in a non-stationary stream . Instead , we would like our network to optimize over the stationary distribution of all examples seen so far . Experience replay ( Lin , 1992 ; Murre , 1992 ) is an old technique that remains a central component of deep learning systems attempting to learn in non-stationary settings , and we will adopt here conventions from recent work ( Zhang & Sutton , 2017 ; Riemer et al. , 2017b ) leveraging this approach . The central feature of experience replay is keeping a memory of examples seen M that is interleaved with the training of the current example with the goal of making training more stable . As a result , experience replay approximates the objective in equation 3 to the extent that M approximates D : θ = argmin θ E ( x , y ) ∼M [ L ( x , y ) ] , ( 5 ) M has a current size Msize and maximum size Mmax . In our work , we update the buffer with reservoir sampling ( Appendix F ) . This ensures that at every time-step the probability that any of the N examples seen has of being in the buffer is equal to Msize/N . The content of the buffer resembles a stationary distribution over all examples seen to the extent that the items stored captures the variation of past examples . Following the standard practice in offline learning , we train by randomly sampling a batch B from the distribution captured by M . Prioritizing the current example : the variant of experience replay we explore differs from offline learning in that the current example has a special role ensuring that it is always interleaved with the examples sampled from the replay buffer . This is because before we proceed to the next example , we want to make sure our algorithm has the ability to optimize for the current example ( particularly if it is not added to the memory ) . Over N examples seen , this still implies that we have trained with each example as the current example with probability per step of 1/N . We provide algorithms further detailing how experience replay is used in this work in Appendix G ( algorithms 4 and 5 ) . Concerns about storing examples : Obviously , it is not scalable to store every experience seen in memory . As such , in this work we focus on showing that we can achieve greater performance than baseline techniques when each approach is provided with only a small memory buffer .

Summary: The authors frame continual learning as a meta-learning problem that balances catastrophic forgetting against the capacity to learn new tasks. They propose an algorithm (MER) that combines a meta-learner (Reptile) with experience replay for continual learning. MER is evaluated on variants of MNIST (Permutated, Rotations, Many) and Omniglot against GEM and EWC. It is further tested in two reinforcement learning environments, Catcher and FlappyBird. In all cases, MER exhibits significant gains in terms of average retained accuracy.