Experience replay for continual learning

1 INTRODUCTION . Modern day reinforcement learning ( RL ) has benefited substantially from a massive influx of computational resources . In some instances , the number of data points to feed into RL algorithms has kept in step with computational feasibility . For example , in simulation environments or in self-play RL , it is possible to generate fresh data on the fly . In such settings , the continual learning problem ( Ring , 1997 ) is often ignored because new experiences can be collected on demand , and the start states of the simulation can be controlled . When training on multiple tasks , it is possible to train on all environments simultaneously within the same data batch . As RL is increasingly applied to problems in industry or other real-world settings , however , it is necessary to consider cases , such as robotics , where gathering new experience is expensive or difficult . In such examples , simultaneous training may be infeasible . Instead , an agent must be able to learn from only one task at a time . The time spent on different tasks and the sequence in which those tasks occur are not under the control of the agent . The boundaries between tasks , in fact , will often be unknown – or tasks will deform continuously and not have definite boundaries at all . Such a paradigm for training eliminates the possibility of simultaneously acting upon and learning from several tasks , and leads to the danger of catastrophic forgetting , wherein an agent forgets what it has learned previously when it encounters a new situation . Here , we consider the setting of reinforcement learning where compute and memory resources are large , but the environment is not stationary : this may arise because an RL agent is encountering a task curriculum or sequence of unrelated tasks , engaged in a budgeted physical interaction within a robot , or learning from unstructured interaction with humans . In this setting , the problem of continual learning rears its head : the distribution over experiences is not controlled to facilitate the agent ’ s maintenance of previously acquired ability . An ideal continual learning system should meet three requirements . First , it should retain previously learned capacities . When a previously encountered task or situation is encountered , performance should immediately be good – ideally as good as it was historically . Second , maintenance of old skills or knowledge should not inhibit further rapid acquisition of a new skill or knowledge . These two simultaneous constraints – maintaining the old while still adapting to the new – represent the challenge known as the stability-plasticity dilemma Grossberg ( 1982 ) . Third , where possible , a continual learning system should learn new skills that are related to old ones faster than it would have de novo , a property known as constructive interference or positive transfer . We here demonstrate the surprising power of a simple approach : Continual Learning with Experience And Replay ( CLEAR ) . We show that training a network on a mixture of novel experience on-policy and replay experience off-policy allows for both maintenance of performance on earlier tasks and fast adaptation to new tasks . A significant further boost in performance and reduction in catastrophic forgetting is obtained by enforcing behavioral cloning between the current policy and its past self . While memory is rarely severely limited in modern RL , we show that small replay buffers filled with uniform samples from past experiences can be almost as effective as buffers of unbounded size . When comparing CLEAR against state-of-the-art approaches for reducing catastrophic forgetting , we obtain better or comparable results , despite the relative simplicity of our approach ; yet , crucially , CLEAR requires no information about the identity of tasks or boundaries between them . 2 RELATED WORK . The problem of catastrophic forgetting in neural networks has long been recognized ( Grossberg , 1982 ) , and it is known that rehearsing past data can be a satisfactory antidote for some purposes ( McClelland , 1998 ; French , 1999 ) . Consequently , in the supervised setting that is the most common paradigm in machine learning , catastrophic forgetting has been accorded less attention than in cognitive science or neuroscience , since a fixed dataset can be reordered and replayed as necessary to ensure high performance on all samples . In recent years , however , there has been renewed interest in overcoming catastrophic forgetting in RL contexts and in supervised learning from streaming data ( Parisi et al. , 2018 ) . Current strategies for mitigating catastrophic forgetting have primarily focused on schemes for protecting the parameters inferred in one task while training on another . For example , in Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , weights important for past tasks are constrained to change more slowly while learning new tasks . The Progressive Networks approach ( Rusu et al. , 2016 ) freezes subnetworks trained on individual tasks , and Progress & Compress ( Schwarz et al. , 2018 ) uses EWC to consolidate the network after each task has been learned . Kaplanis et al . ( 2018 ) treat individual synaptic weights as dynamical systems with latent dimensions / states that protect information . Outside of RL , Zenke et al . ( 2017 ) develop a method similar to EWC that maintains estimates of the importance of weights for past tasks , Li & Hoiem ( 2017 ) leverage a mixture of task-specific and shared parameters , and Milan et al . ( 2016 ) develop a rigorous Bayesian approach for estimating unknown task boundaries . Notably all these methods assume that task identities or boundaries are known , with the exception of Milan et al . ( 2016 ) , for which the approach is likely not scalable to highly complex tasks . Rehearsing old data via experience replay buffers is a common technique in RL . However , their introduction has primarily been driven by the goal of data-efficient learning on single tasks ( Lin , 1992 ; Mnih et al. , 2015 ; Gu et al. , 2017 ) . Research in this vein has included prioritized replay for maximizing the impact of rare experiences ( Schaul et al. , 2016 ) , learning from human demonstration data seeded into a buffer ( Hester et al. , 2017 ) , and methods for approximating replay buffers with generative models ( Shin et al. , 2017 ) . A noteworthy use of experience replay buffers to protect against catastrophic forgetting was demonstrated in Isele & Cosgun ( 2018 ) on toy tasks , with a focus on how buffers can be made smaller . Previous works ( Gu et al. , 2017 ; O ’ Donoghue et al. , 2016 ; Wang et al. , 2016 ) have explored mixing on- and off-policy updates in RL , though these were focused on speed and stability in individual tasks and did not examine continual learning . Here , in CLEAR , we demonstrate that a mixture of replay data and fresh experience protects against catastrophic forgetting while also permitting fast learning , and performs better than either pure onpolicy learning or pure off-policy learning from replay . We provide a thorough investigation and robust algorithm in CLEAR , conducting a wide variety of tests on both limited-size and unbounded buffers with complex RL tasks using state-of-the-art methods , and improve the stability of simple replay with the addition of behavioral cloning . 3 THE CLEAR METHOD . CLEAR uses actor-critic training on a mixture of new and replayed experiences . In the case of replay experiences , two additional loss terms are added to induce behavioral cloning between the network and its past self . The motivation for behavioral cloning is to prevent network output on replayed tasks from drifting while learning new tasks . We penalize ( 1 ) the KL divergence between the historical policy distribution and the present policy distribution , ( 2 ) the L2 norm of the difference between the historical and present value functions . Formally , this corresponds to adding the following loss functions , defined with respect to network parameters θ : Lpolicy-cloning : = ∑ a µ ( a|hs ) log µ ( a|hs ) πθ ( a|hs ) , Lvalue-cloning : = ||Vθ ( hs ) − Vreplay ( hs ) ||22 , where πθ denotes the ( current ) policy of the network over actions a , µ the policy generating the observed experience , and hs the hidden state of the network at time s. Note that computing KL [ µ||πθ ] instead of KL [ πθ||µ ] ensures that πθ ( a|hs ) is nonzero wherever the historical policy is as well . We apply CLEAR in a distributed training context based on the Importance Weighted Actor-Learner Architecture ( Espeholt et al. , 2018 ) . A single learning network is fed experiences ( both novel and replay ) by a number of acting networks , for which the weights are asynchronously updated to match those of the learner . The network architecture and hyperparameters are chosen as in Espeholt et al . ( 2018 ) . Training proceeds according to V-Trace . Namely , define the V-Trace target vs by : vs : = V ( hs ) + s+n−1∑ t=s γt−s ( t−1∏ i=s ci ) δtV , where δtV : = ρt ( rt + γV ( ht+1 ) − V ( ht ) ) , ci : = min ( c̄ , πθ ( ai|hi ) µ ( ai|hi ) ) , and ρt = min ( ρ̄ , πθ ( at|ht ) µ ( at|ht ) ) , for constants c̄ and ρ̄ . Then , the value function update is given by the L2 loss : Lvalue : = ( Vθ ( hs ) − vs ) 2 . The policy gradient loss is : Lpolicy-gradient : = −ρs log πθ ( as|hs ) ( rs + γvs+1 − Vθ ( hs ) ) . We also use an entropy loss : Lentropy : = ∑ a πθ ( a|hs ) log πθ ( a|hs ) . The loss functions Lvalue , Lpolicy-gradient , and Lentropy are applied both for new and replay experiences . In addition , we add Lpolicy-cloning and Lvalue-cloning for replay experiences only . In general , our experiments use a 50-50 mixture of novel and replay experiences , though performance does not appear to be very sensitive to this ratio . Further implementation details are given in Appendix A . 4 RESULTS . 4.1 CATASTROPHIC FORGETTING VS . INTERFERENCE . Our first experiment ( Figure 1 ) was designed to distinguish between two distinct concepts that are sometimes conflated , interference and catastrophic forgetting , and to emphasize the outsized role of the latter as compared to the former . Interference occurs when two or more tasks are incompatible ( destructive interference ) or mutually helpful ( constructive interference ) within the same model . Catastrophic forgetting occurs when a task ’ s performance goes down not as a result of incompatibility with another task but as a result of the second task overwriting it within the model . As we aim to illustrate , the two are independent phenomena , and while interference may happen , forgetting is ubiquitous . We considered a set of three distinct tasks within the DMLab set of environments ( Beattie et al. , 2016 ) , and compared three training paradigms on which a network may be trained to perform these three tasks : ( 1 ) Training networks on the individual tasks separately , ( 2 ) training a single network examples from all tasks simultaneously ( which permits interference among tasks ) , and ( 3 ) training a single network sequentially on examples from one task , then the next task , and so on cyclically . Across all training protocols , the total amount of experience for each task was held constant . Thus , for separate networks training on separate tasks , the x-axis in our plots shows the total number of environment frames summed across all tasks . For example , at three million frames , one million were on task 1 , one million on task 2 , and one million on task 3 . This allows a direct comparison to simultaneous training , in which the same network was trained on all three tasks . We observe that in DMLab , there is very little difference between separate and simultaneous training . This indicates minimal interference between tasks . If anything , there is a small amount of constructive interference , with simultaneous training performing slightly better than separate training . We assume this is a result of ( i ) commonalities in image processing required across different tasks , and ( ii ) certain basic exploratory behaviors , e.g. , moving around , that are advantageous across tasks . ( By contrast , destructive interference might result from incompatible behaviors or from insufficient model capacity . ) By contrast , there is a large difference between either of the above modes of training and sequential training , where performance on a task decays immediately when training switches to another task – that is , catastrophic forgetting . Note that the performance of the sequential training appears at some points to be greater than that of separate training . This is purely because in sequential training , training proceeds exclusively on a single task , then exclusively on another task . For example , the first task quickly increases in performance since the network is effectively seeing three times as much data on that task as the networks training on separate or simultaneous tasks .

Summary: The authors propose an approach to augment experience replay buffers with properties that can alleviate issues with catastrophic forgetting. The buffers are augmented by storing both new and historical experiences, along with the desired historical policy & value distribution. The AC learning now couples two additional losses that ensures the new policy does not drift away from old actor distribution (via KL) and new value does not drift away from old critic distribution (via L2 loss).