Using Hindsight to Anchor Past Knowledge in Continual Learning

In continual learning , the learner faces a stream of data whose distribution changes over time . Modern neural networks are known to suffer under this setting , as they quickly forget previously acquired knowledge . To address such catastrophic forgetting , many continual learning methods implement different types of experience replay , re-learning on past data stored in a small buffer known as episodic memory . In this work , we complement experience replay with a meta-learning technique that we call “ anchoring ” : the learner updates its knowledge on the current task , while keeping predictions on some anchor points of past tasks intact . These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task , in hindsight , when the learner is fine-tuned on the episodic memory of past tasks . Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories . 1 INTRODUCTION . This manuscript studies the problem of continual learning , where a machine learning model experiences a sequence of tasks . Each of these tasks is presented as a stream of input-output pairs , where each pair is drawn from the corresponding task probability distribution . Since the length of the learning experience is not specified a-priori , the learner can only assume a single pass over the data , and store nothing but a few examples into a fixed-size episodic memory . At all times during the lifetime of the model , predictions on examples from all tasks can be requested . Addressing continual learning is an important research problem , since it would waive the long-obsolete assumption of “ identically and independently distributed data ” impeding progress towards artificial intelligence , and allow the deployment of models learning in-the-wild . However , continual learning presents one major challenge , catastrophic forgetting ( McCloskey & Cohen , 1989 ) . That is , as the learner experiences new tasks , it quickly forgets previously acquired knowledge . This is a hindrance for state-of-the-art deep learning models , where all parameters are updated after observing each example . Continual learning has received increasing attention from the scientific community during the last decade . The state of the art in algorithms for continual learning fall into three categories . First , regularization-based approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks ( Kirkpatrick et al. , 2016 ; Aljundi et al. , 2018 ; Chaudhry et al. , 2018 ; Nguyen et al. , 2018 ) . However , when the number of tasks are large , the regularization of past tasks becomes obsolete , leading to representation drift ( Titsias et al. , 2019 ) . Second , modular approaches ( Rusu et al. , 2016 ; Lee et al. , 2017 ) add new modules to the learner as new tasks are learned . While modular architectures overcome forgetting by design , the memory complexity of these approaches scales with the number of tasks . Third , memory-based methods ( Lopez-Paz & Ranzato , 2017 ; Hayes et al. , 2018 ; Isele & Cosgun , 2018 ; Riemer et al. , 2019 ; Chaudhry et al. , 2019a ) , store a few examples from past tasks in a so-called episodic memory , to be revisited when training for a new task . Memory-based methods are the reigning state of the art , but remain a far-cry from the achievable performance by a simple multi-task learning baseline accessing all data at once . Despite intense research efforts , such gap in performance renders the problem of continual learning an open research question . Contribution We propose Hindsight Anchor Learning ( HAL ) , a continual learning approach to improve the performance of memory-based continual learning algorithms . HAL leverages metalearning to regularize the training objective with one representational point per class per task , called anchor . Since it is desirable to preserve the performance of a learner during its lifetime , HAL constructs anchors as to maximize forgetting on the current task throughout the learning experience . We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight : that is , by measuring forgetting on a temporary predictor that has been fine-tuned on the episodic memories of past tasks . Then , the main parameter update of HAL is to minimize the loss on the currently observed mini-batch , while keeping the predictions at all anchors invariant . Results We compare HAL to EWC ( Kirkpatrick et al. , 2016 ) , AGEM ( Chaudhry et al. , 2019a ) , experience replay ( Hayes et al. , 2018 ; Riemer et al. , 2019 ) , and MER ( Riemer et al. , 2019 ) , across four standard benchmarks in continual learning ( MNIST permutations , MNIST rotations , split CIFAR100 , and split miniImageNet ) . In these experiments , HAL achieves state-of-the-art performance , improving accuracy by 5 % and reducing forgetting by 20 % . We show that these results hold for various sizes of episodic memories ( between 1 and 5 examples per class per task ) . Finally , we perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-art performance . We now begin our exposition by reviewing the continual learning setup . The rest of the manuscript then presents our new algorithm HAL ( Section 3 ) , showcases its empirical performance ( Section 4 ) , surveys the related literature ( Section 5 ) , and offers some concluding remarks ( Section 6 ) . 2 CONTINUAL LEARNING SETUP . In continual learning , we experience a stream of data triplets ( xi , yi , ti ) , each containing an input xi , a target yi , and a task identifier ti ∈ T = { 1 , . . . , T } . Each input-target pair ( xi , yi ) ∈ X × Yti is an identically and independently distributed example drawn from some unknown distribution P ( Xti , Yti ) , representing the ti-th learning task . Without loss of generality , we assume that the tasks are experienced in order ( ti ≤ tj for all i ≤ j ) , and that the number of total tasks T is not known a priori . Under this setup , our goal is to estimate a predictor f = ( w ◦ φ ) : X × T → Y , composed by a featurizer φ : X → H and a classifier w : H → Y , that minimizes the multi-task error 1 T T∑ t=1 E ( x , y ) ∼Pt [ ` ( f ( x , t ) , y ) ] , ( 1 ) where Y = ∪t∈T Yt , and ` : Y × Y → R is a suitable loss function . In the sequel , and similarly to prior literature in continual learning ( Lopez-Paz & Ranzato , 2017 ; Hayes et al. , 2018 ; Riemer et al. , 2019 ; Chaudhry et al. , 2019a ) , we consider streams of data , which are experienced only once . Therefore , the learner can not store or revisit any but a small amount of data triplets chosen to be stored in its episodic memoryM . More specifically , we consider tiny “ ring ” episodic memories , which contain the last m observed examples per class for each of the experienced tasks , where m ∈ { 1 , 3 , 5 } . That is , considering as variables the number of experienced tasks t and examples n , we study continual learning algorithms with a O ( t ) memory footprint . Following Lopez-Paz & Ranzato ( 2017 ) , we monitor two statistics to evaluate the quality of continual learning algorithms : final average accuracy , and final maximum forgetting . First , the final average accuracy of a predictor is defined as Accuracy = 1 T T∑ j=1 aT , j , ( 2 ) where ai , j denotes the test accuracy on task j after the model finished experiencing task i . That is , the final average accuracy measures the test performance of the predictor at each of the tasks after the continual learning experience has finished . Second , the final maximum forgetting is defined as Forgetting = 1 T − 1 T−1∑ j=1 max l∈ { 1 , ... , T−1 } ( al , j − aT , j ) , ( 3 ) that is , the decrease in performance at each of the tasks between their peak accuracy and their accuracy after the continual learning experience has finished . Finally , following Chaudhry et al . ( 2019a ) , we use the first k < T tasks to cross-validate the hyperparameters of each of the considered continual learning algorithms . These first k tasks are not considered when computing the final average accuracy and maximum forgetting metrics . 3 HINSDIGHT ANCHOR LEARNING ( HAL ) . Our starting point to describe our proposed continual learning algorithm is the current state of the art methods based on experience replay ( Hayes et al. , 2018 ; Riemer et al. , 2019 ; Chaudhry et al. , 2019b ) . These are predictors fθ that , during training , store a small amount of past observed triplets into an episodic memoryM = { ( x′ , y′ , t′ ) } . When facing a new mini-batch of observations B : = { ( x , y , t ) } from task t , they employ the rule θ ← θ − α · ∇θ ` ( B ∪ BM ) to update their parameters , where ` ( A ) = 1 |A| ∑ ( x , y , t ) ∈A ` ( fθ ( x , t ) , y ) ( 4 ) denotes the average loss across a collection of triplets A , and BM is a mini-batch constructed by sampling triplets of previous tasks fromM at random . In general , BM is constructed to have the same size as B , but can be smaller if the episodic memoryM does not yet contain enough samples . In the previous , the episodic memoryM reminds the predictor about how to perform at past tasks using only a very small amount of datum . As such , the behaviour of the predictor outside the data stored inM is not guaranteed , and subject to worsen . Because of this reason , we would like to be more conservative , and propose to further fix the behaviour of the predictor at a collection of carefully constructed anchor points et′ , one per class per past task t′ , after a parameter update1 . To implement this , we propose a two-step parameter update rule : θ̃ ← θ − α · ∇θ ` ( B ∪ BM ) , θ ← θ − α · ∇θ ( ` ( B ∪ BM ) + λ · ∑ t′ < t ( fθ ( et′ , t ′ ) − fθ̃ ( et′ , t ′ ) ) 2 ) . ( 5 ) The first step computes a temporary parameter vector θ̃ by minimizing the loss at a minibatch from the current task t , and the episodic memory of past tasks ( this is the usual experience replay parameter update ) . The second step employs nested optimization to perform the final update of the parameter θ , which trades-off the minimization of ( a ) the loss value at the current minibatch and the episodic memory , as well as ( b ) the change in predictions at the anchor points for all past tasks . We use L2 over the unnormalized logits as a distance measure in Eq . 5 , as we experimentally find it to be superior than cosine similarity or L1 distance . This two-step nested optimization bears a similarity to gradient-based meta-learning approach ( Finn et al. , 2017 ) where we restrict the inner update to only a single gradient step . Next , let us discuss how to choose the anchor points . Recall that our objective is to preserve the performance of the current task throughout the entire continual learning experience . Ideally , we would like to choose each et as a tool to maximally suppress the amount of forgetting about current task throughout the entire learning experience . For this , we are interested in letting et be the example from task t on which the loss would increase maximally had training with future tasks been performed . Such et would capture instances where the current task would be forgotten the most in future training . Then , requiring the predictions to remain invariant at such et , by using Eq . 5 , would minimize forgetting on the current task . Mathematically , the desirable et and its label yt is given by : ( et , yt ) ← argmax ( x , y ) ∼Pt ` ( fθT ( x , t ) , y ) − ` ( fθt ( x , t ) , y ) , ( 6 ) where θt is the parameter vector after training on task t and θT is the final parameter vector after the entire learning experience . Thus , keeping predictions intact on the pair ( et , yt ) above would 1To ease the exposition , the remaining of the section assumes only one anchor et′ per task t′ . The extension to one anchor per class per task , as we use in our experiments , is straightforward . maximally preserve the performance of task t. However , the idealistic Eq . 6 requires access to ( a ) the entire distribution Pt as to compute the maximization , and ( b ) access to all future distributions t′ > t as to compute the final parameter vector θT . Both are unrealistic assumptions . To circumvent ( a ) , we can recast Eq . 6 as to learn et by initializing it at random and using k gradient ascent updates for a given label yt : et ← et + α · ∇et ` ( fθT ( et , t ) , yt ) − ` ( fθt ( et , t ) , yt ) ︸ ︷︷ ︸ Forgetting loss −γ ( φ ( et ) − φt ) 2︸ ︷︷ ︸ Mean embedding loss , ( 7 ) where we recall that φ denotes the feature extractor of the predictor , and φt is the neural mean embedding ( Smola et al. , 2007 ) of all observed examples from task twhich regularizes the constructed et against the outliers . Since the feature extractor is updated after experiencing each data point , the mean embeddings φt are computed as running averages . That is , after observing a minibatch B = { ( x , y , t ) } of task t , we update : φt ← β · φt + ( 1− β ) 1 |B| ∑ x∈B φ ( x ) , ( 8 ) where φt is initialized to zero at the beginning of the learning experience . In our experiments , we learn one et per class for each task . We fix the yt to the corresponding class label , and discard φt after training on task t. Learning et in this manner circumvents the requirement of storing the entire distribution Pt for a task . Still , Eq . 7 requires the parameter vector θT , to be obtained in the distant future after all learning tasks have been experienced . To waive this impossible requirement , we will approximate the future by simulating the past . That is , instead of measuring the forgetting that would happen after the model is trained at future tasks , we will measure the forgetting that happens when the model is fine-tuned at past tasks . In this way , we say that forgetting is estimated in hindsight , using past experiences . More concretely , after training on task t and obtaining the parameter vector θt , we minimize the loss during one epoch on the episodic memoryM to obtain the temporary parameter vector θM , and recast Eq . 7 as : et ← et + α · ∇et ( ` ( fθM ( et , t ) , yt ) − ` ( fθt ( et , t ) , yt ) − γ ( φ ( et ) − φt ) 2 ) . ( 9 ) Our experiments show that the proposed approximation of future by replaying past data in hindsight reduces forgetting by 20 % when compared to a standard experience replay baseline . This completes the description of our proposed algorithm for continual learning , which combines experience replay with anchors learned in hindsight . We call our approach Hindsight Anchor Learning ( HAL ) and summarize the entire learning process as follows : Hindsight Anchor Learning ( HAL ) • Initialize θ ∼ P ( θ ) and { et ∼ P ( e ) } Tt=1 from normal distributions P ( θ ) and P ( e ) . • InitializeM = { } • For each task t = 1 , . . . , T : – Initialize φt = 0 – For each minibatch B from task t : * Sample BM fromM * Update θ using Eq . 5 * Update φt using Eq . 8 * UpdateM by adding B in a first-in first-out ( FIFO ) ring buffer – Fine-tune onM to obtain θM – Build et using Eq . 9 k times – Discard φt • Return θ .

Summary: This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of "anchor samples" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.