Year: 2019

Abstract:
In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call “anchoring”: the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

text: In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call “anchoring”: the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

heading: 1 INTRODUCTION
text: This manuscript studies the problem of continual learning, where a machine learning model experiences a sequence of tasks. Each of these tasks is presented as a stream of input-output pairs, where each pair is drawn from the corresponding task probability distribution. Since the length of the learning experience is not specified a-priori, the learner can only assume a single pass over the data, and store nothing but a few examples into a fixed-size episodic memory. At all times during the lifetime of the model, predictions on examples from all tasks can be requested. Addressing continual learning is an important research problem, since it would waive the long-obsolete assumption of “identically and independently distributed data” impeding progress towards artificial intelligence, and allow the deployment of models learning in-the-wild. However, continual learning presents one major challenge, catastrophic forgetting (McCloskey & Cohen, 1989). That is, as the learner experiences new tasks, it quickly forgets previously acquired knowledge. This is a hindrance for state-of-the-art deep learning models, where all parameters are updated after observing each example.
Continual learning has received increasing attention from the scientific community during the last decade. The state of the art in algorithms for continual learning fall into three categories. First, regularization-based approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018; Chaudhry et al., 2018; Nguyen et al., 2018). However, when the number of tasks are large, the regularization of past tasks becomes obsolete, leading to representation drift (Titsias et al., 2019). Second, modular approaches (Rusu et al., 2016; Lee et al., 2017) add new modules to the learner as new tasks are learned. While modular architectures overcome forgetting by design, the memory complexity of these approaches scales with the number of tasks. Third, memory-based methods (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Isele & Cosgun, 2018; Riemer et al., 2019; Chaudhry et al., 2019a), store a few examples from past tasks in a so-called episodic memory, to be revisited when training for a new task. Memory-based methods are the reigning state of the art, but remain a far-cry from the achievable performance by a simple multi-task learning baseline accessing all data at once. Despite intense research efforts, such gap in performance renders the problem of continual learning an open research question.
Contribution We propose Hindsight Anchor Learning (HAL), a continual learning approach to improve the performance of memory-based continual learning algorithms. HAL leverages metalearning to regularize the training objective with one representational point per class per task, called anchor. Since it is desirable to preserve the performance of a learner during its lifetime, HAL constructs anchors as to maximize forgetting on the current task throughout the learning experience. We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight: that is, by measuring forgetting on a temporary predictor that has been fine-tuned on the episodic memories of past tasks. Then, the main parameter update of HAL is to minimize the loss on the currently observed mini-batch, while keeping the predictions at all anchors invariant.
Results We compare HAL to EWC (Kirkpatrick et al., 2016), AGEM (Chaudhry et al., 2019a), experience replay (Hayes et al., 2018; Riemer et al., 2019), and MER (Riemer et al., 2019), across four standard benchmarks in continual learning (MNIST permutations, MNIST rotations, split CIFAR100, and split miniImageNet). In these experiments, HAL achieves state-of-the-art performance, improving accuracy by 5% and reducing forgetting by 20%. We show that these results hold for various sizes of episodic memories (between 1 and 5 examples per class per task). Finally, we perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-art performance.
We now begin our exposition by reviewing the continual learning setup. The rest of the manuscript then presents our new algorithm HAL (Section 3), showcases its empirical performance (Section 4), surveys the related literature (Section 5), and offers some concluding remarks (Section 6).

heading: 2 CONTINUAL LEARNING SETUP
text: In continual learning, we experience a stream of data triplets (xi, yi, ti), each containing an input xi, a target yi, and a task identifier ti ∈ T = {1, . . . , T}. Each input-target pair (xi, yi) ∈ X × Yti is an identically and independently distributed example drawn from some unknown distribution P (Xti , Yti), representing the ti-th learning task. Without loss of generality, we assume that the tasks are experienced in order (ti ≤ tj for all i ≤ j), and that the number of total tasks T is not known a priori. Under this setup, our goal is to estimate a predictor f = (w ◦ φ) : X × T → Y , composed by a featurizer φ : X → H and a classifier w : H → Y , that minimizes the multi-task error
1
T T∑ t=1 E(x,y)∼Pt [ `(f(x, t), y) ] , (1)
where Y = ∪t∈T Yt, and ` : Y × Y → R is a suitable loss function. In the sequel, and similarly to prior literature in continual learning (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a), we consider streams of data, which are experienced only once. Therefore, the learner cannot store or revisit any but a small amount of data triplets chosen to be stored in its episodic memoryM. More specifically, we consider tiny “ring” episodic memories, which contain the last m observed examples per class for each of the experienced tasks, where m ∈ {1, 3, 5}. That is, considering as variables the number of experienced tasks t and examples n, we study continual learning algorithms with a O(t) memory footprint.
Following Lopez-Paz & Ranzato (2017), we monitor two statistics to evaluate the quality of continual learning algorithms: final average accuracy, and final maximum forgetting. First, the final average accuracy of a predictor is defined as
Accuracy = 1
T T∑ j=1 aT,j , (2)
where ai,j denotes the test accuracy on task j after the model finished experiencing task i. That is, the final average accuracy measures the test performance of the predictor at each of the tasks after the continual learning experience has finished. Second, the final maximum forgetting is defined as
Forgetting = 1
T − 1 T−1∑ j=1 max l∈{1,...,T−1} (al,j − aT,j), (3)
that is, the decrease in performance at each of the tasks between their peak accuracy and their accuracy after the continual learning experience has finished.
Finally, following Chaudhry et al. (2019a), we use the first k < T tasks to cross-validate the hyperparameters of each of the considered continual learning algorithms. These first k tasks are not considered when computing the final average accuracy and maximum forgetting metrics.

heading: 3 HINSDIGHT ANCHOR LEARNING (HAL)
text: Our starting point to describe our proposed continual learning algorithm is the current state of the art methods based on experience replay (Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019b). These are predictors fθ that, during training, store a small amount of past observed triplets into an episodic memoryM = {(x′, y′, t′)}. When facing a new mini-batch of observations B := {(x, y, t)} from task t, they employ the rule θ ← θ − α · ∇θ `(B ∪ BM) to update their parameters, where
`(A) = 1 |A| ∑ (x,y,t)∈A `(fθ(x, t), y) (4)
denotes the average loss across a collection of triplets A, and BM is a mini-batch constructed by sampling triplets of previous tasks fromM at random. In general, BM is constructed to have the same size as B, but can be smaller if the episodic memoryM does not yet contain enough samples. In the previous, the episodic memoryM reminds the predictor about how to perform at past tasks using only a very small amount of datum. As such, the behaviour of the predictor outside the data stored inM is not guaranteed, and subject to worsen. Because of this reason, we would like to be more conservative, and propose to further fix the behaviour of the predictor at a collection of carefully constructed anchor points et′ , one per class per past task t′, after a parameter update1.
To implement this, we propose a two-step parameter update rule:
θ̃ ← θ − α · ∇θ `(B ∪ BM),
θ ← θ − α · ∇θ ( `(B ∪ BM) + λ ·
∑ t′<t ( fθ(et′ , t ′)− fθ̃(et′ , t ′) )2) . (5)
The first step computes a temporary parameter vector θ̃ by minimizing the loss at a minibatch from the current task t, and the episodic memory of past tasks (this is the usual experience replay parameter update). The second step employs nested optimization to perform the final update of the parameter θ, which trades-off the minimization of (a) the loss value at the current minibatch and the episodic memory, as well as (b) the change in predictions at the anchor points for all past tasks. We use L2 over the unnormalized logits as a distance measure in Eq. 5, as we experimentally find it to be superior than cosine similarity or L1 distance. This two-step nested optimization bears a similarity to gradient-based meta-learning approach (Finn et al., 2017) where we restrict the inner update to only a single gradient step.
Next, let us discuss how to choose the anchor points. Recall that our objective is to preserve the performance of the current task throughout the entire continual learning experience. Ideally, we would like to choose each et as a tool to maximally suppress the amount of forgetting about current task throughout the entire learning experience. For this, we are interested in letting et be the example from task t on which the loss would increase maximally had training with future tasks been performed. Such et would capture instances where the current task would be forgotten the most in future training. Then, requiring the predictions to remain invariant at such et, by using Eq. 5, would minimize forgetting on the current task. Mathematically, the desirable et and its label yt is given by:
(et, yt)← argmax (x,y)∼Pt `(fθT (x, t), y)− `(fθt(x, t), y), (6)
where θt is the parameter vector after training on task t and θT is the final parameter vector after the entire learning experience. Thus, keeping predictions intact on the pair (et, yt) above would
1To ease the exposition, the remaining of the section assumes only one anchor et′ per task t′. The extension to one anchor per class per task, as we use in our experiments, is straightforward.
maximally preserve the performance of task t. However, the idealistic Eq. 6 requires access to (a) the entire distribution Pt as to compute the maximization, and (b) access to all future distributions t′ > t as to compute the final parameter vector θT . Both are unrealistic assumptions.
To circumvent (a), we can recast Eq. 6 as to learn et by initializing it at random and using k gradient ascent updates for a given label yt:
et ← et + α · ∇et `(fθT (et, t), yt)− `(fθt(et, t), yt)︸ ︷︷ ︸ Forgetting loss −γ (φ(et)− φt)2︸ ︷︷ ︸ Mean embedding loss  , (7) where we recall that φ denotes the feature extractor of the predictor, and φt is the neural mean embedding (Smola et al., 2007) of all observed examples from task twhich regularizes the constructed et against the outliers. Since the feature extractor is updated after experiencing each data point, the mean embeddings φt are computed as running averages. That is, after observing a minibatch B = {(x, y, t)} of task t, we update:
φt ← β · φt + (1− β) 1 |B| ∑ x∈B φ(x), (8)
where φt is initialized to zero at the beginning of the learning experience. In our experiments, we learn one et per class for each task. We fix the yt to the corresponding class label, and discard φt after training on task t. Learning et in this manner circumvents the requirement of storing the entire distribution Pt for a task.
Still, Eq. 7 requires the parameter vector θT , to be obtained in the distant future after all learning tasks have been experienced. To waive this impossible requirement, we will approximate the future by simulating the past. That is, instead of measuring the forgetting that would happen after the model is trained at future tasks, we will measure the forgetting that happens when the model is fine-tuned at past tasks. In this way, we say that forgetting is estimated in hindsight, using past experiences. More concretely, after training on task t and obtaining the parameter vector θt, we minimize the loss during one epoch on the episodic memoryM to obtain the temporary parameter vector θM, and recast Eq. 7 as:
et ← et + α · ∇et ( `(fθM(et, t), yt)− `(fθt(et, t), yt)− γ(φ(et)− φt)2 ) . (9)
Our experiments show that the proposed approximation of future by replaying past data in hindsight reduces forgetting by 20% when compared to a standard experience replay baseline.
This completes the description of our proposed algorithm for continual learning, which combines experience replay with anchors learned in hindsight. We call our approach Hindsight Anchor Learning (HAL) and summarize the entire learning process as follows:
Hindsight Anchor Learning (HAL)
• Initialize θ ∼ P (θ) and {et ∼ P (e)}Tt=1 from normal distributions P (θ) and P (e). • InitializeM = {} • For each task t = 1, . . . , T :
– Initialize φt = 0 – For each minibatch B from task t:
* Sample BM fromM * Update θ using Eq. 5 * Update φt using Eq. 8 * UpdateM by adding B in a first-in first-out (FIFO) ring buffer
– Fine-tune onM to obtain θM – Build et using Eq. 9 k times – Discard φt
• Return θ.

heading: 4 EXPERIMENTS
text: We now evaluate the performance of HAL against a variety of state-of-the-art methods, across four popular continual learning benchmarks.

heading: 4.1 DATESETS AND TASKS
text: We perform experiments on four popular benchmarks for continual learning.
• Permuted MNIST is a variant of the MNIST dataset of handwritten digits (LeCun, 1998) where each task applies a fixed random pixel permutation to the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.
• Rotated MNIST is another variant of MNIST, where each task applies a fixed random image rotation (between 0 and 180 degrees) to the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.
• Split CIFAR is a variant of the CIFAR-100 dataset (Krizhevsky & Hinton, 2009; Zenke et al., 2017), where each task contains the data pertaining 5 random classes (without replacement) out of the total 100 classes. This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.
• Split miniImageNet is a variant of the ImageNet dataset (Russakovsky et al., 2015; Vinyals et al., 2016), containing a subset of images and classes from the original dataset. This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.
For all datasets, the first 3 tasks are used for hyper-parameter optimization (grids available in Appendix B). The learners can perform multiple epochs on these three initial tasks, that are later discarded for evaluation.

heading: 4.2 BASELINES
text: We compare our proposed model HAL to the following baselines.
• Finetune is a single model trained continually on the stream of data, without any regularization or episodic memory.
• EWC (Kirkpatrick et al., 2016) is a continual learning method that limits changes to parameters critical to past tasks, as measured by the Fisher information matrix.
• AGEM (Chaudhry et al., 2019a) is a continual learning method improving on (Lopez-Paz & Ranzato, 2017), which uses an episodic memory of parameter gradients to limit forgetting.
• MER (Riemer et al., 2019) is a continual learning method that combines episodic memories with meta-learning to limit forgetting.
• ER-Ring (Chaudhry et al., 2019b) is a continual learning method that uses a ring buffer as episodic memory.
• Multitask is an oracle baseline that has access to all data to optimize Eq. 1, useful to estimate an upper bound on the obtainable Accuracy (Eq. 2).
• Clone-and-finetune is an oracle baseline training one independent model per task, where the model for task t′ is initialized by cloning the parameters of the model for task t′ − 1.
All baselines use the same neural network architectures: a perceptron with two hidden layers of 256 ReLU neurons in the MNIST experiments, and a ResNet18, with three times less feature maps across all layers, similar to Lopez-Paz & Ranzato (2017), their CIFAR and ImageNet experiments. The task identifiers are used to select the output head in the CIFAR and ImageNet experiments, while ignored in the MNIST experiments. Batch size is set to 10 for both the stream of data and episodic memories, across experiments and models. The size of episodic memories is set between 1 and 5 examples per class per task. The same unified code base runs all experiments, and is available at https://bit.ly/2mw8bsE.2
2The code base is anonymized and implements the Permuted MNIST experiments.

heading: 4.3 RESULTS
text: Table 1 summarizes the main results of our experiments. First, our proposed HAL is the method achieving maximum Accuracy (Eq. 2) and minimal Forgetting (Eq. 3) at all benchmarks. This does not include oracle baselines Multitask (which has access to all data simultaneously) and Clone-andfinetune (which trains a separate model per task). Second, the relative gains from the second-best method ER-Ring to HAL are significant, confirming that the anchoring objective (Eq. 5) allows experience-replay methods to generalize better from the same amount of episodic memory.
Figure 1 shows a more fine grained analysis of average accuracy as new tasks are learned on Permuted MNIST and Split CIFAR. HAL preserves the performance of a predictor more effectively than other baselines.
Table 2 shows the Accuracy of methods that employ an episodic memory, when we allow the size of this memory to increase; 3 or 5 examples per class per task, resulting in a total memory size of 600 or 1000 for MNIST experiments, and 255 or 425 for CIFAR and ImageNet experiments. The corresponding numbers for Forgetting are given in Appendix Table 4. HAL outperforms all competitors at all benchmarks.
Figure 2 provides the training time of all the continual learning baselines on MNIST benchmarks. Although HAL adds a overhead on top of experience replay baseline, it is significantly faster than, MER, another meta-learning approach to reduce forgetting.
Finally, Table 3 summarizes the results of an ablation study to better understand the impact of anchor selection. We compare selecting anchors as random noise (Random-Anchors), as a random example for the associated class and task (Data-Anchors), or our proposed optimization in hindsight procedure (HAL). Our results confirm that optimizing anchors in hindsight is the most effective strategy in terms of both accuracy and forgetting metrics.
5 RELATED WORK
In continual learning (Ring, 1997), also called lifelong learning (Thrun, 1998), a learner addresses a sequence of changing tasks without storing the complete datasets of these tasks. This is in contrast to multitask learning (Caruana, 1997), where the learner assumes simultaneous access to data from all tasks. The main challenge in continual learning is to avoid catastrophic interference or forgetting (McCloskey & Cohen, 1989; McClelland et al., 1995; Goodfellow et al., 2013), that is, the learner forgetting previously acquired knowledge when learning new tasks. The state-of-the art methods in continual learning can be categorized into three classes.
First, regularization approaches discourage updating parameters important for past tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018; Nguyen et al., 2018; Zenke et al., 2017). While efficient in terms of memory and computation, these ap-
proaches suffer from brittleness due to feature drift as the number of tasks increases (Titsias et al.,
2019). Additionally, these approaches are only effective when we can perform multiple passes over each dataset (Chaudhry et al., 2019a), a case deemed unrealistic in this work.
Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu & Zhu, 2018; Ferran Alet, 2018). Modular approaches do not scale to a large number of tasks, as they require searching over combinatorial space of module architectures. Another modular approach (Rusu et al., 2016; Lee et al., 2017) adds new parts to the prediction function as new tasks are learned. By construction, modular approaches have zero forgetting, but their memory requirements increase with the number of tasks.
Third, episodic memory approaches maintain and revisit a small episodic memory of datum from past tasks. In some of these methods (Li & Hoiem, 2016; Rebuffi et al., 2017), examples in the episodic memory are replayed and predictions are kept invariant by means of distillation (Hinton et al., 2014). In other approaches (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019b) the episodic memory is used as an optimization constraint that discourages increases in loss at past tasks. More recently, several works (Hayes et al., 2018; Riemer et al., 2019; Rolnick et al., 2018; Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance. Our contribution in this paper has been to improve experience replay methods with task anchors learned in hindsight.
There are other definitions of continual learning, such as the one of task-free continual learning. The task-free formulation does not consider the notion of tasks, and instead works on undivided data streams (Aljundi et al., 2019a,b). We have focused on the task-based definition of continual learning and, similar to many recent works (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a), assumed that only a single pass through the data was possible.
Finally, our gradient-based learning of anchors bears a similarity to (Simonyan et al., 2014) and (Wang et al., 2018). In Simonyan et al. (2014), the authors use gradient ascent on class scores to find saliency maps of a classification model. Contrary to them, our proposed hindsight learning objective optimizes for the forgetting metric, as reducing it is necessary while learning continually. Dataset distillation (Wang et al., 2018) proposes to encode the entire dataset in a few synthetic points at a given parameter vector by a gradient-based optimization process. Their method requires access to the entire dataset of a task for optimization purposes. We, instead, learn anchors in hindsight from the replay buffer of past tasks after training for current task. While Wang et al. (2018) aim to replicate the performance of the entire dataset from the synthetic points, we focus on reducing forgetting of an already learned task.

heading: 6 CONCLUSION
text: We introduced HAL, a meta-learning objective for continual learning. In our approach, we learn one “anchor point” per class per task, where predictions are requested to remain invariant. These anchors are learned using gradient-based optimization, and represent points that would maximize the forgetting of the current task throughout the entire learning experience. We simulate the forgetting that would happen during the learning of future tasks in hindsight, that is, by taking temporary gradient steps across a small episodic memory of past tasks. As shown in our experiments, anchoring
in hindsight complements and improves the performance of continual learning methods based on experience replay, achieving a new state of the art on four standard continual learning benchmarks.

heading: A MORE RESULTS

heading: B HYPER-PARAMETER SELECTION
text: In this section, we report the hyper-parameters grid considered for experiments. The best values for different benchmarks are given in parenthesis.
• Multitask
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
• Clone-and-finetune
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
• Finetune
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
• EWC
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
– regularization: [0.1, 1, 10 (MNIST perm, rot, CIFAR, miniImageNet), 100, 1000]
• AGEM
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
• MER
– learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR, miniImageNet), 0.1, 0.3, 1.0]
– within batch meta-learning rate: [0.01, 0.03, 0.1 (MNIST, CIFAR, miniImageNet), 0.3, 1.0]
– current batch learning rate multiplier: [1, 2, 5 (CIFAR, miniImageNet), 10 (MNIST)]
• ER-Ring
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
• HAL
– learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0]
– regularization (λ): [0.01, 0.03, 0.1 (MNIST perm, rot), 0.3 (miniImageNet), 1 (CIFAR), 3, 10]
– mean embedding strength (γ): [0.01, 0.03, 0.1 (MNIST perm, rot, CIFAR, miniImageNet), 0.3, 1, 3, 10]

heading: C HAL ALGORITHM
text: Algorithm 1 provides pseudo-code for HAL.
Algorithm 1 Training of HAL on sequential data D = {D1, · · · ,DT }, with total replay buffer size ‘mem sz’, learning rate ‘α’, regularization strength ‘λ’, mean embedding decay ‘β’, mean embedding strength ‘η’. 1: procedure HAL(D,mem sz, α, λ, β) 2: M← {} ∗mem sz 3: {e1, · · · , eT } ← {} 4: for t ∈ {1, · · · , T} do 5: φt ← ~0 6: for B ∼ Dt do . Sample a batch from current task 7: BM ∼M . Sample a batch from episodic memory 8: θ̃ ← θ − α · ∇θ `(B ∪ BM) . Temporary parameter update 9: θ ← θ − α · ∇θ ( `(B ∪ BM) + λ · ∑ t′<t (fθ(et′ , t ′)− fθ̃(et′ , t ′)) 2 ) . Anchoring objective (Eq. 5) 10: φt ← β · φt + (1− β) · φ(B) . Running average of mean embedding 11: M← UpdateMemory(M,B) . Add samples to a ring buffer 12: end for 13: et, θ ← GetAnchors(M, θ, φt, η) . Get anchors for current task 14: end for 15: return θ,M 16: end procedure
1: procedure GETANCHORS(M, θt, φt, γ) 2: θ ← θt 3: for BM ∼M do 4: θ ← θ − α · ∇θ`(BM) . Finetune θt by taking SGD steps on the episodic memory 5: end for 6: θM ← θ . Store the updated parameter 7: et ← rand() . Initialize the task anchors 8: for 1, · · · , k do 9: et ← et + α · ∇et ( `(fθM(et, t), yt)− `(fθt(et, t), yt)− γ(φ(et)− φt) 2 ) . Maximize forgetting (Eq. 9)
10: end for 11: return et, θt 12: end procedure

heading: D APPROXIMATION OF ANCHORING GRADIENT
text: Here we derive the approximation of anchoring objective (Eq. 5) gradient. In particular, we are more interested in the regularization part of the anchoring objective that involves second-order terms. We refer to this gradient as ganch. We follow similar arguments as (Nichol & Schulman, 2018).
Let θ0 be the parameter vector before the temporary update, `ce be the cross-entropy loss, and `L2 be the L2 loss in the anchoring objective ( Eq. 5 in the main paper). We will use the following definitions:
g0 = ` ′ ce(θ0) (gradient of cross-entropy loss at initial point on B ∪ BM) H0 = ` ′′ ce(θ0) (Hessian of cross-entropy loss at initial point on B ∪ BM)
g1 = ` ′ L2(θ0) (gradient of L2 loss at initial point on anchors) H1 = ` ′′ L2(θ0) (gradient of L2 loss at initial point on anchors)
Let U0 = θ0 − αg0 be the operator giving a temporary update in the two-step process of (Eq. 5), and let θ1 be the temporary update (note that θ̃ is used in the main paper). The ganch is given by:
ganch = ∂
∂θ0 `L2(U0) = U ′0 · `′L2(θ1) = ( I − αH0 ) · `′L2(θ1) (10)
Let’s calculate the first order Taylor’s approximation of `′L2(θ1)
`′L2(θ1) = ` ′ L2(θ0) + ` ′′ L2(θ0) · (θ1 − θ0) +O(||θ1 − θ0||2)
= g1 +H1 · (θ0 − αg0 − θ0) +O(α2) = g1 − αH1 · g0 +O(α2) (11)
Putting Eq. 11 in Eq. 10 and simplifying yields:
ganch = g1 − αH1 · g0 − αH0 · g1 +O(α2) (12)
This form is very similar to second-order MAML gradient formulation, Eq. 25 in (Nichol & Schulman, 2018). Further simplification of terms like (H1 · g0) would yield inner product between the gradients g0 and g1. However, unlike MAML (Finn et al., 2017), Reptile (Nichol & Schulman, 2018) or MER (Riemer et al., 2019), in anchoring objective these gradients correspond to different losses.

Review summaries:
This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of "anchor samples" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.