Title: USING HINDSIGHT TO ANCHOR PAST KNOWLEDGE IN CONTINUAL LEARNING

Abstract: In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call "anchoring": the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

1. INTRODUCTION:

This manuscript studies the problem of continual learning, where a machine learning model experiences a sequence of tasks. Each of these tasks is presented as a stream of input-output pairs, where each pair is drawn from the corresponding task probability distribution. Since the length of the learning experience is not specified a-priori, the learner can only assume a single pass over the data, and store nothing but a few examples into a fixed-size episodic memory. At all times during the lifetime of the model, predictions on examples from all tasks can be requested. Addressing continual learning is an important research problem, since it would waive the long-obsolete assumption of "identically and independently distributed data" impeding progress towards artificial intelligence, and allow the deployment of models learning in-the-wild. However, continual learning presents one major challenge, catastrophic forgetting (McCloskey & Cohen, 1989) . That is, as the learner experiences new tasks, it quickly forgets previously acquired knowledge. This is a hindrance for state-of-the-art deep learning models, where all parameters are updated after observing each example.

Continual learning has received increasing attention from the scientific community during the last decade. The state of the art in algorithms for continual learning fall into three categories. First, regularization-based approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018; Chaudhry et al., 2018; Nguyen et al., 2018) . However, when the number of tasks are large, the regularization of past tasks becomes obsolete, leading to representation drift (Titsias et al., 2019) . Second, modular approaches (Rusu et al., 2016; Lee et al., 2017) add new modules to the learner as new tasks are learned. While modular architectures overcome forgetting by design, the memory complexity of these approaches scales with the number of tasks. Third, memory-based methods (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Isele & Cosgun, 2018; Riemer et al., 2019; Chaudhry et al., 2019a) , store a few examples from past tasks in a so-called episodic memory, to be revisited when training for a new task. Memory-based methods are the reigning state of the art, but remain a far-cry from the achievable performance by a simple multi-task learning baseline accessing all data at once. Despite intense research efforts, such gap in performance renders the problem of continual learning an open research question.

Contribution:

We propose Hindsight Anchor Learning (HAL), a continual learning approach to improve the performance of memory-based continual learning algorithms. HAL leverages metalearning to regularize the training objective with one representational point per class per task, called anchor. Since it is desirable to preserve the performance of a learner during its lifetime, HAL constructs anchors as to maximize forgetting on the current task throughout the learning experience. We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight: that is, by measuring forgetting on a temporary predictor that has been fine-tuned on the episodic memories of past tasks. Then, the main parameter update of HAL is to minimize the loss on the currently observed mini-batch, while keeping the predictions at all anchors invariant.

Results:

We compare HAL to EWC (Kirkpatrick et al., 2016) , AGEM (Chaudhry et al., 2019a) , experience replay (Hayes et al., 2018; Riemer et al., 2019) , and MER (Riemer et al., 2019) , across four standard benchmarks in continual learning (MNIST permutations, MNIST rotations, split CIFAR-100, and split miniImageNet). In these experiments, HAL achieves state-of-the-art performance, improving accuracy by 5% and reducing forgetting by 20%. We show that these results hold for various sizes of episodic memories (between 1 and 5 examples per class per task). Finally, we perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-art performance.

We now begin our exposition by reviewing the continual learning setup. The rest of the manuscript then presents our new algorithm HAL (Section 3), showcases its empirical performance (Section 4), surveys the related literature (Section 5), and offers some concluding remarks (Section 6).

2. CONTINUAL LEARNING SETUP:

In continual learning, we experience a stream of data triplets (x i , y i , t i ), each containing an input x i , a target y i , and a task identifier t i ∈ T = {1, . . . , T }. Each input-target pair (x i , y i ) ∈ X × Y ti is an identically and independently distributed example drawn from some unknown distribution P (X ti , Y ti ), representing the t i -th learning task. Without loss of generality, we assume that the tasks are experienced in order (t i ≤ t j for all i ≤ j), and that the number of total tasks T is not known a priori. Under this setup, our goal is to estimate a predictor f = (w • φ) : X × T → Y, composed by a featurizer φ : X → H and a classifier w : H → Y, that minimizes the multi-task error

EQUATION

where Y = ∪ t∈T Y t , and : Y × Y → R is a suitable loss function.

In the sequel, and similarly to prior literature in continual learning (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a) , we consider streams of data, which are experienced only once. Therefore, the learner cannot store or revisit any but a small amount of data triplets chosen to be stored in its episodic memory M. More specifically, we consider tiny "ring" episodic memories, which contain the last m observed examples per class for each of the experienced tasks, where m ∈ {1, 3, 5}. That is, considering as variables the number of experienced tasks t and examples n, we study continual learning algorithms with a O(t) memory footprint.

Following Lopez-Paz & Ranzato (2017) , we monitor two statistics to evaluate the quality of continual learning algorithms: final average accuracy, and final maximum forgetting. First, the final average accuracy of a predictor is defined as

EQUATION

where a i,j denotes the test accuracy on task j after the model finished experiencing task i. That is, the final average accuracy measures the test performance of the predictor at each of the tasks after the continual learning experience has finished. Second, the final maximum forgetting is defined as

EQUATION

that is, the decrease in performance at each of the tasks between their peak accuracy and their accuracy after the continual learning experience has finished.

Finally, following Chaudhry et al. (2019a) , we use the first k < T tasks to cross-validate the hyperparameters of each of the considered continual learning algorithms. These first k tasks are not considered when computing the final average accuracy and maximum forgetting metrics.

3. HINSDIGHT ANCHOR LEARNING (HAL):

Our starting point to describe our proposed continual learning algorithm is the current state of the art methods based on experience replay (Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019b) . These are predictors f θ that, during training, store a small amount of past observed triplets into an episodic memory M = {(x , y , t )}. When facing a new mini-batch of observations B := {(x, y, t)} from task t, they employ the rule θ ← θ -α • ∇ θ (B ∪ B M ) to update their parameters, where

EQUATION

denotes the average loss across a collection of triplets A, and B M is a mini-batch constructed by sampling triplets of previous tasks from M at random. In general, B M is constructed to have the same size as B, but can be smaller if the episodic memory M does not yet contain enough samples.

In the previous, the episodic memory M reminds the predictor about how to perform at past tasks using only a very small amount of datum. As such, the behaviour of the predictor outside the data stored in M is not guaranteed, and subject to worsen. Because of this reason, we would like to be more conservative, and propose to further fix the behaviour of the predictor at a collection of carefully constructed anchor points e t , one per class per past task t , after a parameter update1 .

To implement this, we propose a two-step parameter update rule:

θ ← θ -α • ∇ θ (B ∪ B M ), θ ← θ -α • ∇ θ (B ∪ B M ) + λ • t <t f θ (e t , t ) -f θ (e t , t ) 2 .

(5)

The first step computes a temporary parameter vector θ by minimizing the loss at a minibatch from the current task t, and the episodic memory of past tasks (this is the usual experience replay parameter update). The second step employs nested optimization to perform the final update of the parameter θ, which trades-off the minimization of (a) the loss value at the current minibatch and the episodic memory, as well as (b) the change in predictions at the anchor points for all past tasks. We use L2 over the unnormalized logits as a distance measure in Eq. 5, as we experimentally find it to be superior than cosine similarity or L1 distance. This two-step nested optimization bears a similarity to gradient-based meta-learning approach (Finn et al., 2017) where we restrict the inner update to only a single gradient step.

Next, let us discuss how to choose the anchor points. Recall that our objective is to preserve the performance of the current task throughout the entire continual learning experience. Ideally, we would like to choose each e t as a tool to maximally suppress the amount of forgetting about current task throughout the entire learning experience. For this, we are interested in letting e t be the example from task t on which the loss would increase maximally had training with future tasks been performed. Such e t would capture instances where the current task would be forgotten the most in future training.

Then, requiring the predictions to remain invariant at such e t , by using Eq. 5, would minimize forgetting on the current task. Mathematically, the desirable e t and its label y t is given by:

EQUATION

where θ t is the parameter vector after training on task t and θ T is the final parameter vector after the entire learning experience. Thus, keeping predictions intact on the pair (e t , y t ) above would maximally preserve the performance of task t. However, the idealistic Eq. 6 requires access to (a) the entire distribution P t as to compute the maximization, and (b) access to all future distributions t > t as to compute the final parameter vector θ T . Both are unrealistic assumptions.

To circumvent (a), we can recast Eq. 6 as to learn e t by initializing it at random and using k gradient ascent updates for a given label y t :

EQUATION

where we recall that φ denotes the feature extractor of the predictor, and φ t is the neural mean embedding (Smola et al., 2007) of all observed examples from task t which regularizes the constructed e t against the outliers. Since the feature extractor is updated after experiencing each data point, the mean embeddings φ t are computed as running averages. That is, after observing a minibatch B = {(x, y, t)} of task t, we update:

EQUATION

where φ t is initialized to zero at the beginning of the learning experience. In our experiments, we learn one e t per class for each task. We fix the y t to the corresponding class label, and discard φ t after training on task t. Learning e t in this manner circumvents the requirement of storing the entire distribution P t for a task.

Still, Eq. 7 requires the parameter vector θ T , to be obtained in the distant future after all learning tasks have been experienced. To waive this impossible requirement, we will approximate the future by simulating the past. That is, instead of measuring the forgetting that would happen after the model is trained at future tasks, we will measure the forgetting that happens when the model is fine-tuned at past tasks. In this way, we say that forgetting is estimated in hindsight, using past experiences. More concretely, after training on task t and obtaining the parameter vector θ t , we minimize the loss during one epoch on the episodic memory M to obtain the temporary parameter vector θ M , and recast Eq. 7 as:

EQUATION

)

Our experiments show that the proposed approximation of future by replaying past data in hindsight reduces forgetting by 20% when compared to a standard experience replay baseline.

This completes the description of our proposed algorithm for continual learning, which combines experience replay with anchors learned in hindsight. We call our approach Hindsight Anchor Learning (HAL) and summarize the entire learning process as follows:

Hindsight Anchor Learning (HAL)

• Initialize θ ∼ P (θ) and {e t ∼ P (e)} T t=1 from normal distributions P (θ) and P (e).

• Initialize M = {}:

• For each task t = 1, . . . , T :

-Initialize φ t = 0 -For each minibatch B from task t:

* Sample B M from M * Update θ using Eq. 5 * Update φ t using Eq. 8 * Update M by adding B in a first-in first-out (FIFO) ring buffer -Fine-tune on M to obtain θ M -Build e t using Eq. 9 k times -Discard φ t • Return θ.

4. EXPERIMENTS:

We now evaluate the performance of HAL against a variety of state-of-the-art methods, across four popular continual learning benchmarks.

4.1. DATESETS AND TASKS:

We perform experiments on four popular benchmarks for continual learning.

• Permuted MNIST is a variant of the MNIST dataset of handwritten digits (LeCun, 1998) where each task applies a fixed random pixel permutation to the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes. • Rotated MNIST is another variant of MNIST, where each task applies a fixed random image rotation (between 0 and 180 degrees) to the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes. • Split CIFAR is a variant of the CIFAR-100 dataset (Krizhevsky & Hinton, 2009; Zenke et al., 2017) , where each task contains the data pertaining 5 random classes (without replacement) out of the total 100 classes. This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes. • Split miniImageNet is a variant of the ImageNet dataset (Russakovsky et al., 2015; Vinyals et al., 2016) , containing a subset of images and classes from the original dataset. This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.

For all datasets, the first 3 tasks are used for hyper-parameter optimization (grids available in Appendix B). The learners can perform multiple epochs on these three initial tasks, that are later discarded for evaluation.

4.2. BASELINES:

We compare our proposed model HAL to the following baselines.

• Finetune is a single model trained continually on the stream of data, without any regularization or episodic memory. • EWC (Kirkpatrick et al., 2016 ) is a continual learning method that limits changes to parameters critical to past tasks, as measured by the Fisher information matrix. • AGEM (Chaudhry et al., 2019a ) is a continual learning method improving on (Lopez-Paz & Ranzato, 2017) , which uses an episodic memory of parameter gradients to limit forgetting. • MER (Riemer et al., 2019 ) is a continual learning method that combines episodic memories with meta-learning to limit forgetting. • ER-Ring (Chaudhry et al., 2019b ) is a continual learning method that uses a ring buffer as episodic memory. • Multitask is an oracle baseline that has access to all data to optimize Eq. 1, useful to estimate an upper bound on the obtainable Accuracy (Eq. 2). 

4.3. RESULTS:

Table 1 summarizes the main results of our experiments. First, our proposed HAL is the method achieving maximum Accuracy (Eq. 2) and minimal Forgetting (Eq. 3) at all benchmarks. This does not include oracle baselines Multitask (which has access to all data simultaneously) and Clone-andfinetune (which trains a separate model per task). Second, the relative gains from the second-best method ER-Ring to HAL are significant, confirming that the anchoring objective (Eq. 5) allows experience-replay methods to generalize better from the same amount of episodic memory.

Figure 1 shows a more fine grained analysis of average accuracy as new tasks are learned on Permuted MNIST and Split CIFAR. HAL preserves the performance of a predictor more effectively than other baselines.

Table 2 shows the Accuracy of methods that employ an episodic memory, when we allow the size of this memory to increase; 3 or 5 examples per class per task, resulting in a total memory size of 600 or 1000 for MNIST experiments, and 255 or 425 for CIFAR and ImageNet experiments. The corresponding numbers for Forgetting are given in Appendix Table 4 . HAL outperforms all competitors at all benchmarks.

Figure 2 provides the training time of all the continual learning baselines on MNIST benchmarks.

Although HAL adds a overhead on top of experience replay baseline, it is significantly faster than, MER, another meta-learning approach to reduce forgetting.

Finally, Table 3 summarizes the results of an ablation study to better understand the impact of anchor selection. We compare selecting anchors as random noise (Random-Anchors), as a random example for the associated class and task (Data-Anchors), or our proposed optimization in hindsight procedure (HAL). Our results confirm that optimizing anchors in hindsight is the most effective strategy in terms of both accuracy and forgetting metrics. In continual learning (Ring, 1997) , also called lifelong learning (Thrun, 1998) , a learner addresses a sequence of changing tasks without storing the complete datasets of these tasks. This is in contrast to multitask learning (Caruana, 1997) , where the learner assumes simultaneous access to data from all tasks. The main challenge in continual learning is to avoid catastrophic interference or forgetting (McCloskey & Cohen, 1989; McClelland et al., 1995; Goodfellow et al., 2013) , that is, the learner forgetting previously acquired knowledge when learning new tasks. The state-of-the art methods in continual learning can be categorized into three classes.

First, regularization approaches discourage updating parameters important for past tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018; Nguyen et al., 2018; Zenke et al., 2017) . While efficient in terms of memory and computation, these approaches suffer from brittleness due to feature drift as the number of tasks increases (Titsias et al., (Chaudhry et al., 2019a) , a case deemed unrealistic in this work.

Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu & Zhu, 2018; Ferran Alet, 2018) . Modular approaches do not scale to a large number of tasks, as they require searching over combinatorial space of module architectures. Another modular approach (Rusu et al., 2016; Lee et al., 2017) adds new parts to the prediction function as new tasks are learned. By construction, modular approaches have zero forgetting, but their memory requirements increase with the number of tasks.

Third, episodic memory approaches maintain and revisit a small episodic memory of datum from past tasks. In some of these methods (Li & Hoiem, 2016; Rebuffi et al., 2017) , examples in the episodic memory are replayed and predictions are kept invariant by means of distillation (Hinton et al., 2014) .

In other approaches (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019b) the episodic memory is used as an optimization constraint that discourages increases in loss at past tasks. More recently, several works (Hayes et al., 2018; Riemer et al., 2019; Rolnick et al., 2018; Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance. Our contribution in this paper has been to improve experience replay methods with task anchors learned in hindsight.

There are other definitions of continual learning, such as the one of task-free continual learning. The task-free formulation does not consider the notion of tasks, and instead works on undivided data streams (Aljundi et al., 2019a,b) . We have focused on the task-based definition of continual learning and, similar to many recent works (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a) , assumed that only a single pass through the data was possible.

Finally, our gradient-based learning of anchors bears a similarity to (Simonyan et al., 2014) and (Wang et al., 2018) . In Simonyan et al. (2014) , the authors use gradient ascent on class scores to find saliency maps of a classification model. Contrary to them, our proposed hindsight learning objective optimizes for the forgetting metric, as reducing it is necessary while learning continually. Dataset distillation (Wang et al., 2018) proposes to encode the entire dataset in a few synthetic points at a given parameter vector by a gradient-based optimization process. Their method requires access to the entire dataset of a task for optimization purposes. We, instead, learn anchors in hindsight from the replay buffer of past tasks after training for current task. While Wang et al. (2018) aim to replicate the performance of the entire dataset from the synthetic points, we focus on reducing forgetting of an already learned task.

6. CONCLUSION:

We introduced HAL, a meta-learning objective for continual learning. In our approach, we learn one "anchor point" per class per task, where predictions are requested to remain invariant. These anchors are learned using gradient-based optimization, and represent points that would maximize the forgetting of the current task throughout the entire learning experience. We simulate the forgetting that would happen during the learning of future tasks in hindsight, that is, by taking temporary gradient steps across a small episodic memory of past tasks. 

D APPROXIMATION OF ANCHORING GRADIENT:

Here we derive the approximation of anchoring objective (Eq. 5) gradient. In particular, we are more interested in the regularization part of the anchoring objective that involves second-order terms. We refer to this gradient as g anch . We follow similar arguments as (Nichol & Schulman, 2018) . Let θ 0 be the parameter vector before the temporary update, ce be the cross-entropy loss, and L2 be the L2 loss in the anchoring objective ( Eq. 5 in the main paper). We will use the following definitions: Let U 0 = θ 0 -αg 0 be the operator giving a temporary update in the two-step process of (Eq. 5), and let θ 1 be the temporary update (note that θ is used in the main paper). The g anch is given by:

EQUATION

Let's calculate the first order Taylor's approximation of L2 (θ 1 )

EQUATION

)

Putting Eq. 11 in Eq. 10 and simplifying yields:

EQUATION

)

This form is very similar to second-order MAML gradient formulation, Eq. 25 in (Nichol & Schulman, 2018) . Further simplification of terms like (H 1 • g 0 ) would yield inner product between the gradients g 0 and g 1 . However, unlike MAML (Finn et al., 2017) , Reptile (Nichol & Schulman, 2018) or MER (Riemer et al., 2019) , in anchoring objective these gradients correspond to different losses.

:

To ease the exposition, the remaining of the section assumes only one anchor e t per task t . The extension to one anchor per class per task, as we use in our experiments, is straightforward.

The code base is anonymized and implements the Permuted MNIST experiments.

Figures and Tables:

Figure 1: Evolution of Accuracy (Eq. 2) as new tasks are learned. When used, episodic memories contain up to one example per class per task.

Figure 2: Training time (s) of MNIST experiments for the entire continual learning experience. MER and HAL both use meta-learning objectives to reduce forgetting.

ce (θ 0 ) (gradient of cross-entropy loss at initial point on B ∪ B M )H 0 = ce (θ 0 )(Hessian of cross-entropy loss at initial point on B ∪ B M )g 1 = L2 (θ 0 )(gradient of L2 loss at initial point on anchors)H 1 = L2 (θ 0 )(gradient of L2 loss at initial point on anchors)

Clone-and-finetune is an oracle baseline training one independent model per task, where the model for task t is initialized by cloning the parameters of the model for task t -1.

Table content: <table><tr><td>All baselines use the same neural network architectures: a perceptron with two hidden layers of 256</td></tr><tr><td>ReLU neurons in the MNIST experiments, and a ResNet18, with three times less feature maps across</td></tr><tr><td>all layers, similar to Lopez-Paz &amp; Ranzato (2017), their CIFAR and ImageNet experiments. The</td></tr></table>

Accuracy (Eq. 2)  and Forgetting (Eq. 3) results of continual learning experiments. Averages and standard deviations are computed over five random seeds. When used, episodic memories contain up to one example per class per task. Last two rows are oracle baselines.

Table content: <table><tr><td>Method</td><td colspan="2">Permuted MNIST</td><td colspan="2">Rotated MNIST</td></tr><tr><td/><td>Accuracy</td><td>Forgetting</td><td>Accuracy</td><td>Forgetting</td></tr><tr><td>Finetune</td><td colspan="4">53.5 (±1.46) 0.29 (±0.01) 41.9 (±1.37) 0.50 (±0.01)</td></tr><tr><td>EWC</td><td colspan="4">63.1 (±1.40) 0.18 (±0.01) 44.1 (±0.99) 0.47 (±0.01)</td></tr><tr><td>AGEM</td><td colspan="4">62.1 (±1.39) 0.21 (±0.01) 50.9 (±0.92) 0.40 (±0.01)</td></tr><tr><td>MER</td><td colspan="4">69.9 (±0.40) 0.14 (±0.01) 66.0 (±2.04) 0.23 (±0.01)</td></tr><tr><td>ER-Ring</td><td colspan="4">70.2 (±0.56) 0.12 (±0.01) 65.9 (±0.41) 0.24 (±0.01)</td></tr><tr><td>HAL (ours)</td><td colspan="4">73.6 (±0.31) 0.09 (±0.01) 68.4 (±0.72) 0.21 (±0.01)</td></tr><tr><td colspan="2">Clone-and-finetune 81.4 (±0.35)</td><td>0.0</td><td>87.5 (±0.11)</td><td>0.0</td></tr><tr><td>Multitask</td><td>83.0</td><td>0.0</td><td>83.3</td><td>0.0</td></tr><tr><td>Method</td><td colspan="2">Split CIFAR</td><td colspan="2">Split miniImageNet</td></tr><tr><td/><td>Accuracy</td><td>Forgetting</td><td>Accuracy</td><td>Forgetting</td></tr><tr><td>Finetune</td><td colspan="4">42.9 (±2.07) 0.25 (±0.03) 34.7 (±2.69) 0.26 (±0.03)</td></tr><tr><td>EWC</td><td colspan="4">42.4 (±3.02) 0.26 (±0.02) 37.7 (±3.29) 0.21 (±0.03)</td></tr><tr><td>AGEM</td><td colspan="4">54.9 (±2.92) 0.14 (±0.03) 48.2 (±2.49) 0.13 (±0.02)</td></tr><tr><td>MER</td><td colspan="4">49.7 (±2.97) 0.19 (±0.03) 45.5 (±1.49) 0.15 (±0.01)</td></tr><tr><td>ER-Ring</td><td colspan="4">56.2 (±1.93) 0.13 (±0.01) 49.0 (±2.61) 0.12 (±0.02)</td></tr><tr><td>HAL (ours)</td><td colspan="4">60.4 (±0.54) 0.10 (±0.01) 51.6 (±2.02) 0.10 (±0.01)</td></tr><tr><td colspan="2">Clone-and-finetune 60.3 (±0.55)</td><td>0.0</td><td>50.3 (±1.00)</td><td>0.0</td></tr><tr><td>Multitask</td><td>68.3</td><td>0.0</td><td>63.5</td><td>0.0</td></tr></table>

Accuracy (Eq. 2) results for large (3 to 5 examples per class per task) episodic memory sizes.Here we only compare methods that use an episodic memory. Averages and standard deviations are computed over five random seeds.

Table content: <table><tr><td>Method</td><td colspan="2">Permuted MNIST</td><td/><td>Rotated MNIST</td></tr><tr><td/><td colspan="4">|M| = 600 |M| = 1000 |M| = 600 |M| = 1000</td></tr><tr><td>AGEM</td><td colspan="4">63.2 (±1.47) 64.1 (±0.74) 49.9 (±1.49) 53.0 (±1.52)</td></tr><tr><td>MER</td><td colspan="4">74.9 (±0.49) 78.3 (±0.19) 76.5 (±0.30) 77.3 (±1.13)</td></tr><tr><td>ER-Ring</td><td colspan="4">73.5 (±0.43) 75.8 (±0.24) 74.7 (±0.56) 76.5 (±0.48)</td></tr><tr><td colspan="5">HAL (ours) 76.2 (±0.52) 78.4 (±0.27) 77.0 (±0.66) 78.7 (±0.97)</td></tr><tr><td>Method</td><td colspan="2">Split CIFAR</td><td/><td>Split miniImageNet</td></tr><tr><td/><td>|M| = 255</td><td colspan="2">|M| = 425</td><td>|M| = 255</td><td>|M| = 425</td></tr><tr><td>AGEM</td><td colspan="4">56.9 (±3.45) 59.9 (±2.64) 51.6 (±2.69) 54.3 (±1.56)</td></tr><tr><td>MER</td><td colspan="4">57.7 (±2.59) 60.6 (±2.09) 49.4 (±3.43) 54.8 (±1.79)</td></tr><tr><td>ER-Ring</td><td colspan="4">60.9 (±1.44) 62.6 (±1.77) 53.5 (±1.42) 54.2 (±3.23)</td></tr><tr><td colspan="5">HAL (ours) 62.9 (±1.49) 64.4 (±2.15) 56.5 (±0.87) 57.2 (±1.54)</td></tr><tr><td>5 RELATED WORK</td><td/><td/><td/></tr><tr><td/><td/><td>Wall Clock Time (s)</td><td>50 100 150 200 250</td><td>Finetune EWC AGEM MER ER-Ring HAL</td></tr><tr><td/><td/><td/><td>0</td><td>Permutations MNIST Experiments Rotations</td></tr></table>

Impact of anchor selection, where we compare random-noise anchors (Random-), one random example-per-class anchors (Data-), and our optimized anchor selection (HAL).

Table content: <table><tr><td>Anchor type</td><td colspan="2">Permuted MNIST</td><td colspan="2">Split CIFAR</td></tr><tr><td/><td colspan="4">Accuracy Forgetting Accuracy Forgetting</td></tr><tr><td>ER-Ring</td><td>70.2</td><td>0.12</td><td>56.2</td><td>0.13</td></tr><tr><td>Random-Anchors</td><td>72.7</td><td>0.10</td><td>57.8</td><td>0.13</td></tr><tr><td>Data-Anchors</td><td>73.2</td><td>0.10</td><td>59.0</td><td>0.12</td></tr><tr><td>HAL (ours)</td><td>73.6</td><td>0.09</td><td>60.4</td><td>0.10</td></tr><tr><td colspan="5">2019). Additionally, these approaches are only effective when we can perform multiple passes over</td></tr><tr><td>each dataset</td><td/><td/><td/><td/></tr></table>

As shown in our experiments, anchoring

Table content: <table><tr><td/><td colspan="3">-learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR,</td></tr><tr><td/><td>miniImageNet), 0.1, 0.3, 1.0]</td><td/></tr><tr><td/><td colspan="3">-within batch meta-learning rate: [0.01, 0.03, 0.1</td></tr><tr><td/><td colspan="2">(MNIST, CIFAR, miniImageNet), 0.3, 1.0]</td></tr><tr><td/><td colspan="3">-current batch learning rate multiplier: [1, 2, 5 (CIFAR,</td></tr><tr><td/><td>miniImageNet), 10 (MNIST)]</td><td/></tr><tr><td/><td>• ER-Ring</td><td/></tr><tr><td/><td colspan="3">-learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet),</td></tr><tr><td/><td>0.1 (MNIST perm, rot), 0.3, 1.0]</td><td/></tr><tr><td/><td>• HAL</td><td/></tr><tr><td/><td colspan="3">-learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet),</td></tr><tr><td/><td>0.1 (MNIST perm, rot), 0.3, 1.0]</td><td/></tr><tr><td/><td colspan="3">-regularization (λ): [0.01, 0.03, 0.1 (MNIST perm, rot),</td></tr><tr><td/><td colspan="2">0.3 (miniImageNet), 1 (CIFAR), 3, 10]</td></tr><tr><td/><td colspan="3">-mean embedding strength (γ): [0.01, 0.03, 0.1 (MNIST</td></tr><tr><td/><td colspan="3">perm, rot, CIFAR, miniImageNet), 0.3, 1, 3, 10]</td></tr><tr><td colspan="2">C HAL ALGORITHM</td><td/></tr><tr><td>2:</td><td>M ← {}  *  mem sz</td><td/></tr><tr><td>3:</td><td>{e1, • • • , eT } ← {}</td><td/></tr><tr><td>4:</td><td>for t ∈ {1, • • • , T } do</td><td/></tr><tr><td>5:</td><td>φt ← 0</td><td/></tr><tr><td>6:</td><td>for B ∼ Dt do</td><td/><td>Sample a batch from current task</td></tr><tr><td>7:</td><td>BM ∼ M</td><td colspan="2">Sample a batch from episodic memory</td></tr><tr><td/><td/><td/><td>Temporary parameter update</td></tr><tr><td/><td/><td>2</td><td>Anchoring objective (Eq. 5)</td></tr><tr><td/><td/><td colspan="2">Running average of mean embedding</td></tr><tr><td>11:</td><td>M ← UpdateMemory(M, B)</td><td/><td>Add samples to a ring buffer</td></tr><tr><td>12:</td><td>end for</td><td/></tr><tr><td>13:</td><td>et, θ ← GetAnchors(M, θ, φt, η)</td><td/><td>Get anchors for current task</td></tr><tr><td>14:</td><td>end for</td><td/></tr><tr><td>15:</td><td>return θ, M</td><td/></tr><tr><td colspan="2">16: end procedure</td><td/></tr><tr><td colspan="2">1: procedure GETANCHORS(M, θt, φt, γ)</td><td/></tr><tr><td>2:</td><td>θ ← θt</td><td/></tr><tr><td>3:</td><td>for BM ∼ M do</td><td/></tr><tr><td>4:</td><td>θ ← θ -α • ∇ θ (BM)</td><td colspan="2">Finetune θ t by taking SGD steps on the episodic memory</td></tr><tr><td>5:</td><td>end for</td><td/></tr><tr><td>6:</td><td>θM ← θ</td><td/><td>Store the updated parameter</td></tr><tr><td>7:</td><td>et ← rand()</td><td/><td>Initialize the task anchors</td></tr><tr><td/><td/><td/><td>Maximize forgetting (Eq. 9)</td></tr><tr><td>10:</td><td>end for</td><td/></tr><tr><td>11:</td><td>return et, θt</td><td/></tr><tr><td colspan="2">12: end procedure</td><td/></tr></table>

Peer review (iclr2020):
This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of *anchor samples* from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks. The paper is well-written, it is a novel and well-explained idea, and the results are compelling. I lean towards acceptance, but I think there are a number of things that could be improved before final publication. 1) On how the anchors are estimated: a) It is stated on page 3 that *... choose each e_t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e_t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience....* This appears to be contradictory - is the aim to find the example that would maximise the amount of forgetting for a given task after training on all tasks, in order to keep samples that are most *crucial*? This section could be clearer, and further discussion and intuition would be beneficial for the reader. b) Does it always make sense to choose anchors that maximise forgetting on the current task? Is this akin to finding examples that would optimally alter the decision boundary, in a similar fashion to support vectors? c) I really like the intuition behind optimizing for the anchors via gradient descent (since finding examples that would be most forgotten in future violates the continual learning assumption). It would be interesting to visualize what anchors are found. For example, do they tend to be close to the centroid of a class (such that the mean embedding regulariser is low)? Are they examples that tend to be visually close to other classes, or are they outliers? d) Given that anchors are already estimated and used to alleviate forgetting, what*s the benefit of the additional episodic memory? An ablation is performed with different memory sizes, but what if *only* anchors are used (ie. a size of zero)? This would be useful in delineating the benefits of the anchors versus standard replay. 2) On the two-step optimisation (Eqn. 5): a) It*s not clear to me why this constitutes a meta-learning process: there is no adaptation at test time, nor does this perform task inference or learn how to learn. In this case, the method just uses a single gradient step to compute the change in predictions at the anchor points. b) I wonder if there is a relationship to the loss used in Riemer et al, ICLR 2019. In particular, it seems that the gradient dot product term (used in that work to maximise transfer) may be related to a first-order Taylor expansion of the L2 term in this work (the change in predictions at anchor points, Eqn 5). It would be good to clarify whether this is the case, and add further discussion and intuition as needed. 3) On the relationship to existing work: a) The approach seems conceptually most similar to iCARL (cited in the paper), which maintains *exemplars* from each previous task in order to avoid forgetting. I think the writing could be much more explicit about this, and both (i) compare against the performance of iCARL, and (ii) clearly discuss the differences between the iCARL exemplars and the anchors obtained in this work. The equivalent result for splitCIFAR in the iCARL paper (for 20 tasks x 5 classes) appears to be around 45-50%, so I think it is necessary to compare and evaluate the source of this large improvement in performance. b) Another paper worth discussing and comparing further is Variational Continual Learning - which is effectively a regularization-based method augmented with a coreset / episodic memory. c) The paper seems quite centred on memory/replay- based approaches to continual learning; the abstract in particular says *SOTA continual learning methods implement different types of experience replay*, but this fails to focus on non- memory-based methods. I would temper the claim to *many approaches to continual learning...*, and spread the focus beyond replay-based approaches.