Using Hindsight to Anchor Past Knowledge in Continual Learning
Authors: Arslan Chaudhry
Albert Gordo
David Lopez-Paz
Puneet K. Dokania
Philip Torr
Venue:  (2020)
Keywords: Continual Learning
Lifelong Learning
Catastrophic Forgetting

Abstract: In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, state-of-the-art continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call anchoring: the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the state of the art in terms of both accuracy and forgetting metrics and for various sizes of episodic memories. 

Under review as a conference paper at ICLR 2020
USING HINDSIGHT TO ANCHOR PAST KNOWLEDGE IN CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In continual learning, the learner faces stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic for- getting, many continual learning methods implement different types of experience replay, re-learning on past data stored in small buffer known episodic memory. In this work, we complement experience replay with a meta-learning technique that we call "anchoring": the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay terms of both accuracy and forgetting metries and for various sizes of episodic memories.
INTRODUCTION
This manuseript studies the problem of continual learning, where a machine learning model ex- periences a sequence of tasks. Each of these tasks is presented as a stream of input-output pairs, where each pair is drawn from the corresponding task probability distribution. Since the length of the learning experience is not specified a-priori, the learner can only assume a single pass over the data, and store nothing but a few examples into a fixed-size episodic memory. all times during the lifetime of the model, predictions on examples from all tasks can be requested. Addressing continual learning is an important research problem, since would waive the long-obsolett assumption of "identically and independently distributed data" impeding progress towards artificial intelligence, and allow the deployment of models learning in-the-wild. However, continual learning presents one major challenge, catastrophic forgetting (McCloskey & Cohen, 1989). That is, the learner experiences new tasks, quickly forgets previously acquired knowledge. This is a hindrance for state-off -ice-of-th thehhartt art deep learning models, where all parameters updated after observing each example.
Continual learning has received increasing attention from the scientific community during the last decade. The state of the art in algorithms for continual learning fall into three categories. First, regularization-busee approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks (Kirkpatrick al., 2016; Aljundi et al., 2018; Chaudhry al., 2018: Nguyen et al., 2018). However, when the number of tasks are large, the regularization of past tasks becomes obsolete, leading to representation drift (Titsias et al., 2019). Second, modular approaches (Rusu et al., 2016; Lee et al., 2017) add new modules to the learner as new tasks are learned. While modular architectures overcome forgetting by design, the memory complexity of these approaches scales with the number of tasks. Third, memory-based methods (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Isele & Cosgun, 2018; Riemer et al., 2019; Chaudhry et al., 2019a), store few examples from past tasks in a so-called episodic memory, to be revisited when training for a new task. Memory- based methods are the reigning state of the art, but remain a far-cry from the achievable performance by simple multi-task learning baseline accessing all data at once. Despite intense research efforts, such gap in performance renders the problem of continual learning an open research question.

---

Under review as a conference paper at ICLR 2020
Contribution We propose Hindsight Anchor Learning (HAL), continual learning approach to improve the performance of memory-based continual learning algorithms. HAL leverages meta- learning to regularize the training objective with representational point per class per task, called anchor. Since it is desirable to preserve the performance of a learner during its lifetime, HAL constructs anchors We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight: that is, by measuring forgetting on a temporary predictor that has been fine-tuned on the episodic memories of past tasks. Then, the main parameter update of HAL is to minimize the loss on the currently observed mini-batch while keeping the predictions all anchors invariant.
maximize forgetring
the current task throughout the learning experience.
Results We compare HAL to EWC (Kirkpatrick et al., 2016), AGEM (Chaudhry et al., 201 9a), experience replay (Hayes et al., 2018; Riemer et al., 2019), and MER (Riemer al., 2019), across four standard benchmarks in continual learning (MNIST permutations, MNIST rotations, split CIFAR- 100, and split minilmageNet). In these experiments, HAL achieves ofate-oftthe-- art performance, improving accuracy by 5% and reducing forgetting by 20%. We show that these results hold for various sizes of episodic memories (between and 5 examples per class per task). Finally, perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-ar performance.
We now begin our exposition by reviewing the continual learning setup. The rest of the manuscript then presents our new algorithm HAL (Section 3), showcases its empirical performance (Section 4), surveys the related literature (Section 5), and offers some concluding remarks (Section 6).
2 CONTINUAL LEARNING SETUP
In continual learning, we experience a stream of data triplets (I,, (i, 42), each containing an input T, target Vi and a task identifier 4 E T== (.... Each input-target pair (I,, V/) X x Vi, is an identically and independently distributed example drawn from some unknown distribution P(X,, Y.)). representing the t,-th learning task. Without loss of generality, we assume that the tasks are experienced in order (4; S f, for all i S j), and that the number of total tasks T is not known a priori. Under this setup, our goal is to estimate a predictor f f= (w Φ) : X' x T -> V, composed by a featurizer Φ X' + H. and a classifier w H. - V, that minimizes the multi-task error
1
T B( (x(f(x,t),)))
Σ
where y Us, and f: y R is suitable loss function.
In the sequel, and similarly to prior literature in continual learning (Lopez-Paz & Ranzato, 2017: Hayes al., 2018; Riemer et al., 2019; Chaudhry et al., 1019,), we consider streams of data, which are experienced only once. Therefore, the learner cannot store or revisit any but small amount of data triplets chosen to be stored in its episodic memory M. More specifically, we consider tiny "ring" episodic memories, which contain the last observed examples per class for each of the experienced tasks, where m E (1,3,5), That is, considering as variables the number of experienced tasks and examples n, we study continual learning algorithms with Ott) memory footprint.
Following Lopez-Paz Ranzato (2017), we monitor two statistics to evaluate the quality continual learning algorithms: final average accuracy, and final maximum forgetting First, the final average accuracy of a predictor is defined as
T
Accuracy Σ OT
(2)
where di,j denotes the test accuracy on task j after the model finished experiencing task i. That is, the final average accuracy measures the test performance of the predictor each of the tasks after the continual learning experience has finished. Second, the final maximum forgetting is defined as
Forgetting
max (ar. aTJ), EE T-1
(3)
T
2

---

Under review as a conference paper at ICLR 2020
that is, the decrease in performance each of the tasks between their peak accuracy and their accuracy after the continual learning experience has finished.
Finally, following Chaudhry et al. (2019a), use the first & < T tasks to cross- validate the hyper- parameters of each of the considered continual learning algorithms. These first tasks are not considered when computing the final average accuracy and maximum forgetting metries.
HINSDIGHT ANCHOR LEARNING (HAL)
Our starting point to describe proposed continual learning algorithm is the current state of the art
methods based on experience replay (Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019b).
These are predictors fo that, during training, store small amount of past observed triplets into episodic memory M = {(x,, 3// (')}. When facing a new mini-batch of observations B == {(x, y, })}
from task they employ the rule + 0 V E(B U By) to update their parameters, where
((A)
1
Σ
((o((,))..))
(4)
A
(Z.B.t)EA
denotes the average loss across collection of triplets A, and BM mini-batch constructed by sampling triplets of previous tasks from M at random. In general, BM is constructed to have the same size as B, but can be smaller if the episodic memory M does not yet contain enough samples.
In the previous, the episodic memory M reminds the predictor about how to perform at past tasks using only a very small amount of datum. As such, the behaviour of the predictor outside the data stored in M is not guaranteed, and subject to worsen. Because of this reason, we would like to be more conservative, and propose to further fix the behaviour of the predictor at a collection of carefully constructed anchor points ev, one per class per past task t', after parameter update'.
To implement this, we propose a two-step parameter update rule:
- 0 - a Vo (B),
0 0 - a v ((BUBM) + Σ ()) eaa
(5)
The first step computes a temporary parameter vector Ô by minimizing the loss at a minibatch from the current task t, and the episodic memory of past tasks (this is the usual experience replay parameter update). The second step employs nested optimization to perform the final update the parameter 8, which trades-off the minimization of (a) the loss value at the current minibatch and the episodic memory, as well as (b) the change in predictions at the anchor points for all past tasks. We use L. over the unnormalized logits distance measure in Eq. 5, as we experimentally find it to be superior than cosine similarity or LI distance. This two-step nested optimization bears similarity to gradient-base meta-learning approach (Finn al., 2017) where we restrict the inner update only single gradient step.
Next, let us discuss how to choose the anchor points. Recall that our objective is to preserve the performance of the current task throughout the entire continual learning experience. Ideally, we would like to choose each é, as a tool to maximally suppress the amount of forgetting about current task throughout the entire learning experience. For this, we are interested in letting ea be the example from task on which the loss would increase maximally had training with future tasks been performed. Such e, would capture instances where the current task would be forgotten the most in future training. Then, requiring the predictions to remain invariant at such éx, by using Eq. 5, would minimize forgetting the current task. Mathematically, the desirable e, and its label y is given by:
(et, Vr) + arg max E(fer (I, t), v) f(.fe, (æ,, t),
(6)
(x.n)~P
where is the parameter vector after training task f and Or is the final parameter vector after the entire learning experience. Thus, keeping predictions intact on the pair (et, ve) above would
'To ease the exposition, the remaining of the section assumes only anchor ez per task f'. The extension anchor per class per task, we in our experiments, is straightforward.
3

---

Under review as a conference paper at ICLR 2020
maximally preserve the performance of task t. However, the idealistic Eq. 6 requires access to (a) the entire distribution Pt as to compute the maximization, and (6) access to all future distributions t' to compute the final parameter vector 0r. Both unrealistic assumptions.
To circumvent (a), we can recast Eq. as to learn e, by initializing it at random and using gradient ascent updates for given label y/:
6x c a Ver
((o((e)))),)) (et, (), Me) -7 (o(e)) ( 2i)
Forgetting loss
Mean embedding loss
where recall that ø denotes the feature extractor of the predictor, and Φi is the neural mean embedding (Smola et al., 2007) of all observed examples from task which regularizes the constructed e, against the outliers. Since the feature extractor is updated after experiencing each data point, the mean embeddings Φb, are computed as running averages. That is, after observing minibatch {(z, y, t)} of task we update:
+ 3- B
Σ Φ(z), SEB
(8)
where de is initialized to zero at the beginning of the learning experience. our experiments, we learn one per class for each task. We fix the Y/ to the corresponding class label, and discard o after training on task f. Learning e, in this manner circumvents the requirement storing the entire distribution P. for task.
Still, Eq. requires the parameter vector Or, to obtained the distant future after all learning tasks have been experienced. To waive this impossible requirement, will approximate the future by simulating the past. That is, instead of measuring the forgetting that would happen after the model trained at future tasks, will measure the forgetting that happens when the model is fine-tuned past tasks. In this way, we say that forgetting estimated in hindsight, using past experiences. More concretely, after training task and obtaining the parameter vector O,, minimize the loss during one epoch on the episodic memory M to obtain the temporary parameter vector 0M, and recast Eq. 7 as:
+ Ve, (e(fomm (et, t). v.) e(Se, (er, f), w:) ((((e)) o.)3).
(9)
Our experiments show that the proposed approximation of future by replaying past data in hindsight reduces forgetting by 20% when compared to a standard experience replay baseline.
This completes the description of proposed algorithm for continual learning, which combines experience replay with anchors learned in hindsight. We call our approach Hindsight Anchor Learning (HAL) and summarize the entire learning process follows:
Hindsight Anchor Learning (HAL)
Initialize ~ P(0) and (et P(e))T= from normal distributions P(0) and P(e).
Initialize M = 0
For each task I:...,T:
Initialize di 0
For each minibatch from task
Sample BM from M
Update using Eq. 5
Update o, using Eq.
Update M by adding B in first-in first-out (FIFO) ring buffer
Fine-tune on M to obtain 0.
Build e, using Eq. 9 times
Discard dz
Return 0.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
We now evaluate the performance of HAL against a variety of state-of-the-art methods, across four popular continual learning benchmarks.
4.1 DATESETS AND TASKS
We perform experiments four popular benchmarks for continual learning.
Permuted MNIST is variant of the MNIST dataset of handwritten digits (LeCun, 1998) where each task applies fixed random pixel permutation to the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.
Rotated MNIST is another variant of MNIST, where each task applies fixed random image rotation (between and 180 degrees) the original dataset. This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.
Split CIEAR is variant of the CIFAR- 100 dataset (Krizhevsky & Hinton, 2009; Zenke et al., 2017), where each task contains the data pertaining 5 random classes (without replacement) out of the total 100 classes. This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.
Split minilmage/Ner is variant of the ImageNet dataset (Russakovsky et al., 2015; Vinyals et al., 2016), containing a subset of images and classes from the original dataset. This benchmark contains 20 tasks, each with 250 samples per each of the classes.
For all datasets, the first 3 tasks used for hyper-parameter optimization (grids available in Appendix B). The learners can perform multiple epochs on these three initial tasks, that are later discarded for evaluation.
4.2 BASELINES
We compare our proposed model HAL to the following baselines.
Finetune is single model trained continually on the stream of data, without any regulariza- tion or episodic memory.
EWC (Kirkpatrick et al., 2016) is continual learning method that limits changes parame- ters critical to past tasks, as measured by the Fisher information matrix.
AGEM (Chaudhry et al., 2019 a continual learning method improving on (Lopez-Paz episodic memory of parameter gradients to limit forgetting.
Ranzato, 2017), which
MER (Riemer et al., 2019) is continual learning method that combines episodic memories with meta-learning to limit forgetting.
ER-Ring (Chaudhry et al., 2019b) continual learning method that uses ring buffer as episodic memory.
Multitask is an oracle baseline that has access to all data to optimize Eq. 1, useful to estimate upper bound the obtainable Accuracy (Eq. 2).
Clone-and- fineettn-- is an oracle baseline training one independent model task, where the model for task is initialized by cloning the parameters the model for task 1.
All baselines use the same neural network architectures: a perceptron with two hidden layers 256 ReLU neurons in the MNIST experiments, and ResNet18, with three times less feature maps across all layers, similar to Lopez-Paz & Ranzato (2017), their CIFAR and ImageNet experiments. The task identifiers used to select the output head in the CIFAR and ImageNet experiments, while ignored the MNIST experiments. Batch size is set to 10 for both the stream of data and episodic memories, across experiments and models. The size of episodic memories set between and examples per class per task. The same unified code base runs all experiments, and is available at https: //bit 1y/ 1 2mw8bsE.
"hhe code base is anonymized and implements the Permuted MNIST experiments.
5

---

Under review as a conference paper at ICLR 2020
Table Accuracy (Eq. 2) and Forgetting (Eq. 3) results of continual learning experiments. Averages and standard deviations are computed over five random seeds. When used, episodic memories contain up to one example per class per task. Last two rows are oracle baselines.
Method
Permuted MNIST
Rotated MNIST
Accuracy
Forgetting
Accuracy
Forgetting
Finetune
53.5 (#1.46)
0.29 (+0.01)
41.9 (+1.37)
0,50 (+0.01)
EWC
63.1 (+1.40)
0.18 (+0.01)
44.1 (#0.99)
0.47 (#0.01)
AGEM
62.1 (#1.39)
0.21 (#0.01)
50.9 (#0.92)
0,40 (#0.01)
MER
69.9 (#0.40)
0.14 (#0.01)
66.0 (+2.04)
0.23 (#0.01)
ER-Ring
70.2 (+0.56)
0.12 (#0.01)
65.9 (#0.41)
0.24 (#0.01)
HAL (ours)
73.6 (#0.31)
0.09 (#0.01)
68.4 (#0.72)
0.21 (#0.01)
Clone-and-finest
81.4 (+0.35)
0.0
87.5 110.
0.0
Multitask
83.0
0.0
83.3
0.0
Method
Split CIFAR
Split minilmageNet
Accuracy
Forgetting
Accuracy
Forgetting
Finetune
42.9 (+2.07)
0.25 (#0.03)
34.7 (+2.69)
0.26 (#0.03)
EWC
42.4 (#3.02)
0.26 (#0.02)
37.7 (#3.29)
0.21 (+0.03)
AGEM
54.9 (+2.92)
0.14 (#0.03)
48.2 (+2.49)
0.13 (+0.02)
MER
49.7 (#2.97)
0.19 (+0.03)
45.5 (#1.49)
0.15 (+0.01)
ER-Ring
56.2 (+1.93)
0.13 (#0.01)
49.0 (+2.61)
0.12 (#0.02)
HAL (ours)
60.4 (+0.54)
0.10 (+0.01)
51.6 (+2.02)
0.10 (=0.01)
Clone-and-finttur
60.3 (+0.55)
0.0
50.3 (+1.00)
0.0
Multitask
68.3
0.0
63.5
0.0
4.3 RESULTS
Table 1 summarizes the main results of our experiments. First, proposed HAL is the method achieving maximum Accuracy (Fq. 2) and minimal Forgetting (Eq. 3) at all benchmarks. This does not include oracle baselines Multitask (which has access to all data simultaneously) and Clone-and- finetune (which trains separate model per task). Second, the relative gains from the second-best method ER-Ring to HAL are significant, confirming that the anchoring objective (Eq. 5) allows experience-replay methods to generalize better from the same amount of episodic memory.
Figure shows more fine grained analysis of average accuracy as new tasks are learned on Permuted MNIST and Split CIFAR. HAL preserves the performance of a predictor more effectively than other baselines.
Table 2 shows the Accuracy of methods that employ an episodic memory, when we allow the size of this memory to increase; 3 or examples per class per task, resulting total memory size of 600 or 1000 for MNIST experiments, and 255 or 425 for CIFAR and ImageNet experiments. The corresponding numbers for Forgetting are given in Appendix Table 4. HAL outperforms all competitors at all benchmarks.
Figure provides the training time of all the continual learning baselines on MNIST benchmarks. Although HAL adds a overhead on top of experience replay baseline, it is significantly faster than, MER, another meta-lcarning approach to reduce forgetting.
Finally, Table 3 summarizes the results of ablation study to better understand the impact of anchor selection. We compare selecting anchors as random noise Handom--Achhryy as a random example for the associated class and task (HAL). Our results confirm that optimizing anchors in hindsight is the most effective strategy in terms of both accuracy and forgetting metries.
or proposed optimization in hindsight procedure

---

Under review as a conference paper at ICLR 2020
Avg Accuracy
Avg Accuracy
0.65
HAL BWCC
0.
FINETUNS
BIR-Ring
0.7
0.50
0.8
0.45
10 12
16
8
9
10
12
16
Tasks
Tasks
Permuted MNIST
Split CIFAR
Evolution of Accuracy (Eq, 2) as new tasks are learned. When used, episodic memories contain up to one example per class per task.
Figure 1:
Table 2:
Accuracy (Eq. 2) results for large (3 to 5 examples per class per task) episodic memory sizes. Here we only compare methods that use an episodic memory. Averages and standard deviations are computed over five random seeds.
Method
Permuted MNIST
Rotated MNIST
|M| 600
M 1000
|M| 600
[M] 1000
AGEM
63.2 (#1.47)
64.1 (#0.74)
49.9 (#1.49)
53.0 (#1.52)
MER
74.9 (#0.49)
78.3 (#0.19)
76.5 (#0.30)
77.3 (#1.13)
ER-Ring
73.5 (#0.43)
75.8 (+0.24)
74.7 (+0.56)
76.5 (#0.48)
HAL (ours)
76.2 (+0.52)
78.4 (+0.27)
77.0 (40.66)
78.7 (+0.97)
Method
Split CIFAR
Split minilmageNet
|M| = 255
[ = 425
[M] = 255
[M| 425
AGEM
56.9 (+3.45)
59.9 (+2.64)
51.6 (#2.69)
54.3 (+1.56)
MER
57.7 (+2.59)
60.6 (+2.09)
49.4 (#3.43)
54.8 (#1.79)
ER-Ring
60.9 (#1,44)
62.6 (#1.77)
53.5 (#1.42)
54.2 (#3.23)
HAL (ours)
62.9 (#1,49)
64.4 (+2.15)
56.5 (#0.87)
57.2 (+1.54)
RELATED WORK
In continual learning (Ring, 1997). also called lifelong learning (Thrun, 1998), learner ad- dresses a sequence of changing tasks without storing the complete datasets of these tasks. This is in contrast to multitask learning (Caruana, 1997), where the learner assumes simultaneous access to data from all tasks. The main chal- lenge in continual learning is to avoid catas- trophic interference or forgetting (McCloskey & Cohen, 1989; McClelland al., 1995; Good- fellow et al., 2013), that is, the learner forget- ting previously acquired knowledge when learn- ing new tasks. The state-of-the art methods in continual learning can be categorized into three classes.
250
Finetune
Wall Clock Time (s)
EWC
200
AGEM
MER
ER-Ring
150
HAL
100
Permutations
Rotations
MNIST Experiments
Figure 2:
Training time (s) of MNIST experiments for the entire continual learning experience. MER and HAL both meta-learning objectives re- duce forgetting.
First, regularization approaches discourage up- dating parameters important for past tasks (Kirk- patrick et al., 2016; Aljundi et al., 2018; Nguyen al., 2018; Zenke et al., 2017). While efficient in terms of memory and computation, these ap- proaches suffer from brittleness due feature drift the number of tasks increases (Titsias et al.,
7

---

Under review as a conference paper at ICLR 2020
Table 3:
Impact of anchor selection, where we compare random-noiss noise anchors (Random-), one random example-per-clan anchors (Data-), and our optimized anchor selection (HAL).
Anchor type
Permuted MNIST
Split CIFAR
Accuracy
Forgetting
Accuracy
Forgetting
ER-Ring
70.2
0.12
56.2
0.13
Random-Anchors
72.7
0.10
57.8
0.13
Data-Anchors
73.2
0.10
59.0
0.12
HAL (ours)
73.6
0.09
60.4
0.10
2019). Additionally, these approaches are only effective when we can perform multiple passes over each dataset (Chaudhry al., 2019a), case deemed unrealistic in this work.
Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu & Zhu, 2018; Ferran Alet, 2018). Modular approaches do not scale to large number of tasks, as they require searching combinatorial space of module architectures. Another modular approach (Rusu et al., 2016; Lee et al., 2017) 7) adds new parts to the prediction function as new tasks are learned. By construction, modular approaches have zero forgetting, but their memory requirements increase with the number of tasks.
Third, episodic memory approaches maintain and revisit a small episodic memory of datum from past tasks. In some of these methods (Li & Hoiem, 2016; Rebuffi et al., 2017), examples in the episodic memory are replayed and predictions are kept invariant by means of distillation (Hinton et al., 2014). In other approaches (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019b) the episodic memory used as an optimization constraint that discourages increases in loss at past tasks. More recently, several works (Hayes al., 2018; Riemer al., 2019; Rolnick et al., 2018; Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known experience replay, is cheaper than constraint- based approaches and improves prediction performance. Our contribution in this paper has been to improve experience replay methods with task anchors learned in hindsight.
There are other definitions of continual learning, such as the one task-free continual learning. The task-free formulation does consider the notion of tasks, and instead works undivided data streams (Aljundi et al., 201 9), We have focused on the task-based definition of continual learning and, similar to many recent works (Lopez-Paz & Ranzato, 2017; 7; Hayes et al., 2018: Riemer et al., 2019; Chaudhry et al., 2019a), assumed that only a single pass through the data was possible.
Finally, our gradient-based learning of anchors bears a similarity (Simonyan et al., 2014) and (Wang et al., 2018). In Simonyan et al. (2014), the authors use gradient ascent on class scores to find saliency maps of classification model. Contrary them, our proposed hindsight learning objective optimizes for the forgetting metric, as reducing it is necessary while learning continually. Dataset distillation (Wang et al., 2018) proposes to encode the entire dataset in a few synthetic points at a given parameter vector by gradient-basee optimization process. Their method requires access to the entire dataset of a task for optimization purposes. We, instead, learn anchors in hindsight from the replay buffer of past tasks after training for current task. While Wang et al. (2018) aim to replicate the performance of the entire dataset from the synthetic points, we focus on reducing forgetting of an already learned task.
CONCLUSION
We introduced HAL, a meta-learning objective for continual learning. In our approach, we learn one "anchor point" per class per task, where predictions are requested to remain invariant. These anchors are learned using gradient-basee optimization, and represent points that would maximize the forgetting of the current task throughout the entire learning experience. We simulate the forgetting that would happen during the learning of future tasks in hindsight, that is, by taking temporary gradient steps across small episodic memory of past tasks. As shown in our experiments, anchoring
8

---

Under review as a conference paper at ICLR 2020
in hindsight complements and improves the performance of continual learning methods based on experience replay, achieving a new state of the art on four standard continual learning benchmarks.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, pp. 7120-7129, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In CVPR, pp. 11254-11263. 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Online continual learning with no task boundaries. arXiv preprint 19X3::900.00000 2019b.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically composing
representation transformations as means for generalization. In ICML workshop Neural Abstract Machines and Program Induction v2, 2018.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint
10486 2019b.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 1701.0833, 2017.
Leslie P. Kaelbling Ferran Alet, Tomas Lozano-Perez. Modular meta-learning. arXiv preprint arXiv:1806. 10166v1, 2018
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariigg for fast adaptation of deep networks. In ICML- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 131ivv1312.62111 2013.
Tyler I. Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning, arXiv preprint arXiv:1809.05992 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS, 2014.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. arXiv preprint arXiv: 1802. 10269, 2018.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. PNAS, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. https://www..cstoromuu edu/ kriz/cifarhuml, 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann.ecua.com//eebbbb
1998.
Jeongtae Lee, Jachong Yun, Sungju Hwang, and Eunho Yang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 300iv:/708.01544 2017.
10

---

Under review as a conference paper at ICLR 2020
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, pp. 614 2016.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continuum learning. In NIPS, 2017.
James L McClelland, Bruce L McNaughton, and Randall C O'Reellly Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109-165, 1989.
Cuong V Nguyen, Yingzhen Li, Thang Bui, and Richard E Turrne. Variational continual learning. ICLR, 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:/803.02999, 2018, 2018.
S-V. Rebuffi, A. Kolesnikov, and C. H. Lampert. iCaRL: Incremental classifier and representation learning. CVPR, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
Mark B Ring. Child: A first step towards continual learning. Machine Learning. 28(1):77-104, 1997.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. CoRR, abs/1811, 1622, 2018. URL http:: /ttp:///axvvvv/ii org/abs/
1811.11682
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In ICLR, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, 115(3):211-22) 252, 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.00 04601, 2016.
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In ALT, pp. 13-31. Springer, 2007.
Sebastian Thrun Lifelong learning algorithms. Learning to learn, pp. 181-209. Springer, 1998.
Michalis Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning using gaussian processes. arXiv preprint 1rXiv:1901.11356 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NIPS, pp. 3630-3634 2016.
Tongzhou Wang, Jun-Yan Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv preprint arXiv:1811. 2018.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In arXiv preprint arXiv:1805.12369/1 2018.
F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.
11

---

Under review as a conference paper at ICLR 2020
APPENDIX
In Section A, we are report the forgetting 3 metric when large episodic memories are used. Section B provides the grid considered for syper-parameters. Section C provides pseudo-code for HAL.
A MORE RESULTS
Table 4: Forgetting (Eq. 3) results for large (3 to 5 examples per class per task) episodic memory sizes. Here we only compare methods that episodic memory. Averages and standard deviations are computed over five random seeds.
Method
Permuted MNIST
Rotated MNIST
[M] 600
M 1000
[M| 600
[M| 1000
AGEM
0.20 (#0.01)
0.19 (#0.01)
0.41 (#0.01)
0.38 (#0.01)
MER
0.14 (+0.01)
0.09 (#0.01)
0.12 (#0.01)
0.11 (#0.01)
ER-Ring
0.09 (=0.01)
0.07 (#0.01)
0.15 (+0.01)
0.13 (#0.01)
HAL (ours)
0.07 (#0.01)
0.05 (#0.01)
0.12 (+0.01)
0.11 (#0.01)
Method
Split CIFAR
Split minilmageNet
MM 255
[M] 425
[M| 255
[M| 425
AGEM
0.13 (+0.03)
0.10 (#0.02)
0.10 (#0.02)
0.08 (+0.01)
MER
0.11 (=0.01)
0.09 (+0.02)
0.12 (#0.02)
0.07 (+0.01)
ER-Ring
0.09 (=0.01)
0.06 (=0.01)
0.07 (#0.02)
0.08 (#0.02)
HAL L. (ours)
0.08 (=0.01)
0.06 (#0.01)
0.06 (#0.01)
0.06 (+0.01)
B
PAPEERRPRRAETER SELECTION
In this section, we report the hyper-parameters grid considered for experiments. The best values for different benchmarks are given in parenthesis.
Multitask
learning rate: [0.003, 0.01, 0.03 (CTFAR, mini ImageNet)) 0.1 (MNIST perm, rot), 0.3, 1.0]
Clone-and cone-aadd-iince and-finetune
learning rate: [0.003, 0.01, 0.03 (CTFAR, mini ImageNet))
0.1 (MNIST perm, rot), 0.3, 1.0]
Finetune
learning rate: [0.003, 0.01, 0.03 (CTFAR, mini ImageNet))
0.1 (MNIST perm, rot), 0.3, 1.0]
EWC
learning rate: [0.003, 0.01, 0.03 (CIFAR, mini MiiigetttNNet)
0.1 (MNIST perm, rot), 0.3, 1.0]
regularization: [0.1, 1, 10 (MNIST perm, rot, CIFAR, miniImageNet),, 100, 1000]
AGEM
learning rate: [0.003, 0.01, 0.03 (CTFAR, mini ImageNet)). 0.1 (MNIST perm, rot), 0.3, 1.0]
MER
12

---

Under review as a conference paper at ICLR 2020
learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR, miniImageNet), 0.1, 0.3, 1.0]
within batch meta-learning rate: 10.01, 0.03, 0.1 (MNIST, CIFAR, miniImageNet), 0.3, 1.0]
current batch learning rate multiplier: [1, 2, 5 (CIFAR, miniImageNet), 10 (MNIST)
ER-Ring
-learning rate: 10.003, 0.01, 0.03 (CIFAR, miniImageNet)
0.1 (MNIST perm, rot), 0.3, 1.0]
HAL
learning rate: 10.003, 0.01, 0.03 (CIFAR, mini ImageNet)))
0.1 (MNIST perm, rot), 0.3, 1.0)
reqularization (A): [0.01, 0.03, 0.1 (MNIST perm, rot), 0.3 (mini
1 (CIFAR), 3, 10]
mean embedding strength (Y): [0.01, 0.03, 0.1 (MNIST
perm, rot, CIFAR, miniImageNet)) 0.3, 1, 3, 10]
C HAL ALGORITHM
Algorithm provides pseudo-code for HAL.
Algorithm Training of HAL on sequential data D = (Di, ,Dr,, with total replay buffer size 'mem_sz', learning rate 'a', regularization strength 'X' mean embedding decay "3", mean embedding strength 'ry".
1: procedure HAL(D, mem nzm az, X, 3)
2: M + 0 mem SZ
3: (ei...,,er t [}
4:
fort (1.... ,T' do
5:
6: +0
6:
for B Di do
Sample
BM M
Sample brtch from episodic memory
0 r a Vo (o((BBB))
Tomperary gammeter update
9:
0 0- 0 a Vo (MBBBMM) BM) +X- Evea (faler, - Salee, ('))²)
Aschoring objective (Eq.
10:
6, B- (1 B3 (BB)
Running average embedding
M UpdateMemory(MM (M,
Add samples a ring buffer
end for 12:
13:
e.. 0 GetAnchors (1.))
anchoes for current task
14:
end for
return O.M end procedure
16:
1: procedure GETANCHORS(M,BI, 01,
2:
0 0,
BM M
4:
0 0+ 0 a Vol(BM)
D Pinetune 9c by taking SGD on the episodic memary
5:
end for
6:
updated pammeeter
ee rand()
b Inisialize the task anchors
1,...........
9:
6, +- a +a V., (C, (e,,,(es) -((((e)) - -60))) D Maximize forgetting (Eq. 9)
10:
end for
11: return e., O,
end procedure
13

---

Under review as a conference paper at ICLR 2020
D APPROXIMATION OF ANCHORING GRADIENT
Here we derive the approximation of anchoring objective (Eq. 5) gradient. In particular, we are more interested in the regularization part of the anchoring objective that involves second-order terms. We refer to this gradient as fanch- We follow similar arguments as (Nichol & Schulman, 2018).
Let On be the parameter vector before the temporary update, foe be the cross-entropy loss, and fr. be the L2 loss in the anchoring objective Eq. 5 in the main paper). We will use the following definitions:
9n (00)
(gradient of cross-entropy loss initial point B B BM)
Ho 0 l'c (9))
(Hessian of cross-entropy loss at initial point on B BM)
I1 (0o)
(gradient of L2 loss at initial point anchors)
H1 (0o)
(gradient L.2 loss at initial point on anchors)
Let Ug = 00 ogo be the operator giving temporary update in the two-step process of (Eq. 5), and let 0, be the temporary update (note that 0 is used in the main paper). The Janch is given by:
Janch
(12(U))
00,
U0 (2(())
-T-aTH)) (01)
(10)
Let's calculate the first order Taylor's approximation of (01)
(21( (21((0)) (0)) (01 0o) + 0(10; 0oll3)
= D1 + H (0p ago 0o) O(a?)
D1 - all Do + O(a2)
(11)
Putting Eq. 11 in Eq. 10 and simplifying yields:
Janch = 91 aHl In aHo 31 + 0(a²)
(12)
This form very similar to second-order MAML gradient formulation, Eq. 25 in (Nichol Schulman, 2018). Further simplification of terms like (H1 9o) would yield inner product between the gradients To and 9I. However, unlike MAML (Finn al., 2017). Reptile (Nichol Schulman, 2018) or MER (Riemer et al., 2019), in anchoring objective these gradients correspond to different losses.
14


Reviews:
[{"review_id": "rJeuVn7q9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of \"anchor samples\" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.\n\nThe paper is well-written, it is a novel and well-explained idea, and the results are compelling. I lean towards acceptance, but I think there are a number of things that could be improved before final publication.\n\n1) On how the anchors are estimated:\na) It is stated on page 3 that \"... choose each e_t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e_t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience....\"\nThis appears to be contradictory - is the aim to find the example that would maximise the amount of forgetting for a given task after training on all tasks, in order to keep samples that are most \"crucial\"? \nThis section could be clearer, and further discussion and intuition would be beneficial for the reader.\nb) Does it always make sense to choose anchors that maximise forgetting on the current task? Is this akin to finding examples that would optimally alter the decision boundary, in a similar fashion to support vectors?\nc) I really like the intuition behind optimizing for the anchors via gradient descent (since finding examples that would be most forgotten in future violates the continual learning assumption). It would be interesting to visualize what anchors are found. For example, do they tend to be close to the centroid of a class (such that the mean embedding regulariser is low)? Are they examples that tend to be visually close to other classes, or are they outliers?\nd) Given that anchors are already estimated and used to alleviate forgetting, what's the benefit of the additional episodic memory? An ablation is performed with different memory sizes, but what if *only* anchors are used (ie. a size of zero)? This would be useful in delineating the benefits of the anchors versus standard replay.\n\n2) On the two-step optimisation (Eqn. 5):\na) It's not clear to me why this constitutes a meta-learning process: there is no adaptation at test time, nor does this perform task inference or learn how to learn. In this case, the method just uses a single gradient step to compute the change in predictions at the anchor points.\nb) I wonder if there is a relationship to the loss used in Riemer et al, ICLR 2019. In particular, it seems that the gradient dot product term (used in that work to maximise transfer) may be related to a first-order Taylor expansion of the L2 term in this work (the change in predictions at anchor points, Eqn 5). It would be good to clarify whether this is the case, and add further discussion and intuition as needed.\n\n3) On the relationship to existing work:\na) The approach seems conceptually most similar to iCARL (cited in the paper), which maintains \"exemplars\" from each previous task in order to avoid forgetting. I think the writing could be much more explicit about this, and both (i) compare against the performance of iCARL, and (ii) clearly discuss the differences between the iCARL exemplars and the anchors obtained in this work.\nThe equivalent result for splitCIFAR in the iCARL paper (for 20 tasks x 5 classes) appears to be around 45-50%, so I think it is necessary to compare and evaluate the source of this large improvement in performance.\nb) Another paper worth discussing and comparing further is Variational Continual Learning - which is effectively a regularization-based method augmented with a coreset / episodic memory.\nc) The paper seems quite centred on memory/replay- based approaches to continual learning; the abstract in particular says \"SOTA continual learning methods implement different types of experience replay\", but this fails to focus on non- memory-based methods. I would temper the claim to \"many approaches to continual learning...\", and spread the focus beyond replay-based approaches.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkgF_woAYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Summary\nThis paper proposes to formulate the continual learning problem as a meta-learning problem where the multi-tasking and forgetting can be addressed simutaneously. To assess the forgetting, this paper suggests using the anchor points learned in hindsight. Empirical results show that this formulation performs better than previous methods.\n\n*** Strengths\n1. The idea of using meta-learning to deal with the forgetting issue is interesting. Using anchor points to represent previous tasks is a sensible solution.\n\n2. The comparison with the previous shows that the proposed method performances consistently well on multiple tasks.\n\n\n*** Weakness\n1. The results in Table 3 deserves more discussion. The differences between different anchor are marginal compared with the standard deviations in Table 1. Especially, the result of Data-Anchors and HAL can be treated as the same. I wonder if the authors have any intuition behind this. Basically, this result is not persuasive for the effectiveness of the proposed anchor point learning method. Also, I’m curious to see what would happen if you have oracle access to future tasks which means we don’t need to learn the anchor points in hindsight.\n\n2. The writing can be polished. There are several typos. For example, in the third line of page 3, “and following“ might be “following”. In the first line of Section 3, the colon after “state of the art” should be removed.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxT9IAdYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The technique of replay is well established. You have added a selection criterion for what to replay. You show a modest improvement over a random sample. You choose to view this as proof of the criterion, I on the other hand see it as proof of a very small result. To me it seems more like a negative result. I believe negative results are as important as positives. So I recommend acceptance. \n\nYou may want to expand the selection criterion you evaluate. You selected the most leverage on forgetting what happens as you select less and less leveraged cases? How strong is this effect? How linear? etc... ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a continual learning method that uses anchor points for experience replay. Anchor points are learned with gradient-based optimization to maximize forgetting on the current task. Experiments MNIST, CIFAR, and miniImageNet show the benefit of the proposed approach.

As noted by other reviewers, there are some grammatical issues with the paper. 

It is missing some important details in the experiments. It is unclear to me whether the five random seeds how the datasets (tasks) are ordered in the experiments. Do the five random seeds correspond to five different dataset orderings? I think it would also be very interesting to see the anchor points that are chosen in practice. This issue is brought up by R4, and the authors responded that anchor points do not correspond to classes. Since the main idea of this paper is based on anchor points, it would be nice to analyze further to get a better understanding what they represent.

Finally, the authors only evaluate their method on image classification. While I believe the technique can be applied in other domains (e.g., reinforcement learning, natural language processing) with some modifications, without providing concrete empirical evidence in the paper, the authors need to clearly state that their proposed method is only evaluated on image classification and not sell it as a general method (yet).

The authors also miss citations to some prior work on memory-based parameter adaptation and its variants.

Regardless all of the above issues, this is still a borderline paper. However, due to space constraint, I recommend to reject this paper for ICLR.
Decision: Reject