Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of "anchor samples" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.

The paper is well-written, it is a novel and well-explained idea, and the results are compelling. I lean towards acceptance, but I think there are a number of things that could be improved before final publication.

1) On how the anchors are estimated:
a) It is stated on page 3 that "... choose each e_t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e_t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience...."
This appears to be contradictory - is the aim to find the example that would maximise the amount of forgetting for a given task after training on all tasks, in order to keep samples that are most "crucial"? 
This section could be clearer, and further discussion and intuition would be beneficial for the reader.
b) Does it always make sense to choose anchors that maximise forgetting on the current task? Is this akin to finding examples that would optimally alter the decision boundary, in a similar fashion to support vectors?
c) I really like the intuition behind optimizing for the anchors via gradient descent (since finding examples that would be most forgotten in future violates the continual learning assumption). It would be interesting to visualize what anchors are found. For example, do they tend to be close to the centroid of a class (such that the mean embedding regulariser is low)? Are they examples that tend to be visually close to other classes, or are they outliers?
d) Given that anchors are already estimated and used to alleviate forgetting, what's the benefit of the additional episodic memory? An ablation is performed with different memory sizes, but what if *only* anchors are used (ie. a size of zero)? This would be useful in delineating the benefits of the anchors versus standard replay.

2) On the two-step optimisation (Eqn. 5):
a) It's not clear to me why this constitutes a meta-learning process: there is no adaptation at test time, nor does this perform task inference or learn how to learn. In this case, the method just uses a single gradient step to compute the change in predictions at the anchor points.
b) I wonder if there is a relationship to the loss used in Riemer et al, ICLR 2019. In particular, it seems that the gradient dot product term (used in that work to maximise transfer) may be related to a first-order Taylor expansion of the L2 term in this work (the change in predictions at anchor points, Eqn 5). It would be good to clarify whether this is the case, and add further discussion and intuition as needed.

3) On the relationship to existing work:
a) The approach seems conceptually most similar to iCARL (cited in the paper), which maintains "exemplars" from each previous task in order to avoid forgetting. I think the writing could be much more explicit about this, and both (i) compare against the performance of iCARL, and (ii) clearly discuss the differences between the iCARL exemplars and the anchors obtained in this work.
The equivalent result for splitCIFAR in the iCARL paper (for 20 tasks x 5 classes) appears to be around 45-50%, so I think it is necessary to compare and evaluate the source of this large improvement in performance.
b) Another paper worth discussing and comparing further is Variational Continual Learning - which is effectively a regularization-based method augmented with a coreset / episodic memory.
c) The paper seems quite centred on memory/replay- based approaches to continual learning; the abstract in particular says "SOTA continual learning methods implement different types of experience replay", but this fails to focus on non- memory-based methods. I would temper the claim to "many approaches to continual learning...", and spread the focus beyond replay-based approaches.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: *** Summary
This paper proposes to formulate the continual learning problem as a meta-learning problem where the multi-tasking and forgetting can be addressed simutaneously. To assess the forgetting, this paper suggests using the anchor points learned in hindsight. Empirical results show that this formulation performs better than previous methods.

*** Strengths
1. The idea of using meta-learning to deal with the forgetting issue is interesting. Using anchor points to represent previous tasks is a sensible solution.

2. The comparison with the previous shows that the proposed method performances consistently well on multiple tasks.


*** Weakness
1. The results in Table 3 deserves more discussion. The differences between different anchor are marginal compared with the standard deviations in Table 1. Especially, the result of Data-Anchors and HAL can be treated as the same. I wonder if the authors have any intuition behind this. Basically, this result is not persuasive for the effectiveness of the proposed anchor point learning method. Also, I’m curious to see what would happen if you have oracle access to future tasks which means we don’t need to learn the anchor points in hindsight.

2. The writing can be polished. There are several typos. For example, in the third line of page 3, “and following“ might be “following”. In the first line of Section 3, the colon after “state of the art” should be removed.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The technique of replay is well established. You have added a selection criterion for what to replay. You show a modest improvement over a random sample. You choose to view this as proof of the criterion, I on the other hand see it as proof of a very small result. To me it seems more like a negative result. I believe negative results are as important as positives. So I recommend acceptance. 

You may want to expand the selection criterion you evaluate. You selected the most leverage on forgetting what happens as you select less and less leveraged cases? How strong is this effect? How linear? etc... 
review_title: Official Blind Review #2

final_score: rating: null
review_content: Hi, in table 3 we see that the selected anchors vs randomly selected case makes almost zero difference. This seems like a worthwhile result in showing the current selection mechanism is not useful. 

You need to proof read for grammar, particularly between eq 4 and 5. 
review_title: table 3 

Meta-review: This paper proposes a continual learning method that uses anchor points for experience replay. Anchor points are learned with gradient-based optimization to maximize forgetting on the current task. Experiments MNIST, CIFAR, and miniImageNet show the benefit of the proposed approach.

As noted by other reviewers, there are some grammatical issues with the paper. 

It is missing some important details in the experiments. It is unclear to me whether the five random seeds how the datasets (tasks) are ordered in the experiments. Do the five random seeds correspond to five different dataset orderings? I think it would also be very interesting to see the anchor points that are chosen in practice. This issue is brought up by R4, and the authors responded that anchor points do not correspond to classes. Since the main idea of this paper is based on anchor points, it would be nice to analyze further to get a better understanding what they represent.

Finally, the authors only evaluate their method on image classification. While I believe the technique can be applied in other domains (e.g., reinforcement learning, natural language processing) with some modifications, without providing concrete empirical evidence in the paper, the authors need to clearly state that their proposed method is only evaluated on image classification and not sell it as a general method (yet).

The authors also miss citations to some prior work on memory-based parameter adaptation and its variants.

Regardless all of the above issues, this is still a borderline paper. However, due to space constraint, I recommend to reject this paper for ICLR.
Decision: Reject