Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a technique for replay-based continual learning that is applicable to both task-aware and task-free settings.
Instead of replaying the same memory over and over, each sample from the memory is updated to maximize the forgetting (the increase of loss) through gradient ascent.
Its effectiveness is backed up by extensive experiments.
main_review: ### Weak connection to task-free continual learning
The proposed method is indeed applicable to task-free continual learning.
However, the main technique itself is not a necessary component for task-free continual learning.
I think it is better to introduce it as a general technique that can cover both task-aware and task-free CL.

### Extensive experiments
The authors tested multiple datasets and baselines with repeated experiments.
All numbers are reported with error bars.
Various ablation studies are also the strength of this paper.

### Somewhat marginal improvements
According to the reported scores, the proposed GMED consistently outperforms the baselines.
However, the improvements seem rather small compared to added complexity.

### Lack of deeper insights
GMED performs better than other techniques, but there is not enough analysis about the reasons.
Although I do not think this is a critical flaw but hope there are some explanations or hypotheses.

### Summary
This paper proposes a simple technique that can improve replay-based CL methods.
The idea is not groundbreaking but has simple and understandable intuition behind it.
Although the performance improvements are rather small, the experiments are thorough enough to show the efficacy of the technique.
Overall, I think this is a borderline paper, but I am more on the side of accepting it since there is no significant issue.
limitations_and_societal_impact: The performance improvement is rather small.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the Gradient based Memory Editing (GMED) framework for task-free online continual learning that edits stored examples using gradient updates. GMED can be easily applied with other memory-based continual learning algorithms to improve their performance. The paper conducts several experiments to validate their method.
main_review: The GMED framework proposed in this paper is interesting and novel. It helps improve the quality and flexibility of the memory/coreset in continual learning without the need to train a complex generative model as in other replay-based continual learning methods. The paper is well-written and the experiments are extensive. 

The main drawback, in my view, is that the accuracy improvements of GMED in the experiments seem modest compared to the versions that don't use GMED. For example, in Table 1, there are several cases where the p-value is greater than 0.05, which is not a significant improvement.

Below are some other comments:
1. In Algorithm 1, do we need to know the number of replays k for each example in the memory?
2. In this algorithm, would it be better to edit all examples in the memory instead of only one example x_m? How about editing a mini-batch of examples?
3. In Section 4.4, what are the accuracies when using the optimal edits? Maybe we can add them into Table 4 for a comparison.
limitations_and_societal_impact: No potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a replay-based method (GMED) for tackling the challenging and important problem of task-free continual learning (CL), which is concerned with mitigating catastrophic forgetting when learning from a non-i.i.d. stream of data without knowledge of task boundaries. The key idea of the method is to *edit* samples from the replay buffer during training in a way that maximises the interference that would be caused to the performance on these examples by a gradient update on the minibatch from the current task. The method is motivated by previous work that has shown that prioritising replay of samples that would be most interfered with mitigates catastrophic forgetting (e.g.[1]).

Experiments are run demonstrating that GMED yields a statistically significant performance increase on a number of standard image recognition CL tasks (with discrete task boundaries and also with fuzzy task boundaries, which is more challenging and less common) when combined with a handful of existing replay-based methods ((namely experience replay (ER), ER with augmentations and Maximally Interfered Retrieval (MIR)). Further experiments are run to show that the method is robust to different memory sizes and to a range of values for the hyperparameters that the method uses to balance terms in the editing update. Additionally, experiments are also run to show that the type of editing used by GMED results in a better performance than random edits or adversarial edits to replay samples.

[1] Aljundi, Rahaf, et al. "Online Continual Learning with Maximal Interfered Retrieval." Advances in Neural Information Processing Systems 32 (2019): 11849-11860.
main_review: POST-REBUTTAL update:
I am increasing my score to a 7, after discussion with the authors has provided me with a better intuition of the motivation for editing examples, eg with reference to adversarial example construction, and the assurance from the authors that the paper will be updated accordingly to elaborate on the motivation behind the method. 

------------------------
This paper tackles a particularly challenging version of the CL problem, *task-free* continual learning, which is of great interest to the CL community as it moves closer to a real-world setting of streaming data with no task boundaries. One of the main positives of the paper is the extensive set of experiments demonstrating the benefit of the method over a range of tasks, with comparisons to several other methods, along with a good range of ablations and sensitivity analyses. The main drawbacks are that (i) the method has not got a strong theoretical justification and thus it’s not obvious why it works, and (ii) while many of the performance improvements are statistically significant, they are not that large in magnitude and most fall within a standard deviation of the baseline performance. Overall, I recommend for acceptance on the margin as I think the paper demonstrates that editing replay memory might be a promising direction for (a particularly challenging version of) continual learning and could inspire the community to pursue it.

Comments:
- Lacking theoretical grounding. The loss function for the edit updates does not have a clear link to the main objective of mitigating forgetting on previous data. It’s not intuitive that modifying datapoints as GMED does should help to prevent forgetting since it moves them away from the original distribution of data - eventually one might imagine that this would increase forgetting if they move far enough away from the original datapoints. Encouragingly, the paper shows experiments that implement multiple edits (up to 5) per example, which in most cases marginally improves performance (at a computational cost), so perhaps this is not an issue. There is also a visualisation of some examples of edits, showing that there is some blurring or exaggeration of contours, but it’s hard to intuit why these would be helpful. Ideally, some more work would be done to understand why the model works, otherwise it is difficult to build on.
- It was useful to run experiments that increased the number of replayed examples with MIR - this shows to some extent that the benefit of GMED is not simply due to a shift in the balance of replay updates to current task updates during training.
- Having to tune the hyperparameters over the first three tasks is a limitation, though it has precedence in at least one paper as pointed out by the authors (A-GEM).
- It was an interesting idea to correlate GMED’s edits with those of the optimal editing direction (that minimises forgetting on previous tasks by “cheating” and calculating with past data). Is there any intuition for why GMED aligns with the optimal editing direction better for split mnist vs rotated mnist? It would be interesting to see how the optimal editing direction version performs during training, to gauge what the upper bound of memory-editing is.
- While the results show that combining GMED with other algorithms (namely experience replay (ER), ER with augmentations and Maximally Interfered retrieval) yields a statistically significant improvement in performance on numerous tasks, there are a few caveats: the absolute improvement is marginal (on the order of a percentage point), in each case the improvement falls well within 1 s.d. Of the baseline performance, and some of the improvements are only significant with p=0.1, which is an atypically large p-value.

Typos/grammar:
- Line 125. “Found often the case” -> “found often to be the case”.
- Line 154 “an” -> “a”
- Line 181-182. Do you mean to say that you replay the original example and the *augmented* example, rather than the *edited* example, since you are talking about ER_aug, which does not have example editing?
- Line 235 and 236. “Graident” -> “Gradient”
- Line 290. “MNSIT” -> “MNIST” 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a method called Gradient based Memory EDiting (GMED) for task-free continual learning. The motivation is to address one drawback of existing experience replay methods for continual learning, that is, the utility of the stored examples may diminish over time during training. The idea is to update the example in the memory to maximize its interference. The proposed method can be combined with existing continual learning methods to further boost the performance. The authors conduct extensive experiments and ablation studies to show the benefits of the approach. 
main_review: 
Originality: The idea of modifying examples in the memory for continual learning is novel and interesting. Also, the proposed method is well motivated and simple to implement.


Quality: The paper is technically sound and the authors include extensive ablation studies to show the benefit of the proposed GMED-edits. 


Clarity: The paper is generally well-written and the method is clearly presented. 


Significance: The improvements are not significant but meaningful. And the proposed method can be easily integrated with exising continual learning methods which is another benefit. 


Questions:

1. It would be interesting to see the predictions of the model on the edited examples, i.e., would the edits change the model predictions?

2. It seems that how the examples are selected to store in the memory is another factor which can affect the performance of the method. Several strategies exist: i. Random Sampling, ii. Reservoir Sampling iii. GSS [1]. It would be interesting to see how the ways of storing examples would change the performance of the method. 




[1] Gradient based sample selection for online continual learning. 	Neurips 2019

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper introduces the Gradient based Memory Editing (GMED) framework for task-free online continual learning that edits stored examples using gradient updates. The paper is well-written. The proposed method is interesting, novel, and has simple understandable intuition, though theoretical justifications are lacking. Experiments are extensive. While improvements are somewhat marginal, the experiments are thorough enough to show efficacy of the technique.
Decision: Accept (Poster)