Gradient-based Editing of Memory Examples for Online Task-free Continual Learning
Authors: Xisen Jin
Arka Sadhu
Junyi Du
Xiang Ren
Venue: NeurIPS 2021 Poster (2021)
Keywords: Task-free continual learning
memory-based continual learning
continual learning

Abstract: We explore task-free continual learning (CL), in which a model is trained to avoid catastrophic forgetting in the absence of explicit task boundaries or identities. Among many efforts on task-free CL, a notable family of approaches are memory-based that store and replay a subset of training examples. However, the utility of stored seen examples may diminish over time since CL models are continually updated. Here, we propose Gradient based Memory EDiting (GMED), a framework for editing stored examples in continuous input space via gradient updates, in order to create more "challenging" examples for replay. GMED-edited examples remain similar to their unedited forms, but can yield increased loss in the upcoming model updates, thereby making the future replays more effective in overcoming catastrophic forgetting. By construction, GMED can be seamlessly applied in conjunction with other memory-based CL algorithms to bring further improvement. Experiments validate the effectiveness of GMED, and our best method significantly outperforms baselines and previous state-of-the-art on five out of six datasets. 

Gradient-based Editing of Memory Examples for Online Task-free Continual Learning
Xisen Jin
Arka Sadhu Junyi Du Xiang Ren
University of Southern California
(xisenjin, asadhu, junyidu, xiangren@usc.
Abstract
We explore task-free continual learning (CL.) in which model is trained to avoid catastrophic forgetting in the absence of explicit task boundaries or identities. Among many efforts on task-free CL, notable family of approaches are memory- based that store and replay a subset of training examples. However, the utility of stored seen examples may diminish over time since CL models are continually updated. Here, we propose Gradient based Memory EDiting (GMED), framework for editing stored examples in continuous input space via gradient updates, in order to create more "challenging" examples for replay. GMED-edited examples remain similar to their unedited forms, but can yield increased loss in the upcoming model updates, thereby making the future replays more effective in overcoming catastrophic forgetting. By construction, GMED can be seamlessly applied conjunction with other memory-basee CL algorithms to bring further improvement. Experiments validate the effectiveness of GMED, and our best method significantly outperforms baselines and previous state-of-the-art on five out of six dataset!]
Introduction
Learning from a continuous stream of data referred to as continual learning (CL) or lifelong learning has recently seen a surge in interest, and many works have proposed ways mitigate CL models' catastrophic forgetting of previously learned knowledge 20 32 33 Here, we study online task-free CL (3], where task identifiers and boundaries absent from the data stream. This setting reflects many real-world data streams 66 25
and offers a challenging testbed for online CL research.
Memory- based methods, prominent class of approaches used for task-free continual learning, store a small number of training examples (from the data stream) in a memory and replay them at the later training iterations 33 34]. Existing methods operate over the original examples in the data-stream and focus on identifying samples to populate the memory [4] 9 and finding samples in the memory to be replayed (2). However, for continually updating models, using stored-seen examples in their original form, may lead to diminishing utility over time - i.e., model may gradually memorize the stored examples after runs of replay, the memory refreshes slowly. An alternate approach is to generative models to create samples that suffers more from forgetting such as in GEN-MIR in (2) In practice, training the generator network with limited data challenging and leads low-quality generated examples. Further, in the online learning setup, the generative model itself suffers from forgetting. As such, generative models perform worse than their memory counter-parts.
In this paper, we present a novel cemory-based CL framework, Gradient based Memory EDiting (GMED), which looks to directly "edit" (via small gradient update) examples stored in the replay memory. These edited examples are stored (replacing their unedited counterparts), replayed, and further edited, thereby making the future replays more effective in overcoming catastrophic forgetting. Code can found httttps///gthhubb
USC/GMED
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Since no explicit generative model is involved, GMED approach retains the advantages of memory- based methods and is straightforward train only inducing a small computation overhead.
The main consideration in allowing "editing" via a gradient update is the choice of the optimization objective. In light of recent work designing alternative replay strategies [2 7 40], we hypothesize that "interfering" examples (i.e., past examples that suffer from increased loss) should be prioritized for replay. For a particular stored example, GMED finds a small update over the example ("edit") such that the resulting edited example yields the most increase in loss when replayed. GMED additionally penalizes the loss increase the edited example to enforce the proximity of the edited example the original sample, so that the edited examples stay in on-distribution. As result, replaying these edited examples is more effective in overcoming catastrophic forgetting. Since GMED focuses only on editing the stored examples, by construction, GMED is modular, i.e., it can be seamlessly integrated with other state-of-the-arr memory-bases replay methods 2 5 26
We demonstrate the effectiveness of GMED with a comprehensive set of experiments over six benchmark datasets. In general, combining GMED with existing memory-based approaches results in consistent and statistically significant improvements with our single best method establishing a new state-of-art performance on five datasets. Our ablative investigations reveal that the gains realized by GMED are significantly larger than those obtained from regularization effects in random perturbation, and can be accumulated upon data augmentation to further improve performance.
To summarize, our contributions are two-fold: (i) we introduce GMED, a modular framework for task- free online continual learning, to edit stored examples and make them more effective in alleviating catastrophic forgetting (ii) we perform intensive set of experiments to test the performance of GMED) under various datasets, parameter setups (e.g., memory size) and competiting baseline objectives.
2 Related Work
Continual Learning studies the problem of learning from data stream with changing data distribu- tions over time (20, 23]. A major bottleneck towards this goal is the phenomenon of catastrophic forgetting (33 where the model "forgets" knowledge learned from past examples when exposed
to new ones. To mitigate this effect, a wide variety of approaches have been investigated such as adding regularization [9 30 43), separating parameters for previous and new data [24] 36 37), replaying examples from memory or a generative model [26] 33 38). meta-len [88. In this work, we build memory-base -base approaches which have been more successful in the online task-free continual learning setting that we study.
Online Task-free Continual Learning [3] is specific formulation of the continual learning where the task boundaries and identities are not available to the model. Due to its broader applicability to real- world data-streams, number of algorithms have been adapted to the task-free setup 2 14] 15 22 44)
In particular, memory-based CL algorithms which store a subset of examples and later replay them during training, have seen elevated success in the task-free setting. Improvements in this space have focused on: storing diverse examples as in muddient-based Sample Sclection (GSS) [4,, and replaying examples with larger estimated "interference" as in Maximally Interfered Retrieval (MIR) with experience replay [2). In contrast, GMED is used in conjunction with memory-based approaches and explicitly searches for edited example which is optimized to be more "effective" for replay.
Replay Example Construction. Moving away from replaying real examples, a line of works on deep generative replay [7 35 38 generates synthetic examples to replay with generative model trained online. GEN-MIR 2 is further trained to generate examples that would suffer more from interference for replay. However, training a generative network is challenging, even more so in the online continual learning setup where the streaming examples are encountered only once leading to poorly generated examples. Moreover, the forgetting of generative networks themselves cannot be perfectly mitigated. As a result, these methods generally perform worse than their memory- counter-parts. Instead of generating new example via a generator network, GMED uses gradient updates to directly edit the stored example thereby retaining advantages of memory-based techniques while creating new samples.
Novel examples can also be constructed via data augmentation (e.g. random crop, horizontal flip) to help reduce over-fitting over the small replay memory [5). Unlike GMED, these data-augmentations are usually pre-defined and cannot adapt to the learning pattern of the model. Constructing edited examples has also been used for adversarial robustness 31 39 The key difference lies in the

---

Estimate Forgetting (te.te
(b) Memory Example
(c) Update Model
Modal 9r
Trin
Tomporeiiyy Updated Model
(routro)
Model &
Updased Model Mien
(pp.Fo)
Stram Frsompies
6
Memory
(Ki Ned Updased esamples
Xin
(ow M Menarry Exsmpiics
Tiiin
dlecs Ctay Yad in Woe 8.) Ya Rai
Vyderes (Kis Yer)
Piis 6 -Ro (tp:
Figure/ 1: A schematic of GMED framework. (a) Given an example from the data-stream (ID, at time 4, the model randomly draws an example from the memory (Zm, Ym and estimates "interference" after one-step roll-out (Bq. D (b) The example drawn from the memory is then updated using our proposed editing objective (Eq. 3) via gradient ascent, resulting in (&m, ym), and written back into the memory. (c) Finally, the model updated using the edited example (fm; ym) and the example from the data-stream (#D.yD) (Eq.
optimization objective: while adversarial construction focuses on finding mis-classified examples [13
28 GMED aims constructing "interfered" examples that would suffer from loss increase in future training steps.
3 Method
We introduce the formulation of online task-free continual learning (Sec. 3.1 and then present our gradient-based memory editing method GMED (Sec. 3.2) and detail how GMED can be integrated with experience replay (Sec 3.3 and other memory-based CL algorithms (Sec. 3.4.
3.1 Problem Formulation
In continual learning (CL), we consider a (potentially infinite) stream D of labeled examples (x, y). having a non-stationn non-stationary data distribution - i.e., the data distribution p(x, y) evolves over time. Let (z,, yt) denote the labeled example (or mini-batch of examples) received by the model at time-step t in the data-stream D. We assume, for simplicity, that (o,, 34) is generated by first sampling a latent "task"
p(z; followed by sampling data example from conditional data distribution p(x, y|æ), - i.e., (Iz, y1) ~ p(z, yl2). Here p(2; t) is non-i.i.d and time-dependent. In task-free CL setting, the latent task 2 is not revealed to the model. Finally, we emphasize that while models in task-aware setup proceed to the next task only after convergence on the current task 119 43 this work focuses online task-free CL setup g 9
It simulates practical setting where models must perform online update on every incoming example, without accumulating examples within a task.
Following above definitions, our goal is to learn classification model f(; 0) on the data stream D that can preserve its performance over all the tasks in D. At time step t during the training process, the model is updated to minimize predefined empirical loss f(I,, Yt: 0) on the newly received example (Ia, yt), without increasing the loss on the previously visited examples (before t). Specifically, let Pa( y: T) denotes the distribution of the examples visited until the time step T. We look minimize the expected loss ( Ex , y; 0) - i.e., retaining performance on tasks encountered before T.
3.2 Gradient based Memory Editing (GMED)
In online task-free continual learning, examples visited earlier cannot be accessed (revisited) and thus computing the loss over all the visited examples (in D) is not possible. To deal with this challenge, memory-based based CL algorithms store and (continuously) maintain a set of visited examples fixed-size memory and use them for replay or regularization in the future training steps [26] 33 For instance, in Experience Replay (ER) 33 the examples are randomly sampled from the memory for replay; whereas recent works explore more sophisticated replay strategies, such as Maximally Interfered Retrieval with experience replay (ER-MIR) (2) where memory examples which interfere the most with the newly visited example are selected for replay. However, these algorithms only train over samples drawn from small replay memory in their original form. As such, the utility of the stored examples could diminish over time as the model could potentially memorize these examples the memory examples refreshed in slow pace which is found often the case (5].
We address the above limitation in our proposed approach Gradient based Memory Editing (GMED) by allowing examples stored in the memory to be edited in the continuous input space, illustrated in Figure T] The editing step is guided by an optimization objective instead of being pre-defined and involves drawing examples from the memory, editing the drawn examples, replaying the edited
3

---

examples and at the same time writing the edited examples back to the memory. We now state our optimization objectives of example editing followed by algorithmic details of GMED.
Editing Objective. Clearly, the most crucial step involved in GMED is identifying "how" should the stored examples in the memory be edited reduce catastrophic forgetting of early examples. If (Im, Mmm) denotes an example drawn from the memory M, the goal is to de- sign a suitable editing function o to generate the edited example 8m where =m==(<mm ). Such "editing" process can also be found in white-box adversarial example construction for robustness literature [3] where the task is to find an "adver- sarial" example (a', .y that is close to an orig-
Algorithm 1: Gradient Memory EDiting with ER
(ER+GMED)
1: Input: learning rate T, edit stride a, regularization
strength 88, decay rate 7, model parameters
2: Receives: stream example (2n,yD)
3: Initialize: replay memory M
4: for t to T do
5:
//when y=1, k is not required (Tm ym ) ~ M; k f Sepplayd__imm(/ Eoetore t- loss(rm. (Imm ymu 0.);
6:
7:
inal example (zx, y) but is mis-classified by the model. Typically, adversarial example construc-
8:
Exream +- loss(In, WD. 0.)
9:
/ //update model with stream examples
tion utilizes the gradients obtained from the clas- sifier and edits the examples to move towards a different target class. While, in the context of continual learning, the editing objective should be different. We employ similar hypothesis
10:
0/ 4- SGD((>srum 0,,
11:
//evaluate forgetting of memory examples
12:
later + loss((mm, Ymn, 6%)
as previous works [2 (4) that examples that
13:
d Eater Cbefore
likely forgotten by models should be prior-
14:
//edit memory examples
itized for replay. Accordingly, we edit examples
15:
a'm f- Im + u Bleeron)
so that they are more likely be forgotten in future updates. With this objective of editing
16:
f == loss( Mm)) (LD, )D) 0t)
17:
Oct1 + SSDD(E, T,
in place, we detail the process of incorporating GMED with Experience Replay (ER).
18: replace (2m, ymm with (2'm fmm in M
19: reservoir cetrvoir_uddat((r..... yo, M)
3.3 The GMED Algorithm with ER
20: end for
Algorithm| T summarizes the process and Figure provides a schematic of the steps involved in GMED. At time step f, the model receives a mini-batch of the stream examples (a yD) from the training stream D, and randomly draws same number of memory examples (ath ym) from the memory M, which we assume has already been drawn for replay for k times. We first compute the "interference" (i.e., loss increase) on the memory example (a ym) when the model performs one gradient update on parameters with the stream example (ED.UD)
0g = 0. Vol(ED 0D)
de 1 (n Mm)) = E(x Vm 0() (( Ymn Oz),
(2)
where 0, and 6, are model parameters before and after the gradient update respectively. Then, perform a step of gradient update on (a+h vi. by maximizing its "loss increase" in the next one step of training, while using a regularization term É( ym 0e) to penalize the loss increase evaluated with the current model checkpoint. The iterative update is written as,
++1 + 2tt + + [dea+1 (2),, Fm) - 3l(=%. Vm 0,)]
(3)
where the hyper-parameter a controls the overall stride of the edit and tuned with first three tasks together with the regularization strength 3. y is decay factor of edit performed on the model. A decay factor y less than 1.0 could effectively prevent 2 from drastically deviating from their original state, while 1.0 indicates no decay. We note that we cannot perform constrained edits that strictly adhere a distance budget w.r.t original examples, as it requires storing original examples which introduces extra memory overhead.
Following Eq.
perform gradient update on increase its "interference". The algorithm then discards 6/- and updates model parameters 9₂ using the edited memory example (a++1 9m) and the stream example (ID. VD). in a similar way to ER.
VoE([ .yn), We replace the original examples in the memory with the edited example. In this way, we continuously edit examples stored in the memory alongside training.
0++1 02
yD)}; 0.). (x .wn),
(4)

---

3.4 Applying GMED with Data Augmentation, MIR and GEM
Since the process to edit the original examples in GMED is modular, we can integrate GMED with a range of existing memory-based CL algorithms. In addition to ER, also explore ER with data augmentation agg)
[5) ER-MIR
2 and GEM (66 in our experiments.
ERang applies standard data augmentations (e.g., random cropping, horizontal flipping, denoted as 'T) to examples (I'm Dm) drawn from the memory which are replayed at each time step. In ERaug GMMDD we edit original example Im and replay both the augmented example T (z and the edited example &m. To keep the number of replayed examples the same, for Eaaug method, we also replay both the edited example T (3m) and the original example Tm Finally, we write the edited example 2m to the memory.
For integration with ER-MIR (denoted henceforth as MIR for brevity), recall that MIR retrieves and then replays the most "interfering" examples in the memory at each time-step. Thus, making GMED edits on the MIR MIR-retrieved examples may induce a loop of further increasing "interference" of the examples that are already the most "interfering" ones. Instead, in our GMED implementation, we edit mini-batch of examples randomly sampled from the memory a process that is indepen- dent of the MIR replay operation This random sampling may help prevent GMED editing from intensifying potential biases created from the MIR retrieval process (e.g., retrieved examples are edited and thus become more interfered). Similarly, For GEM+GMED, we also apply GMED to edit mini-batee of randomly sampled examples.
Details of the integrated algorithms (i.e., ERaug + +GMED, MIR+ and GEM+ can be and
found in Algorithms 2 and 4 ij Appendix B We leave more sophisticated integration of GMED to
existing CL. algorithms e.g. by optimizing the retrieval and editing operations jointly to future work.
4 Experiments
Our experiments address the following research questions: (i) what are the gains obtained by integrating GMED with existing memory-based CL algorithms and how these gains compare across datasets, methods and memory sizes? (ii) how useful are GMED edits alleviating catastrophic forgetting, and what part of it can be attributed to the design of the editing objective function?? (iii) what role do the various components and parameters in the GMED play and how they affect the performance. In the rest of this section, we first briefly describe the datasets (Sec. 4.T and the compared baselines (Sec. 4.2).. We then detail our main results comparing across memory- algorithms (Sec. (.3). validate the effectiveness of GMED-edits and the editing objective in mitigating catastrophic forgetting (Sec. 4.4). followed by ablative study over its components (Sec. 4.5).
4.1 Datasets
We use six public CL datasets in our experiments. Split/ Permuted Rotated MNIST are con- structed from the MNIST (21) dataset which contains images of handwritten digits. Split MNIST [12] creates 5 disjoint subsets based on the class labels and considers each subset as separate task. The goal then is to classify over all 10 digits when the training ends. Permuted MNIST 12] consists of 10 tasks, where for particular task a random permutation in the pixel space chosen and applied to all images within that task. The model then has to classify over the 10 digits without knowing which random permutation was applied. Rotated MNIST 26 rotates every sample in MNIST by fixed angle between 0 to 180. Similar to the previous datasets, the goal is to classify over 10 digits without any knowledge of the angle of rotation. For MNIST experiments, each task consists of 1,000 training examples following (2). We also employ Split CIFAR-10 and Split CIFAR- 100. which comprise of 5 and 20 disjoint subsets respectively based on their class labels. The model
then classifies over the space of all class labels. Similarly, Split mini- ImageNet (2) splits the mini- ImageNe [10 42 dataset into 20 disjoint subsets based on their labels. The models classify over all 100 classes.
Following the taxonomy of the Split MNIST, Split CIFAR-10, Split CIFAR- 100, and Split mini- ImageNet experiments are categorized under incrementre setup, while Permuted and Rotated MNIST experiments belong to domain-incremental setup. We note that our results are not comparable
"This also ensures the integrated approach will replay the same number of examples as the baselines, yielding a fair comparison.
5

---

Table 1: Mean and standard deviation of final accuracy (%) for on 6 datasets. For Split mini- ImageNet and Split CIFAR- 100 datasets, we set the memory size to 10,000 and 5,000 examples; we use 500 for other datasets. and over GMED methods indicate significant improvement over the counterparts without GMED with p-values less than 0.1 and 0.05 respectively in single-tailed paired t-tests. We report results in 20 for GEM, ER, MIR, and aaa and their (MED-inteeratee versions, and 10 runs for others. over "previous SOTA" results indicates that the best GMED method (bolded for each dataset) outperforms the previous SOTA with statistically significant improvement (p < 0.1).
based approaches
Split MNIST
Permuted MNIST
Rolated MNIST
Split CIFAR-10
Split CIFAR-100
Split mini-Imaged Net
Methods/ / Datasets
Fine tunine
18.80 = 0.6
66.34 2.6
41.24 = 1.5
18.49 0.2
3.06 0.2
2.84 = 0.4
AGEM X
29.02
72.17 = 1.5
50.77 = 1.9
18.49 0.6
2 0.2
2,92 0.3
ESS-Greddy
4
84.16 2.6
77.43 + 1.4
73.66 + 1.1
28.02 13
19:53 + 1.3
16.19 + 0.7
BGD 44
13.54 5.1
19.38 3.0
77.94
0.9
18:23 0.5.
3.11 1
24.71 0.8.8
ER 33
81.07 2.5
78.65 = 0.7
76.71
=: 1.6
33 2.9
20.11
25.92 1: 1.2
ER GMED
82.67"
78.86
0.7
77.09"
34.84"
20.93* +
27.27
MIR (2)
85.72 1. 1,2
79.13 1. 0.7
77.50 1: 1.6
34.42 2.4
20.02 1 1.7
25.21 1. 2.2
MIR GMED
86.52 1.4
79.25 0.8
79.08"
0.8
36. +
21.22
26.50"
ERRwy 5
80.14 3,2
78.11 = 0.7
80.04 = 1.3
46.29 2.7
18.32 + 1.9
30.77 = 2.2
ERasa GMED
82.21 = 2.9
78.13
0.6
80.61 1.2
47.47' 3.2
19.00 + 1.5
31.81" = 1.3
Previous SOTA
85.72|
79.23 0.7
80.04|
46.29| 1:
20,02!
30.77| 2.2
iid online
85,99 = 0.3
73.58 = 1.5
81.30 = 1.3
62.23 + 1.5
18.13 + 0.8
17.53 = 1.6
iid offline (upper bound)
93.87 0.5
87.40 1.I
91.38 0.7
76,36 0.9
42,00 0.9
37,46 1.3
to works that employ a different setup over the same dataset (e.g., results on Split CIFAR-100 are not comparable).
4.2 Compared Methods
We compare against several task-free memory based continual learning methods namely, Experience
Replay (ER) B3 Averaged Gradient Episodic Memory (AGEM) (8), Gradient based Sample
Selection (GSS) 4 and Maximally Interfering Retrieval (MIR) (2). We omit the generative replay
method GEN-MIR proposed together in 2 as it underperforms their memory- counterparts even on simple datasets such as Split MNIST.
We also compare with data augmentation flipping applied to memory examples drawn for replay in ER, noted as ERang (except for MNIST datasets). We also include regularization-based model expansion- based and task-aware approaches, namely Bayesian Graident Descent (BGD) [44), Neural Dirichlet Process Mixture Model (CN-
5 such as random rotations, scaling, and horizontal
DPM) P Progressive Networks (Prog.NN) 36). Compositional Lifelong Learning
(29 Graident Episodic Memory (GEM) (26 and Hindsight Anchor Learning (HAL) 7 respectively.
Finally, we report three baseline models: (i) Fine Tuming, where no continual learning algorithms are used for online updates to model parameters, (ii) iid Online, where we randomly shuffle the data stream, that the model visits an i.i.d. stream of examples, and (iii) iid Offline, where multiple passes over the dataset is allowed. Appendix A provides more details on compared methods and their implementation details. We build our proposed GMED approach upon four baselines, noted as ER+GMED, MIR+GMED, GEM+GMED, and +GMED
Implementation Details. We set the size of replay memory 10K for split CIFAR-100 and split mini-ImageNee. and 500 for all remaining datasets. Following [8. we tune the seper-parameters a (editing stride) and 3 (regularization strength) with only the first three tasks. While y (decay rate of the editing stride) is a hyper-parameter that may flcxibly control the deviation of edited examples from their original states, we find 1=01.0 (i.e., no decay) leads to better performance in our experiments. Results under different y setups are provided in Appendix C and in the remaining sections we assume no decay is applied. For model architectures, MNIST datasets, we use a MLP classifier with hidden layers with 400 hidden units each. For Split CIFAR-10, Split CIFAR-100 and Split mini-ImageNet datasets, we use a ResNet- 18 classifier with three times less feature maps across all layers. See Appendix C for more details.
mostly follow the setup of (2): for the three
4.3 Performance Across Datasets
We summarize the results obtained by integrating GMED with different CL algorithms. In Table [ we report the final accuracy and the standard deviation and make the following key observations.
Effectiveness of Memory Editing. As can be seen in Table [ GMED significantly improves perfor- mance on 5 datasets when built upon ER and MIR respectively. The improvement of MIR+GMED

---

90-
40
Accurpay (8))
HaP SPED
30
Aucuracy
-30
Accuraay
25
65-
1.00
2000
500
200
100
.000
530
50000
10000
5002
15000
20002
(a) Split MNIST (b) Rotated MNIST (c) Split CIFAR-10 (d) Split CIFAR-100 (e) Split mini-ImgNet
Figure 2: Performance of ER, GMED + ER, MIR, and GMED + +MIR with different memory sizes. For Mmane-l dataset, we memory sizes of 1K, 5K, 10K. and 20K examples: for Split CIFAR-100 dataset, use 1K, 2K, 5K and 10K; for other datasets, we use 100, 200, 500, and 1000. indicates whether the improvement of ER+GMED or MIR+GMED is significant with p < 0.05.
corroborates that the optimization in the continuous input spaces of GMED complementary to sample selection over real examples as in MIR. We also notice significant improvement of ER ERaug over GMED on 5 datasets. This indicates that the benefits induced by GMED go beyond regularization effects used to mitigate over-fitting
Comparison across CL methods. From Ta- ble [ we find MIR+GMED achieves the best performance on Split MNIST, Permuted MNIST, and Split CIFAR-100 100 datasets; while on Rotated MNIST, Split CIFAR-10 and Split mini-ImageNet dataset, ER +GMED achieves the best performance. Performance of the best performing GMED method could significantly improve over previous SOTA on five datasets.
Table 2: Comparison with model-expannson--assee ap- proaches under the same memory overhead as CN-DPM The overhead is the size of the replay memory plus the extra model components (e.g. a generator or modules to solve individual tasks), shown in the equivalent number of memory examples (#. Mem). indicates quoted numbers taken from the respective papers.
Method
Split MNIST
Split CIFAR-10
Split CIFAR-100
Acc.
Mom
Acc.
Mem
Acc.
#. Mem
CN-DPM
93.23
2,581
45.
6,024
20. 10
21,295
Prog. NN
89.46
9.755
49.68
6.604
19.17
31,766
We further compare with an non-memory based CL approach, CN-DPM (22). which employs generative model, a dynamically expanding classifier, and utilizes short-term memory (STM). Following the setup in [T6], we set the memory size for GMED so that two methods introduces the same amount of the overhead. Table shows the results of ER, ER+GMED and the reported results of CN-DPM. Interestingly, ER by itself achieves comparable performance to model expansion approaches. ER+GMED further outperforms CN-DPM without any extra memory overhead compared to ER. Similarly, GMED outperforms lask-aware model expansion approaches such as Prog. NN and the recently proposed compositional model expansion (CompCL) with smaller memory overhead.
CompCL
91.27
9.755
45 15.62
6,604
20.51
31,766
ER
92.67
2.581
62.96
6,024
21.79
21,295
ER+GMED
94.16
2,581
63.28
6.024
22.12
21,295
Performance under Various Memory Sizes. Figure 2 shows the performance of ER, ER+GMED, MIR, and MIR+GMED under var- ious memory sizes. The improvement on Split MNIST and Split mini-ImageNet are significant with p < 0.05 over memory size setups. On Rotated MNSIT and Split CIFAR-10 the im- provements are also mostly significant. The e improvement Split CIFAR- 100 is less com- petitive, probably because the dataset is overly difficult for class-incremental learning, from the accuracy around or less than 20% in all setups.
Table 3: Performance of methods over data streams with fuzzy task boundaries. In this setup, examples from the next tasks are introduced and gradually dom- inate the stream when half of the examples from the current task is visited. indicates whether the improve- ment is significant (p- < 0.05)
Split MNIST
Split CIFAR-10
Split mini-ImagoNet
Methods Datasets
Vanilla ER MIR ER...
21.53 - 0.1
20,69
: 2.4
3:05 = 0.6
79.74 1
37.15 4: 1.6
26.47 1: 2.3
38,70
1.7
81.30 4:
47.97 1:
30.75 :: 1.0
ER + GMED MIR ER.... GMRD
82,73* 2 2.6
40.57"
1.7
28.20" 0.6
Performance on Data Streams with Fuzzy Task Boundaries. The experiments Table assume clear task boundary. In Table 3 we report the result of using data-streams with fuzzy task
$2.39'
3.7
$1.38*
2.2
3183 = 0.8
boundaries on four datasets. We leave the complete results in Table Appendix. this setup, the probability density of a new task grows linearly starting from the point where 50% of examples of
7

---

Cosine Similal Similarity
0.5
6
/
/
2
3
0.0
Spilt MNST
6 / ( 2 2
2 2 8 3 3 3
-1.0
0
1/T 1/2 T
Time step
Figure 3: Cosine similarity between the optimal editing direction and direction from ER+GMED over training. The averaged similarity is 0.523 0.014 and 0.035 + 0.009.
Figure 4: Visualization of the edited examples in Split MNIST in the ER-GMED approach. The first two rows show examples before and after editing, and the third row shows the differences.
the current task are visited. In particular, GMED improves performance in three datasets across ER, MIR and ERang-
Effectiveness of GMED in Task-Aware Setting, We additionally compare performance of other task-aware approaches (HAL., GEM, GEM+GMED) in Appendix H
4.4 Ablation to Study the Effect of Memory-Editing
We present a set of experiments to validate that the gains obtained through the memory-cditing step of GMED are indeed helpful towards alleviating catastrophic forgetting. Further, we show that these gains are distinct from those obtained through random perturbations or simple regularization effects.
Comparison to Optimal Editing Direction. At particular time-step GMED identifies the edit direction using the stream example (rp. YD) and the memory sample (Im= Fm To validate whether the edits proposed by GMED are indeed helpful, compare GMED-edit to an "Optimal Edit" that minimizes the loss increase ("interference") of all early training examples in one future time step.
To compute this Optimal Edit, we note that the total loss increase over all previously encountered examples would be t1++1 Et dea+1 (xi, Wk) _E=1(
[6(2., y:; 02+1) f(x;, M:; 0z)]. where 0.+1- 02 Val(Tm, ym; 02) Vol(ap, yD; is the model parameters after the training update on (I YD) and (Im: ). The Optimal Edit direction for the memory example 2im would be the gradient of d w.r.t Im. Computing such optimal edits requires access to early training examples and is not practical in online continual learning setup; present it only for ablative purposes.
Figure shows the cosine similarity of update directions (i.e., the gradient of am between GMED and the optimal editing strategy over Split MNIST and Rotated MNIST datasets. The averaged similarity over time is 0.523 0.014, 0.035 0.009 on two datasets, averaged across 10 runs. Recall that random editing has expectation of zero similarity. On Split MNIST dataset, where the improvement the most significant, we notice a high similarity between the update directions of GMED and optimal editing. On Rotated MNIST where the improvement is less significant, the similarity is still positive on average but is lower than Split MNIST. It implies GMED-edits are generally consistent with explicitly reducing forgetting, but there is still space to improve. The results also imply the whether the edits of GMED aligns with editing is highly dependent on certain properties of datasets. We further include the classification performance of optimal editing in Appendix L
Comparison with alternative Editing Objec- tives. While GMED objective editing correlates with that of Optimal Edit, we further validate the choice of the objective function and consider two alternatives to the proposed editing objec- tive (Eq. 3 (i) Random Edit: memory examples are updated in a random direction with a fixed stride; (ii) Adversarial Edit: following the litera- ture adversarial example construction [[3], the edit increases the loss of memory by following the gradient sgn V, V_((xm) ym f). We report the comparison in Table 4
Table 4: Alternative memory editing objectives or simply increasing the number of replayed examples. Comparison of Random Edit, Adversarial Edit (Adv. Edit) our proposed objective in GMED. indicates significant improvement (p<0.005 compared to adver- sarial edit.
Methods Datusets
Rotated MNIST
Split CIFAR-10
Split mini- Netge-
33.30
25:92 1.2
+ Random ER + Adv. Edit
76.42
1.3
32.26
1.9
26 1.3
76.13
1.5
31.69 0.8
26.09 1.6
ER + GMED
77.50
1.6
34.84
1.2.
27.27' 1.8
Extra batch
78.07 1.1
33.81 +
24.94 + 1.5
Extra batches
77.10
32.86
24.65 1.5
We notice Randao Edit outperforms ER on split mini-ImageNNt, which indicates adding ran- dom noise to memory examples helps in regu- larization. Even so, GMED-edits are good or
34.42 2/4
25.21
Random
77.19
35.39
0.7
MIR
Adv. Edit
78.05
1.5
35.79 0.4
25.48 1.3
MIR GMED
79.08'
0.8
36.17 + 2.5
26.29" 1.2
8

---

Aouany FM
02
1..1 Scregge
Edting sorde
Dap. strungt
Eag: stweggn
(a) Split MINSTT
(b) Split CIEAR-10 10 (c) Split IigNe
(d) Split MNIST
(e) Split CIFAR-I0 ( Split mini -mgge
Figure 5: Parameter sensitivity analysis of the editing stride (a.b.c) and the regularization strength 3 (d.e.f) for ER+GMED across various datasets. The dashed horizontal line indicates the performance of ER.
outperforms both random and adversarial edits across multiple datasets and models. This validates choice of the editing objective.
Comparison to Increasing the Number of Replayed Examples. Recall that in MIR-GMED we sample two independent subset of memory examples to perform editing and replay (Sec. 3.4.. For fairer comparison, we replay one (or more) extra subset of examples. Table fable4shows the performance does not improve replaying more examples. We hypothesize that the performance is bottle-necked by the size of the memory and not the number of examples replayed.
Case study on Edited Memory Examples. In Figure we show visualizations of editing of memory examples. The examples are drawn from first two task (0/1, 2/3) in the Split MNIST dataset using ER+GMED. The first and second row show the original and edited examples, noted as Fbefore and Tafter respectively. The third row shows the difference between two Az = Tafter Tbefore. While there no significant visual differences between original and edited examples, in the difference Az, we find examples with exaggerated contours (e.g. examples (1) and (12)) and blurring (e.g. examples (2), (3), (5), and (6)). Intuitively, ambiguous examples are exaggerated while typical examples are blurred. Our visualizations supports this intuition: examples (1) and (12) are not typically written digits, while examples (2), (3), (5), and (6) are typical. Appendix Ceernddxxppprooe provides t-SNE (27) plot of the edit vector.
4.5 Analysis on the GMED Framework
Here, we analyze and ablate the various components used in the GMED framework and their effect on the final performance.
Hyper-parameter Sensitivity. In Figure we plot the sensitivity of the performance of GMED with respect to two hyper-parameters: the editing stride a and the regularization strength 3 (Eq. 33. Clearly, ER+GMED outperforms ER over broad range of and setups. Furthermore, in Figure fd,e,f), better performance for non-zero 3 confirms the benefit of the regularization term. Recall that in our main experiments we the dyper-paramenn the chosen hyperparameters improve the performance despite they are not always optimal ones.
and 3 with only first three tasks; we note
Computational Efficiency. We analyze the additional forward and backward computation required by ER+GMED and MIR. Compared to ER, ER+GMED adds 3 forward and backward passes to estimate loss increase, and backward pass to update the example. In comparison, MIR adds 3 forward and 1 backward passes with 2 of the forward passes are over a larger set of retrieval candidates. In our experiments, found GMED has similar training time cost MIR. In Appendix the wall-clock time, and observe the run-time of ER+GMED 1.5 times of ER.
B we report
Increasing Steps of Editing. For experi- ments in Table [ we performed one editing step over the sampled memory example I'm at time step f. In general, we can increase the number of editing steps. The direction for the edit is computed at each step, which makes the approach different from increas- ing the editing stride (a). Table [5jindicatss that 3-step and 5-step edits in general don't
Table Editing examples for multiple gradient steps in ER+GMED, tune the editing stride (a) using the first three tasks as described in Sec. 3.3
Split MINSTT
Rotated MNIST
Split CIFAR-10
Split mini-ImageNet
Methods/ Datasets
1-step Edit 3-step Edit
82.21 :: 29
77,50 à: 1.6
34.54 4: 2.2
27.27 4 1.8
83.55 1.9
77.37 1
3193 14
27,36 1.
S-step Edit
83.11
77.53
3682
26.36
lead to significant improvement in performance while incurring additional computational cost. As
such, we only 1 1-edit step across all experiments.

---

5 Conclusion
In this paper, we propose Gradient based Memory Editing (GMED), a modular framework for memory-based task-free continual learning where examples stored in the memory can be edited. Importantly, memory examples edited by GMED are encouraged to remain in-distribution but yield increased loss in the upcoming model updates, and thus are more effective at alleviating catastrophic forgetting. We find that combining GMED with existing memory-based CL approaches leads to consistent improvements across multiple benchmark datasets, only inducing a small computation overhead. Finally, we perform a thorough ablative study to validate that the gains obtained by GMED can indeed be attributed to its editing operation and careful choice of editing objective.
Acknowledgements
This research is supported in part by the Office of the Director of National Intelligence (ODNI). Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N66001 1924033, the Defense Advanced Research Projects Agency with award W91 INF-1920027, NSF IIS 2048211, NSF SMA 1829268, and gift awards from Google, Amazon, JP Morgan and Sony. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.
References
[1] Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (CLAW). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[2] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NeurIPS, 2019.
[3] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 11254 1 1263. Computer Vision Foundation IEEE, 2019.
[4] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' Alché Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 11816-11888 2019.
[5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experi- ence replay: a bag of tricks for continual learning. ArXiv, abs/2010.05595, 2020.
[6] Massimo Caccia, P. Rodríguez, O. Ostapenko, Fabrice Normandin, Min Lin, Caccia, Issam H. Laradji, I. Rish, Alexande Lacoste, D. Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. NeurIPS, 2020.
[7] Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip H. Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. ArXiv, abs/2002.08165. 2020.
[8] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A A-G.M. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[9] Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1952-1961. PMLR, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255 IEEE Computer Society, 2009.
10

---

[11] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty- guided continual learning with bayesian neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
[12] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
[13] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. ICLR, 2015.
[14] James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. In Hugo Larochelle, Marc" Aurelio Ranzato, Raia Hadsell, Maria- Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[15] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning, ArXiv, abs/1906.052000 2019.
[16] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating con- tinual learning scenarios: A categorization and case for strong baselines. arXiv preprint 1810: 10.10..14880 12488, 2018.
[17] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[18] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buee Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1818-1822 1828, 2019.
[19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(13)3521-3355
et al.
2017.
[20] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jía, A. Leonardis, Gregory Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv: Computer Vision and Pattern Recognition, 2019.
[21] Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits, 1998. URL http://yann. lecun. com/exdb/unist, 10:34, 1998.
[22] Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In International Conference Learning Representations, 2020.
[23] Timothée Lesort, V. Lomonaco, A. Stoian, D. Maltoni, David Filliat, and N. Rodríguez. Con- tinual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Inf. Fusion, 58:52 68, 2020.
[24] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97
of Proceedings of Machine Learning Research, pages 3925-3934, PMLR, 2019.
[25] B. Liu. Learning the job: Online lifelong and continual learning. In AAAI, 2020.
[26] David Lapez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 December 4-9, 2017 Long Beach, CA. USA, pages 6467-6476, 2017.

---

[27] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Novv:2557-266)) 2008.
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[29] Jorge A Mendez and ERIC EATON. Lifelong learning of compositional structures. In Interna- tional Conference on Learning Representations, 2021.
[30] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net 2018.
[31] Nicolas Papernot, P. McDaniel, Ian J. Goodfellow, S. Jha, Z. Y. Celik, and A. Swami. Practical black-box attacks against machine learning. Proceedings of the 2017 ACM on Asia Conference on Computer and Communicationn Security, 2017.
[32] G. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural networks: ; the official journal of the International Neural Network Society, 113:54-71, 2019.
[33] Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 7:123-146, 1995.
[34] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 348-358, 2019.
[35] Mohammad Rostami, Soheil Kolouri, and Praveen K. Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay. In Sarit Kraus, editor, Proceedings of the Twenty-Eigtth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3339-3345. ijeai.org, 2019.
[36] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, abs/1606.0467// 1, 2016.
[37] Joan Serrå, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4555 PMLR, 2018
[38] Hanul Shin, Jung Kwon Lee, Jachong Kim. and Jiwon Kim. Continual learning with deep gen- erative replay. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2990-2999, 2017.
[39] Yang Song. Rui Shu, Nate Kushman, and S. Ermon. Constructing unrestricted adversarial examples with generative models. In NeurIPS, 2018.
[40] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[41] Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. ArXiv, abs//904.07734, 2019.
[42] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Sys- tems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3638-3638, 2016
12

---

[43] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. ICML. 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995. PMLR, 2017.
[44] Chen Zeno, Itay Golan, E. Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. ArXiv, httss/8803.10022. 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL.?? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "z9qNT_wmXY", "reviewer": "Reviewer_CT4C", "summary": "In this paper, the authors propose a method called Gradient based Memory EDiting (GMED) for task-free continual learning. The motivation is to address one drawback of existing experience replay methods for continual learning, that is, the utility of the stored examples may diminish over time during training. The idea is to update the example in the memory to maximize its interference. The proposed method can be combined with existing continual learning methods to further boost the performance. The authors conduct extensive experiments and ablation studies to show the benefits of the approach. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality: The idea of modifying examples in the memory for continual learning is novel and interesting. Also, the proposed method is well motivated and simple to implement.\n\n\nQuality: The paper is technically sound and the authors include extensive ablation studies to show the benefit of the proposed GMED-edits. \n\n\nClarity: The paper is generally well-written and the method is clearly presented. \n\n\nSignificance: The improvements are not significant but meaningful. And the proposed method can be easily integrated with exising continual learning methods which is another benefit. \n\n\nQuestions:\n\n1. It would be interesting to see the predictions of the model on the edited examples, i.e., would the edits change the model predictions?\n\n2. It seems that how the examples are selected to store in the memory is another factor which can affect the performance of the method. Several strategies exist: i. Random Sampling, ii. Reservoir Sampling iii. GSS [1]. It would be interesting to see how the ways of storing examples would change the performance of the method. \n\n\n\n\n[1] Gradient based sample selection for online continual learning. \tNeurips 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Y5edUPgtxfV", "reviewer": "Reviewer_vw9x", "summary": "This paper introduces the Gradient based Memory Editing (GMED) framework for task-free online continual learning that edits stored examples using gradient updates. GMED can be easily applied with other memory-based continual learning algorithms to improve their performance. The paper conducts several experiments to validate their method.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The GMED framework proposed in this paper is interesting and novel. It helps improve the quality and flexibility of the memory/coreset in continual learning without the need to train a complex generative model as in other replay-based continual learning methods. The paper is well-written and the experiments are extensive. \n\nThe main drawback, in my view, is that the accuracy improvements of GMED in the experiments seem modest compared to the versions that don't use GMED. For example, in Table 1, there are several cases where the p-value is greater than 0.05, which is not a significant improvement.\n\nBelow are some other comments:\n1. In Algorithm 1, do we need to know the number of replays k for each example in the memory?\n2. In this algorithm, would it be better to edit all examples in the memory instead of only one example x_m? How about editing a mini-batch of examples?\n3. In Section 4.4, what are the accuracies when using the optimal edits? Maybe we can add them into Table 4 for a comparison.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XHasm4DqFV", "reviewer": "Reviewer_BUX3", "summary": "In this paper, the authors propose a technique for replay-based continual learning that is applicable to both task-aware and task-free settings.\nInstead of replaying the same memory over and over, each sample from the memory is updated to maximize the forgetting (the increase of loss) through gradient ascent.\nIts effectiveness is backed up by extensive experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Weak connection to task-free continual learning\nThe proposed method is indeed applicable to task-free continual learning.\nHowever, the main technique itself is not a necessary component for task-free continual learning.\nI think it is better to introduce it as a general technique that can cover both task-aware and task-free CL.\n\n### Extensive experiments\nThe authors tested multiple datasets and baselines with repeated experiments.\nAll numbers are reported with error bars.\nVarious ablation studies are also the strength of this paper.\n\n### Somewhat marginal improvements\nAccording to the reported scores, the proposed GMED consistently outperforms the baselines.\nHowever, the improvements seem rather small compared to added complexity.\n\n### Lack of deeper insights\nGMED performs better than other techniques, but there is not enough analysis about the reasons.\nAlthough I do not think this is a critical flaw but hope there are some explanations or hypotheses.\n\n### Summary\nThis paper proposes a simple technique that can improve replay-based CL methods.\nThe idea is not groundbreaking but has simple and understandable intuition behind it.\nAlthough the performance improvements are rather small, the experiments are thorough enough to show the efficacy of the technique.\nOverall, I think this is a borderline paper, but I am more on the side of accepting it since there is no significant issue.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The performance improvement is rather small.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9MMmHsAJlpI", "reviewer": "Reviewer_ggFa", "summary": "This paper proposes a replay-based method (GMED) for tackling the challenging and important problem of task-free continual learning (CL), which is concerned with mitigating catastrophic forgetting when learning from a non-i.i.d. stream of data without knowledge of task boundaries. The key idea of the method is to *edit* samples from the replay buffer during training in a way that maximises the interference that would be caused to the performance on these examples by a gradient update on the minibatch from the current task. The method is motivated by previous work that has shown that prioritising replay of samples that would be most interfered with mitigates catastrophic forgetting (e.g.[1]).\n\nExperiments are run demonstrating that GMED yields a statistically significant performance increase on a number of standard image recognition CL tasks (with discrete task boundaries and also with fuzzy task boundaries, which is more challenging and less common) when combined with a handful of existing replay-based methods ((namely experience replay (ER), ER with augmentations and Maximally Interfered Retrieval (MIR)). Further experiments are run to show that the method is robust to different memory sizes and to a range of values for the hyperparameters that the method uses to balance terms in the editing update. Additionally, experiments are also run to show that the type of editing used by GMED results in a better performance than random edits or adversarial edits to replay samples.\n\n[1] Aljundi, Rahaf, et al. \"Online Continual Learning with Maximal Interfered Retrieval.\" Advances in Neural Information Processing Systems 32 (2019): 11849-11860.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "POST-REBUTTAL update:\nI am increasing my score to a 7, after discussion with the authors has provided me with a better intuition of the motivation for editing examples, eg with reference to adversarial example construction, and the assurance from the authors that the paper will be updated accordingly to elaborate on the motivation behind the method. \n\n------------------------\nThis paper tackles a particularly challenging version of the CL problem, *task-free* continual learning, which is of great interest to the CL community as it moves closer to a real-world setting of streaming data with no task boundaries. One of the main positives of the paper is the extensive set of experiments demonstrating the benefit of the method over a range of tasks, with comparisons to several other methods, along with a good range of ablations and sensitivity analyses. The main drawbacks are that (i) the method has not got a strong theoretical justification and thus it’s not obvious why it works, and (ii) while many of the performance improvements are statistically significant, they are not that large in magnitude and most fall within a standard deviation of the baseline performance. Overall, I recommend for acceptance on the margin as I think the paper demonstrates that editing replay memory might be a promising direction for (a particularly challenging version of) continual learning and could inspire the community to pursue it.\n\nComments:\n- Lacking theoretical grounding. The loss function for the edit updates does not have a clear link to the main objective of mitigating forgetting on previous data. It’s not intuitive that modifying datapoints as GMED does should help to prevent forgetting since it moves them away from the original distribution of data - eventually one might imagine that this would increase forgetting if they move far enough away from the original datapoints. Encouragingly, the paper shows experiments that implement multiple edits (up to 5) per example, which in most cases marginally improves performance (at a computational cost), so perhaps this is not an issue. There is also a visualisation of some examples of edits, showing that there is some blurring or exaggeration of contours, but it’s hard to intuit why these would be helpful. Ideally, some more work would be done to understand why the model works, otherwise it is difficult to build on.\n- It was useful to run experiments that increased the number of replayed examples with MIR - this shows to some extent that the benefit of GMED is not simply due to a shift in the balance of replay updates to current task updates during training.\n- Having to tune the hyperparameters over the first three tasks is a limitation, though it has precedence in at least one paper as pointed out by the authors (A-GEM).\n- It was an interesting idea to correlate GMED’s edits with those of the optimal editing direction (that minimises forgetting on previous tasks by “cheating” and calculating with past data). Is there any intuition for why GMED aligns with the optimal editing direction better for split mnist vs rotated mnist? It would be interesting to see how the optimal editing direction version performs during training, to gauge what the upper bound of memory-editing is.\n- While the results show that combining GMED with other algorithms (namely experience replay (ER), ER with augmentations and Maximally Interfered retrieval) yields a statistically significant improvement in performance on numerous tasks, there are a few caveats: the absolute improvement is marginal (on the order of a percentage point), in each case the improvement falls well within 1 s.d. Of the baseline performance, and some of the improvements are only significant with p=0.1, which is an atypically large p-value.\n\nTypos/grammar:\n- Line 125. “Found often the case” -> “found often to be the case”.\n- Line 154 “an” -> “a”\n- Line 181-182. Do you mean to say that you replay the original example and the *augmented* example, rather than the *edited* example, since you are talking about ER_aug, which does not have example editing?\n- Line 235 and 236. “Graident” -> “Gradient”\n- Line 290. “MNSIT” -> “MNIST” \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the Gradient based Memory Editing (GMED) framework for task-free online continual learning that edits stored examples using gradient updates. The paper is well-written. The proposed method is interesting, novel, and has simple understandable intuition, though theoretical justifications are lacking. Experiments are extensive. While improvements are somewhat marginal, the experiments are thorough enough to show efficacy of the technique.
Decision: Accept (Poster)