Gradient Based Memory Editing for Task-Free Continual Learning
Authors: Xisen Jin
Junyi Du
Xiang Ren
Venue:  (2021)
Keywords: Continual learning
task-free continual learning

Abstract: Prior work on continual learning often operate in a “task-aware” manner, by assuming that the task boundaries and identifies of the data examples are known at all times. While in practice, it is rarely the case that such information are exposed to the methods (i.e., thus called “task-free”)–a setting that is relatively underexplored. Recent attempts on task-free continual learning build on previous memory replay methods and focus on developing memory construction and replay strategies such that model performance over previously seen examples can be best retained. In this paper, looking from a complementary angle, we propose a novel approach to “edit” memory examples so that the edited memory can better retain past performance when they are replayed. We use gradient updates to edit memory examples so that they are more likely to be “forgotten” in the future. Experiments on five benchmark datasets show the proposed method can be seamlessly combined with baselines to significantly improve the performance.

Under review as a conference paper at ICLR 2021
GRADIENT BASED MEMORY EDITING FOR TASK-FREE CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning often assumes knowledge of (strict) task boundaries and identities for the instances in a data stream-i.e, a "task-aware" setting. However, in practice it is rarely the case that practitioners can expose task information the model; thus needing "task-free" continual learning methods. Recent attempts towards task-free continual learning focus on developing memoryycconstruction and replay strategies such that model performance over previously seen instances is best retained. In this paper, looking from a complementary angle, we propose "edit" memory examples to allow for the model to better retain past performance when memory replayed. Such memory editing is achieved by making gradient updates to memory examples so that they are more likely to be "forgotten" by the model when viewing new instances in the future. Experiments on five benchmark datasets show proposed method can be seamlessly combined with baselines significantly improve performance and achieve state-of the-art results.
INTRODUCTION
Accumulating past knowledge and adapting to evolving environments are one of the key traits in human intelligence (McClelland et al. 1995). While contemporary deep neural networks have achieved impressive results in range of machine learning tasks Goodfellow et al. (2015). they
haven't yet manifested the ability of continually learning over evolving data streams (Ratclifi 1990).
These models suffer from catastrophic forgetting (McCloskey & Cohen 1989; Robiin 1995 when
trained in an online fashion i.e., performance drops over previously seen examples during the sequential learning process. To this end, continual learning (CL) methods are developed alleviate catastrophic forgetting issue when models are trained on non-stationary data streams (Goodfellow al. 2013).
Most existing work on continual learning assume that, when models are trained on a stream of tasks sequentially, the task specifications such as task boundaries or identities are exposed to the models. These task-aware CL. methods make explicit use of task specifications to avoid catastrophic forgetting issue, including consolidating important parameters on previous tasks (Kirkpatrick et al.
2017 Zenke et al., Nguyen al. 2018). distilling knowledge from previous tasks (Li & Hoiem Rannen et al. 2017). ) or separating task-specific model parameters (Rusu et al. 2016 Serrà al. 2018). However, in practice, it is more likely that the data instances comes in a sequential, non non-stationary fashion without task identity or boundary- a setting that is commonly termed as task-free continual learning (Aljundi et al.) 2018). To tackle this setting, recent attempts task-free
2017
CL methods have been made (Aljundi et al. 2018 Zeno et al.] 2018 Lee et al., 2020). These efforts revolve around regularization and model expansion based approaches, which rely on inferring
task boundaries or identities (Aljundi et al. 2018 Lee et al. (2000) and perform online paramater importance estimation (eno al. 2018), consolidate or separate model parameters.
In another line of efforts, memory- CL methods have achieved strong results in task-free set- ting Aljundi et al. (2019b). These methods store small set of previously seen instances in a fix-sized memory. and utilize them for replay (Robins] 1995] Rolnick et al.] 2019) or regularization (Lopez- Paz Ranzato 2017 Chaudhry et al.) 2019a). The core problem in memory- -base CL methods is how to manage the memory instances (e.g., which to replace with new instances) and replay them given a restricted computation budget, so that the model performance can be maximally preserved or
Code has been uploaded in the supplementary materials and will be published.

---

Under review as a conference paper at ICLR 2021
Gradieut Based Memory Editing
Reservoir Sampling
Sumpling from Memary
(GMED) (A02oo1))
Model 0
Training Model
Hindsight Ancher Leaning (HAL)
Gradient Based Sample Selection Aljundi (20196)
Maximally inserfered
Retsioval (MIR) Aljundi et al. (20196)
(9--9) -M
Chaudhry et al. (20220
(1.3) M
Stream Examples
Memory Construction
Memory Examples
Replay Strategy
Figure 1: Categorization memory-based methods on Task-free Continual Learning. Reservoir sampling and Sampling from Memory are the ways that Experience Replay (ER) uses construct and replay memory respectively. In the recent line of works, Gradient based Sample Selection (GSS) and Hindsight Anchor Learning (HAL) explored ways to construct memory, while Maximally Interfering Retrieval (MIR) focused on replay strategy. Our method, Gradient-based Memory Editing (GMED) angle to memory construction.
falls under the former category but provides
enhanced. Prior work developing these methods have tried to either identify: 1) what instances to include in memory from a data stream (Aljundi al., 2019b Rebuffi et al. 2017 Chaudhry et al. 2019b); and 2) which instances in memory need to be replayed at what training step (Aljundi et al. 2019a).
In this paper, we provide a new approach to solving the memory management problem in task-free continual learning by studying how to make gradient updates on stored memory examples. We develop novel memory editing algorithm which complements existing memory-replay methods and data-sampling strategies for memory management (updates). The challenge to propose plausible and sound optimization objective of editing. We employ the same intuition as previous study (Toneva) (ct al.) 2019: Chaudhry et al. 2020 Aljundi et al.) 9019a): examples that are likely to be forgotten should be prioritized. Our proposed method, named Gradient traseent-based Memory EDiting (GMED), edits examples stored in the memory with gradient-basee updates so that they are more likely to be forgotten. Specifically, we estimate the "forgetting" of stored example by its loss increase in the upcoming one online model update. Finally, we perform gradient ascent on stored examples so that they more likely to be forgotten.
Experiments show that our algorithm consistently outperforms baselines on five benchmark datasets under various memory sizes. Our ablation study shows the proposed editing mechanism outperforms alternative editing strategies such as random editing. We demonstrate that the proposed algorithm general enough be used with other strong (more recent) memory-basee based CL methods further enhance performance, thus allowing for improvements in many benchmark datasets.
RELATED WORKS
Task-aware Continual Learning. Most of continual learning algorithms studied under "task- aware" settings, where the model visits a sequence of clearly separated "tasks". A great portion of algorithms make explicit of task boundaries (Kirkpatrick et 2017 Rusu al. 2016 Lopez-Paz & Ranzato 2017). by learning separate parameters for each task, or discourage changes of parameters that are important to old tasks. Existing continual learning algorithms can be summarized into three categories: regularization-base based, architecture-bast
and data-based approaches. Regularization
based approaches (Kirkpatrick et al. 2017 Zenke al. 2017 Nguyen et al. 2018 Adel al. 2020) discourage the change of parameters that are important to previous data. Model expansion- based approaches (Rusu et al.) 2016 Serra et al. 2018 Li et al. 2019) allows expansion of model
architecture to separate parameters for previous and current data. Data-based approaches (Robins) 1995 Shin et al. [2077] Lopez-Paz & Ranzato 2017 replay or constrain model updates with real or synthetic examples.
Task-free Continual Learning. Recently, task-free continual learning (Aljundi et al., 2018) have drawn increasing interest, where we do not assume knowledge about task boundaries. To the best of our knowledge, only a handful number of regularization based (Zeno et al.) [0188] Aljundi et al. 2018), model-expansion based (Lee et al. 2020), generative replay based (Rao al. 2019), continual meta-learning and meta-continual learning (He et al. 2019 Caccia et al. 2020 Harrsonn et al. 2020) approaches are applicable in the task-free CI. setting. Meanwhile, most memory based continual
2

---

Under review as a conference paper at ICLR 2021
(x) Estinutee forgetting (t.y)
(b) Upuate x
(c) Updlate 92
Model
Train
Tempurally Updaited Model
v y0000,
Stramm Esamples
y(01)
Model 9:
Upditedd Model Ores
(x ((0)))())) Stream Examples
(x,))
KGx 02)
xx
Mensor Esamples
Updated
(r,y)- M Mamory Eesmppos
des-- (x.9)) y: (0) P(x.y:0)
(x.)
Pe-s 6 Vo (((t') ()) (x0))))))))))))))))))))))))))))) ):)))
Figure 2: Overview of the (a) forgetting estimation, (b) example editing, (c) model updating in the proposed GMED method. See detailed formulations for forgetting estimation and example editing at Section 4.3
learning algorithms are applicable to the task-free setting (Aljundi et al. 2019a b). Memory-basee CL algorithms such as Experience Replay (ER) (Robins) 1995) store a subset of examples in fix-sized replay memory and utilize them later at training to alleviate forgetting. Recent research has studied online strategies to improve the performance gain when examples get replayed from two dimensions: in terms of which examples to store, and which examples to replay. For example, in terms of deciding which examples to store, Gradient based Sample Selection (GSS) (Aljundi et al., 2019b) proposes to store most diverse examples. In terms of deciding which examples to replay, maximally Interfering Retrieval (MIR) (Aljundi et al., (019a) select examples with the largest estimated forgetting. In particular, task-aware approach, Hindsight Anchor Learning (HAL) (Chaudhry et al.) (2020), shares the same assumption that forgettable examples should be prioritized more. However, HAL only applies task-aware settings and requires extra memory storage keep track of the learned anchors. Figure I shows categorization of memory-based fask-free continual learning.
3 PRELIMINARIES
In this section we first present the problem formulation of task-free continual learning and then introduce preliminaries on memory-based continual learning methods.
3.1 PROBLEM FORMULATION
In task-free continual learning, we consider a (potentially infinite) stream of data examples D, which have non-stationary data distribution, i.e., the data distribution P(x., y) over time.At each time step I, the model receives a single or a mini batch of labeled examples (#x Me) from the data stream D. For simplicity, here we assume that example (Ir, Vt) from D is generated by: first sampling a latent "task" ~ P(z; t). followed by sampling data example from a joint data distribution P(x, 2|2; that is conditioned on task 2, i.e., (I2, yc) P(x, t|2; Here P(2; t) is non-i.i.d and lime-dependent Similarly, P(z, 8/2: t) also changes over time.
The goal of free online continual learning by 0, over new example(s) (xc. y) from the data stream D that minimizes predefined loss {(r, y; O) while not increasing the loss on previously seen examples. This capability evaluated by testing the model over test set of all visited tasks.
seek classification model f(z; (); parameterized
3.2 MEMORY-BASED CL METHODS
Briefly, memory- based CL algorithms maintain fix-sized replay memory M which is used to store (subset of) previously seen examples (z,, from the stream D. When the memory is full. the algorithm needs to either identify a memory example (æ, y) to be replaced by new example, or to discard the new example it just received. Following the same setup in previous memory- methods, our experiments use reservoir sampling (Vitter 1985) to determine how the memory will be updated with new examples received from stream D. Every time the model receives new example, it draws an integer j between 0 and N randomly, where is the number of examples visited so far. If < |M| (i.e., the memory size budget), it replace the example at the j-th position in the memory with the new example; otherwise, this newly received example will be discarded. Reservoir sampling ensures at each time step each visited example is kept with equal probability |M/N..
CL
3

---

Under review as a conference paper at ICLR 2021
At each time step i, the algorithm also needs determine the memory examples to be used for replay. Similar to previous methods, we randomly sample one or a mini batch of examples (z, y) from the memory M. As alternative replay strategy, MIR (Aljundi al.] 2019a) identifies subset of memory examples based on predefined optimization objective (i.e, perform one step of training on (ar, y)), and then replays the selected examples.
GRADIENT BASED MEMORY EDITING
We propose Gradient based Memory Editing (GMED), a novel algorithm for updating stored memory examples in an online fashion. We state our hypothesis about which examples should be stored in Sec. We then formulate an online optimization objective for example editing in Sec. 4.2 In we introduce algorithmic details of GMED and its integration with MIR.
Sec.
4.1
HYPOTHESIS FOR MEMORY EDITING
As there is no prior knowledge about the forthcoming examples in a data stream D, previous task-free CL methods usually impose (implicit) assumptions regarding what kinds of examples may improve model's test performance after replaying these examples. For example, GSS (Aljundi et al. (019b) assumes that the diversity of memory examples contributes the model performance; MIR (Aljundi et al. 2019a) and HAL [Chaudhry et al. (0200) assume that replaying examples that are likely to be "forgotten" can benefit the performance. Empirical study by TTonva al. (2019) also shows that there are constantly forgotten examples, which benefit overall performance when they get replayed compared to other examples.
Our work is based on similar hypothesis: replaying examples that are likely to be forgotten by the current model helps retain its test performance. Specifically, suppose we train the model on until a time step T, the "forgetting" measurement of an example (ær, y) for the model at t, denoted by de.T (z, y). is defined as the "loss increase" at time T compared to that at time t, shown as follows.
der(x,)) y) = E(x, y; 0r) - f(x, y; 0t).
where 0₂ is the model parameters at the current time step t. A larger (x. y) indicates that the example suffers more forgetting at the end of training. The hypothesis is formally stated as follows.
Hypothesis 1. Given budget of C examples new examples from a (latent) task ls (e.g.. in test set), an ideal strategy is to replay the most most forgettable examples, denoted Sk, selected from the training examples of the (latent) task k; denoted Da-
replay at time t, in order minimize the loss over
Following Hypothesis the set of examples Sp can be obtained by solving the following optimization problem.
S arg max
Σ de.((r
S. S.C. (a,
where Dr is the training examples of the latent task k. Unfortunately, the sample selection problem in Eq. cannot be solved in online setting, even when we know task identities, because the objective function in Eq. 2 can only be evaluated at a distant future time step T. Therefore, approximations are necessary to enable online optimization.
4.2 ONLINE OPTIMIZATION FOR MEMORY EDITING
We propose optimization problem that is tractable in an online fashion, which shares the same goal of making memory examples used for replay more likely to be forgotten. We modify Eq. Eq.2 where we (I) relax the constraint S₁ C D₁ and edit individual examples stored in memory instead of selecting the most forgettable examples from D₁ to store, and (2) estimate the forgetting measure in an online fashion.
Formally, suppose that the model is at the t-th time step and that (I, y) is an example from a certain task ht In order to retain the performance on test examples from the same task k; we propose the following objective:
x arg max
(x.41(x,)) 36(x, 0z).
(3)
x
4

---

Under review as a conference paper at ICLR 2021
where d(.) is the forgetting defined in Eq. | and ( y; 0t) is the loss of the example (x. y) the current time step t. , is a trade-off hyper-paramen deciding the regularization strength. The optimal (x* y) has the same label y as the original example (æt, y).
Specifically, we discuss two main differences in Eq. compared to Eq. 2 in the Hypothesis !
Estimating Forgetting Online. We maximize the forgetting of the example (xx, y) in the upcoming update (time step t + 1, noted as de:s-1 (z, 8)), instead of the forgetting when the model get evaluated, i.e. dzr y). The former can be evaluated online efficiently without any overhead on the replay memory.
Relaxed Constraints. Eq. do do not constrain (2, E Dr. It allows (x", v) be arbitrary example in the input space without being a real example from D3: the sample selection problem posed as an optimization problem in the continuous space. Nevertheless, the editing is made conservatively, so that the edited example is still likely to be an example from the original latent task. In practice, (z. y) is initialized as different input examples, and we perform only one or a few gradient updates each time it is drawn from memory for replay. We also add a regularization term 3l(x, y; 0+) to discourage the loss increase on the example.
The objective in Eq. is differientiable with respect to allowing us to update x with gradient ascent. In the rest of this section, we introduce algorithmic details of GMED.
4.3 THE GMED ALGORITHM
We start by introducing the algorithm of building GMED upon the ER (Experience Replay) idea. It in- troduces an additional "editing" step before replaying examples drawn from the memory.
Algorithm 1:
ER with Memory Editing
Input: learning rate т, edit stride regularization strength 3, model parameters B; Receives:
stream example (xx(D) y))))
We assume time step the model receives a stream example (xx(D), y/D) from the training stream D, and randomly draws memory example (x., y) from the memory M. We first compute the forgetting (i.e. loss increase) on the memory example (zz, y) when the model performs one gradient update on parame- ters with the stream example (z:(D), (((D),(()) (P))).
Initialize: replay memory M;
for L = 1 to T do
(x, 8) M: feefore + loss( y: oo): Exrca loss (D) y (D) Bs);
//pdatte model parametera with stream exemgies, discarded later;
0, SGD((anmmm Oz, T);
0f 0==0- - (o((x()) (P) d 1 v) Mx, y, 03) lxx yi 0t),
(4)
/eealluaee forgetting memozy examples:
(5)
Later f- loss(2, y, 0(); Latte Pxctorei
where Or and 0% are model parameters before and after the gradient update respectively. Figure 2[a) visualize the steps to compute forgetting.
//edtt nemory examples
x' a Bloctce);
Following the optimization objective proposed in Eq. 2 we perform gradient update on z to increase its forgetting, while using a regularization term to discourage the loss increase the example at the current time step.
= (2s 1) U (D) .y (P)) ),0,); 0,+1 SGD(E, 0,, T):
replace (I. with (u', y) in M; reservoir reservoir.update( (D)), M);
r'=xth + V (2,441(0,))) - 30(x, y: Bc)).
(6)
end
where a is a hyperparameter for the stride of the update. Figure 2(b) visualize the editing step.
The algorithm then discards the updated parameter 6%, and updates model parameters 6r with the updated memory example (a', y) and the stream example (xx(P), ((()),(()) ((i)), in a similar way to ER.
Vol(((>', v), (x (P))): 02). 0++ 0,
We replace the original examples in the memory with the edited example. In this way, continuously edit examples stored in the memory alongside training. Algorithm summarize the proposed ER+GMED algorithm.
5

---

Under review as a conference paper at ICLR 2021
GMED is studied from a complementary direction compared to most prior approaches. Therefore, can combine GMED with existing memory-basee CL algorithms without much effort. We illustrate the point by proposing hybrid approach of GMED and MIR. We include the details of the algorithm in the Appendix.
5 EXPERIMENTS
We compare the performance of GMED against state-of-the aftt CL algorithms five benchmark datasets. We introduce our experimental setup and discuss our results on comparisons with baselines and performance analysis.
5.1 DATASETS
We consider six public CL datasets our experiments.
Split / Permuted/ / Rotated MNIST are constructed from the MNIST (LeCun et al. 1998) dataset of handwritten digit classification. Split MNIST (Goodfellow et al. 2013) partitions the dataset into 5 disjoint subsets by their labels different tasks. The goal is to classify over all 10 digits when the training ends. Permuted MNIST [Goodfellow et al., 2013) applies a fixed random pixel permutation to the MNIST dataset different tasks. The dataset consists of 10 tasks. The models classify over 10 digits without knowing the permutation applied. Rotated MNIST (Lopez-Paz & Ranzato (017) applies fixed image rotation between 180 degree the MNIST dataset. Similarly, the goal is to classify over 10 digits without knowing the rotation applied. Following Aljundi et al. (2019a), for MNIST experiments, each task consists of 1,000 training examples.
Split CIFAR-10 and Split CIFAR- 100 (Zenke et al. 2017) are constructed by splitting the CIFAR- 10 or CIFAR-100 (Krizhevsky 2009) of image classification into 5 or 20 disjoint subsets by their labels. The model classifies over all 10 or 100 classes when the training ends.
Split mini-ImageNet (Aljundi al. 2019a) splits the mini-ImageNe (Deng et al. 2009 Vinyals et al. 2016) image classification dataset into 20 disjoint subsets by their labels. Similarly, the models classify over all 100 classes.
We do not provide information about task identities or boundaries to the model at both training and test time. Under the taxonomy of the prior literature (van de Ven & Tolias 2019), our Split MNIST, Split CIFAR- 10, and Split mini-ImageNet experiments are under the class-incremental setup, while Permuted MNIST and Rotated MNIST experiments are under the domain-inccemmental setup.
5.2 COMPARED METHODS
For our methods, we report the performance of ER + GMED and MIR + GMED, where we build
GMED upon ER or MIR as introduced in section 4.3 We compare with several frek memory
based continual learning methods: Experience Replay (ER) (Robins) 1995 Rolnick et al., 2019).
Averaged Gradient Episodic Memory (AGEM) (Chaudhry et al.) 2019a). Gradient based Sample
Selection (GSS) (Aljundi et al. (2019b), Maximally Interfering Retrieval (MIR) (Aljundi et al. 2019a)
We also compare with Bayesian Graident Descent (BGD) (Zeno et al. 2018) and Neural Dirichlet Process Mixture Model (CN-DPM) (Lee et al., 2020), which are regularization and model-expansion based approaches respectively. We also include Graident Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017) and Hindsight Anchor Learning (HAL) (Chaudhry et al. (220)). which are task-aware methods. We also reports the results of simple fine tuning, which performs online updates on model parameters without applying continual learning algorithms; and iid Online, where we randomly shuffle the data stream, so that the model visits an i.i.d. stream of examples; and also iid Offline. where we further allow multiple pass over the dataset. See appendix for detailed descriptions of compared methods and their implementation details.
By default, we set the size of replay memory as 10,000 for split CIFAR-100 and split mini- mini-ImageNNtt and 500 for all other datasets. We also report performance under various memory sizes. Follow- ing (Chaudrry& et al. (2019a), we tune the hyperparameters with only the training and validation set of first three tasks. We mostly follow the training setup of Aljundi et al. (2019).. For three MLP classifier with 2 hidden layers with 400 hidden units each. For Split
MNIST datasets, CIFAR-10, 10, Split CIFAR- 100 and Split mini- Mimi-ImageNet datasets, we use a ResNet- 18 classifier. See Appendix for the details of the hyperparameter tuning and model achitectures.

---

Under review as a conference paper at ICLR 2021
Split MNIST
Permuted MNIST
Rotated MNIST
Split CIFAR-10
Split CIFAR-100
Split mini-ImageNet
Methods / Datasets
Fine tuning
18.80 0.6
66. 34
2.6
41.24 + 1.5
18.49 + 0.2
3.06 0.2
2.84 + 0.4
iid online
85.99
0.3
73.58
1.5
81.30 1
62.23 1.5
18.13
0.8
17.53
1.6
AGEM
Chaudhry et al. 2019a
29.02
+ 5.3
72.17
1.5
50.77 + 1.9
18.49 0.6
2.40
1.0+003 0.2
2.92
+ 0.3
GEM
Lopez Paz Kanzalo 2017
87.18
1.3
78.23
1.2
76.49 0.8
20.05 1.4
8.75
0.4
11.27
3,4
GSS Greedy Aljunni el d. 20196
84.16
2.6
77.43
1.4
73.66
1.1
28.02
13
19.53
13
16.19
U.7
BGD
Zend ml.
13.54
5.1
19.38
1. 3.0
77.94
0.9
18.23 0.5
3.11
1111103 0.2
24.71
HAL
[ Daaulry 0 sl. 2020
77.92 1: 4.2
77.55
:: 4.2
78.48
1.5
32,06 1.5
21.11 +
21.18
2.1
ER (RobLis 1995
80.96 80.96=2..3 2.3
79.69
1.0
76.95
+ 1.7
20.65 = 1.3
26.00 10:00=10 1.0
MIK Aijundi at
2019
84.88 1: 1.7
79.96 2: 1.3
78.30
1.0
34.47 1: 2.0
20.18 1 1.7
25.01 1.3
FR GMFD
82.68"* 2.1
79.70
77.89'
0.9
35.01"
+ 1.5
20.87 + 1.4
27.79' + 0.7
MIR + GMED
87.86" 1.1
80.11
1.2
79.16" 11 0.9
35.54
1.9
21.49" 1. 0.6
26.29' 1 1.2
iid offline (upper bound)
93.87
87,40
91.38
75.17
0.7
41.45
+ 0.9
36.54
Table I: Mean and standard deviation of final accuracy($$ (%) in 10 runs. For Split mini- and Split CIFAR- --FFR--10OA datasets, we set the memory size to 10,000 examples; we use 500 for other datasets. and indicate significant improvement over the counterparts without GMED with p-values less than 0.05 and 10-3 respectively in single-tailed paired t-tests.
85
80
ER
Accuracy 1%6
40
GMED ER
80
75
Accuracy
Accuracy
20
75
70
30
ER
ER
+
ER GMED-ER
GMED ER
65
GMED-ER
10
200 500
1000
100
1000
500
1000
1000
5000 10000 20000 Memory size
Memory size
Memory size
Memory size
(a) Split MNIST
(b) Rotated MNIST
(c) Split CIFAR-10
(d) Split mini- miniet
Figure 3: Performance of ER and GMED-ER in different memory sizes. For mini-ImageNet dataset, we use memory sizes of 1,000, 5,000, 10,000, and 20,000 examples; for other datasets, we use 100, 200, 500, and 1,000.
5.3 RESULTS AND PERFORMANCE ANALYSIS
We report the final accuracy achieved by different methods in Table I We summarize the following findings.
Overall Performance. From the results, we see MIR GMED achieves best performance among memory based continual learning algorithms. The improvement of GMED differs by datasets.
ER+GMED clearly improves over ER by an absolute margin of 1.72%. 1.67%. and 1.79% accuracy respectively in Split MNIST, Split CIFAR-10, and Split mini-ImageNee datasets. On Rotated MNIST and Split CIFAR-100 the improvement less significant, with an absolute accuracy improvement of 0.94% and 0.22%, while we do not see a meaningful improvement on Permuted MNIST dataset. MIR+GMED improves performance on all the datasets. The improvement is clear on Split MNIST, Split CIFAR- 10, Split CIFAR- 1000 and Split mini-ImageNet with an absolute accuracy improvement 2.98%, 1.07%, 1.31%.
Performance Under Various Memory Sizes. Figure 3 shows the performance under various mem- ory sizes. We see in Split MNIST, Rotated MNIST, Split CIFAR-10 and Split Mini-ImageNet. the improvement of ER+GMED over ER is consistent under various memory sizes.
Split MNIST
Permuted MNIST
Rotated MNIST
Split CIFAR-10
Split CIFAR-100
Split mini-ImageNet
Methods / Datasets
ER + GMED
82.68
2.1
79.70 1 1.1
77.89 1 0.9
35.01 1 1.5
20.87 1 1.4
27.79 1 0.7
Random
81.04
78.99 +
77.59
32.26 + 19
20.73
26.23
MIR GMED
87.86
80.11 10.11-1..2 1.2
79.16 + 0.9
35.54 + 1.9
21.49 + 0.6
26.29 + 1.2
MIR
Random Edit
84.76
1.4
79.76 1:76--00 0.6
78.19 = 1.0
35.39 3.0
19.77 +
24.86 = 0.7
Table 2: Performance when we updates memory examples to a random direction (Random Edit), compared to ER+GMED and MIR+GMED.
7

---

Under review as a conference paper at ICLR 2021
Method
Split MNIST
Split CIEAR-10
Split CIFAR-100
Acc.
Mem
8. Mem
Mem
ER
92.67
2,581
62,96
6,024
21.79
21,295
ER+GMED
94.16
2,581
63.28
6,024
22.12
21,295
CN-DPM
93.23
500
Gen.
45.21
1,000 Gen.
20.10
1,000 + Gen.
Table 3: Comparison with CN-DPM under the same memory overhead. The overhead is the size of the replay memory plus the extra model components (e.g. a generator (Gen.)), shown in the equivalent number of memory examples (#. Mem).
O 2
2
0 O
2
2
2
8
3
3
3
Figure 4: A I-SNE visualization of the editing performed on data examples. We use labels from the first two tasks in Split MNIST.
Figure 5: Visualization of the editing on exam- ples in Split MNIST. The first two rows show examples before and after editing, and the third row shows the differences.
Comparison with Random Editing. As an ablation study, we show the result of a random editing baseline in Table The baseline edits memory examples to a random direction with fix stride, tuned a similar way as GMED. We see GMED outperforms random editing all cases. We further notice ER+ Random Edit outperforms ER split CIFAR-10 10 and CIFAR-100 100. We conjecture the reason is that the random editing alleviates the overfitting to memory examples.
Comparison with Model Expansion Approach (CN-DPM), Non-memory based continual learning approaches introduce extra overhead in storing model parameters - for example, CN-DPM introduces extra overhead by employing generative model component and short-term memory (STM) in addition to the classifier. Following Hsu et al. (2018), we set the memory size for GMED so that two methods introduces the same amount of the overhead. Table 3 show the results of ER, ER+GMED and the reported results of CN-DPM. We see ER+GMED outperforms CN-DPM.
5.4 CASE STUDY AND DISCUSSION
Visualization Edited Examples. Figure visualize the editing on memory examples. We show examples from first two task (0/1, 2/3) in the Split MNIST dataset. The first and second rows show the original and edited examples, noted as Thefore and Fafter The third row shows the difference between two Az Jafter Thefore We see no significant visual differences between original and edited examples. However, by looking at the difference Az, we see there are examples whose contours get exaggerated, e.g., examples 1 and 12. and some get blurred, e.g., examples 2, 3, 5, and 6. Intuitively, to make an ambiguous example more forgettable, the editing should exaggerate its features; while to make typical example more forgettable, the editing should blur its features. Our visualizations align with the intuition above: examples 1 and 12 are not typically written digits, while examples like 2, 3, and 6 typical.
Visualization of Editing Directions. In Figure we show the t-SNE Maaten & Hinton (2008)
visualization of the editing vector Ax Fafter Thefore for examples from first 2 tasks in Split MNIST. We see the editing vectors cluster by the labels of the examples. It implies the editing performed is correlated with the labels and is clearly not random.
6 CONCLUSION
In this paper, we propose Gradient based Memory Editing for task-free continual learning. The approach estimates forgetting of stored examples online and edit them that they more likely to be forgotten upcoming updates. Experiments on benchmark datasets show our method can be combined with existing approaches to significantly improve over baselines several benchmark
8

---

Under review as a conference paper at ICLR 2021
datasets. Our analysis further show the method is robust under various memory sizes, and outperforms alternative editing methods.
REFERENCES
Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (claw). In International Conference Learning Representations, 2020. URL
net Borzum?ddHHlsso
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In CVPR, 2018.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NeurIPS, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS, 2019b.
Massimo Caccia, P. Rodríguez, O. Ostapenko, Fabrice Normandin, Min Lin, L. Caccia, Issam H. Laradji, Rish, Alexande Lacoste, D. Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. NeurlPS, 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. International Conference on Learning Representations, 2019a.
URL
net meet/forne forum fdt-eiiddikkkkCkcCe
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. 2019b.
Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip H. S. Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. ArXiv, abs/2002.08 165, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guidee continual learning with bayesian neural networks. In International Conference Learning Representations, 2020. URL ht httptt::://ppeeeeeeeee net httpaaarwweeeeeeeeeeooooemmeeeea htte:/ooormmmmmd1000000
I. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep learning. Nature, 521:436-444, 2015.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation catastrophic forgetting gradient-based neural networks. arXiv preprint arXiv: 1312,221,, 2013.
Harrison, Apoorva Sharma, Chelsea Finn, and M. Pavone. Continuous meta-learning without tasks. NeurIPS, 2020.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. ArXiv, abs/ 0ttss/990.0520... 2019.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: categorization and case for strong baselines. arXiv preprint arXiv: 00.//////488. /2488, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinska et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits, 1998. URL http::/yann. lecun. com/exdb/nmist, 10:34, 1998.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task- fase-free continual learning. In International Conference on Learning Representations, 2020. URL httph::t///// /estpss:/oppnreest eew net. forum?
n dd=SJx50JStP
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. ArXiv, abs/1904.003/ 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on patlern analysis and machine intelligence, 40(12):2935-2947 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In NIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov))2579-2665, 2008.
James I.. McClelland, Bruce McNaughton, and Randall C. O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102 3:419-45 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference Learning Representations, 2018. URL httl/ttt////opeeeeiieee
Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1320-1328, 2017.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, pp. 7645-7655, 2019.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2))285, 1990.
Sylvestre-Alvee -Aviie Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5533-5542, 2017.
Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 7:123-146. 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. In NeurlPS, 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, abs//600.00677 2016.
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In ICML, 2018
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, 2017.
10

---

Under review as a conference paper at ICLR 2021
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL URLAattps://peeeeeviee
nnt/ i d=3J1 xm30cKn
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. ArXiv, abs/1904.07734, 2019.
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukeuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 1177:737-57, 1985.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. 2018.
A DETAILS OF COMPARED METHODS
We included detailed descriptions, and implementation details of some selected baselines in this section.
Experience Replay (ER) Robins 1995]; Rolnick et al. (2019) stores examples in a fix-sized memory for future replay. We use reservoir sampling to decide which examples to store and replace. Following prior works Aljunndi et al. (2018) (211a)); Chaudhry et al. (2020), at each time step we draw the same number of examples as the batch size from the memory to replay, which are both set to 10. The algorithm applies to the task-free scenario.
Gradient Episodic Memory (GEM) Lopez-Paz & Ranzato 2017) also stores examples in memory. Before each model parameter update, GEM project gradients of model parameters so that the update does incur loss increase on any previous task. The approach
task-free.
Averaged Gradient Episodic Memory (AGEM) Chaudhry et al. (019a) prevents the average loss increase on randomly drawn subsets of examples from the memory. We draw 256 examples to compute the regularization at each iteration. The approach is task-free.
Bayesian Gradient Descent (BGD) Zeno al. (0188) is eggularization-basee continual learning algorithm. It adjust learning rate for parameters by estimating their certainty, which notes for their importance to previous data. The approach task-free.
Gradient based Sample Selection (GSS) Aljundi et al. (2019b) builds upon ER by encouraging the diversity of stored examples. We use GSS-Greedy, which the best performing variant in the paper. The approach is task-free.
Hindsight Anchor Learning (HAL) Chaudhry al. (200)) learns an pseudo "anchor" example task per class in addition to the replay memory by maximizing its estimated forgetting, and tries fix model outputs on the anchors training. However, unlike GMED, they estimate forgetting with loss increase on examples when the model train for a pass on the replay memory (and thus forgetting estimated with "hindsight"). The approach is not task-free.
Maximally Interfering Retrieval (MIR) Aljundi et al. (2019a) improves ER by selecting top forgettable examples from the memory for replay. Following the official implementation, we evaluate forgetting candidate set of 25 examples for mini-Imagen Imai-em dataset, and 50 examples for others. While the approach is task-free the official implementation filter out memory examples that belong to the same task the current data stream, which assumes knowledge about tasks boundaries. We remove this operation to adapt the method to the fiek-free setup. Therefore, our results are not directly comparable the official results.
11

---

Under review as a conference paper at ICLR 2021
Neural Dirichlet Process Model for Continual Learning (CN-DPM) Lee et al. (2020) is task-free expel-exn based continual learning algorithm. We report the official results in the paper. In the comparison study between ER/ER+GMED with CN-DPM, for the base model in ER/ER+GMED, we use the full expanded model in CN-DPM (iee, the model architecture when the training ends in CN-DPM). We use the same optimizer and the learning rate CN-DPM in this set of experiments.
B HYPERPARAMETER SETUP
We use a batch size of 10 throughout the exper- iment. We use SGD optimizer with learning rate of 0.05 for MNIST datasets, 0.1 for Split CIFAR-10 and Split mini-ImageNet datasets, and 0.03 for the Split CIFAR-100 100 dataset. We perform three steps of model parameter up- dates for each example we visit in the Split mini-ImageNeet dataset, following Aljundi al. (2019)), and one step for others.
Editing stride a
Regularization strength 8
Dataset Hyper-param
Split MNIST
0.01
Permuted MNIST
0.001
Rotated MNIST
0.01
Split CIFAR-I0
0.05
0.001
Split CIFAR-I00
0.05
0.001
Split mini- Imni-lmaeeNtt
1.0
0.1
Table 4: Hyperparamters of the editing stride and
GMED introduces two hyperparameters: the stride of the editing a, and the regularization strength 3. As we assume no access to the full data stream in the online learning setup, cannot select hyperparameters according validation performance after training on the full stream. Therefore, We tune the hyperparameters with only the training and validation set of first three tasks, following Chaudhry al. (2019a). The models are trained until convergence before they proceed to the next task. The tasks used for hyperpa- rameter search are included for reporting final accuracy, following [Ebrhhii et al. (2020). We perform grid search over all combinations of a and /8 and select the one with the best validation performance on the first three tasks. We select a from [0.01, 0.05, 0.1, 0.5. 1.0, 5.0, 10.0), and se- lect B3 from [0, 10-3, 10-2 10-1, 1). We tune hyperparameters for ER-GMED and apply the same hyperparameters on MIR+GMED. Table show the optimal hyperparameters selected for each dataset.
the regularization strength selected for GMED,
C IMPLEMENTATION DETAILS OF THE HYBRID METHOD OF GMED AND MIR
As mentioned in Sec. 4.3 GMED can be built upon MIR to further improve the performance. At each time step, MIR retrieves the most forgettable examples from the memory with the forgetting defined as Eq. T We do not edit the selected examples directly; instead, additionally draw another random mini-batch from the memory to apply editing. The motivation is that examples drawn by MIR are already most forgettable ones; if we directly perform editing them, would fall into loop that letting forgettable examples more forgettable, which is not desired.
12


Reviews:
[{"review_id": "WEys4feIdki", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe authors propose a gradient-based memory editing scheme for replaying samples that are undergoing the most forgetting, coined GMED. They also propose a hybrid method with MIR [1]. An extensive evaluation on standard benchmarks shows some improvements throughout. \n\n**Concerns**\n\nMy main concern is that the authors seem to have completely missed that gradient-based memory editing has been proposed in [1]. Specifically, in their GEN-MIR and AE-ER methods, gradient-based optimization is performed on some latent codes of the data to maximize the \"forgetting\" measurement. \n\nGEN-MIR seems superior to GMED as it can perform multiple gradient-based edits on the data to achieve a significantly different replay sample. GMED can't because it re-uses the initial label as the label of the editable sample. More GMED edits will thus increase the probability that the initial label is wrong. I think this is why the authors only do one edit. You can see, however, that one edit doesn't seem to change the replay data, as exemplified in Figure 5. [1] also proposes AE-MIR when $p(x)$ is difficult to model online. Furthermore, I would argue that gradient-based editing is more sensible in latent space (GEN-MIR and AE-MIR) than in input space (GMED) because it will be much easier to stay on the data manifold. It is also much more computationally efficient. \n\nAll of section 4.1 and most of section 4.2, including \"estimating forgetting online\" can be found in [1]. Algorithm 1 in section 4.3 is also astonishingly similar to the ones in MIR. \n\nI think this work is too incremental to merit a publication at a top conference like ICLR. Thus, I encourage the authors to either submit to a lower-tier conference or further develop the methodology.\n\n**minor concerns**\n\n- The related work seems to be missing a branch of task-free CL, namely Continual-Meta Learning. \n  \n- no need to re-explain Reservoir Sampling\n\n- typos: a lot of hyphens are missing. e.g. *gradient-based* memory editing\n\n\n__________\n\n**POST-REBUTTAL**\n\nSadly, I'm decreasing my score a notch because the authors lack a deep understanding of MIR [1] that would make it evident that GMED is *much* closer to the three methods proposed in MIR. Specifically, the authors have responded to my concern about the lack of novelty:\n\n    In GEN-MIR, the classifier and the generator separately retrieve most forgettable examples for themselves. The generator is indeed optimized for “maximizing the forgetting”, but the “forgetting” here is measured for the generator - i.e., the generator retrieves most forgettable examples for the generator itself. The approach does not learn a generator that can “generate examples that are more forgettable for the classifier”; instead, feeding more forgettable examples in GEN-MIR aims at reducing the forgetting of the generator.\n\nThis is simply not true. If you take a look at Equation 2 in MIR, you will see that the generator is generating forgettable examples **for the classifier**. It also uses it for itself, see Equation 3. GEN-MIR is thus a gradient-based memory editing for CL.\n    \n    AE-MIR (...) It is a hybrid approach of example compression and ER-MIR. There is no online optimization towards more forgettable examples for the classifier like GMED\n\nAgain, just like GEN-MIR, AE-MIR uses gradient-based memory editing for CL for the classifier.\n\nIn GMED lies somewhere between ER-MIR and [GEN-MIR, AE-MIR]. My guess is that the MIR's author didn't propose the GMED method because it doesn't make a lot of sense to update edit a sample **and not edit its label**. \n\nHere is an intuition on the behavior of each method: let's say your models sequentially learning to visually classify objects. The model is now learning about zebras and it's causing some interference on the horse's representation.\n\n- ER-MIR will search in its buffer and retrieve a horse for the classifier to do replay on.\n- GEN-MIR will search inside the latent space of a generative model to find generated horses for the classifier to do replay on.\n- AE-MIR will search the latent space of an autoencoder to retrieve past horses that appeared in the data stream for the classifier to do replay on.\n- GMED randomly samples some data in the buffer, e.g. a car, and takes one gradient update on the car such that it resembles more a horse. Then the classifier is fed that modified image (i.e. x) as well as the unchanged horse label (i.e. y)\n\nThe empirical section shows us that one needs to add MIR to GMED to obtain the best results. This comes as no surprise. Combining methods with each other and increasing computing needs and/or replay will give you a better performance on forgetting.\n\n\n\n\n_________\n\n\n[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, andTinne Tuytelaars.  Online continual learning with maximally interfered retrieval.  In NeurIPS 2019.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RM3qGt9pr8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis paper proposes a task-free continual learning method called GMED that extends experience replay.\nThe key idea is to modify the individual data points in the replay memory to maximize the one-step forgetting at the current time step.\n\n---\n\n## Pros\n\n- The idea of editing the data in a replay buffer with gradient descent is novel.\n- The experiments show a significant performance gain compared to the baselines.\n\n---\n\n## Cons\n\n### Lack of justification for editing memory\n\nGiven a somewhat arbitrary objective function, updating the data in memory by gradient descent will make the data points move away from their original distribution. Since continual learning aims to learn the data's distribution, I think modifying each data point does not align with this goal.\n\nOne possible explanation for the performance improvement is that the editing acts as a kind of data augmentation. As shown in Figure 5, the difference between the edited data and the original data is minuscule. Since the performance would drop if the editing is significant, the magnitude of updates should be small such that the data do not deviate too far from the original distribution.\n\nFrom this point of view, I think it is critical to show that GMED is better than other data augmentation tricks.\nThe authors already performed an ablation study showing that GMED is better than adding random noise to the data.\nHowever, there is no much detail about the experiment to verify its validity. Since it is an important experiment, I strongly recommend the authors to provide enough details for reproducing.\nAlso, I would like to see a comparison with other standard data augmentation methods.\n\n### Confusing motivation and writing\n\nThe fundamental hypothesis of this paper is:\n> replaying examples that are likely to be forgotten by the current model helps retain its test performance.\n\nHowever, the final algorithm is far from this hypothesis. As I mentioned previously, I think it is closer to a data augmentation algorithm.\n\n### Unreasonable experimental setting\n\nFor the Split CIFAR-100 and Split mini-ImageNet experiments, the authors use a memory size of 10,000. However, the size of the whole training set is 50,000, which means the memory can store up to 20% of the whole data. In my opinion, the current trend is to keep the memory size under 1-2% of the whole dataset, which corresponds to 500-1000 in the case of CIFAR100. Note that the authors also follow this trend for MNIST and CIFAR-10. With 20% of the dataset, I suspect that even i.i.d. training on the memory would not differ much from i.i.d. training on the whole dataset.\n\nAlso, I think it is more constructive to compare models under a fixed number of examples in memory, not the number of bytes as in Table 3. While trying to equalize the memory usage of ER(+GMED) and CN-DPM, the number of examples in memory became unreasonably large for ER(+GMED) (over 40% of the whole data).\n\nThe brutal fact is that the current level of continual learning is far from practical. At present, most CL papers work on toy problems like CIFAR-100. At this level, I think comparing the actual memory usage is not meaningful. In my opinion, what the community should look for is an efficient algorithm that can scale to real-world problems. For this reason, I think it is more meaningful to constrain the number of examples in memory. Also, note that if we scale up the image size, the size of image data can be more dominant than CNN's parameter size.\n\n---\n\n## Overall evaluation\n\nThe proposed method does not have any theoretical justification, and several settings in the experiments are unrealistic. Nonetheless, since the experiments show significant improvements over baselines, I think it can be an effective technique. Unfortunately, at the current state of the paper, I cannot exclude the possibility that simpler data augmentation tricks on the replay memory may outperform the proposed method. Although I currently vote for rejection, I am willing to raise the score if the authors can offer enough evidence that GMED is better than trivial data augmentation tricks.\n\n---\n\n## Post-rebuttal\n\nDuring the rebuttal period, the experiments largely improved, resolving the majority of my concerns. I raise my rating accordingly. I expect the authors to reflect other suggestions as well in the final version. Especially, R1 raised a serious concern about the similarity between GMED and GEN/AE-MIR. A proper comparison should be included in the final version.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JAiWUXy2XRE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Differentiation with respect to forgetting looks a promising idea that can complement differentiation with respect to current task accuracy and leads the CL algorithm towards a more stable solution by a good tradeoff between stability and plasticity.\n\nOne major concern about this paper is the empirical validations. Some baselines like AGEM are underperforming compared to reported numbers on other papers. The idea of memory editing can also be added as an additional loss to other methods too (not only to ER and MIR). Is that true?\n\nRandom edits look quite promising. It may suggest that most of improvement is coming from replay (it can be anything) and the regularization effect that it provides to not to overfit to the data.\n\nIsn't figure 4 telling something trivial? Because each class has a unique pattern of active inputs and that leads similar activation patterns on data and subsequently to a similar gradient profile within the class.  \n\nIn equation 6 when we replace the forgetting measure (d_t) with equation 5 loss function \\ell is going to have a different coefficient (-\\alpha-\\beta) which basically is a special way of weighing the two loss terms on \\theta and \\theta'. It's good to elaborate on this more.\n\nThe connection of eq. 1 and eq. 2 to te final editing formula (stated in eq. 3) is a little loose. Especially starting from eq. 2 approximation using eq. 3 is not the most natural or organic step. \n\nThe paper is well written and easy to follow. \npage 2 --->... one online model...\npage 3 ---> an categorization ...\npage 4 ---> also show that ...\nAlso note that citations are not properly using parentheses. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hg7njTZ8L7o", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with continual learning. Specifically, given a stream of tasks we want to maximise performance across all tasks. Typically neural networks suffer from catastrophic forgetting which results in worse performance on tasks seen earlier in training. There are many proposed solutions to this problem. One specific set of approaches are \"memory based\" algorithms. Here we store some training examples in memory from the tasks seen thus far. These are then mixed in with new training data so as to encourage the model to not forget past tasks. \n\nThere are two central question for approaches of these kinds: what examples to store and how to use them. This paper details a method that is complementary to most approaches. Specifically, given a system of storing examples (here reservoir sampling) and using them, this methods deals with how best to extract performance from an example. Specifically, it \"edits\" the example before storing and replaying it, to make it more useful to the model. \n\nThe main motivation for the editing process is as follows: past works have shown that the most useful examples to replay are those that are most likely to be forgotten. Thus, if we take our examples in memory and update them (via gradient ascent) to be \"more forgettable\", then when replayed these will result in the most benefit to the model. \n\n* One comment I have here is that the way this motivation is presented in the paper can be improved; perhaps with a more clearly written paragraph in the preliminaries. On my first reading of the introduction for example, this sentence for example, \"_edits examples stored in the memory with gradient-based updates so that they are more likely to be forgotten_\" sounds like you're trying to get the model to forget examples more which is the opposite of what we want (as opposed to making the example inherently more forgettable so it is better when replayed). \n\n* The reasoning here is that because naturally forgettable examples help during replay, _making_ an example \"forgettable\" will naturally improve its usefulness. I can't fault this reasoning based on the results but to me it is not naturally clear that this follows. Would the authors be able to provide some intuition for what that means from an optimisation perspective? \n\nFurther, the authors propose an online metric for how much the network forgets an example (by comparing the loss at consecutive timesteps). Then when taking an example from memory, we perform a gradient update on the image to optimise this metric. \n\nLastly, we replay the edited example to the network and perform a gradient update based on both this edited example and the current training example. Figure 2 is very useful and well done to illustrate this. \n\nThe experiments are performed across standard tasks and show modest to good improvement over the range of datasets. \n\n * Two baselines that would be useful to include in the table are EWC (https://arxiv.org/pdf/1612.00796.pdf) and MbPA ((https://arxiv.org/pdf/1802.10542.pdf), the latter of which is a memory based method that also does not require tasks IDs. \n\nLastly there is some good visualization and discussion of what editing does. \n\nOverall this is a good paper with that deals with an interesting problem and proposes an interesting method of increasing performance.\n\n*  I think some ways to strengthen this paper would include addressing some of the comments above, specifically around motivating the \"editing\" process better and clarifying some of the language around it. \n* Finally, the field of continual learning in general could do with moving to more large scale tasks. This would allow testing these methods more thoroughly. While the results here are promising, they do not increase performance uniformly across all tasks -- for example on mini Imagenet which is harder than MNIST -- this method does not help. In general considering larger scale tasks (based on Imagenet, Omniglot or reinforcement learning like Atari, for example) would provide more evidence and allow further improvement of the method. \n\nIn the current form however, this is still a very good submission and I would recommend acceptance. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The range of the initial reviews was fairly high with overall scores ranging from 4 to 7.

The authors provided a good response that answered most of the reviewers' comments and questions. One of the reviewers even increased their score following the authors' response. 

The focus of some of our discussions and what ultimately led to my suggestion was the related work of MIR [1]. The methodological differences between (the three versions of) MIR in [1] and GMED [this paper] appear less significant than what the current submission suggests. While there is some disagreement between the authors and Reviewer1 about the exact differences, I find that the current manuscript does not acknowledge the close relationship between these two contributions. Further, from the experimental standpoint and without further justifications the gains from GMED+MIR could be attributed to using more replay (from combining GMED and MIR).


In their response, the authors disputed the view of Reviewer1. I believe the source of the confusion between the author and the reviewer might be captured in this sentence from the author response to Reviewer1: The approach does not learn a generator that can “generate examples that are more forgettable for the classifier”; instead, feeding more forgettable examples in GEN-MIR aims at reducing the forgetting of the generator.

Looking at Equations 2, 3 and Algorithm 1 from [1], in GEN-MIR while two different procedures are used to obtain forgettable examples for the generator (B_G in Alg. 1) and the classifier (B_C in Alg. 1), the generator is used in both cases. In other words, the generator is used to generate examples for both itself and for the classifier. So, I think it's fair to say that the generator does indeed generate examples that are more forgettable for the classifier (Eq. 2). 


I strongly encourage the authors to prepare another version of their work where the differences between MIR [1] and their contribution are clearly highlighted and the results show the advantages of GMED (including memory-editing in data space). 


[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NeurIPS 2019. https://arxiv.org/abs/1908.04742
Decision: Reject