Improving Meta-Continual Learning Representations with Representation Replay
Authors: Lawrence Ki-On Chan
James Kwok
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
continual learning
meta-continual learning
replay

Abstract: Continual learning often suffers from catastrophic forgetting. Recently, meta-continual learning algorithms use meta-learning to learn how to continually learn. A recent state-of-the-art is online aware meta-learning (OML). This can be further improved by incorporating experience replay (ER) into its meta-testing. However, the use of ER only in meta-testing but not in meta-training suggests that the model may not be optimally meta-trained. In this paper, we remove this inconsistency in the use of ER and improve continual learning representations by integrating ER also into meta-training. We propose to store the samples' representations, instead of the samples themselves, into the replay buffer. This ensures the batch nature of ER  does not conflict with the online-aware nature of OML. Moreover, we introduce a meta-learned sample selection scheme to replace the widely used reservoir sampling to populate the replay buffer. This allows the most significant samples to be stored, rather than relying on randomness. Class-balanced modifiers are further added to the sample selection scheme to ensure each class has sufficient samples stored in the replay buffer. Experimental results on a number of real-world meta-continual learning benchmark data sets demonstrate that the proposed method outperforms the state-of-the-art. Moreover, the learned representations have better clustering structures and are more discriminative.

Under review as a conference paper at ICLR 2022
IMPROVING META-CONTINUAL LEARNING REPRE- SENTATIONS WITH REPRESENTATION REPLAY
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning often suffers from catastrophic forgetting. Recently, meta- continual learning algorithms use meta-learning to learn how to continually learn. A recent state-of-the-art is online aware meta-learning (OML) Javed & White (2019). This can be further improved by incorporating experience replay (ER) into its meta-testing. However, the use of ER only in meta-testing but not in meta- training suggests that the model may not be optimally meta- trained. In this paper, we remove this inconsistency representations by integrating ER also into meta-training. We propose to store the samples' representations, instead of the samples themselves, into the replay buffer. This ensures the batch nature of ER does not conflict with the online-aware nature OML. Moreover, we introduce a meta-learned Predictive Sample Selection to replace the widely used reservoir sampling to populate the replay buffer. This allows the most significant samples to be stored, rather than relying on randomness. Experimental results on number of real-world meta-continual learning benchmark data sets demonstrate that the proposed method outperforms the state-of-ther Moreover, the learned representations have better clustering structures and are more discriminative.
the use of ER and improve continual learning
INTRODUCTION
Humans can acquire knowledge incrementally throughout their lives. continual learning (De Lange et al. 2019: Parisi et al. 2019), the machine learning model attempts to imitate this behavior by learning number of tasks sequentially. This allows for lifelong learning, which acquires knowledge tirelessly and improves continuously over time. It also has diverse real-world applications, from recommendation systems (Yuan et al. (020), clinical applications (Lee & Lee 2020) to natural language processing (Biesialska al. (020). However, while machine learning models can sometimes surpass humans on specialized tasks, they often suffer from catastrophic forgettnn (Goodfellow al. 2015 Kirkpatrick al. 2017). When new tasks learned, the old knowledge learned from previous tasks is overwritten, leading to poor performance on these old tasks.
To alleviate this problem, meta-learning (Hospedales et al.) 2020) has been recently incorporated into continual learning, leading to meta-continua learning (Caccia et al. 2020). By optimizing models for learning to learn as meta-learning, meta-continual learned models are meta-trained to avoid catastrophic forgetting. (Riemer et al. 2019) show that the meta-lcarning objective aligns with the continual learning goals of minimizing interference and maximizing transfer.
A recent state-of-the-ar in meta-continudl learning is online aware meta- meta-learning (OML) (Javed & White 2019), which optimizes the model for continual learning through online sequential updates in meta-training- Javed & White (2019) show that the performance of OML can be further improved by incorporating experience replay (ER) (Chaudhry et al. 2019), which is an effective rehearsal strategy for continual learning by replaying tiny episodic memories during the training process, into its meta-testing. However, the use ER only in meta-testing but not in meta-training suggests that the OML model may not be optimally meta-trained. Moreover, the effectiveness of ER heavily relies on the samples stored for replay, the use of reservoir sampling (Vitter 1985) to populate the replay buffer heavily relies randomness.
In this paper, we improve the continual learning representations and remove inconsistency in the use of ER between meta-training and meta-testing by integrating ER also into meta-training. However, the

---

Under review as a conference paper at ICLR 2022
batch nature of ER training interferes with the online nature of OML during meta-training resulting in a loss of gradient and Hessian information during the meta-updates. To alleviate this problem, propose to store the samples' representations, instead of the samples themselves, into the replay buffer. This allows both OML's online training and ER's ' batch replay to be preserved. Besides, reservoir sampling to populate the replay buffer, we use the meta-learning
instead of relying procedure to select samples that can best help overcome catastrophic forgetting. Experimental results on number of real-world benchmark data sets demonstrate that the proposed method outperforms the state-of-the-an
2
RELATED WORK
2.1 CONTINUAL LEARNING
Continual learning aims to learn a series of tasks incrementally (Mai et al. 2021). ). As data from only the current task is available during training, it is important to overcome catastrophic forgetting. There are three continual learning settings of increasing difficulty (van de Ven & Tolias (201)): lusk-incremental, domain-ineremental. and class-incremental. In this paper, we focus on the class- incremental setting, which is the most difficult. At test time, the model has to both decide the task identity and solve all the tasks encountered so far. We also use the online (or streaming) setting (Aljundi et al. 2019b; Hayes et al. 2019; Borsos et al. 2020 in which each sample only appears once during training.
A common approach to avoid catastrophic forgetting is by regularization (Zenke et al. 2017 Huszar 2018, [i & Hoiem| 2018). However, |esorrt et al. (2019) show both theoretically and empirically that regularization insufficient. Knoblauch et al. (2020) also demonstrate that some form of memory or replay buffer, as used in rehearsal methods Lopez-Pat & Ranzato 2017: Alljuni et al. 2019a van de Ven al. 2020), is necessary.
2.2 META-LEARNING
Meta-learning, also known as "learning to learn", aims to enable models to learn more effectively (Hospedales et al. 2020). A well-known example algorithm is model-agnostic meta-learning (MAML) (Finn et al.) 2017)) which meta-learns model initialization mneta-parammttr 0) that can be quickly adapted to a new task by training on only a few samples. During the training phase of meta-training (meta-tratte imeta-training training), Wo. a copy of 9 learns from batch of tasks in the meta meta-training training set and is updated to We after / iterations. During the testing phase of meta-training (mela-training testing), 0 is updated by back-propagating the losses of each of Wi...., W. on the meta-training test set. In meta-testing, the meta-testing training phase adapts the meta-trained model using the same procedure meta- training training on each of the unseen tasks. Finally, each adapted model is evaluated on the corresponding task in the meta-testing test set during meta-testing testing.
2.3 COTA--CONINUAL LEARNING
Meta-continual learning meta-leame a model for effective continual learning (i.e., "learning to learn not to forget" (Caccia al., 2020)). The procedure follows closely that of meta-learning, except that continual learning is performed instead of batch learning. In other words, in each meta-training training iteration, sequence (instead of batch) of tasks is sampled, and Wo is updated on each of the tasks sequentially (thus simulating continual learning).
The state-of the is Online aware Meta-Learning (OML) (Javed & White 2019). Its model fa.w has two components: (i) a representation-lcaneee learns a good representation for continual learning, and (ii) a prediction learning network (PLN) fw with parameter W, that performs prediction using RLN's representation. Algorithm shows its meta-training procedure (Figure ((a).. A task sequence T; is sampled at outer iteration (step 3).
network (RLN) fo with parameter 6, which meta-
A training set Strain (of size k) and a test set Stest is constructed from these tasks (step 4). The inner loop (meta-training training) online updates the PLN using the loss l on each sample of Strain (step 8), in the online class-incrementee continual learning setting (van de Ven & Tolias 2019).
Note that online update, instead of batch update, is key to OML as it simulates continual learning and allows the effect of catastrophic forgetting be taken into account. The RLN is not explicitly updated
2

---

Under review as a conference paper at ICLR 2022
(a) OML.
(b) Naive approach.
(c) Representation replay.
Figure I: OML and ways integrate with experience replay into its meta-training.
during meta-training training. However, it is implicitly updated from 0, = 0 to 02,... 0k++1 so that the meta-loss gradient (to be computed in meta-training testing) can sack-propagate through all intermediate models and allow the RLN to improve its representation based on each Strain sample. Step 10 (meta-training testing) updates the RLN by minimizing the meta-loss [(0x+1, Ws+1) = =[(foe+.We+t (Steatf:, 0)), Stest [:. 1)) with gradient descent (using learning rate :)::
- 0 3V (0 WK41).
(I)
On meta-testing, a sequence of tasks that have not been seen during meta-trrining is used. The meta-testing training phase uses the training set in T, with the same procedure as in meta-training training (online updating the PLN by each sample in T once). The model is then evaluated in meta-testing testing using the test set of T.
Javed & White (2019) demonstrate that OML can also be used in conjunction with experience replay (ER) (Chaudhry et al. 2019) during meta-testing. A replay buffer stores previous tasks' inputs to approximate the previous tasks' distributions. random batch from the buffer is interleaved with a new sample each meta-testing training iteration.
Another state-of-the-art is ANML (Beaulieu et al. (020), which is similar to OML but meta-learns a network that selective gates the PLN outputs. ANML is shown to outperform OML in (Beaulieu et al.) 2020). However, Javed & White (2019) later showed that the two have comparable performance after fixing bug in the meea-egrddent computation of OML
3 PROPOSED METHOD
3.1 USING EXPERIENCE REPLAY IN
In OML, inner updates to the PLN are performed an online manner so as to simulate continual learning. However, these updates perform simple SGD and do not take explicit measures to combat catastrophic forgetting. Moreover, while Davee & White (2019) demonstrate that using experience replay (ER) in meta-testing alone improves OML's meta-testing testing accuracies, the lack of ER in meta-training creates inconsistency. More extensive experiments in Section 4. shows that it is indeed worse than standard OML on more challenging data sets. To alleviate these problems, we propose integrating ER also into the meta-tran veta-training phase of OML.
3.1.1 NAIVE APPROACH
In this section, we first present straightforward integration. At inner iteration j of Algorithm I the new meta-training sample X, is simply combined with b - 1 random samples from the replay buffer (Figure T(b).. The resultant batch B, goes through both the RLN and PLN as in standard OML. The PLN is updated by SGD as: W,+1 = W, a w; {[Se,, ww; (Byle [:. 0)), [y 1)), where Se Se,w, (B) 0)) the prediction on this batch, and [; 1] is the target output.
By expanding the meta-loss gradient v ( W in I to leading order as in (Nichol et al. 2018). we obtain the following Proposition. All the proofs are in the appendix.
Here, the inputs of all samples in S is denoted St:. 0), and the corresponding labels denoted S|; 1). https//gituhb.om//aarraaggg www..eraaaaaaaadd/////ssssssss m0j0-ma0000--0---------- bug-rrbuuuuuu hhu--ix--nd-refiann ann-aenffctorrnn--ogg
3

---

Under review as a conference paper at ICLR 2022
Proposition 1. VoL(Oxx11 Wx11) = =6 ag (B, aH aH I 0f + O(a2), where O = (ol Wi) and H = V2I.(, are the gradient and Hessian of the meta-loss respectively. (B,
(B, 0)), By[:, 1)), and H' = Vo,(fo.w, (Bylff By 11).
As (B;) and (B;) are computed from all samples Bj. the loss's gradient and Hessian contributions from Xj may be canceled out by some replay samples in Bj. This loss of information can hinder the RLN from learning representations from all meta-training samples. The benefits of obtaining gradients for each sample to train the RLN, which is crucial in OML, are thus lost.
3.1.2 REPRESENTATION REPLAY
To alleviate the problem in Section 3.1.1 the RLN needs to be online updated by only the current sample X, (as in OML). Instead of storing the raw samples directly, we propose representation replay, which stores the latent representations of the input samples into the replay buffer (Figure [(c). Algorithm 2 shows the meta-training training procedure with representation replay (which replaces steps 6-9 of OML's Algorithm []. At inner iteration j. X, still passes through the RLN to obtain its representation fo(X)). ). A batch of representations Rg is also sampled from the replay buffer. As R2 are not raw samples, they only need to go through the PLN fw.. Using the combined batch of representations (ffo.w, (X,)) U R2), the PLN is updated by SGD as W,+1 W VVw, (l(fo,w, (X),) Y3) + E(fw, (Ryl. O),, (gft,1]))
After k inner updates, the RLN is updated. As R2 does not depend on 0 only the gradient and Hessian for X, need to be computed when meta-updating the RLN, The meta-loss gradient can be obtained by the following Proposition.
Proposition 2. VoL(O.+1, We+1) = [f- #xXx) - att ,
0] + O
where g' d'(x;) (x), Y,). (1,));) = Vil(fo.w,(X)) (X)), and I and Ht are as in
Proposition 7
Compared with Proposition 1 (B,) and BB,) are now replaced by d'(X,) and (X())), re- spectively. Thus, information X, will no longer be interfered by replay samples in the same batch. This allows the RLN to learn representations for each individual sample while minimizing interference to the previously learned tasks. Note that the proposed method can also be used with other meta-continual representation learning methods, such as ANML (Beaulieu et al. 2020 and La-MAML (uupta et al. 2020).
A related technique is latent replay (Pellegrini et al. 2019). It trics to reduce the time and space required for rehearsal by storing the activations some intermediate layer (instead of the samples themselves), so that the forward and backward passes only need to go through the model's top layers but not the lower levels. Though latent replay also stores the representations, differs fundamentally from the proposed representation replay in the following aspects: (i) Latent replay aims to reduce time and space in continual learning, while representation replay aims improving the meta-continual learning representations; (ii) In latent replay, one has to tune and decide which particular intermediate layer for storing the representations, while representation replay always stores the outputs of the RLN; (iii) Latent replay performs slow training on the lower layers. With long training process, the stored feature representations may become unsuitable for current and future training. For representation replay, the RLN is not updated in both meta- training training and meta-testing training, and thus does not suffer from the aforementioned problem.
3.2 SELECTING SAMPLES INTO THE REPLAY BUFFER
To select samples into the replay buffer, reservoir sampling /itter 1985 has been commonly used (Robins 1995 Lopez-Paz & Ranzato 2017 Chaudhry et al. 2019). However, reservoir sampling uses random selection which does not reflect how each sample may benefit continual learning. Another simple approach is to implement the replay buffer of each task (Chaudhry et al.) (02I). where M is the buffer size and T is the total number tasks. However, these last few samples may not be the most representative.
ring buffer and store the last M/T samples
4

---

Under review as a conference paper at ICLR 2022
Recently, AAljundi al. (2019b) propose greedy sample selection scheme, which encourages diversity in the replay buffer by choosing samples whose gradients are least similar to all gradients of random batch from the buffer. However, this may be problematic when the task has outlying samples. More recently, [Borsos et al. (2020) formulate sample selection as a bilevel optimization using greedy forward selection, it is still very computationally expensive.
problem. However,
In this section, we propose a novel scheme, called Predictive Sample Selection (PSS), for selecting samples into the replay buffer based how they affect the loss. When the buffer is full, the buffered sample that leads to the smallest loss increase is replaced. For computational efficiency, the effect on loss is estimated by simple FC layer. Note that sample selection only takes place during meta-tres veta-training training and meta-testing training.
The procedure is shown in Algorithm 3 First, the current batch of representations R (containing the replay buffer samples and new sample (fo(X,), Yj)) goes through a forward pass of the FC layer. For each sample (X,),, Y,)= E R. the FC outputs a score FC(fo(X,)) which a prediction of the kR. Here, ln and (rx,((f((x.)xx))) are the model's (PLN's) losses on class Y, when trained on R with and without (fa(X, Yx) respectively. The loss is measured on all meta-training samples belonging class Y,, as (Xy,, Y.) mainly used to prevent forgetting on Y,. It thus takes into account of how this sample prevents catastrophic forgetting of its class if it resides in the buffer. Step 3 then selects sample (fo(X), Y) with the lowest score. If this comes from the replay buffer, will be replaced by the new sample (fo(X)), X2) when the replay buffer full.
difference & (
The FC layer is trained during meta-training training, by minimizing the square loss between FC((f(X)) and the actual loss increase (x),)( LR. To evaluate (xx.))(X.))... (step 5), we need to first update the PLN on ((){([((Xr)) Y.)). This can be done very efficiently by perform- ing only one SGD step on W, (step 4). During meta-testing training, the FC layer not updated, and only steps 1 and 2 in Algorithm need to be performed.
Algorithm Meta- in OMI. (Javed & White 2019).
Algorithm 2 Representation replay.
Require: M: the replay buffer
Require: p(T): distribution over task se- quences;
function REP REPLAY(( Wi, M)
2:
for j 1,2, k do
1: randomly initialize 0
3:
(X,, Y2) = Strain
2: while not done do
4:
sample batch R from M
3: sample task sequence T; p(T)
5:
R +- R U (Xfo(X))
4: obtain Strain- Stnst from Tj
I1 update PLN
5: randomly initialize W1 // meta-training training // (updating PLN on each sample j)
6:
Wi+1 =W; - aV {(Jw, (R(: 0)), Ri: 1J)
7:
(fo(X), 8)+ SRED_SSMPLE_SSS 0, 0,W, R),
6:
for j = 1, 1.2.....k do
7:
(X,, 1,) Staai Lsl
1/ update replay buffer
8:
W W,+1 W
8:
M M (Uo(())) Y3)
a aww, (So.w (X)).
9:
if M > max then
9:
end for
10:
M + M M 9)}
// meta-training testing
11:
end end for
// (updating RINN
12:
10:
0 0= LoL Wk+1)
13:
return W M
11: end while
14:
end function
EXPERIMENTS
summary of their meta-training and meta-testing data sets (which have disjoint sets of classes) is shown in Table The experimental setup follows (Javed & White 2019). The meta-training training set contains task sequences. Each sequence Strain randomly samples tasks (classes) from the meta-traine training data set, and each task has 5 randomly sampled images for each class. The meta training test set Stest randomly samples another 5 images for each of the same 3 classes in Strains well random batch of images from previously seen classes (Omniglot uses 15 random
5

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Predictive Sample Selection (PSS).
Require: FC: the FC layer for the Predictive Sample Selection (PSS)
1: function PRED_ PRED_SAMPLE_SS
W W
2: scores = [Fc(Rt
3: () Rlarg min(
4: W1+1 W - aV w,, (ww, (F)((((()))))
5: compute n using Wj+1 and 'R (1) using W111
6:
update FC based on actual loss increase ((((16(8))))) F
7: return (X)( Y)
8: end function
images, while mini-lmageNet and CITAR-FS use 10 random images). The meta-testing training set is a task sequence containing all classes in the meta-testing set, while the meta-testing testing set is a set of unseen samples for all classes in the meta-testing set. Each task (class) randomly samples training and test images from the meta-testing set (15 training and 5 test images for split- Omniglor; 30 training and 100 test images for mini- ImageNe and CIFAR-FS). Further details on the setup can be found in Appendix C
meta- training
meta- testing
#classes
#samples/class
split-
meta-train
963
20
Omniglot
meta-test
600
20
ANML OML OML(ER)
mini-
meta-train
64
600
ImageNet
mela-test
20
600
ER REP
OML(REP)
meta-train
64
600
CIFAR-FS
OMER
ER
meta-test
20
600
OMREP
REP
REP
Table I: Summary of the meta- training and meta- testing sets for the data sets used.
OMREP(
REP
Table 2: Use of replay in each method.
We will experiment with the state-of-the-art of OMI 3 and ANML Beaulieu et al. 2020), and the following OML variants.
(i) OML(ER), which combines OML with ER (and reservoir sampling) as in (Javed & White 2019). Meta-training follows that of standard OML, but ER is used during meta-testing training;
(ii) OML(REP): Meta-training follows OML, but representation replay (REP) is used during meta- testing training;
(iii) Online aware Reea-Eexpeieece Replay (OMER): The naive approach for incorporating ER into OMI. as discussed in Section 3.I.I.
(iv) Online aware MetaaRRepresnntttion Replay (OMREP): The proposed method (Section 3.1.2):
(V) (aRRE((no-REP) an OMREP variant which does not use REP in meta-testing training.
The use of replay each of these methods is shown in Table 2 Performance measured by the accuracy on the meta- meta-testing sets from all previously learned tasks.
4.1 USING EXPERIENCE REPLAY IN META- TEETA-EETING ONLY
We first demonstrate that using experience replay only in meta-testing, in (Javed & White, 2019). may not be useful. Experiments are performed on ANML, OMI and its variants (MR) and OMEP) In OML (EL sampling. Its size set to 20% of the meta-testing training set (i.e., 1,800 for 120 for mini-ImageNNe and CIFAR-FS), Similar experiments have been performed in (Javed & White 2019). However, here, we use longer task sequences to further explore the effects of catastrophic forgetting, and use data sets with varied complexities.
the replay buffer is initially empty and populated by reservoir and
Figure ((a) shows the meta- testiettttng testing accuracies with the number of classes on Split- Siiit-Omnigoo.. As can be seen, the testing accuracies of ANML and OML are very similar. The curves for OML(ER)
We used the version after bug fix for correct meta-gradients.

---

Under review as a conference paper at ICLR 2022
and OML(REP) are exactly the same, the RLN is not updated during meta-testing. The only difference between them is that the stored representations in OML(REP) do not need to go through the RLN during meta-testing training. Hence, OML(REP) is more efficient than OML(ER) and takes less memory a Both methods significantly outperform ANMI. and OMI..
DVL (21)
OVL EM
85
DC) (917)
USA (107)
00), part
100
205
200
452
(a) Omniglot.
(b) Net
(c) CIFAR-FS.
Figure 2: Meta-testing testing accuracies of ANML,, OMI, OML(ER), OML(REP). Note that the curves for OML(ER) and OML(REP) completely overlap with each other.
As for mini- NmageNrr and CIFAR-FS, they are more difficult than split-Omniglos and the observations in Figures 2(b) and 2(c different. First, ANML fails to learn the meta-testing classes. Second, while (MR(ER)) WOMLOMER has good testing accuracies on Split-Omnigle (as also reported in (Javed & White 2019)) is now outperformed by OML. This can be attributed to the increased effects of the stability ptability-lasstiiity tradeoff (Parisi et al. 2019). Though revisiting previous tasks can help maintain the accuracies for learned tasks, this hinders the model's ability learn new tasks.
4.2 USING EXPERIENCE REPLAY IN BOTH META-TRAINING AND META-TESTING
In this experiment, experience replay is also used in meta-training. Since OML(ER) is identical OML(REP) in terms of accuracies, compare ANML, OML and OML(REP) with the OML variants of OMER, OMREP(nO-REP) and OMREP. The setup is the same as Section 4.1
Figure shows the meta-testing testing accuracies. OMREP has the best accuracies all data sets. Thus, by making the meta-training training procedure consistent with that of meta-testing training, OMREP can learn the effects of ER on continual learning during meta-training. On the other hand, OMER (which uses ER in meta-training and meta-testing) does not have good accuracies overall. As discussed in Dection3.1.I1 information from some meta-training samples may have been canceled out by replay samples. Thus, the RLN fails to learn good meta-continual learning representations.
CILL
CIRL
DML
CMA- (RDF)
83
Accewery D50
32
300
12
(a) Omniglot.
(b) mini--maggeNet.
(c) CIFAR-FS.
Figure 3: Meta-testing testing accuracies of ANML, OMI, OMI(REP), OMER, OMREP(no-REP), and OMREP.
*5.3GB for OMI((RE)) vs 6.4GB for OMI(ER) in meta-training, and 2.4GB for OMI((RP)) vs 7.1GB for OML(ER) in meta-testing.
7

---

Under review as a conference paper at ICLR 2022
4.3 BETTER META-CONTINUAL LEARNING REPRESENTATION
In this section, we demonstrate that the RLN representation learned by OMREP has a better clustering structure and is more discriminative. Since the meta-trained RLNs for OML(ER) and OML(REP) the same as that of OMI, we only compare ANML, OMI, OMER and OMREP
Figure shows the visualizations by t-SNE (Matten Hinton 2008 on three random classes from the meta-testing set of CIFAR-FS. Visualizations for the meta- perta-training set and other data sets are shown in Figures 88 and 7 respectively, in Appendix D As can be seen, for ANML, OML, and OMER, their RLN output representations for different classes are mixed together. This indicates the failure in capturing representations useful for classification. Overall, the clustering structure from the OMREP's RLN ouiput representations is the best, with most samples of the same class close each other. Table quuatittaivvel evaluates the clustering quality by the intra-class to inter-class variance (Goldblum al. 2020). A lower ratio means better class separation. OMREP yields the best class separation on the more difficult data sets mini- mani-ImageNer and CIFAR-FS. On split- Omniglot, OMREP is comparable with OMER and are better than the others.
(a) ANML.
(b) OML.
(c) OMER.
(d) OMREP.
Figure 4: T-SNE visualizations for the RLN output representations learned from the meta-testing of
CIFAR-FS. Circles represent training samples, and crosses represent testing samples (best viewed in color).
split- Ommiglot
mini- ImageNet
split- Omniglot
mini- ImageNet
CIFAR-FS
CIFAR-FS
ANML
70.6
12.8
25.2
ANML
192.1
8.8
6.0
OML
60.5
16.8
29.5
OML
186.8
8.7
4.4
OMER
70.6
32.9
28.3
OMER
129.5
5.2
4.2
OMREP 71.8
32.3
33.3
OMREP 138.0
4.8
3.4
Table 4: Meta detaatesting testing accuracies by a ANN classifier on the learned RI.N output representa- tions.
Table 3: Intra-class inter-class variance ratios for the learned RLN output representations.
Following (Beaulieu et al. 2020),
k-nearest-neighhee (kNN) classifier of training samples in the meta-testing training task sequence for each class on the RLN representations learned by various methods. The kNN classifier is trained the RLN representations from the meta-testing training set and is evaluated on the RLN representations from the meta-testing testing set. As can be seen from Table OMREP outperforms all the other methods on aplit-Omniglo and CIFAR-FS RS,, and has comparable accuracy as OMER on mini-ImageNNtt.
4.4 SAMPLE SELECTION
In this experiment, we compare the proposed Predictive Sample Selection (PSS) with the following ways to populate the replay buffer in OMREP: (i) reservoir sampling (Vitter 1985); (ii) ring buffer (Chaudhry et al.) 202T) (iii) greedy sample selection (GSS) (Aljundi et al. 2019b); and (iv) bilevel
optimization [Borsos al. 2020).
Figure shows the meta-testing testing accuracies. As can seen, PSS outperforms the others. Table shows the time for meta-testing. Reservoir sampling and ring buffer are simple and so have
We do not compare with OMREP(no-REP), it has the same representations as OMREP, and only differ in that it does not use replay in meta-testing
'The ratio is defined as L
C Σ., () where fo(2v.) is the feature representation of Σ, ||pe
ohevem
N
sample Ji. in class i, μ4 is the mean representation of class i, is the mean representation for the whole data set, C is the number of classes, and N the number of samples per class.
8

---

Under review as a conference paper at ICLR 2022
small time requirements. PSS only requires forward pass of the trained FC layer and thus is also efficient. On the other hand, GSS requires computing the gradients for both the new sample and the replay buffer batches at each step and is much more expensive. Performing bilevel optimization at each step is, as expected, the most expensive.
Satandr
Ting
755
ICCassss
#Cassss
#CBasss
(a) Omniglot.
(b) mini- Meeii-.
(c) CIFAR-FS.
Figure 5: Meta-testing testing accuracies for the different memory selection schemes in OMREP.
Omniglot
mini-ImageNet
CIFAR-FS
reservoir
0.15
0.01
0.01
ring
0.15
0.01
0.01
GSS
7.5
0.5
0.5
bilevel
30.0
2.0
2.0
PSS
0.75
0.05
0.05
Table 5: Meta-testing time (in GPU hours) for different memory selection schemes.
4.5 EFFECT OF REPLAY BUFFER SIZE
In previous experiments, the replay buffer size is set to 20% of the meta-testing training set. In this experiment, we vary the size from minimum of samples per class (i.e., 13.33% for Omniglot and 6.67% for miti-ImageNNe and CIFAR-FS) to 20%. We consider the most difficult setting where the largest number of tasks (classes) has to be continually learned (i.e., 600 classes of Omniglot, and 20 classes mini-ImageNer and CIFAR-FS).
Figure 6 shows the meta-testing testing accuracies. For reference, we also show the accuracies of OM which does not use a replay buffer. As can be seen, on all data sets, OMREP with PSS consistently outperforms all the baselines.
DM
caree (PK)
DM)
1022) O6IPEr (P30)
Oli-
Okinii (522)
53
ceaoci (003)
ckieer (750)
curter (P50:
Placancc
plus araia
23
15
17
20
Omniglot.
(b) Nitie-mageeee..
(c) CIFAR-FS.
Figure 6: Meta-testing testing accuracies with different replay buffer sizes. Note that curves for OML(ER) and OML(RE)) completely overlap.
5 CONCLUSION
In this paper, we proposed representation replay, which improves the meta-continual learning repre- sentations while ensuring that the batch nature of ER does not conflict with the online-aware nature of meta-continn continuul learning. We also proposed the Predictive Sample Selection, which selects the most sensitive samples into the replay buffer. Experimental results demonstrate that the proposed method outperforms the state-of-the the- and the meta-continual learning representations obtained have better clustering structures and more discriminative.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
We have reviewed the ICLR Code of Ethics and have confirmed that this paper adheres to them. There no privacy and security issues all data sets used have been publicly released. The findings and research works do not have any negative social impacts or inappropriate applications.
7
REPRODUCIBILITY STATEMENT
For all theoretical results in this paper, the assumptions have been explained clearly and the complete proof' of the propositions are included in Appendix A and B The methodologies used are detailed in
Section 3.1 Appendix
The data sets used and the detailed experimental setups are provided in Section and and
REFERENCES
R. Aljundi, L. Caccia, E. Belilovsky, M. Caccia, M. Lin, L. Charlin, and T. Tuytelaars. Online continual learning with maximally interfered retrieval. Technical report, 1rXiv:1908.0477 2019a.
R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio. Gradient based sample selection for online continual learning. In NeurlPS, pp. 11816-1188 1125, 2019b.
S. Beaulieu, I.. Frati, T. Miconi, J. Lehman, K. Stanley, J. Clune, and Cheney. Learning to continually learn. In ECAI, pp. 992-1001, 2020.
L. Bertinetto, Henriques, P. Torr, and A. Vedaldi. Meta-leaminng with differentiable closed-form solvers. In ICLR, 2019.
M. Biesialska, K. Biesialska, and M. Costa-jussà, Continual lifelong learning in natural language processing: A survey. Technical report, 2rX0v22012.09822, 2020.
Z. Borsos, M. Mutny, and A. Krause. Coresets via bilevel optimization for continual learning and streaming. In NeurlPS, 2020.
M. Caccia, P. Rodríguez, O. Ostapenko, F. Normandin, M. Lin, L. Caccia, I. Laradji, I. Rish,
A. Lacoste, D. Vázquez, and Charlin. Online fast adaptation and knowledge accumulation (OSAKA): A new approach continual learning. In NeurlPS, 2020.
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, Dokania, P. Torr, and M. Ranzato. On tiny episodic memories in continual learning. Technical report, arXiv:1902.1 10486, 2019.
A. Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In AAAI, pp. 6993-7001. AAAI Press, 2021.
M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuyte- laars. A continual learning survey: Defying forgetting classification tasks. Technical report, 1XXv:1909.08383 2019.
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pp. 1126-10 1135, 2017.
M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In JCML, pp. 3607-3616. PMLR, 2020.
I. Goodfellow, M. Mirza, X. Da, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgeting in gradient- neural networks. Technical report, arXiv:1312.6211 2015.
G. Gupta, K. Yadav, and L. Paull. La-maml: Look-ahead meta learning for continual learning. Technical report, arXiv:2007.13900 2020.
Hayes, N. Cahill, and C. Kanan. Memory efficient experience replay for streaming learning, In ICRA, pp. 9776. IEFE, 2019.
10

---

Under review as a conference paper at ICLR 2022
T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey. Technical report, arXiv:2004.05439, 2020.
F. Huszar. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, 11(()):449-2497, 2018.
K. Javed and M. White. Meta-learning representations for continual learning. In NeurIPS, pp- 1818-1828, 2019.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A.A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinnkk, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. PNAS, 114(13):321-3352 2017.
Knoblauch, H. Husain, and T. Diethe. Optimal continual learning has perfect memory and is NP-HARD. In ICML, 2020.
C. Lee and A. Lee. Clinical applications of continual learning machine learning. The Lancet Digital Health, 2(6):e279- 2020.
Lesort, A. Stoian, and D. Filliat. Regularization shortcomings for continual learning. Technical report, arXiv:1912.03000 2019.
Z. Li and D. Hoiem. Learning without forgetting. IEEE, 20(12):9955-2947, 2018.
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. NIPS, pp. 6467-6476, 2017.
L. Maaten and G. Hinton. Visualizing data using t-SNE. Journal Machine Leurning Research, (N0v):2579-2605, 2008
Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner. Online continual learning in image classification: An empirical survey. Technical report, 11Xiv:2101.10423, 2021.
A. Nichol, J. Achiam, and Schulman, On first-order meta-learning algorithms. Technical report, MXXiv:100300299% 2018.
G. Parisi, R. Kemker, J. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: review. Neural Networks, 113:54-71, 2019.
L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni. Latent replay for real-time continual learning. Technical report, 1uXiv:1912.01110 2019.
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Tu, and G. Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
A. Robins. Catastrophic forgetting, rehearsal, and pseudorehearsal. Connection Science, 7:123-146. 1995.
G. van de Ven and A. Tolias. Three scenarios for continual learning. arXiv:1904.07734, 2019.
Technical report,
G. van de Ven, H. Siegelmann, and A. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):1-14, 2020.
J. Vitter. Random sampling with reservoir. ACM Transactions on Mathematical Software, 11(1): 37-57, 1985.
Yuan, G. Zhang, A. Karatzoglou, X. He, J. Jose, B. Kong, and Y. Li. One person, one model, one world: Learning continual user representation without forgetting. Technical report, arXiv:2009.13724 2020.
Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML, volume 70, pp. 3987-3995, 2017.
11

---

Under review as a conference paper at ICLR 2022
A PROOF OF PROPOSITION 1
Proof. Let U, Wal) = 10,, W3] - a 1o,0, where Ej (B, B, I1). As in Nichol et al. (2018), we have
VoL(Ok+1,Wezz = VOL(UR( (U7(U)( ((),, Wil))))
= (o(, ... Ve. (UR,, W&D)) Bess L(Ou+1, W 1)22))
As V = -1- -V fj. -a w; Ggl. VoL(Ox+1. Wx+1) in reduces to:
- -a vw, Von 101121100+1 We+1).
By Taylor's theorem,
Vor.i [(Ox+1, Wx+1)
= VeL + VV - -1) + +0100x+1 -0111²)
5 + 11 0()))
k
3- -alll Vo,ly+0(00)
j=l
k
= 5 all Σ
(Vel + +(())
j=1
k
= g - all Σ +(())+
j=1
Expanding this to leading order as Nichol al. 2018).
J:
VoL(On, (LL00,,W++1) = II - (B)) VV (a-aft all Σ F((B)) + +0(a2)
a=1
(1
iT((:)+ (0(2))0(02)0000)) 1)
1 a
g all Σ
((B,) +0(a3)
j=1
j=1
H'(B)) a Σ (B,), 0
j=1
B PROOF OF PROPOSITION 2
Proof. Let U1, W2D) = 10,, W v 10 wy where = e(So.w (Xy), Note that Ey is now the loss on the new sample (X,, j, Y2) instead of batch of samples By in Appendix A As in Nichol et al. (2018), we have
V VoL(0 ( = VoL(Ua(.... (10),, Wi))))
= (o(U)((0,,W:i)) (U= W&I)) ocnr 1(0+11, W 13(3)
Vo,,; w. Lyl voL(OA+1. W in reduces to: As Vo, U,,,, = II VV ly. vV
II V b, -a a w,, Ver.i Verri L(0k+1, W
12

---

Under review as a conference paper at ICLR 2022
By Taylor's theorem,
Ver.i L(Ou+1, Wk+1) =
- -1) + +0100++1- - -1113)
= J+ -
= j-aHf Vo,0z+0(a7
j=l
k
= 0 aHl
(oll + 0(a))
j=1
k
9 oH Σ
j=1
Expanding this to leading order as in Nichol et al. 2018).
VoL(On, Wk+1)
II ( (X)) (x)) -a - v,,
g all Σ [((x)) + +
))
(X,) + +
0 - aH Σ
8((3))
+ +
j=1
==1
k (X aH > g'(x)).
+ +0(a²).
C EXPERIMENTAL SETUP
The model architecture used for Omniglot follows (Javed White 2019). and the architectures for mini-ImageNee and CIFAR-FS are the same as (Finn et al. [0017 77 and (Bertinetto et al., 2019) but with
White 2019). The additional filters also added by (Javed & White, 2019) because of the added difficulty for continual learning. The model architectures for Omniglot, mini-ImageNNe and CIFAR- FS are shown in Table
more filters and no batch normalization layers as in (Javed
Melii-Omniglot
mini-ImageNet
CIFAR-FS
Input size
84 84
84 x 84
32 x 32
Number of convolutional layers
4
Number of filters
256
256
256
Kernel size
3
3
3 1
Stride
[2,1,2,1,2.2]
Padding
Maxpool kernel size
None
2 x 2 [2,2,2.1]
2 2
Maxpool stride
None
[2,2,2.1]
Maxpool padding
None
0
0
Number of fully-connecter layers
1
Activation function
ReLU
ReLU
ReLU
Table 6: Networks used for Split-Omniglot, mini-ImageNer, and CIFAR-FS.
For methods using a replay buffer, the buffer size is set to 15 and the batch size for replay is 10. The inner learning rate is 0.03 and the meta-learning rate is 10 The models meta-updated by the Adam optimizer and are trained for 200,000 epochs.
Following the meta-testing setup in (Javed & White 2019), we perform five validation runs choose the best learning rate to use from (0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003, 0.000013. Us- ing the selected learning rate, the meta-testing phase is performed 50 times. The average is shown
13

---

Under review as a conference paper at ICLR 2022
(a) ANML.
(b) OML.
(c) OMER.
(d) OMREP.
(c) ANMI.
(f) OMI..
(g) OMER.
(h) OMREP.
(i) ANML.
(i) OML.
(k) OMER.
(I) OMREP.
Figure 7: T-SNE visualizations for the RLN output representations learned from the meta-testing sets of Omniglot (top), mini--magell (middle), and CIFAR-FS (bottom). Circles training samples, and represent testing samples.
with 95% confidence interval (drawn using 1,000 bootstraps). For methods using replay strategies, a batch size of 100 used during meta-testing to adapt to a large number of tasks.
For ANML, the model architecture follows (Beaulieu et al. 2020 for all data sets (the input images are resized accordingly). but the hyperparameters values are ones provided the tables.
We use one NVIDIA GeForce RTX 2080Ti GPU to train the models. The meta-training phase takes about 3-4 days, while the meta-testing phase takes one day for Split-Omniglot and 2-3 hours for mini-ImageNet and CIFAR-FS.
D T-SNE VISUALIZATIONS
Figure (resp. shows the t-SNE visualizations on the meta-testing (resp. meta-training) sets for ten random classes of Omniglot, and three random classes of Imiie-e and CIFAR-FS. From the visualizations,
see that OMER and OMREP generally have better clustering structures, with OMREP having a farther separation between classes and fewer mixups between different classes for both the meta-training and meta-testing sets.
14

---

Under review as a conference paper at ICLR 2022
(a) ANML.
(b) OML.
(c) OMER.
(d) OMREP.
(c) ANMI
OMI..
(g) OMER.
(h) OMREP
(i) ANML.
(i) OML.
(k) OMER.
(1) OMREP.
Figure T-SNE visualizations for RLN output representations learned from the meta-training sets Omniglot (top), mini-ImageNer (middle), and CIFAR-FS (bottom). Circles are training samples, and crosses represent testing samples.
15


Reviews:
[{"review_id": "oHLJAKDjLzJ", "reviewer": "Reviewer_XC4V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1: Integrating the experience replay (ER) during the meta-training phase is the key contribution. The paper mentioned that using ER during meta-test only but not meta train creates inconsistency. What do you mean by inconsistency? During the meta-test and meta-train, the classes are disjoint; how consistency is defined?\n\n2: The paper extends the OML with ER, which does not seem a significant contribution. The buffer replacement policy is novel, and it may be useful for the other replay based approaches. PSS improves the model performance compared to the Ring, GSS and Bilevel. \n\n3: In Algorithm-2, meta-training procedures are provided for the ER, while meta-test procedures are unclear. How are meta-tests performed? Would you please provide the Algorithm for the same?\n\n4: The direct comparison with OML is not fair since OML does not leverage on ER. It seems that results are trivial since compared to without ER, replay always improves the performance. Also, since during the meta-test, only classifiers are trained, therefore it will be beneficial to store the feature representation is also trivial.\n\n5: Another recent work [a] also improves the representation over the OML for continual learning; I request the author, please discuss the result with [a].\n\n[a]: knowledge consolidation based class incremental online learning with limited data, IJCAI-21", "summary_of_the_paper": "The paper extends the online meta-learning (OML) model with the ER during the meta-training. Also, the paper proposes a replacement buffer policy for samples replacement from the reservoir. Instead of storing the raw samples, since the backbone model is static during the meta-test, it's better to store the feature representations that paper does. ", "summary_of_the_review": "Paper has some novel part but the main contributions compared to OML is not significant.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bWSgdApriz", "reviewer": "Reviewer_iSAe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The paper identifies some issues with one of the current methods for continual learning and propose simple extensions to reduce the effect of those issues\n2. The empirical evaluations/visualizations show clear improvement in performance by adding the proposed ideas\n3. The paper is written very well. Easy to read and understand.\n\nQuestions/Weaknesses\n1. Could you elaborate a bit more on the inconsistencies (how exactly it affects learning/performance) of using ER only during testing? Is the inconsistency of concern here the inconsistency in inner loop updates between the meta-training and meta-testing?\n2. [Representation replay] \na. Wouldn’t not using ER for RLN potentially lead to forgetting?\nb. Wouldn’t representation replay still have the interference issue for PLN? \nc. Is there a tradeoff that is being made here? If so any thoughts on what exactly it is and how to choose that tradeoff?\n3. Isn’t the target for the prediction in PSS always shifting? If thats the case, what is the rationale behind not updating the FC during meta-testing training? Wouldn't the loss for the same sample change over the course of meta-testing training?\n4. In ER, Would having multiple, different (batches of data) updates from the R + new sample alleviate the problem of losing the information from the new sample? \n5. Not clear if the contributions of the paper are significant enough for ICLR  or very incremental. Could you clarify the contributions of the papers in more detail please, including the issues identified, the ideas contributed (novel and point out where ideas were borrowed and applied in a different setting) and evaluations etc.\n6. Are there methods other than ANML for the setup of interest in the paper following OML which was proposed in 2019? While evaluating the proposed changes with OML and variations of OML is essential, it is also important to compare with other methods that look at the same setting to situate the proposed method in the literature. \n\nOther Comments\n1. [Related works section] Note that apart from regularization and rehearsal based methods, there are also several architecture based (modular networks type and parameter isolation methods) that have been proposed for avoiding catastrophic forgetting in continual learning.\n", "summary_of_the_paper": "The paper looks at the problem of catastrophic forgetting in continual learning. They propose a method that extends OML to use ER both during training and testing. In the ER, they store the embedding of the samples instead of the samples themselves to avoid the interference caused by the batch nature of ER and the online nature of OML. They use meta-learning to select samples that are useful for avoiding catastrophic forgetting to be in the ER instead of reservoir sampling. They predict the difference in loss in predicting the class caused by removing a sample from replay and use that score to decide what to remove.", "summary_of_the_review": "Simple extension to a current method. Unclear about the significance of the contribution. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8QXD5v2d7Uy", "reviewer": "Reviewer_Quoa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The authors proposed improvement for OML by integrating ER into  meta-training. \n* Experiment shows the effectiveness of the proposed method. \n\n\n\nWeakness:\n\n\n* The technical novelty of the proposed method is limited and is incremental compared to online aware meta-learning (OML). The proposed method is just a simple improvement based on the framework of OML. The difference is that the proposed method  integrates ER  into the meta-training of OML \n\n* The clarity of presentation could be improved.  For example, the terms meta-training training, meta-training testing, meta-testing training are not comfortable to read. It would be better to use some common simple words that people widely used to improve the clarity of presentations. \n\n\n* For the proposed representation replay, there is a lack of concrete analysis why the proposed method of integrating ER into meta-training could better alleviate catastrophic forgetting compared to OML. It is unclear to see the benefit and necessity of using ER during meta-training only from the derived gradients. \n\n* Sample selection method lacks in-depth analysis about how different task samples are selected. What are the sample selection results for different seeds?  Is the proposed sample selection method sensitive to data streams or task ordering?\n\n* For the experiment, lack of comparisons to some existing methods which consider the memory selection, e.g., [1]\n\n\nReference: \n\n[1] Online Continual Learning with Maximal Interfered Retrieval. Rahaf Aljundi, et al. NeurIPS 2019\n", "summary_of_the_paper": "This paper proposes to  integrate ER into meta-training of online aware meta-learning (OML) Javed & White (2019). The authors propose to store the samples’ representations, not the samples themselves into a replay buffer.  In addition, the authors propose a sample selection method with a meta-learned Predictive Sample Selection to select the most informative examples. Experimental results on several datasets demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "The technical contributions of this paper is limited compared to OML. Also, the presentation clarity, detailed analysis of the proposed method, experiments need to be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0u3GHCg6Kum", "reviewer": "Reviewer_oDEb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of using representations instead of samples for buffer is interesting. \n2. The Predictive Sample Selection method is effective.\n\nWeaknesses:\n1. Lack of experiments or theoretical analysis to support using representations instead of samples. \n2. Sample selection method lacks concrete analysis.", "summary_of_the_paper": "The paper studies the meta-continual learning problem. Representations replay instead of samples replay are used to improve the performance of meta-continual learning. And a new strategy called Predictive Sample Selection is used to select samples into the replay buffer. Experiments demonstrate the effectiveness of the proposed methods. ", "summary_of_the_review": "The topic is intersting. However, the contribution of this paper is minor. It is a combination of several incremental improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Addressing the problem of catastrophic forgetting in continual learning, this paper extends OML to use experience replay (ER) during training, instead of the original approach which uses ER during test phase only. The paper proposes a policy for samples replacement from the reservoir. Experiments show the superiority of the approach in three standard benchmarks compared with several baselines. 

Reviewers were unanimously concerned that the technical contribution of the paper is not sufficient. The authors addressed several issues, including experiments to compare with additional baselines, but the technical novelty remains limited for an ICLR publication. 
The paper cannot be accepted at its current form.
Decision: Reject