A simple but strong baseline for online continual learning: Repeated Augmented Rehearsal
Authors: Yaqian Zhang
Bernhard Pfahringer
Eibe Frank
Albert Bifet
Nick Jin Sean Lim
Alvin Jia
Venue: NeurIPS 2022 Accept (2022)
Keywords: continual learning
experience replay
online learning
forggetting
online continual learning

Abstract: Online continual learning (OCL) aims to train neural networks incrementally from a non-stationary data stream with a single pass through data. Rehearsal-based methods attempt to approximate the observed input distributions over time with a small memory and revisit them later to avoid forgetting. Despite their strong empirical performance, rehearsal methods still suffer from a poor approximation of past data’s loss landscape with memory samples. This paper revisits the rehearsal dynamics in online settings. We provide theoretical insights on the inherent memory overfitting risk from the viewpoint of biased and dynamic empirical risk minimization, and examine the merits and limits of repeated rehearsal.
Inspired by our analysis, a simple and intuitive baseline, repeated augmented rehearsal (RAR), is designed to address the underfitting-overfitting dilemma of online rehearsal. Surprisingly, across four rather different OCL benchmarks,
this simple baseline outperforms vanilla rehearsal by  9\%-17\% and also significantly improves the state-of-the-art rehearsal-based methods MIR, ASER, and SCR. We also demonstrate that RAR successfully achieves an accurate approximation of the loss landscape of past data and high-loss ridge aversion in its learning trajectory. Extensive ablation studies are conducted to study the interplay between repeated and augmented rehearsal, and reinforcement learning (RL) is applied to dynamically adjust the hyperparameters of RAR to balance the stability-plasticity trade-off online.

Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning
Yaqian Zhang University of Waikato New Zealand yaqianz&waikato. ac.nz
Bernhard Pfahringer University of Waikato New Zealand bernhard@waikato..er ac.nz
Eibe Frank University of Waikato New Zealand e eibe@waikato..
Albert Bifet
Nick Jin Sean Lim University of Waikato New Zealand
Yunzhe Jia University of Waikato New Zealand alvin.jia@ue jiriajjiaaaikkto.. ac nz
University of Waikato
New Zealand LTCI, Télécom Paris, France abifet@waikato. ac
nick www.waakawtatooo.mmmmm.m.....oom ac. nz
Abstract
Online continual learning (OCL) aims to train neural networks incrementally from non-stationary data stream with single pass through data. Rehearsal-based meth- ods attempt to approximate the observed input distributions over time with small memory and revisit them later to avoid forgetting. Despite their strong empirical performance, rehearsal methods still suffer from poor approximation of past data's 's loss landscape with memory samples. This paper revisits the rehearsal dynamics online settings. We provide theoretical insights on the inherent memory overfitting risk from the viewpoint of biased and dynamic empirical risk minimization, and examine the merits and limits of repeated rehearsal. Inspired by our analysis, a simple and intuitive baseline, repeated augmented rehearsal (RAR), is designed address the underfitting- ovvrrfitingg dilemma of online rehearsal. Surprisingly, across four rather different OCL benchmarks, this simple baseline outperforms vanilla rehearsal by 17% and also significantly improves the state-of-the tiffthe-ar art rehearsal-Nee bbsed methods MIR, ASER, and SCR. We also demonstrate that RAR successfully achieves accurate approximation of the loss landscape of past data and high-loss ridge aversion in its learning trajectory. Extensive ablation studies are conducted study the interplay between repeated and augmented rehearsal, and reinforcement learning (RL) is applied to dynamically adjust the hyperparameters of RAR balance the stability-plasticity trade-off online. Code available hww.:///wwwwnaanooaoeneoooonnnnn
Introduction
Despite its recent success, deep learning largely relies on the assumption of independent and identi- cally distributed (i.i.d.) data that can be repeatedly revisited during training. Non-i.i.d settings challenging for neural networks due to catastrophic forgetting: previously learned knowledge can easily overwritten when training
new data because this data may follow a different distribu- tion [Li and Hoiem 2017] Rebuffi et al. 2017 Delange et al. 2021]. Online continual learning (OCL Online CL) studies how enable deep learning an online manner from non-stationary data stream. As the data stream can be vast or even infinite, it is infeasible to store and shuffle the dataset for multiple epochs of training. Therefore, fundamental assumption is that the data stream can only
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

be accessed one batch at a time and training is performed with a single pass over the data Aljundi al. 2019]
Experience replay (ER), also known as rehearsal [Chaudhry et al. 2019 Delange et al. 2021 is a key idea in OCI. It stores subset of previously seen data 2 in a fixed-size memory M and revisits the memorized samples during training to mitigate forgetting of previous tasks. To update the model, a batch sampled from the memory is combined with the incoming batch from the stream to compute the gradient [Chaudhry et al. 20191. Different variants of ER have been developed to improve memory management policies and representation learning. achieving state-of oh-the-rrt performance in number of standard OCL benchmarks (Aljundi et al. 2019 Mai et al. 2021 Shim et al. [02T].
However, whether rehearsal is appropriate for continual learning, considering the risk of overfitting the memory when using data from the memory to directly contribute to the gradient computation, has heen dehated vigorously [Lopez-Paz and Ranzato 2017 Chaudhry et al. (2019) Verwimp et al. [022]] I. The potential for overfitting has motivated the development of constraint- Paz and Ranzato 2017 e.g., GEM and A-GEM, which use memory samples solely constrain the gradient direction. However, there is empirical evidence suggesting that rehearsal-based methods consistently outperform methods that do not train directly on the memory [Chaudhry al., 2019]. This indicates rehearsal on the memory does not necessarily prevent effective generalization, possibly due to the implicit regularization effect of incoming data [Chaudhry al., 2019]. Nevertheless, recent work analyzing the loss landscape when applying rehearsal to offline continual learning finds that memory samples indeed provide poor approximation of the loss landscape of past tasks, especially near high-loss ridge. As result, "instead of ending up near the high-loss ridge in perspective of the rehearsal memory, the solution in reality resides on the high-loss ridge for the training data" [Verwimp) et al. 2021]. This latest finding poses the question of how to better approximate the loss surface of past data [(D; 0) with memory samples' loss ().
replay methods [Lopez,
To better approximate past data's loss surface, previous work studies which samples should be memorized for rehearsal. Instead, we examine the optimization process during rehearsal and study how to effectively perform rehearsal with the memorized samples. Focused on online CL, our study extends the previous understanding of rehearsal along two directions. First, we provide theoretical considerations that reveal two insights regarding the extent of overfitting to memory: it is a) related to the inherent attributes of the OCL problem concerned and b) varies across the different stages of continual learning. Second, we highlight the limits of applying rehearsal with multiple iterations a trick used to maximally utilize the incoming batch [Aljundi et al. 2019) and identify an underfiitingg-ovefitinng dilemma for online rehearsal.
Based on our analysis, we design simple baseline to deal with the underfitting-overfiie in online CL problems, dubbed repeated augmented rehearsal (RAR), that can be easily integrated into existing rehearsal-bass based methods. Surprisingly, this simple baseline leads to a large performance boost for ER, as well as state-of ifate-of-the-at the-harrt art ER-based approaches, across four OCL benchmarks. More importantly, the loss landscape analysis shows that RAR can help memory samples reliably approxi- mate the distribution of past data and successfully avoids the high-loss ridge of past tasks. To better understand the behavior of RAR, we further investigate the interplay between repeated rehearsal and augmented rehearsal via an ablation study. We also propose reinforcement learning- beanning-based method to dynamically adjust the hyperparameters of RAR and balance the saaility-pllasiicie trade-off in an online manner.
dilemma
2 Related Work
We consider the online continual learning setting with non-stationary (potentially infinite) stream of data D,: at each time step f, the continual learning agent receives an incoming batch of data samples B, (x,, yi li. B. that drawn from the current data distribution P(D,). The period of time where the data distribution stays the same is often called a task or experience in the continual learning literature. An abrupt change in the data distribution occurs when the task changes. The standard objective during training is to minimize the empirical risk on all the data seen so far:
Online continual learning:
min R0) = min
E.B.
Σ
Σ L(f((x),),
(I)
0 with loss function C, the CL network function f. and its associated parameters 0.
Σ,
X.EEB
2

---

Metrics: A common metric is the end accuracy after training on T tasks, defined as Ar + شi=1 arj, where dn denotes the model's accuracy the held-out test set of task after training on task i. Other metrics are "forgetting" [Chaudhry et al.) 2018). which is defined as Fr
(ar. ma.. (1,z) and the related metric "backward transfer" (Lopez.
Paz. and Ranzato 2017]: B - E @T 04,5
Online setting: A key difference between online and offline CL is that the latter assumes full access to the whole training data for the task that is eurrently being processed. Therefore, it allows training on each single task with multiple epochs (e.g.. 70-200 epochs) (Rebuffi et al. 2017 Wu et al. 2001 Cha al. 202T]. The online CL setting is more challenging because the agent can only access the current batch of incoming data and performs training with a single pass through the data.
Experience Replay (Rehearsal): Chaudhry et al. (2019) propose experience replay (ER), which performs joint training on memory samples and incoming samples. A simple but strong baseline approach to sampling in ER is reservoir sampling Vitter (1985). Aljundi et al. (2019) propose Maximally Interfered Retrieval (MIR), which retrieves the samples that will be most negatively impacted by the foreseen parameter updates. Shim al. (2021) propose ASER, which selects samples to best preserve existing memory-basee based class boundaries. In terms of model training, Mai et al. (2021) propose to replace the cross-entropy loss with the supervised contrastive loss learn better representation. We consider all these variants of ER in our experiments.
Augmentation: In the standard i.i.d. setting, data augmentation is a widely used method to improve deep learning [Cubuk et al. 2020]. In the offline CL sctting, Mai et al. (202I) use augmentation to construct supervised contrastive loss and Bang et al. [202]] employ it together with an uncertainty- based memory management strategy. However, these papers apply augmentation together with other advanced techniques, and there is no ablation study on the effect of augmentation per se. Thus, it is unclear whether augmentation itself is beneficial to rehearsal or not, especially in the online setting.
Repeated Rehearsal (Multiple Iterations): Vanilla online continual learning employs single gradient update given an incoming batch of data. To maximally utilize the current incoming batch, Aljundi et al. [2199) propose to perform multiple gradient updates instead. Their experiment with CIFARTO shows using MIR with five iterations leads to a 1.7% improvement in accuracy. In this paper, we systematically analyze the effect of multiple iterations on online rehearsal and provide theoretical and empirical insights on when this trick may improve or harm performance.
Hyperparameter Tuning for OCL: Hyperparameter tuning is particular challenge in OCL due to the lack of a dedicated validation set and the constraint of single pass through the data. Chaudhry eet al. [2018) and Mai et al.] [2022] employ hyperparameter tuning protocol that uses an external validation data stream with small number of tasks. Offline hyperparameter tuning applied to this validation data with multiple passes to identify optimal values, which are then used for the actual online continual learning tasks. A limitation of this method is that relies on external validation data.
3 Revisiting Online Rehearsal: Is Repeated Rehearsal a Good Idea?
We revisit rehearsal from two directions. First, while previous work demonstrates its strong empirical performance [Chaudhry et al. 2019) and provides conceptual analysis for offfine CL [Verwimp et al., 2021]. we focus on the online setting and provide theoretical insights through the lens of empirical risk minimization (ERM). Second, we examine the dynamics of employing rehearsal with multiple iterations. This has been proposed as a trick for online CL to maximally utilize the incoming batch already (Mai et al. 2022 Aljundi et al. 2019]; we investigate whether it is always better than rehearsal with single iteration.
3.1 Empirical Risk Minimization in Online Rehearsal: a Biased and Dynamic Objective
For rehearsal in OCI., at each iteration a batch of data B, obtained from the incoming task, where B, ~ DT and B, (x,, 3/ b= Bx) and a batch B.M is sampled from memory, where B.M ~ D' and B.M {xx, y/ 1......18.. The gradient-based update rule of ER is:
0++1 0,
7
Σ (L(((()))))
71
Σ
VL(((()),))
(2)
Bd
IBM
X.VEB,
3

---

Given this update rule, we would like to establish the corresponding objective function, but this not straightforward to derive because the memory immediately updated after each incoming batch, which means the memory data samples Diy changes all the time. For a widely used memory management policy, where the memory is updated using reservoir sampling [Vitter [985] Chaudhry et al. 2019], we prove in the appendix that the empirical risk for online ER follows Proposition 1.
Proposition (ERM for online rehearsal): Assume an incoming task stream DT and an initial memory set DM with different data distribution P(DT) & ((D))) Assume further that the memory updated at the end of each iteration using reservoir sampling. Then, Eq iiplements unbiased stochastic gradient descent for the following loss function:
min (0) = min Σ ((fo(x). v)
+ +BA
Σ (()o(x),
(3)
X.VEDT
X.YEDA
where À [Prl and [ and [DT| are the memory size and incoming task data size respectively; D
3. = 1/(1
2N
), and N E [B,] denotes the number of samples of the current task that
Myast
T= [Dy] denotes the number of samples pertaining to the tasks so far, excluding the current task. We immediately have Br E (0,1), 1].
have been seen so far and N "past" = ==
The proposition reveals several interesting properties:
Bias: compared with the true objective in continual learning in Eq | the loss of online ER in 3 is actually a biased approximation of the former, as it puts a different weight (3,A) memory samples while the true objective treats all samples with equal weight. This bias, introduced by ER's 's objective function, can contribute to the risk of memory overfitting.
Problem-dependence: given that Be E (0, 1], the biased weight on the memory samples is mostly influenced by X. In other words, the memory overfitting risk is related to inherent property of the CL problem concerned: the ratio À between the current task data size and the memory data size [] While previous works extensively report empirical results on the influence of memory size, to our knowledge, we are the first to point out that the relative data size of incoming task also plays an important role. Empirical evidence is provided in Section (6.3 to support this claim. With a 2k memory, performing rehearsal on the COR550 dataset with larger task data size (A 6) faces high level of memory overfitting while the CLRS dataset with a smaller task data size (A = 1.12) enjoys lower risk of memory overfitting.
Dynamic: ER in online CL optimizes towards a dynamic objective, which varies with each incoming batch f, as the weight on the memory sample depends on N Ner and NT This analysis shows that memory overfitting may be relatively slight for the first few tasks when NT past is small. As more tasks arrive, memory overfitting worsens as fr increases with NT
past- In the case of an infinite data stream, lim Ny Ne.. +00 3r = the memory weight is solely determined by X.
3.2 Repeated Rehearsal: The Decaying Regularization Effect of Incoming Data
We now investigate whether performing multiple iterations is beneficial to rehearsal not. We refer to applying multiple iterations in ER as "repeated experience replay" (Repeated ER) or "repeated rehearsal" and formalize it as follows: for each incoming data batch B, ~ DD from the data stream, we perform multiple gradient updates (K in total) using stochastic gradient descent (SGD) or variants thereof. At each gradient update &o- 1,...K, data batch B. is chosen from memory M and concatenated with the current incoming batch B, to perform replay iteration as follows:
0a 0
VC (x),
" BMM
Σ X.EERM
VL (fo, (x).
(4)
B.
X
'The experiments in this paper mainly consider a balanced CL setting where incoming tasks have the same data size. For imbalanced CL cases, Ả is also dependent on different tasks and should be expressed as X", The finding remains the same.

---

lest
Wa.reer
W
2.
W
2,
2,
W
W2,f
13
(a) RER (K- 10)
(b) RAR (K = 10)
Figure 1: Loss contours of RER and RAR. Memory data overfitting can be observed in (a) for RER but not in (b) for RAR. Note how the shape and position of the loss contour of "TI test" differs from the "TI memory" loss contour in (a). At the CL solution point, the test loss is 7.9 (left blue arrow) while the memory loss only 2.1 (right blue arrow).
Note that when k 1, this update rule provides unbiased stochastic gradient for Eq 3 in the case of l > 1, it provides biased gradient estimate as the same incoming batch is used for consecutive gradient updates. To study the influence of this biased gradient update in repeated rehearsal, we examine the internal dynamics of repeated rehearsal by studying the loss landscape.
We compare the loss surfaces regarding the memory samples and the test data of past tasks during repeated rehearsal. Following the visualization method used in Verwimp et al.) [2021] Mirzadeh et al. [2020], we examine the learning process on the first two tasks in the Split Mini Meai-lmaeNNet dataset and plot the loss landscape the 2D plane defined by three model parameter vectors the model W1 obtained by training on the first task until convergence, the model Wy obtained on the second task using experience replay, and the model W2, obtained after training on the second task using finetuning without replay. Verwimp et al. (2021) use this method to demonstrate the memory overfitting in ER for offline continual learning setting with epochs. Our results show that applying ER in the online setting yields severe memory overfitting with 10 iterations (see Fig (a)). In other words, increasing the number of iterations means the loss landscape of the rehearsal memory provides poorer approximation of the loss landscape of previous tasks, shown by the differences in the positions and the shapes between the left and right red contours Fig T (a). As a result, the learning trajectory of RER avoids the high-loss ridge region for the memory data but goes right into the high-loss ridge region for the past tasks' test data.
Memory Overfitting:
Regularization Effect of Incoming Data: To investigate why repeated rehearsal may suffer from even more memory overfitting than vanilla rehearsal, we analyze the regularization effects of incoming data during repeated ER. To this end, we examine the training process given an incoming batch and compare the training loss on the memory batch and incoming batch with respect to memory iteration k (see Fig An interesting observation is that during the training session of a given incoming data batch, the decrease in the training loss on the incoming batch is much faster than the decrease in loss on the memory batches. One intuitive explanation is that the former is computed over fixed batch during multiple iterations and the latter is computed over different memory batch samples. As result, even though the incoming loss is larger than the memory loss the start of training session (k = 1), at later iterations (i.e., k >> 3) the training loss of the incoming batch becomes 10 - 102 times lower than that of the memory batch. This means that at this stage the regularization effect of the incoming data batch is greatly undermined: the joint training on the memory batch and the incoming batch becomes similar training on memory only.
In summary, our findings imply that the performance online rehearsal is constrained by the dilemma between overfitting locally and underfitting globally. Specifically, online rehearsal faces the challenge of underfitting of the large data stream but overfitting of a small memorized data subset. Applying repeated rehearsal ameliorates the former problem but aggravates the latter problem. Therefore, the performance gain from repeated rehearsal is quite limited.
When training and w2, the model is initialized from Wh. The memory contains 100 samples/task (see Fig I (b) left and right). The 2-d coordinate system is built by orthogonalizing w2 and we. W1.
5

---

4 Repeated Augmented Rehearsal
To deal with the overfiting-underrttinng dilemma, we explore simple strategy, "repeated augmented rehearsal" (RAR), which combines repeated rehearsal with data augmentation. Consider a group of transforms G that acts on the input space X' and invariant under function f. i.e., f(gx) fx), G, x E X. Given an incoming batch from the data stream, B,, multiple replay iterations are conducted using this batch. At each replay iteration k, a random memory batch B is sampled and concatenated with the incoming batch. Then, a random transform gt.k E G is sampled and applied to each data point x, in the concatenated minibatch. The model parameters are updated as:
Σ X.YEBA1
V
gror
Be K,VE
B
Intuitively, the augmentation can help alleviate memory overfitting in two ways. First, we observe that upplying augmentation on the incoming batch helps strengthen the regularization effect. As shown in Fig the decaying regularization effect of incoming data is alleviated in RAR, as the loss of the incoming batch stays comparable the loss of the memory batch during multiple iterations. Second, rehearsal on augmented memory batches can help to more accurately reflect past tasks' data distributions. With RAR, the loss landscapes of memory data and past tasks' test data become very similar (see Fig (b)), which suggests that the model ends up in part of the parameter space where the rehearsal memory approximates the past tasks' distribution well. Moreover, the continual learning solution identified with RAR avoids the high-loss ridge not only in the memory data loss landscape but also the test data loss landscape.
Theoretically, we prove that augmented rehearsal reduces the generalization error in OCL. Specifi- cally, assume the augmentation group G is a compact topological group and follows a probability distribution Q. Similar to Proposition 1, it can be easily proven that the augmented rehearsal gradient corresponds to unbiased SGD on an augmented empirical risk (see Proposition 3 in Appendix A}:
R₂(0)= = Σ J. ((folox).)) + B.A X.DEDT
[ [(folox).
Σ N.DEDM
(5)
This result shows that applying augmented rehearsal is equivalent to performing an averaging operation of the loss of rehearsal in Eq 3 over the orbits of certain group that keeps the data
distribution approximately invariant. In the standard i.i.d. learning setting, Chen et al. (2020) found that such an orbit-averaging operation can reduce both the variance and generalization error. Based on B Eq we show that this theoretical benefit of using augmentation to boost model invariance is also applicable to rehearsal in continual learning. In fact, as discussed in Section 3.1 rehearsal in online CL has a biased empirical risk Eq 3 which leads to inherent memory overlitting and poor generalization ability. Thus, this benefit of augmentation in reducing generalization error is particularly important when applying rehearsal in online CL.
The modifications required for the RAR procedure are summarized in Lines 3 and 5 of Algorithm uses a general framework for Eb-ba continual learning that consists of three key components: sampling from memory, joint training on memory data and incoming data, and updating of the memory. As mentioned in the related work section, different ER variants have been proposed to improve these components. RAR can flexibly be combined with any of these ER variants, and investigate the effectiveness RAR on these different ER variants in the experiment section.
5 Reinforcement Learning-based Adaptive Repeated Augmented Rehearsal
There are two key components in RAR: repeated rehearsal and augmented rehearsal. The interplay of the two is determined by the number of memory iterations and the strength of augmentation. A key question how to choose these hyperparameters. remains an unsolved challenge for online CL due to the single-pase -aass assumption [Chaudhry et al. 2018. Finding suitable RAR hyperparameters needs to account for the severity of memory overftting
general, hyperparameter tuning (HPT) still
"Note that the theoretical analysis of the loss functions in Eq 3 and Eq 5 also applicable to offline continual learning, which may be of independent interest. More discussion of the influence of augmentation in offline continual learning can be found in Appendix D.6

---

Algorithm 1: RL-based RAR
(a) RER
M is the memory with fixed size, B3 is the incoming batch from the current task,
Training loss fin togi
Incoming
Merrory betch
0 are the parameters of the CL network, are the parameters the RL agent, K is the number of memory iterations, P. Q are the augmentation hyperparameters
-1
1: procedure RAR(M, B,, f,. w, )
2: Ke, Ps, Q1 Sample
Rehearsal
10
3: for 1, .... K₁ do
(b) RAR
4:
Bi Metrieeria 5: Bang f- aug U B,, Pt, Q+)
Training loss (in logi
6:
Tt 4 ComputeRenve Reward Baug 01,k)
7:
OLA+I + 4 SGD(Baug 01,kk)
8:
end for
Incoriing batch Meroory batch
9:
Me+1 MemU MMmmUpdateee ydate(Ms, BB)
1
10: Wt++1 t (pddteRL(rr)
10
Rehearsal iteration (k)
11: end procedure
Figure 2: Memory loss vs. incoming loss.
and poses extra challenges. In particular, as shown in the ERM analysis, the extent of memory overfitting is related the CL problem features (c.g., task data size and memory size) and also varies at different training stages of the continual learning process. To automatically select suitable RAR hyperparameters for the different CL problems and different training stages, propose to reinforcement learning to adaptively adjust the hyperparameters (see Algorithm[[]
In particular, we design the hyperparameters of RAR as the action space and use the training statisties as the reward (see lines 2, 6, and 10 Algorithm DT. A major challenge of applying RL in online HPT is sample efficiency. The exploration horizon (i.e, training steps) in the OCL environment is quite limited due to the constraint of single pass through the data and poor action choices (undesirable hyperparameters) may lead to bad gradient update step and hurt the OCL training process. To address the sample efficiency issuc, we employ the armti- bandit framework and apply bootstrapped policy gradient (BPG) [Zhang and Goh! 2019]. The key idea of BPG is to incorporate prior knowledge to bootstrap the policy gradient to achieve stable and fast convergence with limited samples. To obtain prior knowledge in OCL problems,
the training accuracy on the memory batch as the overfitting feedback, as a higher training memory accuracy suggests higher chance of memory overfitting. Reward defined as the distance between the current memory accuracy and target memory accuracy. Compared against a target memory accuracy (e.g., 0.9), the current memory accuracy is used indicate whether the current choice rehearsal iteration or augmentation causes too much memory overfitting and then the action selection probability is C.2 adjusted following BPG (see Appendix C.2 for more RL design and implementation details). The algorithm details of applying BPG as a specific RL method for hyperparameter tuning is summarized in Algorithm of Appendix
Experiments
6.1 Experiment Setup
Baseline: We apply RAR to four ER- based continual learning algorithms: ER CCaudhry et al. 2019].
MIR [Aljundi et al. 2019], ASER (Shim al. 2021 and SCR [Mai et al. 202 We also compare
with other continual learning methods, including the regularization- based method LWF [Li and
Hoiem 2017] and the constrained optimization- based method A-GEM (Chaudhry et al. 2018
Dataset: Four CL benchmarks are used the experiments: Seq-CIFAR100 (20 tasks), Seq- MinilmageNer (10 tasks) Vinyals et al. (016)), CORE50-NC (9 tasks) Lomonaco and Maltoni [2017] and CLRS25-NCC (5 tasks) Li et al. 2020 (see Appendix for more details). Additionally, we also investigate ER and RAR on the large-scale ImageNet- 1 dataset in Appendix D.3]
7

---

Table 1: Accuracy four OCL benchmarks with 2k and 5k memory. The performance boost of RAR over ER and ER variants shown.
SEQ
CIFAR TOD
SEQ MINI IMAGENET
CORESO NC
CLRS25 NC
2K
5K
2K
5K
0.1
0.9
LWF
8.7 =
0.5
10.94000
0.5
9.6 =
12.4 + 2.2
AGEM
8.5 d:
0.4
9.2 à: 0.2
11.6 0.1
13.1
1: 0.3
18.6 0.4
19.4 di 1.8
14.6 4: 1.4
14.4 d: 0.3
ER
0.6
26.2 0.2
0.8
24.0 2.6
27.8
18.7 + 1.6
19.2 +0.3
ER-RAR
0.5
0.7
3.9
3.4
39.3
45.0 2.7
28.6 2.7
28.9
GAINS
8.8
10.0
10.01
13.51
15.31
17.2
9.91
9.7
MIR
1..4
25.7
0.66
25.2 2.2
26.9
14.3
1
RAR
0.2
36.1 0.3
0.6
0.79
39.1
44.6
27.8
29.2 2.6
GAINS
9.1
10.4
10.11
12.61
13.9
17.77
13.5
14.0
ASER
0.3
24.3 2.0
15.7
0.7
16
16.7 2.3
19.4 1.3
19.7 + 1.4
ASER-RAR
35.8 1.0 2.3
0.3
0.62
2..2
30.0
28.7
29.5 0.2
GAINS
7.2
11.51
11.31
14.71
7,8
13.3
9.31
9.8
SCR
32.0
0.2
29.7
3.91
4.11
50.3 1.9
2.2
2..6
SCR-RAR
0.7
0.2
0.7
4.,7
53.4
61.1
374 1.0
41,5 0.9
GAINS
5.1
8.4
5.71
10.61
8.3
10.8
14.9
17.9
ERw
1.0
0.2
0.8
0.6
27.8 0.66
10.2
19.2 0.3
ERe--RA
3..8
36,5 0.4
30.4
36,5 0.4
4.2
50,8
28.6
28.9 1.5
GAINS
9.8t
10.31
10.31
13.51
20.71
23.0
9,41
9.71
23.8
1.1
DER-RAR
30,0
0.5
26.2
+
37.7
3.7
28.4 3.2
27,4 3.8
GAINS
21.61
32.81
14.41
23.21
11.9
18.6
16.6
14.81
FR (win wigl RAR nasuug)
FR leto aug) RAR (wibug)
-0.1
P0-K2
-0.10
PO-K1
Stability
6.20
Stability
-0.15
0.25
PL-K2
-020
P1
PL-K2
P1-22
-0.35
PL K3
F1-55
PL K20
P1-100
P1-120
P14220
Plasticity
Plasticity
Figure 3: Stability and plasticity trade-off: CIFAR100 (left) and Mini-ImaggNee(righh))
Implementation: We use a reduced ResNet-18 for datasets following Mai et al. (2021), Aljundi et al. [219]. Single-head evaluation is employed with a shared final layer trained for all the tasks. SandAuumentationn Cubuk et al. (2000] is used for auto augmentation. Given set of augmentation operations, it randomly selects P augmentation operations and exerts augmentation magnitude of Q for all the selected augmentation operations on each image. All the experimental results we present are averages of three runs. We summarize all hyperparameter details in Appendix C The running time of different algorithms is shown in Appendix D.5
6.2 Main Results
RAR with ER and its variants We first analyze RAR's performance with pre-defined hyperparam-
eter set (K = 10, P Q = 14). As shown in Table RAR greatly improves the ER method on the four datasets, by +8.8% +17.2%. Moreover, RAR also leads to substantial gains for the other algorithmic variants of ER for all datasets (MIR: +9.1% ~ +17.7%. ASER: +7.2% ~ +14.7%, SCR: +5.1% + 17.9%). These results suggest that with advanced memory management strategies, such as MIR or ASER, or representation learning techniques, e.g., SCR, OCL still benefits substantially from repeated augmented rehearsal.
RAR with Modified Rehearsal Loss Besides using the vanilla online rehearsal loss Eq 3 we investigate the effectiveness of RAR with another two, more advanced, rehearsal loss designs: 1) Reweighted memory loss: ER-rw introduces a reweighting hyperparameter
the gradient of Eq 2 to deal with the biased ER loss by balancing the weight of the memory loss and incoming loss; 2) Distillation-based memory loss: DER [Bzzzeaa et al. 2020 employs the logits-based distillation loss for memory samples, instead of the cross entropy loss. The results in Table 1 show

---

Table 2: Accuracy of variants of RAR and different hyperparameter tuning methods.
SEQ-CIFARI00
SEQ-MINI-NAGENET
CORE50-NC CLRS25-NC
19.0 0.0
20.0 208
24.0 =.
RAR
25.4
27.4
38.6
28.8
RAR
21.6
24.5 +.1.1
35.7 L1
29.1
RAR worn
27.8 0.0
30.0 6:00
39.3 #
28.6 #:7
RAR-HTOCL
23.4 0.2
26.0 0.0
40.8 +.0.
26.9 0.5.
RAR-RL
29.6 0.4
32.1
44.4 0.8.8
35.0 0.7
N1992 2
wikug? 2
W0.00g
wing
Wasg2
(a) CIFARIO0
(b) MINI-MAGEEET
(c) CORESO
(d) CLRS25
Figure 4: Effects of augmentation and rehearsal iterations (red stars: accuracy in Table Tb.
RAR leads to large performance gains for ER-rw (for the best a choice; further results can be found in Appendix D.2) and DER for all four datasets (ER-rw:-9.4%% ~ +33.0%. DER:+111 11.9% ~ +32.8%) This suggests that even with more advanced rehearsal loss designs, repeated augmented rehearsal important for online rehearsal.
Stability and Plasticity Trade-off Based the definition of accuracy Ar, forgetting Fr and 2 we find that these three metrics have the following relationship:
backward transfer B7 in Section
Ar
T 01,1
T
1
Br
1
T a,
T 1
T Fr.
T
Plasticily
Stubility
Interestingly, this finding shows that accuracy is related to the ability to learn new tasks, quantified by: + E (i and the ability avoid forgetting past tasks, quantified by T Br, which draws connection to the more general problem of the cability-plasticiie trade-off in neural networks and continual learning [Grossberg] 2012 Delange al., 2021]. Plasticity refers to the ability to integrate new knowledge and stability refers to the ability to retain old knowledge. Fig presents the stability and plasticity trade-off in RAR. Generally, we observe increasing the repeated rehearsal iterations (K) leads to a higher level of plasticity. However, this may also cause decrease of stability, i.e., introduce forgetting. On the other hand, the of augmentation generally improves stability. More importantly, the use of augmentation repeated rehearsal shifts the plasttiityy trade-off curve towards the upper right, thus creating a better stability-plasticity trade-off frontier.
Hyperparameter Tuning for RAR We compare the RL-based hyperparameter tuning method with the hyperparameter tuning framework for continual learning (HTOCL) method used in Chaudhry al. [2018] Mai et al. [2022) (see Section 2 The results in Table 2 show that RL-based RAR significantly outperforms using HTOCL to select hyperparameters for RAR. One reason is that the extent memory overfitting varies
different training stages of CL. HTOCL only uses the first few tasks select hyperparameters for RAR, which may not optimal for later stages of CL. In fact, we observe HTOCL tends to select a large number of repeats and small augmentation strength. This selection strategy may be desirable for problems with short task sequences but problematic for long task sequences with increased memory overfitting risk. contrast, the RL-based method can take into account the latest feedback (e.g., train memory accuracy) to adjust hyperparameter choices. The selected iteration numbers and augmentation are shown in Appendix D.4
6.3 Ablation Studies
Interplay between Repeated and Augmented Rehearsal We investigate the interaction of the number of replay iterations with the augmentation strength Fig For three out of four datasets, using augmentation alone without repeated rehearsal leads to even worse performances than rehearsal

---

without augmentation (see K = 1 in Fig 4. One explamation is the underfitting challenge of online CL. Training on augmented samples can make model underfitting even worse. The only exception is the CORE50 dataset, which inherently has high memory overfitting risk with À = 6 making it benefit more from augmentation. Similarly, employing repeated rehearsal alone (see the blue solid line in Fig + also harms performance in three out of four datasets, with the CLRS dataset as the only exception, which enjoys a low memory overfitting risk with À = 1.12. An important takeaway message is that repeated rehearsal or augmented rehearsal is not always helpful in OCL settings and whether they will benefit or harm the performance is dependent on the structure of the OCL problem at hand (e.g., the task data size and memory data size).
RAR's Robustness to Large Numbers of Repeats Although the performance curve flattens out around 10 iterations, there is no evident drop in performance even with 20 iterations. This result reinforces how RAR can help with the mnderfitting-vverittin more training in OCL without having to worry about the performance drop introduced by memory overfitting. In comparison, for repeated rehearsal without augmentation (solid lines), the accuracy starts to drop quickly on CIFARI00 and MinilmageNet when using more than two iterations.
dilemma: can support the use of
Augmenting the Memory vs. Augmenting the Incoming Data RAR applies augmentation both the memory batch and the incoming batch. We examine the effectiveness of the two separately. RAR applied (a) solely with memory augmentation (RAR- mem) and (b) solely with incoming data augmentation (RAR-inc). We find that RAR achieves the best performance compared to -em and RAR-inc. This shows that it is beneficial to apply augmentation to both the memory batch and the incoming batch. Interestingly, RAR-inc itself also achieves consistent performance improvements over RER, e-g.. gain of 7.7% in CORE50. As discussed in Section 3] adding augmentation on the incoming batch can strengthen the regularization effect of the incoming task and indirectly alleviate memory overfitting.
Table 2 presents the performance
RAR with MIR, ASER, SCR We also perform ablation studies to investigate RAR's strong perfor- mance with the ER variants (see Appendix D.I). Similar to ER, the performance gains for RAR-MIR and RAR-ASER come from the combination of repeated rehearsal and augmented rehearsal. However, for SCR, the repeated rehearsal itself also leads a consistent performance boost. One reason is that SCR already includes a strong augmentation procedure with four augmentation operations to construct the supervised contrastive loss used in this method. also works poorly without augmentation.
Augmentation for Offline Rehearsal Although this paper focuses on online CL., the (augmented) empirical risk analysis is also relevant offline CL (see Propositions 2 and 3 in Appendix (A). This
suggests the memory overfitting risk in offline rehearsal is related to the ratio of task-to-memory size X and can be alleviated by augmented rehearsal (see more empirical results in Table of Appendix (D6).
7 Discussion and Conclusion
Rehearsal- methods play central role in fighting catastrophic forgetting when learning from non-stationary data streams. Compared to offline rehearsal, online rehearsal has faced particular challenges in tackling complex CL datasets due to the single-pas--hrouugh data constraint. This work tries to analyze the internal workings of online rehearsal from a theoretical and conceptual perspective and identifies the fundamental challenge that it faces as the dilemma between overfitting locally and underfitting globally. To deal with this challenge, we propose a simple baseline: repeated and augmented rehearsal (RAR). Surprisingly, despite its simplicity, RAR achieves a large performance boost for set of different rehearsal-bar based methods. Additionally, we propose an RL-based method to tune the hyperparameters of RAR to balance the cuability-plastionn trade-off in an online manner. It achieves promising results compared to hyperparameter tuning based on validation data. This work is focused on continual learning for classification problems with image data. An interesting future research direction is look at other CL domains, e.g., text/audio inputs or RL. problems.
Acknowledgments and Disclosure of Funding
This research is funded by the New Zealand MBIE TAIAO data science programme.
10

---

References
R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:1 32:11849-1188 1860, 2019.
J. Bang, H. Kim, Y. Yoo, J.-W. Ha, and J. Choi. Rainbow memory: Continual learning with memory of diverse samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218-8227, 2021.
Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara. Dark experience for general continual learning: strong, simple bascline. Advances in neural information processing systems, 33:15920- 15930, 2020.
H. Cha, J. Lee, and Shin. Co2L: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9516-9525, 2021.
A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: /0486, 2019.
S. Chen, E. Dobriban, and J. H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21 21(245):1-71, 2020.
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
M. Delange, R. Aljundi. M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
T. Grossberg. Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
H. Li, Jiang, X. Gu, J. Peng. Li, L.. Hong, and C. Tao. CLRS: Continual learning benchmark for remote sensing image scene classification. Sensors, 20(4):1226, 2020.
7. Li and D. Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, (0(12):2935-29) 2017.
V. Lomonaco and D. Maltoni. CORe50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17-26. PMLR, 2017.
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Z. Mai, R. Li, H. Kim, and S. Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3589-3599, 2021.
Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
Mirzadeh, M. Farajtabar, D. Gorur, R. Pascanu, and H. Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference Learning Representations, 2020.
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lamperr. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.

---

D. Shim, Z. Mai, J. Jeong, S. Sanner, H. Kim, and J. Jang. Online class-incrent incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9630-9638, 2021.
E. Verwimp, M. De Lange, and T. Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9385-9392 2021.
O. Vinyals, C. Blundell, T. Lillierap, and D. Wierstra. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
S. Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):77-7, 1985.
Y. Wu, Y. Chen, L. Wang. Y. Ye, Z. Liu, Y. Guo, and Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 374-382, 2019.
Y. Zhang and W.-B. Goh. Bootstrapped policy gradient for difficulty adaptation intelligent tutoring systems. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 711-719,2011
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
12

---

(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "jSZN2HSKJk", "reviewer": "Reviewer_P1CH", "summary": "This paper presents an online Experience replay (ER) based continual learning technique to mitigate forgetting of the past data. The paper considers a repeated rehearsal setting that allows training for multiple steps on each batch of data (previously proposed by Aljuni et al. 2019). In this repeated rehearsal setting, the paper studies the source of replay-memory overfitting. \n\nThe paper proposes a formulation of ER loss function that captures not only the dependence on the number of samples from the current and past tasks (shown in previous work), but also accounts for the ratio of incoming data size and the replay-memory size (novel contribution claim). When the relative size of replay-memory is smaller as compared to the size of the incoming task, memory overfitting can occur. As the number of previously-seen tasks increases, the memory overfitting worsens. \n\nFurther, the paper analyzes the training process for any given task and observes that repeated rehearsal leads to a quick drop in the training loss of the incoming data. Therefore, during the later iterations of training, the memory loss dominates the incoming data loss, and reduces the regularization effect of incoming data. The authors describe this phenomenon as an underfitting-overfitting dilemma – where repeated rehearsal can reduce underfitting of incoming data but at the same time can cause overfitting of the replay-memory data. \n\nIn order to solve the over-fitting problem, the authors propose data-augmentation of both the incoming data and the replay-data during mini-batch training. A RL-based HPO scheme is employed during training to tune the two key hyper-parameters – number of repeated rehearsals and strength of augmentation. \n\nThe proposed method is evaluated on a suite of computer-vision tasks and is shown to outperform ER baselines. \n", "questions": "1. The training loss for the incoming batch in Figure 2 (a) goes down very quickly. Why do you not observe overfitting on the incoming training batch during repeated rehearsal (say k=10)?\n2. It is not clear to me what is the reward function used for RL-based HPO. \n3. What is the compute overhead of RL-based HPO?\n", "limitations": "The authors mention some limitations of RL-based HPO but do not specify the the compute overhead. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper analyzes the replay-memory overfitting problem in detail. \n2. The paper combines existing techniques of repeated rehearsal with data-augmentation to alleviate overfitting on replay-memory data.\n3. The paper performs ablation studies to understand the effect of repeated rehearsal and augmentation. \n4. Use of RL-based HPO during online learning is an interesting direction.  \n5. The paper is mostly well-written. \n\nWeaknesses:\n1. The paper does not cite and compare against Dark ER [Buzzega et al. 2020]. This prior work also employs data-augmentation and is also applicable to the online learning setting.  \n2. Previous ER formulations often use an additional hyper-parameter that balances the loss between incoming data and the new data. Not sure why this hyper-parameter is missing in the problem formulation. \n3. I could not find any example of the type of augmentations used in the main paper. \n4. Writing improvements:\n   (a) It is difficult to follow Figure 1 due to smaller font and lack of enough details in the figure caption.\n   (b) Typo in Line 213. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4ise5vydxUz", "reviewer": "Reviewer_EmdQ", "summary": "The paper studies the effect of data augmentation in online continual learning. It proposes *repeated augmented rehearsal*, a simple method that adds data augmentation when using multiple iterations on the same batch in online continual learning.", "questions": "1. Did you consider the (partial) remedy of reweighting the memory loss term in Eq. (2)? If yes, why did you not include this option in your method? If no, do you see any reasons not to use this simple (partial) remedy in conjunction with data augmentation?\n\n2. Why did you choose to focus exclusively on the online CL setting? Do you think it is of great practical relevance? Would you agree that some of the main questions the paper is raising are also relevant in the offline CL setting?", "limitations": "I think the limitations of the paper could be discussed more honestly. Some of the consideration around the \"regularization effect\" of incoming data are mostly anecdotal (e.g., Figures 1 and 2) but that is not clearly stated. Also, data augmentation seems to be limited to image data, which could have been mentioned in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. I consider the simplicity of the proposed approach a major strength. Continual learning research suffers from a flood of overly complex methods without proper ablation studies. Revisiting the seemingly simple baselines, with proper ablation studies, is an important contribution!\n\n2. The empirical evaluation is of high quality, includes most relevant baselines and competing methods, and provides a number of ablation studies. (A slight caveat is that the same architecture, ResNet18, is used on all datasets.)\n\n3. The results of the proposed method in the online continual learning setting are very convincing. RAR provides significant performance boosts across the board and can be combined with any rehearsal-based CL method.\n\n\n\n### Weaknesses\n\n1. I think this paper has one major flaw. It assumes the ER formulation in Eq. (2), which weights loss contributions from memory and incoming batch equally. One can easily upweight or downweight the memory by reweighting the first gradient term in Eq. (2), which controls the memory overfitting issue and (if done properly) can also remove the undesirable dependence on the task size. Therefore, in some sense the paper highlights a problem of vanilla ER, but neglects the most straight-forward (partial) remedy. Since the goal of the paper is to revisit a simple baseline, this constitutes a major shortcoming in my opinion.\n\n2. I find it somewhat regrettable that the authors chose to exclusively focus on the *online* continual learning setting, which seems of limited practical relevance to me. (One can always buffer data from a stream into mega-batches and run an \"offline\" continual learning algorithm on the resulting \"pseudo-tasks\".) The key questions the paper studies, the effect of data augmentation in rehearsal-based continual learning, is just as relevant in the \"offline\" CL setting.\n\n3. Finally, I want to register that the resulting method is of limited novelty, simply combining repeated ER with data augmentation. (Personally, I don't think that is a big caveat, but should be mentioned.) I would also add that augmentation can only ever be a partial remedy. It can ameliorate the memory overfitting issue and is certainly useful in practice. But, in some sense, it will never tackle the fundamental problem.\n\n4. Minor Comments\n\n    - The notation in Eq. (2) is sloppy, since the examples are indexed with $i$, but this is not the summation index of the sum. Why not just $\\sum_{x, y \\in \\mathcal{B}_t}$? This would also avoid the currently overloaded notation for data points from memory and incoming batch.\n    - Calling Theorem 1 a Theorem is a bit much in my opinion. I would go with a more modest \"Proposition\", but I guess that is a question of personal taste.\n\n### Rating\n\nI explicitly commend the effort to revisit simple, principled baselines. The paper is well-written and the empirical evaluation is sound. Unfortunately, I have to recommend rejection for this paper. I think omitting the option to reweight the memory loss (Weakness 1) is a fundamental shortcoming when trying to study the issue of memory overfitting in continual learning.\n\n### Update after rebuttal\n\nThe authors' rebuttal has resolved my main concern regarding the reweighted ER baseline. I have increased my score accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3vdZthOYWoJ", "reviewer": "Reviewer_enAz", "summary": "The paper addresses a rehearsal-based approach for online continual learning. It gives theoretical insights on overfitting and unfitting problem in rehearsal-based approaches. Inspired by the investigation, the paper proposed a simple, novel method called repeated augmented rehearsal. ", "questions": "When the RL agent is removed, what is recommended to keep a good performance?  ", "limitations": "No experiments on large-scale datasets such as ImageNet-1K. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- strong theoretical analysis of rehearsal-based methods from the perspective of empirical risk minimization. \n- the experimental results support the effectiveness of the suggested method. \n\nWeaknesses: \n- No experiments on large-scale datasets such as ImageNet-1K. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3Ne1E4qwVW", "reviewer": "Reviewer_ga23", "summary": "The authors first point out an underfitting vs overfitting dilemma in online continual learning (OCL) with experience replay (ER) on the data distribution and on the samples in the replay buffer, respectively.\n\nThey show that the amount of iteration on an incoming batch in Repeated ER helps with the underfitting of the data at the expense of more overfitting on the buffer. This is both proved theoretically and empirically.\n\nThe authors propose to combine data augmentation with repeated ER, coined repeated augmented rehearsal (RAR). The motivation for the data augmentation is that it should reduce the buffer's overfitting.\n\nThe authors also point out the open problem of Hyperparameter Tuning in OCL. Accordingly, they propose an RL-based hyperparameter tuning algorithm that adapts the hyperparameter in an online manner.\n\nThe method is then empirically validated against relevant ER baselines on relevant benchmarks. The paper includes useful ablations and analyses of the results.\n\n", "questions": "None for now", "limitations": "The main limitation I see is that the method requires prior knowledge of data augmentation techniques, which makes it less general than other OCL methods.\nCould someone w/o prior knowledge in e.g. audio, RL, use the same RandAugmentatiomn technique? If not, a general protocol to tackle different modalities would be useful.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Theorem 1 is helpful in showcasing how different OCL problem characteristics will impact the underfitting/overfitting dilemma, specifically the data to buffer size ratio.\n\nThe repeated rehearsal analysis is useful for the field. \nThe loss landscape analysis helps the reader to grasp the increasing overfitting problem as the number of iterations (k) increases. \nThe regularization effect of the incoming data is also something useful, and contrary to the aforementioned observation, I have never given this one previous thoughts.\n\nThe method (RAR) is simple and effective. \n\nI also appreciate that the others addressed the open problem of hyperparameter-tuning in OCL with an adaptive RL approach. Its adaptive feature could be particularly useful in OCL with very long task sequences.\n\nThe plasticity/stability was particularly interesting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept