Learn the Time to Learn: Replay Scheduling for Continual Learning
Authors: Marcus Klasson
Hedvig Kjellstrom
Cheng Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learning
Replay Memory
Task Incremental Learning

Abstract: Replay-based continual learning has shown to be successful in mitigating catastrophic forgetting. Most previous works focus on increasing the sample quality in the commonly small replay memory. However, in many real-world applications, replay memories would be limited by constraints on processing time rather than storage capacity as most organizations do store all historical data in the cloud. Inspired by human learning, we illustrate that scheduling over which tasks to revisit is critical to the final performance with finite memory resources. To this end, we propose to learn the time to learn for a continual learning system, in which we learn schedules over which tasks to replay at different times using Monte Carlo tree search. We perform extensive evaluation and show that our method can learn replay schedules that significantly improve final performance across all tasks than baselines without considering the scheduling. Furthermore, our method can be combined with any other memory selection methods leading to consistently improved performance. Our results indicate that the learned schedules are also consistent with human learning insights.

Under review as a conference paper at ICLR 2022
LEARN THE TIME TO LEARN: REPLAY SCHEDULING FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Replay-based continual learning has shown to be successful in mitigating catas- trophic forgetting. Most previous works focus on increasing the sample quality the commonly small replay memory. However, in many real-world applications, replay memories would be limited by constraints on processing time rather than storage capacity most organizations do store all historical data in the cloud. Inspired by human learning, we demonstrate that scheduling over which tasks revisit is critical to the final performance with finite memory resources. To this end, we propose to learn the time to learn for continual learning system, in which learn schedules over which tasks to replay different times using Monte Carlo tree search to illustrate this idea. We perform extensive evaluation and show that method can learn replay schedules that significantly improve final performance across all tasks than baselines without considering the schedul- ing. Furthermore, our method can be combined with any other memory selection methods leading to consistently improved performance. Our results indicate that the learned schedules are also consistent with human learning insights.
INTRODUCTION
Many organizations deploying machine learning systems receive large volumes of data daily where these new data are often associated with new tasks. Although all historical data are stored in the cloud in practice, retraining machine learning systems on daily basis is prohibitive both in time and cost. In this setting, the systems must continuously adapt to new tasks without forgetting the previously learned abilities. Continual learning methods (De Lange et al., 2019; McCloskey & Cohen, 1989 Parisi et al., 2019) address this challenge where, in particular, replay-based meth- ods (Chaudhry et al., 2019; Hayes et al., 2020) have shown to be very effective in achieving great prediction performance and retaining knowledge of old tasks. Replay-based methods mitigate catas- trophic forgetting by revisiting a small set of samples, which is feasible to process compared to the size of the historical data. In traditional continual learning literature, the replay memory is limited due to the assumption that historical data are not available. In the real-world setting where historical data are in fact always available, the requirement of small memory remains due to processing time and cost issues.
Most research on replay-based continual learning has been focused on the sample quality in the memory (Aljundi et al., 2019; Borsos et al., 2020; Chaudhry et al., 2019; Chrysakis & Moens, 2020; Nguyen et al., 2017; Rebuffi al., 2017: Yoon al., 2021) or data compression increase the memory capacity (Hayes et al., 2020; Iscen et al., 2020; Pellegrini et al., 2019). Common for these methods is that the memory allocates an equal amount of space for storing samples from old tasks. When learning new tasks, the whole memory is replayed to mitigate catastrophic forgetting. How- ever, in life-long learning settings, this simple strategy would inefficient as the memory must store large number of tasks. Furthermore, these methods ignore the time to learn old tasks again which is important in human learning. Humans are continual learning systems, and different meth- ods have been developed to enhance memory retention, such as spaced repetition (Dempster, 1989; Ebbinghaus, 2013; Landauer & Bjork, 1977) which is used often in education. These education methods focus the scheduling of learning and rehearsal of previous learned knowledge.
In this work, we argue that finding the proper schedule what tasks to replay in the fixed memory setting is critical for continual learning. To demonstrate our claim, perform simple experiment

---

Under review as a conference paper at ICLR 2022
Task 1 - Task 2
- Task 3
- Task 4
- Task 5
Replay
ACC: 89.66%
ACC: 93.85%
ACC: 93.17%
ACC: 94.49%
Accuracy (%)
100
100
100
100
90
90
90
90
80
80
80
80
70
70
70
70
5
Task
Task
Task
Task
Figure 1: Task accuracies on Split MNIST (Zenke et al., 2017) when replaying only 10 samples of classes 0/1 a single time step. The black vertical line indicates when replay is used. ACC denotes the average accuracy over all tasks after learning Task 5. Results are averaged over seeds. These results show that the time to replay the previous task is critical for the final performance.
on the Split MNIST (Zenke et al., 2017) dataset where each task consists of learning the digits 0/1, 2/3, etc. arriving in sequence. The replay memory contains data from task 1 and can only be replayed at one point in time. Figure shows how the task performances progress over time when the memory is replayed at different time steps. In this example, the best final performance is achieved when the memory is used when learning task 5. Note that choosing different time points replay the same memory leads to noticeably different results in the final performance. These results indicate that scheduling the time when to apply replay can influence the final performance significantly of continual learning system.
To this end, we propose learning the time to learn, in which we learn replay schedules of which tasks to replay at different times inspired from human learning (Dempster. 1989). We illustrate the ad- vantages with replay scheduling by using Monte Carlo tree search (MCTS) (Coulom, 2006) to learn policies for replay. More specifically, we train a neural network on the current task dataset mixed with the scheduled replay samples and measure the final performance of the network to evaluate the replay schedules selected by MCTS. summary, our contributions are:
We demonstrate the importance of replay scheduling in continual learning and propose to learn the time to learn which tasks to replay (Section 3.1 and 3.2).
We use MCTS an example method to illustrate how replay schedules for continual learn- ing can be learned by establishing a finite set of memory compositions that can be replayed at every task (Section 3.3).
We demonstrate with several benchmark datasets that learned scheduling can improve the continual learning performance significantly in the fixed size memory setting (Section 4.1 and 4.4). Furthermore, we show that our method can be combined with any other memory selection methods (Section 4.3), as well as being efficient in situations where the memory size is even smaller than the number of classes (Section 4.5).
2 RELATED WORK
In this section, we give brief overview of continual learning methods, essentially replay-based methods, as well as spaced repetition techniques for human continual learning.
Continual Learning. Traditional continual learning can be divided into three main areas, namely regularization-basse architecture-based, and replay-based approaches. Regularization-basel meth- ods aim to mitigate catastrophic forgetting by protecting parameters influencing the predictive per- formance from wide changes and use the rest of the parameters for learning the new tasks (Adel al., 2019: Chaudhry al., 2018a; Kirkpatrick et al., 2017; Li & Hoiem, 2017: Nguyen et al., 2017: Rannen et al., 2017; Schwarz et al., 2018: Zenke et al., 2017). Architecture- based methods isolate task- task-specific parameters by either increasing network capacity (Rusu al., 2016; Yoon et al., 2019; 2017) or freezing parts of the network (Mallya & Lazebnik, 2018; Serra et al., 2018) to main- tain good performance on previous tasks. Replay-based methods mix samples from old tasks with the current dataset to mitigate catastrophic forgetting, where the replay samples are either stored in an external memory (Chaudhry et al., 2019; Hayes et al., 2020; Isele & Cosgun, 2018; Lopez- Paz & Ranzato, 2017) or generated using generative model (Shin al., 2017; van de Ven & Tolias, 2018). Regularization-ba based approaches and dynamic architectures have been combined with replay-based approaches methods to overcome their limitations (Chaudhry al., 201 Sa;b; Douillard et al.,
2

---

Under review as a conference paper at ICLR 2022
2020: Ebrahimi et al., 2020; Joseph & Balasubramanian, 2020; Mirzadeh et al., 2020; Nguyen al., 2017: Pan et al., 2020; Pellegrini et al., 2019; Rolnick et al., 2018; von Oswald et al., 2019). Our work relates most to replay- based- methods with external memory which we spend more time describing in the next paragraph.
Replay-based Continual Learning. A commonly used memory selection strategy of replay sam- ples is random selection. Much research effort has focused on selecting higher quality samples to store in memory (Aljundi et al., 2019; Borsos et al., 2020; Chaudhry et al., 2019; Chrysakis & Moens, 2020; Hayes et al., 2019; Isele Cosgun, 2018; 8: Lopez-Paz Ranzato, 2017: Nguyen et al., 2017; Rebuffi et al., 2017; Yoon et al., 2021). Chaudhry et al. (2019) reviews several selection strate- gies in scenarios with tiny memory capacity, e.g., reservoir sampling (Vitter, 1985), first-in first-out buffer (Lopez-Paz & Ranzato, 2017), k-Means, and Mean-of-Features (Rebuffi et al., 2017). How- ever, more elaborate selection strategies have been shown to give little benefit over random selection for image classification problems (Chaudhry et al., 2018a; Hayes et al., 2020). More recently, there has been work on compressing raw images to feature representations to increase the number of memory examples for replay (Hayes et al., 2020; Iscen al., 2020; Pellegrini et al., 2019). Our approach differs from the above mentioned works since we focus on learning to select which tasks to replay at the current task rather than improving memory selection or compression quality of the samples in the memory. Replay scheduling can however be combined with any selection strategy as well as storing feature representations.
Human Continual Learning. Humans are continual learning systems in the sense of learning tasks and concepts sequentially. Furthermore, humans have impressive ability to memorize ex- periences but can forget learned knowledge gradually rather than catastrophically (French, 1999). Different learning techniques have been suggested for humans to memorize better (Dunlosky al., 2013; Willis, 2007). An example is spaced repetition which gradually increases time-intervals be- tween rehearsals for retaining long-term memory (Dempster, 1989). This technique has been studied frequently and was inspired from the works of Ebbinghaus (2013) on memory retention. For exam- ple, Landauer & Bjork (1977) demonstrated that memory training schedules using adjusted spaced repetition were better at preserving memory than uniformly spaced training. Hawley et al. (2008) studies the efficacy of spaced repetition on adults with probable Alzheimer's discase for learning face-name association. Several works in continual learning with neural networks are inspired by or have a connection to human learning techniques, including spaced repetition (Amiri, 2019; Amiri et al., 2017; Feng et al., 2019; Smolen et al., 2016), mechanisms of sleep (Ball et al., 2020; Mallya & Lazebnik, 2018; Schwarz et al., 2018), and reactivation of memories (Hayes et al., 2020; van de Ven et al., 2020). Our replay scheduling method is inspired by spaced repetition; we learn schedules of which memory samples to use for replay at different time steps.
3 METHOD
In this section, we describe method for learning replay schedules for continual learning. The idea is to learn schedules of which memory examples the network should rehearse at different times. We use Monte Carlo tree search (MCTS) (Browne et al., 2012; Coulom, 2006) to learn a scheduling policy by encouraging searches for promising replay schedules based on the classification accuracy.
3.1 PROBLEM SETTING
We focus on the setting considering the real-world continual learning needs where all historical data are available but are prohibitively large. Therefore. only a small amount of historical data can be used when adapting the model to new data due to processing capability consideration. Thus, the goal to learn how to select subsets of historical data to efficiently mitigate catastrophic forgetting when learning new tasks. We refer to these subsets of historical data the replay memory throughout the paper, where the size of the replay memory affects the processing time when learning a new task t. Moreover, focus composing the replay memory based on the seen tasks in the historical data rather than single stored instances.
Next, introduce the notation of our problem setting which resembles the traditional continual learning setting for image classification. We let a neural network fo- parameterized by 0, learn T tasks sequentially given their corresponding task datasets Di,..., Dr arriving in order. The 1-th
3

---

Under review as a conference paper at ICLR 2022
) N consists of N, samples where at and 3/2 are the i-th data point and class label respectively. The training objective at task is given by
dataset D. - [((r 3/
(6) (4)
(6)
min ((So(zl )..))
(1)
where E(-) is the loss function, e.g., cross-entropy loss in our case. Since Dr is only accessible at time step the network fo is at risk of catastrophically forgetting the previous - 1 tasks when learning the current task. Replay-based continual learning methods mitigate the forgetting of old tasks by storing old examples in an external replay memory, that mixed with the current task dataset during training. Next, we describe our method for constructing this replay memory.
We assume that historical data from old tasks accessible at any time step l. However, since the historical data is prohibitively large, we can only fill small replay memory M with M historical samples for replay due to processing time constraints. The challenge is how to fill the replay memory with M samples that efficiently retain the knowledge of old tasks when learning new tasks. We focus on selecting the samples on task-level by deciding on the task proportion (01,...: 0t-1) of samples to fetch from each task, where a, 2 is the proportion of M examples from task i and Σi= a 1. Consequently, we need method for choosing these task proportions of which old tasks replay. To simplify this selection, we construct a discrete set of choices for possible task proportions telling how many samples from each task to use when constructing the replay memory M.
3.2 REPLAY SCHEDULING FOR CONTINUAL LEARNING
In this section, we describe our replay scheduling method for selecting the replay memory at different time steps. We define a replay schedule as a sequence S = (a)..... ar-1). 1). where a; = (a)....,TT-.....
for S S T 1, is the sequence of task proportions used for deter- mining how many samples per task to fill the replay memory with at task i. To make the selection of task proportions tractable, we construct an action space with a discrete number of choices for the task proportions from old tasks. We use the following ways to construct this action space: At the time to learn task 1, we have - historical tasks that we can choose from. We create t - 1 bins and choose a task index to sample for each bin by 1....t 1. We treat the
b. = 1b,,b2,.... bins as interchangeable and only keep the unique choices. For example, at task 3, we have seen task I and 2; so the unique choices of vectors are [1, 1), [1, 2), [2, 2). where [1, 1] indicates that all memory samples are from task 1, [1, indicates that half memory is from task I and the other half are from task etc. The task proportion are then computed by counting the number of occurrences of each task index in b, and dividing by the t - 1, such that a, = bincount (bz)/(t 1). If the memory size M is not divisible by the task proportion value for task i, we round the number of replay samples from task i, i.e., d; M, up or down accordingly while keeping the memory size fixed. From this specification, we can build a tree of different replay schedules to evaluate with the network.
Figure 2 shows an example of such replay schedule tree with Split MNIST (Zekke et al., 2017) where the memory size has been set to M = 8. The figure shows the current tasks with an example image of the task classes the left, and the right side shows examples of possible replay memories that can be evaluated. The memory starts as the empty set, i.c. M1 = 0. at task 1. Before learning task 2. M2 is filled with M task 1 examples since this is the only task seen so far. At task 3, the memory compositions we can choose from are M examples from either task 1 or 2, as well as equally filling M3 with four examples each from both tasks. A replay schedule is represented path traversal of different replay memory compositions from task to task 5, We have color-coded three examples of possible schedules in Figure use for illustration: the blue path represents replay schedule where only task 1 examples are replayed at all future tasks. The red path represents using an equally distributed amount of memory samples per task in the memory, and the purple path represents a schedule where the memory is filled M examples from the previously visited task. Note that all other possible paths in the tree are also valid replay schedules.
3.3 MONTE CARLO TREE SEARCH FOR REPLAY SCHEDULES
Our proposed setup to find replay schedules is to discretize the number of possible task proportions per task. The action space of task proportions then turn into a tree shape where each node represents
4

---

by ignoring the Under review as a conference paper at ICLR 2022
Task
O
Ø
Task
5069
a 3
Task 3 H
GONN
Task
6
Task
i 9
Figure 2: An exemplar tree of replay memory compositions from the proposed discretization method described in Scction 3.2 for Split MNIST. The replay memories from replay schedule found by traversing from task 1-5 through the tree the right hand side. The replay memory compositions have been structured accord- ing to the task where they can used replay. Note that replay memory task is the empty set, i.e. M = 6. Example images for each task are shown on the left.
a task proportion that corresponds to a specific composition of the replay memory M, as can be seen in Figure 2. To find the efficient replay schedule S in mitigating catastrophic forgetting, can perform an exhaustive search (e.g., breadth-first search) where all possible replay schedules evaluated. However, since the grows fast with the number of tasks, we need scalable method that enables tree searches in large search spaces. To this end, we propose to use Monte Carlo tree search (MCTS) (Browne et al., 2012; Coulom, 2006) since MCTS has been successful in applications with similar conditions (Chaudhry & Lee, 201 8; Gelly et al., 2006; Silver et al., 2016). In our setting, MCTS can help us to concentrate the search directions towards promising replay schedules in terms of classification performance and thus learn the time to learn.
Before outlining the steps for performing MCTS, we describe the notation for the tree search. The tree has T' levels where level / corresponds to task t. Each tree level & contains set of nodes V, { Ke where K, is the number of nodes at level l. Every node vg has a corresponding sequence of task proportions af. which can be used to retrieve the replay memory from the his- torical data. Referring back to Figure 2, the replay memory composition on the root node ol is the empty set M = 0, since the historical data is empty. At task 2, the only task proportion is a = (a1, 022 ..., -1)
(1.0, 0.0, ...: 0.0), so the replay memory at node vi is only filled with samples from task 1. In the rest of the paper, we denote node at task superscript i to avoid clutter in the notation.
At every visited node Us, store the corresponding task proportions a; in the replay schedule S. The final replay schedule then used for constructing the replay memory at each time step during the continual learning training. Next, briefly outline the steps for performing MCTS in the search for replay schedules.
Selection. An MCTS iteration begins by selecting the next node to visit from the root node. If the current node v, has visited all its children during the previous iterations, the next node is selected by evaluating the Upper Confidence Tree (UCT) (Kocsis & Szepesvári, 2006) function for all children. We use the following UCT function proposed by Chaudhry & Lee (2018) evaluate the score for moving from node to its child u++1:
2 llg((N(v))
UCT(V, v441) = max (Q(v++1)) +C
N(1++))
5

---

Under review as a conference paper at ICLR 2022
The reward function Q(-) contains the rewards for previous searches that has passed through the child 2/+1- The exploration constant C' 2 0 determines the degree of exploration of less visited replay schedules based on the number of visits (1, and (e+++) ) to the corresponding nodes in the tree. The child node to visit next is the one with the highest UCT score.
Expansion. Whenever the current node Ut has unvisited child nodes, the search tree is expanded with one of the unvisited child nodes 1r+1 selected with uniform sampling.
Simulation and Reward. After the expansion step, the search proceeds by selecting the succeed- ing nodes uniformly at random until a terminal node ur reached. The replay schedule S is then collected by appending the task proportions from the visited nodes during the iteration. We train the network with the replay memories retrieved from and evaluate the average accuracy over all
tasks after learning the final task T as the reward r, i.e., r = 7 E (ral)) where (eat) is the validation accuracy of task after learning task T.
Backpropagation. The reward r is backpropagated from the expanded node it up to the root node v1, where the number of visits N-) and reward function Q(-) for each node is updated.
We provide pseudocode in Algorithm 1 in Appendix B. Finally, we emphasize that we use MCTS to illustrate in simple setting the importance of learning the time to learn old tasks again.
4 EXPERIMENTS
We evaluated our replay scheduling method empirically on six common benchmark datasets for continual learning. We show that scheduling of the replay memory improves significantly over replaying equal proportions across the tasks. Furthermore, we demonstrate that replay scheduling can be as efficient as replaying all available memory samples in settings where only I example per class can be stored for replay. We denote our method Replay Scheduling MCTS (RS-MCTS) and select the result on the held-out test sets from the replay schedule that yielded the best reward on the validation set during the search. We perform experiments using 5 different seeds on all datasets. Our code is available as part of the supplementary material. The full details on experimental settings are in Appendix A and additional results are in Appendix D.
NEW (dhXe)
Datasets. We conduct experiments on six datasets commonly used as benchmarks in the continual learning literature: Split MNIST (LeCun et al., 1998; Zenke et al., 2017), Fashion-MNIST (Xiao et al., 2017), Split notMNIST (Bulatov, 2011), Permuted MNIST (Goodfellow et al., 2013), Split CIFAR-100 (Krizhevsky & Hinton, 2009), and Split minilmagenet (Vinyals et al., 2016). We ran- domly sample 15% of the training data from each task to use for validation when computing the reward for the MCTS simulations.
NEW (8z4B)
Baselines. We compare RS-MCTS to using 1) random replay schedules (Random), 2) equal task schedules (ETS), and 3) a heuristic scheduling method (Heuristic). The ETS baseline uses equal task proportions, such that M/(1- - 1) samples per task are replayed during learning of task I, and use both training and validations sets for training such that they use the same amount of data as RS- MCTS. The Hcuristic bascline replays the old tasks which accuracy on the validation set is below a certain threshold proportional to the best achieved validation accuracy the task. Here, the replay memory is filled with M/k where is the number of tasks that need to be replayed according to their decrease in validation accuracy. See Appendix C for more details the Heuristic baseline.
Network Architectures. We use multi-head output layer for all datasets except for Permuted MNIST where the network uses single-head output layer. We use 2-layer MLP with 256 hid- den units for Split MNIST, Split FashionMNISTT. Split notMNIST, and Permuted MNIST. For Split CIFAR- 100, we use the CNN architecture used in Vinyals et al. (2016) and Schwarz et al. (2018). For Split mnilmagenet, we apply the reduced ResNet- 18 from Lopez-Paz & Ranzato (2017).
Evaluation Metric. We use the average test accuracy over all tasks after learning the final task, i.e., ACC = $ ET Ar, where Ar, is the test accuracy of task i after learning task T.
4.1 RESULTS ON REWARD FOR RS-MCTS
In the first experiments, we show that the replay schedules from RS-MCTS yield better performance than replaying an equal amount of samples per task. The replay memory size is fixed to M 10 for Split MNIST, FashionMNIST. and notMNIST, and M 100 for Permuted MNIST, Split CIFAR- 100, and Split minilmagenet. Uniform sampling is used the memory selection method for all

---

Under review as a conference paper at ICLR 2022
S S-MNIST
S FashiooMNIST
S nOMNNIST
P-MNIST
S. S-CIFAR 100
5-miilmmgenet
Random
99
95
98
75
Heuristic
ACC
97
Ours
96
71
50
100
100
50 Iteration
Iteration
Iteration
Iteration
Iteration
Iteration
Figure 3: Average test accuracies over tasks after learning the final task (ACC) over the MCTS simulations for all datasets, where 'S' and 'P' used short for 'Split' and 'Permuted'. We compare performance for RS-MCTS (Ours) against random replay schedules (Random). Equal Task Schedule (ETS), and Heuristic Scheduling (Hcuristic) baselines. For the first three datasets, we show the best ACC found from a breadth first search (BFS) as an upper bound. All results have been averaged over 5 seeds. These results show that replay scheduling improve significantly over ETS and outperform perform par with Heuristic across different datasets and network architectures.
methods this experiment. For the 5-task datasets, we provide the optimal replay schedule found from breadth-first search (BFS) over all possible replay schedules in our action space as upper bound for RS-MCTS. As the search space grows fast with the number tasks, it will even with only 5 continual learning tasks (which corresponds to a tree with depth of 4) yield tree with 1050 leaf nodes. Thus, BFS becomes computationally infeasible when we have 10 or more tasks.
NEW (8z4B)
Figure 3 shows the progress of ACC over the MCTS iterations by RS-MCTS for all datasets. We also show the best ACC metrics for ETS, Heuristic, and BFS as straight lines. We observe that RS-MCTS outperforms Random and ETS successively with more iterations. Furthermore, RS- MCTS approaches the upper limit from BFS on the 5-task datasets. For Permuted MNIST and Split CIFAR-100, the Heuristic baseline and RS-MCTS perform on par after 50 iterations. This shows that Heuristic with careful tuning of the validation accuracy threshold can be a strong baseline when comparing against replay scheduling. The top row of Table shows the ACC for each method for this experiment. We note that RS-MCTS outperforms ETS significantly for most datasets and performs on par with Heuristic.
4.2 REPLAY SCHEDULE VISUALIZATIONS
We visualize a learned replay schedule from Split CIFAR-100 with memory size M 100 to gain insights into the behavior of the scheduling policy from RS-MCTS. Figure 4 shows a bubble plot of the task proportions that are used for filling the play memory at every task. Each color of the cir- cles corresponds to historical task and its size represents the proportion of examples that are re- played the current task. The sum of all points from all circles at each column is fixed at the dif- ferent time steps since the memory size M is fixed. The task proportions vary dynamically over time in a sophisticated nonlinear way which would be hard to replace by heuristic method. Moreover, we can observe space repetition style scheduling in many tasks, e.g., task 1-3 are replayed with similar pro- portion at the initial tasks but eventually starts varying the time interval between replay. Also, task and 6 need less replay in their carly stages, which could potentially be that they are simpler corre- lated with other tasks. We provide a similar visualization for Split MNIST Figure 7 in Appendix D.1 to bring more insights the benefits of replay scheduling.
19
Replayed Task
2 3
4 5
8 9
10
12 15 17 18 19
Curreni Task
Figure 4: Visualization using bubble plot of play schedule learned from Split CIFAR-100. The task proportions vary dynamically over time which would be hard to replace by a heuristic method.
NEW (kfup)
4.3 ALTERNATIVE MEMORY SELECTION METHODS
In this section, we show that our method can be combined with any memory selection method for storing replay samples. In addition to uniform sampling, apply various memory selection methods commonly used in the continual learning literature, namely k-means clustering, k-center clustering (Nguyen al., 2017), and Mean-of- (MoF) (Rebuffi al., 2017). We compare
7

---

Under review as a conference paper at ICLR 2022
Split MINIST
Split FushionMNIST
Split noMMNIST
Random
ACC (18)
98
ETS
Heuristic
96
92
Ours
95
8
24
80
120
200
400 800
24
30
120
200
400
800
24
30
120
200
400 800
Permuted MNIST
Split CIFAR-100
Split minilmagennet
ACC (%)
8.5
70
75
35
55
90
270 450
900 2250
955
2855
475
950
1900
955
265
4775
950
1900
Memory size AM
Memory size Ad
Memory size
Figure 5: Average test accuracies over tasks after learning the final task (ACC) sizes M for the RS-MCTS (Ours) and the Random, ETS, and Heuristic baselines on all datasets. results have been averaged over 5 seeds. The results show that replay scheduling can outperform replaying with random and equal task proportions. especially for small M, on both small and large datasets across different backbone choices. Furthermore, method requires less careful tuning than the Heuristic baseline M increases.
different replay memory
Table I: Accuracy comparison between RS-MCTS (Ours), Random choice scheduling (Random). Heuristic Scheduling (Heuristic), and Equal Task Schedulc (ETS) with various memory selection methods evaluated across all datasets. We use 'S' and 'P" as short for 'Split' and 'Permuted' for the datasets. The replay memory size is MM = 10 and M = 100 for the 5-task and 10/20-task datasets respectively. We report the mean and standard deviation averaged over 5 seeds. RS-MCTS performs better par than the baselines on most datasets and selection methods, where MoF yields the best performance general.
5-task Datasets
10- and 20-task Datasets
Selection
Method
S MNIST
PishionMNIST
S SosMNIST
P. M.NISST
C-FAR 100
S-minimugenee
Random
95.91 + 1.56
95.82 1.45
92.39 + 1.29
72.44 + 1.15
53.99 + 0,51
48.08 + 1.36
ETS
94.02 + 4,25
95.81 + 3,53
91.01 + 1.39
71.09
+ 2,31
47.70
+ 2.16
46.97 + 1.24
Uniform
Houristic
96.02
97.09
91.26
3.99
76.68
57.31
1.21
49.66
Ours
97.93 1: 0.56
98.27 1 0.17
94.64 1 0.39
76.34
1 0.98
56.60
+: 1.13
50.20 1: 0.72
Random
94.24 + 3.20
96.30 + 1.62
91.64 + 1.39
74.30 + 1.43
53.18 + 1.66
49.47 + 2.70
ETS
92.89 1: 3.53
96.47 + 0.85
93,80 +: 0.82
69.40
+: 1.32
47.51
1.14
45.82 1 0.92
& meaas
Heuristic
96.28 1: 1.68
95.78 1: 1.50
91.75 1: 0.94
75.57
1.18
54.31
3.94
49.25 +: 1.00
Oars
98.20 +: 06
98.48 + 0.26
93.61 + 0.71
77.74 + 0.80
56.95 +: 0.92
50.47 + 0.85
Random
96.40 1. 0.68
95.57 J: 3.16
92.61 1: 1.70
71.41 2.75
48.46
0,31
44.76 1. 0.96
ETS
94.84 +
97.28 0.50
91.08
2.48
69.11 + 1.69
44.13
1.06
41.35 +
Ascenter
Heuristic
94.55 1. 2.79
94.08 1. 3.72
92,06 1: 1.20
74.33 1: 2,00
50.32
1.97
44.13 1 0.95
Ours
98.24 +: 0.36
98.06 + 0.35
94.26 + 0.37
76.55 + 1.16
51.37 + 1.63
46.76 + 0.96
Random
95.18 1. 3.18
95.76 1: 1.41
91.33 1: 1.75
77.96 1: 1.84
61.93 1: 1.05
$4.50 1: 1.33
MoF
ETS
97.04 +
96.48
1.33
92.64
77.62
1.12
60.43
+
Houristic
96.46 + 2.41
95.84 + 0.89
93.24 + 0.77
77.27 = 1.45
2.770
$2.30 + 0.59
Ours
98.37 - 0.24
97.84
0.32
94.62 + 0.42
81.58 + 0.75
64.22 - 0.65
57.70 + 0.51
our method and ETS combined with these different selection methods. As in Section 1.1, we set the replay memory size M 10 for the 5-task datasets and M 100 for the 10- and 20-task datasets. Table 1 shows the results across all datasets. We note that using the replay schedule from RS-MCTS outperforms the baselines when using the alternative selection methods, where MoF performs the best on most datasets.
NEW (824B)
4.4 RESULTS WITH VARYING REPLAY MEMORY SIZE
We show that our method can improve the performance across different choices of memory size. In this experiment, we set the replay memory size to ensure the ETS baseline replays equal number of samples per class at the final task. The memory size is set to A = Tlept nape (T - 1), where Nepp is the number of classes per task in the dataset and Nape are the number of samples per class wish to replay at task T for the ETS baseline. In Figure 5, we observe that RS-MCTS yield better task accuracies than ETS, especially for small memory sizes. Both RS-MCTS and ETS perform better than Heuristic as M increases showing that Heuristic requires careful tuning of the validation accuracy threshold.
FIX (874B)
8

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracy comparison in the example per class memory setting evaluated across all datasets. We 'S" and 'P' as short for 'Split' and 'Permuted' for datasets. RS-MCTS has replay memory size and M = 50 for the 5-task and 10/20-task datasets respectively. The baselines replay all available memory samples. We report the mean and standard deviation averaged over 5 seeds. RS-MCTS performs on par with the best baselines on all datasets except S-CIFAR-100.
S-lask Datasets
10. and 20-task Datasets
Method
S-MNIST
S-FashionMNIST
S-notMNIST
P-MNIST
S-CIFAR-100
S-minilmagenet
Random
92,56 1: 2,90
92,70 - 3.78
89.53 1 3,96
70.02 1: 1.76
48.62
1.02
48.85 1: 1,38
A-GEM
94.97 1.50
94.81 0.86
92.27 + 1.16
64.71
1.78
42.22
2.13
32.06 +:
ER Ring
94.94 1,56
95.83 2.15
91.10 1 1.89
69.73
1.13
53,93
1,13
49.82 1: 1.69
Uniform
95.77 = 1.12
97.12 = 1.57
92.14 - 1.45
69.85
4 1.01
52.63
1.62
50.56 +: 1.07
RS MCTS (Ours)
96.07 + 1.60
97.17 = 0.78
93.41 +: 1.11
72,52 1 0.54
$1.50
1.19
50.70 1: 0.54
4.5 EFFICIENCY OF REPLAY SCHEDULING
Here, we illustrate the efficiency of replay scheduling with com- parisons to several common replay- based continual learning base- lines in an even more extreme memory setting. Our goal to investigate if scheduling over which tasks to replay be more efficient in situations where the memory size is even smaller than the number of classes. To this end,
Ours Baselines
#Samples
set the replay memory size for our method to M = 2 for the 5-task datasets, such that only 2 samples can be selected for replay at all times. For the 10- and 20-task datasets which have 100 classes, we set M = 50. We then compare against the most memory efficient continual learning baselines, namely A-GEM (Chaudhry et al., 2018b), ER- Ring (Chaudhry al., 2019) which show promising results with I sample per class for replay after learning each task, and with uniform memory selection as reference. Additionally, we compare to using random replay sched- We visualize the memory usage
Task
Figure 6: Number of replayed sam- ples per task for the 5-task datasets in our tiny memory setting. Our method uses fixed M = 2 sam- ples for replay, while the baselines increment their memory per task.
NEW (8z4B)
ules (Random) with the same memory setting for RS-MCTS when training on one the 5-task datasets for our method and the baselines in Figure 6. We also visualize the memory usages for the other benchmarks in the Appendix E.
Table 2 shows the ACC for each method across all datasets. Despite using significantly fewer sam- ples for replay, RS-MCTS performs better on par with the best baselines on all datasets except Split CIFAR-100. These results show that replay scheduling can be even more efficient than the state-of-the-ars memory efficient replay-based methods, which indicate that learning the time to learn is an important research direction in continual learning.
5 CONCLUSIONS
We propose learning the time learn, i.e., in real-world continual learning context, learning sched- ules of what previous tasks to replay at different times. To the best of our knowledge, we are the first consider the time learn in machine learning inspired by human learning techniques. We demonstrate with an example method that learned replay schedules produce significantly improved results under the same memory budget when comparing with the method without scheduling. Fur- thermore, the dynamic behavior of the learned schedules showed similarities to human learning techniques, such as spaced repetition, by rehearsing previous tasks with varying time intervals. Fi- nally, we showed that replay scheduling allows for utilizing the memory more efficiently compared to standard benchmarks replaying all memory samples in the tiny memory setting.
In future work, we would like to explore choosing memory samples on an instance level as the current work selects samples on task level. This would need a policy learning method, that scales to large action spaces which is a research challenge by itself. Also, our demonstrated method with MCTS can be inefficient, especially in continual learning settings, in terms of training time and since it needs to learn for each dataset and application separately. In future work, will incorporate reinforcement learning methods that generalize. We will therefore extend replay scheduling to learn general policies that can be directly applied to any new application and domain.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tameem Adel, Han Zhao, and Richard E Turrer. Continual learning with adaptive weights (claw). arXiv preprint arXiv:1911.000 11.094444, 2019.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. arXiv preprini ToXXvv:903.086777 2019..
Hadi Amiri Neural self-training through spaced repetition. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 21-31, 2019.
Hadi Amiri, Timothy Miller, and Guergana Savova. Repeat before forgetting; Spaced repetition for efficient and effective training of neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2401-2410, 2017.
Philip J Ball, Yingzhen Li, Angus Lamb, and Cheng Zhang. A study on efficiency in continual learning inspired by human learning. arXiv preprint urXiv22010.51187. 2020.
Zalán Borsos, Mojmír Mutný, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. arXiv preprint arXiv:2006.03875, 2020.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43, 2012.
Yaroslav Bulatov. The notMNIST dataset.
http::/yarosllavbbcoom wwwp://wwww..oooooooo.......
n
Arslan Chaudhry, Punect K Dokamia, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2a.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint urXiv: /812.00000 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 12:02.0440 10486, 2019.
Muhammad Umar Chaudhry and Jee-Hyong Lee. Feature selection for high dimensional data using monte carlo tree search. IEEE Access, 6:76036-76048, 2018.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In ICML, 2020.
Rémi Coulom. Efficient selectivity and backup operators monte-carlo tree search. International conference on computers and games, pp. 72-83. Springer, 2006.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forget- ting in classification tasks. arXiv preprint 1ixXiv::909.08 2(6), 2019.
Frank N Dempster. Spacing effects and their implications for theory and practice. Educational Psychology Review, 1(4):309-330 1989.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision ECCV 2020, pp. 86-102. Springer International Publishing, 2020.
John Dunlosky, Katherine A. Rawson, Elizabeth J. Marsh, Mitchell J. Nathan, and Daniel T. Will- ingham. Improving students' learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1): 4-58, 2013. ISSN 15291006, URL http: //ww:/ jstor. http:/saalle22222
10

---

Under review as a conference paper at ICLR 2022
Henmann Ebbinghaus. Memory: A contribution to experimental psychology. Annals of neuro- sciences, 20(4):155, 2013.
Sayna Ebrahimi,
Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. arXiv preprint arXiv:2003.09553, 2020.
Kanyin Feng, Xiao Zhao, Jing Liu, Ying Cai, Zhifang Ye, Chuansheng Chen, and Gui Xue. Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions.
Journal of Neuroscience, 39(27):5351-5360 2019.
Robert M. French. Catastrophic forgetting in connectionist networks.
Trends in Cog-
nilive Sciences, 3(4):128-135, 1999. ISSN 1364-6613. doi: http://doi.org/I//
10166S1366-66113000012444 URL https:/ /www. sci encedirect. com/ acieenenn ence/ artticlle/pt//1136666119999000
Sylvain Gelly, Yizao Wang, Rémi Munos, and Olivier Teytaud. Modification uct with patterns in monte-carlo go. Technical Report RR-6062, INRIA, 2006. inria-001 1726663f..
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri-
cal investigation of catastrophic forgetting gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
Karri S Hawley, Katie Cherry, Emily O Boudreaux, and Erin M Jackson. comparison of ad- justed spaced retrieval versus a uniform expanded retrieval schedule for learning name-face association in older adults with probable alzheimer's discase. Journal of Clinical and Experimen- tal Neuropsychology, 30(6):639-644 2008.
Tyler L Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp- IEEE, 2019.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European Conference on Computer Vision, pp. 466-483. Springer, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incre- mental learning through feature adaptation. In European Conference on Computer Vision, pp. 699-715. Springer, 2020.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
KJ Joseph and Vineeth N Balasubramanian. Meta-consolidation for continual learning. arXiv preprint arXiv:2010.00355 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35213355 -352,, 2017.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference machine learning, pp. 282-293. Springer, 2006.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
11

---

Under review as a conference paper at ICLR 2022
T. Landauer and Robert Bjork. Optimum rehearsal patterns and name learning. Practical aspects of memory, 1, 1977.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2224 1998.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual learning. arXiv preprint arXiv: 17X6v:7006.8880 08840, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7765-7773, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Seyed Iman Mirzadeh and Hassan Ghasemzadeh. Cl-gym: Full-featured pytorch library for con- tinual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3621-3627, June 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXiv preprint arXiv:2010.04495, 2020.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turrer. Variational continual learning. arXiv preprint arXiv: 10628, 2017.
Pingho Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mo- hammad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. arXiv preprint arXiv:2004. 14070, 2020.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real-time continual learning. arXiv preprint arXiv: 19/2.01 100, 2019.
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1320- 1328, 2017.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne, Experience replay for continual learning. arXiv preprint arXiv:1811.11682 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: IrXiv:1606.04:0 0467, 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Stabska-Barwinsse Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International Conference on Machine Learning, pp. 4528-4537. PMLR, 2018.
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548 4557. PMLR, 2018
12

---

Under review as a conference paper at ICLR 2022
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. arXiv preprint 1xXiv:170..0888 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(75877:484448, 2016.
Paul Smolen, Yili Zhang, and John H Byrne. The right time to learn: mechanisms and optimization of spaced learning. Nature Reviews Neuroscience, 17(2):77, 2016.
Gido M van de Ven and Andreas Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv preprint arXiv:/809 /0635, 2018.
Gido M van de Ven, Hava T Siegelmann, and Andreas $ Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 1 11(1):1-14 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. arXiv preprint urXiv: 040000 2016.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: 190.0099,,, 2019.
Judy Willis. Review of research: Brain-based teaching strategies for improving students' memory, learning. and test-taking success. Childhood Education, 1355:33 2007.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 2017.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXiv: 09432, 2019.
Jaehong Yoon, Divyam Madaan, Funho Yang, and Sung Hwang. Online coreset selection for rehearsal-based continual learning, arXiv preprint arXiv:2106 wvXiv:2106.01085, 2021.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
This supplementary material is structured as follows:
Appendix A: Full details of the experimental settings.
Appendix B: Pseudocode RS-MCTS in Algorithm 1 to provide more details about our method.
Appendix C describes the Heuristic scheduling baseline.
Appendix shows additional experimental results.
Appendix includes additional figures.
A EXPERIMENTAL SETTINGS
In this section, we describe the full details of the experimental settings used in this paper.
Datasets:
We conduct experiments on six datasets commonly used in the continual learning lit- erature. Split MNIST (Zenke et al., 2017) is variant of the MNIST (LeCun et al., 1998) dataset where the classes have been divided into tasks incoming in the order 0/1, 2/3, 4/5, 6/7, and 8/9. Split MaSSi (Xiao et al., 2017) is of similar size to MNIST and consists of grayscale images of different clothes, where the classes have been divided into the 5 tasks T-shin//rouser,, Pullover/Dress, Coat/Sandals, Shirt/Sneaker, and Ank/Ankle boots. Similar to MNIST, Split notM- NIST (Bulatov, 2011) consists of 10 classes of the letters A-J with various fonts, where the classes divided into the 5 tasks A/B, C/D, E/F, G/H, and I/J. We use training/test split provided by Ebrahimi et al. (2020) for Split notMNIST. Permuted MNIST Goodfellow et al. (2013) dataset con- sists of applying a unique random permutation of the pixels of the images in original MNIST to create each task, except for the first task that is to learn the original MNIST dataset. We reduce the original MNIST dataset 10k samples and create 9 unique random permutations get 10-task version of Permuted MNIST. In Split CIFAR- 100 (Krizhevsky & Hinton, 2009), the 100 classes are divided into 20 tasks with 5 classes for each task (Lopez-Paz & Ranzato, 2017: Rebuffi al., 2017). Similarly, Split minilmagenet Vinyals et al. (2016) consists of 100 classes randomly chosen from the original Imagenet dataset where the 100 classes are divided into 20 tasks with 5 classes per task.
Network Architectures: We 2-layer MLP with 256 hidden units and ReLU activation for Split MNIST, Split FashionMNIST, Split notMNIST. and Permuted MNIST. We use multi-head output layer for each dataset except Permuted MNIST where the network uses single-head output layer. For Split CIFAR-100, we use a multi-head CNN architecture built according to the CNN in (Adel et al., 2019; Schwarz et al., 2018; Vinyals al., 2016), which consists of four 3x3 convo- lutional blocks, i.e. convolutional layer followed by batch normalization (loffe & Szegedy, 2015), with 64 filters, ReLU activations, and 2x2 Max-pooling. For Split mnilmagenet, we use the reduced ResNet- 18 from Lopez-Paz & Ranzato (2017) with multi-head output layer.
Hyperparameters: We train all networks with the Adam optimizer (Kingma & Ba, 2014) with starting learning rate 7) = 0.001 and hyperparameters 31 0.9 and 82 0.999. Note that the learning rate for Adam is not reset before training on a new task. Next, we give details on number of training epochs and batch sizes specific for each dataset:
Split MNIST: 10 epochs/task, batch size 128.
Split FashionMNIST: 30 epochs/task, batch size 128.
Split notMNIST: 50 epochs/task, batch size 128.
Permuted MNIST: 20 epochs/task, batch size 128.
Split CIFAR- 100: 25 epochs/task, batch size 256.
Split minilmagenet: epoch/task (task trained for epochs as warm up), batch size 32.
We run RS-MCTS for 100 iterations in all experiments. The replay schedules used in the reported results the held-out test are from the replay schedule that gave the highest reward on the validation sets. The exploration constant for UCT in Equation 2 is set to C' = 0.1 in all experiments (Chaudhry & Lee, 2018).
Monte Carlo Tree Search:
14

---

Under review as a conference paper at ICLR 2022
Computational Cost: All experiments were performed
NVIDIA GeForce RTW 2080Ti. The wall clock time for ETS on Split MNIST was around 1.5 minutes, and RS-MCTS and BFS takes 40 seconds on average to run iteration, where BFS runs 1050 iterations in total for Split MNIST.
Implementations: We adapted the implementation released by Borsos et al. (2020) for the mem- ory selection strategies Uniform sampling. k-means clustering, k-center clustering (Nguyen et al., 2017), and Mean-of-Featuree (Rebuffi et al., 2017). Furthermore, we follow the implementations released by Chaudhry et al. (2019) and Mirzadeh & Ghasemzadeh (2021) for A-GEM (Chaudhry et al., 2018b) and ER-Ring (Chaudhry et al., 2019).
Experimental Settings for with Single Task Replay Memory Experiment: We motivated the need for replay scheduling in continual learning with Figure in Section I. This simple experiment was performed on Split MNIST where the replay memory only contains samples from the first task, i.e. learning the classes 0/1. Furthermore, the memory can only be replayed at one point in time and show the performance the memory size to M = 10 samples such that the memory holds samples from both classes. We use the same network architecture and hyperparameters ACC metric above each subfigure corresponds to the ACC for training a network with the single task memory replay at different tasks. We observe that choosing different time points to replay the same memory leads to noticeably different results in the final performance, and in this example, the best final performance is achieved when the memory is used when learning task 5. Therefore, we argue that finding the proper schedule of what tasks to replay at what time in the fixed memory situation can be critical for continual learning.
each task when the memory is replayed at different time steps. We set
described above for Split MNIST. The
B REPLAY SCHEDULING MONTE CARLO TREE SEARCH ALGORITHM
We provide pseudo-code in Algorithm 1 outlining the steps for our method Replay Scheduling Monte Carlo tree search (RS-MCTS) described in the main paper (Section 3.3). The MCTS pro- cedure selects actions over which task proportions to fill the replay memory with at every task, where the selected task proportions to EVALUATEREPLAY PLYTEREEPLLSSIEDULE(( SCHEDULE(.) where the continual learning part executes the training with replay memories filled according to the schedule. The reward for the schedule S is the average vali- dation accuracy over all tasks after learning task T, i..., ACC, which is backpropagated through the tree update the statisties of the selected nodes. The schedule Sbest yielding the best ACC score is returned to be used for evaluation on the held-out test sets.
stored in the replay schedule S. The schedule is then passed
The function GETREPLAY MEMORY(() is the policy for retrieving the replay memory M from the historical data given the task proportion a. The number of samples per task determined by the task proportions are rounded up or down accordingly to fill M with M replay samples in total. The function GETTASK TweeTTASKPRRRORRT PROPORTION()) simply returns the task proportion that is related to given node.
15

---

Under review as a conference paper at ICLR 2022
Algorithm Replay Scheduling Monte Carlo tree search
Require: Tree nodes 01:T, Datasets DEr, Learning rate 1)
Require: Replay memory size M
I: ACCbezt <- 0, Sbest <- ()
2: while within computational budget do
+ ()
4: Uz, U, + TREEPOLICY (vi
5: UT, S DEFAULIPOLICY((H:. S)
6: ACC
M)
BACKPROPAGATE<(M, ACC)
ACCC > ACCbest then
9: ACCheat 4- ACC
10: Sheat +- 5
return ACCoest, Sbest
12:
function TREEPOLICY(w,,
13: while non-terminal do
if v not fully expanded then
15:
return EXPANSION(1r,
16:
else
17:
1/2 BESTCHILD((E))
18:
S.append(ar), where a GETTASSPROPORTION((e))
19:
return vr,
20:
function EXPANSION(Te,
21:
Sample 1 1 uniformly among unvisited children S.append (a,41). where a. +- GETTASKKROPORTION(Co..))
22:
23:
Add new child 1241 to node
24:
return S
25:
function BESTCHILD(U:)
26:
Wi+11 arg max
max(Q(v++1))+ + OV
Alaet Mcvel
N(v441)
Me41 children of
27:
return Ve+1
28:
function DEFAULTPOLICY(V,55
29:
while non non-termiinal
30:
Sample U++1 uniformly among children ih S. S.append((:+4 where d++1 + GET PAKK
31:
32:
Update 4- 2141
33:
return Ve, S
34:
function WWALLATEREPLLYSSHEDUEEEEE
35:
Initialize neural network fo
36:
for 1,..., do
37:
a 4 Sit
38:
M < GETREPLA a)
for ~ ((train) do
39:
40:
+ SGD(BU M,
41:
A (va!)
1:T +- EVALUUAEEACCURACY(OO... (val))
42:
ACC + T. ET, (val)
43:
return ACC
44:
function BACKPROPAGATE(VV, H))
45:
while 24 is not root do
46:
N(v)) + N(x)) +
47:
Q(v:)
24 <- parent V:
16

---

Under review as a conference paper at ICLR 2022
C HEURISTIC SCHEDULING BASELINE
NEW (824B)
We implemented a heuristic scheduling baseline to compare against RS-MCTS. The baseline keeps validation for the old tasks and replays the tasks which validation accuracy below a certain threshold. We set the threshold in the following way: Let Ar, be the validation accuracy for task t evaluated at time step The best evaluated validation accuracy for task time is given by A (beat) = (4,1, A,,))) The condition for replaying task on the next time step is then 1,3
A,, 2 (, where E [0, 1] is ratio controlling how much the current accuracy on task t is allowed to decrease w.r.t. the best accuracy. The replay memory is filled with M/k, where the number of tasks that need be replayed according to their decrease in validation accuracy. This heuristic scheduling corresponds to the intuition of re-learning when task has been forgotten. Training on the current task performed without replay if the accuracy on all old tasks above their corresponding threshold.
Grid search for T. We performed a coarse-to- tosse-to-fine fine grid search for the ratio 7 on each dataset. The best value for is selected according the highest mean accuracy on the validation set averaged over 5 seeds. The validation set consists 15% of the training data and is the same for RS-MCTS. We use the same experimental settings described in Appendix A. The memory sizes are set M = 10 and M = 100 for the 5-task datasets and the 10/20-task datasets respectively, and we apply uniform sampling as the memory selection method. We provide the ranges for that was used on each dataset and put the best value in bold:
Split MNIST: 7 = [0.9, 0.93, 0.95, 0.96, 0.97, 0.98, 0.99]
Split MashionMNISS:: [0.9, 0.93, 0.95, 0.96, 0.97, 0.98, 0.99]
Split notMNIST: [0.9, 0.93, 0.95, 0.96, 0.97, 0.98, 0.99]
Permuted MNIST: [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.9, 0.95, 0.97, 0.99]
Split CIFAR- 100: -= [0.3, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.8, 0.9, 0.95, 0.97, 0.99]
Split minilmagenet:
[0.5, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 0.97, 0.99]
Note that we use these values for 7 on all experiments with Heuristic for the corresponding datasets. The performance for this heuristic highly depends careful tuning for the ratio when the memory size or memory selection method changes, as can be seen in in Figure 5 and Table 1.
17

---

Under review as a conference paper at ICLR 2022
Finctune (ACC; 89.889888 (9.84%)
ETS (ACC: 96.13%)
RS-MCTS (ACC: 98.24%)
Replay Schedulee
100
100
100
Roplayed Task
Taak
Accuracy
90
Accuracy
90
Accuracy 20
80
80
88
Tank
20
Tisk
1
2
3
3 Task
Task
Task
Current Task
Figure 7: Comparison of test classification accuracies for Task 1-5 on Split MNIST from a network trained without replay (Fine-tuning) ETS, and RS-MCTS. The ACC metric for each method is shown on top of each figure. We also visualize the replay schedule found by RS -S-MCTS as bubble plot the right. The memory size set to M 10 with uniform memory selection for ETS and RS-MCTS, Results are shown for seed.
D ADDITIONAL EXPERIMENTAL. RESULTS
In this section, we bring more insights to the benefits of replay scheduling in Section D.I as well as provide metrics for catastrophic forgetting in Section D.2.
D.1 REPLAY SCHEDULE VISUALIZATION FOR SPLIT MNIST
NEW
In Figure 7, we show the progress in test classification performance for each task when using ETS and RS-MCTS with memory size M = 10 on Split MNIST. For comparison, we also show the performance from network that is fine-tuning
(kfup,
dhXe)
the current task without using replay. Both ETS and RS-MCTS overcome catastrophic forgetting to large degree compared to the fine-tuning network. Our method RS-MCTT further improves the performance compared to ETS with the same memory, which indicates that learning the time to learn can be more efficient against catastrophic forgetting. Especially, Task and seems to be the most difficult task to remember since has the lowest final performance using the fine-tuning network. Both ETS and RS-MCTS manage to retain their performance on Task using replay, however, RS-MCTS remembers Task 2 better than ETS by around 5%.
To bring more insights to this behavior, we have visualized the task proportions of the replay ex- amples using bubble plot showing the corresponding replay schedule from RS-MCTS in Figure 7(right). At Task and 4, we see that the schedule fills the memory with data from Task 2 and discards replaying Task 1. This helps the network to retain knowledge about Task 2 better than ETS at the cost of forgetting Task 3 slightly when learning Task 4. This shows that the learned policy has considered the difficulty level of different tasks. At the next task, the RS-MCTS schedule has decided to rehearse Task 3 and reduces replaying Task 2 when learning Task 5. This behavior is similar to spaced repetition, where increasing the time interval between rehearsals helps memory retention. We emphasize that even on datasets with few tasks, using learned replay schedules can overcome catastrophic forgetting better than standard ETS approaches.
D.2 ANALYSIS OF CATASTROPHIC FORGETTING
NEW (dhXe)
In this section, we compare the degree of catastrophic forgetting for our method and the baselines. We use backward transfer (BWT) from Lopez-Paz & Ranzato (2017) for measuring catastrophic forgetting, which is given by
BWT
1 T-1
Σ Ar. Ai.
where Ar, is the test accuracy for task t after learning task é. Table 3 shows the ACC and BWT met- rics for the experiments in Section 4.3. In general, the BWT metric is consistently better when the corresponding ACC is better. We find exception in Table 3 on Split CIFAR-100 and Split minilm- agenet between Ours and Heuristic with uniform selection method, where Heuristic has better BWT while its mean of ACC is slightly lower than ACC for Ours. Table 4 shows the ACC and BWT metrics for the experiments in Section 4.5, where we see a similar pattern that better ACC yields better BWT. The BWT of RS-MCTS is on par with the other baselines except on Split CIFAR-100 where the ACC on our method was a bit lower than the best baselines.
18

---

Under review as a conference paper at ICLR 2022
Table 3: Performance comparison with ACC and BWT metrics for all datasets between RS MCTS and the baselines with various memory selection methods. The memory size is set to M = 10 and M = 100 for the 5-task and 10/20-task datasets respectively. We report the mean and standard deviation of ACC and BWT. where all results have been averaged over 5 seeds. RS-MCTS performs better on par than the haselines most datasets and selection methods, where MoF yields best performance in general.
Split MNIST
Split FashionMNIST
Split BoLMNIST
Selection
Method
ACC(5)
BWT
ACC(3]]
BWT(S)]
ACC
BST
Random
95.91 +: 1.56
-4.79 +: 1.95
95.82
1.45
-435 + 1.79
92.39 + 1.29
4.56 1.29
Uniform
ETS
5.33
3.53
4.45 4.34
1.39
1.82
Heuristic
96.02 + 2.32
-,64
= 2,90
97.09
0.62
2.82
91.26
+ 3.99
-6.06
4.70
Ours
97.93 + 0.56
-2.27 = 0.71
98.27
= 0.17
0.20 -1.29
0.39 94.64
0.79 -1.47
Random
94.24 3.20
-6.88 + 4.00
96.30 + 1.62
-3.77 2.05
91.64 1.39
-5.64 4:44:00 1.77
ETS
-8.66 4.42
0.85
-3.55
1.07
93.80
-2.84
R-means
Heuristic
96.28 1: 1.68
4,32 # 2.11
95.78 +: 1.50
4.46 + 1.87
91.75 + 0.94
5560 +: 2.07
Ours
98.20 0.16
-1.94 =
0.26 98.48
0.31 -1.04
0.71 93.61
-3.11 0.55
Random
96.40 +: 0.68
4,21 + 0.84
95.57 3.16
-7.20 3.93
92.61 + 1.70
4.14 1. 2.37
k-oenter
ETS
94.84
4.20
97.28 0.50
-2.58 0.66
2.48
-6:39 3.46
Heuristic
94.55 + 2.79
-6.47 = 3.50
94.08 + 3.72
-6.59 + 4.57
92.06 + 1.20
4.70 2.09
0.37
Random
95.18 + 3.18
-5.73 = 3,95
95.76 1.41
4,41 1.75
91.33 1.75
-5.94 1.92
MoF'
ETS
97.04 +: 1.23
-3.46 +: 1,50
96.48
1.33
-3.55
1.73
92.64 0.87
-457 1.59
Hcuristic
96.46
2.41
-4,09
= 3.01
95 84
0.89
-4339
+ 1.15
93.24 + 0.77
-3.48
1.37
+:
0.42
0.56
Permuted MNIST
Split CIFAR-100
Split miniImagenet
Selection
Method
ACC(%)
BWT(3T
ACC
BWT(
ACC
BWT(%
Random
72.44 +: 1.15
-25.56
1.39
$3.99
34.19
+ 0.66
48.08 1.36
-15.98 +: 1.08
Uniform
ETS
71.09 1: 2.31
-27.39
2.59
47.70
2.16
41.68
2.37
46.97 1: 1.24
-18.32 1: 1.34
Heuristic
2.13
4 2.41
+ 1.21
1.45
1.10
Ours
76.34 - 0.98
-21.21 2 1.16
56.60
+ 1.13
-31.9 1.11
50.20
-13.46
1.22
Random
74.30 +: 1.43
-23.50
1.64
53.18 1.66
-36.15
49.47 2.70
-14.10 2.71
k-means
ETS
69.40
1.32
-29.23 + 1.47
47.51 1.14
-1.77 + 1.30
45.82 + 0.92
-19.53
1.10
Houristic
75.57
-22.11
$4.31 3.99
-3,800 +
1.00
-12.92
1.22
Ours
77.74 1: 0.80
0.95 -19.66
$6.95 1: 0.92
30.92 1: 0.83
50.47 1 0.85
-13.31 +: 1.24
Random
71.41
-2
48.46 0.31
-39899 + 0.27
44.76 0,96
-18.72 + 1.17
k-center
ETS
69.11 1.69
-29.58
1811
44.13 1.06
-5.28
1.04
41.35 0.96
-23.71
+: 1.45
Houristic
74.33 2,00
-23.45 2.27
50.32 1.97
-3.99 +::99:2.11
44.13 0.95
-18.26
1.05
Ours
76.55 +: 1.16
1.32 -21.06
$1.37 :: 1.63
-37.01 +: 1.62
46.76 + 0.96
-16.56 :: 0.90
Random
77.96 1.84
61.93 1.05
-2.899 + 1.07
$4,50 + 1.33
-8.64 1.26
Mol'
ETS
77.62 +: 1.12
1.26 -20.10
60.43
1.17
28
1.26
56.12 1.12
-8.93 0.83
Heuristic
77.27 1. 1.45
20.15
1.63
55.60 2.70
32.57
1: 2.77
$2.30
1 0.59
9.61 0.67
Ours
81.58 +: 0.75
0.86 -15.41
64.22 0.65
-3.48 + 1.02
57.70 - 0.51
-5.31 0.55
Table 4: Performance comparison with ACC and BWT metrics for all datasets between RS-MCTS and the baselines in the setting where only sample per class can be replayed. The memory sizes are to 10 and M 100 for the 5-task and 10/20-task datasets respectively. We report the mean and standard deviation of ACC and BWT, where all results have been averaged over 5 seeds, RS-MCTS performs on par with the best baselines for both metrics all datasets except S-CIFAR-100
Split MNIST
Split MASSI
Split notMNIST
Method
ACC(S)T
BWT
ACC(S)
BWT(3T
ACC
BWT(3)
Random
2.90
-8.77
3.62
92.70
3.78
-8.24
4.75
89.53 +
A A-GEM
94,97 1.50
..03 1: 1.87
94.81
1 0.86
-5.65
1.06
92.27
1 1.16
.4.17 1: 1.39
ER-Ring
94.94 1.56
-6.07
1.92
95.83
= 2.15
-4.38
2.59
91.10
+ 1.89
-6.27
2.35
Uniform
95.77 1.12
-5.02
1.39
97.12 1.57
2.79
1.98
92.14
: 1.45
-4,90
4: 1.41
RS-MCTS (Ours)
96.07 1.60
-4.59
2.01
97.17
0.78
-2.64
2.69
93.41
-3.36 + 1.56
Permuted
Split CIFAR-100
Split minilmagenet
Method
ACC
BWT
ACC
BWT
ACC BWT(W)T
Random
70.02 1.76
-28.22 + 1.92
48.62
1.02
-39.95 + 1.10
48.85 + 1.38
-14.55
1.86
A-GEM
64.71 1.78
-34.41
4 2.05
42.22
4 2.13
-46.90
+: 2.21
32.06
+: 1.83
-30.81
+ 1.79
Ring
1.13
28.87
1.29
53.93
1.13
Uniform
69.85 1.01
.28.74
+ 1.17
52.63
+ 1.62
-3.44 + 1.81
50.56 + 1.07
-13.52
+ 1.34
RS MCTS (Ours)
72.52 0.54
22.43 1 0.65
$1.50 1: 1.19
.37.01 1. 1.08
50.70 1 0.54
-1266 1. 1.13
19

---

Under review as a conference paper at ICLR 2022
E ADDITIONAL FIGURES
Memory usage. We visualize the memory usage for Permuted MNIST and the 20-task datasets Split CIFAR-I00 and Split minilmagenet in Figure for our method and the baselines used in the experiment in Section 4.5. Our method uses fixed memory size of M = 50 samples for replay on all three datasets. The memory size capacity for our method is reached after learning task 6 and task 11 on the Permuted MNIST and the 20-task datasets respectively, while the baselines continue incrementing their replay memory size.
Memory Usage for 10-task Permuted MNIST
100
Ours Baselines
#eeplayed Samples
30
20
10
2
6
10
Task
Memory Usage for the 20-task Datasets
100
Ours Baselines
#Replayed Samples
50
40
30
3
7
9
10
11 Task
12
13
14
15
16
17
18
19
20
Figure 8: Number of replayed samples per task for 10-task Permuted MNIST (top) and the 20-task datasets our method reached after learning
experiment Section 4.5. The fixed memory size task and task 1l on the Permuted MNIST and the 20-task datasets respectively, while the baselines continue incrementing their number of replay samples per task.
20


Reviews:
[{"review_id": "qQSZM7DWLb3", "reviewer": "Reviewer_8z4B", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I overall found this work interesting and refreshing, since it proposes a novel approach to the improvement of rehearsal-based methods by inquiring what is the best order for revising past tasks. I believe that this issue is very relevant and that there is plenty of room for improvements in this specific area.\nI found this paper to be very well written, well organized and easy to understand, for which I congratulate the authors.\n\nHowever, I find that this work suffers from the following weaknesses:\n\n+ The biggest concern I have with this work is that randomly sampling what information to replay might not be too relevant as a baseline. On the one hand, it is certainly true that this is how the majority of rehearsal approaches work; on the other, it is fairly atypical for these methods to make the assumptions that the authors make here, i.e. that data from past tasks are fully accessible and that the choice of what to replay is up to the model. For this reason, I think it might be appropriate to introduce a somewhat more sophisticated heuristic baseline in addition to the random choice (e.g. keeping a small validation set for past tasks and replaying the ones on which accuracy decreases below a certain level). This would make the experimental section much stronger in my opinion: the finding that the proposed approach outperforms random choice is rather expected in my view.\n+ Along the lines of the previous point, I found the initial experiment to be rather unsurprising. Of course, if given the choice when to replay past data, the decision should be made as late as possible: this minimizes forgetting and lets the model come closer to the overall joint trained baseline, similar to [1].\n+ I am unsure of the efficiency implications of RS-MCTS in a setting such as continual learning, which is generally quite focused on having a reduced model footprint and execution time. It would make for a very interesting ablation study to see a walltime comparison between it and random replay. I suspect that MCTS might be very demanding, which could naturally bring up the point whether simpler approximations of it could be also proposed for a more effective trade-off.\n\nThe following point were minor and did not affect my evaluation\n+ I am not sure I correctly understood how robust the proposed policy is to changes in the order of classes: can the findings of Figure 4 generalize to other orders? Can I learn a policy on a specific task order and test it on another?\n+ the reference to Figure 5 in the text seems to be missing.\n+ While the paper makes it very clear that this approach is effective when the memory buffer is small, what does it happen when we assume that the memory buffer is very large? Do the differences between random and MCTS blur or not? This could be an interesting point to further explore.\n\n[1] GDumb: A Simple Approach that Questions Our Progress in Continual Learning, Prabhu et al.", "summary_of_the_paper": "The authors propose a reinforcement-learning based approach to mitigate catastrophic forgetting by elaborating on the replay schedule for rehearsal methods. They preliminarily show that not all replay schedules are equally effective and then propose a Monte Carlo Tree Search-based approach that is shown to outperform random replay.", "summary_of_the_review": "I feel that the paper is relevant and clearly written, however I am insecure about its impact due to the lack of a valid baseline method and its efficiency implications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "me8N-eRjwpM", "reviewer": "Reviewer_dhXe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. As the authors already mentioned, there are many relevant methods that adaptively select samples (rather than task-uniform selection) to memorize in an on-/offline manner during continual learning. A naive comparison with ETS is not attractive to validate the effectiveness of the proposed method and it requires in-depth comparisons with recent adaptive sample selection-based works.\n\n2. The buffer should memorize a larger number of past tasks' instances to control the instance occupation per task. It enables the model to reuse the discarded instances at previous time steps, which is not allowed standard replay-based continual learning scenario.\n\n3. The requirement of the validation dataset is also the weakness of the proposed method. The authors store and use 15% of training instances as a validation during the continual learning process. This is essential for computing scheduling rewards. To this end, the proposed method indeed needs more room to memorize previous samples compared to baselines. To this end, I'm quite negative about fairness in comparison. \n\n4. In general, there is a lack of analyses in the proposed method, such as \n- the size of the validation set,\n- the possible design of rewards, \n- analysis of catastrophic forgetting,\n- continual learning settings including class-incremental learning,\n- the change of per-task performance according to the selection of adaptive task-selection schedule,\n- the classwise performance when the size of M is smaller than the number of classes like the case of Section 4.5 (i.e., the case that the buffer cannot cover all classes as a replay),\n...", "summary_of_the_paper": "The paper proposes a new scheduling technique for constructing the replay buffer during training. As different to existing baselines which use a task-equal selection, they suggest a dynamic selection of the past tasks' replay exemplars, based on the observation that the time to revisit past tasks affects the averaged performance of the continual learner. The technique is simple yet seems to be effective compared to task-equivalent selection schedule.\n", "summary_of_the_review": "Motivation is reasonable and the proposed technique is simple yet seems to be effective compared to a task-equivalent selection schedule. However, there are several concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GN3_mcx2zxt", "reviewer": "Reviewer_kfup", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Questions**\n- It is not clear if the memory selection process through approaches like k-means chooses datapoints or is it a replacement to MCTS for choosing a sequence of tasks? I have assumed the former while reading the paper.\n- How does this algorithm scale in life-long learning paradigms. Are the K (number of branches) models stored in memory, so it is easier to increase the number of models from T to T+1 without training from scratch? Or am I getting something wrong?\n\n### **Strengths**\n- All the contributions ion the introduction have been verified empirically on 6 reasonably diverse datasets.\n- Table 1 is quite interesting, as it shows a significant boost due to an optimal selection of tasks for replay memory strongly supporting the claim that replay schedule can be quite important for continual learning\n- For extreme scenarios where the replay memory does not have some of the classes that were previously seen, the approach of selecting the correct schedule is able to match performance for continual learning baselines that were designed specifically for memory-efficient training. \n- The paper (along with the appendix) is generally well written and mostly self-contained.\n\n### **Weaknesses** and **Suggestions**\n- The concerns I have with using MCTS to search for an optimal replay sequence are as follows:\n  - The time taken to run the training for each replay sequence from the root to leaf node is equal to the time taken to train all the T tasks. And this process has to be done K number of times to get enough branches for a decent estimate. O(TK) training time does not sound very appealing when K or T is very high. \n  - As T increases, the value of K might need an update, as the tree will progressively become sparser. At a high enough T, wouldn't the greediness of the approach result in the suboptimality of the replay sequence? To prevent that K would need to be increased, which is often possible using a lot of parallel compute, but that is also limited to the processing constraints posed on the problem in the introduction.\n- In the bubble plot in Figure 4, the explanation is that tasks 4 and 6 need lesser replay in the early stages probably because they are less correlated with the other tasks. I would argue that if two tasks are correlated, then there is a greater need to replay that information to expose the model to high-quality negative samples (negative samples that are close to the positive/anchor points; hence the decision boundary is more certain). Either way, It would have been interesting to see a bubble plot for a smaller and analyzable dataset such as MNIST. Examples like the tasks for classifying digits 8, 0 and 3 which can often be confused would have been an interesting analysis tool.\n- There is potential inconsistencies between Figure 5 plots and Figure 3. For M=100 in Figure 5, the accuracies for both ETS and Ours on SplitImagenet and SplitCIFAR are quite different from the ones reported in Figure 3.\n", "summary_of_the_paper": "The key motivation of this work is that the bottleneck of replay in continual learning is the processing time in each training cycle and not storage space for the historical dataset. Hence, this work has been approached from the angle of fixed-sized memory allowance for each experience training cycle. The main research question of this work is: Can the replay schedule (i.e. which task is replayed is which time) significantly affect the training process over time? After demonstrating the effect of the replay schedule, a monte carlo tree search is introduced as a methodology to learn an optimal replay schedule for a series of tasks. This approach shows improvement over naive selection process of replay memory as well as gives the flexibility to apply any selection process for individual sample points for each task. Finally, the efficiency of this approach is tested in extreme scenarios where the replay memory is smaller than the number of classes (i.e. in the training cycles, samples of some classes will not be sampled). Experiments on 6 datasets (varying over 5, 10, and 20 tasks) is used as empirical proof of performance. ", "summary_of_the_review": "While the ideas in this paper are sound and are justified with ample experiments, I have some concerns about how this approach would scale with a significantly larger number of tasks, which was one of the main motivations of the work. Hence my initial rating is borderline", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "99Ggiduaznr", "reviewer": "Reviewer_gm3R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning to schedule replay for continual learning is an interesting and novel idea. Clear writing helps readers appreciate the importance of this topic and understand the proposed solution. The behaviour of the method is well illustrated with the analysis in Fig. 4.  Unfortunately, the experiments fall short of showing the benefit of the solution in a more realistic setting. \n\nHere, the paper focuses more on memory efficiency and ignore computing efficiency. It seems that training with MCTS is more expensive as multiple trials need to be done (around 40 iterations to converge-Fig. 3) in exchange for a few % of final accuracy improvement. Moreover, the performance gap reduces if more sophisticated sampling techniques are employed (Table 1). I wonder if there is still improvement as MCTS is applied to recent replay methods [1,2,3]. Also, as M  increases (e.g. up to 100-Fig. 5), the performance gap is marginal. Here, we need an explanation of the practicality of the trade-off between training and memory complexity (i.e. 1 iteration+M=100 vs 40 iterations+M=10, which one is better?). \n\nAnother concern is the need for a complicated model like MCTS if we can perform multiple runs. In Fig. 3, some tasks only need one iteration (i.e. the found policy is random) to achieve significantly better results. To prove the benefit of MCTS, the authors need to compare MCTS with simple baselines such as random search and/or standard UCB bandit algorithms. \n\n \n\n[1] Riemer, Matthew, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. \"Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference.\" In International Conference on Learning Representations. 2018. \n[2] Buzzega, Pietro, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. \"Dark Experience for General Continual Learning: a Strong, Simple Baseline.\" In NeurIPS. 2020.\n[3] Chaudhry, Arslan, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. \"Using Hindsight to Anchor Past Knowledge in Continual Learning.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, pp. 6993-7001. 2021.", "summary_of_the_paper": "This paper proposes a new continual learning method that learns to select samples for the replaying process. The replay memory is filled with samples from previous tasks according to a specific proportion corresponding to the action performed at the current task. The paper proposes to find the optimal sequence of actions by using Monte-Carlo Tree Search (MCTS) with the reward as the average accuracy over all tasks after seeing the final task. The experiments demonstrate that MCTS improves the continual learning performance especially when the memory size is small and is compatible with some sample-selection strategies. ", "summary_of_the_review": "Overall, I like the topic and proposed method. However, the exepriments need improvement. I may consider raising my score if the authors can: (1) clarify the practicality of the trade-off mentioned above, (2) combine MCTS with recent replay techniques and show clear improvement, and (3)  prove that MCTS is better than simple search/optimization methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of dynamically selecting samples to replay given that all previous data is stored. The paper shows that in this setting, selecting which samples to replay outperforms several baselines over a variety of datasets. 

I believe that the reviewers understood this work, but their initial opinions were quite mixed. 

Two of the reviewers did not "accept" this setting (all past data stored and accessible) as a reasonable one for continual learning. The discussion did not lead to a reconciliation. 

I found truth in both views. On one side, I can believe that the proposed setting has applications (recommender systems where historical data is kept seems like a reasonable one). I also find the approach reasonable since "compute" is often the bottleneck and not memory/storage. On the other, I also see that this is specializing the CL problem a bit and so, while immediately useful, may or may not help to improve more general continual-learning approaches. This is highly speculative. Another argument against this setting is that it is not absolutely clear that in this setting CL approaches are necessarily required. This really depends on the specifics of the problems.

Several of the questions and weaknesses discussed by other reviewers were also discussed and addressed by the reviewers. 

Overall, the final score from the reviewers makes this a very borderline paper. Further, even amongst the positive reviewers, one provides an overall recommendation of a 6 (marginally above the acceptance threshold). In the end, the paper was in the category of papers that were examined closely for possible acceptance, but the broad view of the area chair and the reviewers was that the paper could benefit from additional work before publication.
Decision: Reject