Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learning
Authors: Yejia Liu
Wang Zhu
Shaolei Ren
Venue: NeurIPS 2022 Accept (2022)
Keywords: continual learning
experience replay
pseudo-task simulation
global optimization
dynamic memory construction

Abstract: Continual learning faces a crucial challenge of catastrophic forgetting. To address this challenge, experience replay (ER) that maintains a tiny subset of samples from previous tasks has been commonly used. Existing ER works usually focus on refining the learning objective for each task with a static memory construction policy. In this paper, we formulate the dynamic memory construction in ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. We first apply three tactics to solve the problem in the offline setting as a starting point. To provide an approximate solution to this problem under the online continual learning setting, we further propose the Global Pseudo-task Simulation (GPS), which mimics future catastrophic forgetting of the current task by permutation. Our empirical results and analyses suggest that the GPS consistently improves accuracy across four commonly used vision benchmarks. We have also shown that our GPS can serve as the unified framework for integrating various memory construction policies in existing ER works.

Navigating Memory Construction by Global Pseudo-Tase Simulation for Continual Learning
Yejia Liu"t t
Wang Zhu
Shaolei Ren
University of California Riverside
University of Southern California {yliu807, shaolei]@ucr. edu wangzhudusc. edu
Abstract
Continual learning faces a crucial challenge of catastrophic forgetting. To address this challenge, experience replay (ER) that maintains tiny subset of samples from previous tasks has been commonly used. Existing ER works usually focus on refining the learning objective for each task with a static memory construction policy. In this paper. we formulate the dynamic memory construction in ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. We first apply three tactics to solve the problem in the offline setting as starting point. To provide an approximate solution to this problem in the online continual learning setting, we further propose the Global Pseudo-task Simulation (GPS), which mimics future catastrophic forgetting of the current task by permutation. Our empirical results and analyses suggest that the GPS consistently improves accuracy across four commonly used vision benchmarks. We have also shown that our GPS can serve the unified framework for integrating various memory construction policies in existing ER works.
Introduction
Data real world is non-stationary and keeps changing. Adapting new knowledge while maintaining the old skills learned from previous data is an idealism for intelligent systems. However, deep artificial neural networks suffer from catastrophic forgetting of old behaviors as the learning of new 34]. To combat this issue, numerous novel algorithms have been designed in continual learning in recent years [37 13] 28 20 2 (27]. Amongst them, the experience replay (ER) methods have been attested as of the few methods that consistently achieve strong results across different continual learning setups 6
tasks keeps overwriting the past 30 32 33
21. In the ER, a slightly relieved continual learning setting is considered, where learner stores subset of old examples from previous tasks in fixed-sized memory and jointly trains the memory with the upcoming task.
Most existing ER-based works put rigorous efforts in refining the local learning objective to update the model parameters on one task at a time. In the methods, they stick to a single static memory construction policy, which is prone to failing in the long task sequence. For example, selecting random data examples from experienced tasks for the memory buffer can effect good generalization the very beginning, but the forgetting rate would soon climb up when the tasks sequence becomes longer, as some class representations are totally squcezed out from the memory [[]] This major drawback has inspired to explore the optimal memory construction with dynamic policy.
In this work, we formulate the memory construction problem in ER as combinatorial optimization problem. Unlike previous memory construction methods 1 15), we explicitly optimize the global objective, i.e., the minimum loss of the final model on all observed tasks, by finding the best memory
Author contributed equally
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

configuration strategy. As starting point, we approach this problem in an offline setting, where we can go through the task sequence for multiple trials independently. By utilizing three tacties, we reduce the intractable search space to a significantly smaller one. Specifically, for each task, we blend random and class-balanced memories according to a parameter, the switching point Then, we use the binary search to find switching points in ((T log [Ml), where T is the total number of tasks and [M] is the size of the memory buffer.
Based on the offline solution, in the online continual learning setup Pseudo-task Simulation (GPS) as an approximate solution to the problem. The GPS mimics the catastrophic forgetting pattern for the current task by creating future pseudo-tasks, and finds approximate switching points based on the pseudo-task simulation. We examine few simulation methods and find permutation is the favorable way to synthesize pseudo-tasks.
2 we propose our method Global
We conduct experiments on four widely used vision benchmarks. Our results have shown that the GPS achieves higher accuracy compared to baselines, especially when we have a long task sequence. In addition, our empirical analysis verifies that the dynamic memory construction by using GPS is close to the offline solution. Meanwhile, GPS can be easily applied to other ER variants 6 9 to further improve their performance.
2 Preliminary and Notations
Continual Learning In continual learning, the model f(0) experiences a stream of data points (24,34) ~ P. from sequence of tasks t, where i E T {1, ..., T.,,,"). and P. is unknown i.i.d. distribution of task 4, Without experience replay, the model f(0) is optimized on one task at a time following the task sequence under the tight constraint that the examples from previous tasks cannot be accessed (38]. We denote f2 as the parameter of f(-) after training task t,, and we refer to the function (()) that updates 0, for each task t, as the local updating method. Once the local updating method is determined, 0y can be derived recursively by 6 = g(0, 1, P.) from fg, which is the initialization point.
After sequentially training T tasks, the objective of continual learning to achieve the minimum loss across all observed tasks with the final model in the end. We write the summed global loss L as in Eqn. [I) where Or is the final parameter after T tasks and ((.) is the cross-entropy loss. For convenience, we also denote the global loss for single task i as L B( (I,,, (zz OT)).
C
E Bor Eyy, I 0T)
Experience Replay Previous works 13
38 46 have proposed series of local updating methods to
optimize the global objective. Amongst them, one effective way is experience replay (ER) from M. Experience replay relieves a bit the tight constraint in continual learning by adding a fixed-sized memory buffer M to store a limited subset of seen examples.
We denote the memory after training task & as M,. The modified local updating method g treats the memory as another input, and jointly optimizes examples of the current task and examples stored in the memory, with factor X on the loss of memory examples as shown in Eqn. ( Thus, 0, is
iteratively updated by 0g = g(0,-1, P., M.-1).
g P. P) arg min
( C. (L)(),
(2)
8
Lo P) = [e f(x;; 0))
(3)
Both empirical results [1 and theoretical analysis 21 have suggested that the local updating
method g of experience replay is effective reducing the global loss L If not specially specified, we refer to local updating method as Eqn. ( in the following text.
3 Problem Formulation: Dynamic Memory Construction
Most previous works based on ER regard the global loss L as a function of g with a static memory construction strategy for M. They utilize various techniques like regularization 6 or memory
We use online as opposed the multi multi-trial offline setup, while still use the multi pass training here

---

Task
Roplay
Replay
Store
Replay
Replay
Equal Memory Heuristic
Maa3
Ma1
Maa
Mig
Soparate
Separate
Separate
Separate
Pollcy Assumption
Mi
M
M
ER-Res
M
Mi
ER-Ring-Full
Search
Search
Search
Switching
Point
49
TA JiMul
JM Mi IiMMal
IM A M/Maal
Figure I: Using three tactics the S-CIFAR-10 dataset solve the dynamic memory construction problem in the offline setting. The equal memory heuristic and the base policy assumption reduce the search space. The switching point existence enables the binary search. For each task, we split the part (ER- FER-Ring-Full) according to the
memories into a random part (ER-Rest and a switching point.
sampling [1] to further refine the local updating method Unlike them, view the global loss _ as a function of the memory M, and explicitly minimize L by optimizing the memory without modifying the local updating methods in Equ. (
Following the setup of ER, we propose dynamic memory construction, which aims at finding the best memory construction M to optimize the global objective L as defined in Eqn. (1). This combinatorial optimization problem is expressed by the Eqn. { where we minimize the global objective by considering the memory M; after task as variables.
L7 ((Mi)itT) (Miher
min
We denote the part in M; storing the examples of task tú as Mi.j- There are four intrinsic constraints for the optimization problem. In all constraints, i, T.
M; is the union of memories {MigheT.. The collection {Misheer is mutually disjoint.
No future examples in the memory, ie., Ma,j = Ø when i <-j-
The size of memory is the same across the training procedure from 1 to lrs denoted as |.M].
As we cannot access previous examples that are not stored in the memory, once the set for task tú in the memory is constructed after training task ty. its size will not increase. Thus, the memory set for task j in M; is always a subset of that in My given j & i' < i.
4 An Offline Solution: Reduce the Search Space
In order to reduce the intractable search space of My quantitatively, we first consider the dynamic memory construction problem in an offline setup as the starting point for analyzing the realistic online setting. In the offline setup, we can go through the task sequence for multiple trials independently, but we are still strictly not allowed to access previous examples that are not M in each trial. We take three tactics to reduce the search space, referred to as the equal memory heuristic, the base policy assumption and the switching point existence. Fig. M illusttraees how we apply three tacties to the S S-CIFAR- 10 dataset. Based on these tactics, provide an approximate solution to the problem.
3

---

4.1 Equal Memory Buffer for Each Task
We first take the equal memory heuristic from [T] which has shown that an equal size of memory buffer for each observed task is effective to avoid catastrophic forgetting R
Based on this tactic, we
add |M,i = |MI/i, where 2 j. the previous constraints list. Given a fixed size of each Mij. instead of optimizing M; from previous observed tasks jointly, we can independently optimize each Mi,j and construct M, from Mijj. To this end, we replace the optimization objective Eqn. (4) by a
simplified version Eqn. (5), taking Mi.j as variables.
min c ((MMs)esse)) (Mig).JeT
(5)
However, given the equal memory heuristic, the search space in Eqn. (5) is still very large. For each task 1j, suppose the total number of examples for task i is my and nj > MM Maj has more than [M]] different constructions.
(Mi))
4.2
Mix of Base Policies
To further reduce the search space in Eqn. ( Full [T]. For given task t,, ER-Res builds a memory by random samples, while ER- ER-Ring-Fiis -Ful builds a memory by selecting the same number of samples from each class.
we introduce two base policies, ER-Res and ER-Ring-
We take the base policy assumption, where we deem that the memories having the same size and taking the same base policies are the same. It implies that even though two memory buffers contain different data examples, they still identified as the same long as they meet these two rules.
This assumption ignores randomness of data selection within the same policy. It is backed up by analysis in DER (6). which shows the standard deviation of accuracy on different benchmarking datasets quite small (~0.5) when using the same policy with a relatively large [M Based on the assumption, we separate Ma, into two disjoint parts and take the mixed policy, where MIS and M represent two parts in Ma.j. namely the former taking the ER-Res and the latter taking ER- Ring-Full policies, respectively. For completeness, we extend the subset constraint to M C CM M fing--full CM where j <.
Previous studies of ER [TT] have shown the randomness (ER-Res) is crucial in a large memory, while guaranteeing the equal representation of each class (ER-Ring-Full (uul)-Full) is crucial under a tiny memory. As the number of tasks grows, the size of memory for each task t, becomes smaller. Under mixed policy, it is natural to first shrink the ER-Res part of the memory and then shrink the ER-Rin ER-Ring-Fiis Ruul part of the memory. By doing so, given Mag and finl- we can determine .M and M fill accordingly for each > j. We denote the size of Minnn-fulll as aj and derive the size of M (IMIjj) - a,. Substituting the collection of sets {MMsjjise by an integer variable aj. we therefore further simplify the optimization objective, as in Eqn. 16)
jJ as
min L (faG)iET) (ag)EeT
(6)
where each aj has [MIj different choices, significantly smaller than the search space for Ma. in Eqn. (5). Notice that the memory construction for the last task ty is not required. Thus, the total search space equals to |M| - 1)!,, which is still large.
4.3 Existence of a Switching Point
For the memory allocation after each task Lj. there exists a gold point a, = that assigns exactly the required ring-full memory to keep the best balance of M and MM for the following task f, If the ring-full memory size is not large enough, we lose the power of forcing an equal representation of classes as the task sequence grows. Conversely, if the ring-full memory is more than enough, we sacrifice the randomness when the task number still small. To this end, we assume sj is a unique switching point, i.e., there exists switching point 8, for each aj. which satisfies the monotonicity
C (Has)) J (aT)ee//))) > (La,)) U (athee/(s)), af < aj S3 or sj < <a (7)
For simiplicity of the formulation, we assume each task has the same number of classes in M.

---

Notice that the global loss of task t, should depend closely on what data we store and replay for task Ej but very loosely on what data we store and replay for other tasks. Following this intuition. though it is hard to verify a specific point satisfies the condition of Eqn. 7 for all combinations of (a:)ee//U) we can simplify the switching point condition to
(d,)) > (a,);
d aj < or 83
< a < dj
(8)
Given the switching point existence, though we still have the same search space, instead of a linear search, we can apply binary search algorithm to reduce the time complexity O [M])- During the binary search, if we search by comparing against the exact left and right integer of the point, Le.,, comparing the loss computed by a; + 1, 0; 1 and aj, the large variance of sampling may cause the algorithm to exit unexpectedly. To increase the robustness of the algorithm, we take a search stride e, where we compare the loss computed by aj + e, aj and aj. The stride is determined by the benchmark and the size M, The detailed binary search algorithm for 8, can be found in our Appendix A.
This assumption is valid for over 85% of the cases from the observations in Fig. ( the S-CIFAR- 10 dataset) and Appendix B (for other benchmarks). For each 0j (i.e., |M 1) in the figures, the global loss _ initially decreases monotonically and then increases monotonically Rarely but possible, due to the variance of sampling, we cannot find the switching point 8j. In such cases, we will choose aj with the minimum loss amongst the values searched.
ring jú
aj grows.
5 Global Pseudo-task Simulation (GPS)
The offline solution of dynamic memory construction requires going through the task sequence for O(T log M)) times find {sy]JeT, which violates the online continual learning setup. To circum- vent this issue, we propose Global Pseudo-task Simulation (GPS), which provides an approximate solution {$ jET the problem by simulating the future training process under the online setup. Specifically, we simulate the local updating process Eqn. (2) by creating pseudo-future tasks.
5.1 Objective Function for Simulation
Perfectly simulating the future is mission impossible in continual learning (1 As we have no information about the future tasks, the distribution of the pseudo-future tasks we create could be quite different from the distribution of the real future ones. To find each approximated switching point 5; more precisely, we intend to use more real tasks and less pseudo-future tasks as possible. As we are required to allocate examples of task t, solve sj by a simulation process from f, to 02, without future modifications. In theory, we could overwrite the previous switching point sy by a more accurate simulation after task t,, j. However, one risk of overwriting the 8j is that we may not have enough examples of task Lj in the memory to support the reallocation brought by the new switching point, since more examples of task L cannot be accessed anymore except the ones stored in MjJ-- Another drawback comes from the drastically increased simulation complexity if we take the overwriting mechanism.
non-empty set Mj.j right after training the task tj.
To this end, we modify the offline objective function Eqn.
6 to the online simulation objective
Eqn. where f is the simulated A, from the real 0,. for é > j.
3j arg min (E (1y, ((zz;00.)))
a,
Note that 0z. is derived recursively from o,.: - ((0),111), P., Mi-1), initialized with 0,ا 0,. P. the task distribution of the synthesized pseudo-tasks L, and M; is the simulated pseudo-memory after training the pseudo-task f,- In the online setup, we cannot access the previous tasks as well as the future tasks. Thus, we will not evaluate the summed global objective LG Following the offline Eqn. (8) evaluate on C. the global objective of task tz-
5.2 Synthesizing Pseudo-tasks
The goal of our pseudo-task simulation is to find 8, precisely, i.e,, 8y Ri 8j- Concretely, instead of accurately simulating the future training process, synthesized pseudo-tasks to mimic the
5

---

forgetting patterns of task i3 caused by the future tasks. To achieve this aim, we believe if the real task sequence holds certain properties, it is essential for pseudo-tasks to hold the same set properties to mimic the same forgetting pattern.
We find most of the existing widely used vision CL benchmarks 6
hold two properties: 1) similar learning difficulty of individual tasks; 2) limited zero-shot transfer ability. The experimental validation of these properties in Appendix C. To induce the same level of catastrophic forgetting, we expect a pseudo-task i, to have similar difficulty as its real counterpart i in the future, measured by the accuracy an identical end-to-end training setup [4]. To avoid the similarity between tasks to lead to little forgetting after training on pseudo pseudo-tasks, pseudo-tasks should also have low zero-shot accuracy with the model trained on the current task. If pseudo-taske ttasks can achieve high zero-shot accuracy with the model trained on the current task, the similarity between tasks could result in very little forgetting after training on the pseudo-tasks, hindering our purpose of mimicking the catastrophic forgetting.
25 +
Based on these two properties, we synthesize pseudo-future tasks from the task Ey by apply- ing different permuting seeds to its input 2j to create series of future tasks {f+ .... (r). We achieve this by permuting the pixels of each im-
M
Memory Buffer Mr
Store
8g
Pseudo-task
(1)
(19221)
Update
Replay
(311)))11)
age on image datasets, i.c., a frrsh permutation would be generated and applied to all images within a synthesized task [54].
Miss
Store
0 J+1
Pseudo-task /+2
(z)
Besides permutation, we have also considered two other synthesizing techniques that lack certain property we discussed for comparison. One is rotation, where we rotate the image in- puts of the task ' gradually by 15 degrees to create pseudo-future tasks (6). The other is blur- ring, where we apply the Gaussian blurring into the image using a 5 X filter with growing stan- dard deviation. Rotation creates pseudo-tasks with similar difficulties as the real tasks, but the zero-shot transfer ability is far too good. Blur- ring creates pseudo-tasks with limited zero-shot transfer ability, yet the task difficulty is increas- ing along the pseudo-task sequence.
P113) (2) (2112. 9
1 Select
Update
Replay
M Sul
ay
Mr
Store
argmin
i pr_
Pseudo-task 7'
(1)
Update
Cr (+)
Eval
6r
task)
Figure 2: The Global Pseudo-task Simulation pro- cess to solve aj after training task ty.
5.3 Construct Pseudo-memories
Fig. 2 visualizes the process of pseudo-task training and pseudo-memory construction. During the simulation process after task fy. we try different 0, with binary search and pick the best one as 8, based on the online objective Eqn. (
We start from My. which is the same as the real Mj. The construction for Mi (i >>> j) different for real tasks and pseudo-tasks. For real tasks J' E (1.....)) Mig' is the real Mi, determined for given aj', we discussed in $ 4.2 Note that the approximated switching points {5 (1..)_...))1)
for the previous j 1 tasks are solved before 8j. For pseudo-tasks J E ( + 1,.,, i), Mas is constructed by randomly selecting |M/i pseudo-date from Ey.
5.4 Other Implementation Details
For the efficiency of simulation, the number of examples of the synthesized pseudo-tasks we use in the simulation is equal to [M] As the simulation quickly converges on a small number of examples, we train fewer epochs for each task compared with the real training process. Besides, when the number of tasks T are large or unknown, we extend GPS by simulating fixed-sized sliding window of future tasks for the sequence, e.g., simulating 10 pseudo-future tasks for each task. Also, during the local update of cach task t,, for a smooth transition, allow the current training task t; to take

---

Table 1: Accuracy of GPS using different simulation techniques and baselines four vision benchmarks. ER-Oracle shows the performance of the offline solution as described in Reported numbers all averaged over 5 runs.
Method
Simulation
P-MNIST
S-CIFAR-10
S-CIFAR-100
TinyLmageNet
[M
1000
200
2000
2000
ER-Res
86,55+0.4x
92.01
81.38+0.51
57.50-0.54
ER-RE ER-RRin--uul
84.33
91.5310.56
81.1620.65
54.73=0.32
ER-Hybrid
86.84.0.35
92.0610.88
81.47
57.97=0.44
GPS
Permutation
07
92.77
82.46
59.26=0.31
Rotation
85.38 20
91.61 40.49
81.50
57.45=0033
Blurring 86.03.0.33
91.96:20.38
81.49
56.85_0.27
ER-Oracle
Offlin
88.26
93.0910.35
82.88
60.56_0.23
up to MM/i the size of the memory without changing Mas for any previous task t the detailed algorithm for GPS in Appendix A.
We also put
Experiments
6.1 Experimental Setup
Datasets We carry out evaluations on four widely used vision benchmarks in continual learning, P-MNIST, S-CIFAR-10, ACIFAR-100 and TinyImageNet [43
25). P-MNIST was proposed in
[8. contains 10 tasks where the first task the MNIST dataset 23 while the later ones are constructed by permuting each image in MNIST with an unique permutation seed. The S-CIFAR-10 constructed by splitting CIFAR-I0 [17] into 5 sequential tasks where each task contain classes and 12,000 images [22 6). Similarly, split CIFAR- 100 [17] into 10 tasks where each one contains 10 classes and 6,000 images to construct S-CIFAR-100 The Tinylmagenet [48] is a subset of ImageNet [1 with 200 classes. We split it into 10 consecutive tasks with 20 classes per task.
Architectures For P-MNIST, we apply a fully connected network with two hidden layers. Each comprises 100 ReLU units. For S-CIFAR-I0, S-CIFAR- 100 and TinylmageNet, we use Resnet18 following and
Baselines The baselines we used in experiments include ER-Res, ER-Ring-Fai Full and ER-Hybrid [
ER-Hybrid is mix of ER-Res and ER-Ring-Full, where the memory construction strategy would switch from the former to the latter once observing only one sample of some class is left M. We
also compare to non-ER methods: online EWC (oEWC) 44 iCaRL 37 A-GEM 10 and GSS 3
Training Details: Our training all use stochastic gradient descent (SGD) with a learning rate of 0.1. We use À = in the local updating method Eqn. (2). For P-MNIST, we train 5 epochs for each task while increasing the number of epochs to 50 for S-CIFAR-10, 100 for both S-CIFAR-100 and TinyImageNet regarding their data complexity.
For P-MNIST and S-CIFAR-10, 10, we set the batch size as 10. For S-CIAFR- and TinylmageNer, batch size set to 50. Following the implementation of ER [1I. we set the same batch size for training the current task and the memory. Note the permutation seeds we use for simulation in P-MNIST are different from the ones used in P-MNIST itself to avoid peeping into test datasets. More training details and hyperparameter values can be found in the Appendix
done by works 6 4
6.2
Main Results
In [ we compare GPS to ER baselines and the offline oracle on the four vision benchmarks. We also compare GPS using different simulation techniques.
We release code httttss///gghhuu..c/////ggggg/gg
7

---

oiler
Gr.ikman
Method
T= 20
T 40
ER-Res
71.2521.07
47.3312.75
Re-Riing-uul
71.92+1.47
49.26
ER- ER-Hybrid
72.27
49.63+2.01
GPS
74.63
53.5710.86
ER- ER-Oracle
75.21=0.17
54.44+0.51
1 1830
(a)
(b)
Figure 3: (a). Global loss of task 23 from the S-CIFAR-100 100 benchmark w.r.t. different configurations of M3.3 (or M3,3). The Offline and Rotation curves use the right y-axis, while the Permutation and Blurring curves use the left y-axis. The blue stars mark the switching points. (b). The loss of task l after training future tasks. (c). Accuracy of GPS and baselines on long sequence P-MNIST.
GPS using permutation performs better GPS using permutation shows improved accuracy compared to ER baselines. Its performance close to the performance of the offline oracle solution, which implies it is good approximation. We can also see that GPS using permutation performs better than using rotation and blurring, in terms of both accuracy and stability. The results support our hypotheses on the properties that synthesized tasks should bear, i.e., similar level difficulties to previous tasks and limited zero-shot transfer ability.
Dynamic memory construction provides stability Besides, we notice that the offline ER oracle, and GPS using different simulation techniques, are more stable than ER baselines as they achieve lower standard deviation on most of the evaluated benchmarks. We attribute this to the reduced interference from random seeds take the global objective into explicit consideration.
6.3 Analysis of Simulation Methods
To further understand why permutation works better than other simulation techniques, analyze the task l3 of the S-CIFAR- 100 benchmark. To make fair comparisons with the offline oracle, we train the pseudo-tasks with the same amount of training data and training epochs here. We plot the curve of global loss w.r.t. the ratio of ER-Ring-Full in the memory Ma.3 (or pseudo memory M2.3). 3). and the forgetting curve of task la after training future tasks (or future pseudo-tasks), as in Fig.
We can see that the rotation induces less forgetting (lower loss) than the real tasks (offline cases) along the sequence, as shown in Fig.3((b) (3b as rotation creates tasks sequences that bear good zero-shot transfer from the previous tasks. The pseudo-task sequence created by rotation is therefore too easy such that the switching point of the curve close to 0, as shown in the Fig. (()) which implies is
similar an ER-Res statie policy. As for the blurring, the forgetting curve first climbs slowly, it
allows some zero-shot transfer from the previous tasks. And then, the loss increases dramatically
since the task difficulty goes up. Its pattern of forgetting is quite different from the real tasks (offline case) as shown in the Fig. 3(b)
We also observe that the GPS using permutation has the closest switching point to the offline compared to the rest. Though permutation creates pscudo-task sequence that result in higher losses than the real task sequence, the pattern of forgetting is similar. We compute the Ll-norm of the offline sampling ratios vs. permuting-simat simulltte sampling ratios, which are 0.9, 0.7, 1.2, 1.2 for P-MNIST, S-CIFAR- S-CIFAR-100 100 and Tiny Imngent,eeet, respectively. The method selects sampling ratios for other benchmarks as good as that for the P-MNIST, which implies the similarity of pseudo- -tauks is not the major factor for the enhanced results
6.4 Long Task Sequence
We extend the 10-task P-MNIST benchmark 20 and 40 tasks create longer task sequences, and make the same comparison on GPS, ER-Orack and three ER baselines in Fig. 3(c)
In the following text, refer to "GPS using permutation" GPS.
8

---

Table 2: (a). Time cost (in minutes) of training vs. simulation for P-MNIST and S- S-CIFAR- (b).
Accuracy of GPS when incorporating existing ER variants, DER++ [6] and HAL comparing to other methods. For these two ER variants, we use memory size of 1000 on P-MNIST and memory size of 2000 on the TinylmageNet.
P-MNIST
TinyImageNet
oEWC
69.21=2.02
20.8
Short Seq
# Tasks
Training
Simulation
iCaRL
38.77+3.68
P-MNIST
10
26.32
2.30
GSS
86.34
S-CIFAR-10
545.56
10.02
A-GEM
77.36 1:28
25.30
S-CIFAR-I00
1187.93
137.63
OGD
81.52=2.21
TinylmageNet
10
2418.20
209.98
HAL
87.69.0.84
Long Seq
# Tasks
Training
Simulation
GPS+HAL
88.2310.08
P-MNIST P-MNIST
20
55.29
6,21
DER +
91.14-0.22
60.67+1.08
40
108.17
13.37
GPS+DER+
91.64+0.16 61.01+0.95
(a)
(b)
Longer task sequence requires more careful memory construction The offline solution, i.e., ER-Oracle, outperforms the baseline policies by ~5% accuracy when T 40. We attribute the performance gain to the global objective oriented memory construction. Clearly, for our global objective oriented memory construction, longer task sequence means larger search space of memory construction. Notice for ER baselines in (c)], the standard deviations are larger than those on shorter task sequences, which implies that the larger space of memory construction choices makes larger performance gap between the worst and the best memory construction.
We also that GPS solves the problem significantly better than the baselines. Interestingly, GPS still achieves close performance to the offline solution when the task sequence is long.
6.5 Simulation Time Cost
We show how many extra time, i.e., pseudo-task generation and pseudo-task training time, GPS adds to the standard ER training. We compute the time cost of the standard ER training vs. the extra time from GPS in Table a)..
GPS is time efficient We take the asynchronous simulation, which applies a binary search to determine 5 sequentially in O(T log (M) sweeps. Suppose simulating the training of each pseudo- task takes a unit time, each sweep is in O(T). Then, the total simulation process is in O log [M]), which is dependent on the memory buffer size and the number of simulation training epochs. To ensure efficiency, we train pseudo-tasks with fewer epochs. We disclose those values for each dataset in Appendix When the task sequence short, from Table we can see the simulation time over ten times less than the standard training time. When the task sequence is long, we can see the simulation cost grows linearly w.r.t. the number of tasks, as we restrict the search window from T to 10 to prevent the simulation cost increasing quadratically.
6.6 Exploration of Other Local Updating Methods
We show the performance of adopting other local updating methods from the existing ER variants besides Eqn. (2). together with GPS. The exemplar base policies we take are from the DER+ and HAL. DER+ + leverages knowledge distillation in the episodic memory construction. More specifically, the DER + stores both the network output logits of examples and their ground truth labels in the memory. HAL selects pivotal learned data points besides random samples to store in the memory. We change the local updating method from Eqn. ( to the local updating method used in DER+ or HAL respectively to transplant them into GPS, without any other modifications to the algorithm.
GPS integrates well with advanced ER variants From Table we can see the performance of DER++ and HAL has been further improved by taking the optimized memory construction for M

---

by GPS, which implies the ability of GPS as a framework to incorporate advanced memory -aanor--baeed continual learning methods. Besides, as the state-of-the-ari method DER++ outperforms other non-ER methods, GPS+DER+ naturally outperforms them.
7 Related Works
Continual Learning Enabling intelligent agent to learn progressively and adaptively without
forgetting old knowledge is long-standing objective in Al 49 39 To combat the catastrophic forgetting problem (30 18, few methods have been proposed in continual learning
12 2 [3]
They usually can be categorized into three classes. One is the regularization-biit based methods, which introduce an additional regularization term in the loss function to consolidate old behaviors when 36 [3]. One is the replay methods, which store tiny amount size-bounded memory or condense previous knowledge in a generative model to 1 9
learning new tasks [20 44 53 8 19
of old examples
generate pseudo samples [26] 46
The data stored in the buffer would be revisited and trained together with each current training task. The third the parameter isolation methods, which usually allocate additional neural resources for new knowledge without The memory cost of these methods would therefore
40
24
47
51
7 B5I.
constraints on the model size [42 29
scale with the number of tasks.
The ER Family Experience replay falls into the replay method category, which takes a fixed-sized buffer to store old examples. Existing experience replay variants have adopted the same setup as ER, and focus on refining the local updating method, i.e. how optimally update model parameters by joint training of current task and examples in memory. The advanced techniques used to improve local updating step include additional regularization [9, knowledge distillation [5] 6 and selective memory sampling [ (35.. Distinct to vanilla ER or its variants, which implicitly minimize the global loss of the final model parameters by designing a powerful local updating method, our work focus on directly optimizing the global objective function by creating pseudo-tasks to mimic the catastrophic forgetting for the current task.
8 Conclusion
In this paper, we propose the dynamic memory construction optimization problem for continual learning under the experience replay setup. The problem aims at finding the best memory construction strategy to optimize the global objective function in continual learning. We simplify the problem to a small space by taking three tacties, and find a solution in time complexity O log [M] in an offline setup. We then officially introduce our Global Pseudo-task Simulation (GPS), which provides an approximate solution to the simplified problem under the realistic online setup by creating pseudo-tasks to mimic the future catastrophic forgetting pattern for the current task. Our empirical results have shown our approach outperforms baselines and can improve the accuracy of existing ER variants (9 [).
Future Studies We focus on the task- and commain-incrementaa in this work. Some future improve- ments could be extending Global Pseudo-task Simulation to the class-incremental (class-IL.) setup. There are two potential challenges under this setup. 1) the task identity is known. This could be achieved by triggering the simulation after experiencing a certain amount of data points instead of task. though the ER-Res and ER-Ring-Full mixed policies are good enough under task and domain IL. we might involve a complex mixture policies to find the offline oracle under class-IL, where new assumptions are required.
Further, we hope our work could inspire the community to design new datasets closer to the real- world setting. For example, when task sequences have specific zero-shot transfer patterns. In such cases, we might first infer the zero-shot transfer pattern from a few data points and then inject the bias into the simulation process.
Acknowledgements
We would like to express our sincere gratitude Sébastien M. R. Arnold, Robin Jia and Jesse Thomason from USC for their valuable suggestions writing.
10

---

References
[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval, 2019.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts, 2017.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. 2019.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
[5] Ari S. Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space, 2019.
[6] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline, 2020.
[7] Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning, 2018.
[8] Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Rieman- nian walk for incremental learning: Understanding forgetting and intransigence. Lecture Notes Computer Science, page 556-572, 2018.
[9] Arslan Chaudhry, Albert Gordo, Puneet K. Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning, 2021.
[10] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem, 2019.
[11] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. On tiny episodic memories continual learning, 2019.
[12] Zhiyuan Chen, Bing Liu, Ronald Brachman, Peter Stone, and Francesca Rossi. Lifelong Machine Learning. Morgan amp; Claypool Publishers, 2nd edition, 2018
[13] Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1-1, 2021.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[15] Arthur Douillard, Eduardo Valle, Charles Ollion, Thomas Robert, and Matthieu Cord. Insights from the future for continual learning, 2020.
[16] Jeffrey Elman. Learning and development in neural networks: the importance of starting small. Cognition, 48:71-99, 08 1993.
[17] Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
[18] Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient- baade neural networks, 2015.
[19] Heechul Jung. Jcongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks, 2016.
[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks, 2017.

---

[21] Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and np-hard, 2020.
[22] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
[23] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 1):2277-22,,, 1998.
[24] Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Wor Woo Ha, and Byoung-Tae Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching, 2018.
[25] Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity, 2021.
[26] Timothée Lesort, Alexander Gepperth, Andrei Stoian, and David Filliat. Marginal replay vs conditional replay for continual learning, 2019.
[27] Zhizhong Li and Derek Hoiem. Learning without forgetting, 2017.
[28] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning, 2017.
[29] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning, 2018.
[30] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem, 1989.
[31] Melike Nur Mermer and Mehmet Fatih Amasyali. Training with growing sets: A simple alternative to curriculum learning and self paced learning, 2018.
[32] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning, 2020.
[33] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?, 2021.
[34] Cuong Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning, 2019.
[35] Ameya Prabhu, Philip Torr, and Puneet Dokania. Gdumb: A simple approach that questions our progress in continual learning. In The European Conference on Computer Vision (ECCV), August 2020.
[36] Amal Rannen, Rahaf Aljundi, Matthew B. Blaschko, and Timne Tuytelaars. Encoder based lifelong learning. 2017 IEEE International Conference Computer Vision (ICCV), Oct 2017.
[37] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning, 2017.
[38] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference, 2019.
[39] Mark B. Ring. Child: A first step towards continual learning. In Learning to Learn, 1998.
[40] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experi- ence replay for continual learning, 2019.
[41] Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian optimization, 2017.
[42] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, 2016.
12

---

[43] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. 2021.
[44] Jonathan Schwarz, Jelena Luketina, Wojciech M. Czameeki, Agnieszka Grrbska-Barwinske Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning, 2018.
[45] Joan Serrà, Didac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task, 2018.
[46] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay, 2017.
[47] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting, 2017.
[48] Stanford. Tiny ImageNet Challenge (CS23 In), 2015. http:///tin-immagenes com/
herokuapp hew
[49] Sebastian Thrun. Lifelong learning algorithms. In Learning Learn, 1998.
[50] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work?, 2021.
[51] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. 2019.
[52] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks, 2017.
[53] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence, 2017.
[54] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See our Future Studies of
Section g
(c) Did you discuss any potential negative societal impacts of your work? [No] No potential negative social impact observed from our perspective
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See our Section
(b) Did you include complete proofs of all theoretical results? [No] The time complexity self-explanable from the given pseudo algorithms
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Please refer to our Section
6 and Appendix F to reproduce the experimental results. We will release the github code to the public at hhttps:///// //gt:::///httut com/ h Liuyeeiaaggaar _cc_cc
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer Appendix F
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Results averaged over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix F.3.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A] We use public data
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] No offensive contents observed from our perspectives
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] No human subjects/research involved
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "w0rCsJRfrK", "reviewer": "Reviewer_BsxV", "summary": "The work formulate a dynamic memory construction process for ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. The author proposes Global Pseudo-task Simulation for online continual learning memory construction, which is based on permutation of current task.", "questions": "Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks\n\nSome parts of the work is not clear in meaning, e.g. line 80 “each memory Mi after task ti as a variable”, revision on the work is recommended.", "limitations": " the authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks, as these two can be completely different in practice.\n\n2. Constructing the pseudo tasks is basically a data augmentation process, it is better to include related work of this field into the context introduction also.\n\n3. The expectation of “assume that tasks in a sequence have similar difficulties” in Line 177 don’t hold generally in practice.\n\n4.  2 out the 3 strategies that the author come up with for offline continual learning is intractable, which can serve as good prelimenary ideas but not valid approaches.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gTKJDvTrd3H", "reviewer": "Reviewer_a3T6", "summary": "A method is presented to optimally populate the memory in a Continual Learning context. The authors simplify this problem by reducing it to determining the optimal ratio of populating the memory between two baselines: randomly populating the memory and a class balanced random population. The algorithm is shown to be computationally efficient and to perform better than the proposed baselines.", "questions": "- Few questions about definitions: Is g only the optimization function? According to Eq. 4, the model is optimized using only the memory M or is it used to find the elements of the memory? \n- Why present Section 4 as an offline study? It does not appear that the offline assumption is being used to present different simplifications of the optimization problem.\n- Did you compare your proposal to a method without data augmentation?\n- It would be a good idea to study how the selection of a_j evolves during the different tasks.\n", "limitations": "Few limitations of the own method were mentioned in the weaknesses.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S:\n- The questions presented are relevant to the community. Given the popularity of memory-based methods in CL it is relevant to understand if there’s a better way to populate this memory.\n- The proposed solution is a creative alternative to how to populate memory. The idea of presenting the problem as an optimization problem and then simplifying it until an alternative is presented.\n- The online scenario in CL on which the experiments of this paper are performed is of significant interest to the community.\n\nW:\n- The methodology of the paper is unclear. Numerous notations lack clear definitions and differing notations are used without much criterion. \n- It is not clear in which part of the training procedure the pseudo tasks are added. Is it only for selecting which elements to store in the memory? \n- It is unclear why the 2 proposed baselines are used over other possible choices. \n- There’s a bit of confusion on which message this paper is conveying: is it the optimality of the memory selection or the computation feasibility of the method. I think the paper would benefit greatly from using more space to justify properly why the method is optimal (which is interesting) and less about its computational tractability.\n- It is unclear why the selection of elements that are stored in memory works. By training the model with the \"augmented\" data we are looking for elements that can bring us more benefit in the future. Would it be similar to not using transformations? I think that calling them pseudo-future tasks can be misleading.\n- It is unclear how fair the comparisons to different methods are. Do different methods see the data the same amount of times or do they share a compute budget? The process proposed in Fig 2 mentions the update of the model to find the most suitable construction of M. This implies that the model is trained more than once in each task.\n- In Table 2.a, it would be helpful to show the time cost of other methods.\n- It is not well defined what is s_j\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UqAL1PXOVy5", "reviewer": "Reviewer_Jeht", "summary": "The paper investigates the question of how to find the best policy for constructing the buffer in continual learning with experience replay. The authors first consider an offline setting, where one can go multiple times through the whole training sequence in order to check how different buffer compositions would affect the learning process. In particular, the authors focus on the question of when to apply the ER-Res strategy and when to apply the ER-Ring-Full strategy. Since the offline setting is infeasible in practice, the authors introduce the notion of pseudo-tasks that allow one to simulate future changes in distribution and consider three different pseudo-tasks: permutation, rotation, and blurring. Through empirical analysis, they show that permutation performs best and outperforms ER-Res, ER-Ring-Full, and ER-Hybrid. Finally, the authors investigate the correctness of pseudo-tasks, performance in long sequences, and compatibility with other update methods such as DER.", "questions": "- Can you explain, theoretically or empirically, why the permutation pseudo-task approximates the true global loss well?\n- Did you consider including other policies in your search space? I understand that this would significantly increase the complexity and potentially make the algorithm intractable, but I would be interested to know if you think that in principle increasing the policy space would improve the results much further.", "limitations": "The paper does not discuss the potential negative societal impact and I don't think such a discussion is necessary.\n\nThe authors briefly discuss limitations of their work in the \"Future Studies\" paragraph at the end of the paper. Although some of the major points are covered there (omitting class-incremental CL, problems with simulating forward transfer), I would suggest making this section longer and explaining the problems more clearly. For example:\n- The study is limited to vision datasets, we have no idea how the permutation pseudo-task would perform in other domains.\n- The algorithm is quite slow, so if we care about quick training it will not be a viable solution.\n- The decision space is rather small (see Weakness #2)", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea seems novel, intriguing, and for the most part intuitive. Although there have been several papers on the best way to construct the buffer in continual learning, to the best of my knowledge none of them approached this problem from the perspective of simulating the future.\n- The empirical analysis is satisfactory. Although the improvements introduced by the methods are not revolutionary (usually accuracy increases by 1 or 2 percentage points over ER), they are consistent over different datasets and settings. The authors also investigate important questions such as the computational complexity of the proposed method and the relation between the true global loss and the pseudo-task losses.\n- The paper is nicely written, with proper notation, reasoning that is easy to follow, and clearly explained assumptions.\n\nWeaknesses:\n- My main doubt is how good the pseudo-tasks really are and how well they approximate the true loss function. The authors find that permutation works best by far in this regard and support this claim by additional analysis in Section 6.3. However, I still don't think learning on permutations has similar dynamics as learning new tasks in CL -- the former should be especially difficult for convnets which rely on the assumption that lots of features are local and this assumption is broken if we apply shuffling. I would argue that for most CL sequences the changes in the input space are not as drastic, as the images stay on the same manifold. So my worry is that the correlation between global loss and the permutation pseudo-task loss is spurious and can break easily under different circumstances. In general, this seems like an important question -- if we could reliably approximate the dynamics of future learning in CL, many different applications and solutions may follow from that (e.g. better regularization methods that know which parameters to protect thanks to simulation). In the end, I think this problem should be investigated further, even though I don't have a clear solution on how to do that. One possible hypothesis I can put forward is memorization -- maybe permutation works well because it mimics the memorization phenomenon [1, 2] which might turn out to be crucial for learning in CL setting.\n- In the end, I think the decision space considered in this paper is somewhat limited. There are a lot of possible design decisions in terms of constructing the replay buffer (considering \"freshness\" of the examples, loss value for the current model, clustering the examples into different subsets, etc) and the paper focuses on two policies: ER-Res and ER-Ring-Full which seem pretty similar in the end. It's still interesting that with this simple choice one can get noticeable performance gains, but I would say that the limited decision space is an important limitation of this study.\n- The paper does not consider incremental class learning which is in general much more difficult and interesting. This is especially problematic as ER methods are the most useful in this challenging scenario. Authors do discuss this limitation in the \"Future Studies\" section at the end of the paper, but I still think this setting does not lie outside of the scope of this paper.\n\nIn the end, I would say that the strengths of this paper outweigh the weaknesses and that some of the weaknesses can be addressed by clearly discussing the limitations. As such, my score at the moment is weak accept.\n\nVarious comments:\nLine 108: Given that the notions of ER-Res and ER-Ring-Full are crucial for the paper, it would be nice to explain them a little bit better, as for now the reader has to go to [3] to get a detailed explanation. For example, including the algorithm in the appendix would be nice.\nFigure 3(a) and Figure 3(b): This is very interesting. Do you have similar plots for other datasets? A quantitative metric such as an average L2 loss between pseudo-task loss and global loss would also be useful.\nTable 2a: Is it the cost of a single simulation or all simulations throughout the training? Also, what would the numbers look like if the simulation wasn't parallelized?\nLine 293: You refer to Appendix D.3 but actually the Appendix does not use letters for indexing. You probably mean Appendix 6.3?\nSection 6.6: It might be interesting to see how GPS performs when combined with different methods of choosing the examples from the buffer such as MIR.\n\n[1] Zhang, Chiyuan et al. “Understanding deep learning requires rethinking generalization.” ArXiv abs/1611.03530 (2017)\n[2] Arpit, Devansh et al. “A Closer Look at Memorization in Deep Networks.” ArXiv abs/1706.05394 (2017)\n[3] Chaudhry, Arslan et al. “On Tiny Episodic Memories in Continual Learning.” arXiv: Learning (2019)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept