Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learning

Abstract: Continual learning faces a crucial challenge of catastrophic forgetting. To address this challenge, experience replay (ER) that maintains a tiny subset of samples from previous tasks has been commonly used. Existing ER works usually focus on refining the learning objective for each task with a static memory construction policy. In this paper, we formulate the dynamic memory construction in ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. We first apply three tactics to solve the problem in the offline setting as a starting point. To provide an approximate solution to this problem under the online continual learning setting, we further propose the Global Pseudo-task Simulation (GPS), which mimics future catastrophic forgetting of the current task by permutation. Our empirical results and analyses suggest that the GPS consistently improves accuracy across four commonly used vision benchmarks. We have also shown that our GPS can serve as the unified framework for integrating various memory construction policies in existing ER works.

Reviews and author responses:

 Thank you for the feedback! We would like to clarify our response as follows:

**Q1 – Pseudo-tasks**

Thanks for the comment, and we will make the wording clarification of forward transfer in the draft. We experiment with the zero-shot forward transfer in Appendix C.2 to show the superiority of permutation over the other compared methods (which allow zero-shot forward transfer). We do agree that evaluating the few-shot transfer is also interesting. 

**Q2 – Other policies**

Here we mean ER-Res and ER-Ring-Full are good strategies and complementary.
"Good": These two methods are shown to be good baselines in many previous works, e.g., [2] and [3].
"Complementary": Sec 5.4 of [1] mentioned this, like *"ring buffer for tiny episodic memories, and reservoir
sampling for bigger episodic memories..."*, impling these two strategies are somehow complementary.

[1] On Tiny Episodic Memories in Continual Learning, Chaudhry et al. 2019

[2] Gradient Projection Memory for Continual Learning, Saha et al. 2021

[3] Using Hindsight to Anchor Past Knowledge in Continual Learning, Chaudhry et al. 2020

**Q4 – Class IL**

Thanks for the comments! We agree the class IL setup is applicable with some change of the algorithm, as we mentioned in the Future Study. However, even if the forward transfer is not a big issue here, we might still need to change the algorithm due to the unknown task identity.



 Thank you for the thorough response and additional improvements made to the paper. For the time being, given arguments of other reviewers and some remaining doubts (e.g. the class IL setting), I decided to keep the score unchanged (Weak Accept).

**Q1 - Pseudo-tasks**

I appreciate the response and the additional experiments. Although I still have doubts whether pixel permutation is a good enough approximation of future tasks, authors provide empirical results showing that at least in the considered settings it works well. One more minor comment after reading Appendix C - I would be careful not to equate forward transfer with zero-shot forward transfer. I agree that in CL you usually do not see any zero-shot forward transfer, but in practice this is not something we really care about, as the focus is on few-shot/many-shot transfer. That is, in practice we usually want to get good performance quick when training on a new task. The experiment presented in Appendix C.2 is still very useful and interesting, I would just recommend being more explicit about the semantics and avoid general statements about forward transfer.

**Q2 - Other policies** 
> ER-Res and ER-Ring-Full have known to be good complementary memory construction strategies

Could you elaborate on that? Is this statement supported by previous work?

**Q4 - Class IL**

I appreciate the answer, but if I understand correctly, in principle, you could apply your method to class IL, even though it might not work very well. I'm also not convinced that forward transfer would be the greatest issue here -- the pseudo-tasks you build are explicitly created to limit zero-shot forward transfer and I think there would also be virtually none (or even negative) zero-shot forward transfer in CL. In the current scope of the work, using this quite sophisticated replay method to attack the task IL problem seems like an overkill, so I really think that including class IL experiments would improve this work significantly.

 We thank all the reviewers for their efforts and thoughtful comments. We have submitted a revised manuscript. In particular,

 - We further clarified the intuition and motivation of designing pseudo-tasks.

 - We provided more explanations of the simulation time table (Table 2a).

 - We rewrote the Future Studies paragraph following the suggestion from reviewers.

 - We added detailed ER-Res and ER-Ring-Full algorithm in the supplementary material to help readers understand the base policies.

 - We fixed typos and indexing of the main paper and the supplementary material.

Revisions and new contents are marked in blue.


 **Q1 – Notations:** “Few questions about definitions: Is g only the optimization function? According to Eq. 4, the model is optimized using only the memory M or is it used to find the elements of the memory? what is s_j?”

**Response:** We would like to clarify our notations. As explained in the L64-67, $g$ is a pre-determined local updating function,  not the global optimization function. The local updating function $g$ aims to find the parameter value of the next step $\theta_i$ given the current parameter value $\theta_{i-1}$, memory construction $\mathcal{M}_{i-1}$ and the next task distribution $P_i$. The global loss takes the memory construction $\mathcal{M}$ as variables, and updates $\theta$ with $g$. As mentioned in L135, $s_j$ is the switching point for memory allocation after task $t_j$.

**Q2 – Motivation for the offline study:** “Why present Section 4 as an offline study? It does not appear that the offline assumption is being used to present different simplifications of the optimization problem.”

**Response:** Thanks for your comments. The offline assumption enables the search of the memory construction across multiple runs. Without the offline assumption, we cannot search on the future tasks, where we come up with simulating the future tasks. The reasons we are doing offline study before exploring the online solution are two-fold: 1) different from other ER-based works, our goal is to optimize the global loss. We therefore believe an offline study can help us develop useful assumptions and tactics to reduce search space, which are then reused in our online approach; 2) the solution of the offline setting serves as an oracle to evaluate the performance of the online solutions as they are under the same set of assumptions.


**Q3 – Non data augmentation methods:**: “Did you compare your proposal to a method without data augmentation?”

**Response:** If the data augmentation refers to our pseudo-task constructions, we compared to ER-Res, ER-Ring and ER-Hybrid, which are without pseudo-task constructions. If the data augmentation refers to the ER memory construction, the baselines we used like oEWC, iCaRL and OGD are not using any data augmentation and the results can be found in the Table 2b.

**Q4 – On pseudo-tasks:** “It is not clear in which part of the training procedure the pseudo tasks are added. Is it only for selecting which elements to store in the memory?”

**Response:** The pseudo-tasks are only added in the selection stage to construct the memory. The detailed algorithm can be found in our Supplementary material Section A.


 **Q5 – Two baselines:** “It is unclear why the 2 proposed baselines are used over other possible choices”

**Response:** Thanks for your helpful comments. The ER-Res and ER-Ring-Full have known to be simple and effective memory construction strategies. On the other hand, we want to point out that our GPS is a general framework to integrate various base policies like DER and HAL and the corresponding empirical results are reported in the Table 2b. ER-Res and ER-Ring-Full are widely used and referred in the ER-based works, so we believe using these methods as our base policies can better help readers understand our ideas. We have also explored and used loss-based curriculum policies at Section E of the supplementary material.

**Q6 – Computation Feasibility and Optimality:** “There’s a bit of confusion on which message this paper is conveying: is it the optimality of the memory selection or the computation feasibility of the method…”

**Response:** We would like to clarify that we come up with three key assumptions following the logic of reducing searching space from intractable to tractable. Thus, we believe the computation feasibility parts are naturally part of our logic and necessary. Moreover, the offline study reduces the memory construction to a binary search problem under the three assumptions, which could clearly find an optimal solution.

**Q7 – What if no transformations:** “It is unclear why the selection of elements that are stored in memory works. Would it be similar to not using transformations?”

**Response:** Thanks for the comments. As explained in our Section 5.2, the goal of selecting data examples is to mimic the forgetting pattern. We do not think using exactly the same current task without any editing would be a good mimic, as there would be no forgetting introduced by this way.

**Q8 – Comparison of different methods:** “It is unclear how fair the comparisons to different methods are. Do different methods see the data the same amount of times or do they share a compute budget?”

**Response:** As described in the Section 6.1, we follow the setting of ER (Chaudhry et al. 2019), where different methods use the same memory construction budget, e.g., they could store at most 200 data examples in the episodic memory. The comparison is fair based on our understanding and experiments.

**Q9 – Time cost:** “In Table 2.a, it would be helpful to show the time cost of other methods”

**Response:** Thanks for pointing this out. The training time of other ER methods including ER-Res, ER-Ring-Full, ER-Hybrid is the same as the one in the Table 2a ‘Training’ column. As for the simulation time of our method, we would like to clarify that the simulation time numbers shown in Table 2a is the total simulation time, including the required pseudo-task training time. The overall time cost of our method therefore is about 1.1 times of other ER methods. 
We have changed the text to make this clearer in Section 6.5. 


 **Q1 – Pseudo-tasks, how good and why:** *“Can you explain, theoretically or empirically, why the permutation pseudo-task approximates the true global loss well?”*

**Response:** 
Thanks for your constructive comments and hypotheses.  For the "how good" part, we provide detailed statistics the reviewer requested.

Average difference of switching point between offline solution and simulated online solution, $\frac{1}{n}\sum^{n}_{1} |s_j-\tilde{s_j}|$ (x-axis of Figure 3a):
| Method   | P-MNIST | S-CIFAR-10 | S-CIFAR-100 |  TinyImageNet |
|---|---|---|---|---|
| Permutation  |  0.09 |    0.14	|    0.12	| 0.12 |
| Rotation | 0.20 |    0.22	|   0.29 | 0.24 |
| Blurring 	|  0.24 |	0.20	|     0.22	| 0.23 |
 
Average L2 distance of the global loss and pseudo-task loss (L2 of y-axis on the last point x=10 of Figure 3b):
| Method   | P-MNIST | S-CIFAR-10 | S-CIFAR-100 |  TinyImageNet |
|---|---|---|---|---|
| Permutation  |  0.005 |    0.009 |    0.010 | 0.018 |
| Rotation | 0.008 |    0.015 |   0.022 | 0.024 |
| Blurring 	|  0.009 |	0.008 |   0.013	| 0.018 |

The permutation clearly finds the closest switching point (i.e., memory configuration) to the offline oracle. As for approximating the global loss, permutation and blurring outperforms rotation. Though blurring also approximate the global loss well, as shown in Figure 3b, the trend of loss changing is different from the real future tasks. Thus, the forgetting pattern of the blurring tasks is quite different from the real tasks, making it fails to search the switching point.

For the "why" part, please refer to the response of Q1 for reviewer BsxV. In Section C.1 of the Supplementary Material, Table 1-3 verify the permutation created pseudo-tasks have similar difficulty and forward transfer ability as the real future tasks. We agree that the learning dynamics of pseudo-tasks could be different from the real tasks, which might result in some discrepancy between global loss and pseudo-task loss at the end. However, if a pseudo task have a similar task difficulty and a similar relationship to the previous tasks as a real task, we expect the additional model capability required to learn this pseudo-task is also similar, resulted in a similar forgetting of the model. Thanks again for putting forward the memorization phenomenon of networks, which relates to our intuition for designing pseudo tasks.

**Q2 – Other policies:** *"Did you consider including other policies in your search space? ..."*

**Response:** ER-Res and ER-Ring-Full have known to be good complementary memory construction strategies. Though not optimal, the mixed policy seems good enough for task IL. We explored loss-based curriculum policies at Section E of the supplementary, but the results are no better than the mixture of ER-Res and ER-Ring-Full. We suspect including more policies into the search space would not lead to large performance gain for task IL. On the other hand, for the class IL setup, it is possible that more policies in the search space with a more careful memory construction would improve the performance greatly, as the setting is more challenging and requires that.

**Q3 – Time concern:** *"Is it the cost of a single simulation or all simulations throughout the training? Also, what would the numbers look like if the simulation wasn't parallelized? The algorithm seems quite slow"*

**Response:** We would like to clarify that the time cost results in Table 2a is using asynchronous simulations. Also, the simulation time includes all simulations throughout the training, which shows the total simulation time is only ~10% of the whole training time. Note that we train pseudo-tasks with fewer epochs than real tasks, details in Section F.1 of the supplementary.

We have changed the text to make this clearer in Section 6.5.

**Q4 – Class-IL:** *"... but I still think this setting does not lie outside of the scope of this paper"*

**Response:** Our offline study could be applied to class IL, while for the online simulation, task properties like forward transfer ability might be complex under the class IL setup. E.g., learning what is a cat might be beneficial for learning what is a dog. To generate pseudo tasks, we might first infer the forward transfer pattern from a few data points and then inject the bias into pseudo-tasks. This sounds like a very different method from ours, thus, we leave it for future studies.






 **Q1 – Intuition of why pseudo-task works:** *"Intuitive explanation is needed on why the $s_j$ searched with data augmentation of current task is a good replacement of real future tasks."*

**Response:** As explained in Section 5.2, our goal of simulation is to find the best memory construction. Thus, the aims of the data augmentation is not to predict what the future tasks are, but to create pseudo-task sequences that *mimic the future forgetting pattern of the real tasks*. As long as we mimic the future forgetting pattern, we could search the best memory construction with these pseudo-tasks, and these pseudo-tasks would be considered as good replacement.

We believe if the real task sequence holds certain properties, it is essential for pseudo-tasks to *hold the same set of properties* in order to mimic the same forgetting pattern. The existing widely used vision CL benchmarks hold two properties: 1) similar learning difficulty of individual tasks; 2) limited forward transfer ability. The permutation data augmentation creates pseudo tasks holding both two these two properties.

We have put the revised text in the paper to make this point clearer in Section 5.2. 

**Q2 – On similar difficulties of tasks:** *"The expectation of 'assume that tasks in a sequence have similar difficulties' in Line 177 don’t hold generally in practice."*

**Response:** We politely disagree with that viewpoint. Confirmed by our experiments, displayed in the Supplementary Material C.1, most of the existing widely used CL vision benchmarks hold the property – "tasks have similar difficulties". Note that benchmarks like S-CIFAR-10, S-CIFAR-100 and TinyImageNet are all using real world images, not synthetic ones. This observation also implies CL benchmarks created by splitting a real-world image dataset into sequential classification tasks, which is a widely used approach of creating vision CL benchmarks, generate tasks of similar difficulties.

We agree it is possible to create CL benchmarks with tasks of very different difficulties by special design. In the Future Studies of Section 8,  we discussed this limitation and the possible solution.

**Q3 – Intractable search space:** "2 out the 3 strategies that the author come up with for offline continual learning is intractable, which can serve as good preliminary ideas but not valid approaches."

**Response:** We would like to clarify that our offline solution use 3 strategies together, not independently, to achieve the final tractable research space.


 The work formulate a dynamic memory construction process for ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. The author proposes Global Pseudo-task Simulation for online continual learning memory construction, which is based on permutation of current task. 1. Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks, as these two can be completely different in practice.

2. Constructing the pseudo tasks is basically a data augmentation process, it is better to include related work of this field into the context introduction also.

3. The expectation of “assume that tasks in a sequence have similar difficulties” in Line 177 don’t hold generally in practice.

4.  2 out the 3 strategies that the author come up with for offline continual learning is intractable, which can serve as good prelimenary ideas but not valid approaches.
 Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks

Some parts of the work is not clear in meaning, e.g. line 80 “each memory Mi after task ti as a variable”, revision on the work is recommended.  the authors adequately addressed the limitations and potential negative societal impact of their work

 The paper investigates the question of how to find the best policy for constructing the buffer in continual learning with experience replay. The authors first consider an offline setting, where one can go multiple times through the whole training sequence in order to check how different buffer compositions would affect the learning process. In particular, the authors focus on the question of when to apply the ER-Res strategy and when to apply the ER-Ring-Full strategy. Since the offline setting is infeasible in practice, the authors introduce the notion of pseudo-tasks that allow one to simulate future changes in distribution and consider three different pseudo-tasks: permutation, rotation, and blurring. Through empirical analysis, they show that permutation performs best and outperforms ER-Res, ER-Ring-Full, and ER-Hybrid. Finally, the authors investigate the correctness of pseudo-tasks, performance in long sequences, and compatibility with other update methods such as DER. Strengths:
- The idea seems novel, intriguing, and for the most part intuitive. Although there have been several papers on the best way to construct the buffer in continual learning, to the best of my knowledge none of them approached this problem from the perspective of simulating the future.
- The empirical analysis is satisfactory. Although the improvements introduced by the methods are not revolutionary (usually accuracy increases by 1 or 2 percentage points over ER), they are consistent over different datasets and settings. The authors also investigate important questions such as the computational complexity of the proposed method and the relation between the true global loss and the pseudo-task losses.
- The paper is nicely written, with proper notation, reasoning that is easy to follow, and clearly explained assumptions.

Weaknesses:
- My main doubt is how good the pseudo-tasks really are and how well they approximate the true loss function. The authors find that permutation works best by far in this regard and support this claim by additional analysis in Section 6.3. However, I still don't think learning on permutations has similar dynamics as learning new tasks in CL -- the former should be especially difficult for convnets which rely on the assumption that lots of features are local and this assumption is broken if we apply shuffling. I would argue that for most CL sequences the changes in the input space are not as drastic, as the images stay on the same manifold. So my worry is that the correlation between global loss and the permutation pseudo-task loss is spurious and can break easily under different circumstances. In general, this seems like an important question -- if we could reliably approximate the dynamics of future learning in CL, many different applications and solutions may follow from that (e.g. better regularization methods that know which parameters to protect thanks to simulation). In the end, I think this problem should be investigated further, even though I don't have a clear solution on how to do that. One possible hypothesis I can put forward is memorization -- maybe permutation works well because it mimics the memorization phenomenon [1, 2] which might turn out to be crucial for learning in CL setting.
- In the end, I think the decision space considered in this paper is somewhat limited. There are a lot of possible design decisions in terms of constructing the replay buffer (considering "freshness" of the examples, loss value for the current model, clustering the examples into different subsets, etc) and the paper focuses on two policies: ER-Res and ER-Ring-Full which seem pretty similar in the end. It's still interesting that with this simple choice one can get noticeable performance gains, but I would say that the limited decision space is an important limitation of this study.
- The paper does not consider incremental class learning which is in general much more difficult and interesting. This is especially problematic as ER methods are the most useful in this challenging scenario. Authors do discuss this limitation in the "Future Studies" section at the end of the paper, but I still think this setting does not lie outside of the scope of this paper.

In the end, I would say that the strengths of this paper outweigh the weaknesses and that some of the weaknesses can be addressed by clearly discussing the limitations. As such, my score at the moment is weak accept.

Various comments:
Line 108: Given that the notions of ER-Res and ER-Ring-Full are crucial for the paper, it would be nice to explain them a little bit better, as for now the reader has to go to [3] to get a detailed explanation. For example, including the algorithm in the appendix would be nice.
Figure 3(a) and Figure 3(b): This is very interesting. Do you have similar plots for other datasets? A quantitative metric such as an average L2 loss between pseudo-task loss and global loss would also be useful.
Table 2a: Is it the cost of a single simulation or all simulations throughout the training? Also, what would the numbers look like if the simulation wasn't parallelized?
Line 293: You refer to Appendix D.3 but actually the Appendix does not use letters for indexing. You probably mean Appendix 6.3?
Section 6.6: It might be interesting to see how GPS performs when combined with different methods of choosing the examples from the buffer such as MIR.

[1] Zhang, Chiyuan et al. “Understanding deep learning requires rethinking generalization.” ArXiv abs/1611.03530 (2017)
[2] Arpit, Devansh et al. “A Closer Look at Memorization in Deep Networks.” ArXiv abs/1706.05394 (2017)
[3] Chaudhry, Arslan et al. “On Tiny Episodic Memories in Continual Learning.” arXiv: Learning (2019)
 - Can you explain, theoretically or empirically, why the permutation pseudo-task approximates the true global loss well?
- Did you consider including other policies in your search space? I understand that this would significantly increase the complexity and potentially make the algorithm intractable, but I would be interested to know if you think that in principle increasing the policy space would improve the results much further. The paper does not discuss the potential negative societal impact and I don't think such a discussion is necessary.

The authors briefly discuss limitations of their work in the "Future Studies" paragraph at the end of the paper. Although some of the major points are covered there (omitting class-incremental CL, problems with simulating forward transfer), I would suggest making this section longer and explaining the problems more clearly. For example:
- The study is limited to vision datasets, we have no idea how the permutation pseudo-task would perform in other domains.
- The algorithm is quite slow, so if we care about quick training it will not be a viable solution.
- The decision space is rather small (see Weakness #2)

 A method is presented to optimally populate the memory in a Continual Learning context. The authors simplify this problem by reducing it to determining the optimal ratio of populating the memory between two baselines: randomly populating the memory and a class balanced random population. The algorithm is shown to be computationally efficient and to perform better than the proposed baselines. S:
- The questions presented are relevant to the community. Given the popularity of memory-based methods in CL it is relevant to understand if there’s a better way to populate this memory.
- The proposed solution is a creative alternative to how to populate memory. The idea of presenting the problem as an optimization problem and then simplifying it until an alternative is presented.
- The online scenario in CL on which the experiments of this paper are performed is of significant interest to the community.

W:
- The methodology of the paper is unclear. Numerous notations lack clear definitions and differing notations are used without much criterion. 
- It is not clear in which part of the training procedure the pseudo tasks are added. Is it only for selecting which elements to store in the memory? 
- It is unclear why the 2 proposed baselines are used over other possible choices. 
- There’s a bit of confusion on which message this paper is conveying: is it the optimality of the memory selection or the computation feasibility of the method. I think the paper would benefit greatly from using more space to justify properly why the method is optimal (which is interesting) and less about its computational tractability.
- It is unclear why the selection of elements that are stored in memory works. By training the model with the "augmented" data we are looking for elements that can bring us more benefit in the future. Would it be similar to not using transformations? I think that calling them pseudo-future tasks can be misleading.
- It is unclear how fair the comparisons to different methods are. Do different methods see the data the same amount of times or do they share a compute budget? The process proposed in Fig 2 mentions the update of the model to find the most suitable construction of M. This implies that the model is trained more than once in each task.
- In Table 2.a, it would be helpful to show the time cost of other methods.
- It is not well defined what is s_j
 - Few questions about definitions: Is g only the optimization function? According to Eq. 4, the model is optimized using only the memory M or is it used to find the elements of the memory? 
- Why present Section 4 as an offline study? It does not appear that the offline assumption is being used to present different simplifications of the optimization problem.
- Did you compare your proposal to a method without data augmentation?
- It would be a good idea to study how the selection of a_j evolves during the different tasks.
 Few limitations of the own method were mentioned in the weaknesses.

Meta-review: This paper introduces an approach for improving the efficacy of experience replay approaches in continual learning, by cleverly deciding what should be in the replay buffer. Specifically, for each task the approach blends random and class-balanced memories according to a parameter that is fit explicitly (in the offline case) or fit approximately via simulated future tasks (in the online case). The authors show this approach outperforms baselines in avoiding forgetting. The reviewers are somewhat split on this paper, with the main criticisms raised concerning to what extent the simulated tasks accurately portray the challenges posed by real tasks. After reading the paper and reviews, I think this concern is valid and the paper is somewhat close to the border, but I believe (alongside 2 of the 3 reviewers) that overall the paper's innovations outweigh its weaknesses; and I recommend it for acceptance.

Decision: Accept