Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The work formulate a dynamic memory construction process for ER as a combinatorial optimization problem, which aims at directly minimizing the global loss across all experienced tasks. The author proposes Global Pseudo-task Simulation for online continual learning memory construction, which is based on permutation of current task.
strengths_and_weaknesses: 1. Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks, as these two can be completely different in practice.

2. Constructing the pseudo tasks is basically a data augmentation process, it is better to include related work of this field into the context introduction also.

3. The expectation of “assume that tasks in a sequence have similar difficulties” in Line 177 don’t hold generally in practice.

4.  2 out the 3 strategies that the author come up with for offline continual learning is intractable, which can serve as good prelimenary ideas but not valid approaches.

questions: Intuitive explanation is needed on why the s_j searched with data augmentation of current task is a good replacement of real future tasks

Some parts of the work is not clear in meaning, e.g. line 80 “each memory Mi after task ti as a variable”, revision on the work is recommended.
limitations:  the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the question of how to find the best policy for constructing the buffer in continual learning with experience replay. The authors first consider an offline setting, where one can go multiple times through the whole training sequence in order to check how different buffer compositions would affect the learning process. In particular, the authors focus on the question of when to apply the ER-Res strategy and when to apply the ER-Ring-Full strategy. Since the offline setting is infeasible in practice, the authors introduce the notion of pseudo-tasks that allow one to simulate future changes in distribution and consider three different pseudo-tasks: permutation, rotation, and blurring. Through empirical analysis, they show that permutation performs best and outperforms ER-Res, ER-Ring-Full, and ER-Hybrid. Finally, the authors investigate the correctness of pseudo-tasks, performance in long sequences, and compatibility with other update methods such as DER.
strengths_and_weaknesses: Strengths:
- The idea seems novel, intriguing, and for the most part intuitive. Although there have been several papers on the best way to construct the buffer in continual learning, to the best of my knowledge none of them approached this problem from the perspective of simulating the future.
- The empirical analysis is satisfactory. Although the improvements introduced by the methods are not revolutionary (usually accuracy increases by 1 or 2 percentage points over ER), they are consistent over different datasets and settings. The authors also investigate important questions such as the computational complexity of the proposed method and the relation between the true global loss and the pseudo-task losses.
- The paper is nicely written, with proper notation, reasoning that is easy to follow, and clearly explained assumptions.

Weaknesses:
- My main doubt is how good the pseudo-tasks really are and how well they approximate the true loss function. The authors find that permutation works best by far in this regard and support this claim by additional analysis in Section 6.3. However, I still don't think learning on permutations has similar dynamics as learning new tasks in CL -- the former should be especially difficult for convnets which rely on the assumption that lots of features are local and this assumption is broken if we apply shuffling. I would argue that for most CL sequences the changes in the input space are not as drastic, as the images stay on the same manifold. So my worry is that the correlation between global loss and the permutation pseudo-task loss is spurious and can break easily under different circumstances. In general, this seems like an important question -- if we could reliably approximate the dynamics of future learning in CL, many different applications and solutions may follow from that (e.g. better regularization methods that know which parameters to protect thanks to simulation). In the end, I think this problem should be investigated further, even though I don't have a clear solution on how to do that. One possible hypothesis I can put forward is memorization -- maybe permutation works well because it mimics the memorization phenomenon [1, 2] which might turn out to be crucial for learning in CL setting.
- In the end, I think the decision space considered in this paper is somewhat limited. There are a lot of possible design decisions in terms of constructing the replay buffer (considering "freshness" of the examples, loss value for the current model, clustering the examples into different subsets, etc) and the paper focuses on two policies: ER-Res and ER-Ring-Full which seem pretty similar in the end. It's still interesting that with this simple choice one can get noticeable performance gains, but I would say that the limited decision space is an important limitation of this study.
- The paper does not consider incremental class learning which is in general much more difficult and interesting. This is especially problematic as ER methods are the most useful in this challenging scenario. Authors do discuss this limitation in the "Future Studies" section at the end of the paper, but I still think this setting does not lie outside of the scope of this paper.

In the end, I would say that the strengths of this paper outweigh the weaknesses and that some of the weaknesses can be addressed by clearly discussing the limitations. As such, my score at the moment is weak accept.

Various comments:
Line 108: Given that the notions of ER-Res and ER-Ring-Full are crucial for the paper, it would be nice to explain them a little bit better, as for now the reader has to go to [3] to get a detailed explanation. For example, including the algorithm in the appendix would be nice.
Figure 3(a) and Figure 3(b): This is very interesting. Do you have similar plots for other datasets? A quantitative metric such as an average L2 loss between pseudo-task loss and global loss would also be useful.
Table 2a: Is it the cost of a single simulation or all simulations throughout the training? Also, what would the numbers look like if the simulation wasn't parallelized?
Line 293: You refer to Appendix D.3 but actually the Appendix does not use letters for indexing. You probably mean Appendix 6.3?
Section 6.6: It might be interesting to see how GPS performs when combined with different methods of choosing the examples from the buffer such as MIR.

[1] Zhang, Chiyuan et al. “Understanding deep learning requires rethinking generalization.” ArXiv abs/1611.03530 (2017)
[2] Arpit, Devansh et al. “A Closer Look at Memorization in Deep Networks.” ArXiv abs/1706.05394 (2017)
[3] Chaudhry, Arslan et al. “On Tiny Episodic Memories in Continual Learning.” arXiv: Learning (2019)

questions: - Can you explain, theoretically or empirically, why the permutation pseudo-task approximates the true global loss well?
- Did you consider including other policies in your search space? I understand that this would significantly increase the complexity and potentially make the algorithm intractable, but I would be interested to know if you think that in principle increasing the policy space would improve the results much further.
limitations: The paper does not discuss the potential negative societal impact and I don't think such a discussion is necessary.

The authors briefly discuss limitations of their work in the "Future Studies" paragraph at the end of the paper. Although some of the major points are covered there (omitting class-incremental CL, problems with simulating forward transfer), I would suggest making this section longer and explaining the problems more clearly. For example:
- The study is limited to vision datasets, we have no idea how the permutation pseudo-task would perform in other domains.
- The algorithm is quite slow, so if we care about quick training it will not be a viable solution.
- The decision space is rather small (see Weakness #2)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: A method is presented to optimally populate the memory in a Continual Learning context. The authors simplify this problem by reducing it to determining the optimal ratio of populating the memory between two baselines: randomly populating the memory and a class balanced random population. The algorithm is shown to be computationally efficient and to perform better than the proposed baselines.
strengths_and_weaknesses: S:
- The questions presented are relevant to the community. Given the popularity of memory-based methods in CL it is relevant to understand if there’s a better way to populate this memory.
- The proposed solution is a creative alternative to how to populate memory. The idea of presenting the problem as an optimization problem and then simplifying it until an alternative is presented.
- The online scenario in CL on which the experiments of this paper are performed is of significant interest to the community.

W:
- The methodology of the paper is unclear. Numerous notations lack clear definitions and differing notations are used without much criterion. 
- It is not clear in which part of the training procedure the pseudo tasks are added. Is it only for selecting which elements to store in the memory? 
- It is unclear why the 2 proposed baselines are used over other possible choices. 
- There’s a bit of confusion on which message this paper is conveying: is it the optimality of the memory selection or the computation feasibility of the method. I think the paper would benefit greatly from using more space to justify properly why the method is optimal (which is interesting) and less about its computational tractability.
- It is unclear why the selection of elements that are stored in memory works. By training the model with the "augmented" data we are looking for elements that can bring us more benefit in the future. Would it be similar to not using transformations? I think that calling them pseudo-future tasks can be misleading.
- It is unclear how fair the comparisons to different methods are. Do different methods see the data the same amount of times or do they share a compute budget? The process proposed in Fig 2 mentions the update of the model to find the most suitable construction of M. This implies that the model is trained more than once in each task.
- In Table 2.a, it would be helpful to show the time cost of other methods.
- It is not well defined what is s_j

questions: - Few questions about definitions: Is g only the optimization function? According to Eq. 4, the model is optimized using only the memory M or is it used to find the elements of the memory? 
- Why present Section 4 as an offline study? It does not appear that the offline assumption is being used to present different simplifications of the optimization problem.
- Did you compare your proposal to a method without data augmentation?
- It would be a good idea to study how the selection of a_j evolves during the different tasks.

limitations: Few limitations of the own method were mentioned in the weaknesses.
review_title: null

Meta-review: metareview: This paper introduces an approach for improving the efficacy of experience replay approaches in continual learning, by cleverly deciding what should be in the replay buffer. Specifically, for each task the approach blends random and class-balanced memories according to a parameter that is fit explicitly (in the offline case) or fit approximately via simulated future tasks (in the online case). The authors show this approach outperforms baselines in avoiding forgetting. The reviewers are somewhat split on this paper, with the main criticisms raised concerning to what extent the simulated tasks accurately portray the challenges posed by real tasks. After reading the paper and reviews, I think this concern is valid and the paper is somewhat close to the border, but I believe (alongside 2 of the 3 reviewers) that overall the paper's innovations outweigh its weaknesses; and I recommend it for acceptance.
confidence: Less certain
Decision: Accept