Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The key motivation of this work is that the bottleneck of replay in continual learning is the processing time in each training cycle and not storage space for the historical dataset. Hence, this work has been approached from the angle of fixed-sized memory allowance for each experience training cycle. The main research question of this work is: Can the replay schedule (i.e. which task is replayed is which time) significantly affect the training process over time? After demonstrating the effect of the replay schedule, a monte carlo tree search is introduced as a methodology to learn an optimal replay schedule for a series of tasks. This approach shows improvement over naive selection process of replay memory as well as gives the flexibility to apply any selection process for individual sample points for each task. Finally, the efficiency of this approach is tested in extreme scenarios where the replay memory is smaller than the number of classes (i.e. in the training cycles, samples of some classes will not be sampled). Experiments on 6 datasets (varying over 5, 10, and 20 tasks) is used as empirical proof of performance. 
main_review: ### **Questions**
- It is not clear if the memory selection process through approaches like k-means chooses datapoints or is it a replacement to MCTS for choosing a sequence of tasks? I have assumed the former while reading the paper.
- How does this algorithm scale in life-long learning paradigms. Are the K (number of branches) models stored in memory, so it is easier to increase the number of models from T to T+1 without training from scratch? Or am I getting something wrong?

### **Strengths**
- All the contributions ion the introduction have been verified empirically on 6 reasonably diverse datasets.
- Table 1 is quite interesting, as it shows a significant boost due to an optimal selection of tasks for replay memory strongly supporting the claim that replay schedule can be quite important for continual learning
- For extreme scenarios where the replay memory does not have some of the classes that were previously seen, the approach of selecting the correct schedule is able to match performance for continual learning baselines that were designed specifically for memory-efficient training. 
- The paper (along with the appendix) is generally well written and mostly self-contained.

### **Weaknesses** and **Suggestions**
- The concerns I have with using MCTS to search for an optimal replay sequence are as follows:
  - The time taken to run the training for each replay sequence from the root to leaf node is equal to the time taken to train all the T tasks. And this process has to be done K number of times to get enough branches for a decent estimate. O(TK) training time does not sound very appealing when K or T is very high. 
  - As T increases, the value of K might need an update, as the tree will progressively become sparser. At a high enough T, wouldn't the greediness of the approach result in the suboptimality of the replay sequence? To prevent that K would need to be increased, which is often possible using a lot of parallel compute, but that is also limited to the processing constraints posed on the problem in the introduction.
- In the bubble plot in Figure 4, the explanation is that tasks 4 and 6 need lesser replay in the early stages probably because they are less correlated with the other tasks. I would argue that if two tasks are correlated, then there is a greater need to replay that information to expose the model to high-quality negative samples (negative samples that are close to the positive/anchor points; hence the decision boundary is more certain). Either way, It would have been interesting to see a bubble plot for a smaller and analyzable dataset such as MNIST. Examples like the tasks for classifying digits 8, 0 and 3 which can often be confused would have been an interesting analysis tool.
- There is potential inconsistencies between Figure 5 plots and Figure 3. For M=100 in Figure 5, the accuracies for both ETS and Ours on SplitImagenet and SplitCIFAR are quite different from the ones reported in Figure 3.

summary_of_the_review: While the ideas in this paper are sound and are justified with ample experiments, I have some concerns about how this approach would scale with a significantly larger number of tasks, which was one of the main motivations of the work. Hence my initial rating is borderline
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new scheduling technique for constructing the replay buffer during training. As different to existing baselines which use a task-equal selection, they suggest a dynamic selection of the past tasks' replay exemplars, based on the observation that the time to revisit past tasks affects the averaged performance of the continual learner. The technique is simple yet seems to be effective compared to task-equivalent selection schedule.

main_review: 1. As the authors already mentioned, there are many relevant methods that adaptively select samples (rather than task-uniform selection) to memorize in an on-/offline manner during continual learning. A naive comparison with ETS is not attractive to validate the effectiveness of the proposed method and it requires in-depth comparisons with recent adaptive sample selection-based works.

2. The buffer should memorize a larger number of past tasks' instances to control the instance occupation per task. It enables the model to reuse the discarded instances at previous time steps, which is not allowed standard replay-based continual learning scenario.

3. The requirement of the validation dataset is also the weakness of the proposed method. The authors store and use 15% of training instances as a validation during the continual learning process. This is essential for computing scheduling rewards. To this end, the proposed method indeed needs more room to memorize previous samples compared to baselines. To this end, I'm quite negative about fairness in comparison. 

4. In general, there is a lack of analyses in the proposed method, such as 
- the size of the validation set,
- the possible design of rewards, 
- analysis of catastrophic forgetting,
- continual learning settings including class-incremental learning,
- the change of per-task performance according to the selection of adaptive task-selection schedule,
- the classwise performance when the size of M is smaller than the number of classes like the case of Section 4.5 (i.e., the case that the buffer cannot cover all classes as a replay),
...
summary_of_the_review: Motivation is reasonable and the proposed technique is simple yet seems to be effective compared to a task-equivalent selection schedule. However, there are several concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new continual learning method that learns to select samples for the replaying process. The replay memory is filled with samples from previous tasks according to a specific proportion corresponding to the action performed at the current task. The paper proposes to find the optimal sequence of actions by using Monte-Carlo Tree Search (MCTS) with the reward as the average accuracy over all tasks after seeing the final task. The experiments demonstrate that MCTS improves the continual learning performance especially when the memory size is small and is compatible with some sample-selection strategies. 
main_review: Learning to schedule replay for continual learning is an interesting and novel idea. Clear writing helps readers appreciate the importance of this topic and understand the proposed solution. The behaviour of the method is well illustrated with the analysis in Fig. 4.  Unfortunately, the experiments fall short of showing the benefit of the solution in a more realistic setting. 

Here, the paper focuses more on memory efficiency and ignore computing efficiency. It seems that training with MCTS is more expensive as multiple trials need to be done (around 40 iterations to converge-Fig. 3) in exchange for a few % of final accuracy improvement. Moreover, the performance gap reduces if more sophisticated sampling techniques are employed (Table 1). I wonder if there is still improvement as MCTS is applied to recent replay methods [1,2,3]. Also, as M  increases (e.g. up to 100-Fig. 5), the performance gap is marginal. Here, we need an explanation of the practicality of the trade-off between training and memory complexity (i.e. 1 iteration+M=100 vs 40 iterations+M=10, which one is better?). 

Another concern is the need for a complicated model like MCTS if we can perform multiple runs. In Fig. 3, some tasks only need one iteration (i.e. the found policy is random) to achieve significantly better results. To prove the benefit of MCTS, the authors need to compare MCTS with simple baselines such as random search and/or standard UCB bandit algorithms. 

 

[1] Riemer, Matthew, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. "Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference." In International Conference on Learning Representations. 2018. 
[2] Buzzega, Pietro, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. "Dark Experience for General Continual Learning: a Strong, Simple Baseline." In NeurIPS. 2020.
[3] Chaudhry, Arslan, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. "Using Hindsight to Anchor Past Knowledge in Continual Learning." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, pp. 6993-7001. 2021.
summary_of_the_review: Overall, I like the topic and proposed method. However, the exepriments need improvement. I may consider raising my score if the authors can: (1) clarify the practicality of the trade-off mentioned above, (2) combine MCTS with recent replay techniques and show clear improvement, and (3)  prove that MCTS is better than simple search/optimization methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a reinforcement-learning based approach to mitigate catastrophic forgetting by elaborating on the replay schedule for rehearsal methods. They preliminarily show that not all replay schedules are equally effective and then propose a Monte Carlo Tree Search-based approach that is shown to outperform random replay.
main_review: I overall found this work interesting and refreshing, since it proposes a novel approach to the improvement of rehearsal-based methods by inquiring what is the best order for revising past tasks. I believe that this issue is very relevant and that there is plenty of room for improvements in this specific area.
I found this paper to be very well written, well organized and easy to understand, for which I congratulate the authors.

However, I find that this work suffers from the following weaknesses:

+ The biggest concern I have with this work is that randomly sampling what information to replay might not be too relevant as a baseline. On the one hand, it is certainly true that this is how the majority of rehearsal approaches work; on the other, it is fairly atypical for these methods to make the assumptions that the authors make here, i.e. that data from past tasks are fully accessible and that the choice of what to replay is up to the model. For this reason, I think it might be appropriate to introduce a somewhat more sophisticated heuristic baseline in addition to the random choice (e.g. keeping a small validation set for past tasks and replaying the ones on which accuracy decreases below a certain level). This would make the experimental section much stronger in my opinion: the finding that the proposed approach outperforms random choice is rather expected in my view.
+ Along the lines of the previous point, I found the initial experiment to be rather unsurprising. Of course, if given the choice when to replay past data, the decision should be made as late as possible: this minimizes forgetting and lets the model come closer to the overall joint trained baseline, similar to [1].
+ I am unsure of the efficiency implications of RS-MCTS in a setting such as continual learning, which is generally quite focused on having a reduced model footprint and execution time. It would make for a very interesting ablation study to see a walltime comparison between it and random replay. I suspect that MCTS might be very demanding, which could naturally bring up the point whether simpler approximations of it could be also proposed for a more effective trade-off.

The following point were minor and did not affect my evaluation
+ I am not sure I correctly understood how robust the proposed policy is to changes in the order of classes: can the findings of Figure 4 generalize to other orders? Can I learn a policy on a specific task order and test it on another?
+ the reference to Figure 5 in the text seems to be missing.
+ While the paper makes it very clear that this approach is effective when the memory buffer is small, what does it happen when we assume that the memory buffer is very large? Do the differences between random and MCTS blur or not? This could be an interesting point to further explore.

[1] GDumb: A Simple Approach that Questions Our Progress in Continual Learning, Prabhu et al.
summary_of_the_review: I feel that the paper is relevant and clearly written, however I am insecure about its impact due to the lack of a valid baseline method and its efficiency implications.
review_title: null

Meta-review: This paper studies the problem of dynamically selecting samples to replay given that all previous data is stored. The paper shows that in this setting, selecting which samples to replay outperforms several baselines over a variety of datasets. 

I believe that the reviewers understood this work, but their initial opinions were quite mixed. 

Two of the reviewers did not "accept" this setting (all past data stored and accessible) as a reasonable one for continual learning. The discussion did not lead to a reconciliation. 

I found truth in both views. On one side, I can believe that the proposed setting has applications (recommender systems where historical data is kept seems like a reasonable one). I also find the approach reasonable since "compute" is often the bottleneck and not memory/storage. On the other, I also see that this is specializing the CL problem a bit and so, while immediately useful, may or may not help to improve more general continual-learning approaches. This is highly speculative. Another argument against this setting is that it is not absolutely clear that in this setting CL approaches are necessarily required. This really depends on the specifics of the problems.

Several of the questions and weaknesses discussed by other reviewers were also discussed and addressed by the reviewers. 

Overall, the final score from the reviewers makes this a very borderline paper. Further, even amongst the positive reviewers, one provides an overall recommendation of a 6 (marginally above the acceptance threshold). In the end, the paper was in the category of papers that were examined closely for possible acceptance, but the broad view of the area chair and the reviewers was that the paper could benefit from additional work before publication.
Decision: Reject