Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new memory sampling strategy which replaces random/herding. The strategy consists of two parts: first level determining the distribution of #samples of each class in memory and second level determining samples to be selected for every class. This is learnt via RL policy. The sampler is trained by emulating a pseudo-CL problem with the data available in the pretraining phase-- where there are an equal number of tasks made as expected in the CL setup to enable transfer. Given a sampling configuration, a CL method trains the model and the performance on the pseudo-CL test set is given as reward to the RL algorithm. This trains the policy components, which optimize sample size and samples per class and train CL. Then, the best policy is used to create the memory which is used to train the actual CL problem with state-of-the-art continual learning approaches. This memory selection achieves better performance than previously used memories across CIFAR100, Imagenet100 and Imagenet1000 datasets.
main_review: 2.1) Well written: The paper is very well written and was an enjoyable read. The contributions are clearly stated, acknowledgement is given to the existing body of work to the best of my knowledge. Experiments back motivation the problem as well, demonstrating the need. The paper then proceeds to describe a sufficient solution, results are clear and well-presented.

2.2) Novel idea: The paper introduced a an interesting idea which might seem currently inefficient but can potentially be the first step which can result in nice & efficient future works about memory selection aspects of CL (similar to NASNet {1} for the NAS domain)

2.3) Thorough ablations and supporting experiments: The paper analyzed the proposed approach well ablating each of the components and comparing with a maximum limit (Table 2) and exploring the memory size (Figure 4)/growing memory (supplementary) pre-emptively answering many of my questions.

Weaknesses:

3.1) Potentially violating the CL protocol [Critical]: Since the future is unknown-- continual learning formulation and in real world, formulations don’t have access to how many tasks will arrive in the future. Isn't using this information violating the CL protocol? (and this seems to be critical to the proposed method)

3.2) Complexity and expensiveness don’t justify performance improvement [Major]: The introduced method requires extensive retraining of models to learn the optimal policy function. Furthermore, the memory selection process seems very complex in design for a memory sampler. The accuracy increase is relatively minor, which makes me far less excited about the proposed approach. Another major shortcoming is this requires datasets which have good transferability properties (NAS literature demonstrates that CIFAR and Imagenet happen to have similar properties so transfer works between them, but it's often unlikely)

3.2.1) Why not only use the first level and not second-- it would massively simplify the learning complexity and time while still preserving most performance increases (acc. to Table 2).

Overall:

This seems to be a well-written and explained paper with a novel idea which bridges two domains together and could potentially allow interesting directions to explore. However it seems to me that (i) it violates the CL protocol in a way which is critical to the method’s performance (ii) performance improvements does not justify the complexity introduced: it's very expensive to train and complex in design, the improvement in accuracy is not as great which makes me less excited (if more parameters & information are introduced, improvements in performance are expected). However, I might have misunderstood some of these claims. I request the authors to clarify the points presented in the weaknesses, so that I can consider it in my final rating. I am not familiar with RL literature and hence my feedback is primarily about the CL aspects of the work.

{1} Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, Learning Transferable Architectures for Scalable Image Recognition, CVPR18

Post rebuttal:

I am somewhat convinced by the author's rebuttal. I presumed assuming the number of tasks was critical, but it was not. I am quite surprised by the generalizability of the proposed approach. 

I suspect that there might be a simpler phenomena happening here aiding the generalizability-- eg: balancing to keep extra samples of older classes in a certain manner or picking certain kind of samples. Discovering what it is might vastly simplify the approach and lead to better understanding of the underlying mechanism. This is potentially nice future direction to build upon, and I encourage the authors to pursue it if possible.

Overall, the method seems too complicated for its performance gains currently. However, I think this provides a method for exploring this space and uncovering better sampling strategies. It is generalizable enough to be valuable tool for the community without violating any critical assumptions. I would upgrade my rating to 6.
limitations_and_societal_impact: No substantial negative societal impact that must've been discussed in my view but the section was missing from the draft.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work addresses the problem of memory management in Class Incremental Learning (CLI). In this setup, models store exemplars from old classes and must decide what exemplars to drop to make space to new memories. The authors propose a hierarchical reinforcement learning approach where an agent first chooses how much of the old memory to release to make space for new samples and then it chooses how much of the new space allocate to high-entropy samples and low-entropy samples (they divide new classes in two groups). Experiments show that the the proposed method attains better performance than POD-AANets and LUCIR on CIFAR and ImageNet. In the supplementary material they show that the proposed memory management system also improves the performance of already-existing systems. Additional studies show how policies transfer between datasets, the impact of hierarchical RL, and how old and new memories balance over time.

main_review: # Overall Review
The proposed method is sound, interesting, and it improves the performance of class-incremental learners on multiple datasets. The authors provide ablations, a detailed description of their algorithm, and code. I have some questions about the level-2 actions and the chosen baselines but for the moment I recommend to accept this work. 

## Strengths
* The proposed method is technically sound and improves the performance of CI learners.
* The text is clear and easy to understand, the authors did a good job explaining their method.
* The authors provide an ablation study

## Weaknesses
* In the text you cite other CIL works such as DER [45] that you do not include in your tables. Could you explain why? I suspect it is due to a difference in setups but some explanation would be welcome by me and those who are not experts in the particular field of CIL.
* The use of the entropy for splitting the data in groups is not well-motivated. Could you reference previous work or include some experiment showing that using entropy is better than directly learning to predict a score?

# Detailed comments
## Originality
* The authors apply existing techniques in RL to the memory management problem in CIL but to the best of my knowledge they are the first to do such thing.

## Quality
* The overall technical quality is good. My only concern is that the group separation by entropy is not explained.

## Clarity
* In general, the text is well-written and easy to follow. 

## Significance
* The proposed idea is sound and interesting for the research community. The experiments show significant improvement. 

## Reproducibility
* The authors did a good job towards reproducibility. They include an Algorithm, the code, the implementation details, and how to execute the code in the supplementary material.



limitations_and_societal_impact: The authors vaguely discuss limitations and potential negative societal impact in the supplementary material. They could broaden the discussion by including some discussion about the additional computational cost of training a model with their method or the privacy issues when keeping exemplar data.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a RL-based memory management policy for CIL problem. They have two-step formulation of policy, which first determines how to allocate the memory to old and new tasks, then determines how many samples to store per each class within the task. A standard RL framework of policy gradient method is used to optimize their policy. Experimental results how that their RMM can further improve the recent state-of-the-art methods, such as LUCIR or PODNet. 
main_review: Applying RL for the memory management is novel, and it does seem that the CIL performance improves with the proposed method. However, my major concern is that the policy should be learned with a pseudo-CIL tasks, utilizing the separate or base classes. Namely, one needs to build exact same setting (i.e., with the same number of incremental learning phases, N) as the target CIL setting, and it does not seem to be practical. At least, it would have been nice if the experiments have the results for the mismatched scenarios -- namely, for the case in which the setting of target and pseudo-CIL is different. 

Followings are some more comments:

Pro:
   - the problem of memory management for CIL and applying RL is new. 
   - Hierarchical formulation of RL is also new. 
   - Positive experimental results are shown. (Improvements over the state-of-the-art)

Con & Questions:
   - Major concern is mentioned above. CIL capability should not be confined to the benchmark data setting (Requirement 2) in line 206). Namely, what if the number of phases N becomes 13 or 17, for example? It seems like the whole RL policy should be re-trained, and this is a critical limitation. 
   - How does the performance of RMM change with the total memory budget? Maybe, if the total memory budget increases, the effect of RMM would decrease? 

Overall, I think the paper is on the borderline. While the setting and methodology is novel, the practicality of the method seems to be limited. I would also see what other reviewers think and finalize my rating. N/A

----
I have read the author's rebuttal and finalized my score to 6. Thanks. 

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: A common approach in class-incremental learning is to retain a fixed set of examples of the old classes to help avoid forgetting. The paper presents an approach to dynamically retain exemplars for the class-incremental learning problem. Reinforcement learning is used for dynamic memory management.
main_review: Pros:
+ The idea of using reinforcement learning for memory allocation is interesting. The way in which policy function is learned on pseudo-CIL tasks is novel to my knowledge. 
+ A bi-level action space is defined i.e., first deciding the split between old and new classes and then the memory allocated to each specific class. 
+ Extensive experiments are provided on three splits and two datasets. 
+ The generalizability of the proposed approach is showcased by plugging it in two recent baseline CIL approaches [25+18, 25+13]. 
+ Experiments with fixed and growing memory budgets are also performed, the proposed approach appears to help in both cases. 

Cons/Questions:
- One main issue I see with the current approach is that the policy is learned on pseudo tasks which may have totally different task distributions, intrinsic properties as compared to the future tasks that we are going to encounter. How much the approach is sensitive to such distribution shifts, e.g., can we learn a robust policy on an altogether different distribution or a set of classes that are much different from the ones going to be encountered in future tasks (as it is quite likely in a realistic setting)?
- RL component will result in additional computational costs. I could not find any comparisons on that. It will be useful to compare the training times with recent approaches and also report the overhead caused by policy learning.
- The paper mentions that the policy can be learned with either an in-domain or out-of-domain dataset, there is a single ablation experiment shown to support this. Can authors provide additional experimental results to support this claim, e.g., leveraging a medical-scan dataset, stylized ImageNet or iNaturalist?
- Since Table 1 reports average accuracies, can the authors report confidence intervals or standard deviations over all tasks (for all/available methods)?
- The memory budget in Table S4 seems to have used high storage. Can the authors confirm its in line with the literature? My understanding is that it's the total memory and not necessarily the memory used for storing the exemplars only. If this is the case, it will be clear to provide exemplars only memory statistics as well. 
- In figure 1, it will be helpful to annotate the meaning of different colors (i.e., old exemplars, new exemplars etc.). 
- The related works section can be improved with related papers:
-- Zhu et al. Prototype Augmentation and Self-Supervision for Incremental Learning. CVPR'21.
-- Wu et al. Incremental Learning via Rate Reduction. CVPR'21.
-- Simon et al. On Learning the Geodesic Path for Incremental Learning. CVPR'21.
-- Iscen et al. Memory-Efficient Incremental Learning Through Feature Adaptation. ECCV'20.
-- Rajasegaran et al. Random path selection for incremental learning. NeurIPS'19.

limitations_and_societal_impact: The limitations are not listed. Societal impact questionairre is referred to as supplementary material, it would have been nice to answer the questions in the main paper.
review_title: null

Meta-review: This paper proposes a reinforcement learning-based for memory management in Class-Incremental Learning (CIL). The reviewers find the paper well written and the idea novel. They also think that the experiments are thorough and positive. There were initially some concerns on the generalizability of the proposed method. The reviewers are generally satisfied by the author responses on the issue. Despite remaining reservations regarding the complexity of the method relative to the performance gains, there is a clear consensus among the reviewers that the paper should be accepted.


Decision: Accept (Poster)