Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper targets the problem of Class-Incremental-Learning with two major contributions. First, the authors propose to evaluate previous methods with aligned memory budgets to fairly compare exampler-based and model-based approaches. Second, the authors conduct empirical observations on the effect of different layers during model updates. Based on the observation, the authors propose MEMO to create deep layers for new tasks to save memory budget.
strength_and_weaknesses: Strength:

(1)	Reasonable experimental analysis. The authors conduct extensive experiments to investigate the pros and cons of exampler-based methods and model-based methods given the same memory budget. It will also be interesting to know where the regularization-based methods sit.

(2)	Simple method based on interesting observation. Observation of the effect of different layers is informative. Given the observation, MEMO seems to be a reasonable proposal to save memory for the model-based method.


Weakness: 

(1)	This paper focuses on vision benchmarks, which is understandable when comparing performance. However, for observational conclusions such as "shallow layers stay unchanged while deeper layers change more during incremental learning," experiments could cover more settings. For example, have the authors looked at Vision Transformer as the transformer is also popular architecture for vision tasks? Have the authors investigated datasets from another domain, e.g., NLP? Does the observation conclusion still hold in various settings?

Questions:

(1)	MEMO only extend specialized layers to save memory budget for exemplars. One choice here is what layers are considered specialized. The last layer? The last two-layer? The authors provide a study in SectionB.1 using ResNet32. Does the result here generalize to other ResNets and other architectures? What would the authors recommend as the rule of thumb in deciding the specialized layer?

clarity,_quality,_novelty_and_reproducibility: Quality: Overall quality is high. Related work seems to cover reasonable background information. Figures are nicely presented and easy to read. Visualization, such as Figure7, helps understand the method.

Clarity: Paper is generally well-written with no obvious grammar mistakes. Figure 6 may need some clarification.  I understand how the data are split, but I don't quite sure about the comparison setting. Are the methods compared under the same memory budget? If yes, how did you decide on the memory budget? If not, what are the hyper-parameters of each approach?

Novelty: Novelty is fine. Layer-wise decompose is not new, however,  the benchmark result and empirical observation may be interesting for the community. 


summary_of_the_review: Overall, this paper could bring useful insight to the community. (1) It provides a practical evaluation perspective to compare orthogonal direction. (2) Its empirical observation on the gradient norm and shifting range could help understand the training dynamic of Class-Incremental-Learning. (3) The proposed method is a simple and intuitive next step for model-based methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to tackle the class-incremental learning (CIL) problem from a novel perspective. The current class-incremental learning community is somehow chaotic, where more and more methods tend to design extra model components. These additional components are not counted in the total memory budget, resulting in an unfair comparison between different methods. To this end, this paper unifies the comparison of different methods by introducing the extra dimension for comparison, i.e., memory budget. By varying the memory budget of different methods and designing new performance measures, this paper aims to evaluate incremental models holistically. The results are diverse, where strong models shall fail with limited memory budgets. These results inspire the authors to analyze the importance of each part in CIL and design a simple yet effective method that suits the characteristics of CIL models. Vast experiments on benchmark datasets verify the effectiveness of the proposed method.
I really like the way that the authors study the CIL problem. They performed very systematic and reasonable experiments. The second part of the paper, which looks at the characteristics of different layers, is interesting. Also, the proposed approach (MEMO) just decouples the representations and expands the task-specific layers to strike a trade-off between model expansion and exemplar saving. It is a very interesting way to solve the problem.
strength_and_weaknesses: Strengths:
- New empirical findings shall shed light on the class-incremental learning community, e.g., the experimental evaluations in Figure 3 show that deep and shallow layers have different characteristics in CIL. The results are clear and straightforward, which correspond to the intuitive solutions in model expansion.
- Intuitive method to solve CIL problem. Observing that shallow layers yield higher similarities than deeper layers, the authors argue that expanding deep layers is enough for CIL. The extra memory budget for expanding shallow layers can be exchanged for saving exemplars, which is more memory-efficient. Experimental results verify that the current method can boost performance for free with the same budget.
- Novel performance measures to holistically and fairly compare different CIL methods. I agree with the point that current CIL methods are not compared fairly, especially for the model-based methods with multiple backbones. I think the new performance measure and the way of model evaluation shall inspire the community to design fair comparisons in the future.

Weaknesses:
- The current evaluation protocol depends on some specific and unusual network structures, e.g., ConvNet2, ResNet14, etc. As a result, it would be better for the authors to release the systematical codebase for this research in the final version, which will ease the burden when calculating these new performance measures.
clarity,_quality,_novelty_and_reproducibility: The contributions are significant and do not exist in prior works. The paper is well-written and easy to follow. The details in this paper are enough to reproduce the experimental evaluations.
summary_of_the_review: Finally, I found the paper clear and easy to read. All the experiments are detailed, as well as the thought process behind them. I think this is a thorough study of a very important problem and can improve many papers that are based on it. As a result, I vote for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors explore how to balance the memory budget usage between exemplars and model parameters in class-incremental learning. Based on their findings, the authors propose a simple yet effective class-incremental learning method named Memory-efficient Expandable Model (MEMO). MEMO extends specialized layers based on the shared generalized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on benchmark datasets validate MEMO’s competitive performance.
strength_and_weaknesses: ### Strengths

- It is very meaningful to explore how to balance the memory budget usage between exemplars and model parameters in CIL. Saving the model parameters definitely requires a memory budget.

- This paper is well-written and easy to follow. 

- Extensive visualizations and figures are provided. 

### Weaknesses 

- It would be better to discuss some strategies that help to decide which blocks should be frozen. In Figure 5, we can observe that different blocks should be frozen in different settings (e.g., Base0 vs. Base 50). However, we cannot know these settings in advance in real-world applications. Thus, it is important to design some methods that help us to decide how many blocks should be frozen. 

- It would be better to include a table in the main paper to show the accuracies of different baselines. Currently, Figure 6 shows the Top-1 accuracy. However, it is more clear to have a table with numerical results.

clarity,_quality,_novelty_and_reproducibility: ### Clarity & Quality

This paper is well-written and easy to follow. 

### Novelty 

Evaluating the memory allocation between the exemplars and model parameters is an interesting direction.

### Reproducibility

The authors provide detailed configurations in the appendix. 
summary_of_the_review: Overall, I think this is a high-quality paper. It is well-written and easy to follow. The task studied is very meaningful in my view. I recomannd acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: It is known that methods that store examples in memory have positive results for the Continual Learning problem, specifically in Class Incremental Learning. Instead of saving examples from previous tasks, some works recommended storing model checkpoints to retain the information. The authors of this paper point out that unfair comparisons are often performed since the model buffer is not always considered part of the saved memory. Because of this, the authors propose to compare the methods by adding both memories (exemplar and model), revealing that weight-saving methods tend to suffer when memory is limited. With these results, the authors propose MEMO, a more efficient method to store checkpoints. MEMO takes advantage of the fact that the first tends to have generic information, keeping those shared across tasks and only adding later layers to incorporate specific knowledge. The newly freed memory can then be used to save more examples of previous tasks. MEMO achieves comparable results and sometimes outperforms current methods in different scenarios.
strength_and_weaknesses: S:
- Based on current works, the community agrees with the authors that there is a need to improve how we compare methods. A suitable example of this is the framework proposed in [1].
- The idea of reducing the number of layers saved in the "memory buffer" to save more examples or using better models is good. The authors' experiments that show that these layers store general information and take advantage of that, are intriguing.

W:
- The CL definition is unclear, making it unclear whether or not the authors employ task-ids during inference. If they use the same definition as Yan et al., 2021. One can assume they do not use the task id. However, it would be helpful to make it clear.
- "… Since a single backbone can depict a limited number of features, learning and overwriting new features will undoubtedly trigger the forgetting of old ones …" I would be careful with these kinds of statements. It may seem accurate at first, but we do not know how many concepts can be stored in each feature. For example, the model rarely used the entire weights based on the Lottery Tickets Hypothesis. Therefore, treating this as a limitation without giving further reasons is complex.
- In implementation details, it would be helpful to know how many times the experiments were run. Running the experiments only once is not good practice in CL, since the variance between different initializations and task orders can be significant.
- The motivation of Fig1 is straightforward but may be out of place. Several things are not explained until Sec4 (for example, Base50, Inc5, or MEMO). The explanation in the text could be improved or Fig1 could be left in Sec4.
- In sec4.2. Since the authors advocate a fair comparison, I do not think it is right to change the base model. Each model has its characteristics that can harm or benefit some methods or scenarios. The results in Fig2 validate this concern.
- Sec4.3. The experiments are interesting, but as the authors mention, it is something that is known. It may be possible to reduce this section and add experiments about how to properly divide general and specific layers. There is no mention in the paper of how this division is made.
- Sec5.2. If models are in a similar memory context, why not just compare ACC? The motivation for using AUC is not straightforward.

Q:
- Understand that this paper and [2] are different. In your opinion, what is the motivational difference between them?
- Could give an intuition for why not freezing the general layers improves the acc? If there is general knowledge in those layers, why change it? How much can we modify those layers and still prevent forgetting?

[1] Mundt, Martin, et al. "CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability." arXiv preprint arXiv:2110.03331 (2021).
[2] Ostapenko, Oleksiy, et al. "Continual learning via local module composition." Advances in Neural Information Processing Systems 34 (2021): 30298-30312.
clarity,_quality,_novelty_and_reproducibility: The authors present their motivations clearly, however the contributions are not completely new, they are a small step to what was already known.

I leave a few questions to the authors regarding the experiments.I think the experiments are good, but they are not capable of answering all the questions raised as motivations by the authors.
For example, one of the questions that I did not see answered is where to make the cut between global and specific weights. I think this is a fundamental point that the authors do not dispute. I understand the space limitations, but it is a very important point.
summary_of_the_review: Despite having limited contributions in my opinion, the ideas presented are good and may be interesting for the community.

However, the authors leave many questions raised that, if answered, could significantly improve the work. In my opinion, this research work is being developed and has not yet reached its maximum potential.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary of Contribution

This paper addresses the trade-off between memory budget and performance in class-incremental learning (CIL). The authors begin with a critical examination of how overall CIL memory budget should be determined by considering both model parameters and exemplar memory size. They propose a simple (but effective) baseline for their comparisons based on an expandable memory (MEMO). They also propose new metrics to analyze the trade-off between model size and exemplar memory in terms of CIL performance. Extensive and critical experimental results are given for MEMO and a variety of consolidated baselines from the state-of-the-art. The experimental comparison is given on small- (CIFAR) and large-scale image (ImageNet) recognition benchmarks.

# Strengths

+ **Clarity**: The paper is very well-written and the main technical points and arguments are well-motivated from first principles. The visualizations are well thought-out and complement the analyses and experimental results throughout.

+ **Experimental Analysis**: The main contribution of the work is in the empirical analysis of the model/exemplar memory trade-off. The experiments and accompanying analyses shed light on one of the key issues in class-incremental learning, and although the problem of memory/model trade-off is well-known this seems to be the first very thorough and decisive exploration of this aspect. 

+ **Impact**: The metrics and new baselines proposed in the paper are likely to have lasting effect on the broader community involved in incremental learning research (and potentially also in practice). Though not entirely comprehensive in its approach, this work represents an excellent starting point for deeper understanding of the many trade-offs in continual learning.

# Weaknesses

+ **Non-vision Benchmarks and Other Architectures**: The paper exclusively addresses vision benchmarks using CNN backbones. During the discussion period the authors performed additional experiments using Transformers (both ViT and BERT), providing some evidence that the main conclusions of the paper hold also for other tasks and backbone architectures.

+ **Mitigation Specifics**: The contribution would be stronger if the experimental analyses delved deeper into more specifics on *how* to divide between specific and general layers.

+ **Other Orthogonal Directions**: This paper looks at one specific trade-off in class-incremental learning. A discussion on how other types approaches (e.g. regularization) could be brought into the same analytical framework.

# Summary

The reviewers are nearly unanimous in their very positive evaluation of the paper. During a very lively back-and-forth during the discussion period the authors addresses all of the main reviewer concerns, provided selected new experiments on new tasks and using new backbones, and in general significantly improved the presentation and clarity of the underlying contributions of the work. The research community in continual learning is coming out of an expansive, breadth-first phase of exploration and experimental search for effective techniques. This work attempts to abstract, generalize, and accurately measure one specific aspect of these recent advances, which is something very useful to the entire community working on class-incremental learning.

justification_for_why_not_higher_score: I think this paper could make a decent oral presentation, however it is fairly narrow in scope as it looks at a specific problem in class-incremental learning and makes no general, technical contribution (the methodological contributions are its strength). Thus I feel it has somewhat limited appeal to the broader ICLR audience.
justification_for_why_not_lower_score: The research community in continual learning is coming out of an expansive, breadth-first phase of exploration and experimental search for effective techniques. I think this paper represents an important step towards consolidating results from this exploration phase, a step that should provide a more solid foundation for moving forward.
Decision: Accept: notable-top-25%