Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use C-scores as a score to select representative samples to insert into a memory buffer in continual learning. More concretely the authors:
- describe current limitations of memory selection methods
- propose to use C-score, an expensive but accurate score capable to identify learnable samples, to select a memory that is highly representative of the class and can be efficiently learned
- identify two limitation in a naive use of C-scores and try to alleviate them:
    - C-scores are expensive to compute, and the authors propose more efficient proxies to approximate them
    - selecting a memory greedily w.r.t. C-scores can result in lack of diversity, so the author propose to select uniformly at random (promotes diversity) but only out of samples that achieve a C-score above a certain threshold (promotes relevancy) 

The proposed method is evaluated using several ablations and baselines, but only within the context of other memory-based CL methods.
strength_and_weaknesses: The strong points of the method are its conceptual simplicity (C-scores are an intuitive metric of relevance) and good performance compared to other memory-based baselines. The motivation section is especially insightful and helps to highlight how the memory selection problem is still very open and hard in general. The algorithm exposition is clear and its most relevant hyperparameter are ablated. The experimental section covers multiple datasets.

However the work also has some clear weaknesses
- One of the main contributions is the derivation of new proxies for C-scores, but the approximation accuracy (e.g. proxy C-score/actual C-score) of these proxies w.r.t. the real score or Jiang et al.'s approximation. The only metric reported is downstream accuracy of the classification task, but without checking the quality of the C-score approximation it is possible that the proxies are injecting additional biases that could actually be helpful for these specific tasks.
- The memory budgets considered in the experimental section are quite small (i.e. ~100-1000), and much smaller than what off-the-shelf hardware is capable of handling currently (i.e. 10k and up). It would be interesting to see how the various proxies and the baseline behave with a larger memory, and if the gain are preserved.
- For the real algorithm (i.e. the one using C-score proxies) only accuracy is reported, while usually the more relevant metrics in CL are forgetting or some sort of transfer.
- None of the experiments are reporting error bars or uncertainty, and it is not clear if the experiments with proxies have also been run with 3 seeds.

Minor comments:
- The baselines considered are only memory-based. While this is acceptable (CL is a broad field) it is a bit of a limitation of the comparison.
- The paper seems to only consider the setting where sample are explicitely labeled with their tasks during training. This is acceptable but it should be stated that it is a special case of the more general CL problem without explicit tasks or task boundaries.
- Accuracy and forgetting are never defined as metrics in the paper.
- In A.2 CAWS should include points with C(x) >= delta
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear and the results are interesting. The novelty is limited since C-score were already applied in CL, and the proxies are relatively straightforward. The authors included code for their experiments so reproducibility should not be an issue.
summary_of_the_review: Overall the paper presents an interesting approach and is overall acceptable, but the empirical evaluation undermine the message that C-scores are an interesting metric in CL. In particular
- the quality of the proposed proxies are never truly investigated
- it is not clear which computational regime the authors are targeting for their approach, and they should justify their choiches for memory size, number of epochs, and size of datasets in each epoch in terms of memory and compute, providing back-of-the-envelope estimations supporting their choice (e.g. if this approach should target compute on a phone, a consumer desktop, an industry server, or a whole cluster).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an idea to select important examples to be buffered in continual learning. The proposed Consistency Aware Sampling (CAWS) considers examples that are easy to learn and representative of previous tasks. Empirical investigation has been conducted to evaluate the effectiveness of the proposed method.
strength_and_weaknesses: The main strength of the paper is the topic itself, selective buffering in continual learning.

However, the weaknesses of the paper overweigh the strength by a large margin:
- The organization of the paper is scattered and hard to follow. Interleaving methods and experiments are not clear. Moreover, the writing style makes it hard to distinguish the motivation and original parts of the paper.
- Saving the "easy to learn" examples in the buffer is counter-intuitive. An easy-to-learn example should already been learned by the model in early iterations, why bother saving it in the replay buffer instead of giving the model more exposure to hard examples? 
- Lack of latest competing methods. As far as I am concerned, there has been recent work that studies the same topic as this paper, including but not limited to GSS [1], LARS[2] and DDR [3]. The authors should definitely compare the proposed method against these recent methods.


[1] Aljundi, Rahaf, et al. "Gradient based sample selection for online continual learning."NeurIPS 2019.

[2] Buzzega, Pietro, et al. "Rethinking experience replay: a bag of tricks for continual learning." ICPR 2021.

[3] Wang, Zifeng, et al. "SparCL: Sparse Continual Learning on the Edge." NeurIPS 2022.
clarity,_quality,_novelty_and_reproducibility: The overall clarity and quality of the paper do not meet the ICLR standards.
The novelty of the paper is quite limited.
The reproducibility is not ensured since the codebase will be public after acceptance.
summary_of_the_review: Although the topic of selective buffering is interesting, the proposed method is not well justified neither intuitively or empirically. Therefore, I recommend rejection of the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates continual learning under limited memory setup  and proposes a storage policye CAWS (Consistency AWare Sampling), that leverages a learning consistency score (C-Score) to enable effective and efficient learning. CAWS adds diversity to the memory while allowing the model to learn a far more detailed decision boundary. Practical proxies that require no extra training can still achieve similar results according to some experimental results.
strength_and_weaknesses: Pros: 
The paper focused on a very interesting scenario in continual learning. The problem setting of continual learning with efficient learning is well motivated. It is reasonable to apply C-score in evaluating the learnablity of a sample and this method could be extended to other areas including Curriculum Learning.

Cons:
1. The structure of the paper is not clear. The authors proposed to use C-Score to measure how learnable a specific sample can be concerning a set of models, and C-Score of some datasets have already be calculated. They mentioned they needed a proxy to evaluate the score for the other datasets, but the definition of the proxy appear after several sections. The writing order will put extra reading burden for the readers. Also the original paper appears in Section 4.1, while the C-Score definition appears in Section 2.

2. For the datasets we already have the C-Score computed by Jiang 2021, does this C-Score rely on the model architecture? If we switch to a new architecture, does that mean this C-Score does not work any more? 

3. There are a few proxies proposed by Jiang 2021, what is the difference between the proxy in this paper and the proxies in Jiang 2021?

4. How to determine the threshold of high C-score and low C-score? How to determine the X in CAWS? 

5. The results in Figure 4 a and Figure 4 b are leading to two different directions when we increase the threshold. Could you please explain why the 1000 memory capacities would increase with threshold = 0.9 in the CIFAR100 and all the other curves seens going downwards?
clarity,_quality,_novelty_and_reproducibility: Although the problem setting is new, my main concern is the limited novelty of the proposed CAWS method.The proposed CAWS method, as illustrated in Section 3, simply sampling randomly from the top C-Score samples.

- The paper is not easy to follow because they split the method and the results into multiple pieces, and some references and definitions are not set close to their first appearance.
- How the dataset is splited into 5 tasks is not explained explicitly. 
- Why the accuracy would decrease with more training epochs in CIFAR-10-100 in Figure 3？
summary_of_the_review: Overall, even though the problem is important, I find the structure of the paper needs further modifications. The method CAWS they proposed seems like just sampling from the top C-Score samples under the continual learning framework, I think the authors could put more emphasize on the novelty of this method and how much of the memory size is saved or the performance boost in using CAWS sampling. 
On top of this, there are a few technical aspects (inconsistent results, hyperparameter thresholds etc) I do not completely agree with, therefore, would request authors to answer above questions for clarity.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper explores the idea of addressing catastrophic forgetting through rehearsal using a memory buffer. It studies strategies to select the most important samples for storage in a memory buffer. The core idea is to design a consistency score that ranks samples for storage according to how easy they are to learn and at the same time are representative of previous tasks. Experiments using three common continual learning benchmarks are performed to demonstrate that the proposed method is effective.
strength_and_weaknesses: Strengths:

1. The paper is clear and easy to follow.

2. Experiments on three major continual learning frameworks are provided.

Weaknesses:

1. The idea is incremental.

2. The approach is heuristic and without much theoretical justification.

3. Comparison against prior methods is extremely limited.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. However, the novelty of the paper is incremental. It merely proposes another heuristic approach for selecting samples for experience replay. However, the strategy is not supported by theoretical justification. Experimental results do not demonstrate that the method leads to competitive performance against prior works. Hence, the novelty and contribution of this work are limited and do not offer an improved understanding of the sample selection strategies for continual learning.
summary_of_the_review: The idea of experience replay using selected samples is not new and many existing works have been proposed. The proposed idea is a heuristic idea without no theoretical justification. Experiments also do not demonstrate that the proposed method leads to state-of-the-art performance. Hence, the paper does not provide theoretical novelties or sufficient empirical results to demonstrate its effectiveness. In conclusion, it is not ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers agreed that the novelty of this work was limited and the proposed method lacked a theoretical justification. Reviewers also found the experimental validation and ablation studies to provide limited insights in the practical trade-offs. Reviewers appreciated the author's rebuttal as well as the work done to ensure results are reproducible, but overall concluded that this work was not ready for publication.

justification_for_why_not_higher_score: Only one reviewer voted for acceptance and he did not try to sway other reviewers.
Decision: Reject