Coreset Selection via Reducible Loss in Continual Learning

Authors: Ruilin Tong, Yuhang Liu, Javen Qinfeng Shi, Dong Gong
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual learning, Coreset selection

Abstract:
Rehearsal-based continual learning (CL) aims to mitigate catastrophic forgetting by maintaining a subset of samples from previous tasks and replaying them. The rehearsal memory can be naturally constructed as a coreset, designed to form a compact subset that enables training with performance comparable to using the full dataset. The coreset selection task can be formulated as bilevel optimization that solves for the subset to minimize the outer objective of the learning task. Existing methods primarily rely on inefficient probabilistic sampling or local gradient-based scoring to approximate sample importance through an iterative process that can be susceptible to ambiguity or noise. Specifically, non-representative samples like ambiguous or noisy samples are difficult to learn and incur high loss values even when training on the full dataset. However, existing methods relying on local gradient tend to highlight these samples in an attempt to minimize the outer loss, leading to a suboptimal coreset. To enhance coreset selection, especially in CL where high-quality samples are essential, we propose a coreset selection method that measures sample importance using reducible loss (ReL) that quantifies the impact of adding a sample to model performance. By leveraging ReL and a process derived from bilevel optimization, we identify and retain samples that yield the highest performance gain. They are shown to be informative and representative. Furthermore, ReL requires only forward computation, making it significantly more efficient than previous methods. To better apply coreset selection in CL, we extend our method to address key challenges such as task interference, streaming data, and knowledge distillation. Experiments on data summarization and continual learning demonstrate the effectiveness and efficiency of our approach.

Decision: accept (poster)
Reviewer scores: 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730596964722

thread_type:
official_review

<thread_object>
creator: reviewer_ungg

<content>
summary:
The authors focus on the coreset selection method used in continual learning scenario. They believe that the not well learned samples, like ambiguous and noise samples should not be included in the coreset. Therefore, a new method CSRL is designed with reducible loss to filter the not-well-learned samples during coreset selection. The experiments are conducted on MNIST, CIFAR-10, and CIFAR-100 with several earlier competitor methods.

strengths:
1. I find the authors make great efforts in polishing this work. I found multiple supplementary explanations for the main text in the supplementary materials, which I believe is commendable and should be encouraged. 
2. The paper is well-organized and writing is almost clear.

weaknesses:
About Method:

1. The proposed CSRL seems to be severely affected by the selection order and subset division way. If Algorithm 1 is correctly implemented, this means that the selected samples in the coreset will not be considered in the following selection process. Obviously, the gain score G of samples stored in the coreset will change as the model gradually learns from all subsets. The authors ignore this possibility, which may lead to the result that previously selected candidates are more likely to be stored in the coreset. In other words, the gain score may not be fair for all candidates.

2. The gain score G seems only to assess which samples are easier to learn fully, rather than which are more core. From the expressions in Equations (4) to (6), it is evident that the authors do not differentiate between difficult samples and noise when applying this gain score, instead treating both as noise. Clearly, difficult samples should also be treated appropriately during coreset selection.

3. I notice that the authors provide the results in Appendix H. However, I believe these results do not verify that CSRL is more reasonable and may even exacerbate the concerns I raised in W1. Since the overall difficulty level of the entire dataset is not provided, we cannot determine whether CSRL regulates the coreset selection more effectively. As I expressed in W1, the results can also be interpreted to mean that CSRL is severely affected by the selection order (which seems to be random) during application, leading to selection results that might also be random as the size of the coreset increases. This suggests that CSRL has less impact on regulating coreset selection. In contrast, the other competing method appears to perform better. Additionally, the authors claim that selecting fewer hard-to-learn samples will make a difference, which further confirms that hard samples are considered the same as noise samples.


About Experiments:

4. I noticed that the settings used by the authors are somewhat outdated. First, the largest network structure in this paper is only ResNet-18, and the authors further provide training and selecting time results based on this backbone to show the efficiency of their method. From both the performance and efficiency perspectives, I find this unconvincing. With such simple network structures, it is difficult to compare performance differences between new and old methods. Let alone that many recent state-of-the-art methods are not included in the main experimental results.

5. In terms of the experimental results, I find them unconvincing because the authors only adopt competitor methods from 3 and 5 years ago in all the experimental results presented in the main text. It becomes even more puzzling when I notice that the authors are aware of the latest state-of-the-art methods (e.g., BCSR, GCR) but only compare CSRL with them on one of the datasets used in BCSR in the appendix. Why are the latest state-of-the-art methods not compared under the same settings in this paper's main text? I believe that both the absence of state-of-the-art methods in the main text and the partial comparison results of these methods shown only in the appendix do not support the authors' claim that the proposed CSRL is state-of-the-art.

questions:
Please see my questions in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627334128

thread_type:
official_review

<thread_object>
creator: reviewer_kuqk

<content>
summary:
This paper introduces a method for coreset selection using reducible loss to improve continual learning (CL) performance. Traditional experience replay-based CL methods face challenges when selecting representative samples from non-stationary data streams, often retaining ambiguous or noisy data that can degrade model performance. The proposed method uses reducible loss as a selection criterion, estimating each sample's performance gain to determine its representativeness and informativeness. This selection framework is computationally efficient, requiring only a forward computation without gradient backpropagation, thus avoiding the computational cost associated with implicit gradient methods. Additionally, the approach is extended to handle challenges in CL, including task interference and data streaming, demonstrating robustness against noisy data and effective scalability for large datasets like ImageNet. Experimental results show that this method consistently improves CL model accuracy and reduces forgetting when applied to various datasets.

strengths:
This paper introduces a coreset selection method based on reducible loss for continual learning, aiming to improve model performance by selecting representative samples efficiently. By focusing on performance gain, this approach reduces computational demands and mitigates the impact of noisy data. It addresses key continual learning challenges, such as task interference and data streaming, making it particularly valuable for dynamic and large-scale applications.

weaknesses:
The coreset selection method based on reducible loss presented in this paper shows promise for continual learning applications, but some of the design choices lack thorough validation, and certain aspects of the experimental comparisons and theoretical evidence could be more comprehensive.

questions:
1.	This paper builds upon the limitations identified in Greedy Coreset (NIPS 2020). However, given that four years have passed since its publication, does this work lack timeliness? Can its performance surpass current state-of-the-art methods?
2.	This paper uses a coreset-based replay method, so it should include more comparative experiments with other coreset selection approaches. The comparative experiments in this paper are insufficient.
3.	Since the paper mentions that reducible loss eliminates the need for backpropagation and reduces computational cost compared to traditional bi-level coreset selection strategies, an additional experiment should be included to validate this conclusion.
4.	Does training a holdout model using all the data contradict the continual learning (CL) setting?
5.	The paper mentions that “samples with high performance gain are both representative and informative.” Could the distribution of high-performance samples in feature space be provided as evidence for this claim?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667770110

thread_type:
official_review

<thread_object>
creator: reviewer_muyb

<content>
summary:
The work presents a concise way of preventing select noisy data into the coreset due to their high loss value. By using a holdout model as the baseline, it incrementally selects samples with high log prob loss. Which is mathematically shown by the authors that such samples are informative but not well represented yet in the coreset. The work shows improvements on various common continual learning datasets.

strengths:
1. A detailed, structured, and rigorous paper with extensive derivations and experiments.
2. Conducts a variety of experiments, including those on standard continual learning datasets, more challenging ImageNet variants, training overhead, and extensive ablation studies.
3. Presents a clear and straightforward idea for selecting good samples to add to the coreset, supported by sound equations.

weaknesses:
1. Not easy to understand at the first glance. It might be better to add some teaser figure.

questions:
1. How do you train the holdout model? I believe it is trained only on the initial dataset, without any information about the incoming task. What if the holdout model is not good enough to calculate a meaningful loss for later tasks? For example, if the incoming task is very different from the previous one (a huge domain shift), the loss of holdout model might be poor for every input. In such a scenario, CSRL might not work.
2. What's the difference between eq.4 and eq.7? What's the difference between $\mathcal{S}$ and $\mathcal{M}$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732151940852

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. Your thoughtful questions, constructive suggestions, and detailed feedback have been invaluable in helping us refine and improve our work.

We have carefully reviewed all the comments and addressed the key concerns raised. We provided additional clarifications on the methodology, expanded experimental comparisons, and included new results to further support the contributions of our approach. Your insights have also guided us in identifying areas for potential future improvements, which we acknowledge in our responses and revisions.

We are committed to engaging actively in the discussion phase to ensure that all concerns are addressed thoroughly. Once again, thank you for your thoughtful evaluations and for helping us strengthen this work.


**Contribution**

"Presents a clear and straightforward idea for selecting good samples to add to the coreset, supported by sound equations." --Reviewer MUYB

"By focusing on performance gain, this approach reduces computational demands and mitigates the impact of noisy data. It addresses key continual learning challenges, such as task interference and data streaming, making it particularly valuable for dynamic and large-scale applications." --Review kUqK

**Writing**

"A detailed, structured, and rigorous paper with extensive derivations and experiments." --Reviewer MUYB

"The paper is well-organized and writing is almost clear." --Reviewer Ungg

**Experiments**

"rigorous paper with extensive derivations and experiments." --Reviewer MUYB

"I found multiple supplementary explanations for the main text in the supplementary materials, which I believe is commendable and should be encouraged." --Reviwer Ungg

**Concerns**

Reviewer MUYB raised concerns about the clarity of our work. In response, we have provided additional clarifications.

Reviewer kUqK raised key concerns regarding timeliness, comprehensiveness, and clarity. In response, we have provided the necessary experiments, analyses, and clarifications to address these issues.

Reviewer Ungg raised major concerns regarding the soundness and comprehensiveness of our work. In response, we have provided detailed analysis and clarifications to address soundness and conducted additional experiments to enhance the comprehensiveness of our work.

We have provided detailed responses to all questions from each individual reviewer and updated our submission, using orange highlights to indicate the changes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a coreset selection method for continual learning, using reducible loss to efficiently select representative samples and improve model performance. Reviewers feel positive about this paper due to its detailed derivations, extensive experiments on various datasets, and rigorous ablation studies, capable of addressing challenges like task interference and data streaming. In the meantime, reviewers raised several major concerns, including the usage and motivation of bi-level coreset selection strategies, the effectiveness against current methods, and the validity of the experimental comparisons. The authors successfully addressed these concerns in the discussion, and the AC was convinced.