Active Learning for Continual Learning: Keeping the Past Alive in the Present

Authors: Jaehyun Park, Dongmin Park, Jae-Gil Lee
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: active learning, continual learning, Fisher information

Abstract:
*Continual learning (CL)* enables deep neural networks to adapt to ever-changing data distributions. In practice, there may be scenarios where annotation is costly, leading to *active continual learning (ACL)*, which performs *active learning (AL)* for the CL scenarios when reducing the labeling cost by selecting the most informative subset is preferable. However, conventional AL strategies are not suitable for ACL, as they focus solely on learning the new knowledge, leading to *catastrophic forgetting* of previously learned tasks. Therefore, ACL requires a new AL strategy that can balance the prevention of catastrophic forgetting and the ability to quickly learn new tasks. In this paper, we propose **AccuACL**, **Accu**mulated informativeness-based **A**ctive **C**ontinual **L**earning, by the novel use of the Fisher information matrix as a criterion for sample selection, derived from a theoretical analysis of the Fisher-optimality preservation properties within the framework of ACL, while also addressing the scalability issue of Fisher information-based AL. Extensive experiments demonstrate that AccuACL significantly outperforms AL baselines across various CL algorithms, increasing the average accuracy and forgetting by 23.8% and 17.0%, respectively, on average.

Decision: accept (poster)
Reviewer scores: 8, 8, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729751206426

thread_type:
official_review

<thread_object>
creator: reviewer_as7j

<content>
summary:
This paper focuses on the scenario that performs Active Learning (AL) in the Continual Learning (CL) process. The paper states that previous AL methods only focus on learning new tasks, which leads to catastrophic forgetting. The authors introduce a novel method called **AccuACL** that improves fisher information-based AL to make a balance between learning new tasks while preserving old knowledge. Empirical results show that **AccuACL** outperforms existing AL methods in standard CL metrics. Overall, this paper opens an important problem and propose a reasonable method to mitigate it.

strengths:
* The problem is important, especially for online deployed models. 
* The extensive experiments partly support paper's claims and the effective of the method. Meanwhile, ablation studies help to understand the proposed method. 
* The paper provides sufficient details and code in Appendix for readers to reproduce the experiments.

weaknesses:
### Method and theory 
1. The proposed method only works for memory-based continual learning methods. It can not be applied on naive continual learning or regularization-based continual learning methods. I understand that memory buffer is necessary for leveraging existing AL method, but I encourage authors to add this as limitations or propose future directions to handle this issue. 
2. In Figure 1, it seems like **AccuACL** is good for selecting new samples that related to the learned *features*. However, it is not clear what are those features, and Section 4 does not explain this part clearly either. 
3. More background for Active Learning is needed for CL community readers.

### Experiment
1. While theoretical space complexity shows **AccuACL** is more efficient than BAIT, no experiment results are support this claim. An additional experiment for memory usage is needed, like Figure 2(c) for time complexity.
2. In Introduction and Section 5.2, authors claims that:
> "these AL strategies typically pay more attention to new features...", "most AL baselines that most AL baselines that focus only on learning new tasks often perform worse the Uniform."

However, these claims are not confirmed by the experiments. Compared with the Uniform strategy, some existing AL baselines have better forgetting score, but worse average accuracy. This phenomenon might indicates that existing AL baselines are not learning new tasks well either. Learning accuracy metric [1] may be able to support paper's claim. If the claim is true, I expect to see existing AL methods might have higher Learning accuracy. But I am not sure what will the final results be. I encourage authors to do quick additional experiments on CIFAR-10 to see their claims is true or not. 

# Reference
[1] Wide Neural Networks Forget Less Catastrophically, ICML 2022

questions:
### Method
1. In Section 4.5, why uses distribution score first and then magnitude score later. Does the order of these two score matters? This is not discussed in Section 5.3 either. 
### Future work
1. Does using regularization-based methods can further improve ACL performance? Maybe the authors can consider this idea to further improve ACL.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729934720431

thread_type:
official_review

<thread_object>
creator: reviewer_fggu

<content>
summary:
The paper focuses on active continual learning (ACL) and proposes AccuACL (Accumulated Informativeness-based Active Continual Learning) to effectively balance the prevention of catastrophic forgetting and the rapid learning of new tasks. This method proposes the accumulated informativeness modeled by the Fisher information matrix, which is further approximated by the diagonal Fisher information embedding to reduce the calculation cost. Experimental results show that AccuACL outperforms existing active learning baselines across various continual learning algorithms.

strengths:
1. The motivation is clear and well-justified, aiming to enhance previous methods by addressing both forgetting and rapid learning in subset selection for training.

2. The proposed method is based on the two theorems, making the algorithm be more confident and efficient.

3. Experimental results demonstrate the strength of the proposed method, significantly improving accuracy while reducing forgetting.

4. The paper is well-organized and most components are clear and easy to understand.

weaknesses:
**Major**

1. The method employs buffer data to evaluate the current parameter importance with respect to the historical task. However, a small buffer can not adequately represent the complete data distribution of historical tasks, which is easy to overfit. Then the network may memorize these buffer samples and exhibit low parameter sensitivity, leading to an incorrect Fisher information matrix.

2. The paper focuses solely on the Fisher information matrix of linear classifiers. However, linear classifiers in CL often exhibit task-recency bias, making them prone to favoring new classes [1,2,3]. This raises concerns about the accuracy of the Fisher information matrix calculation. How can an unbiased Fisher information matrix be derived from a biased linear classifier?

3. The description of the greedy algorithm for subset selection is unclear (line 6 in Algorithm 1 requires further explanation). My understanding is as follows:

   a) Randomly sample a subset and calculate the Fisher information embedding according to Theorem 4.2.

   b) Calculate the magnitude score and distribution score of the current subset.

   c) Randomly sample a lot of subsets and repeat steps a) and b). After that, comparing scores across these subsets, and then, selecting the subset with the highest score as the final selection subset $X$.

4. If my understanding is correct, how to guarantee the convergence of the greedy algorithm and ensure that it can choose the optimal subset? Additionally, detailed settings should be provided, such as the subset sampling method, whether subsets intersect, and the size of the subsets.

5. Regarding the budget experiment. There is no clear definition of budget and no experiments pointing out the budget limitation.


**Minor**

1. In Table 3 (lines 472-474), how to incorporate historical knowledge into the greedy algorithm? Why use the memory buffer as the initial subset?

[1] Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. CVPR, 2021.

[2] Cba: Improving online continual learning via continual bias adaptor. ICCV, 2023.

[3] Overcoming recency bias of normalization statistics in continual learning: Balance and adaptation. NIPS, 2024.

If all my concerns are addressed, I'll gladly increase the score.

questions:
1. In active learning, why is unlabeled data not utilized in an unsupervised manner to further enhance the training process?

2. In Figure 3, the choice of lambda depends on the amount of unlabeled data. Does this conclusion also hold in imbalanced datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561374307

thread_type:
official_review

<thread_object>
creator: reviewer_fzgq

<content>
summary:
The paper proposes an active learning method for continual learning that balances the prevention of catastrophic forgetting and the ability to quickly learn new tasks.

strengths:
Studying active learning in the continual learning setting is interesting.

weaknesses:
1.	The novelty of the paper is somewhat limited, as the use of the Fisher information matrix and replay mechanisms are already well-known methods in continual learning.

2.	The citations in the related work section are outdated, giving the impression that the authors may not be fully aware of the latest advancements in continual learning, particularly beyond replay and regularization-based methods. Please see the survey [1] and the related work section of [2] and [3]. 

3.	It would be beneficial to examine how the system and baseline models perform as the number of queries increases.

4.	The baseline methods used in this paper are also outdated. See those in [2] and [3]. 

5.	The current accuracy results are quite low and significantly fall short of existing state-of-the-art methods.

6.	Given that your method selects only a small number of samples, it would be essential to compare it with recent few-shot continual learning approaches [4], [5], [6], and [7].

7.	Catastrophic forgetting does not seem to be the main problem anymore. Inter-task class discrimination may pose a greater challenge. Please refer to [2] and [3]. 

8.	The results on the full dataset are notably low too, and it is unclear what method was used in the “full” setting. Please see the papers mentioned above for comparison.

9.     The paper is overly formal and not easy to follow. Some more descriptions of intuitions will be helpful. 

10.   The space completely is very high for continual learning.

[1] Wang et al. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.

[2] Lin et al., "Class Incremental Learning via Likelihood Ratio-Based Task Prediction," ICLR 2024.

[3] Wang et al. BEEF: Bi-compatible class-incremental learning via energy-based expansion and fusion. ICLR 2023. 

[4] Mazumder et al. Few-Shot Lifelong Learning. AAAI-2021 

[5] Zhou et al. Forward compatible few-shot class-incremental learning. CVPR- 2022.

[6] Song et al. Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-incremental learning. CVPR-2023 

[7] Tian et al. A survey on few-shot class-incremental learning. Neural Networks. 2024.

questions:
See the previous sections
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611322547

thread_type:
official_review

<thread_object>
creator: reviewer_xnjr

<content>
summary:
This paper introduces a novel active learning (AL) approach for active continual learning (ACL) that effectively balances the prevention of catastrophic forgetting with the capacity to quickly learn new tasks. The authors present an accumulated informativeness-based method, termed AccuACL, which extends Fisher information-based AL to a class-incremental setting. By leveraging Fisher information embeddings, AccuACL estimates informativeness through a combination of magnitude and distribution scores, offering a more efficient way to rank data for selection. Experiments across four continual learning (CL) methods on three CL benchmarks show that AccuACL significantly improves CL performance, outperforming traditional AL baselines in terms of both accuracy and forgetting metrics.

strengths:
- The paper addresses active continual learning (ACL), an underexplored area in the intersections of AL and CL.
- In addition to promising experimental results, the paper provides a solid theoretical foundation for the proposed method, as well as an efficient approximation strategy for computing accumulated informativeness scores.
- The authors include analyses of both time and space complexity, demonstrating the feasibility of implementing AccuACL.

weaknesses:
-  Some experiment details like annotation budgets and number of runs are not provided, which are crucial for reproducing the results.
-  Further analysis of the specific data points chosen by each AL method across different CL settings could offer more insight into optimal query selection. Additionally, an examination of annotation budget sensitivity (as many methods peak around round 10 in Figure 2, suggesting diminishing returns) and the effect of task order (since CL can be highly sensitive to task sequencing) would add depth to the findings.

questions:
1. In the main experiments, it’s intriguing to see ACL methods, especially AccuCL and random, outperform training on the full dataset. What annotation budget is used, and how many runs are conducted in this experiment? Also, why is there no standard deviation for the BAIT rows?
2. Figure 2 shows an interesting trend: accuracy and forgetting metrics peak around AL round 10 and then decline for all AL methods. Could you provide an explanation for this phenomenon?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620901039

thread_type:
official_review

<thread_object>
creator: reviewer_k3jb

<content>
summary:
This paper addresses Active Continual Learning (ACL) and particularly the problem of catastrophic forgetting when active learning and continual learning meet. The authors propose propose AccuACL, (Accumulated informativeness-based Active Continual Learning,) which attempts to achieve an optimal balance between learning new knowledge while preserving past knowledge in ACL. They model the accumulated informativeness via the Fisher information matrix,  through the approximation with a small memory buffer, the model parameters, and the unlabeled data pool for the new task. The are theoretical guarantees and extensive experimentation.

strengths:
The novelty of the method is the application of active learning in continual learning. A technique that is applied in active learning is modified to balance learning and prevention of forgetting in the continual setting domain.

Since the proposed solution has a theoretical basis, this basis can be extended to continual learning. The paper includes a theoretical analysis of space and time complexity and also the Fisher Optimality preserving properties.

The proposed method and the intuition behind it are clearly explained in the paper.

The experimentation evaluates the proposed methods from a number of perspectives.

weaknesses:
What are the open problems or future directions related to your work? Adding this to the paper would improve the paper's discussion of broader impact and potential future work. This should be additional to the limitations section in the Appendix.

Typos: Please do a grammar check on your paper. There is a typo in the Abstract “informativeness”

questions:
What are the characteristics of problems where the method works well? Additionally, what are the characteristics of problems where the method does not perform well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732364794808

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response Summary to All Reviewers

comment:
We are extremely grateful for the reviewers' constructive comments and valuable feedback, which have significantly enhanced our paper. Overall, the reviewers have recognized the need for an AL strategy in CL scenarios (all reviewers), strong theoretical background ([k3jB](https://openreview.net/forum?id=mnLmmtW7HO&noteId=tiHphRX72n), [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f), [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc)), and extensive experimental results ([k3jB](https://openreview.net/forum?id=mnLmmtW7HO&noteId=tiHphRX72n), [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f), [as7J](https://openreview.net/forum?id=mnLmmtW7HO&noteId=dxK0Fju6qn)). Additionally, the reviewers have suggested insightful additional experiments to further establish the efficacy of **AccuACL**. In accordance with the reviewers' feedback, the revised sections of the paper have been highlighted in **blue**. The revisions encompass the followings:

- **Additional Evaluation \& Analysis**
    - Evaluating the performance of **AccuACL** across different labeling budgets (Figure 2(a), Reviewer [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc), [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f))
    - Evaluating **AccuACL** on *inter-task discrimination-based CL* (Figure 4, Reviewer [Fzgq](https://openreview.net/forum?id=mnLmmtW7HO&noteId=FWIJVdfD2E))
    - Evaluating **AccuACL** on more recent CL method (Table 4, Reviewer [Fzgq](https://openreview.net/forum?id=mnLmmtW7HO&noteId=FWIJVdfD2E))
    - Evaluating the effectiveness of the Fisher information matrix via memory buffer (Table 5, Reviewer [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f))
    - Evaluating the ACL performance with *Learning Accuracy* (Table 6, Reviewer [as7J](https://openreview.net/forum?id=mnLmmtW7HO&noteId=dxK0Fju6qn))
    - Evaluating robustness of **AccuACL** with varying task orders (Table 7, Reviewer [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc))
    - Adding space complexity analysis of AL baselines (Figure 2(b), Reviewer [Fzgq](https://openreview.net/forum?id=mnLmmtW7HO&noteId=FWIJVdfD2E), [as7J](https://openreview.net/forum?id=mnLmmtW7HO&noteId=dxK0Fju6qn))
- **Enhanced Presentation \& Discussion**
    - Improving Algorithm 1 for readability (Algorithm 1, Reviewer [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f))
    - Improving the limitation and future works (Appendix I, Reviewer [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc), [fGgu](https://openreview.net/forum?id=mnLmmtW7HO&noteId=6Jz6jpBM8f), [as7J](https://openreview.net/forum?id=mnLmmtW7HO&noteId=dxK0Fju6qn))
    - Improving related works for both AL and CL (lines 98-100, lines 110-112, lines 115-116 Reviewer [Fzgq](https://openreview.net/forum?id=mnLmmtW7HO&noteId=FWIJVdfD2E), [as7J](https://openreview.net/forum?id=mnLmmtW7HO&noteId=dxK0Fju6qn))
    - Providing detailed experimental setup (line 1092-1094, Reviewer [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc))
    - Providing visualization of query results (Figure 6, Reviewer [XnjR](https://openreview.net/forum?id=mnLmmtW7HO&noteId=XJJyiihdzc))


We believe that the following responses address all the issues and questions raised by the reviewers. Again, we are grateful for the opportunity to address these comments and improve our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732467622362

thread_type:
official_comment

<thread_object>
creator: area_chair_8j5q

<content>
comment:
Dear Reviewers,

This is a gentle reminder that the authors have submitted their rebuttal, and the discussion period will conclude on November 26th AoE. To ensure a constructive and meaningful discussion, we kindly ask that you review the rebuttal as soon as possible and verify if your questions and comments have been adequately addressed.

We greatly appreciate your time, effort, and thoughtful contributions to this process.

Best regards,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732716686735

thread_type:
official_comment

<thread_object>
creator: area_chair_8j5q

<content>
comment:
Dear Reviewers,

We wanted to let you know that the discussion period has been extended to December 2nd. If you haven't had the opportunity yet, we kindly encourage you to read the rebuttal at your earliest convenience and verify whether your questions and comments have been fully addressed.

We sincerely appreciate your time, effort, and thoughtful contributions to this process.

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

This work proposes an active learning strategy for continual learning where given a pool of unlabeled data and an annotation budget, the algorithms selects the most relevant samples to annotate for each task increment. It achieves state-of-the-art results compared to other ACL methods.

**Strengths**:
* The framework bridges active learning (AL) with continual learning (CL), proposing an informative sample selection mechanism based on Fisher information.
* The authors provide a solid theoretical justification demonstrating the effectiveness of the sample selection strategy.
* Extensive experiments show significant improvements in accuracy and forgetting metrics over ACL baselines across multiple datasets and continual learning methods.
* The approach introduces efficient approximations for Fisher information, making the method computationally feasible.
* Robustness: Tests with different budgets, task orders, and class-incremental setups demonstrate the adaptability of the method.

**Weaknesses** 
* ACL is an artificial problem which might not translate to a real world situation.

Overall, this work is interesting, the algorithm is sound, and it produces state-of-the-art results in ACL. Another point I have taken into consideration is that the field of active continual learning has not progressed much during the recent years and this work could serve as a spark to reignite the field.