Continual Learning via Learning a Continual Memory in Vision Transformer

Authors: Chinmay Savadikar, Michelle Dai, Tianfu Wu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Lifelong Learning, Continual Learning, Vision Transformers

Abstract:
This paper explores continual learning (CL) using Vision Transformer (ViT) in streaming tasks under the challenging exemplar-free class-incremental (ExfCCL) setting. We formulate ExfCCL as a learning problem consisting of two key sub-systems: (i) task ID inference for test data, which selects appropriate task-specific head classifiers to accounting for varying class distributions across tasks and streams, and (ii) a dynamic learning-to-grow feature backbone that balances stability and plasticity, mitigating catastrophic forgetting through task synergies. Following the common protocol that the first task can train a ViT sufficiently well as the base model, we address these sub-systems from a continual memory learning perspective. To support task ID inference, we utilize an external memory mechanism that maintains task centroids computed by the base ViT throughout CL. For the feature backbone, we identify optimal placements for internal (parameter) memory to enable a dynamic, task-synergy guided growing feature backbone. We propose a Hierarchical Exploration-Exploitation (HEE) sampling-based neural architecture search (NAS) method that effectively learns task synergies by continually and structurally updating internal memory with four basic operations: _reuse_, _adapt_, _new_, and _skip_. Our approach, dubbed **Continual Hierarchical-Exploration-Exploitation Memory (CHEEM)**, is evaluated on the challenging Visual Domain Decathlon (VDD) and ImageNet-R benchmarks, demonstrating its effectiveness.

Decision: reject
Reviewer scores: 5, 8, 3, 3

Reviews and discussion:
<thread>
creation date:
1730430791269

thread_type:
official_review

<thread_object>
creator: reviewer_wv9n

<content>
summary:
This paper proposes a new model for task-incremental continual learning (TCL) by Vision Transformers. To improve the overall streaming-task performance of TCL, this work proposes a NAS-based method cooperated with a Hierarchical tasksynergy Exploration-Exploitation (HEE) sampling method, which effectively learns task synergies by structurally updating the identifed memory component. In this way, the proposed model can tackle various tasks in TCL by adaptively using reuse, adapt, new or skip strategies. The authors conduct experiments mainly on the challeging Visual Domain Decathlon benchmak, which demonstrates the effectiveness of the proposed model.

strengths:
1. The work conducts experiments on the challenging Visual Domain Decathlon (VDD) benchmark, and obtains good results. 
2. The work proposes a new model based on NAS and MoE, which is somewhat novel in continual learning field.

weaknesses:
1. The writing and presentation of the paper is poor. 
2. Why the authors ignore class-incremental learning in this work? Compared with task-incremental learning, class-incremental learning is more practical. Although the authors have explained some in Line 97-102, it is still confused to me. 
3. The authors show an example of learned network architecture in Figure 4, but the conclusion is not clear to me. What insights can we get from the figure?
4. Please clarify the detailed differences between the adopted Visual Domain Decathlon (VDD) benchmark and other popular ones. More importantly, why the proposed model is specifically targeted at VDD? Please clarify this from the perspective of model design.  
5. Please clarify why you choose the projection layer after MHSA. As you ilustrated in Line 199-202, please clarify this from the perspectives of forward transfer ability, forgetting, simplicity and invasive implementation. 
6. Please conduct more comparison between your proposed model and existing models. For example, the authors only compare with S-Prompts and L2P in Table 2, it is not enough at all. Furthermore, I wonder if the authors have conducted a comparison with models at the same scale of parameters.

questions:
See the Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730431363834

thread_type:
official_review

<thread_object>
creator: reviewer_874z

<content>
summary:
This paper presents a novel way of designing a dynamic network in continual learning.
Specifically, this work first analyzes the potential component of the ViT network as the synergy memory and concludes with choosing the projection layer.
The paper designs a NAS-based framework and defines operations like Reuse, New, Adapt, and Skip and associated methods to achieve continual learning.
The author also shows experiments under task-incremental learning settings on multiple datasets.

strengths:
1. Strong Motivation. The natural way to conduct continual learning is to re-wire the neural networks and it is good to see the trial on the popular vision transformers.
2. Reasonable designing. I think the design of Reuse, New, Adapt, and Skip is reasonable and complete.
3. The proposed task-synergy exploration-exploitation (HEE) is novel and effective.

weaknesses:
1. Limited applicability. This paper only focuses on the task-incremental learning paradigm, which holds a strong assumption that the incoming streaming samples are organized as different domain groups and that the task IDs are available during inference. I have noticed the authors have explained their thoughts on this in Lines 92-102. However, I am still concerned about this because (1) I don't think assuming a common classification space is a problem since identifying the samples within a domain is much more difficult than identifying which domain is a sample in. In other words, a common space is compatible with separated spaces, i.e., the unified classification space can be easily decomposed into domain-separated spaces but it is hard to merge independent spaces into a unified one. (2) The design of CHEEM is based on the task-incremental learning setting and I cannot see the potential of applying CHEEM in more complex continual learning settings (class-incremental, domain-incremental, online continual learning, etc.).

2. The design and the training of the proposed method CHEEM. The training of the proposed CHEEM relies on the feature mean of each layer's output. My concern is why using the mean of the features is enough to identify the similarities of different tasks but ignores other information like the (co-)variances of the features. I am very interested in how to measure the similarity of various tasks but this paper has not discussed more potential options of this. 

4. Experiment results. (1) Why does the proposed method achieve worse performance on DAD shown in Table 4? I don't recognize the performance shown in Table 5. as SOTA performance since CHEEM loses on almost all datasets except the small dataset Aircraft. I think the authors did not make a reasonable explanation of the advantages of the proposed methods in the part of the experiment.
(2) It is stated in Lines 402 to 407 that the proposed method introduces fewer FLOPs compared to L2P. I am confused about how the number of `2.2G/task' in L2P. As far as I know, the L2P method maintains a fixed-size pool and only a sequential fixed-size prompt will be appended to the MHSA at all tasks so the increase of the FLOPs is fixed rather than increased per task. And the citation of L2P is not correct in the lines.
(3) Recent works on task-incremental learning have shifted into the vision-language models (like CLIP, which also employs the ViT as the backbone) [A, B], and I encourage the authors to explore the application of the proposed method in this challenging task.
```
    [A] Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y. You. Preventing zero-shot transfer degradation in continual learning of vision-language
models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023
    [B] Yu J, Zhuge Y, Zhang L, et al. Boosting continual learning of vision-language models via mixture-of-experts adapters. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23219-23230.
```
5. The paper is hard to follow and the writing can be greatly improved, especially the structure of section 2.2.

6. Lack of citation of more vision-transformer-based continual learning methods:

```
[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.
[2] Tang Y M, Peng Y X, Zheng W S. When prompt-based incremental learning does not meet strong pretraining. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1706-1716.
[3] Zhou D W, Sun H L, Ye H J, et al. Expandable subspace ensemble for pre-trained model-based class-incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23554-23564.
[4]Zhou D W, Ye H J, Zhang D C, Liu z. Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need, arXiv 2023.
```
And related NAS-based and archi-search-based methods:
```
[5]Gao Q, Luo Z, Klabjan D, et al. Efficient architecture search for continual learning. IEEE Transactions on Neural Networks and Learning Systems, 2022, 34(11): 8555-8565.
[6]Abati D, Tomczak J, Blankevoort T, et al. Conditional channel gated networks for task-aware continual learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 3931-3940.
[7]Rajasegaran J, Hayat M, Khan S H, et al. Random path selection for continual learning. Advances in neural information processing systems, 2019, 32.
```

questions:
Please see the weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730462404637

thread_type:
official_review

<thread_object>
creator: reviewer_rora

<content>
summary:
This paper focus on task-incremental continual learning (TCL). The proposed method, CHEEM, presents a hierarchical task-synergy exploration-exploitation sampling based NAS method for learning task-aware dynamic models continually with respect to four operations. The results on experiments conducted on the large-scale, diverse and imbalanced VDD benchmark demonstrate the better performance of CHEEM.

strengths:
1. The proposed method proposes a novel hierarchical task-synergy exploration-exploitation sampling method based on NAS for learning task-aware dynamic models continually with respect to four operations.
2. This paper is good written and easy to follow.
3. The proposed method is evaluated on a persuasive experimental setup with larger datasets, which is similar with the wild scenarios.

weaknesses:
1. There are not enough methods for comparison. In my opinion, the methodology of the paper may need to be compared with more methods，such as DualPrompt [1].

[1] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning. Wang. et.al. European Conference on Computer Vision.

questions:
1. Due to the additional training time for NAS in this method, is the training time of the paper's method longer compared to other methods ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730521641162

thread_type:
official_review

<thread_object>
creator: reviewer_ceyj

<content>
summary:
This work proposed to adapt a network for task-incremental learning. Considering the characteristics of different tasks, the proposed method learns additional parameters or reuse the model weight learned from previous tasks.
Experimental results demonstrate the effectiveness of the proposed method.
The presentation can be improvement.

strengths:
1. Instead of simply learning more parameters, the proposed method considers reusing the model weight when the new task is similar with previous task.
2. Ablation study is conducted for each design component.

weaknesses:
1. The presentation can be improve. For example, the figure 4 in the Introduction is overly detailed and contains redundant content. To illustrate the main idea, showing a case of only two or three sequential tasks are enough. For the Fig. 5, the corresponding description is far away from the figure, and a detailed figure for method will be helpful.

2. The overall designs are not focus to the continual learning. (i) For example, in line 244 "How to Adapt in a sustainable way" is related to the gradient vanishing or exploding when learning deep network. And the design of residual connection is not novel.
(ii) Using the cls-token to guide the task-specific learning is not a significant contribution because previous works also use the cls-token to select and learn task specific prompts as in L2P and DualPrompt. (iii) The proposed HEE-BASED NAS considers the task similarity but the novelty is limited due to (ii). 

3. The methods compared in Table 2 and Table 3 are limited and somewhat outdated; the latest papers in class-incremental learning should be included with necessary modifications.

4. With the parameter-efficient finetuning (PEFT) techniques, the task-incremental learning seems can be directly addressed by learning different PEFT module while keeping the main backbone frozen. The L2P learns the tasks prompt, which can be regarded as prompt tuning in PEFT. Although simply adding more prompts will quickly leads to performance saturation, adding more PEFT modules such as Adapter, LoRA may help. Therefore, a natural question is the comparison of the proposed method and the combination of different PEFT compare in terms of performance and number of learnable parameters.

questions:
Please refer to the Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732457379404

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Brief summary of Results and Changes (1/2)

comment:
We thank the reviewers for their valuable feedback. We have carefully revised the original manuscript based on the comments provided. Below, we briefly summarize our results, followed by an overview of the changes made in the revised manuscript. We believe that the revised manuscript is clearer and that the changes implemented in response to the reviewers' suggestions have significantly enhanced the overall quality of the paper. We have addressed the concerns raised by reviewers through individual comments.

## Summary of Changes
- As suggested by reviewers `ceyj`, `874z`, `wv9n`, we have revised the writing of the Introduction and Approach to improve clarity.
- Addressing the concerns raised by reviewers `874z` and `wv9n` about the applicability of CHEEM to class-incremental scenarios, we have modified the method to handle class-incremental learning. The modifications required to handle class-incremental learning are minor and do not change the core of the method. 

### Section 1: Introduction
- The Introduction has been shortened and states the motivations and contributions more clearly.
- Following is the motivation for CHEEM, which is now clearly stated in the Introduction:

As shown in Figure 1, we formulate continual learning as a problem of learning continual memory in ViT, which has two components: (i) The internal memory enabling a dynamic learning-to-grow feature backbone that balances stability and plasticity, mitigating catastrophic forgetting through **task synergies, in which a new task learns automatically to  reuse/adapt modules from previous similar tasks, or to introduce new modules when needed, or to skip some modules when it appears to be an easier task**. The internal parameter memory learning presents alternative perspectives to the input and prefix prompting based methods. (ii) The external memory enabling task ID inference for test data, for which we adopt a method proposed in~\citep{s-prompts} for its simplicity.

**Core Contributions**
- We propose a hierarchical task-synergy exploration-exploitation sampling based NAS method for learning task-aware dynamic models continually with respect to four operations: Skip, Reuse, Adapt, and New to mitigate  catastrophic forgetting. 
- We identify a "sweet spot" in ViT as the task-synergy internal (parameter) memory, i.e., the output projection layers after MHSA in ViT. It also presents a new usage for the class-token CLS in ViT as the internal memory updating guidance, in addition to leveraging it in maintaining the external (task-centroid) memory for task ID inference on the fly. 
- This is the first work, to the best of our knowledge, to evaluate continual learning with ViTs on the large-scale, diverse and imbalanced VDD benchmark, with better performance than the prior art.

### Approach
- We have organized the contents to improve clarity.
- We have also added a detailed explanation of the method to handle class-incremental learning (Section 2.2).

### Experiments
We have added comparisons with Learn to Prompt, Dual Prompt, and CODAPrompt in a class-incremental setting (Table 2) and reorganised the Results to convey the intended message more clearly. We have also added experiments with the Split ImageNet-R benchmark, where we construct a task sequence with imbalanced number of classes in each task. Please see our response to reviewer `wv9n` for the reason for choosing the VDD benchmark for experiments. 

Briefly, our results show that:

(i) A dynamic backbone is advantageous over a fixed one, as demonstrated by the comparison between S-Prompts and CHEEM.

(ii) Explicit Task ID inference is crucial for scenarios when the number of classes in a task varies, as shown by the comparison between Learn to Prompt, Dual Prompt, CODAPrompt, and CHEEM.

(iii) CHEEM enables a better balance between stability and plasticity compared to regularization based methods and experience replay.

(iv) Learn to Prompt, Dual Prompt, and CODAPrompt show significant drops in accuracy for the imbalanced split ImageNet-R (different number of classes in each task) compared to the balanced (same number of classes in each task) due to the discrepancy in the way the classification head is used in training and testing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732587232870

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Our proposed CHEEM works with Exemplar-free Class-Incremental Continual Learning (ExfCCL) in revision.

comment:
Dear Reviewers, 

Thank you for your efforts.  We have carefully and significantly revise the submission based on your valuable feedback. 

> This paper explores continual learning (CL) using Vision Transformer (ViT) in streaming tasks under the challenging **exemplar-free class-incremental (ExfCCL) setting.**

>  We formulate ExfCCL as a learning problem consisting of two key sub-systems: **(i) task ID inference for test data**, which selects appropriate task-specific head classifiers to accounting for varying class distributions across tasks and streams, and **(ii) a dynamic learning-to-grow feature backbone that balances stability and plasticity, mitigating catastrophic forgetting through task synergies.** 

>> To support task ID inference, we utilize an external memory mechanism that maintains task centroids computed by the base ViT throughout CL. 

>> For the feature backbone, we identify optimal placements for internal (parameter) memory to enable a dynamic, task-synergy guided growing feature backbone. We propose **a Hierarchical Exploration-Exploitation (HEE) sampling-based neural architecture search (NAS)** method that effectively learns task synergies by continually and structurally updating internal memory with four basic operations: _reuse_, _adapt_, _new_, and _skip_. 

> Our proposed CHEEM is evaluated on the challenging Visual Domain Decathlon (VDD) and ImageNet-R benchmarks, demonstrating its effectiveness.

Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732993068881

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A summary of author-reviewer discussions and request reviewers' (re-)evaluation of the changes in revision

comment:
Dear Reviewers, 

We are grateful for your efforts and valuable time spent on our submission and revision, which have helped us a lot in improving the quality of our submission.

Reviewer **roRA** thought we have addressed her/his concerns in the rebuttal and revision. We appreciate the recognition.  

We try to do our diligence to maximize the outcome from this valuable reviewing process.  So, we briefly summarize the current status as follows, and humbly request other reviewers' (re-)evaluation of the changes in our revision. 

Reviewer **ceyj** has yet given feedback on our rebuttal. We hope we will have her/his valuable feedback soon, which we appreciate very much.  

Reviewers **wv9n** and **874z**  have further concerns.  The main concerns and our rebuttal are as follows. 

> Our revision has changed too much.                          

>>  Overall,  **the text has been revised significantly, while the core idea and contribution are kept the same as the original submission.**   By analogy, if we treat the two versions as two streaming "tasks" in the reviewing process,  the two versions are "similar tasks", rather than ``dissimilar tasks" due to sharing the same core idea.  We provided more explanations individually to Reviewers **wv9n** and **874z**.

> After re-evaluating our method using class-incremental continual learning (CCL) settings, more results and comparisons are needed.

>> What we have provided include:  results on VDD and ImageNet-R and the provided comparisons with the state-the-art prompting based methods (L2P++, DualPrompt, CODA-Prompt, S-Prompts),  traditional continual learning methods (EWC, L2 Regularization, Experience Replay), and three state-of-the-art methods under task-to-task transfer based settings (Supermasks in Superposition (SupSup) (Wortsman et al., 2020), Efficient Feature Transformation (EFT) (Verma et al., 2021) and Lightweight Learner (LL) (Ge et al., 2023) in the appendix (Table 6)).

>> We totally agree that more results and comparisons are helpful. What we request for re-evaluating our revision is whether the provided results, together with previous task-incremental results before revision, are sufficient to support the advantages of our proposed method.  We believe our results and analyses provide insights for continual learning and shed lights on future directions.  We respect reviewers' decisions, but would like to make sure we do our best explaining our efforts, as to maximize the outcome of this reviewing process. 

Thank you very much.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings

The paper introduces a method for task-incremental continual learning using a network adaptation strategy. The proposed approach, CHEEM, employs a hierarchical task-synergy exploration-exploitation sampling based on neural architecture search (NAS) to learn task-aware dynamic models. The method focuses on reusing model weights for similar tasks and introduces operations like Reuse, New, Adapt, and Skip to achieve continual learning. Experimental results on the Visual Domain Decathlon (VDD) benchmark demonstrate the method's effectiveness, though the presentation and comparison with other methods could be improved.

(b) Strengths

Reviewer ceyj appreciates the method's consideration of reusing model weights for similar tasks and the inclusion of an ablation study. Reviewer roRA finds the hierarchical task-synergy exploration-exploitation sampling method novel and effective, with a well-written paper and a persuasive experimental setup. Reviewer 874z commends the strong motivation and reasonable design of the operations, as well as the novelty of the task-synergy exploration-exploitation method. Reviewer wv9n notes the good results obtained on the challenging VDD benchmark and the novelty of the NAS and MoE-based model.

(c) Weaknesses

Reviewer ceyj points out that the presentation needs improvement, with figures being overly detailed or poorly placed, and the design not being focused on continual learning. Reviewer roRA mentions the lack of comparison with more methods, such as DualPrompt. Reviewer 874z highlights the limited applicability to task-incremental learning, concerns about the training method, and the lack of convincing experimental results. Reviewer wv9n criticizes the poor writing and presentation, the exclusion of class-incremental learning, and insufficient comparisons with existing models.

(d) Decision Reasons

On balance, AC agrees with negative points raised by the reviewers which outweigh the positive points. Reviewer ceyj rates the paper as marginally below the acceptance threshold due to presentation issues and limited novelty. Reviewer roRA gives a higher rating, appreciating the novel method and good experimental setup. However, Reviewer 874z and Reviewer wv9n both recommend rejection, citing limited applicability, poor presentation, and insufficient comparisons. Importantly, after the rebuttal period, the most postitive reviewer  roRA, states the following 
> I have carefully reviewed the authors' rebuttal, their revisions, and the feedback from other reviewers. My comments are as follows: ﻿ I commend the authors for their effort in addressing the concerns raised during the review process. I concur with reviewers wv9n and 874z that the substantial revisions required, particularly those involving the main claim of the paper, are beyond what is typically feasible or supported within the constraints of this review cycle. ﻿ Considering this, I believe the manuscript would benefit from being submitted to another venue. This would provide the authors with an opportunity to fully address these foundational concerns and present a more cohesive and well-supported narrative.

The overall consensus is that while the method shows potential, the weaknesses outweigh the strengths, leading to a decision to reject. While the method shows promise and novelty in its approach to task-incremental learning, the limited applicability, lack of comprehensive comparisons are significant drawbacks. The paper does not convincingly demonstrate superiority over existing methods, and the focus on task-incremental learning without addressing more practical scenarios like class-incremental learning limits its impact. Additionally, the writing and structure need improvement to enhance clarity and understanding. These weaknesses outweigh the strengths, leading to a decision to reject.