Advancing Prompt-Based Methods for Replay-Independent General Continual Learning

Authors: Zhiqi KANG, Liyuan Wang, Xingxing Zhang, Karteek Alahari
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning, General Continual Learning, Catastrophic Forgetting, Prompt Tuning, Sharpness-Aware Minimization

Abstract:
General continual learning (GCL) is a broad concept to describe real-world continual learning (CL) problems, which are often characterized by online data streams without distinct transitions between tasks, i.e., blurry task boundaries. Such requirements result in poor initial performance, limited generalizability, and severe catastrophic forgetting, heavily impacting the effectiveness of mainstream GCL models trained from scratch. While the use of a frozen pretrained backbone with appropriate prompt tuning can partially address these challenges, such prompt-based methods remain suboptimal for CL of remaining tunable parameters on the fly. In this regard, we propose an innovative approach named MISA (Mask and Initial Session Adaption) to advance prompt-based methods in GCL. It includes a forgetting-aware initial session adaption that employs pretraining data to initialize prompt parameters and improve generalizability, as well as a non-parametric logit mask of the output layers to mitigate catastrophic forgetting. Empirical results demonstrate substantial performance gains of our approach compared to recent competitors, especially without a replay buffer (e.g., up to 18.39, 22.06, and 11.96% points performance lead on CIFAR-100, Tiny-ImageNet, and ImageNet-R, respectively). Moreover, our approach features the plug-in nature for prompt-based methods, independence of replay, ease of implementation, and avoidance of CL-relevant hyperparameters, serving as a strong baseline for GCL research. Our source code is publicly available at https://github.com/kangzhiq/MISA

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730209802627

thread_type:
official_review

<thread_object>
creator: reviewer_1w4v

<content>
summary:
The paper proposes a prompt-based method for General Continual Learning (GCL), an incremental setting where task boundaries between consecutive tasks are not rigid, allowing certain classes to reappear in later tasks. GCL represents a challenging yet realistic scenario, as the absence of clear boundaries limits the applicability of many algorithms and techniques, particularly those relying on model expansion. The authors leverage two key strategies: (1) refining the pre-trained model’s initialization to improve resilience to distribution shifts, thereby preserving performance; and (2) using masking techniques during cross-entropy loss computation, selectively based on the classes present in each batch. Experimental results indicate strong, promising performance relative to existing solutions, complemented by extensive ablation studies.

strengths:
- The writing is brilliant. It is a pleasure to review papers like this one.
- The idea behind FAM is noteworthy and represents the main contribution of this work.
- The experiments are extensive and cover all that is needed to understand the proposed approach.

weaknesses:
I have only a few minor questions and suggestions.

1. While reading Sec. 4.1, I was initially puzzled by the optimization problem outlined in Eq. 2. Specifically, I questioned the advantage of fine-tuning prompts using the same data employed during pre-training. Since the gradients should be near zero around the pre-training weights, this raised some concerns about the utility of prompts. The rationale becomes clearer a few lines later; however, to improve readability, I suggest that the authors provide some context earlier in Sec. 4.1.

2. While reading Sec. 4.3, I began to wonder if FAS could also be effectively applied to standard class-incremental learning. The idea does not seem strictly constrained to the GCL setting.

3. In reviewing Eqs. 5 and 6, I noticed similarities with Meta-Learning, particularly Meta Agnostic Meta Learning (MAML). These equations suggest an optimization setup where some data is used to train the model (inner loop) and other data is used for “differentiable” evaluation (outer loop). I believe that discussing this connection in the main paper would add value, but I’d like to hear the authors' perspective on this.

4. The prompt augmentation is the only aspect that doesn’t fully convince me. Even with an MLP layer, the resulting complexity seems comparable to that of a straightforward learnable prompt. Perhaps the MLP affects the training trajectory, introducing a smoothing effect across iterations. In my experience, concatenation-based prompting strategies tend to be less effective, though this may not be due to reduced parameters or complexity. Instead, I suspect it depends on how these prompts are incorporated into the pre-trained model. I recommend that the authors test their approach with [a] as a fine-tuning strategy (i.e., using addition instead of concatenation).

[a] Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. A. (2022). Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35, 1950-1965.

5. The masking strategy used in this work is identical to that of ER-ACE (ICLR 2022). I think the paper should be more transparent about this similarity.

questions:
See section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712306529

thread_type:
official_review

<thread_object>
creator: reviewer_y3su

<content>
summary:
The paper presents MISA, an innovative approach for tackling General Continual Learning (GCL), arguably the most challenging form of Continual Learning. They do so by starting from a pre-trained model, refined with prompts, and introducing two key components:

ISA-FAM, which is applied at the start of each learning session (essentially, at the beginning of each GCL task). This component ensures proper prompt initialization. Here, "proper" means achieving both robust generalization capabilities and mitigating future forgetting. To enhance the learning effectiveness of prompts in this phase, the authors introduce a “prompt augmentation” technique, where prompts are trained using an approach involving an MLP that is later thrown away at the end of this phase..

Non-parametric Logit Masking: similar to prior CL methods, the authors propose a session mask that preserves the final activations of classes not present in the current learning session. Since GCL involves blurred task boundaries and often lacks task identifiers, they use a unique batch-level mask designed to function universally across scenarios.

The experimental validation utilizes widely recognized CL datasets, with a particularly rich ablation analysis that provides in-depth insights into the method’s effectiveness.

strengths:
1) The method introduced by the authors meets their claim and proves to be the SOTA in GCL;
2) the ablation section is really rich and thorough, probing deeply every aspect of the method;
3) while the components used by the authors are not exactly novel per se, their combination and the proposed modifications prove to be interesting and effective;
4) by cleverly leveraging the pre-trained data, the authors eliminated the need for a replay buffer. At the same time, they showed that the model can benefit from storing past exemplars in case it is allowed;
5) the key components of MISA are plug and play, as shown by one of the ablations (Table 5), meaning that they could work out of the box if applied on other methods;
6) The code is available in the supplementary material, which greatly helps reproducibility.

weaknesses:
1) While it is true that a buffer is not mandatory for MISA, it still depends on the pre-train data for the ISA-FAM phase. This still poses a limitation, as the pre-train data may not be available;
2) the experimental section lacks the case of a completely out-of-distribution dataset.

questions:
1) In the ISA-FAM phase, the classification heads are trained with the prompts but later discarded (replaced by the previously trained heads) when the real learning session begins. Why is that? This way, I suppose prompts would "communicate incorrectly" with the classification heads, as the latter are replaced;
2) is there any reason why DualPrompt was selected as the prompting method? Did you consider more performing methods like Coda-Prompt[1] or HiDe-Prompt[2]?
3) you did cite [3] but not [4] for General Continual Learning, although the latter was published earlier. Is there any particular reason for this? Did you prefer the former over the latter in your GCL formulation?

[1] Smith, James Seale, et al. "Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 

[2] Wang, Liyuan, et al. "Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality." Advances in Neural Information Processing Systems 36 (2024).

[3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. 2020.

[4] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. PAMI, 44(7):3366–3385, 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731258339277

thread_type:
official_review

<thread_object>
creator: reviewer_iguv

<content>
summary:
This paper introduces MISA, a novel method that enhances prompt-based approaches in Generalized Continual Learning (GCL) by incorporating a forgetting-aware initial session adaptation and a non-parametric logit mask. Experiments demonstrate the proposed method's effectiveness and its ability to improve performance on three representative datasets: CIFAR-100, Tiny-ImageNet, and ImageNet-R.

strengths:
1. The paper is well-structured and easy to understand.
2. The authors aimed to propose an effective methodology for a scenario that is closer to real-world conditions, compared to the common CL scenarios assumed by existing prompt-based CL methods.

weaknesses:
1. The methodology appears to lack substantial novelty. The approach of using SAM for effective initialization and leveraging previously learned knowledge with masking for robustness against forgetting has been proposed and utilized numerous times in CL.
2. In the context of GCL, demonstrating the performance of a pre-trained model on natural images only with similar or closely related benchmarks weakens the paper's claims. It would be important to show that the SAM + masking with a pre-trained model approach remains strong when tested on benchmarks composed of datasets from significantly different domains, such as completely different views, 3D data, or medical imagery.
3. The scenario used in the paper is based on an already proposed (limited) GCL scenario that assumes blurry and task-free settings, which may not fully capture the (real) general scenario. I believe including experiments that reflect more realistic situations, such as not knowing the number of classes in the beginning or noisy datasets, would enhance the paper's contribution.

questions:
Please address the concerns mentioned above in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731464991139

thread_type:
official_review

<thread_object>
creator: reviewer_n9s5

<content>
summary:
In this paper, the authors proposed a new prompt based method for general continual learning. Here they utilize forgetting-aware initial session adaption that employs pretraining data to initialize prompt parameters to improve generalizability. Additionally, they propose to use a simple non-parametric logit mask at the output layers to mitigate catastrophic forgetting. With comprehensive experiments and analysis they show the effectiveness of the method in GCL setup.

strengths:
1. The idea of the paper is clearly presented with comprehensive experiments, ablations, and analysis. 
2. Though the paper uses concepts from other previous works the combination of the ideas and their effective implementation in the GCL setup is interesting and useful for the continual learning community.

weaknesses:
1. The paper aims to address the challenging general continual learning (GCL) setup. However, in this setup, the classes can appear one by one and the model may need to learn one class at a time. No discussion on how to handle such a situation is presented in the paper. 
2. Though the authors presented experiment with blurry task/class boundaries, in true GCL setup the labels of the classes might not be known ahead of time. In such cases, the model needs to detect novel classes and then learn that new classes (see [1]). How the current algorithm would handle such scenarios is not discussed in the paper. 
3. The algorithm assumes the availability of the pertaining data. This is a strict constraint. In most practical cases this would not be available. For instance for large opensourced pre-trained vision, vision-language, and language models, the corresponding pretraining data is not publicly released. In the absence of any pertaining data or with very limited accessibility of pertaining data how this algorithm will perform? 
4. The paper failed to cite and/or compare with many related works. For example: [2,3,4].  


[1] M. Gummadi, D. Kent, J. Mendez, and E. Eaton, “SHELS: Exclusive
Feature Sets for Novelty Detection and Continual Learning Without Class
Boundaries,” PMLR Conference on Lifelong Learning Agents, pp. 1065-
1085, 2022
[2] Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, pp. 11909–
11919, 2023.
[3] RanPAC: Random Projections and Pre-trained Models for Continual Learning. (https://arxiv.org/abs/2307.02251)
[4] Visual Prompt Tuning in Null Space for Continual Learning (https://arxiv.org/abs/2406.05658)

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732104972507

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
We sincerely appreciate the valuable feedback provided by all the reviewers. It is encouraging to see many positive comments, especially noting that our presentation is *clear* (Reviewers n9s5, iGuV, 1W4v), our method is *novel* (Reviewers n9s5, y3su, 1W4v) and *effective* (Reviewers n9s5, iGuV, y3su, 1W4v), our experiments are *comprehensive* (Reviewers n9s5, y3su, 1W4v), etc.

We have provided a point-to-point response to all the comments, summarized the corresponding changes, and uploaded a revised version of our manuscript. Our responses and revisions are summarized below.

(1) We have analyzed a variety of advanced prompt-based methods tailored for regular CL, showing their shortcomings for GCL in terms of representation recovery and orthogonality regularization.

(2) We have added more explanations and experiments to address the scenarios where pretraining data is partially or even completely unavailable.

(3) We have included additional explanations and experiments to address scenarios where the pretraining data and subsequent CL tasks are significantly different, as well as the challenging one-class increments.

(4) We have further clarified our framework to address the challenges of CGL, highlighting its innovation compared to previous efforts in regular CL.

We hope these responses provide sufficient materials to address the reviewers' concerns. We would be happy to answer any further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733309646429

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final overall response and Gratitude to the reviewers

comment:
Dear Reviewers,

Thank you for your thoughtful feedback and efforts during the review process of our submission. We would like to kindly address two points as the rebuttal phase concludes.

For Reviewer n9s5 and iGuV: Since no further questions were raised during the rebuttal discussion, we hope this indicates that we have satisfactorily addressed your initial remarks. We would be grateful if you could consider re-evaluating our work and potentially raising the overall score.

For Reviewer y3su and 1W4v: Your engagement during the rebuttal has been invaluable in ensuring that our work receives a thorough and fair evaluation. 

Once again, we are very grateful for your time, thoughtful reviews, and dedication to ensuring a constructive and fair evaluation process.

Best regards,

Authors of paper ID 7052
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates general continual learning and introduces MISA (Mask and Initial Session Adaptation) to advance prompt-based methods. Strong empirical performance is achieved due to a combination of two strategies, refining prompt parameter initialization and output layer masking. The paper has several strengths, including the plug-and-play nature of the approach, its strong empirical performance, a thorough ablation study, clear presentation, and interesting combination of methodology. The weaknesses and points for improvement revolved around additional comparisons, e.g. on completely out-of-distribution scenarios and in scenarios were pre-training data is not available, alongside concerns on the training set-up. These points were addressed in the discussion phase and additional experiments added for clarification. 

The AC agrees with the reviewers that there are several strong ideas in the paper and that the paper provides a meaningful empirical improvement in a very challenging to solve scenario. The AC votes to accept the paper.