Vision and Language Synergy for Rehearsal Free Continual Learning

Authors: Muhammad Anwar Ma'sum, Mahardhika Pratama, Savitha Ramasamy, Lin Liu, H Habibullah, Ryszard Kowalczyk
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, prompt dilemma, language descriptors, prompt generator, catasthropic forgetting

Abstract:
The prompt-based approach has demonstrated its success for continual learning problems. However, it still suffers from catastrophic forgetting due to inter-task vector similarity and unfitted new components of previously learned tasks. On the other hand, the language-guided approach falls short of its full potential due to minimum utilized knowledge and participation in the prompt tuning process. To correct this problem, we propose a novel prompt-based structure and algorithm that incorporate 4 key concepts (1) language as input for prompt generation (2) task-wise generators (3) limiting matching descriptors search space via soft task-id prediction (4) generated prompt as auxiliary data. Our experimental analysis shows the superiority of our method to existing SOTAs in CIFAR100, ImageNet-R, and CUB datasets with significant margins i.e. up to 30% final average accuracy, 24% cumulative average accuracy, 8% final forgetting measure, and 7% cumulative forgetting measure. Our historical analysis confirms our method successfully maintains the stability-plasticity trade-off in every task. Our robustness analysis shows the proposed method consistently achieves high performances in various prompt lengths, layer depths, and number of generators per task compared to the SOTAs. We provide a comprehensive theoretical analysis, and complete numerical results in appendix sections. The method code is available in https://github.com/anwarmaxsum/LEAPGEN for further study.

Decision: accept (poster)
Reviewer scores: 8, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729603710662

thread_type:
official_review

<thread_object>
creator: reviewer_bacr

<content>
summary:
This paper introduces a new prompt-based approach for continual learning that take advantage of language guidance. It uses task-wise generators, soft task ID prediction, and generated prompts as auxiliary data. The method outperforms state-of-the-art models on CIFAR100, ImageNetR, and CUB datasets with significant improvements in accuracy and forgetting measures.

strengths:
1. The paper provides a detailed explanation of current prompt-based continual learning methods and analyzes the weakness associated with these methods, effectively leading to the introduction of the authors' proposed method.
2. The paper presents a novel language-guided continual learning method and thoroughly demonstrates its effectiveness through extensive experiments.

weaknesses:
1. The comparison mentioned in lines 183-184 between "the photo of car" and "the photo of cat," stating they have 15/16 similarity but 1/16 dissimilarity, is confusing. The authors compared the similarity between these two prompts on a **letter-by-letter** basis and concluded that they are similar. However, the similarity between two prompts should not be evaluated in such a manner. After being processed by the tokenizer, the embeddings of the two prompts are not similar, as their meanings differ significantly.
2. The proposed method relies on using ChatGPT to generate descriptive terms for each class and requires an additional Sentence Transformer to obtain embeddings. The introduction of these extra resources creates an unfair comparison with other methods and limits the practical applicability of the approach in real-world scenarios.
3. Although the proposed method achieves state-of-the-art performance on multiple datasets, the improvement is minor on some, such as the comparison with HiDe-Prompt on the CUB200 dataset. This is particularly relevant given the substantial additional resources, like ChatGPT, required by the method, which other approaches do not utilize. Moreover, this suggests that the performance gains on fine-grained datasets may not be significant, as the generated descriptive terms for each class are quite similar.
4. The authors have used excessive line spacing adjustments, which negatively impact the visual presentation of the paper, such as in lines 437-438 and 446-447. They should revise the layout of the entire paper to provide a better reading experience for the reader.

questions:
Please the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448004828

thread_type:
official_review

<thread_object>
creator: reviewer_ugi2

<content>
summary:
This paper presents a novel approach to address the catastrophic forgetting in continual learning through a prompt-based structure. This method incorporates language inputs for prompt generation and utilizes task-wise generators and soft task-ID prediction. The authors highlight the advantages over existing methods across various datasets, showcasing substantial improvements accuracy while minimizing forgetting metrics.

strengths:
- The authors systematically review the limitations of existing prompt-based methods, and Figure 1 commendably illustrates the comparison of these methods.

- The proposed method demonstrates performance that far exceeds current methods, which is pleasantly surprising.

- The paper introduces the novel and meaningful use of large language models to generate more refined descriptions.

weaknesses:
- The writing of this paper needs significant improvement. Additionally, the structure is overly compact, affecting readability. For instance, the theorem presented in the paper has weak relevance to the proposed method, and I believe it is unnecessary. I suggest moving it to the supplementary materials. Given the complexity and sophistication of the method, I encourage the authors to provide an overview initially.

- The provided ablation study is not sufficiently convincing. I recommend that the authors validate the effectiveness of the proposed modules step-by-step starting from a standard baseline.

- The inclusion of the ChatGPT model raises the question of how to ensure the fairness of experimental comparisons under such settings, and to what extent the performance gains are due to more accurate descriptions.

- Considering the authors (are about to) open-source their code, I suggest making the descriptions obtained via ChatGPT publicly available for deeper analysis. I believe this will enhance the impact of the paper.

questions:
Please see the comments of weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677303742

thread_type:
official_review

<thread_object>
creator: reviewer_j5sh

<content>
summary:
This paper proposes a novel method LEAPGen (Language as Prompt Generator) for continual learning that leverages language knowledge for prompt generation.
LEAPGen uses language descriptors for each class as input for prompt generation, rather than shared embeddings. For each task, LEAPGen employs a fixed number of generator networks, a learnable task-level key, and learnable class-level parameters associated with the descriptor embeddings. To generate a prompt, it first predicts the task ID using a soft prediction mechanism, then selects top-k matching descriptor embeddings based on cosine similarity to the input. Experimental results on datasets like CIFAR100, ImageNet-R, and CUB demonstrate that LEAPGen significantly outperforms existing state-of-the-art methods in terms of accuracy and forgetting measure, addressing key limitations of previous prompt-based continual learning approaches.

strengths:
- LEAPGen uses language descriptors as input for prompt generation, leveraging richer semantic information compared to previous methods.

- As a prompt-based method, it doesn't require storing and replaying old data.

- By using fixed language descriptors as input, LEAPGen potentially reduces catastrophic forgetting compared to methods that continuously update shared embeddings.

- Experimental results show significant improvements over state-of-the-art methods.

weaknesses:
- The paper does not explicitly specify some key details about the pre-trained models used. Which Sentence Transformer does the paper employ? Whether this is a pre-trained language model and if any fine-tuning of these models occurs during the continual learning process.

- The paper does not provide a clear comparison of the increase in parameters due to the addition of generators for new tasks. Based on the paper, the generator is not a lightweight module. As the proposed method requires adding new generators for some new tasks, even if not all tasks, this could lead to a significant growth in the number of parameters as the number of tasks increases. And also an unfair comparison with related works.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688945462

thread_type:
official_review

<thread_object>
creator: reviewer_ueby

<content>
summary:
The paper proposes a new method for prompt based continual learning. Unlike previous methods that focus on task-specific prompts or growing prompt components, this work generates prompts using language descriptions of top-k similar classes from all the previously trained tasks to an input image. The generated prompts are plugged in to the Key and Value pairs of the backbone model for the given input image. In addition, the combined encoded language descriptions of the top-k similar classes are appended to the input image embeddings. Result on CIFAR100, ImageNet-R and CUB datasets with pretrained ViT-B/16 backbone shows significant improvements across accuracy and forgetting metrics over previous works. Extensive ablation study on different components provides better insights on the proposed approach.

strengths:
•	Introduction is clear in terms of discussing drawbacks of prior works and motivating the proposed approach. Contributions are listed clearly.

•	I appreciate Section 3 and Figure 1 as it gives provides brief recap of the previous SOTA methods and laid some foundation for discussing the proposed method.

•	Leveraging language descriptions and using all language descriptors to find top-k similar classes to generate the prompts is novel in the context of continual learning.

•	The strategy to incorporate inter-class loss helps the method to achieve better results.

•	Comparisons are made against various previous works and significant improvements are shown with their proposed method.

•	Appreciate the detailed ablation study to understand benefits of different components.

weaknesses:
I don’t find any major weakness but have set of questions (see below) to improve the understanding. However, I find that learning with auxiliary data and inter-task loss are critical components of the method (as shown in ablation study), which are not sufficiently emphasized in the contributions.

questions:
1. Between L155-160, under the context of previous task-specific approaches, it is mentioned that two different tasks could produce similar key vectors. It would make a stronger point to justify this statement with some empirical analysis.

2. What is the value of k for top-k descriptors and how many Generators are chosen for each task? Can we chose a single generator for each task? How {E_i}^t of size |C^t|gets chosen for k set of generators?

3. From the Figure 2 or its caption, it is not clear what is the component that is compared against input x to get the cosine similarity. The sentence “search space for top-k” in the figure gives some hint, but the arrows for entire blue block rising the confusion. Please clarify it in the figure or caption.

4. Why it is interesting to solve equation 3, if the task is to search for E_i in all tasks [1, 2, ….t]? Similarly, why consider K^t and why not just use {L_i}^t ?

5. Under the ablation study, what does absence of generator mean? How do you get prompts in this approach without generator?

6. Is the MLP head (classifier) also trained for each task? How can it produce list of softmax values of all classes for equation 7?

7. Please explain why inter-task loss help improve FAA and CAA? What does it tell us about the learned parameters?

8. Has the importance of inter-task loss already explored in previous continual learning papers? 

Minor suggestions:

a. In Figure 1, reposition the labels “fixed”, “trainable”, and “generated” since these labels affect entire Figure, not just the green block.

b. In section 3.b, I suggest to use emphasized text style for “pool-based approach”, “task-specific prompt approach”, and “growing component approach”.

c. In Line 242, correct the typo “tem” -> “them”

d. Line 267 mentions t_hat, but eq. 3 uses notation as t, but not t_hat.

------------------------------------------------------------------------
Final review: I thank authors for their responses. I find that paper proposes a novel method to leverage language descriptors within the context of continual learning. Authors have conducted thorough experimental evaluation and demonstrated notable improvements across multiple datasets. I read the fellow reviewers concerns and I find that authors have sufficiently addressed the major concerns and revised their manuscript accordingly. In particular, regarding the utilization of GPT, authors have justified that their method does not necessarily rely on compute heavy GPTs or LLMs, and can also work without GPT-generated descriptors. Overall, I believe that this paper would be interesting to the research community and suggest towards acceptance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732239457991

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers and Summary of Updates

comment:
We would like to express our gratitude to all the reviewers for their constructive and insightful comments on our paper. We are honored that all the reviewers (ueBy, J5sH, UGi2, and bACr) found our work to be novel/new. We also appreciate reviewers (ueBy, and bACr) for pointing out the detailed writing of our paper as a positive point. We have revised our paper, following the reviewers' feedback. We have improved the quality and presentation of our paper with the summary of changes as follows:

(1). We added a detailed overview of our method as presented in section 4.1. to improve e better presentation and understanding of our proposed method. We emphasized the uniqueness of our method to existing SOTAs in the section.

(2). We replace our ablation study with a step-by-step scenario to improve the clarity of each component contribution, as presented in section 5.2.d. 

(3). We added a new empirical experiment and analysis on the high similarity of task key (K^t) as presented in Appendix D.1. The high similarity of task keys motivates us to develop a more accurate task-id prediction mechanism rather than the conventional way.

(4).  We added a new empirical experiment and analysis on the high similarity of language embedding generated by the encoded string "the photo of class name" as used in the existing method. The high similarity between classes motivates us to develop a more discriminative way of language-guided learning rather than utilizing the embedding directly as references or loss anchors. Please see Appendix D2. 

(5). We added a new experiment and analysis on the trade-offs between cost and performance that comprehensively evaluate the proposed methods and existing SOTAs in terms of performance (accuracy and forgetting), number of parameters, and running time. We also compare our proposed method with the large version of SOTAs that executes a higher number of parameters. Please see Appendix D3.

(6). We added a new experiment and analysis on the "class name" as descriptors where we have no class descriptors generated by GPT or other LLMs. Our experiment results as presented in Appendix D4. shows that our method outperforms the existing SOTAs with a significant margin in 3 datasets despite substituting class descriptors with the class name. 

(7). We elaborated the detailed setting for all consolidated methods as presented in Appendix E.

(8). We revised Figures 1 and 2 for a better understanding of our paper. 

(9). We improve the readability and presentation of our paper by ensuring a clear space between a text to another text, caption, figure, or table.

(10), Last but not least, We revise the writing errors and typos. 

The changes w.r.t. feedback from reviewers 1(ueBy), 2(J5sH), 3(UGij), and 4 (bACr) are highlighted in red, green, orange, and violet colors respectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732710278352

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Updates

comment:
Dear Reviewers and Area Chairs,

We added additional updates in our latest manuscript that fully address the comments/concerns of reviewers  J5sH, UGi2, and bACr as follows.

1. We added experiments and analysis of the lite version of LEAPGen i.e. LEAPGen-lite that utilizes lightweight Conv1d generators and without class descriptors (it uses class names only). LEAPGen lite has far smaller parameters than ConvPrompt (12.7% of ConvPrompt #params) and is similar to Hide-Prompt parameters. Our experiment shows that LEAPGen-lite outperforms the existing SOTAs i.e. 4.7-25% FAA and 3-11% CAA in CIFAR100, 3-30% FAA and 2-26% CAA in ImageNet-R, and 6-20% FAA and 3.3-14% CAA in CUB. This evidence proves that the significant performance of our ideas (language as input for prompt generation, task-wise generators, soft task-id predictor, and learning with auxiliary) is not bounded by LLM-generated descriptors or the complexity (parameters and size) of the generators. Please kindly see the results and analysis in Tables 2,3,4, Figure 3, and Section 5.2.d.

2. We added a summarized analysis on #Parameters, Running Time, and Storage as presented in section 5.2.g, detailed in Appendix D.3. The analysis shows that LEAPGen spends moderate inference time, but lower training+inference time and total running time, than ConvPrompt and Hide-Prompt, despite having more parameters and storage. LEAPGen-lite requires far smaller cost in all aspects than ConvPrompt and smaller running time and storage than HiDe-Prompt, with just a slightly higher number of parameters i.e. 0.16M vs 0.15M. We move the analysis of LEAPGen performance on various types of generators to the Appendix for the exchanged space in the main paper. 

Both additional updates are presented in brown color in our latest manuscript.

Thank you.

Best Regards

Author of Submission 6684
</content>
</thread_object>

</thread>
<thread>
creation date:
1733295995118

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision and Discussion

comment:
Dear Program Chairs (PC), Senior Area Chairs (SAC), Area Chairs (AC), and Reviewers,


As the discussion phase has ended,

**First**, we thank reviewers PC, SAC, AC, and Reviewers for your effort and time in organizing the review and discussion of our paper. Through these processes, we have improved the technical quality and presentation of our paper significantly.  We thank the reviewers ueBy and J5sH for reviewing our paper and/or its revision thoroughly, confirming the novelty and significance of our works.  


**Second**, we have revised our paper following the reviewers' advice in both writing i.e. **layout, presentation, details, tables, figures, etc**, and technical aspects i.e. **method overview, ablation study, the trade-off between performance and cost, empirical/numerical evidence of task similarity, language embedding similarity,  setting details, etc** as we mentioned in our previous official comments.


**Third**, we would like to confirm that we have addressed all concerns and questions of the reviewers in our revised paper and responses. However, we would like to emphasize a few key points that reviewers UGi2 and/or bACr may not have reviewed thoroughly in the discussion phase, as follows:


1. **Fairness Setting** :   LEAPGen utilizes the same resources as ConvPrompt and LEAPGen-lite(no descriptors with lightweight generators) utilizes less resources than ConvPrompt and comparable resources to the other SOTAs (sec. 5.1, sec. 5.2.g, Appendix E). All methods are evaluated in the same setting and run with respective best (recommended by official paper/code) hyperparameters. Thus we believe we have satisfied the fairness aspect in our study.


2. **The Non-Mandatory Presence of GPT**  : 

- **GPT is not part of our method** (sec 4.2.a, and fig. 2), It is one of the options to generate class descriptors following ConvPrompt. Our method accommodates both class descriptors (as in ConvPrompt) and class names (as in LGCL) language text. Even without GPT-generated descriptors. our method works excellently and significantly outperforms the existing SOTAs, as demonstrated by LEAPGen-lite (sec. 5.2.d) and LEAPGen-CN (Appendix D.4). 

- **GPT is utilized in an online way** (through the Internet), by querying it using API and Python script. It is a common practice nowadays as in our simulation (please see https://platform.openai.com/docs/guides/). Thus we never download and save the GPT model in our storage. LEAPGen indeed saves the descriptors embedding (numerical vector), but not the GPT model.  The embedding adds to the whole model's storage size but is still in a reasonable amount, and we have included it in our storage analysis. 

- **Descriptor generation by GPT** (before training) via only query indeed takes a fair amount of time. However, despite spending extra time on it, our method still has a **lower total running time** than existing SOTAs, i.e. ConvPrompt and HidePrompt. (sec. 5.2.g, Appendix D.3)

- **Descriptor impact** to the model performance has been demonstrated by the performance difference between LEAPGen and LEAPGen-CN (Appendix D.4). We also extend our analysis of the performance of LEAPGen vs ConvPrompt in various types of descriptors (Appendix D.5).


3. **LEAPGen-lite** proves the significant impacts of our ideas i.e. (1) language as input for prompt generation, (2) task-wise generators, (3) soft task-id predictor, and (4)learning with auxiliary despite without GPT-generated descriptors (only utilizes class names), and utilize least parameters, running time and storage.  (sec. 5.2.d, sec. 5.2.g, Appendix D.3) 


We believe these key points **resolve all concerns** regarding the **fairness setting** and **GPT (non-mandatory) presence**, and prove the **significant impacts** of our ideas i.e.  (1) language as input for prompt generation, (2) task-wise generators, (3) soft task-id predictor, and (4) learning with auxiliary, without being bound to the presence of GPT. 


I think that's all, once again thank you to all the committee, we pray that the hard work and dedication of the committee will be paid off by the success and impact of this year's ICLR, Thank you.


Best Regards,

Author of Submission 6684
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper proposes LEAPGen, a novel method for continual learning that generates prompts using language descriptors for classes instead of shared embeddings, with a learnable task key and class-level parameters. LEAPGen predicts the task ID and selects top-k matching descriptor embeddings to generate prompts. Experimental results on CIFAR100, ImageNet-R, and CUB show that LEAPGen outperforms state-of-the-art methods in accuracy and mitigating forgetting.

(b) Strengths: The strengths of the paper lie in its clear and well-motivated introduction, which effectively highlights limitations of prior works and outlines the proposed approach. LEAPGen introduces a novel use of language descriptors for prompt generation, leveraging richer semantic information and avoiding the need to store or replay old data, thereby reducing catastrophic forgetting. The method achieves significant improvements over state-of-the-art methods and incorporates a unique inter-class loss strategy to enhance performance. The experimental results are robust, with thorough comparisons against existing methods and detailed ablation studies to validate the contributions of individual components. Section 3 and Figure 1 provide a concise recap of prior methods, laying a solid foundation for the proposed approach.

(c) Weaknesses: The paper has several weaknesses, including a lack of clarity about key details, such as the specific Sentence Transformer used, whether it is pre-trained, and whether any fine-tuning occurs during the continual learning process. The scalability of the proposed method is a concern, as adding new generators for new tasks could significantly increase the number of parameters, leading to potential unfair comparisons with related works. The writing and structure require improvement, as the compact format affects readability, and some components, like the theorem, seem irrelevant and better suited for supplementary materials. The ablation study lacks rigor and should more systematically validate the contributions of each module starting from a standard baseline. Additionally, the use of ChatGPT for generating descriptions raises concerns about the fairness of comparisons and the source of performance gains, prompting the need for transparency and public availability of these descriptions.

(d) The most important reasons for acceptance are the solid methods and compelling performance, along with the sufficient theoretical & empirical analyses. Since this paper has received 2 negative scores (i.e., 3), the AC has checked very carefully about the details and discovered that the major reasons for low scores, i.e., the inclusion of GPT for accurate description, are well addressed by the authors. The authors have designed a lite version (LEAPGen-lite) of the proposed approach which does not rely on GPT entirely. The LEAPGen-lite still shows significantly superior performance to previous sota methods, proving the significant impacts of the ideas. In the AC's view, all the issues have been addressed during the rebuttal period. By the way, Reviewer UGi2 who scores 3 fails to participate in the rebuttal period despite reminders from the AC.

One more suggestion for the authors to improve the presentation quality: In the current form of this manuscript, the authors show 4 principles. Four principles are too many; for a highly impactful paper, one principle is sufficient—for example, the residual idea in ResNet. The AC suggests the authors reconsider the core problem this paper aims to address, propose a single principle, and then present multiple methods to support this principle. This approach would enhance the clarity of the motivation and increase the paper's overall impact.