Vision and Language Synergy for Rehearsal Free Continual Learning
Authors: Muhammad Anwar Ma'sum
Mahardhika Pratama
Savitha Ramasamy
Lin Liu
H Habibullah
Ryszard Kowalczyk
Venue: ICLR 2025 Poster (2025)
Keywords: continual learning
prompt dilemma
language descriptors
prompt generator
catasthropic forgetting
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: The prompt-based approach has demonstrated its success for continual learning problems. However, it still suffers from catastrophic forgetting due to inter-task vector similarity and unfitted new components of previously learned tasks. On the other hand, the language-guided approach falls short of its full potential due to minimum utilized knowledge and participation in the prompt tuning process. To correct this problem, we propose a novel prompt-based structure and algorithm that incorporate 4 key concepts (1) language as input for prompt generation (2) task-wise generators (3) limiting matching descriptors search space via soft task-id prediction (4) generated prompt as auxiliary data. Our experimental analysis shows the superiority of our method to existing SOTAs in CIFAR100, ImageNet-R, and CUB datasets with significant margins i.e. up to 30% final average accuracy, 24% cumulative average accuracy, 8% final forgetting measure, and 7% cumulative forgetting measure. Our historical analysis confirms our method successfully maintains the stability-plasticity trade-off in every task. Our robustness analysis shows the proposed method consistently achieves high performances in various prompt lengths, layer depths, and number of generators per task compared to the SOTAs. We provide a comprehensive theoretical analysis, and complete numerical results in appendix sections. The method code is available in https://github.com/anwarmaxsum/LEAPGEN for further study.

Published as a conference paper at ICLR 2025
VISION AND LANGUAGE SYNERGY FOR REHEARSAL FREE CONTINUAL LEARNING
M. Anwar Ma'sum' MMhaardika Pratama', Savitha Ramasamy?, Lin Liu', Habibullah Habibullah', and Ryszard Kowalczyk* Univeesiity South Australia, Mawson Lakes, SA, 5095, Australia Institute for Infocomm Research, A STAR & IPAL, INRS@CREATE
(at.ryassaaaaaaaatadd.dd.... masmy039 @ mymail.unisa www.wu.wi.uuuua.a.. edw.aau au, dhika. dhika.pratama pratama @ wwwisa.ed..aaaa. lin.liu @ unisa.edu.au, habibullaahhhabibullal habibullah @ unisa.edu.au, ryszard. kozzard.owwaczykk @ uunisa.edu.a
ABSTRACT
The prompt-based approach has demonstrated its success for continual learning problems. However, it still suffers from catastrophic forgetting due to inter-task vector similarity and unfitted new components previously learned tasks. On the other hand, the language- language-guided guided approach falls short of its full potential due to minimum utilized knowledge and participation in the prompt tuning process. To correct this problem, we propose novel prompt-based structure and algorithm that incorporate key concepts (1) language as input for prompt generation (2) task-wise generators (3) limiting matching descriptors search space via soft task- id prediction (4) generated prompt as auxiliary data. Our experimental analysis shows the superiority of our method to existing SOTAs in CIFAR 100, ImageNet- R. and CUB datasets with significant margins i.e. up to 30% final average ac- curacy, 24% cumulative average accuracy, 8% final forgetting measure, and 7% cumulative forgetting measure. Our historical analysis confirms our method suc- cessfully maintains the stability-plasticity trade-off in every task. Our robustness analysis shows the proposed method consistently achieves high performances in various prompt lengths, layer depths, and number of generators per task com- pared to the SOTAs. We provide a comprehensive theoretical analysis, and com- plete numerical results appendix sections. The method code is available https: /tttps///gte //gi thub. com/ anwa rmax XaaaxxxumLEPPCCO for further study.
INTRODUCTION
Continual learning (CL) attracts tremendous interest in the world of artificial intelligence (AI) for dynamic environments. CL methods address the catastrophic forgetting when handling a sequence of tasks De Lange et al., 2021; Wang et al., 2024b. CL methods are proven promising in various applications c.g. computer vision, NLP, graph. and Automation (Liu et al., 2023; Biesialska et al., 2020: Tian et al., 2024; Shaheen et al., 2022). However, most of the CL methods require a small por- tion of previous task exemplars for a rehearsal process to maintain the knowledge from previously learned tasks (Rebuffi al., 2017; Wu et al., 2019; Boschini et al., 2022). This approach isn't prac- tical since old data can be unavailable anymore due to privacy constraints or data openness policy. Besides, the rehearsal approach incurs additional memory and computational expenses. To this end, a prompt-based approach comes as breakthrough solution. Leveraging frozen pre-trained back- bone, the prompt-base based approach demands only small added learnable parameters (prompt) to adapt to new task. This approach leads to efficient yet accurate solutions for catastrophic forgetting.
Despite its promising performance and efficiency, the prompt-based approach suffers from the fol- lowing dilemmas: First, the task-specific prompt approach e.g. DualPrompt( Wang et al., 2022b), SPrompt( Wang et al., 2022a), HiDE-Prompll Wang et al., 2024a) and CPrompt (Gao et al., 2024) rely on task identifier where misidentified task-id leads to misclassification e.g. in the case where the trained keys two or more tasks have similar vector. Second, the growing component approach e.g. CODA-P(SSmit
al., 2023b), EvoPrompt( Kurmiawwn et al., 2024), and ConvPrompt(Roo et al., 2024) on the other hand increase prompt components to adapt with a new task, instead of the task identifier. However, the (new) extra components optimal for the current task, but not for previous tasks. In addition, the old task components/generatoos that are trained side-by-side with the current.
"Corresponding author

---

Published as a conference paper at ICLR 2025
task componeetss/eeneratoo lead forgetting. Third, the shared learnable parameter approach such as G-Prompt in DualPrompt or semantic embedding in ConvPrompt has the risk of forgetting since it is adjusted in all tasks. This condition is similar to the pool-based approach i.c. L2P(Wang al., 2022c) where any prompt in the pool may be trained in every task, as it is chosen by the input-to- prompt similarity mechanism. On the other hand, language-guidet approach e-g. LGGLL (Khan et al., 2023) and GMM (Cao et al., 2024) do not fully reach the optimum potential of language assistance in prompt tuning since they only use the name of class as additional information.
In retrospect to the aforementioned drawbacks, we propose a novel approach for prompt-based con- tinual learning. We design new prompt generation and its learning mechanism to handle catas- trophic forgetting. The key principles of our approach are (1) we utilize language as input for prompt generation instead of a tasks-shared learnable vector. The input selected from the catalog of lan- guage descriptors based on their similarity to the input. (2) We utilize task-wise generators instead of growing generators to generate a prompt component, thus the current task generators are frozen in the upcoming tasks training. (3) We propose a new soft task-id prediction to limit the search space of matching descriptors. Our mechanism is different from the task-id prediction in the previous methods. (4) We utilize the generated language-bass based prompt as the auxiliary data to be appended to the input embedding. To our knowledge, these four principles are new in the prompt-based based CL method and have not yet been explored in the previous State-of- -ftte-o--hh--ar(()) (SOTAs).
Our contributions are: (1) We propose a novel rehearsal-free prompt-based method for CL. problem named Language as Prompt Generator (LEAPGen) consisting of four main components as afore- mentioned. (2), We design a new task-id predictor and joint loss function for our method. (3). Our rigorous experiment and analysis prove the superiority of our method compared the existing SO- TAs in general and historical performance in terms of accuracy and forgetting index. Our extended analysis proves the robustness of the proposed method in various settings i.e., the number of layers, prompt length, and number of generators. (4). We provide rigorous analysis and discussion of our method theoretically and numerically, and complete numerical results, please see appendices.
2 RELATED WORK
(a) Continual Learning: DualPrompt (Wang et al., 2022b) and CODA-P (Smith et al., 2023b), offer a breakthrough solution for Class Incremental Learning (CIL) by training tiny task-aware parame- ters called prompts where the feature extractor c.g. ViT that contains far bigger parameters remains frozen. The prompt-base based approach is proven to be more effective than the rehearsal approach e.g. ICARL (Rebuffi et al., 2017). GD (Prabhu et al., 2020), exemplars from the previous tasks and replays them along with current task samples, the bias cor- rection approach e.g. BiC (Wu et al., 2019) and LUCIR (Hou al., 2019) that trains an additional task-wise bias layer to balance the model's 's stability-plasticin dilemma, and the regularization ap- proach e.g. EWC (Kirkpatrick et al., 2017), MAS (Aljundi et al., 2018), and DMC (Zhang et al., 2020) that tunes the learner parameters to accommodate the previous and current tasks. However, the prompt-based approach still has above-mentioned dilemmas leading to forgetting-
et al., 2022) that saves
(b) Language Guided Learning: Inspired from CLIP (Radford et al., 2021), the language-guided approach becomes a new alternative method to assist rehearsal and regularization approaches (Ni et al., 2024), contrastive learning (Zhu et al., 2023), few-shot continual learning (Park et al., 2024), generative approachh (Cao et al., 2024), and prompt- based methods (Khan et al., 2023; Roy et al., 2024). However, it is still far from its optimum potential since only utilizes small knowledge from the learned class such as the name of the class or inter-class text similarity to generate several new prompts. The discriminative and representative knowledge of language modality is not yet utilized in the learning process. Please see Appendix F for the detailed literature review.
3 PRELIMINARY
(a) Problem Formulation : In this study, we focus on Class incremental learning (CIL.) since it is the most challenging sub-problem in CL. Class incremental learning (CIL) problem is defined as the problem of learning a sequence of fully supervised tasks { J where after finishing learning a task T', model must recognize all learned tasks i.e. T1,7 .... T'.,,t,,,'' Symbol T represents the number of consecutive tasks. Each task carries pairs of training samples i.e. T
{(xt, ! where x, E X and ys E y² denotes input image and corresponding label, while I-| denotes cardinality. C' denotes the unique class labels in y i.e. C unique())") ), and |C*| denotes the number of classes in T Each task / is disjoint from another task L i.e Vt, L' + t, (T' = 0)
2

---

Published as a conference paper at ICLR 2025
Prok-faned
Growing Corporeet Apgrouch
Lenguage died Approuch
L2P
gol
OCDAUP
LOCL
Teak Spac to Aponoach
QualPronps
sal-eers prccppas
pr
FOR
poral
GVR!
Chocee
peetoctied spaer melorn)
penentor
G²
GM
Exsiroopl
Siraediig
aromect
Previl
AC1
X
velected
Figure 1: Visualization of Pool-Based Prompting (red area), Task-Specific Prompting (blue area), Growing Component Prompting (green area), and Language Guided Approach for Continual Learning (gray area).
(b) Types of Prompt-based Approaches: Figure visualizes the topology of prompt-based ap- proaches. The pool-hased approach e.g. 1.2P(Wang, et al., 2022c) selects top-k prompts from the prompt pool [PL P2_ ... where M is the pool size. Any prompt fm in the pool is trainable at any task & E [1..7]. The task-specific prompt approach e.g. DualPrompt( (Wang et al., 2022b) trains only the corresponding prompt P during the training process on task t while the prompt asso- ciated with the previous tasks i.e [P1, p2 ,... ...P(t-1)) are frozen. Each task / accommodated with a learnable key R" to identify the task-id and select the prompt P during the inference process. The prompt identification is based on the highest cosine similarity to the input. HiDe- HiDDe-Promp((( Wang ct al., 2024a) utilize unstructured representation (centroids) produced by unsupervised learning instead of learnable keys. CPrompt(Gao Gao et al., 2024) randomly selects a frozen prompt from [P], p2 ,.... pt to support the tuning of Pt. The growing component approach dynamically increases the number of prompt components e.g. p',p't ..p" to adapt to a new task. The final prompt is generated by weighted sum of the components with similarity coefficients e.g. P = ΣM Emmp'aa In CODA-P(Smith et al., 2023b), p' is pre-defined learnable parameter, while a' = =<((()), k²) is attention similarity between input query (zx and learnable key k associated to p'. In ConyPromp((Ry et al., 2024), p' is generated by respective generator network G from shared embedding (SE). Similarly, a² is the cosine similarity between input x and k². EvoPrompt(Kurmiann memory W that forms the fusion of referred (previously learned) memory w² and current working memory w/. W is implemented is generated by the formula P = (A()U(()))
et al., 2024) grows prompt
linear model with 2 parts i.e. W and W and the prompt P
(c) Language Guided Approaches: The gray colored area of figure 1 shows how the role of language modality for the prompt-based method (LGCL) and generative method (GMM). LGCI,(Kaan al., 2023) generates task level prototypes i.e. T.2, I.2..... and class level proto- types e.g. L. T.y,....;; then utilizes the prototypes for loss computation i.e. Li(K", I.',T.") and Le(X, LT_ : L where L, and Le are task-wise and class-wise losses respectively, X and L' are the ViT output and language prototype for current learned class respectively, K is the learnable key associated with prompt pt, and L and L are task-level and class-level prototypes taken from the previously learned tasks. GMM(Cao sentences into embedding and then passes the embedding into the LLM decoder. The decoder gen- crates predicted embedding associated with the ground truth embedding. The cross-entropy loss of predicted and ground truth tunes a trainable linear model.
al., 2024) transforms question-and-answee
(ground truth)
(d) Dilemma and Drawbacks: The pool-based approach has a high risk of forgetting since all the Prompts [Pl, p2. J] are possible to be trained in all tasks t E [1..7]. A prompt Pd is optimal for the th task but is tuned again in the + 1h task. Therefore, it is no longer optimal for gth task and leads to forgetting. In a task-specific approach, two different tasks e.g. + L' could produce similar key vectors i.c. 0.73 < cos(K', )² < 0.96 (Please see Appendix D.1) that leads to inaccurate task identification that leads to misclassification and forgetting. Note that K is trained based on all samples of all classes in T T and T'' are indeed disjoint so that X 'x = 0. but they could have similar representation i.e. So (2(') R Jo(2'*) such as in CUB dataset where all the images are the photos of bird. Similarly, HiDe-Prompt suffers from the same drawback where two representations from different tasks have similar values Sa E T N S3 E T In the growing component approach, newly added components i.e. p'pp't1, Ap disrupts the previous components i.e. p² p2 .-p'11
3

---

Published as a conference paper at ICLR 2025
that already optimal for previous tasks. In addition, utiliziig continuously learned parameter e.g- shared embedding (Roy et al., 2024) or task-sharing prompt (Wang et al., 2022b) increases the chance of forgetting. The existing language guidance is not fully explored. LGCL produces class prototype I by encoding string "the photo of class name". However, the prototypes could be misleading due to high similarity between different classes, e.g. the prototype of class "Great White Shark" has 0.9 cosine similarity to the prototypes of class "Tree Frog" and "Iguana", please see Appendix D.2. With such different classes, we can't 't directly utilize language representations as references or guidance for model training.
presents the average accu- racy and average forgetting of all learned classes in the first, sec- ond, and last tasks on ImageNet-R dataset with 10-tasks setting. The table shows that the existing prompt-based methods suffer from 9-15% average accuracy drop between the first and the final tasks. In the second task, the methods already suffer from up to 5% average forgetting and the amount tends to increase to up to 8% in the last tasks. Instead of reducing the average forgetting, almost all methods experience higher forgetting in latest tasks. HiDe--rompt manages to reduce the average forgetting with the highest amount i.e 1%, but in the last task, It achieves 1 - 2% lower accuracy than the best achiever i.. ConyPrompt. Despite its highest performance in the final tasks, ConyPrompt suffers from a fairly high accuracy drop i.e. 12% and an increase of 1.5% forgetting. This preliminary numerical result confirms our aforemenntoned analysis.
(e) Preliminary Analysis:
Table
Method
Avg. Accuray
T10
Avg. Forgetting
TT0
lnc.
L2P
76.89 62.30
14.39
5.52
5.0)
-0.51
DusiPrompt
78.97
68.59
10.39
3,83
4.61
0.79
CODA-P
89.24 73.77
15:47
5.04
7.1
2,90
LGCL
78.59
6846
4:75
1,85
HiDe DiDe-Procppt
85,22
75.75
9.47
3.29
2.29
1100
PGP
78.97 68.62
10.35
3.83
4,53
0,70
EvoPrompt
89,27
76.00
4.22
0,95
CPrompt
6.10
1.54
ConyPromps
89.53 77.08 12:45
2.57
4.17
1.61
Table I:
Preliminary results on
Imagenet-R R with 10 tasks setting.
4 PROPOSED METHOD
4.1 OVERVIEW
In this study, we propose novel LanguagE As Prompt Generator (LEAPGen) accommodating main principles that are emphasized in the introduction section. The structure and flow of LEAPGen are visualized in figure 2. LEAPGen generators produce prompts from top-k selected embedding as input. LEAPGen also produces auxiliary (aux) data from the top-k embedding. The prompts are prepended into ViT MSAs while the aux appended into input patches, thus producing feature and final prediction by ViT layers and MLP head respectively. The top-k embedding is selected based on the cosine similarities between an input and the class-wise keys. LEAPGen limits the search space into task 1 to predicted task f. by performing soft task-id prediction. In each task of the training phase, LEAPGen updates aasoiassoed learnable parameters i.e. generator, task-wise key, and class-wise keys. In the inference phase, LEAPGen selects the generators based on the predicted task LEAPGen utilizes cross-entropy loss and cosine similarity loss to optimize its parameters. Task- wise generators, language embedding for prompt generation, and soft task-id prediction are unique to recent SOTAS of evolving generator methods e.g. (Roy et al., 2024) and (Kurniawan et al., 2024), wise fixed prompt methods such as (Wang et 2024a) and (Gao et al., 2024). and pool-based task-prediction mechanism,
method (Wang et al., 2022c) in terms of prompt ceneration/sellction and modality for prompt generation. The detailed architecture, flow, and learning mechanism presented in sub-section 4.2 and 4.3.
4.2 ARCHITECTURE AND PROMPT GENERATION
(a) Input and Learnable Parameters: while the existing works use the language to compute loss as LGCCL(Kaan et al., 2023) or calculate the number of generators per task as in ConvPrompt (Roy et al., 2024), we take the benefit of language modality as the input for prompt generation. Our method accommodates both language text i.c. generated descriptors by GPT as in ConyPrompt and class names (as utilized in LGCL) as descriptors. Descriptors list carries text describing the visual attributes of the classes, e.g. the deseriptors for class "apple" are ['round shape", "smooth, glossy skin", "red, green, yellow color", "stem and leaves", "five-pointed star shape when cut in half", "white flesh inside"]. The descriptors are more distinctive and representative than the words "the photo of apple" utilized in the existing works. The list of descriptors is then unified into a single long string and encoded into a numerical vector i.e. E. E RR_ called a descriptor embedding. We store the descriptor embedding from the learned classes as the catalog for prompt generation. Each task t contain k generators {G) task-wise leamable key Kt E IR and a set of class-wise learnable parameters [LA) L E RP, where k is a predefined number, and |C*| is the number of classes on task 1, each T.g is associated to Ez. These learnable parameters are tuned only task-I.
4

---

Published as a conference paper at ICLR 2025
top descripor
emoodoiings
A%
Fi
9
%
B
G
PCI
8:
Cloxadier (MLP)
Cosine Sim
PC's
Pl
search space tt
preditte lokkut lae tosling
MUP
Toodo
PGL
Promet E/S,
Gonenstora
of
4
4
MBA
Alletton
Gencrator Nenwork
assodksedd
descttstrr emboediingg
Ei
E
W/
Incut Image
Ri
Segiarcee Trurrafurmerr
A
W
deserpoor
Cosc
Text
Cusc
Tex
Cuso
Tiei
Ther
cim--
cimo-
dem-10
dem--11
clam-112
clmm-13
D| Lourmaate
Pood
Cenweree
Auookkory
Entoodzed
Figure 2: The LEAPGen architecture. Each task contains set of generators, a task-wise learnable, and class- wise learnable parameters associated with the descriptor embedding. Given an input image, our method predicts the task-id and finds top-k k descriptors matched to the input via cosine similarity afterward. The descriptors are dispatched to the generators producing the prompt components. LEAPGen produces the final prompt and an auxiliary embedding by the weighted sum of the prompt components and the descriptors respectively. The prompt prepended into the ViT MSA layer, while the auxiliary is appended into the input patches.
(b) Prompt Generation: Before generating the prompt for an input image x, our method predicts the task-id via soft task prediction (ref. point c). In the training process, the task-id easily given in a supervised way instead of being predicted. Let the task-id be f, then our method selects k descriptor embedding associated with top-k matched L by performing cosine similarity. Note that
(E2) ... {E Let the search space is from the first task (task-1) until task-t i.c. { {EeJe=1""
the k selected embedding be {Ei) and associated similarity values {S: then the embedding is inputted into the respective generators i.c. { producing prompt components {PC:)E-- Note that each generator a trainable network, so it satisfies PC; = G. (F... The final prompt is generated by weighted sum of the prompt components and similarity i.e. P Mg S,P.. In simplified expression, the prompt generation is formulated by equation 1.
P Σ (.G
(c) Generator Networks and Detailed Generation: we design single-head self-attention (SSA) network G parameterized by W w.k and W.Y without bias as visualized in figure 2. Following (Roy et al., 2024) and prefix tuning (Li & Liang, 2021), we implement G; as key-value pair generators i.e. (G4K G:V) and distribute them into corresponding layers and heads. Therefore, a hth head of MSA in 1th layer corresponds to a pair of generators (G G,,,A) parameterized by (WKQ i. WKV., k. W WKY,, (.) and (WV. WKV. WVV..). where W WKR i.1.h is (D/H) x (D/H) matrix and so do the other parameters. The language embedding E, is then divided into H parts i.e. E, = [Ea,,1, E.I.2,,.Ei,,,, where part Ei E [1..H] plays as the input for (G G.V (. Thus, the detailed formula of prompt generation for respective head h on th layer is expressed as in equation 2.
PK = Σイ S,G.
= (F4.A)
(d) Soft Task-ID Prediction: we design soft task-id prediction limit the search space to find top-k Eg. We use the term "soft" as the search of top-k E, is conducted not only within predicted task t but also the previously learned tasks i.c. [1, 1). The reason is that most similar language embedding comes from the respective class from within respective tasks, the second -ranedd and the rest may come from classes from different (previous) tasks. Therefore, they will contribute to a better distinctive prompt than the embedding from the other classes within the same task. For instance, class "pear" has higher similar descriptors to class "apple" than to class "palm tree". But class "apple" is located in different (previous) task, while class "palm tree" is within the same task as class "pear", ; thus selecting "apple" descriptors from the precious task is better than choosing "palm tree" descriptors from the same task. Different from the task-id prediction in previous works (Wang et al., 2022b; 2024a) we utilize both task-wise learnable parameter K' and class wise learnable parameter T.C. We predict the task-id & for an input & by finding the maximum product of cosine
5

---

Published as a conference paper at ICLR 2025
similarity between 2 and K* and similarity between r and L/ as formulated in equation 3.
t = arg max (zx. ((a..))(((mmarr)llb:llll) (ll2..)))))) (0 c))
(3)
ce[1.T].cEC'
(d) Learning with Auxiliary Data: Our method generates auxiliary embedding based on selected language embedding used for prompt generation. The auxiliary embedding Touz RDD helps our method to recognize the input image v as it is generated based on the similarity between the input and the catalog of language embedding. The previous study(Shen et al., 2023) presents insightful knowledge of leveraging the auxiliary modality for deep learning training. The auxiliary embedding Toua generated by the weighted sum of language embedding i.c. Zauæ > S S,E, The auxiliary embedding is concatenated into the input patches list, therefore the patches list that enters the ViT layers now becomes [Tey Begy un FeN," Faur), where Np the number of patches.
4.3 FORMULATION OF LEARNING MECHANISM
(a). Theoretical Foundation: The goal of CIL is to learn sequence of tasks i.e. TI, .?...... where each task T E [1..T] carries a pair of image and label set (X* y²). Suppose that C is the set of available classes in )", and E C' represents class label e available in task f. Suppose that " denotes the concatenation and T = -|=11 T denotes the concatenation of task to task Similarly, X11t and y²lit denotes the concatenation of input and label space respectively, from task to task t. Given a sequence of experienced events T = {T', . .,.,... T²), }. and a deep learning model fo(.). The objective of CIL is to recognize any input E X1:I" by maximizing P(x E X [T, 0). Following (Wang et al., 2024a), P(z E (6, can be decomposed into task prediction and class prediction within task as in equation 4.
TT,
(4) Analogically, we can decompose the P(x E X, 0) in softer task-id prediction i.e. by decom- posing the term P(x E X w.r.t p(x E X (T. Then, the decomposition can be derived into equation 5. P(x X 0) P(x E X X T, E E X²)
P(x () P(æ E X I E X", T,0)) E XT)
(5)
=P(x E E X PT,) E 1
The first line of equation 5 expresses hard decomposition where a classification is conducted into a strict two-step classification i.c. predict the task ID first, then predict the class label within the task. On the other hand, the second line of the equation 5 expresses a soft way of decomposed classification i.e. predict the possible task (1:t) of the sample then predict the class within the task range. In the first approach, a false task prediction directly leads to misclassification, while in the second approach, It still hold a possibility for correct classification as long the predicted task ID is higher than the ground truth task. Since event æ E X' is subset of event x E x Then P(r E X² The second approach has an easier way to predict the possible task but a harder step afterward, while the first approach has a more difficult task prediction but easier way afterward. However, the equation above gives us insight that improving P(x E X (T 7, 9) can be conducted by improving P(x E Xa E X T, P(x E X²I 0). P(x X J 1 T,,), 0). P(x X1: 0), pair to the first line, pair of the second line of equation 5, or all of them.
P(æ (T,)), that implies P(x E X jr E X²) > P(z E X E x²
(b). Learning Objective: Based on the insights above, we develop strategy to improve P(r E X.(T,O) during the learning process. Note that ab is the frozen ViT parameter, 0 = (G,, Kt 1'TE c- are our customized trainable parameters including generators, task-wise keys, and language class-wise keys, and ò is the parameter of MLP head (classifier) and the class label of an input x is predicted by computing hs(f..o(z)) First, utilizing cross-entropp function, we improve P( L. J E X T, 0) by intra-task loss Cintra as formulated in equation 6.
exp(hs (Sv (x))(()
Lintra (x. (1/, 0)) = -
E log EeECI exp(hs (fv. REC
(6)
where "[]" denotes the index filter for the respective class. Note that his (f)) produces list of softmax values for all classes. Second, similar to the intra task loss, utilizing cross entropy function, we improve P(x E X I E X1=t, T, TT0) by inter-task loss Linter as defined in equation 7.
t=
(
Linter (a, (1.
Σ
Σ log CEC
E Evec exp(hs((e

---

Published as a conference paper at ICLR 2025
Then, we improve task prediction P(r computing negative cosine similarity between the imput x and wase and class wise learnable parameters K' and L respectively formulated in equations and 9. Note that designed task-id predictor utilize both Kt and L as explained in sub-section 4.1.c
XT),,0 that leads to improve P(z XT 0) by
Li Kt) = (x. ( | 112, ())
(8)
C, = - -x.
(9)
Finally, we define a joint loss function as a total loss accommodating the four loss components above to train the learnable parameters, generators, and classifier head formulated in the equation 10. We add loss coefficients i.e. \1, >2. As to enhance the flexibility of our proposed method e.g. the higher number of learned classes needs higher capability inter-task discrimination. We don't 't utilize any regularization, since all learned parameters on task-l will be frozen after finishing the task.
Liotal = =Lintrale (zr, (1, 0, 0)) + Linter (z, (1b, 0))) + A2Li(1, K*) + A3 T.)
(10)
(c). Training and Inference procedures for LEAPGen are presented in Appendix B.
4.4 THEORETICAL ANALYSIS
We state Theorem 2, and 3 that prove the importance of the generator network, loss minimization, and better task prediction respectively as presented in Appendix A. Please see Appendix A for our detailed theorems and theoretical analysis.
5 EXPERIMENT AND ANALYSIS
5.1 EXPERIMENT SETTING
Datasets: We evaluate our method in three CIL benchmarks i.e. CIFARR100(Hendryeks et al., 2021), & Popescu, 2019), and CUB(Wah et al., 201 The CIFARI00 has 100 classes of small-scaa images while ImageNet-R contains large-scale images that cover 200 classes. ImageNet-R contains art, cartoon, paint, and deviant images. The CUB dataset includes fine-grained images of 200 classes of birds. We follow 5, 10, and 20 task splits in the SOTAs (Roy al., 2024; Gao et al., 2024; Kurniawan et al., 2024).
Baselines and Performance Metries:
We compare our proposed method with the existing SO- TAs of prompt-based approach i.e. CIFARION L2P(Wang et al., 2022c), DualPrompt Wang et al., 2022b), CODA-P(Smith al., 2023b), LGCL(Khan et al., 2023), PGP(Qiao et al., 2023), HiDE- Prompt Wang et al., 2024a), EvoPrompi(Kurniuvan et al., 2024), CPrompt et al., 2024), and (roy et al., 2024). We also compare LEAPGen to low-Rank Adaptation (LoRA) ap- proach i.e. C(miA(Smith et al., 2023a), LAE(Gao et al., 2023), InfLoRA((Lann & Li, 2024), slow learner approach i.e. SI.CA(Zhag et al., 2023), and generative approach i.e. GMM((ao et al., 2024). Adapted from Hide-Prompt(Wang et al., 2024a), we measure final average accuracy (FAA) and final forgetting measure (FFM) that represent the final performance after learning all tasks, and cumulative average accuracy (CAA) and cumulative forgetting measure (CFM) that represent cumulative historical performance on each task. See Appendix E.
We implemented LEAPGen and its lite version i.c. LEAPGen-lin -lte in Pytorch utilizing the pre-trained ViT-B/16 backbone following the common practice. L.EAGeen- lite utilizes lightweight Conv Id generators (note: different from ConyPrompt) without deseriptors, it uses class names instead. We utilize Adam optimizer, set 128 batch-size, and cosine learning scheduler similar to ConyPrompt and CODA-P. The initial learning rate set to 0.01, 0.05, and 0.005 for CIFARI 100, ImageNet-R, and, CUB respectively, with 20 maximum epochs. All the methods are run under the same environment i.e. a single NVIDIA A100 GPU with 40 GB RAM. The hyper- parameter settings follow their official settings. See Appendix E.
Implementation Details:
5.2 RESULT AND ANALYSIS
a) Overall Performance: Table 2 shows the summarized numerical results of consolidated algo- rithms in split CIFAR 100 and ImageNet-R dataset, while table 3 shows the summarized result in split CUB dataset. The detailed numerical results are presented in Appendix G. In the split CIFAR dataset, proposed method (LEAPGen) achieves the highest accuracy with 5-16% margin of FAA and a 5-11% margin of CAA. The table shows that a higher number of tasks results in a higher gap in performance. For the forgetting measure, our method consistently achieves the lowest FFM

---

Published as a conference paper at ICLR 2025
Spli
Split ImageNet-R
Method
FAA
CAA
FFM
CPM
FAA
CAA
FFM
CPM
5 Task
classes/task
Task @440 classes/task
L2P
84.77 2 0.48
88.67
= 0.30
6.18 + 0.57
5.99
+
0.29
64.62
+ 0.32
68.01
0.42
3.94
+ 0.16
0,20
DualPrompt
86.41 + 0.21
89.95
+ 0.10
5.37
+ 0.21
4.77
0.46
69.71 0.11
72.78
+ 0.14
3.32
+ 0.16
2.78 +
CODA-P
88.22 2:22:110 1.06
92.25
:: 1.28
7.05
+ 2.18
6.06
2.66
74,89
1 0.36
79.71
+: 1.27
8.89
1: 0.65
7.65
0.98
LGCL
86.90 0.40
90.45
0.18
5.01
+ 0.35
4.36
0.13
69.93
0.21
72.91
+ 0.19
3.04
0.36
2.50
+
HiDe-Prommp
91.99-
93.95
+
2.52
75.40
78.88
2.64
0.16
PGP
87.69 0.06
91.26 1: 0.13
5.32
1 0.18
4.60
1: 0.15
69.71
1: 0.15
72.77
1 0.07
3.36 1.66100 0.23
2.85 0.25
EvoPrompt
89.07
92.32 +:
5.25
+:
0.24
77.27
+ 0.40
81.67
0.18
1.79 0.31
1.1 +
CPrompt
89.22
93.09
= 0.06
5.02
+ 0.17
4.31
- 0.35
78.65
+ 0.00*
82.44
+.00
6.00
5.49
+
Con-Prompe
90.26 00266:00.4 0.44
93.49
+ 0.19
3,64
+ 0.28
3.25
+ 0.16
79,36
+ 0.08
82.93
4 0.24
3.42 0.05
2.36 - 0.16
LEAPGen-lite
97.07 1: 0.08
97.28
1 0.10
0.05
1 0.01
0,02
1 0.01
82.44
0.63
84.37
1 0.90
0,43
1 0.08
0.17 1 0.06
LEAPGen
96.84 + 0.12
96.85
+ 0.26 0.08
+ 0.07
0.06
+ 0.04
82.79
+ 0.32
85.06 + 0.29
0.51 + 0.04
0.18 + 0.07
Task
classsstssk
Task 20 classes/task
L,2P
83.84 0.32
88.67 1 0.16
6.55 1 0.34
5.16
1 0.14
62.50 1 0.51
67.05
1 0.47
5.01
1 0.40
4.41
0.43
DualPrompt
85.36 0.20
89.77 + 0.20
5.41
+ 0.33
4.33
+ 0.15
68.59 + 0.24
72.18
+ 0.20
4.61
+ 0.07
3.70
+ 0.18
CODA-P
91.27 0.14
0.56
+
5.09
73.77
0.50
79.38
148
7,94
0.08
6.72
+
LUCL
83.68 0.43
90.16 + 0.29
5.46
+: 0.22
4.25
+ 0.32
68.65
+: 0.25
72.57
+ 0.19
4.75
+ 0.33
3.38 + 0.58
HiDe-Prompt
0.11
95.01 = 0.08
1.98
+ 0.05
1.56
0.15
75.75 0.40
79.27
0.17
2.29
+ 0.27
PGP
86.36
90.83 0.17
5:49
+ 0.35
4.28
+ 0.27
68.62 0.14
72.19 0.20
4.33
3.63
+ 0.35
EvoPrompt
92.18 1: 0.49
5.39
1: 0.45
3.97
1: 0.73
76.00
1: 0.26
80.97
1: 0.30
4.22
1 0.42
3.59
0.52
CPronpt
86.92 1.04
91.73 +: 0.66
5.43 0.74
4.01
1: 0.81
76.32
0.53
81.50
4: 0.30
6.10
0.75
5.60
1.35
ConvPrompe
88.77 -
92.71
+ 0:04
4.12
0.44
2.67
+ 0.11
77.08
= 0.26
81.47
+ 0.10
4.17
+ 0.04
3.11
0.17
LEAPGen-lite
0.03
98.69
1.04 84.09
0.52
2.13 0.60
LEAPGen
98.38 -: 0.15
98.15 : 0.39 0.10 1 0.03
0.05
1 0.00
1: 0.93
85.54 + 0.65
1.46 1: 1.25
2.11 + 1.21
20 Task @5 classes/lssk
20 Task
@ 10 classes/tusk
L2P
81.89
87.16
1
+ 0.33
57,40
0.31
63.33
+ 0.21
7.88
0.17
DualPrompi
82.32 1: 0.22
87.47 1: 0.24
6.88 1: 0.35
5.63
1: 0.23
65.19
1 0.17
70.31
1 0.29
7.30 1. 0.18
5.16 1 0.34
CODA-P
81.29 0.16
87.72 4:
+
4.98
0.95
0.71
77.08
1.02
+
6.95
0.70
LGCL
0.40
88.63 + 0.18
7.22 + 0.47
4.91
+ 0.56
64.96
+ 0.67
70.18 + 0.37
7.35 + 0.65
5
HiDe-Prompto
97.62 0.14
0.74
- 0.03
81.60 0.48
2.23
PGP
83.41
89.23
7.95
+ 0.23
5.66 0.22
65.24
0.25
0.22
7.17
+ 0.21
0.25
EvoPrompt
84.63
0.21
89.47 + 0.21
9.19
+ 0.41
7.39
+ 0.65
74.93 : 0.64
0.13 79.92
6,72 + 0.90
5.67
0.26
CPrompt
0.00*
90.10
0.17
79.82
0.44
ConyPrompt
87.21
0.20
91.60
1: 0.36
5.47
1 0.33
3.92
1: 0.33
73.93 1: 0.36
78.92
0.37
4.87
1 0.57
3.57
0.25
LEAPGen-lite
95.28
3.37
98.38
+ 1.13
+
0.70
+:
83.67
0.39
85.65
0.33
0.47
0.14
LEAPGen
96.51 + 2.16
98.73
= 0.26
0.66
+ 0.75
0.52
+ 0.39
87,03
0.12
87.81
+ 0.48
2.17
+ 0.17
2.54 0.77
Table 2: Summarized numerical result on split CIFAR 100 and ImageNet-R in 5, 10, and 20 tasks scttings,
and o denote the results obtained from sceded run and the first 10 tasks respectively due to system crash.
and CFM in all settings of CIFARI00. In addition, the magnitude of its FFM and CFM is lower than 0.7%, while the best-of-competioor method i.e. HiDE-Prompt suffers more than 1.5 %, and more than 2% for the rest of the SOTAs. Note that PeDee-rrmpt failed to finish the experiment in the 20 tasks setting, and the result is obtained from its first 10 tasks. However, our method still achieves higher performance the setting by approximately 1% margin. In Imagenet-R R dataset, the proposed method outperforms the existing methods with higher margin i.e. 3-30% for FAA and 2-24% for CAA. Our method also attains the lowest forgetting in both FFM and CFM with 2 1% margin to the competitors. Similar to in CIFAR 100 dataset, the difference of accuracy and forgetting are higher the higher number of tasks. In CUB dataset, our method keeps its superiority to the previous methods. For the final performance i.e. FAA and FFM, our method outperforms the SOTAs with 1-21% and 0.6- 60% respectively, while in terms of cumulative performance i.c. CAA and CFM, LEAPGen gains 3-16% and 1. 1.4-8.5% respectively.
b) Comparison to Different Approaches: Ta-
Split CUB 10 Tasks E020 classes/task
ble 4 compares the performance of our method to SOTAS of different approaches i.e. LoRA, slow learner (SLCA), and generative (GMM)
Method
FAA
CAA
CFM
L2P
66.95 0.13
74.03 0.33
0.11
7.39 0.23
DualPrompt
73.95 0.73
80.29
7.87
8.60 0.37
CODA.P
72.99 0.30
0.64
11.71
1.49
1.04
approaches. Compared to LoRA approach, our method achieves better performance with significant margin i.c. 5-22% and 3-17% for FAA and CAA respectively. These margins are even higher than the margins to the prompt- based approach. SLCA achieves a promising performance its performance is on par to ConvPrompt and Hide- PiieePrrompt in ImageNet-R 10 tasks and CIFARIO0 10 tasks settings respectively. However, our method is still in the higher performance than SLCA with 4-7% margin. Our method also achieves higher performance than the GMM method with 3-10% margin.
LGCL
79.93
83.07
5.35
0.39
Prompt
87.21
87.66
1.90
0.16
PGP
78.35 + 0.68 82.21
0.76
0.20
EvoPrompt
76.23 0,51 81,00
3,57 0.43
CPrempt
77.14 1.16 85.67
11.65
0.31
CanyPrompi LEAPGen-lite
80.12 1.37 84.70 0.64
0.97
+.61
86.00 0.32 =
2.46
L61
2.29
2.15
LEAPGen
88.45 + 0.58 90.90 = 0.81
1.32 # 0.52
1.29 a 0.82
Table 3: Summarized result on split CUB (10-tasks).
8

---

Published as a conference paper at ICLR 2025
Net- Tasks
NeTmageNet-R 10 Tasks
S.ImageNet.R 20 Tasks
S-CTFARTOD TO Tasks
Method
FAA
CAA
FAA
CAA
FAA
CAA
FAA
CAA
C-LORA
15.35
2.38
15.23
65.71
10:60
70.63
0.85
$2.9700.77
87.06
0
LAE
1..44
77.29
0.45
7
76.71
0.110
66.98
0.35
73.72
0.05
88.81
0.34
91.59
InfT.RA
07.52+0.37
82.01
0.12
75.68
0.14
80.82
0.224
71.01
+045
77.28 0.45
89.84 0.03
SLCA
77.00
81.17
91.53 0.228
94.09
GMM
80.72
87.59
LEAPGen-lite
84.37 0.64
0.90
82.38
1.04
0.514
83.67
0.39
85.65
0.33
98,58
0.03
98.69 0.10
LEAPGen
82.79 + 0.32
85.06 0.29
84.09 + 0.93
85.54 + 0.68
87.03 + 0.12
87.81 = 0.48
98.38 + 0.15
98.15 = 0.39
Table 4: Comparison of the proposed method to the LoRA approach, SLCA, and GMM. All the results of the competitor methods is taken from respective refrerences.
CIFAR100 Tasks Accuutas
Imageliet Accurasy
Imagehet Forgstiing
Averaqu Accurasy
Austage Furguilting
Average Forgetiing
vis
L04
925
9-2
512-
73
12P
DueiPrompt
20004~P
LGGL
PrDDmpp!
PGP
EvoPrompp
GPrampe
-CorsPrompt
LLAARGen-IRe - APA
Figure 3: Historical performance plot of the consolidated methods in CIL benchmark datasets.
c) Historical Performance: Figure 3 shows the historical average accuracy i.e. final average aceu- racy of all learned classes after finishing learning on each task t. LEAPGen's gentle slop shows that our proposed method experience the least performance degradation compared to the existing meth- ods. In the CIFARI00 dataset, our method even manages to increase its performance in the later tasks. The existing methods suffer from a significant performance drop i.e up to 13%, 15%, 22% in CIFAR 100, ImageNet-R, and CUB datasets respectively. For the average forgetting measure, our method once again demonstrates an excellent trend. Except on the first 5 tasks of ImageNet-R datasets, method suffers from the lowest average forgetting in all tasks and all datasets. In CI- FARI00 datasets, our method experiences nearly zero forgetting from the first until the last tasks. In Imagenet-R R and CUB datasets, our method manages to decrease its average forgetting in the 5 latest tasks, and consistently achieve the lowest magnitude i.e. 1.4 - 2.2% and 1.2 - 1.8 in CUB dataset. On the contrary, the competitor methods suffer from the increasing average forgetting during incre- mental learning, as shown by the rising curves, The methods suffer from a high average forgetting i.e. for up to 7%, 8%, and 15% in CIFAR 100, ImageNet-R and CUB datasets respectively. Their increase of average forgetting is significant i.e. 4%, 3%, and 8% respectively. Overall, the historical analysis confirms how our method handles catastrophic forgetting better than the existing methods.
d) LEAPGen-lite's Performance: As shown in Table 2-4 and Figure 3 Despite utilizing far smaller (2.67% params) generators and without class descriptors generated by L.L,, LEAPGen-lite still outperforms the existing methods significantly i.e. 4.7-25% % FAA and 3-11% % CAA in CIFAR100 100, and 3-30% FAA and 2-26% CAA in ImageNet-R dataset. LEAPGen-lite also achieves a low for- getting rate in these 2 datasets for all task-settings. In the CUB dataset, LEAPGen-lite archives comparable performance to HiDe-Prompy Prompt and outperforms other SOTA with a significant margin i.e. 6-20% FAA and 3.3- 14% CAA. This evidence proves our ideas i.e. language embedding input for prompt generation, task-wise generators, soft task-id predictor, and learning with auxiliary data not bounded by the generated descriptors and the size of generators.
e) Ablation Study: Table 5 shows the impacts of the proposed method's com- ponents on its performance.
Component FT F
Loss
FAA
CAA
FFM
CFM
Cintra
61.5 65.3
5,9
5.6
(1) Descriptor Embedding E as one of LEAPGen's main components, elevates fine tuning (FT) baseline sigmificantly ie. > 11% FAA and > 13% CAA. It shows the promising impact of our idea (2) Task Key K" and Loss L, trans- form the method into task-wise prompt- ing based on input to K and L, similarity. They produce a better model with 1-2% FAA and CAA improvement and reduce FFM and CFM by > 2%. It shows the effectiveness of task-wise decom- position as applied in the task-wise prompt approach. (3) Task-wise generator G*() enhances the model recognition capability by 2% and 1% for FAA and CAA respectively. It also decreases its forgetting rate by %. It means the trainable generator produces more discriminative prompt than
Liatra
72.9 78.5
5.8
5.3
Liatra Ce
74.9 79.5
2.8
FT+G" (F)=K'
Ciatru Ci
76.6 80.5
1.8
24
FT+0" (E)+K*+ +22
Liatra Lx. Co
77.2 81.0
0.5
1.3
FT+G (E)+K' KL +Aux
Ciatras Li, Co
78.4 82.0
0.5
1.2
FT+G (E)+K*+ +1 +Aux
Lintra Lincer. L.. Le
83.7
86.0
2.5
FT+G* EE>K'' +L
Ciatra Cintor, Lc. Co 84.7 86.1 0.9
Soft Task- ID Predictor
Table 5: Ablation study on the impact LEAPGen compo- nents to its performance, run on 10 tasks Imagenet-R.

---

Published as a conference paper at ICLR 2025
the descriptor embedding only. (4) L Class-wise Key L' and Loss Le for top-k embedding se- lection i.c. Eg for i E [1...k] improves the model performance with 1% margin. It also reduces the model forgetting rate with up to 1% margin. Thus, measuring the input similarity to L offers a better embedding selection than to (irectly.(S) Auxiliary embedding improves the LEAPGen's performance with a more than 1% margin. It shows the secondary contribution of descriptors em- bedding as an auxiliary modality along with its primary role in prompt generation. (6) Inter-tasks loss Linter improves LEAPGen accuracy significantly i.e. 5.4% and 4.0% for FAA and CAA re- spectively even though it increases FFM and CFM by 0.5 2%. It contributes to balancing the knowledge of previously learned classes (stability) and currently learned classes (plasticity), and emphasizes the risk of forgetting previously learned classes. (7) Soft task-id predictor substitution to conventional task-id (Wang et al., 2022b) improves the accuracy with up to 1% and reduces FFM and CFM by 1.5% and 0.5% respectively. It implies that our designed task-id predictor outperforms the existing task-id predictor both in accuracy and forgetting.
Sensitivity Analysis: : Figure 4 shows the sensitivity analysis of our proposed method in various settings i.e. different prompt lengths (left), layer depths (center), and number of generators per trih(), LEAPGen main- tains its stable FAA in any prompt length val- ues i.c. within the range of 84.2 84.7%, while the existing methods i.c. ConyPrompt, CODA- DualPrompt, and L2P achieve significantly lower FAA i.e. < 78%. LEAPGen with the shortest prompt (4) has higher FAA than the existing methods with the longest prompt (40), de- spite the 10 times smaller size. LEAPGen holds its stable performance at various depths of prompted VIT layers. LEAPGen consistently manages to achieve 83.5 7% FAA, while other methods achieve far lower FAA i.c < 78%. The best layer depth for LEAPGen and ConvPrompt is 7, while the best depth for the others is The number of generators turns out to be the most sensitive parameter LEAPGen, generators in LEAPGen is the same number as the selected descriptor embedding, it is quite logical that LEAPGen becomes counterproductive after some threshold k.
20
Figure 4: FAA of the consolidated methods under dif- ferent prompt lengths (left). layer depth (middle), and maximum generators per task (right)
it is saturated after 4 generators per tasks. Considering the number of
g) Parameters, Running Time, and Storage: Table compares the number of executed pa- rameters, running time (ImageNet-R) and stor- age of our methods and existing SOTAs. De- spite having higher number of parameters and storage, LEAPGen consumes less running time than existing SOTAs both training-inference and total running time. LEAPGen-lite con- sumes the least costs in total running time and storage and requires relatively low parameters and inference time, LEAPGen and Conv Prompt re- quire additional time to generate descriptors that increase their total simulation time. Despite having the least parameters, Hide- Prommpt requires the longest training and total times since it needs extra operations to generate uninstructed class representations. Please Appendix D) for our detailed extended analysis on task-key similarity, language embedding similarity, performance-and-ee trade-off, class name as descriptor, and performance on various descriptors and types of generators.
Method
Desc- criptors
#Params(M)
Rumning Time (h)
Storage
T.Desc
Trelnt Total
(MB)
HiDe-Prompt
0.15
0.019
3,40
5,40
334
Com-Prompi
1.28
1.07 0.033
1.04
2.11
346
LEAPGen-
0.16
332
I.EAPGen
6.35
1.07
0.025
0.72
1.79
567
Table 6: Cost comparison of the methods, T.Desc. Tr, and Inf denote time for generating descriptors, training, and inference respectively, detailed in Appendix D.3.
6 CONCLUDING REMARKS
We identify the prompt-based approach dilemma and propose a novel prompt-based structure and algorithm that incorporates 4 key concepts (1) language as input for prompt generation (2) task-wise generators (3) limiting matching descriptors search space via soft task-id prediction (4) generated prompt as auxiliary data. Our experimental analysis shows the superiority of our method to existing SOTAS in CIFAR100, ImageNet-R, and CUB dataset with a significant margin i.e. up to 30% final average accuracy, 24% cumulative average accuracy, 8% final forgetting measure, and 7% cumulative forgetting measure. Our historical analysis confirms our method successfully maintains stability pialtitt-p in every task. Our robustness analysis shows our method consistently achieves higher performance in various prompt lengths, layer depths, and number of generators per task compared to the SOTAs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
M. Anwar Ma' sum acknowledges the support of oTTooppedi--UI Centre of Excellence for GPU access run the experiments. Savitha Ramasamy acknowledges the programme DesCartes supported by the National Research Foundation, Prime Minister's Office. Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
REFERENCES
Rahaf Aljundi, F. Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) forget. In ECCV, 2018.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 583-592, 2019.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.0982. 2020.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 45(5):5497-555 2022.
Xusheng Cao, Haori Lu, Linlan Huang. Xialei Liu, and Ming-Ming Cheng. Generative multi-modal models are good class incremental learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 28706-28717, 2024.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-33) 3385, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual leamingg framework with general parameter-efficient tuning. In Proceedings of the JEFF/CVF International Conference on Computer Vision, pp. 11483-11493, 2023.
Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal- continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28463-28473, 2024.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributis generalization. ICCV, 2021.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 831-839, 2019.
Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-based contin- ual learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
11463-1 1473, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan. John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):55113322 2017.
11

---

Published as a conference paper at ICLR 2025
Muhammad Rifki Kurniawan, Xiang Song, Zhiheng Ma, Yuhang He, Yihong Gong, Yang Qi. and Xing Wei. Evolving parameterized prompt memory for continual learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 13301-13309, 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for marameter-efficien prompt tuning. arXiv preprint arXiv:2 112220444009) 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv 2021.
preprint
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638 23647, 2024.
Hao Liu, Yong Zhou, Bing Liu, Jiaqi Zhao, Rui Yao, and Zhiwen Shao. Incremental learning with neural networks for computer vision: a survey. Artificial intelligence review, 56(5):4557 2023.
Bolin Ni, Hongbo Zhao, Chenghao Zhang, Ke Hu, Gaofeng Meng, Zhaoxiang Zhang, and Shiming supervision. In Proceedings of
Xiang. Enhancing visual continual learning with the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10068-24077, 2024.
Keon-Hee Park, Kyungwoo Song, and Gyeong-Moon Park. Pre-trained vision and language trans- formers are few-shot incremental learners. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23881 2024.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540. Springer, 2020.
Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, al. Prompt gra- dient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Anurag Roy, Riddhiman Moulick, Vinay Verma, Saptarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 23616-23626, 2024.
Khadija Shaheen, Muhammad Abdullah Hanif, Osman Hasan, and Muhammad Shafique. Continual learning for real-world autonomous systems: Algorithms, challenges and frameworks. Journal of Intelligent & Robotic Systems. 105(1);9, 2022.
Yu Shen, Xijun Wang, Peng Gao, and Ming Lin. Auxiliary modality learning with generalized curriculum distillation. In International Conference on Machine Learning, pp. 31057-31076. PMLR. 2023.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to- text-to-image diffusion with c-lora. arXiv preprint arXiv:2304. 06027,6077 2023a.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention based prompting for rehearsal-free continual learning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 11909-11919 2023b.
12

---

Published as a conference paper at ICLR 2025
Zonggui Tian, Du Zhang, and Hong-Ning Dai. Continual learning graphs: A survey. arXiv preprint vrXiv:2402.063300 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Liyuan Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu, Hierarchical de- composition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2024a.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024b.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong, S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022a.
Zifeng Wang. Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompr: Complementary prompting for rehearsal-free continual learning. In European Conference Computer Vision, pp. 631-648 Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022c.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Gengwei Zhang, Liyuan Wang, Guoliang Kang. Ling Chen, and Yunchao Wei. Slea: Slow learner pre-trained model. In Proceedings of the
with classifier alignment for continual learning JEEF/CVF International Conference on Computer Vision, pp. 19148-19158, 2023.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1131-1140, 2020.
Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, and Yao Zhao. Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22267-22267, 2023.
13

---

Published as a conference paper at ICLR 2025
A THEORETICAL ANALYSIS
We state the following theorem to prove the effectiveness of our proposed method in a theoretical analysis.
Theorem 1: Let fo and he are the VIT encoder and classifier head respectively, vb is frozen, 6 = {G,, K" Le is the set of trainable parameters and generators, 6' {K*, L:)t-T Lefi=l,eec is the set of trainable parameters only, and C is loss function. Assume that G. f. h, and are continues functions in R. C can produce identity output, then there is Z 0 that satisfies ((ho(fv.r(z)))- Ch (f (x)( = c..
Theorem 2: Assuming pre-trained ViT the backbone, if the loss error C S 5 then exist Lintra S and Linter S 6.
Theorem 3: Let F and F" denote the task-pedictor functions parameterized by ((K*). (L and { respectively, and cos(,) denote cosine similarity function. If K' and L' are trained well enough so that cos(I, K*) Re cos(æ, L.) 2 S, and cos(æ, vr, K*) R cos(æ, 122) R cos(r, L.) s (PIT,FF))
2/38 for any x E Xx then it guarantees P(x X" 0p) 2 P(x
Theorem 1 theoretically shows there is always a chance to achieve a smaller loss with generators than with descriptor embedding only. Theorem shows that minimizing the loss error implies minimizing both inter-task and intra-task losses. Theorem shows that if the learnable parameters are well-trained then it guarantees our task predictor as presented in equation 3 is more accurate than the existing task-id predictor (based on {K* only). Please see in Appendix A for the detailed analysis.
A.1 PROOF OF THEOREM 1
Recall that fo is pre-trained ViT encoder with frozen parameter th, h& is classifier head parameter- ized by ø (learnable), 0 = {G K', T. is the set of learnable parameters and generators, = L L.)t=1,cec* is the set of learnable parameters only. Note that G produces a prompt L:
component PC; (F.)) 0' = {K*, I.J is the same using identity function I as generator. C is the loss function that in this case is cross-entropy. First we adopt the definition of continuity of (x) at a i.e for 1/2 > 0 there is some d1 such that:
Ix al < 81 = 1f(x) - [(a)] < <m
(A11)
Now, following the assumption that f(-) is continuous function in IR, now we substitute f(-) with f=,o(-), subtitute with ((4))) and with T(F.). Not that G is trainable generator that can produce identity function I as well as nearby vectors i.c. (E.).. Also note that fe.0.(G=1)() = (.) Therefore, the equation All can be derived into:
(G,) - I(E.) 8, = (fx,(x)) - (x)) < <
(A12)
Then, incorporating the continuity of h(.)) in IR and the definition of conituinous function then we have
(z) - feror (z) < m1 + [ho(fo,)())) - (2)) m2
(A13)
Analogically, applying the continuity of L(.) in R, then we have
(o)(f))(z)) he(ff.er (x)) < 72 = ((ho(fvv(z))) - ((ho(fu. < 73
(A14)
Combining the equations A12, A13, A14, then have
(G.(E)) T(Ea) 51 = [Chh
L(ho (Sr (2))) 13.
(A15)
In the equation above we can pick constant 0 < 1/ na and 8 < 81 tat satisfies
(F.) T(F..) 5ं = [[(hs(Jv. C(ho (x)) 1/3
(A16)
14

---

Published as a conference paper at ICLR 2025
The term -Chh (f (f - C(h (fl. = n7s if ((h > ((ho(fo.(z)))) In this condition, then state the following statement:
-(ho
ns can be satisfied by two conditions: (1)
[(h - [(ho(lv.0 >0
(A17)
The other condition (2) C(h (fv.(())) - [(ho(fe.0 (x)))
nJ if ((hg(fv.0z)))) (S (x))). However, we didn't use this condition as the optimized G instead, we select G = 7 so that ((lh(((wo(())) best possible value of G is the identity function itself. In the second condition, then we can state
L(ho(So.or (x))). The second condition shows the where the
(( - ((ho(hv.olt))))
(A18)
Combining both inequations A17 and A18, then we prove Theorem that states Assume that G, f. h. and L are continues functions in IR, if G can produce identity output, then there e 2 0 that satisfies C(haffu er (z))) (( ( (x))) = e. Theorem I shows that a possibility to achieve better or at least similar learning performance with learnable generator networks than without them.
A.2 PROOF OF THROREM 2
Given model with loss error L < 6, then
T=
explholfu. (x))(c) 2
Σ log
(A19)
EE Eneco exp(ho(
1=1 CEC*
t=T
log explho((e
Linter (x,, (16,0,0)
E Edecr exp(hs((o, S8
(A20)
1=1 CEC
Or simply
(inter (4,00,0 8
(A21)
From equation 4 We get intra-task probability as T. ; P(x E X () 0) E X*
P(x E X x E X",
(A22)
Since P(æ E 1 E x (,O), P(x E 0), and P(x E X" 0) are in the range of (0, then have:
1)
P(x E le E X T,,) 2 P(æ E X T,0)
(A23)
Applying log both sides, then have
log(P(z E 2 X", T, ()) 2 log E xT
(A24)
Multiplying both sides with 1, then we have
-Po E X X², T,0)) < -log(P<x E xx)
(A25)
Let inter-task probability and intra task probability represented as:
:=T ΣΣ
exp(ho(fv.0())()
P(x E 1T
1l Etsl Edeco exp(h
(A26)
P(x e X/ E X T,0)= = Σ (xp(l)(fee(())dd)
Edecr exp(hs(fe.o
cEC
Subtituting equations A26 into A25 and combining equation A21, then we have:
-log( Σ csp(ho(foo(z)(()) < -log( eEC! Eveco cxp(ha(fuo(()]eee
eap(ho(fu.o(z)))))
D Evecr cxp(ho((r
REC
(A27)
15

---

Published as a conference paper at ICLR 2025
Σ log ())())(f RECE
log cxp(ho(fe.(())lc)
88
Ede cco(ho((osoe)lee))
1=1 nEce 21
Eeeco exp exp(hx(fo.oz)))l
(A28)
Thefore, we have
Lintra (2, -- Σ log exp(he(fo.o())cc)
Execr (() 5
(A29)
oEC
or in the simplified form
Lintra (12, 0)) 8
(A30)
The explanation above proves Theorem 2 that assuming pre-trained ViT as the backbone, if the loss enror L < then exist Lintra S 5 and Linter S 8.
A.3 PROOF OF THEOREM 3
Let F and F' denote the task preedirdtttoor functions parameterized by ((K") (TA)) and { (K') respec- tively therefore, F' is presented in equation and simplified into equation A31 while F* is presented in equation A32.
F(x) =t
arg max {cos(xr coss(, L
(A31)
tE(I..T)nECC
F'(x) =t= arg mmxx((/FK//mmmmaall/l/l111a 1e[1..1.
arg (co((( K*)) sE[L..].
(A32)
Let cos(.) denote cosine similarity function, L denotes similarity loss i.e. difference between the right and wrong representation. Therefore, for any x E X² then we have
((F())) cos(æ, K" (eos(2, L.) - 1) CEP
1
cos(I, K" )cos(I, L()) Σ Σ
(A33)
Σ
cos(æ, Kt )eos(s, (he)
(IC*) 1)
c'SCEE
((F = cos(e, K2)
Σ C'Lt
cos(x, K²)
(A34)
T
Thus minimizing ((F())) and (F)(x) imply maximizing and F' respectively. Suppose that Kt and Li are trained well enough that for any x E X it satisfy cos(æz Kt) ~ S, and cos(æ, K²) Sa = S e1, cos(æ, L.) R S2 and cos(x, L'yze) cos(æ, L() Sn S2 62. where S1, S2, Sa, Sp, é1, and 62 have positive values. Then equations A33 can be derived into
[(F(x)) = SIS2
1
Σ Σ
SaSo (1C*1-1)
Σ c'FEEC
SIS,
(A35)
(T- - 1)((*
((F(x))= = S - S.S, S,Sb
(A36)
and A34 be derived into
((F = T Σ Sa
(A37)
((F((z)) =S SS.
(A38)
Now, assuming that S R S2 s S and 62 then we have
((F(z)) = S2 - (S e) - -) - S(S E)
(A39)
((F(x)) =22 -S2 2S +2)) (52 Se)
(A40)
((F(x)) 52 - 52 + 2Sc 2 52 Se
(A41)
16

---

Published as a conference paper at ICLR 2025
[(F(z)) 3Sc - -52 -
(A42)
and we have,
=(
c)
(A43)
Suppose that want to find condition where ((F(x)) (F)( then have
3Se (5 22) S
(A44)
3Sc S e² S
(A45)
Note that S. Then we can choose 1/35, the equation above becomes
35(1/3)S -5² -
(A46)
S² -
(A47)
-5
(A48)
that always true for any O. Note that (1/3)S means S===== (2/3)S.
Choosing < (1/3)$ also satisfies the incquation above. The derivation above proves Theorem 3ic. let F and F" denote the task-pedictor functions parameterized by ([K*), {L+) and {K*}
respectively, and cos(,) denote cosine similarity function. K and L are trained well enough so that cos(x, K*) ss cos(æ, L2) 2 S, and cos(æ, K*) ss cos(s, L2) as cos(r, L.) < 2/35 for any x Xe then guarantees ((F(2)) I (xF(())) that implies P(x (T) P(x 1T ).
B PSEUDO-CODE OF LEAPGEN ALGORITHM
In this section we present the procedures of LEAPGen training and inference that are shown in algorithm and respectively.
Algorithm LEAPGen Training
1: Input: A sequence of tasks T', T²...,T", frozen L-layered (with H MSA heads/layer) pre- trained ViT fw(.) training epochs E, and batch size B.
2: for t= 1 do
3:
Initiate trainable parameters for task t, 0 ((G)) K", (,,
4: B Split T into sized batches
5:
do 6: for b do
for
7:
Find top-k L/ where j [1.t], [1.|[c|]]
8:
Generate prompt P {PPk PV hi as in eq. 2
9:
Compute logits by forwarding the input i.e. he(fe,(())
Compute Lintra in eq.
11:
Compute Linter as in eq.
Compute C, in eq.
13:
Compute L. as in eq. 9
Compute Crotat as in eq. 10
15:
Update parameters 0f end for
((G)),, Kt, (E based on Listal
16:
17:
end for
18:
end for
19: Output: Optimum parameters = (theta')s T ((G)) K", K" (1
C'IT
C COMPLEXITY ANALYSIS
Following the pseudo-code in Algorithm LEAPGen has several atomic operations e.g. selecting top-k LE (line 7), prompt generation (line 8), computing logits (line 9), computing loss (line 10- 14) and update parameter (line 15). Let N |TI' and accumulating the operations all batches,
17

---

Published as a conference paper at ICLR 2025
Algorithm 2 LEAPGen Inference
Input: An input x, frozen L-layered (with MSA heads/layer) pre-trained VIT fø() and {theta") {(G)) K (22) h
optimized parameters
2: Predict task-id & using soft task-id prediction as in eq.
3: Find top-k L2 where j E [1. [I.
4: Generate prompt P = {(P)k. 1.k:
in eq.
5: Compute logits by forwarding the imput i.c. logits hs(fv.o(2))
6: Compute Pedicted label ) = argmax (logits)
7: Output: Predicted label y
then selecting top-k L costs ((N.TT.|C")) the prompt generation costs O(N"K)) k)), computing losses costs O(N'), and parameters update cost costs O(N"), then the the complexity of LEAPGen will be:
+ +
(A49)
(A50)
OLLEAPGen) E. O(T.N' AA +
(A51)
Note that T.N* N that represents the number of all samples from the first task until the last task, and T.|C*| = |C| that represents the number of all learned classes where |C| << N, and & small constant; 10. Then LEAPGen complexity becomes
OLLEAPGen) = E.(O(NIC)) + O(NA) O(N))
(A52)
O(LEAPGen) = E.(O(NICI)
(A53)
O(LEAPGen) = O(EN(CI)
(A54)
Assuming that learning epoch E and number of classes |C| are constant, then the equation above is derived into:
O(LEAPGen) ((EN(C)) O(N)
(A55)
D EXTENDED ANALYSIS
D. HIGH SIMILARITY OF TASK KEY VECTORS
We investigate the further detail of conventional task-id prediction computed by cosine similarity between input æ and task-key learning on the ImageNet-R dataset, we evaluate the similarity between task keys. Table A7 shows
implemented by DualPrompt. After completing the 10 task
the cosine similarity between K" and Kt for all t, E [1.T]] The tables show that pair of different task keys has high similarity. For example, except for task-3, task-1 has 0.8 0.96 cosine similarity to other task keys. During the inference step, this high similarity between different task keys causes a false task prediction that leads to the wrong selected prompt, unexpected feature value, and finally, misclassification.
This empirical analysis is one of our motivations to develop a better task-id prediction i.e. by utiliz- ing both task-wise key Kt and class-wise key L:- Combining the similarity score of input to both K and key L produces a more discriminative task prediction than using K' only. Another way to solve this issue is creating uninstructed representation where class may have several proto- types as used Prom-Pr However, this mechanism takes a longer time since the uninstructed representation optimization needs prototype augmentation and many iteration updates.
18

---

Published as a conference paper at ICLR 2025
Task-1
Taske2
Task-3
Task-4
Taske5
Task-6
Task-7
Task-8
Task-9
Task-T0
Task-1
1,000
0.961
0.728
0.857
0.932
0.901
0.866
0.827
0.800
0.877
T
0.961
0.866
0.932
0.879
0.844
0.899
3
0.728
0.749
1,000
0.873
0.738
0.785
0.725
0.672
0.660
0.687
Task.4
0.857
0.866
0.873
1,000
0.872
0.828
0.771
0.755
0.806
T
0.932
0.932
0.738
0.872
1,000
0,921
0.893
0.850
0.843
0.908
Task-6
0.901
0,909
0.785
0.893
0
1,000
0.933
0.893
0.873
0.893
Task-7
0.866
0.879
0.725
0.828
0.893
0.933
1,000
0.937
0.920
0.907
Task-8 0.827
0.844
0.672
0.771
0.850
0.893
0.937
1,000
0.905
0.857
Task-9
0.809
0.755
0,920
0:905
Task-10
0,877
0.899
0.687
0,806
0.908
0.893
0.907
0.857
0.872
1,000
Table A7: Cosine similarity of task-key vectors after completing 10-task training on ImageNet-R by DualPrompt method.
D.2 HIGH SIMILARITY OF LANGUAGE EMBEDDING
We continue our deeper investigation the encoded language embedding as implemented in previ- ous SOTAs i.c. LGCL. LGCL generates the language embedding by forming the string "The photo of class name". then patches it into the CLIP encoder. Surprisingly, prototypes from some classes have high similarity to the other classes' prototypes. For example, in class 'Great White Shark' has 0.9 similarities to the prototypes of class 'Tree Frog' and 'Iguana', despite those classes being to- tally different and not the sub-classes of the same superclass. On the contrary, the class 'Great White Shark' has lower similarity to the class 'Goldfish' which is despite both of them are the species from the same super-category i.e. fish.
In that case, can't utilize the language embedding as a reference anchor in the training process as it may pull the adjusted (ViT) prototype closer to its value, while its value is highly similar to other classes' prototypes. Thus, in dealing with this challenge, we choose to use the language embedding as input for prompt generation. In such way, the trainable generators can produce more discriminative prompt than the undiscriminating language embedding.
'Gold-
fish'
Shark"
While
"Ham- mer- head'
"Sting" ruy"
"Hen"
trich"
'Gold- finch"
"Junco"
Eagle'
'Vuli- ture'
'Newt"
lotl"
Prog'
'Tree
'Igu- ana'
Chame- leun'
Afri-
bral
'Scor- plon"
tula"
'Centi- pede'
Pea- cock'
Geldlish
0.732
0.724
0.759
0.595
0,565
0.736
Great
White
0.111
1.000
0.916
0.608
(1760
0.719
0.848
0.887
0.744
01538
0.719
0.901
(1.904
(0.8%)
0.654
0.663
0.751
0.824
0.577
0.882
Shark'
Hammers
beud'
0.732
0.966
1,000
0.631
0.634
0.714
0.811
0.898
0.687
0.445
0.573
0.830
0.877
0.846
0.584
0.008
0.648
0,806
0,575
0.815
'Simgray"
0.724
00.088
0.631
1,000
(0016
0.652
0.093
0.080
0.425
0.669
(1,055
0.051
0.644
0.835
0.606
0.874
0.617
1.000
Hen'
0.659
0.160
0,651
0.816
1.000
00.44
0.79%
0.800
0.358
0.760
0.927
0.791
00,005
0.877
0.613
0.682
0335
0.713
0.389
0,336
Ostricch
0.766
0.719
0.714
0.676
(0.093
1.000
0.026
0.736
0.742
0.425
0.614
0.752
(1,055
0.734
0.646
0.009
0.671
0,742
0,621
1.000
Goldfinch"
0.746
0,811
0.652
0.798
1.000
0.813
0.778
0.685
0.916
0.917
0,677
0.767
0,718
0.564
0.199
Junce'
0.250
0.367
01877
0.695
0.807
0.736
0.133
1.000
0.774
0.542
0.719
0.929
(1.875
0.115
0.390
0.716
0.726
0.859
0,562
"Bald-
Eagle'
0.097
0.744
0.657
0.680
0,338
0.778
0.669
0.768
0.794
0.821
0.805
0.742
0.665
0.856
Vulture'
0.424
0.338
0,445
0.425
0.780
0.425
0.685
0.542
0.009
1,000
0.815
0.462
(1.886
0.090
0.300
0.389
0.725
0,502
0.371
0.718
'Newt'
0.389
0.719
0.573
0.379
0.927
0.722
0.719
0.00
0.815
1,000
0.22
0.762
0.830
0,009
0.62
0335
0.835
0.337
Axoloti'
0.734
00,50
0.609
0.294
0.252
0.778
0.929
0.705
0.462
1000
(1.822
0.879
0.605
0713
0.366
0.597
11304
Frog"
Trre
0.714
0.904
0.871
0.655
0.505
0.665
0.916
0.873
0.794
0.686
0,762
0.832
1,000
0.922
0.584
0.623
0.786
0.753
0.554
0.936
"Igunns"
0.759
0.806
0.846
0.651
0.875
0.734
0.917
0.93
0.831
0.690
0.830
0.879
0.922
1.000
0.657
0.677
0.804
0.815
0.572
0.04
Africun
Chameleon
0.724
0.654
0.584
0.644
0.613
0.646
0.677
0.500
0.639
0.360
0.600
0.665
(0.584
0.657
1.000
0.620
0.594
0.585
0.648
0.645
'Cobra'
0.682
0.668
0.608
0.635
0.633
(.699
0.508
0.716
0.690
00.389
0.623
0.745
(0,633
0.671
0.629
0661
0.709
0.674
0.992
'corpiinn'
0.395
0.751
0.818
0.805
0.838
00671
0.767
0.726
0.503
0.723
0.835
0.713
0.766
0.504
0.594
0.651
10000
0,854
0.665
0.336
'arantula'
0.734
0.674
0.713
0..00
366
00.553
0.585
0666
0.581
Centipede
0.565
0.577
0.375
0.517
0.589
0.621
0.364
0.562
0.663
0.371
0.557
0.397
0..54
0.372
0.648
0.874
0.665
0.5k]
1.000
0.619
"Parnnck"
0.736
0.833
0.815
0.680
0.386
0.696
0.919
193
0.856
0.718
0.844
0 0,364
(1.926 0.944
0.645
0.697
0836
0.794
0.619
1,000
Table A8: Encoded Language Embedding similarity of the first 20 classes of ImageNet-R, encoded by C:IP encoder implemented in LGCL.
D.3 TRADE-OFFS BETWEEN COST AND PERFORMANCE
We extend our analysis on the trade-offs between Cost and Performance. Table A9 shows the cost performance on LEAPGen in comparison to ConvPrompt as the SOTA of the growing component approach and HiDe-Prompt the SOTA of the task-wise prompt approach. We also compare our
19

---

Published as a conference paper at ICLR 2025
proposed method to the bigger version of ConvPrompt and HiDe-Prompp i.e. ConwPrompt-Lange and HiDe-Prompt large that utilize a bigger number of executed parameters than our method. Please note that the number of executed parameters in the table means the total executed parameters in forward operation, excluding ViT backbone and classifiers head.
(a) Performance vs Parameter and Rumning Time: Our method outperforms the default version of Conv ConyPrompt and DiDe-Prompt with up to 9% average accuracy margin and up to 4% average for- getting measure with the drawback of the higher number of executed parameters. However, with the higher number of executed parameters, our method requires a significantly lower running time than both of the method i.e. 1.5 2.1 Ih and 5.4h for ConvPrompt and Hide-Promt respectively. Even though utilizing far smaller parameters, Hide-Promp requires extremely high running time since executes too many operations i.e. augmenting the prototypes and then forming an uninstructed representation that requires high epochs to converge. Thus, it spends far higher running time in total.
The large version of Cony and HiDe-Prompt executes higher number of parameters than LEAPGen. In line to the higher parameters, they require far higher running time than their original version. Unfortunately, with the extremely high number of parameters, they didn't 't achieve higher performance, rather lower than the performance of their standard version. Compared to the large version of ConvPrompt and HiDe-Prompt, LEAPGen has better performance and lower cost. This evidence confirm our sensitivity analysis that a model has the best setting that achieves optimal per- formance. The higher number of parameters or training epochs doesn't imply a higher performance. Thus we evaluate the competitor methods following their advised setting described on the respective papers or official code.
LEAPGen-lite outperforms ConvPrompi and Hide-Prompt with a significant margin i.e. 6-8% even though with utilizing& smallest number of parameters on average and without class descriptors as in ConyPrompt and LEAPGen. However, it achieves smaller final (task- 100 average accuracy than LEAPGen with more than 1.5% margin.
Table A10 shows the simulation time for LEAPGen, ConyPrompt as SOTA in benchmark datasets i.e. CIFAR100, ImageNet-R, and CUB. The table shows a similar trend to the analysis on the ImageNet-R dataset. Descriptor generation consumes fair running time iee 1.53-1.09 since it is conducted via online query through the internet connection. In such a manner, we don't 't need extra storage to save the LLM model. Please note that descriptor generation is conducted only once before the training phase. Thus the LLm is not utilized during the training and testing phases of our method. Despite requiring additional time to generate classes' descriptors, our method still has significantly lower total grinning time than HiDDe-Prompt i.e. 3 times, 3 times, and 2.8 times smaller in CIFAR 100, IeageNet-R and, and CUB datasets respectively. Compared to our method, PonvPrompt requires a higher training time in those 3 datasets with significant gap i.c. up 7.7 hours. Please note that ConvPrompt utilizes evolving generators and the number of its generators grows with the increasing of tasks. Thus in the later tasks, its forward operation takes more time than in the earlier tasks. Having no additional time for descriptor generation LEAPGen- lite spends the smallest total running time and training+inference time.
(b). Total Running Time:
(c). Inference time only: Table A10 shows that our methods i.e. LEAPGen and LFAPCen--lite hae moderate inference time i.e. lower than ConvPrompy and higher than HiDe-Prompt This empir- ical evidence is in line with our previous analysis where growing components/generators approach executes more generators in its forward function with the increasing number of tasks. Thus, on aver- age, takes more inference time than our task-wise generator approach. Hlde-Prompt requires less inference time since it utilizes task-wise prompting where the prompts are not generated, but rather directly prepended into the ViT MSA heads. Thus it takes less operation to execute which implies its lower inference time. In spite of its lower inference time, HiDe-Prompt requires far higher training time which makes it a higher total running time than our method. In summary, this evidence proves the running efficiency of our proposed method both in terms of inference time or the total running time in a simulation, in spite that our method executes a higher number of parameters.
(d). Performance vs Storage: Table All shows the performance vs storage of the consolidated methods in the ImageNet-R R dataset with 10 task settings. Model saving is conducted by executing dicct().iileaamme)
the method
Please note that this storage doesn't 't include the LLM model, since it is used via online (remote) query, not saved in our storage. The table shows that LEAPGen requires a bigger storage size than ConyPrompt, as L.APGen has more at-
20

---

Published as a conference paper at ICLR 2025
tributes and parameters. Please note that the model saving is represented in dictionary object, thus, more attributes require more dictionary keys and items that imply more storage consumption. Different from ConvPrompt and HiDe-Prompp LEAPGen has more complex generator structure and task-id predictors that utilize task-wise key and class-wise keys. In addition, LEAPGen also saves the descriptors embedding for the testing phase, different from ConvPrompt which doesn't t save the embeddings as it uses the descriptors embedding to calculate the number of generators dur- ing the training phase only. Therefore it requires a bigger storage save its model. The storage consumption can be minimized by using lightweight generator. In comparison to large versions of ConvPromp and HiDe-Prommp that have higher number of parameters, our method achieves lower storage consumption while achieving a higher performance i.e. FAA and CAA.
Performance Alter Finishing Task
Running
Method
Metrics
AVG
Time (h)
HiDe-Prempt
Avg. Accuracy
85.22
82,93
82.35
79.74
78.85
77.81
77.37
76.38
76.31
75.75
79.27
LiDe-Prompt-Large
Avg.Accuracy
85.76
83.12
82.03
79.49
78.55
76.55
76.00
74.82
74.98
74.30
78.56
Cony Prompt
Avg Accurccyy
86.24
80.79
78.80
77.74
81.47
Conv Prompi Large
Avg.Accuracy
90.12
86.22
83.96
81 45
80.52
79.73
76.40
74.19
74.56
80.79
I.EePGenIite
Avg. Accuracyy
89.73
88.66
86.04
83.03
82.75
LEAPGen
Avg. Accuracy
89.83
87.63
86.46
85.97
84.71
84,79
84.01
83.77
84.12
84,09
85.54
HiDe. Prompt
Avg. Forgetiing
3.29
1.50
2.41
2.19
2.53
2.20
2.25
2.30
2.29
2.33
2.33
HiDe-Promph Prompt-Large
Avg. Forgetting
2.15
2.91
2.66
Conv Prompt
Avg. Forgetting
2.57
1.57
2.60
2.93
3.15
3,53
3.44
4.07
4.17
3.11
Conv Prompe-Laage
Avg. Forgetting
4.65
4.69
8.91
10.55
10.50
6.61
LEAPGen-lite
Avg. Forgetting
1.11
0.75
0.58
0.56
4.13
3.37
2.92
2.70
3.01
2.13
2.13
LEAPGen
Avg. Forrettet
3.97
1.96
1.38
2.63
2.24
1.99
1.77
1.60
1.46
2,11
HiDe-Prompt
#arameters (M)
0.15
0.15
0.15
0.15
0.15
0.13
0.15
PrDD-Prempt-Large
#Parameters (M)
12.29
12.29
12.29
12.29
12.29
12.29
12.29
12.29
12.29
12.29
12.29
15.15
Conv Prompe
#Parameters (M)
0.55
0.60
0.70
0.80
0.85
0.99
1.09
1.19
1.28
0.89
2.11
Conv Laomet-Large
#Parameters (M)
5.18
5.82
6.65
7.28
7.91
8.60
9.81
10.94
12:01
13.23
8.74
15,28
LEAPGen-lite
#Parameters (M)
0.02
0.03
0.05
0.07
0.08
0.10
0.11
0.13
0.15
0.16
0.09
0.53
LEAPGen
#Parameters (M)
6.21
6.23
6.24
6.26
6.27
6.31
6.34
6.35
6.28
1.79
Table A9: Trade-offs between cost and performance on ImageNet-R dataset with 10 task setting. Con-Prompt-Large is run for task only since it crashed on the final task.
Running time (h)
Method
CIFAR100
ImageNet-RR
CUB
Descriptor
Inference
Training Total Inference
Descriptor Generation
Inference
Training Total
Descriptor Generation
Inference
Training Total Inference
Generation
Only
Only
Inference
Only
HiDe-Prompe
0.037
4.63
4.63
0.019
5.40
5.40
0.017
3.94
3.94
CromPrompt
0.53
0.023
2.01
2.54
1.07
0.033
1.04
2.11
1.09
0.035
8.08
9.17
LEAPGen-lite
0.016
1.2
1.2
0.028
0.53
0.53
0.028
0.38
0.38
LEAPGen
0.53
0.017
0.98
1.51
1.07
0.025
0.72
1.79
1.09
0.025
0.32
Table A10: Total running time including descriptor generation, training, and testing.
Performance
Method
Storage (MB)
FAA
CFA
HiDe-Prompt
75.75
79.27
334
HiDe--rompt-Large
74.30
78.56
797
ConyPrompt
77.08
81.47
346
ConyPrompt--arg
74.56*
80.79
632
LAEPGen-lite
82.38
85.14
332
LEAPGen
84.09
85.54
567
Table All: Performance Storage Trade-off's in ImageNet-R dataset, denotes measurement at task-9 since the method crashed after task-9
D.4 CLASS NAME AS LANGUAGE DESCRIPTOR
We extend our investigation of our proposed method in case there is no descriptor generated by GPT or other LLMs. We evaluate LEAPGen performance in a such condition in those 3 datasets as presented in table A12.LEAPGen CN indicates the variant of LEAPGen that utilizes class names as descriptors embedding. LEAPGen- only utilizes a text encoder similar to the majority of language-guided methods such LGCL. In addition, it doesn't utilize LLM such GPT as in
21

---

Published as a conference paper at ICLR 2025
ConvPrompt and GMM. Table A12 shows that substituting descriptors with class names doesn't t significantly decrease LEAPGen performance i.e. less than 1.1% performance. Despite utilizing class names descriptors LEAPGen-CN to outperforms existing methods with significant margin i.e. 5-14% FAA and 3-9% CAA in 100, 6-20% FAA and 3-17% CAA in ImageNet-R, and 1-21% FAA and 2.8-16% CAA in CUB dataset. This evidence confirms the promising impact of our ideas i.c. task-wise prompt generators and soft task-id prediction. Second, It confirms that utilizing language embedding as input for prompt generator remains effective, rather than utilizing the embedding directly as a guiding prototype or loss anchor.
CIFARIOO 10- Task TO classes/task ImageNet-K 10- Task 6: classes/lask
CUB 10-Task e220 classes/tsk
Method
FAA CAA
FFM
G.Gap
C.Gap
FAA
CAA
FPST
CFM
E.Gap
C.Gap
FAA
CAA
FFM
CFST
F.Gap
C.Gap
1,2
88.67
6.55
5.16
14.46
9.50
62.50 67.05
5.01
4,41
20.73
17.46
66.95 74.03
5.18
7.39
21.41
16.46
DualPrompt
541
12,94
68.59
4.61
3.70
14.64
12.33
73.95
7.87
8.60
14.41
10.20
CODA-P
91.27
6,38
5.09
11.86
6.90
73.77
79.38
7.94
6.72
9.46
5.13
72.99
82,64
11.71
9.91
15.37
7.85
LGCL
85.68 90.16
5.46
4,25
12.62
8,01
68.65
72.57
4.75
3.38
14.58
11.94
79.93 83.07
5.45
5.58
8.43
7.42
HiDe-Prropp
92.89 95.01
1.98
1.56
5.41
3.16
75.75
79.27
2.29
2.33
7.48
5.24
87.21 87.66
1.0
2.66
1.15
2.83
PGP
86.36 90.83
5.49
4,28
11.94
7.34
68.62
72.19
4.53
3.63
14.61
12.32
78.35
82 5.76
6.10
10.01
8.28
EvoPPropt
88.17
5.39
10.13
76.00
4.22
3.59
7.23
3.54
76.23
3.57
12.13
9.49
CPrompt
86.92 91.72
5.43
4.01
11,38
6.44
76.32
81
6.10
5.60
6.91
3.01
77.14
85.67
11.65
8.69
11.22
4,82
Con-Prompe
88.77 92.71
4.12
9.53
77.08
81 47
4.17
3.11
6.15
3.04
80.12
84 70
6.04
4.61
8.24
5.79
LEAPGen
98.38
15.15
0.10
0.05
-0.08
0.02
84.09
85.54
1,46
2.11
-0.86
-1.03
88.45
90.90
1.32
1.29
-0.09
-0.41
LEAPGen-CN
98.30 98.17
0.14
0.14
0.00
0.00
83.23
84.51
1.55
2.79
0.00
0.00
88.36
90.49
1.31
1.23
0.00
0.00
Table A12: Performance of LEAPGen with class name as descriptors compared to existing method and original LEAPGen. E.Gap and C.Gap indicate gap of the respective method to LEAPGen-CNN in terms of final average accuracy (FAA) and cumulative average accuracy (CAA).
D.5 PERFORMANCE ON VARIOUS DESCRIPTORS
We explore 3 types of descriptors i.e. short, long, and narrative generated by popular LLms i.e. GPT, Gemini, and Llama. The examples of descriptors for classes "Strawberry" and "Lemon" for respective types and LLMs are shown in Table A13, The table shows that GPT generates longer phrase than Gemini and Llama for the same descriptor type. We evaluated the performance of LEAPGen and compared it to ConvPrompt which is the second-best achiever in ImageNet-R dataset and also uses descriptors. Note that ConvPrompt utilizes descriptors define the number of new generators, while LEAPGen utilizes them as input to generate prompt components.
Table A14 shows the summarized result of LEAPGen and ConvPrompt performance on various descriptors. Overall, In any descriptors type generated by any LLMs, LEAPGen achieves at least 83.6% FAA and 85.0% CAA, where ConyPrompt achieve at maximum 77.6% FAA and 81.5% CAA. This implies than regardless any type of descriptors generated by any LLM, LEAPGen consis- tently outperforms ConvPrompt with at least 6% and 3.5% margin for FAA and CAA respectively. Looking at the average forgetting metrics, LEAPGEn also consistently outperforms ConvPrompt with least 3% and 1% margin for FFM and CFM respectively. Figure A5 shows the historical per- formance of LEAAPGEn and ConyPrompt in various types of detectors and LLMs. The figure shows that despite LEAPGen achieving slightly lower average accuracy in the first task, It consistently achieves higher accuracy in the second until the last task. In addition, the gap is getting higher with the increasing number of learned tasks. The figure also shows that despite LEAPGen achieving a slightly higher forgetting in the second task, manages to decrease the forgetting measure in the following task, until the last task. On the other hand, ConvPrompt suffers from the increasing av- erage forgetting so its average forgetting becomes far higher than LEAPGen's 's average forgetting in the later tasks.
Table A14 shows that the best result is achieved by LEAPGen with short descriptors generated by LLama. However, LEAPGen with short descriptors generated by Gemini achieves the least performance. Surprisingly, short descriptors generated by Llama are the shortest descriptors among all the evaluated types. This result gives insight that the length of descriptors is not the main factor in achieving maximum performance, rather the discriminative text has more impact. According to table A14, we also get insight that GPT produces more stable descriptors than the other LLSm, the performance difference between types of descriptors is insignificant (< 0.15%). Thus, regardless type of preferred descriptors, GPT is preferrable than the other L.MSs
22

---

Published as a conference paper at ICLR 2025
Type Descriptors
Short
Long
Narralive
"Strawbeery"
"Red. heart-shaped
"Small, leafy crown".
"Strawberry":
"Strawberry" "
"Juicy and sweet".
"Tiny seeds surface",
heart-shaped fruit", "small, green leaves", "white seeds on surface",
"A strawberry rests temptingly,
"Bright red color".
red surface dotted with taay seeds hinting at juiciness.",
"Soft, tender texture".
"Used desserts and salads",
"yellow white center", pointed tip". juicy
"The green leaves crown the top. contrasting beautifully with vibrant
"Often enjoyed fresh or in preserves".
"Distinctive aroma",
"With a sweet spirit, it emhodies the essence summer."
"Popular in summer"
"Lemon":
GPT-4
"Lemon":
"yellow or greenish yellow color".
"Lemon":
"Bright yellow, oval truit".
"oval round shape". smooth, glossy skin", "dimpled surface",
"A lemon sits cheerfully, its bright yellow skin and dimpled texture promising burst of citrus flavor",
"Tangy, acidic flavor", "Thick, dimpled skin"
"Juicy, citrusy interior",
"acidic smell", "sour taste", "thick, white pith". "seeds inside"
"Used for flavoring and zest".
"The zesty aroma vibrant color invite culinary exploration."
"Common cooking and heverages",
"With a tangy spirit, it embodies refreshment."
"Refreshing and tart".
"Variety sizes",
I
"Often used gamishes",
"Distinctive, vibrant color"
"Srawberry"
"red fruit with seeds on The outside".
"Strawberry":
Strawberry":
"red color".
"red fruit with seeds on outside".
"fruit",
taste",
"A small, red fruit with seeds outside.
color".
"red color".
It is a popular fruit and often in jams, jellics, and desserts."
"small size and sweet taste".
"small and
"used in dosserts and jams"
Gemini-1.5
Sweet taste"
"Lemon":
"I "Lemon":
"Lemoon"
"A yellow citrus fruit with
"yellow citrus fruit".
"yellow citrus fruit",
"yellow skin".
"yellow skin".
It is often used in drinks. cooking. baking."
"sour taste"
sour taste"
"citrus fruit".
"yellow
"sour taste".
drinks cooking"
"Srraweerr""
"fruit".
"sweet".
"juicy".
"red color".
"summer""
"Strawherry"
"Strawberry":
"shrrtcake". rich". "antioxidant"
"Sweet delights. luscious landscape indulging, happy heart enjoying like Joyful, little treat."
"fruit" "red color". "sweet"
"delicious" "romantic"
I
Llama-3.0
"Lemon":
"I Teman":
"I "Lemoon":
[
"frait". "yellow skin". "sour"
"fruit" "citrus"
"Zesty refreshes, citrasy landscape revitalizing, Rejuvenated heart refreshing Like bright, little sunrise."
"sour taste". "yellow skin". "juicing".
"cooking";
"favoorull".
"refreshing".
"zesty".
"invigorating"
Table A13: Types and examples descriptors generated by popular LLMs.
23

---

Published as a conference paper at ICLR 2025
LEAPGen
ConyPrompt
LLM
Desc. Type
FAA
CAA
FFM
CFM
FAA
CAA
FFM
CFM
GPT-4
Short
84.70
86.36
0.70
1.74
77.08
81.47
4.17
2.98
GPT-4
Long
84.73
86.14
1.48
77.03
81.40
3.95
3.28
GPT-4
Narrative
84.60
85.72
0.84
1.66
77.33
81.53
3.95
3.28
Gemini-1.. 1.5
Short
83.67
85,07
0.96
1.45
77.27
81.33
3.89
3,43
Gemini-1.5
Long
84.65
85.69
0.81
1.74
77.18
81.37
4.24
3.52
Gemini-1.5
Narrative
84.63
85.68
0.53
1.28
77.22
81.37
3.96
3.40
Llama-3.0
Short
85.22
86.35
0.79
1.54
77.25
81.37
3.94
3.40
Llama-3.0
Long
84.72
86.24
0.77
1.49
77.15
81.45
3.92
3.29
Llama-3.0
Narrative
83.72
85,33
0.80
1.49
77.59
81.49
3,93
3.27
Table A14: Performance of LEAPGEn vs ConvPrompt in various descriptors
Avg. Accuracy with GPT- Short
Avg. Accuracy with GPT Long
Avg. Accuracy with GPT Narrative
Avg. Forgesting with GPT Short
Avg. Forgetting with GPT Long
Avg. Forgetting with GPT Narraiee
K1.0
35.0
30.0
71.5
Accurscy Geminni-33 Long
Forgeetinggg Gemiin: 1.5 Short
Forgetting Gemini-1.5 Long
Forgetting (eem ini Narrative
Gemiini-155 (.5 Short
Gemiini Narrative
00.6
22.
71.
760
Yark Forgetting Liama-33. Narrative
Accuracy
Accurscy Llama-330
Avg. Forgettgg Llmma-d Short
Avg. Forgetting Llama-
L'ama-3.0
Liama-33 Narrative
20.
07.5
17.
51.5
00.8
303
28.0-
Taxt
Tea
75ch
LEAPGen
Figure A5: Visualization of the historical performance of consolidated algorithms in ImageNet-R with 10 tasks setting, in various descriptors and LLM
24

---

Published as a conference paper at ICLR 2025
CIFAR100 5-Tasks Average Accuracy
ImageNet-R 5-Tasks Average Accuracy
CIFAR100 5-Tases dess Average Forgetting
ImageNett-R 5-Taskss Average Forgetting
37-
75
Teat
Text
Ture
-e-1
DisiPrompt CODA-P -010 -iDDe-Prmmp -PPGP
-.hromp
CondPrompt
LEAPGen-lite -4-LFAPGen
Figure A7: Visualization of the historical performance consolidated algorithms in CIFARI00 and ImageNet-R with 5 tasks setting
D.6 PERFORMANCE ON VARIOUS TYPES OF DESCRIPTORS
LEAPGen's 's default generator (G G.T,A)) is single- head self-attention (SSA) network parameterized by 3 matrices explained in section sub-4.d. However, our method accommodates the different types of generators e.g. MLP (3-layered), LSTM, and GRU. Figure A6 shows that LAPGen with any type generator achieves higher performance than without generators (None). The pro- posed SSA generator achieves fairly higher performance than MIP, slightly higher than GRU and comparable to LSTM. However, LSTM and GRU has approximately 2.5 and 2 times larger number of parameters than SSA, while MLP has the same number of parameters as SSA.
NC Figure A6: FAA, CAA, FFM, and CFM of the proposed method w.r.t type of generator.
*N
CW
D.7 HISTORICAL PERFORMANCE ON AND 20-TASK SETTINGS
We continue analysis the historical performance of consolidated methods in CIFARI00 and ImageNet-R with 5 and 20-task settings. Figures A7 and A8 plot the historical performance of the consolidated methods. The figures show that both in 5-task and 20-task settings, both in CIFAR100 and ImageNet-R datasets, LEAPGen consistently outperforms the existing methods. Figure A7 shows that in the 5-task setting, LEAPGen manages to maintain its performance in CIFAR in all tasks, and suffers from less than 4% accuracy degradation in Imagent-R. The figure also shows that our method experiences less than 1% forgetting all tasks in both datasets.
Figure A8 shows that in the 20-task setting, LEAPGen suffers from less than 2% accuracy drop CIFAR100, while the existing methods suffer from up to 17% accuracy drop. In the Imagenet-R dataset, our methods experience less than 7% accuracy drop CIFAR100, while the existing methods suffer from up to 25% accuracy degradation. This is remarkable achievement pf our proposed method since the larger task number implies more challenging setting. In such difficult setting, our method can maintain its performance so that experiences the least degradation among the eval- uated methods. The figure also shows that method experiences nearly zero forgetting in the first ten tasks of CIFARIO0 100, and experiences less than 2% forgetting afterward. Our method also manages to decrease its forgetting measure in the latest tasks on ImageNet-R datasets. On the other hand, the existing methods suffer from increasing forgetting both in CIFAR 100 and Imagenet-R so its magnitude gets higher i.e. up to 9% and 11% in CIFAF100 and ImageNet-- respectively.
This historical analysis emphasizes the robustness of our method in various number task settings and confirms its superiority to the existing SOTAs.
E DETAILED EXPERIMENTAL. SETTING
The existing SOTAs are by executing the official implementation (code) of the respective methods. The hyper-parameters setting is chosen based on the official setting. HiDe-Prompt utilizes S- Prrompt(Waag et al., 2022a) iee. similar DualPrompt but without the global (task-shared) prompt. LGCL and PGP don't 't propose a specific prompt structure but rather utilize 1.2P and DualPrompt. The reported results for PGP and LGCL. are obtained with DralPrompt
Existing Methods:
25

---

Published as a conference paper at ICLR 2025
CIFAR100 20-Tasks
ImageNet-R 20-Tasks Average Accuracy
CIFAR10D 20-Tasks
ImageNet-R 20-Tasks Average Forgetting
Average Accuracy
Average Forgetting
47.
C8
Taat
LmP ---
DualFromet CODAPP - LGCL
Promot -
- Evoorromot
CPrompt -i-
ConnPrompt -- LEAPGen ihe LE-LEen
Figure A8: Visualization of the historical performance of consolidated algorithms in CIFAR100 and ImageNet-R with 20 tasks setting
structure which is the best from their result. The other methods i.e. 1.2,, DualPrompt, CODA-P, EvoPrompt, CPrompt, and ConyPrompt utilize their proposed structure. All the evaluated methods utilize ViT B/16 pre-trained ImageNet 21K the backhone model. L.GCL utilizes pre-trained CLIP text encoder, while ConvPrompt utilizes Transfoorrn pre-trained on BERT as its text encoder.
LEAPGen: Our proposed method is implemented on top of the ViT backbone pre-trained on ImageNet-21K. The prompt structure is as defined in section The prompt length is set to 30, and the prefix tuning layers is set to 7 i.e. 10,1,2,3,4,55, 6,6,6 for all main experiments. We utilize Adam optimizer with a cosine learning rate scheduler. For CIFAR100 dataset, We set 0.01 initial learning rate and 3, 10, and 10 epochs for 5-task, 10-task, and 20-task settings respectively. For ImageNet-R dataset, We choose 5, 10, and 20 epochs for 5-task, 10-task, and 20-task settings respectively. The initial learning rate is chosen from the best of O. 10.04,0.0,,0.00 and 0.005 initial learning rate. Our method accommodates a flat learning rate (not decayed) for K' in different optimizer; the soft task-id predictor is computed in batch-wise manner for convenience. Similar to ConyPrompl we utilize CentenceTransfomee as the text encoder. The pre-trained models i.e. ViT, S Transformer and CLIP(LGCL) parameters frozen (not fine-tuned) following ConvPrompt (Roy et al., 2024) implementation.
For CUB dataset, We choose 20 epochs
The λ2 and Å set to 1.0, while A is set to 1.0 for CIFAR100, and 0.1 for CUB and ImageNet-R. The number of generators per task that is the same as top-k descriptors is set to 3. All the consol- idated methods are run under the same machine and computing environment i.c. single NVIDIA A100 GPU with 40 GB memory. python 3.8 and Pytorch 2.2.0.
Performance Metries: Adapted from HidePrompt, we measure both accuracy and forgetting the methods. Suppose that Ar,, denotes the accuracy on the t-th task after learning the f-th task. The av- erage accuracy all learned task is defined as AA, = (1 (1/t) Ai, Suppose that 7 is the number of all tasks, we measure final average accuracy (FAA), cumulative average accuracy(CAA) final forgetting measure (FFM), and cumulative forgetting measure (CFM) as defined in the equations below.
FAA = AAT 1 CAA
(A56)
T AAc
(A57)
1
FFM-
T max (AAi,
- (AAAA--AA -Ai.))
(A58)
T
1
CFM T-1T-1
23 mar.el..... - - -Ais)
T
(A59)
F EXTENDED LITERATURE REVIEW
Rehearsal-based CL e.g. ICARL (Rebuffi et al., 2017), GD (Prabhu et al., 2020), XDER(Boschini et al., 2022) that saves exemplars from the previous tasks and replays them along with current task samples. The methods tune the whole backbone e.g. ResNet in each to adapt to learn new knowl- edge. The forgetting to previously leamnd tasks is minimized via replaying saved exemplars and distillation between the previous (saved) model and the current tuned model. The bias correction approach e.g. BiC (Wu et al., 2019) and LUCIR (Hou et al., 2019) adapt
task by adding
26

---

Published as a conference paper at ICLR 2025
task-wise bias layer. The bis layer is tuned to learn new task with minimum change of model weight. Therefore minimizes the forgetting on the previous tasks and achieves plaasiity- The bisa correction approach also utilizes memory replay to minimize forgetting. Regularization approach e.g. EWC (Kirkpatrick et al., 2017), MAS (Aljundi et al., 2018), and DMC (Zhang et al., 2020) that tunes the learner parameters accommodate the previous and current tasks. The regu- larization can be implemented as function or a saved gradient to avoid forgetting the previously learned tasks. The knowledge distillation can be applied in the regularization approach using a non-memory exemplar e.g. openly accessed dataset that is not part of learned task samples.
The prompt-based approach proposes a breakthrough solution for Class Incremental Learning (CIL) by training tiny tware parameters called prompts on top of frozen pre- trained feature extractor e.g. visual transformer (ViTsDDssvvssskiy et al., 2020) that contains far bigger parameters. ViT is sequence of I--layered model fo(.) where a layer-/ (1 E [1....) contains H Multi Head Self Attention (MSA) combined with Multi-Layer Perceptron (MLP) block, Norm, addition and residual connections. Given an input patched embedding T the ttth head MSA of layer-l, parameterized by koy-query-value matrices i.c. equation below:
1.h 1,k. W performs self-attention function as defined in
A(QU,, KL,,, VL,A) (oftmar((Qu.
(A60)
1,6. Ki,h = Tei.a W VL,A = Tei, W,,, are query, key, and value and D denotes their dimension. Prompt-based CL attaches a small set of trainable parameters called prompts into frozen pre-trained ViT model (fixed v/) (Wang et al., 2022c:b). The attachment techniques are mainly divided into 2 types i.e. prompt tuning (Lester et al., 2021) and prefix tuning (Li & Liang, 2021). Prompt tuning prepends a prompt vector P E R x D into query, key, and value of MSA. Suppose the prompt is divided into head-wise parts i.c. P = [P, P2 Ps....Pi then PA E DD is prepended into h head of 7th layer, and the attention mechanism is computed as A((P, Qu,al, [P, : K,,,], [Pr Vi,tl). The symbols Lp, D, and ":" denote prompt length, embedding dimension and concatenation operator respectively, Prefix tuning as utilized in DualP- prompt (Wang et al., 2022b) divides Py. into pair of P and Pk for key and value respectively, then the attention mechanism becomes A(QuA,, (PA KL,), [PX Vi,n)).
where Qi = Zel, W
Based on the prompt structure point of view, we can categorize the prompt-based approach into several categories i.e. (1)Pool-based approach e.g. L.2P(Wang et al., 2022c) that uses a pool as prompt container and select prompt based on input-similarity, (2) Task-specific approach e.g. Du- alPrompt((Wang et al., 2022b), SPrompt( (Wang et al., 2022a), HiDE- Prompt(Wang et al., 2024a) and CPrompt (Gao al., 2024) that train specific prompt for specific task and uses task identi- fier in the inference process, (3) Growing component approach e.g. CODA-P(Smith et al., 2023b), et al., 2024), and Conv ConvPrompt(Roy et al., 2024) that increase prompt com- ponents to adapt with a new task, instead of the task identifier. DualPrompt, and CPrompt utilize learnalle keys task predictor, while HiDe-Prompt utilizes unstructured representation (clusters) as the task predictor. CODA-P adds new hand-crafted (fixed size) learnable parameters as new prompt components, while EvoPrompt and ConvPrompt generate the new prompt components using newly added generators. Therefore, EvoPrompt and Conv ConvPrompt can be considered as growing generators approach.
EvoPromp((Kurnint)
G COMPLETE NUMERICAL RESULTS
In this section, we present the complete numerical results on CIFAR100, ImageNet-R and CUB datasets in all settings, as presented in table A15 to A21. Color blue and red denote the highest and runner-up performance respectively. Table A15 to A21 show that our proposed method almost achieves the highest average accuracy and average forgetting in all cases.
27

---

Published as a conference paper at ICLR 2025
Method
Lask
Avg.
2
Task
Avg.
Aecuray
Forgetting
L3P
34.87
90.11
0.45
57.56
0.35
8608
0.40
84.77
0.48
88.67 0.30
6.25
0.54
6.12
5.42
=0228
6.18
0.57
5.99
0.22
DuaiPrompt
95.40
91.21
0.14
59.22
0.15
57.53
0.15
86.41
0211
29.95
4,50 0.95
4.72
4.48 0.29
5.57
0.21
4,77
0.46
CODA
97.33
94.53
91.66
0.99
88.22
92.25 1.28
4.25
6.15
681
1.50
7,05
2.18
91.97
88.15
0.35
HiDe Prompp
97.30
94.24
0.08
93.18
0.11
91.99
200
20.95
2.23
2
2.36
222
0.00
2.52
0.33
2.33
PGP
96.422.0335
22.58:0.11
90.17
0.09
89.17
0.08
87.89
0.06
91.26.
4,45 041
4,57
0.225
4.07
0.04
5.32
0.18
4.60 0.15
0.24
Crrocpp
93.09
2.64
4.62
ConsPronye
18.18
0.25
92.88
0.25
91.18 0.16
0042
93.49
0.19
3.37
0.25
2.22
3.64
0.22
0.16
LEAPGen
97.18 0.67 0..28 96.11 +0.14 96.45 0.10
2..8 96.85
0.26
0,05
0.08
0.06
0.01
0.6
0.05
0.08
a 0.07 0.06 0:04
Table A15: Complete numerical result of the consolidated algorithms
CIFARI00 dataset with tasks setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
Method
Averago
Accuracy
(AA)
L2P
07.20:00.00
93.75
20.30
91.21
0011
89
0.30
86
11.9
85.32
0.38
85.14
0.30
$4.4
0.31
83.84
58.77
1..6
DealPromps
98.13
94.60
0.13
89
67.04
86
0.20
CODA-P
95.83
LGCL
98.2540.12
34.8220.3
0.44
92.99
0.30
0.350
0.56
89
87.94
0.30
87.33
2013
86.83
0.14
86.71
0.31
83.68
2043
90.16
00.29
IaDe Prompt
99.17:0.06
97.27
0.003
96.03
0.15
95.57
0.19
96.77
0.09
93.99
0.09
93.62
0.17
93.
23.30
0.10
93.44
0.16
92.89
0
95.01
0.08
PGP
09.85:0.29
95
90.64
ExcPrompt
96.37 0.36
58.17
Crrompt
98.20
96.28
0.83
94.18
0.43
93.34
0.24
91.79
90.69
1.04
89.19
0.94
88.62
0.96
87.76
0.60
86.92
91.73 0.66
ComPrompt
99.66
0.00
96.83 0:118 a
0.11
93.
0.24
92.77 = 0.13 90.66
0.16
90.18
89.71
0.02
89.74
0.26
98.77
0.24
92.71 0.04
LEAPGen
97,47
0.53
98.08
Average Furgelting
Measure (FM)
LZP
3:27 2.:40
4.15
4.53
0.35
3,60
2.32
2:225
5.55
0.39
5.44 0.19
5.95
2016
6.35
5.16
DealPrompe
3.47 0.22
3.60 0.13
3.58
0
0.26
4,73 =.24
4.66 0.21
4,50 0.19
5.41 0.33
4.33 0.155
6.38
LGCL
3.67 L.15
0.48
3.30
0.67
0.19
4.66
0.221
0.29
4.57 0.51
0.39
3,46 0.22
4.25 2.33
HilDe-Prompl
0.33 0.33
0.13+021
1.26 0.22
0.3
0.00
-.140014
1
1.75 0.05
1.98 0,05
1.56 0.15
PGP
0.77
4.33
4
Excerompt
1.17
1.00
4.47
CPronpt
1.60=1.25
2.60
1:13
2.04
5.53
4.59
4..8
-.580009
5.06
5.48
1.08
5.47
401 +081
Comproomp
1.17 0.33
1.85 0.18
0.31
2.22
2995 0.188
3.00
0.08
3.22
5:22 0.54
4.12 0144
2.67 #011
LEAPGen
0.0000000
0.05
0.06
CIFAR100 dataset with 10 tasks setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
Table A16: Complete mumerical result of the consolidated algorithms
28

---

Published as a conference paper at ICLR 2025
Average Accurncy (AA)
Method
10
Avg.
07.07
96.3
0.09
0.32
0.42
22
DealPrompt
07.73
97.
0.53
93
91
0.58
0.40
89.52
0.35
10
0.30
0.17
86.45
CODA-P
99.13+0.12
96.67
1.48
94.47
92
1.75
91
88
90.34
89.75
3.19
89.23
2.62
88
0.92
87.08
0.51
LGCL
98.20
98
0.10
93.96
94.00
0.36
93.01
0.22
91.06
0.18
90.88
0.29
89.33
0.34
88
0:40
87.99
0.266
HiDe-Prumpt
99.60
0,
96
PCP
08.53+0..33
98.57
0:40
94.42
94
0.18
93.13
94
0.13
91
89.94
88.71
ExaPrompt
99.20
99 10 0.26
94.20
3.35
93
93
0.81
92.93
0.47
91.77
0.40
91
0.60
90.23
0.47
67
0.55
$8.90 0.35
CPrompi
99.40
97.80
0.00
96.93
0.00
95
0:00
94 76
0.00
92.20
0.00
90.74
0.00
90.
72
0.00
07
0.00
89.24
0.00
ComPrompt
0..47
99
LEAPGen
4..73 99.60
2.117
99.58
0.08
99:42
0.12
99 A 0.09
99.36
0.07
99.31
0.08
99.35
0.09
99.19
0.07
4 40.09
11
12
15
14
15
16
17
19
20
Avg.
05.50
51.71
DealPrompt
85.96
84.89
0.23
84.24
0.18
78
0.37
83.01
=0.3
83.37
0.23
R3
0.19
82.52
0.23
87.47
0.34
CODA-P
86.21 85.64
1.58
85.24
1.53
83.82+
0.62
83.35 0.98 82.88
0.87
62.20
-0144
82.50
0.05
81
0.34
81.29
0.16
57.72
0
LGCL
87.44
96.29
0.007
85.29
$1.83
0.66
84.12
A 0.54
$4.15
0.50
63.96
0.51
83,91
0.58
$4.03
0.51
83.18
a
53.63
1.11
HiDe-Prompi
PGP
87.78 8
86.01
4 0.20
85
81
0.13
84.55 2 0.42 85,29 0.43
B4.57 0.41
84,80
0.51
84.52 0.21
83:41
4 0.35
89.23
0.13
ExePrompt
88.17 96.27
1.66
85.73
0.700
85
85.87
0.80
84.90 0.35
$5.40
0.26
$5.47.0.33
85.73
1 0.29
83 50.80.1
0.17
81.63
0.21
89.17
0.21
CPrompi
0..0
0.00
88.08
0.00
57.
0.00
86.6 61 0.00
86
0.00
65.72 = 0.00
85.49
0.00
84
14
0.00
83.60
0.00
90.10
0.00
Com-Propt
59:22
0.57
LEAPGen
09.18:4..00 98,46
1,22
98.45
98.47
1.02
97.83 0.95 97.91 0.89 97.96
0..81 98.03
0.73
97.69
1.35
96.51
A 2.16
0.26
Averige Forgetting Messure (FM)
Melhod
Avg.
1.47 0.50
5.47 0.91
4.29 a 0:21
5.22 0.48
5.28 a 0.56
6.19 0.20
6.32
0:40
7:20
8 0.41
7.19 0.42
DualPrompt
1.27 0.22
4
3.96
5.80 0.44
5.87
5.87
5.47
0.24
5.01
0:21
5.03
0.118
5,16 0.14
CODA-P
1.40
2.60
2.77
2.78 1.14
3.19 1.12
3
3.91
1.59
4,46
1.24
5,35
1.78
LGCL
0..3
061
1
0.4
0.87
HiDe Prompt
0.33 0.33
0.23
0..1
0.68 0.13
0.77 0.00
0.00
0.86
0.89
0.07
0.07
PCP
0.60 0.60
0.67
0.73
5.05 1.33
1.14
4.30
0.22
0.00
5.26 0.19
EvoPrompt
0.00
6.00 2.35
5.24
0.77
1.09
1
7.212
Crromp
1.00
1.00
2.87
3.15
0.00
3,57
5,24 1.00
Cun-Prompt
0.27 0.12
3:20
0.79
2.60
0.50
2.68
0.38
1.229
2.60
3.13
3.23
0.433
3.62 0.29
LEAPGen
0.00 0.00
0.03 à 0.06
0.00 & 0,00
0.000
a 0.00
0.00 & 0,00
0.01 0.02
0.02
0:03
0005
$ 0.03
0,09 4 0.000
14
19
L2P
7.562
1.06
2.555
0.0
7.77
20.41
7,05
0.5
7.69
0.75
8:31
2055
0.33
K.11 2013
H.T
0.30
6.79 0:33
DualPrompe
6.12
6.41
2.33
6.472:222
6.52
0 0.36
6.75
0:29
6.61
0.551
6.51
0.27
6.29
0.21
0.24
6,88
0.35
5.63 0.22
CODA-P
1.27
5:66 L.49
5.79 1.53
6.35
a 1.33
6.42 1.14
6.10
1.25
6.79
1.84
6 1,99
6.57
1.74
6.82 1.60
4.8 1.99
LGCL
0.0
5021
6.34
6.72
1.02
6.47
2087
0.76
6.362004
6.78 0.43
6.74
7.22 0.47
49.5
HiDe Prompt
0.74 0.00
PGP
6.38
6.77 0.33
0.12
6.95 + 0.18
6.72 0.36
0.48
7.35 0.5
21110:66
7.29 = 0.19
7.5 0.23
5.66 0.222
ExoPrompt
8.1520.50
2 0.63
0.89
9.24
0.45
8.46 2.45
0.39
8336
9,19
0.65
CPoopp
0,00
5.69
0:00
0.0
6.29
0.00
6.08
0:00
6.44
0.00
ComPrompp
4.75
4.50
0.33
5.17
5.10
4.74
0.18
4.87
4.96 0.43
5.37
3.92 +.33
LEAPGen
0.11
1.31
1
0.1
1.39
0.32
1,30
0.25
1.15
0.60 0.75
0.52 0.399
Table A17: Complete numerical result the consolidated algorithms CIFARI00 dataset with 20 tasks
setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
Method
Task
Fask
Average Accuracy
(AA)
Average
Forgetting (FMD
L2P
05.14
68.52
0.65
66.35
0.64
65.24 0:49 66
0.33
68.01 0.42
3:25 a
0.48
3.30
0.19
3.89
0.24
3
0
3.55
0.20
DuilPrrempt
70.09
69.71
72.78
0.25
CODA-P
15.90
81.65
2.50
79.28
1.96
7683
0.36
19.71 0036
6,30 155
7.39
1076
8
0065
7.65
0.98
LOCL
77.72
73.83
72.61
0.33
2.19
69.93
1..9
2.49
0.54
2.03
0.49
244
3.04
a 0.36
2.50
0.38
HiDo-Prompe
04.50
80.28
0.32
78
0.16
76 17
0.08
7540+0227
1.72
+.72.00.1
2.79
2.1
0.32
3.15
0446
2.64
0.16
PCP
0..25
00.55
1
0.16
296
3,36
EvoPrompt
8..00
84.07
$1.40
0.20
7860
0.00
77.27
0.40
81.67
0.88
0.90 0:
1,70
0.61
1:27
0:47
1.79
0.1
0.30
CPronpe
88.12+
0.00
83.42
02222-0000 0.00
81.66
0.00
80.36
0.00
78.65
0.00
82.44 0.00
5.75
0.00
4.58
0.00
5.62
0.00
6.00
0:00
5.49
0:00
ConsPromps
88.56
$4,15
82.26
0.37 80.33
0.25 79.36
0.08 82.93 0.24
1.41
0.18
2.24
à 0.28
2.36
0.22
3/42
0.05
2.36 0.16
LEAPGen
58.23
85
82.79 0.30
0.11
000
0.07
Table A A18: Complete numerical result of consolidated algorithm in ImageNet-R dataset with 5 tasks setting.
Color "blue" and "red" denotes the highest and runner-up performance respectively.
29

---

Published as a conference paper at ICLR 2025
Method
Task
Average Accuraty (AA)
Tdp
76.89
72.06
0.33
66
60
63.57
62
DualPronpe
78.97
76.34
0.330
74,96
0.21
0.30
68.59
CODA-P
09.24
#3000
82.10
2.00
50
22 2.11
78.57
+ 2.44
77.84
1.69
16.79
197
75.64
145
74
1.07
73.77
0.50
79.38
1.48
LGCC
7..59
76.61
+0.26
75.51
+ 40
73
0.30
72.24
+ 0.2
71.45
+ 0.17
70.0
0.34
69
+ 041
69
0.14
68.65
0.25
72.57
0.11
H.DDe-rrompt
85.22
82.93
0.59
77.37
76.31
75.75
PGP
0..97
76.44
0:40
74,96
0:45
77.87
0.09
71.32
0.39
70.90
0.26 69.78
0.19
69.20
68
68.62
72.
0.200
ExsPrompt
89.27
84.00
0.33
83.43
+ 0.23
K2.03
0.55
80.97
0.91
80.00
+ 0.72
78.53 0.74
77.43
0.47
76.83
0.25
76.00
0.26
50.97
0.30
CPhongt
90,91 0.770
86.17
2,15
84.25
52.57
+ 0.17
81.11
0:45 79.76
+ 0.38
78.67=0.15
78.14
0.36
77
0.15
76.32
+ 0.53
81.50 0.30
ComPPrompi
0.55
76:50=
77
0.28
LEAPGen
89.83+00338
87.63
0.68
86.46
0.68
1668
94.79
84.01
83.77
1.32
12.1
1.05
84,09
+ 0.93
85.54
0.66
Average Forgelting
Measure (FM)
1.2P
5:52 0.55.
0.20
4.52
4.35
0.48
4:00
4.06 0:40
4.43
5.11
4:41 0043
DualPronpe
0
2.51
0.07
2.97
+
+.29
0.12
4.61
0:18
CODA-P
5.04
6.06
0.85
5.83
1.47
6.61 1.12
6.94
7.37 0:94
7.56
0.37
7.94 0.08
6.72 0.77
LGGC
2.91 1:44
2 0.74
2.72
+ 0.84
0.15
3,23
3.65 062
3.68
0.54
4.39
4.75 0.18
3.38 0.58
Hille-Prampi
+..0
2.41
0.56
0.19
0.5
2.20=055
0.27
0.30
2.29
2:33 2.33
PGP
0.80
0.12
2.94
0 0:26
0.45
0.0
0.28
0.55
4.53
0.35
ExmPrompt
5.17+0.21
3.57 0.58
3.57 0.82
2,98
1.07
2.8+
2.70
1.24+040
0.0
4.22+00.2
3.59 0.55
Crromp
4.55 2.51
2
5.57
5 1.47
5.71
1,48
a 1.15
5.85
5.9
0.68
5.60 6.7
L.35
ComPrrmpt
2.57
0.08
0.24
0.31
0.36
0.11
4,17
00.17
LEAPGen
+.08
0.16+0.3
1.38 0.08
2.6
2.19
1-00
1.57
1 1.35
1.46 1.25
1.21
Table A19: Complete numerical result of the consolidated algorithms in ImageNet-R dataset with 10 tasks setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
Average Accuracy (AA)
Method
Avg.
L2P
0.41
73.00
0.33
72.34
0.85
69.9
0.99
0.99
68.49
à 1.10
67.47
1.11
65.97
=
0.53
64.47
1
0.50
64
0.33
6
DealPrompt
80.7810.31
78.08
0.82
76.46
74.33
0.34
73:22
0.45
73.58
0.52
72.12
0.08
71
0.36
70.99
0.555
0.55
68.67
1..17
CODA-P
9.77
85.37
84.23
80
79.51
7623
75.99
2.112
LGCL
80.7840.81
77.43
0.44
76.07
4 0.38
74.37
0.41
73.56
2 0
73.02
0.53
71.67
0.43
71.21
1 0.14
71.57
0.311
0:335
IaDe Prompt
0.70
65.34
84.22
a 0.20
61
0.52
90.95
4 0.35
80.56
1 0.87
79.77
0.28
78,76
1
0.47
79
0.80
77.56
PGP
00,78
78.11
0.77
76.40
0.58
74
0.54
73.28
73.33
0.28
72.04
0.16
76,93
0.46
71
0.66
EvePrompt
90.13
87.37
0.445
85.47
83.53
82.90
1.56
81
0.15
79.93
CPrompi
91,45
68.61
0.93
87.58
38
54
2.40
83.04
1000
81
81.13
1.01
80.45
1 0.68
79
46
0.49
78.64
1
Com-Proppt
91.89 0.55
87.69
0.6.1
85.29
0.53
93
90
0.11
82.55
56
0.31
82.03
0.71
60.27
8 0.59
78.67
9 0.58
81
0.69
77.10
0.53
LEAPGen
0.85
91.18
89.54
1.66
88,
2.27
$8.14
88.06
1.53
87.89
87
$7.34
1.07
87,01
4.112
14
16
Avg.
L2P
02.15
56.75
2:113
57.61 0.47
58.14
0.08
$7.80
0.30
37.09
2
0,05
57.0
57.21
2 0.00 30.49
0.14
$7.40
2
0.31
03.33
2022T
DealPrompt
67.55
67.65
0.25
67.12
0.41
67.14
0.53
67.02
0.47
66.65
0.34
66.35 0.35
65.87
65
0.38
6 0.17
70.31
0.29
CODA.P
1..3 74.92
1.46
1.35
73.24
72.14
71,82
0.80
0.49
70.55
LGCL
6..67
67.40 0.29 66.83
: 1.01 06.85
0.60
66.91 A
0.54
06.21
1 0.47
20.3
65.74
0.40
65.27
0.64
64.95
A 0.67
70.18 0.37
IbDe Prompt
$1.6
PGP
61,54:0:22
67.68
0.50
67.18
0.53
67.08
66.80
66.7220013
0.52
0.08
65.24
ExePrompt
78.73
76.63
0.55
76.47
75.67
0.29
75.17
0.40
74,93 à
Crrompi
17.59
77.26 1: 0.12
76.66
0.71
76.41
0.62
76.35 2 0.29 75.87 0.17
25.50=0.31
74.95
0.64
74.25
0.10
76,23
0.17
79.82 1 0.51
Com-rrmpt
75.95 00.11
0.270
75.45
23.40
75.58
2 0.62 74.79 0.15
74.25 0.23
74.51
0.41
77.228
0.32
73.95
à (1.6
78.92 a 0.37
LEAPGen
87.15 96.39
0.25
86.33
0.19
36.61
0.11
86.56
0.14 36.60
2.173
36,98
0.15
57.01
87,03
Average Furgelting: Measure (FM)
Method
4
0
9
10
Avg.
L2P
3.70
1
2011
0.33
DualPrompt
3.07 :
6.34
0.57
5,86
a 0.55
5.04
0.49
4.07
0.74
3,95
0.29
3.83
0.30
4.41 : 0.45
3,80 0.23
CODA-P
6.87 1.87
4.98
1.71
5.88
+ 1.24
5.55
1.18
5.97 + 1.21
6.1 0.30
6.81
1.52
7.10 1.30
7.13 1.50
LGCL
3.88 1.80
6.02
0.0
5.05
1.63
0.42
4.29 1.500
024
3.8 3.65
3,04 0.09
HiDDe-rompl
1.50
0.17
1
2.25 0.37
PCP
1.30
0.5
5.4
0.65
4.4
4.100000
4.08
0.13
0.33
ExePrompt
2.20
$.50
584
1.49
1
6.12
0.1
1.07
0.0
4.66 0.38
CPrompi
0.66
3.60
0.95
5.42
1.70
5.00
1000
5.79 0.92
574 1.19
5.72
0.2
5.95
0.64
5.86 0.46
Cum-rrmpt
2.00 10.1
0.71
3.69
3.07
0.20
0.14
LEAPGen
0.0000000 12
4.83 3.18
3,51
4.770
2.85
2.08
2.60
2.26
1.39
2.75
1.16
2.57
11
13
14
15
16
17
19
20
Avg.
8.781022
8.78
0.19
10.27
10
0.98
11.79
10.76
DealPrompt
0.31
4.82
0.32
4 0.00
5.
a 0.31
5.16
0.35
5.46
0.35
5.97
0.12
6.81
0.23
6.97
a 0.22
7,30
0.18
5.16 0.34
CODA-P
1.23
7.19
7.10 1.00
7.25
4 0.79
7.53
0.56
1.34
7.83
0.79
7.89
a 0.54
7.98
a 0.51
8.23 0.86
6.95 2..70
LGCL
4.2910.3
1.12
4 0.99
5.39
1
5.11 0.52
0.55
6.79 0.5
0.40
7.39
1199200 0.62
1 0.65
5.05 0.332
HiDD--rrmpl
PGP
0.57
0.31
0.55
5.05 4 0.35
4,95
10.1
0.10
0.16
0.29
6.72 2 0.08
0.174021
5.09 0.25
Exefrompt
5.56:1..0
5:27
0.54
0.71
0.77
6.09
0.69
0,55
6.28
0.78
6.96
0.59
6.76
6.72 0.90
5.67 1022
CPrompi
0.36
5.73
0.24
5.92
5
0.1
0,19
0.7
5.92
0.06
6.07
0,30
5.98 0.24
5.54 0.48
Cum-rompt
2.93 0.23
0.1
3.65
1.166
3,57
0025
0.55
0.38
4.29
LEAPGen
231
2699 0.26
2.63
10222
2.41
4021
2.14
2.35
2.09
2.16
2.17
2:54 10777
Table A20: Complete numerical result of the consolidated algorithms in ImageNet-R R datasct with 20 tasks
setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
30

---

Published as a conference paper at ICLR 2025
Average Accuracy (AA)
0663
DualPrompt
03.33
93.33
8-47
77.76
0.96
73.95
0.15
CODA-P
05.13+4.31
91.97
2.95
88.8
0.74
83
81.00
2.63
78664
75.94
1.01
74
0.68
72.99
0.30
82.64
LGCL
92.75
57.06
90
0.30
82.54
82.24
0.26
31
81.15 10 0.19
79
0.36
79.93
0.30
83007
0..23
HiTte-Prampt
04.06+00:41
PGP
92.62.10.89
96.97 0.66
82.34
90
0.53
70
0.72
11
0.50
80.11
89
0.56
78
0.66
78.35
0.68
82.21
1.556
ExoPrompt
3 1.27
65.10 0.00
83.83
0.47
52 40
0.95
80.93
0.90
79.93
0.64
76.53
0.84
77.37
0.72
76
0:49
76.23
0.51
81.00
0.40
(Prompt
97.39 1.09 97.82
1.78
90.32
83
85 66
12
83.99
0.84
81
1.57
80.55
79
77.14
85.67
0.56
Cam-Propp
LEAPGen
0.96
92.17
93.38
90.87
0.66
90.11
00.24
89.00
SR:45
0.58
00.90
0.8
Average
Forgetting Measore (F M)
5.18
DualPrompe
3.75 0.49
13.52
0.60
9.89
0.78
0.54
8.25
7.62
0.1
0.19
7,87
CODA
3.92
7.65
10.66
1.91
10.26 1.25
10.33 0.99
10.44 161
11883
11833
11.97
11.71
1.49
LGCL
1.49
10.08 0.74
6.92
0.59
5.82
3.94
a 0.35
1888100 0.29
4.57 0
5.55
5.58 0.339
HiDe Prompt
0.41
2:2
PGP
1.29
(0.92:00777
7.99
1.04
6.63 4.43
6.71 1.222
5.78
5.13
4.70
5.76
6.16
Exafrompt
4.67 2
3.28
1.07
3.28 1,50
3.08
1.08
3.22 0.52
3
3.44
4.08 0054
0.45
3.57 2086
CPromp:
4.25 1.97
1:4
9.18
1.27
Com-rrmpt
5.24
LEAPGen
0.19::0:4
0:13
1.33
1
1.44
1.87 082
1.62 0.67
1.46
1.32
0.82
Table A21: Complete numerical result of the consolidated algorithms in CUB dataset with 10 tasks setting. Color "blue" and "red" denotes the highest and runner-up performance respectively.
Setting
each
CIFAR100
Sitak 97.35 97.88
90.89
96
97.07
97.28
ImageNet
Sissk 87.41
81.98
$4.41
82.59
82.44
10-ask
ImageNes
10-ask
89.75 88
96.94
86.43
86.04
8300
22.75
82.52
82.94
82.38
85.14
CUB200
10-task 96.05 91.56 93
89
85.45
$1.65
93.97
84.07
85
85.06
$5.00
$3.03
CIFAR100
20-msk 99.67
99.50
99
98.28
99.35
99
99
99.17
99
91
99
97.72
97.78
97.27
97.83
97.85
97.91
98.00
95.85
05.25
98.37
ImageNec
54.26
LRAPGen-Ble Average For oorgetiing on each Task
Dutaset
Setting
2
8
10
11
12
13
14
17
18
19
20
Avg.
CIFAR100
Siusk
ImageNes-R
S-ask
8.05
0.04
0.17
0.43
0.17
CIEAR100
104ask
0.03
0.00
0.03
0.05
0.07
0.08
0.09
0.10
0.11
0.06
ImageNet R
10
1.11
0.75
3.37
292
2.70
2:13
10-ack
CIFARI0O
20-msk
8.00
0.07
0.02
0.05
0.03
0.07
0.07
0.07
0.06
0.09
1.64
1.33
2.08
1.38
1.34
133
1.11
1.18
1.08
0.70
ImageNet R. 20 task
0.00
0.09
0.16
0.12
0.17
0.16
0.28
0.31
0.43
0.34
0.49
0.46
0.61
0.67
0.81
0.95
0.88
0.90
1.0
0.47
Table A22: Complete Numerical Results of LEAPGen-lite CIFARI(M, ImageNet-R and CUB dataset.
31


Reviews:
[{"review_id": "9PdQCrAqdW", "reviewer": "Reviewer_ueBy", "summary": "The paper proposes a new method for prompt based continual learning. Unlike previous methods that focus on task-specific prompts or growing prompt components, this work generates prompts using language descriptions of top-k similar classes from all the previously trained tasks to an input image. The generated prompts are plugged in to the Key and Value pairs of the backbone model for the given input image. In addition, the combined encoded language descriptions of the top-k similar classes are appended to the input image embeddings. Result on CIFAR100, ImageNet-R and CUB datasets with pretrained ViT-B/16 backbone shows significant improvements across accuracy and forgetting metrics over previous works. Extensive ablation study on different components provides better insights on the proposed approach.", "questions": "1. Between L155-160, under the context of previous task-specific approaches, it is mentioned that two different tasks could produce similar key vectors. It would make a stronger point to justify this statement with some empirical analysis.\n\n2. What is the value of k for top-k descriptors and how many Generators are chosen for each task? Can we chose a single generator for each task? How {E_i}^t of size |C^t|gets chosen for k set of generators?\n\n3. From the Figure 2 or its caption, it is not clear what is the component that is compared against input x to get the cosine similarity. The sentence “search space for top-k” in the figure gives some hint, but the arrows for entire blue block rising the confusion. Please clarify it in the figure or caption.\n\n4. Why it is interesting to solve equation 3, if the task is to search for E_i in all tasks [1, 2, ….t]? Similarly, why consider K^t and why not just use {L_i}^t ?\n\n5. Under the ablation study, what does absence of generator mean? How do you get prompts in this approach without generator?\n\n6. Is the MLP head (classifier) also trained for each task? How can it produce list of softmax values of all classes for equation 7?\n\n7. Please explain why inter-task loss help improve FAA and CAA? What does it tell us about the learned parameters?\n\n8. Has the importance of inter-task loss already explored in previous continual learning papers? \n\nMinor suggestions:\n\na. In Figure 1, reposition the labels “fixed”, “trainable”, and “generated” since these labels affect entire Figure, not just the green block.\n\nb. In section 3.b, I suggest to use emphasized text style for “pool-based approach”, “task-specific prompt approach”, and “growing component approach”.\n\nc. In Line 242, correct the typo “tem” -> “them”\n\nd. Line 267 mentions t_hat, but eq. 3 uses notation as t, but not t_hat.\n\n------------------------------------------------------------------------\nFinal review: I thank authors for their responses. I find that paper proposes a novel method to leverage language descriptors within the context of continual learning. Authors have conducted thorough experimental evaluation and demonstrated notable improvements across multiple datasets. I read the fellow reviewers concerns and I find that authors have sufficiently addressed the major concerns and revised their manuscript accordingly. In particular, regarding the utilization of GPT, authors have justified that their method does not necessarily rely on compute heavy GPTs or LLMs, and can also work without GPT-generated descriptors. Overall, I believe that this paper would be interesting to the research community and suggest towards acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tIntroduction is clear in terms of discussing drawbacks of prior works and motivating the proposed approach. Contributions are listed clearly.\n\n•\tI appreciate Section 3 and Figure 1 as it gives provides brief recap of the previous SOTA methods and laid some foundation for discussing the proposed method.\n\n•\tLeveraging language descriptions and using all language descriptors to find top-k similar classes to generate the prompts is novel in the context of continual learning.\n\n•\tThe strategy to incorporate inter-class loss helps the method to achieve better results.\n\n•\tComparisons are made against various previous works and significant improvements are shown with their proposed method.\n\n•\tAppreciate the detailed ablation study to understand benefits of different components.", "weaknesses": "I don’t find any major weakness but have set of questions (see below) to improve the understanding. However, I find that learning with auxiliary data and inter-task loss are critical components of the method (as shown in ablation study), which are not sufficiently emphasized in the contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tbyhSJBQUZ", "reviewer": "Reviewer_J5sH", "summary": "This paper proposes a novel method LEAPGen (Language as Prompt Generator) for continual learning that leverages language knowledge for prompt generation.\nLEAPGen uses language descriptors for each class as input for prompt generation, rather than shared embeddings. For each task, LEAPGen employs a fixed number of generator networks, a learnable task-level key, and learnable class-level parameters associated with the descriptor embeddings. To generate a prompt, it first predicts the task ID using a soft prediction mechanism, then selects top-k matching descriptor embeddings based on cosine similarity to the input. Experimental results on datasets like CIFAR100, ImageNet-R, and CUB demonstrate that LEAPGen significantly outperforms existing state-of-the-art methods in terms of accuracy and forgetting measure, addressing key limitations of previous prompt-based continual learning approaches.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- LEAPGen uses language descriptors as input for prompt generation, leveraging richer semantic information compared to previous methods.\n\n- As a prompt-based method, it doesn't require storing and replaying old data.\n\n- By using fixed language descriptors as input, LEAPGen potentially reduces catastrophic forgetting compared to methods that continuously update shared embeddings.\n\n- Experimental results show significant improvements over state-of-the-art methods.", "weaknesses": "- The paper does not explicitly specify some key details about the pre-trained models used. Which Sentence Transformer does the paper employ? Whether this is a pre-trained language model and if any fine-tuning of these models occurs during the continual learning process.\n\n- The paper does not provide a clear comparison of the increase in parameters due to the addition of generators for new tasks. Based on the paper, the generator is not a lightweight module. As the proposed method requires adding new generators for some new tasks, even if not all tasks, this could lead to a significant growth in the number of parameters as the number of tasks increases. And also an unfair comparison with related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "86V2gLU6Eg", "reviewer": "Reviewer_UGi2", "summary": "This paper presents a novel approach to address the catastrophic forgetting in continual learning through a prompt-based structure. This method incorporates language inputs for prompt generation and utilizes task-wise generators and soft task-ID prediction. The authors highlight the advantages over existing methods across various datasets, showcasing substantial improvements accuracy while minimizing forgetting metrics.", "questions": "Please see the comments of weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors systematically review the limitations of existing prompt-based methods, and Figure 1 commendably illustrates the comparison of these methods.\n\n- The proposed method demonstrates performance that far exceeds current methods, which is pleasantly surprising.\n\n- The paper introduces the novel and meaningful use of large language models to generate more refined descriptions.", "weaknesses": "- The writing of this paper needs significant improvement. Additionally, the structure is overly compact, affecting readability. For instance, the theorem presented in the paper has weak relevance to the proposed method, and I believe it is unnecessary. I suggest moving it to the supplementary materials. Given the complexity and sophistication of the method, I encourage the authors to provide an overview initially.\n\n- The provided ablation study is not sufficiently convincing. I recommend that the authors validate the effectiveness of the proposed modules step-by-step starting from a standard baseline.\n\n- The inclusion of the ChatGPT model raises the question of how to ensure the fairness of experimental comparisons under such settings, and to what extent the performance gains are due to more accurate descriptions.\n\n- Considering the authors (are about to) open-source their code, I suggest making the descriptions obtained via ChatGPT publicly available for deeper analysis. I believe this will enhance the impact of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odOumiGhKf", "reviewer": "Reviewer_bACr", "summary": "This paper introduces a new prompt-based approach for continual learning that take advantage of language guidance. It uses task-wise generators, soft task ID prediction, and generated prompts as auxiliary data. The method outperforms state-of-the-art models on CIFAR100, ImageNetR, and CUB datasets with significant improvements in accuracy and forgetting measures.", "questions": "Please the weakness.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a detailed explanation of current prompt-based continual learning methods and analyzes the weakness associated with these methods, effectively leading to the introduction of the authors' proposed method.\n2. The paper presents a novel language-guided continual learning method and thoroughly demonstrates its effectiveness through extensive experiments.", "weaknesses": "1. The comparison mentioned in lines 183-184 between \"the photo of car\" and \"the photo of cat,\" stating they have 15/16 similarity but 1/16 dissimilarity, is confusing. The authors compared the similarity between these two prompts on a **letter-by-letter** basis and concluded that they are similar. However, the similarity between two prompts should not be evaluated in such a manner. After being processed by the tokenizer, the embeddings of the two prompts are not similar, as their meanings differ significantly.\n2. The proposed method relies on using ChatGPT to generate descriptive terms for each class and requires an additional Sentence Transformer to obtain embeddings. The introduction of these extra resources creates an unfair comparison with other methods and limits the practical applicability of the approach in real-world scenarios.\n3. Although the proposed method achieves state-of-the-art performance on multiple datasets, the improvement is minor on some, such as the comparison with HiDe-Prompt on the CUB200 dataset. This is particularly relevant given the substantial additional resources, like ChatGPT, required by the method, which other approaches do not utilize. Moreover, this suggests that the performance gains on fine-grained datasets may not be significant, as the generated descriptive terms for each class are quite similar.\n4. The authors have used excessive line spacing adjustments, which negatively impact the visual presentation of the paper, such as in lines 437-438 and 446-447. They should revise the layout of the entire paper to provide a better reading experience for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)