Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
Authors: Haodong Lu
Xinyu Zhang
Kristen Moore
Jason Xue
Lina Yao
Anton van den Hengel
Dong Gong
Venue: Accepted by TMLR ()

Abstract: Continual learning (CL) enables deep neural networks to acquire new knowledge over time while mitigating catastrophic forgetting of previously learned information. The powerful generalization ability of pre-trained models (PTMs), such as the Contrastive Language-Image Pre-training (CLIP) model, has inspired a range of CL methods targeting new and specialized tasks, further bridging the gap between PTMs and continual adaptation. Leveraging its multi-modal visual and textual representations, CLIP offers a natural paradigm for CL, where new tasks can be accommodated by incrementally learning lightweight parameters, particularly prompts. However, existing prompt-based CL methods for PTMs often rely on complex designs built upon specific assumptions, such as intricate regularization schemes for prompt pools, specialized routing mechanisms, or multi-stage incrementation processes.  While these approaches improve performance, they frequently introduce additional-and possibly unnecessary-complexity, underutilizing CLIP's intrinsic capabilities. In this paper, we propose a concise CL approach for CLIP based on incremental prompt tuning that fully exploits its multi-modal structure and the stability of textual representations. Our method, Textual Prototype-guided Prompt Tuning (TPPT), introduces textual prototypes not merely as static classifiers, as in existing methods, but as stable anchors to guide the learning of visual prompts, thereby shaping the embedding space (i.e., TPPT-V). We show that our bidirectional supervision strategy enables more effective learning of new knowledge while reducing forgetting. To further close the vision-language gap during CL, we activate the language branch and extend our approach to jointly optimize both visual and textual prompts (i.e., TPPT-VT). We also introduce a relational diversity regularization on the textual anchors to prevent embedding space collapse and mitigate correlated forgetting.  Extensive experiments and analyses demonstrate the effectiveness of our proposed approach, highlighting the benefits of leveraging CLIP's intrinsic guidance for continual adaptation.

Published in Transactions on Machine Learning Research (12/2025)
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
Haodong Lu
haodong. Sus..s.. edu xingu. shang...agggg. www..aaanngkklooonno Bauckland. ac ac.nz kristen. moore data611. csiro.ar au jason. vue @ @data61. csiro.
Xinyu Zhang
Kristen Moore
Jason X
Lina Yao!
lina. yao Gunsuun edu on anton. wwwdenhengeeeeeeeeeeee Gadelaiddd eduu dong, gong Sunsw. edu ou
Anton van den Hengel
Dong Gongl*
School of Computer Science and Engineering, University of New South Wales
Datab1, CSIRO
School of Compuler Science, Universily of Auckland
Australian Institute for Machine Learning (AIML), The University of Adelaide
Reviewed on OpenReview: https: // openreview. net/ forum? id= YJnjkek95Y
Abstract
Continual learning (CL) enables deep neural networks to acquire new knowledge over time while mitigating catastrophic forgetting of previously learned information. The powerful generalization ability pre-trained models (PTMs), such as the Contrastive Sanguage-Imagge Pre-training (CLIP) model, has inspired a range of CL methods targeting new and spocialized tasks, further bridging the gap between PTMs and continual adaptation. Leveraging its multi-modal visual and textual representations, CLIP offers natural paradigm for CI., where new tasks be accommodated by incrementally learning lightweight parameters, particularly prompts. However, existing prompt-based CL methods for PTMs often rely on complex designs built. upon specific assumptions, such as intricate regularization schemes for prompt pools, specialized routing mechanisms, or multi-stage incrementation processes. While these approaches improve performance, they frequently introduce additional and possibly munccssary-complexttyy underutilizing CLIP's intrinsic capabilities. In this paper, we propose concise CL approach for CL.IP based incremental prompt tuning that fully exploits its multi-modal structure and the stability of textual representations. Our method, Textual Prololype-guided Prompt Tuning (TPPT), introduces lextual prototypes not merely as static classifiers, as in existing methods, but as stable anchors to guide the learning of visual prompts, theroby shaping the embedding space (i.e., TPPT-V). We show that our bidirectional supervision strategy enables more effective learning of new knowledge while reducing forgetting. To further close the language gap during CL, we activate the language branch and extend our approach to jointly optimize both visual and textual the
prompts (f.e., TPPT-VT). We also introduce a relational diversity regularization textual anchors to prevent embedding space collapse and mitigate correlated forgetting. Extensive experiments and analyses demonstrate the effectiveness of our proposed approach, highlighting the benefits of leveraging CLIP's intrinsic guidance for continual adaptation. Code is available at https:/ haops://githulee fattuub.com/jeff//4////0
1 Introduction
Continual learning (CL) focuses on lcarning from sequence of data or tasks while accumulating knowledge over time and avoiding calastrophic forgetting of previously learned information. It. aims to do so with
*D. Gong the corresponding author.

---

Published in Transactions on Machine Learning Research (12/2025)
Learning on Task t (blue and green learned)
After Learning (purple
(a) Cross Entropy
TPPT-V (Ours)
(c) Naive extension (b) Multi-modal Prompt Tuning
TPPT-VT (Ours)
Visual Feature
Trainable
Repulsion (visual & textual)
Attraction
Textual Feature
Frozen
Repulsion (between textual)
Figure Conceptual illustrations of: (a) standard Cross-Ent ropy (CE), (b) our proposed TPPT-V, (c)
naïve multi-modal extension of TPPT-V, V, and (d) our proposed TPPT-VT. (a) Prior methods Zhou al.
2022e Jia al.) 2022] Wang al. 2022e d Smith et al. 2023 Wang et al. 2023 Zhou et al. 2023c use
CE loss to adapt PTMs, but suffer from representation drift (Gama et al. 2014 Lu et al. 2018 leading
to forgetting. (b) TPPT-V introduces textual prototypical contrastive loss to anchor visual features and
mitigate drift. (c) A naïve extension that also tunes textual prompts may improve textual prototype quality
but risks collapse to trivial solutions (Minderer et al. 2022 Kim el al. 2022 Khaltak el al. 2023a Liang
et al.) )022). (d) TPPT-VT addresses this by regularizing multi-modal prompt learning with diversity constraints on textual prototypes.
minimal storage and computations ional overhead by eliminating the need to retain all previously encountered
data (De Lange et al. 2021 Gomes et al. 2017 Mai et al.
2022). Our primary focus in this study is
class-incremental learning (CIL), a representative CL setting where classification tasks arrive incrementally,
each consisting of distinct, non-overlapping classes (Zhou al.
2023a Rebuffi et al.) 2017] To mitigate
catastrophic forgetting, various CL approaches have been explored, including experience replay (ER) methods
Luo ct al. 2023 Aljundi et al. 2019b Chaudhry et al. 2018a Liu al. 2020 Chaudhry ct al.
2018b Yan
et al.
2022 and regularization-bent
techniques (Kirkpatrick el al. 201 7 Aljundi et al. 2018
Zenke el. al.
2017
Aljundi et al. 2019a Jha et al.
2023), both of which have shown notable success.
Recent developments in pro-trained models (PTMs) based on the Transformer architecture (Vaswani et al. 2017) have demonsstrtted strong generalization capabilities and achieved state-of-the-arl performance across various applications (Dosovitskiy et al. 2020 Devlin al., [2018] Wortsman et al.) 2022). Among these, multi-modal pro-trained models, such Contrastive Language-Iman Image Pro-training (CLIP) RRdford et al.
2021), jointly learn lasion-language embedding spaces, effectively aligning visual and textual representations and enabling robust zero-shot generalization. To better adapt PTMs to downst tasks, parameter-efficient
fine-tuning methods such as prompt tuning (Zhou et al. 2022c Yao et al. 2023: Khattak et al.
2023a b) and
adapters (Gao et al. 2023 Zhang et al. 2021 Sung et al. 2022 have been developed.
In CL settings, PTMs such as Vision Transformer (ViT) DDoovitskiy et al. 2020) and CLIP (Radford
et al. 2021) often serve stable, "frozen" backbones for continually arriving downstream tasks (e.g., in CIL). CL methods built PTMs incrementally learn fixed or dynamically expanding set of trainable
parameters, such soft prompts (Wang et al. 2022e d Lester et al. 2021 Liu et al. 2021 Smith et al.
2023) or adapters (Jha et al. 2024 Zhou et al. 2023b McDonnell et al. 2024 Wang ct al. 2024 Lu et al.
2025 Yu et al. 2024h, to incorporate new tasks encountered in the data stream. CLIP, with its jointly Trained
vision and language representations, provides a natural foundation for CL (Jha et al., 2024 Wang et al., 2023). where textual embeddings can serve as pre-trained universal classifier heads for visual tasks. Due their lightweight and non-intrusive nature, prompt-based methods have gained popularity in CL, particularly
for CIT..
2

---

Published in Transactions on Machine Learning Research (12/2025)
However, existing prompt-based continual learning methods with pre-trained models (including CLIP and ViT)
often rely on task-specific prompt expansion (Smith et al. 2023 Wang et al. 2022d], complex regularization using external priors like ort orthogonality (Smith et al. 2023 Wang et al. 2022e| 2023), or intricate prompt routing schemes (Smith et al.) 2023 Wang et al. 2022d cp. While these mcthods can boost performance, they
often roduce additional and potentially unnecessary complexity, potentially underutilizing the intrinsic strengths of CLIP's multi-modal representations and the stability of its textual embeddings.
In this paper, we propose a concise CL approach for CLIP, called Textual Prototype-guided Prompt Tuning (TPPT), based on incremental prompt tuning without the need for complex regularization architectural modifications. TPPT exploits CLIP's inherent multi-modal structure and the stability of its uaxtual representations.
It accommodates new tasks by incrementally adding visual prompts (Jia et al. 2022 to the inputs of multiple multi-head head self-attention (MSA) layers within the CLIP encoders. While prompt tuning has been widely used for CL with PTMs such as CLIP (Jha et al.] 2024) and VIT (Smith et al.
2023
Wang et al. 2022d).
prior works often introduce complexity through orthogonal regularization (Smith et al.) 2023) probabilistic
modeling (Jha et al. 2024), handerafted prompt selection (Smith et al. 2023 Wang et al.
2023
In contrast, our method is simple and transparent, relying only on basic incremental and mixture mechanisms. This allows us to more clearly validate CLIP's inherent capabilities in CL.
We first present TPPT-V, which applies incremental prompt tuning only to the vision encoder of CLIP, guided by fixed textual embeddings of class labels. Prior works (Wang et al. 2023 Thengane et al., 2022 Ding et al., 2022 Zheng et al., 2023) typically treat these textual embeddings as static classifiers and use cross entropy (CE) loss to align vision features with corresponding textual embeddings. While this is effective when all classes are jointly trained, relying solely on CE leads to representation drift and interference (Gama et al.] 2014 Lu et al.; 2018)) in CL due to partial class visibility and the model's unawareness of future tasks resulting in forgetting (Nguyen et al. 2019 McCloskey & Cohon 1989.. as illustrated in Fig. (a)
and Fig. [(a]
As conceptualized in Fig. [ (a,, this issue partly arises because CE loss focuses only on assigning each sample in isolation to class label, ignoring how the embedding space evolves over time. We show that treating
the fixed lextual embeddings as protolype anchors (Li et al. 2020 Snell et al. 2017: Lu el al., 2024) and reintroducing contrastive learning supervision (Radford et al. 2021 Khosla et al. 2020 Grill et al.) 2020) alleviates this problem. Although the textual prototypes remain fixed, the contrastive loss (specifically, the asymmetric CE formulation) serves as an additional regularizer, guiding visual embeddings away from unrelated prototypes and stabilizing the embedding space (Fig. [b)]. In this way, CLIP's stable textual prototypes serve as anchors that help preserve prior knowledge by regularizing representation drift in the embedding space during CL.
To further enhance vision-language alignment in CL on downstream tasks, we go beyond vision-only tuning and enable lcarning of both visual and textual prompts. However, naïve joint prompt tuning can cause embedding space collapse (Minderer et al. 2022 Kim et al. 2022: Liang el al. 2022 Khattak el al. 2023a (as shown in Fig. [cc) and Fig. 2(b))p, especially in CL, where the model can easily converge to trivial solutions (Minderer et al. 2022: Kim et al. 2022 Liang et al., 2022 Khattak ct al., (023a). To prevent
this, we propose TPPT-VT (illustrated in Fig. [d) which introduces relational diversity regularization on
textual prompts, maintaining diversity among textual prototypes and mitigating collapse. By preserving the original contextual structure of textual representations, TPPT-VT improves performance over CL tasks.
In summary, our contributions are:
We propose concise CL approach (TPPT) for CLIP based incremental prompt tuning, without relying on complex regularization, external priors, or additional architectural overhead. TPPT leverages CLIPPss multi-modal nature and the stability of textual representations to explore its underutilized potential in CL.
We propose two variants of TPPT. TPPT-V learns visual prompts guided by fixed textual prototypes, which serve as stable anchors the embedding spaco-captune both class semantics and past knowledge beyond mere classification. To further bridge the vision-text gap in CL, TPPT-VT jointly learns visual and textual prompts while applying a diversity regularization to prevent embedding collapse and forgetting.
3

---

Published in Transactions on Machine Learning Research (12/2025)
1.4
23-UUP
Ris
1.2
Naio Medlicnnlal Triiing
1.20
for
Naioo Mulli modal Thiling
1.0
2.00
088
0.75
0.00
0.4
0.25
0.2
0.00
Amrndt
0.0
(a) Representation Drift 4.
(b) Diversity
Figure 2: Analysis of (a) representation drift (lower is better) and (b) feature embedding diversity (higher is better to prevent collapse). In (a), we measure how far Task 1 embeddings deviate after learning all tasks. Training with CE loss alone leads to significant drift (Fig. [a]), whereas our proposed LTrCL mitigates this (Fig. [b))- In (b), we assess embedding diversity via average pairwise distances over incremental stages. The red dotted line represents the diversity of the pre-trained CLIP model, denoted as ZS-CLIP. Naäve multi-modal training reduces diversity, especially on CUB, resulting in lower scores and potential collapse (Fig. [c)). Our dliversity-regularizee approach alleviates this, as shown in Fig. [dd..
Comprehensive experiments and analyses that demonstrate TPPT's ability to mitigate catastrophic forget- ting and improve adaptability to new tasks, achieving superior performance across multiple benchmarks.
2 Related Work
Continual learning. Experience replay (ER) based methods (Luo al. 2023 Aljundi al. 2019b;
Chaudhry et al.
2018a Liu et al. 2020 Chaudhry et al. 2018b Yan et al. 2022 Tong et al. 2025 involve storing subset of the original training data and replaying it during the training of new classes to remind
the model of old knowledge. Knowledge distillation-based methods (Li & Hoiem 2017 Rebulli el al.
2017
Douillard et al. 2020) use a teacher-student approach, where the original teacher model helps in training the updated student model by transferring knowledge, thus preserving previously learned information. Parameter regularization methods (Kirkpatrick et al.] 20177] Aljundi al. 2018 Zenke et al. 2017 Aljundi et al, 2019a Tha et al.
2023) add constraints to the learning process to retain important, parameters from changing significantly, preserving prior knowledge. Dynamic networks (Yan et al. 2021 Wang et al. 2022a b Zhou el al. 2022a Wang et al. 2024) adjust their architecture dynamically during learning, allowing different parts of the network to specialize in each task, thus balancing the learning of new classes with the preservation of old ones.
Prompt tuning. Prompt luning, originally proposed the Natural Language Processing (NLP) domain [Lester al. 2021 Liu al. 2022 2021), involves adapting pre-trained foundational models for downstream tasks. This is achieved by introducing small number of learnable embeddings at the input, known prompt tokens. Prompt tuning technique have demonstrated their effectiveness at adapting to downstream vision
tasks as well (Zhou et al. 2022c b Yao et al. 2023 Jia et al. 2022 Khattak et al. 2023a Rasheed et al. Khattak et al. 2023b). by prepending learnable prompt tokens to either the text encoder (Zhou et al. Yao et al. (0233) or the image encoder (Jia et al. 2022), or learning multi-modal prompts (Rasheed
2023:
2022c
et al. 2023; Khattak et al. 2023a b) that are integrated into both text. and image encoders. More recently, multi-modal prompt tuning (Rashoed et al. 2023 Khattak et al.] 2023a b) has emerged, which involves learning deep multi-modal prompts that jointly adapt the vision and language branches of the CLIP model for enhanced performance.
CL based on prompt tuning. In light of the effectiveness of prompt tuning in adapting pre-trained models to downstreem lasks, various prompt tuning-based methods (Wang et al.) 2022e 2023 Smith
et al. 2023) have been developed for continual learning. Generally, these methods employ a set of prompts to preserve previously acquired knowledge, with each component designed to encode sogment of incoming knowledge. Specifically, L2P (Wang al.) (202cc) utilizes a fixed number of prompt sets and selects the top-K nearest. prompts through query-key matching. DualPrompt (Wag et al. 2022d) innovates with both
4

---

Published in Transactions on Machine Learning Research (12/2025)
global prompts and an incremental set of task-specific prompts, also employing query-key matching for selection. S-Prompis (Wang et al. 2022cb adopts a similar set of prompt designs, but they are specifically tailored for Nomain-incrementa learning. CODA-P (Smith et al. 2023) learns an incremental set of visual prompts. It employs a weighted aggregation of visual prompts and learns the prompts relying on orthogonality regularization.
3 Proposed Method
3.1 Preliminaries
Class-incremental continual learning. Class-Tncremental Learning (CIL. is challenging scenario in the domain of CL (De Lange et al.) 2021 Gomes et al. 2017 Mai et al. 2022 where learning system is required to learn new classes over time without forgetting previously acquired knowledge [opee-Paz & Ranzato 2017 Rebufii et al. 2017). Formally, the learning process in CIL can be divided into a sequence of tasks D
[D D2 D'...., ,'' J. where each task D {(x,, )/) contains tuples of the input sample x and its corresponding label y E V. Following conventional CIL set tings (Rebulfi et al. 2017 Hou et. al. 2019 Wu et al. 2019), small number of exemplars from the seen classes selected as the exemplar set E, and are used jointly with training data D, of each task. CIL aims to develop model that predicts the label y given an unseen test sample x from arbitrary T tasks.
CLIP model. In the CLIP model (Radford et al. 2021 the image and text encoders are represented as fo and gw: respectively. For an input image x #xxxx where H, W, represent the height, width, and channel of the image, it undergoes segmentation into patches and subsequent processing through several Transformer blocks. This process yields a latent visual feature representation the dimension of the embedding space. The corresponding class label is integrated into hand-crafted prompls like "a photo of [CLS]" to obtain text input t, where [CLS) denotes the class name associated with class label This prompt is then encoded by the text encoder Ie to produce the text embedding
fe(x) RD. D represents
w (t) E R. The probability given image x being classified into class y; E (1,2, ".r",,")) ,C} is calculated as follows:
x)) = exp(sim(z Wy./T)
(1)
Ee exp(sim(z Wy./T))
where sim(-) denotes the cosine similarity function, and 7 is the temperature parameter.
Prompt tuning for CLIP. Prompt tuning has emerged as a highly effective and efficient technique for adapting pre-trained foundational models to various downstream tasks. A range of methods (Zhou al., 2022c Yao et. al. 2023 Rasheed el al. 2023 Khattak et al.) 2023a b) have been developed for adapting R L. and textual prompt Pt. RL-XD are prepended to the the visual and textual input tokens, steering the intermediate hidden states of the first layer of the model, and subsequently influence the final prediction. Here, Le and L, denote the lengths of the visual and textual prompts. The prompted visual and textual features are then denoted as ă == fx(xx) and w = (t),,, respectively. Here, fo and Go represent the encoders augmented with prompt tokens for image and text, respectively.
VLM. Specifically, visual prompt P P.,11
3.2 Overview
We present the framework of our methods in Fig. 3 Previous CLIP-based CL methods (Wang et al. 2022c 2023: Zhou et al. 2023c) usually treat the text encoder a pro-trained classifier, overlooking the rich representation capability of the language modality. In Sec. 3.3 we introduce TPPT-V, which guides the learning of incremental visual prompts with static textual prototypes. Since learning with static textual prototypes may be limited by the quality of prototypes, See. 3.4 we further propose TPPT-VT learn textual prompts for higher quality and representative textual prototypes. Our approach encourages textual diversity to better regulate the learning process.
5

---

Published in Transactions on Machine Learning Research (12/2025)
3p phoss ora (CLS)
(C1.8)
IPPr- Day roctal Momple sadichov
proklyps
Taxk
panto
Text Encoder
Tack
Image Encoder
Incremental multi-model prompts
Visual
Viuul Prompt Prompi
Trsirable
Frcuen
Duta Sample
Halk 1
Tac-1
Prototype
Anchoring learned visual representations prototypes
Attration
Weighted Aggregation Deep Vieual Prompte:
Repulsion
Figure 3: The overall framework of our 2 proposed methods, (1) The learned visual representations are guided by static textual prototypes (TPPT-V). We alleviate the forgetting issue by guiding visual representations with consistent textual prototypes, preventing drift of representations the embedding space. (2) To improve upon the static textual prototypes, we propose to learn textual prompts for protolypes (TPPT-VT), and regulate the learning process by encouraging diversity. (3) Benefiting from the textual prototype anchors, our proposed methods remain simple yet effective, unlike previous methods that use delicate, complex designs.
3.3 Incremental Visual Prompt Tuning with Textual Prototypes
Incremental visual prompt tuning. Prompt-based methods in CL have been proven to be highly effective in mitigating catastrophic forgetting, primarily by utilizing fixed or incremental set of prompts (Wang et al. 2022e Smith et al., 2023). Building on this, we employ a straight forward method to prepend an incremental set of visual prompts on multiple Transformer layers (Jia et al. 2022; Rasheed et al. 2023 Khattak et al. 2023b). These prompts are denoted as P.,. = (P.. P2 Po.:-... Pa for cach Transformer layer ranging from the hist layer up the dy-th layer of the vision encoder, where M is the total number of prompts, and d, is hyperparameter indicating the depth of Transformer layers at which prompts learned.
During both the training and testing phases, instance-specific visual prompts are generated for each input image, using weighted aggregation mechanism. The prompts used for a given input image x at the I-th Transformer layer during training and inference are computed as follows:
P = Σ
Pm
(2)
v.b
m=1
where the aggregation weights or (g(x)) RM are produced by lightweight trainable linear layer applied at each layer with trainable prompts to obtain the affinities for the M candidate prompts, in contrast
to the complicated attended query-key matching with additional learnable attention vectors used in (Smith
ct al. 2023). Here, the query function q(.) fo(-) R refers to the pre-trained image encoder (Wang
et al. 2022e Smith et al. 2023). Our concise formulation adopts the widely used query key prompt. selection/ aaggriggggggrgttionnn mechanism (Wang et al. 2022e Smith et al. 2023) and simplifies prior designs by omitting incremental Icarnable attention vectors and orthogonal regularization (Smith et al. 2023
The prompt tuning process is guided by consistent and stable knowledge derived from previously learned information through textual prototypes during incremental stages.

---

Published in Transactions on Machine Learning Research (12/2025)
Anchoring the past with textual prototypes. Existing CLIP-based CIL methods [aang et al. 2023 Thengane et al. 2022) often utilize the text embeddings merely as a pre-trained classifier, thereby not fully exploiting the rich representational capacity of the language modality. To this end, we propose TPPT-V to guide the learning of incremental visual prompts through the use of static textual prototypes as anchor points in the embedding space. TPPT-V mitigales rapastrophic lorgelting by consistently ensuring that, visual representations remain closely aligned and compact around corresponding static textual prototypes.
Specifically, our textual prototypes are derived from text embeddings, obtained through hand-crafted prompt templates with class names from OpenA[] whenever available. For datasets where prompt templates not provided, we apply the basic template of "a photo of [cls]". To enhance the alignment (Wang & Isola 2020) between visual features and their corresponding textual prototypes, we propose the Textual Prototypical Contrastive Loss Li et al., 2020: Cui et al.) 2023 Li et al.) 2023). This approach leverages the robust pro-trained representational capacity of these textual prototypes while preventing visual features from being assigned to the classes previously Iearned. It also effectively prevents the drift of previously Iearned semantics throughout the incremental learning phase (Fig. 5 We denole the Textual Protofypical Contrastive ive Loss as:
- log LTPCI. - - =1 E exp(sim(w)) W. ze/T))
1
1 = c) exp(sim(w We 2,/T))
(3)
where 1-) is the indicator function that selects samples corresponding to the given textual prototype. = [x) is the visual embedding obtained through the prompted image encoder, and We is the text embedding for class c, acting as a static textual prototype. N denotes the number of training examples the current mini-batch, and C is the number of classes already seen. At incremental step i, we construct the textual prototype set using only the class labels that have been introduced up to the current task. Text embeddings for future classes are not instantiated or used in any component.
Training objectives. The overall training objective for TPPT-V is formally denoted as:
CTPPT-V LCe + CIPCL.
(4)
where Lce represents the cross-y loss between predicted probabilities and ground truth. All loss terms are computed with the prompted image encoder So and the frozen pre-trained text encoder Pe- This loss shares a similar form with the pre-training loss of the CLIP model (Radford et al. [021), but differs in purpose. While CLIP focuses on the one-to-one mapping between an image and its own caption to form a broad representation space, our method aims to cluster image embeddings to be compact around their corresponding class text embeddings, from the perspective of CL. to mitigate forgetting.
3.4 Incremental modti-modal prompt tuning with regularized textual prototypes
Incremental textual prompt tuning. Although static textural prototypes from hand-crafted text prompts of classes effectively serve as stable anchors TPPT-V, the effoctiveness can be limited by the gap between pre-trained CLIP and specific downstream tasks (e.g., fine-grained classification), as well as the inherent discrepancy between the textual and visual embeddings in CLIP (Liang et all 2022), To address these limitations, we introduce TPPT-VT to further incorporate multi-model prompt tuning using learnable soft text prompts. Similar to the design of visual prompts in TPPT-V, we use an incremental set of textual prompts for each Transformer layer 1, ranging from the first layer up to the de-th layer of the text encoder, where de is set as a hyperparameter indicating the depth of Transformer layers at which prompts are learned. We denote these prompts as Pt, (P),, P....... P.]. where cach class is assigned to a unique textual prompt, and C represents the total number of classes encountered thus far. By using unique prompts for each class, TPPT-VT further enhances the fixed textual prototypes in TPPT-V, making them more representative.
Regularizing textual prompt tuning. While guiding the model with textual prototypes derived from hand hand-crafted prompts has demonstrated superior performance, the joint training of textual and visual prompts introduces new challenges. One critical issue is the model collapse, Le, both branches of the joint-enbedding model produce nearly identical and constant output vectors (Jing et al. 2021 Bardes et al., 2021 (as
hutps///ghhubbcom/oeena///i
7

---

Published in Transactions on Machine Learning Research (12/2025)
illustrated in Fig. 6 This collapse compromises the model's ability to distinguish between diverse inputs and effectively learn meaningful representations. In addition, naïve joint training can adversely affect the uniformity of the pre-trained model (Wang & Isola, 2020). Uniformity is a desirable property for joint- embedding models, ensuring that feature vectors are uniformly distributed roughly on the unit hypersphere. This uniform distribution is essential for preserving the maximum amount. of information from the data.
To address these issues, we introduce the textual prototype diversity loss, which regularizes the model by explicitly enforcing textual prototypes to be uniformly distributed on the unit hypersphere. This is achieved
by promoting high pairwise distances among textual prototypes (Wang & Isola 2020 Cho et al. 2023
formally defined as follows:
DDIV = log E m, exp(-Ilum - wn 12).
(5)
where [C] represents the index set of the already-seen C classes, and # indicates that the computation of pairwise distances excludes the self comparisons.
Training objectives. The overall training objective for TPPT-VT is formally given by:
CTPPT-VT LTPPT-V + aCDIv,
(6)
where a indicates the weight of textual diversity loss, and all loss terms are computed with the prompted image encoder Ja and the prompted text encoder Fø-
4 Experiments
4.1 Experiment Settings
Datasets. In line with common CIL benchmarks (Rebuffi et al. 2017 Wang et al. 2022e d Smith et al. 2023 , we evaluate the performance using widely used datasets CIFA
Krizhevsky et al. 2009), ImageNet- R Deng et al. 2009); TinyImageNNt. (Le & Vang 2015), and CUR200 Wah al. 2011). To further validate the efficacy of our approach, we extend our evaluation to include two additional datasets, FGVCAireraft
MMaji et al. 2013) and Stanford Cars (Krause et al. 2013), following the popular VLM methods (Zhou et al. 2022c Yao et al. 2023 Khattak et al. 2023a b). Specifically, we split the training datasets into 10 tasks, resulting in 10 classes per task for CIFAR100, Aircraft, and Stanford Cars, 20 classes per task for ImageNet-Ri TinylmageNet, and CUB200. We sample 20 exemplars per class as our replay buffer as in (Rebulli et al.,
2017
Training details. For fair comparison, use the same backbone model CLIP with VIT-B/16, using pre-trained weights from (Ilarcco et al. 2021 Radford et al.) 2021). We train all methods with batch size of 64 for 5 epochs, using SGD with momentum for optimization. The learning rate starts from 0.1 and decays with cosine annealing. The visual prompt length Ly and the textual prompt length L, are set to 4, and the depth for both visual prompts d, and textual prompts dN are set to 12. The number of prompts per task is set to 10. The loss weight a set to by default.
Evaluation metrics. To demonstrate the effectiveness of our approach, we report two commonly used metrics (Rebuffi et al. 2017): (1) the final accuracy Ar after learning all the T incremental stages, (2) the average accuracy A T Σ. -- As of the individual accuracies across the incremental stages.
4.2 Experimental Results
Comparisons with prompt-based methods. In Table we evaluate the final and average accuracies across various datasels, comparing current state-of-the-arl methods. Additionally, we illustrate the performance across incremental stages in Fig. | and provide evaluations of multiple independent runs with varied orders of the incoming classes in Appendix Our proposed methods, TPPT-V and TPPT-VT, exhibit a remarkable improvement over previous approaches. In contrast to previous prompt tuning methods like CoOp and VPT, learn an incremental set of prompts, effectively mitigating the issue of incoming tasks overwriting
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table Average and final accuracy different methods. All methods are trained with identical exemplar sizes, utilizing the CLIP model with a ViT-B/16 backbone using the same pre-trained weight from LAION-400M (Ilharco et al. 2021). A denotes the average test accuracy over the incremental stages, Ar denotes the test accuracy at the last incremental stage. The best and second best performances indicated in red and blue, respectively.
CIFARI00
ImageNet-R
CUB
Aircraft
Cars
Methods
A
Ar
A
Ar
A
Ar
A
Ar
A
Ar
ZS-CLIP
81.38
71.26
82.93
76.67
76.01
64.50
26.66
17.22
82.60
76.37
CoOp
83.37
73.36
82.40
76.20
77.34
68.70
44.26
39.87
89.73
84.91
VPT-CL
87.11
78.21
84.60
78.97
80.81
69.89
35.52
25.50
93.71
91.49
L2P
76.42
66.21
75.73
67.22
79.23
68.54
55.06
44.88
83.81
72.44
DualPrompt
79.07
70.06
78.47
70.82
83.21
74.94
50.93
46.53
85.30
74.35
CODA-P
83.53
72.20
74.45
65.88
83.36
76.12
54.51
45.06
96.16
92.17
PROOF
86.70
79.05
85.34
80.10
84.93
79.43
61.00
53.59
93.26
89.84
TPPT-V (Ours)
88.28
80.81
85.62
80.75
85.77
80.28
63.47
54.85
95.26
94.57
TPPT-VT (Ours)
87.20
78.98
84.87
81.02
86.60
81.55
69.34
61.99
96.83
95.37
3004
Accuracy (%)
Accuracy (%)
Accuracy (%)
90-
65
12
PROCR
30
to su to D00
66
the nio nio
the the rio sio nho Number of Classes
Number of Classes
Number of Classes
(a) CIFAR100
(b) ImageNet-R
(c) CUB
Accuracy (%)
Accuracy (%)
60.
50
AUTE
VFF
City
50+
POOOF EOPEATE
TEWAT
20 30 00 30 60 70 90 00
10
d0 50 0 to mm
Number of Classes
Number Classes
(d) Aireraft
(e) Cars
Figure 4: Experiment results across incremental stages. All methods are trained with the same exemplar size using the same pro-trained weights and backbonc model of CLIP with ViT-B/16
previously learned parameters. Compared with prior CIL methods, steers the learning of visual prompts using fixed textual prototypes, providing reliable anchor points in the embodding space. TPPT-VT can perform the best (and better than TPPT-V) consistently the datasets with fine-grained classification characteristies, i.e., CUB, Aireraft, and Cars, which indicates that the textual prompt tuning helps fill the gap downstream tasks that have more specific characteristics. For the normal datascts, e.g. CIFAR100 and ImageNet-R, text prompt tuning may introduce training difficulties, influencing the performance of TPPT-VT. Table 2 we extend the comparisons to include two related methods DIKT and CPP, with further details in Appendix A.2
Comparisons with fine-tuning and adapter/LoRA-Base methods. In Table 2 we compare method against fie-ttnnigg-bsse approaches such as 7SCL, (Zheng et. al. 2023 and recent.

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Comparison with prior methods on CIFAR-100 and Tiny-Imas Liny-ImaeeNet using a 10-task split following
the experiment settings of (Zheng et al. 2023 Yu al.
2024). All methods utilize CLIP (ViT-B/16) with
pre-trained weights from OpenAl (Radford al. 2021
CIFAR100
TinyImageNet
Methods
Ar
Á
Ar
ZS-CLIP
74.47
65.92
69.55
65.59
ZSCL
82.15
73.65
78.61
71.62
InfLoRA
85.66
75.09
79.88
74.42
DIKI
84.43
76.14
81.29
75.66
CPP
84.65
75.57
77.92
72.22
CLAPACLIP
82.09
76.77
80.90
74.44
MoE-Adapter
85.15
75.98
80.23
76.35
TPPT-V (Ours)
83.88
77.85
83.18
77.51
TPPT-VT (Ours)
85.72
77.75
82.41
78.08
1.25
Repoentalon Deitt
0.20
Aack
Reprerciittioo Drib
0.3
Paizwiee Dietanee
1.00-
vid.
Paicwtio Distance
008
0.6
0.15
0.2
0.75
0.10
0.1
0.2
0.05
2.5
Number of Tasls
5,0
7.5
10.0
2.5
Number of Tasis
5.0
1000
122
Number Tusks
5
9
Nuraber
1
Tacks
8
9
10
(a) CUB
(b) Aireraft:
(a) CUB
(b) Aircraft
Figure 5: Representation drift + (lower is better)
Figure Pairwise distance 1 (higher better) be- tween visual class means across incremental stages for (a) CUB and (b) Aircraft. A higher pairwise distance indicates the model has better learning capacity and is less prone to collapse.
across incremental stages for (a) CUB and (b) Air- craft. Representation drift measures the divergence in the means of visual features for each class compared to the last incremental stage.
adapter/ LoRA Hu et al. 2021 based methods, including InfLoRA (Liang & Li 2024). MoE- Adapter Yu al. 2024). and CLAP4CLIP (Jha al.) 2024). Despite utilizing additional reference data (Zheng et al.] )023), fine-tuning- methods suffer Trom rapid forgetting due to full model updates and computationally expensive. Adapter-based methods, the other hand, struggle to maintain vision-text alignment across incremental learning stages, leading to suboptimal knowledge retention. CLAP4CLIP adopts probabilistic fine-tuning with adapters and mixtures of task logits, which leaves the image text embedding space under-constrained. In contrast, our approach leverages textual prototypes as stable anchors to guide the lcarning of incremental visual and multi-modal prompts. Our proposed method effectively mitigates representation drift, enhances cross-lask consistency, and significantly improves continual learning performance while maintaining computational efficiency. Additional comparisons with the fine-tuning based SPU method (Zhang al. 2024 are provided in Appendix Table 10
4.3 Discussion
Learning with textual prototypes prevents representation drift. Catastrophic forgetting in continual learning is often attributed to representation drift (Gama et al. 2014 Lu et al. 2018). phenomenon where the statistical properties of the larget variable, which the model aims to predict, undergo changes during incremental learning. In Fig. 5 we measure representation drift during the incremental learning process by computing the divergence the mcans of visual features for cach class, relative to the class means from the previous incremental stage. We find that the phenomenon of representation drift (Gama et al. 2014 Lu et al. 2018 is severe, especially for the first few incremental stages. By enforcing visual features to be
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Effectiveness Textual Prototypical Contrastive Loss in prompt tuning methods like VPT--L and TPPT-V (Ours), on both CIL (A and Ar) and Joint Training (AJaint).
CUB
Aircraft
Methods
A
Ar
A
A
Ar
Ajoint
w/o LTPL
80.81
69.89
71.97
35.52
25.50
30.36
VPT-CL
w/ CTPCI
81.42
71.57
73.37
41.73
30.57
30.45
w/o w/o LTCCL
83.78
76.97
79.64
60.53
53.20
57.19
TPPT-V
w/ CTeCL
85.77
80.28
79.77
64.58
54.85
59.35
Table Ablation on Textual Diversity.
CUB
Aireraft
Methods
A
Ar
A
Ar
w/o LDIV
85.18
78.24
67.99
59.44
TPPT-VT
w/ CDv
86.60
81.55
69.34
61.99
compact around their fixed textual prototypes through our proposed textual prototypical contrastive loss, we effectively mitigate catastrophic forgetting by significantly reducing representation drift.
Regularized textual prototypes encourage uniformity and prevent collapse. We quantify the pairwise distances between class means of visual features across different incremental stages in Fig. 6 ZS-CLIP with pre-trained weight exhibits consistent uniformity (Wang & Isola 2020 Cho et al. 2023 and is less prone to collapse (Jing et al. 2021 Bardes et al. (021), due to the massive amount of data seen during pre-training. However, challenges arise when textual and visual prompts are learned jointly. In the initial incremental stages, the model tends to exhibit a tendency towards collapse, characterized by class means clustering too closely. The proposed diversity regularization effectively promotes uniformity of the model and prevents collapse.
4.4 Ablation Studies
Effectiveness of textual prototypical contrastive loss. In Table 3 we demonstrate the effectiveness of proposed textual prototypical contrastive loss prompt tuning methods, including VPT-CL. and our proposed method TPPT-V, across CUB and Aireraft datasets under both Class-Incremental Learning and Joint Training settings. For Joint Training, where models are exposed to all classes simultaneously during training, a good decision boundary between classes can be achieved using CE alone, while learning with textual prototypical contrastive loss slightly improves the performance.
Most importantly, for CTL, during the early incremental stages, models initially encounter only a tiny fraction of the total knowledge to be learned. The model has no idea what it will encounter in the future; thus, learning current data does not take future data into consideration. However, when learning new data, some embedding space is indeed needed to accommodate upcoming tasks (as illustrated in Fig. [a),, which causes representation drift (as shown in Fig. 5) of previously learned data. The use of textual prototypical rastive loss helps mitigate this issue by maintaining more stable and meaningful embeddings throughout the incremental learning process.
Effectiveness of Textual Diversity Regularization. Table ablates our diversity loss, which enforces uniform textual prototypes the unit hypersphere to boost per-class retention. Fig. demonstrates robustness across regularization weights, performance peaks at 0.8 and adopt by default for simplicity.
Number of prompts. In Fig. g. we evaluate the impact of varying the number of prompts per task on the performance of our methods. We conducted tests with our methods using a range of prompts per lask, specifically {1, 2, 3, 4, 5, 10, 20, 50}. Our findings reveal that for TPPT-V on the CUB dataset, using only a single prompt per task inadequate. Both TPPT-V and TPPT-VT exhibit favorable performance when utilizing least prompts per task. Consequently, we opt for 10 prompts per task by default, ensuring an effective trade-off between the training budget and performance.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Ascurwey (99)
81
Accnreey
Aucunayy
80
62.5
1.0 Texs wrightt a
1,5
0.5
15
20 Numbor Procupts
20 Prompts
40
Loss weight
(a) CUB
(b) Aircraft
(a) CUB
(b) Aircraft
Figure 7: Effect of the textual diversity loss weight on (a) CUB and (b) Aircraft.
Figure 8: Ablation studies on the number of
prompts per task for (a) CUB and (b) Aircraft.
85
85
Aveu wey (%)
86
Aoou weg
880-
Accur any
84
80
80
3
12
3
12
12
12
24
Visual Prompt Depth
Tectual Prompt Depch
Visual Premutt Length L
Textual Prompp Length Ly
(a) dg
(b) de
(c) by
(d) l
Figure 9: Ablations multi-modal prompt formulation. Sub-figures (a) and (b) analyze the depth of visual and textual prompts, while (c) and (d) investigate the length of visual and textual prompts.
Prompt formulation. In Fig. we ablate the design of visual and textual prompts utilized in our proposed methods. Overall, both TPPT-V and TPPT-VT show improved performance with deeper prompls, where prompts are embedded into more advanced layers Transformer blocks. Regarding the length prompts, extended visual prompts benefit the prediction accuracy for both methods, while the performance these methods appears relatively insensitive to the length of textual prompts.
Discussion computation cost. Similar to design, CODA-P, which also uses incremental visual prompts, introduces 5.68M parameters and reaches average final accuracy of 70.29. Without the need for additional learnable attention vectors, TPPT-V achieves 78.25% with only 4.3 M parameters, and TPPT-VT further elevates this to 80.07% with an extra 2.46 M parameters. More Details in Appendix Table 13
5
Conclusion
Our work introduces TPPT-V and TPPT-VT, which leverage textual prototypes to guide CLIP visual prompts, further aligning text and vision via learned textual prompts and diversity regularization, respectively. Extensive experiments show that our simple, elficient design substantially reduces forgetting while improving adaptation new tasks, representing notable advance in CL.
Limitations and Future Work. We introduce general supervision strategy for vision-language models, demonstrated mainly using CL.PP Our proposed mot is generally applicable to any models and set settings, and we leave evaluation on other architectures and experimental settings to future work.
Broader Impact Statement
This work represents fundamental research in machine learning, aiming to enable pre-trained foundation models to extend their knowledge base downstream tasks without compromising their original strong pre- trained knowledge. The proposed rescarch has the potential to be applied to various real-world applications, including computer vision systems, robotics, embodied AI, and autonomous driving. A polential negative social impact involves privacy concerns, as our methods utilize a replay buffer containing previous data samples, which may not be idcal for privacy-sensitive applications. However, this negative effect is minimal, our methods still perform well even under restricted memory sizes, thereby reducing the reliance on extensive data storage.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Acknowledgments
This work was partially supported by the ARC DECRA Fellowship (DE230101591) awarded to D. Gong. H. Lu affiliated with CSTRO Data61 through a PhD scholarship from CSTRO and acknowledges the support of the Google PhD Fellowship.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the Europeon conference on computer vision (ECCV), pp- 139- 154, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 11254-11263, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019b.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Vataanc-invarianccccovriii regularization for self-supervised learning. aiviv preprinl urXiv:2105.04.00 04906, 2021.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the Europeon conference on computer vision (ECCV), pp. 392-547, 2018a.
Arslan Chaudhry,
Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Xrxiv preprint 1xXi::181..04333 2018b.
Euirang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for rision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22004-22013, 2023.
Yawen Cui, Zitong Yu, Rizhao Cai, Xun Wang, Alex C Kot, and Li Liu. Generalized few-shot continual learning with contrastive mixture of adapters. arXic preprint arXiv:2302. 05936, 2023.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pottern analysis and machine intelligence, 44(7):3366-33) -3856,333,.-1 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, pp. 248 255, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiu preprint arXiv: 04805, 2018.
Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding, Don't stop learning: Towards continual Icarning for the clip model. arXiv preprint Xiw:2207.009.//4 2022.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11..1.1999... 2020.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Eilon-ECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 86-102. Springer, 2020.
Emanuele Frascaroli, Aniello Panaricllo, Pictro Buzzoga, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Clip with generative latent replay: a strong baseline for incremental learning. arXiv preprint 1rXiv:2407.15793, 2024.
13

---

Published in Transactions on Machine Learning Research (12/2025)
João Gama, Indré Zliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey concept drift adaptation. ACM compuling surveys (CSUR), 46(4):1-33 2014.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better lision-language models with feature adapters. International Journal of Computer Vision, pp. 1 15, 2023.
Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet. A survey on ensemble learning for data stream classification. ACM Computing Surveys (CSUR), 50(2):1-36, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sele-supervised learning. Advances in neural information processing systems, 33:21271 2222222422,4, 2020.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 831 839, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arx:2:2 06. 00.09655,, 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi. org/10. 1.00.5221//eoodo 51437733
Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npel: Neural processes for uncertainty- continual learning. arXiv preprint 2xXi::231011927: 2023.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. arXiv preprint orXiv:2408.19197 2024.
Menglin Jia, Luming Tang, Bor- Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:21100 10.
09348, 10.00:2110.09348,
2021.
Muhammad Uzair Khattak, Hanoona Rashoed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt. learning. Tn Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Nascer, Salman Khan, Ming- -suan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15190-15200, 2023b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Tsola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, and Honglak Lee. Transferring pre-trained multi- modal representations with cross-modal similarity matching. Advances in Neural Information Processing Systems, 14:30866-30833 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-35) -3226, 2017.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Jonathan Krause, Michael Stark, Jin Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In TCCV Workshop, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameler-efftcien prompt tuning. arXiv preprint ar 086911 2021.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXi preprint arXin:2005. (:Xi::205504466 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2944 2017.
Zhuowei Ti, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tuning for rehearsal-free continual learning. arXiv preprint or arxi::3033.094/1 2023.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototypes with prompt-tuning for rehcarsal-fis continual learning. In Proceedings of the IEEE/CVF Winler Conference on Ayplications of Compuler Vision, pp. 2523 2533, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638-2366 23647, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. orXiv preprint arXiv:2110.07608 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEFE/CVV conference on Compuler Vision and Pattern Recognition, pp.
2020.
David Lopez-Pat Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Haodong Lu, Dong Gong, Shuo Wang, Jason Xue, Lina Yao, and Kristen Moore. Learning with mixture of prototypes for dittrf-distribuution dotection. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://pperreriew..t htt.://wwoer.eeeeeeooooooeeaaaaa
Haodong Lu, Chongyang Zhao, Jason Xue, Lina Yao, Kristen Moore, and Dong Gong, Adaptive rank, reduced forgetting: Knowledge retention in continual learning vision- language models with dynamic rank-selective lora, 2025. URL htttps:/arxiv..rrtt wwrg/abs//444 2412. 2410401002..
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions knowledge and data engineering, (1(1))23346 2363, 2018.
Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun. Class-incremental exemplar compression for class- incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11371-11380, 2023.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocompuling, 469:28 51, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint (Xix:1306.5154 2013.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivalion, volume 24, pp. 109 165. Elsevier, 1989.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Aduances in Newral Information Processing Systems, 36, 2024.
Minderer, A Gritsenko, Stone, M Neumann, Weissenborn, A Dosovitskiy, Mahendran, A Arnab, M Dehghani, Z Shen, et al. Simple open-vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint orXiv:2205002200 2, 2022.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning, arXiv preprint arXiv: 1900.000911,,, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545 6554, 2023.
Sylvestre-Alvise Rebulfi, Alexander Kolesníkov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision ond Pattern Recognition, pp. 2001-2010, 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonille Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pottern Recognition, pp. 11909 11919, 2023.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficiet transfer learning for vision-and- language tasks. In Proceedings of the /CV//CVV Conference on Computer Vision and Pattern Recognátion, pp. 1227-5231 2022.
Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision- language models. In European conference computer vision, pp. 346-365. Springer, 2024.
Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan. Clip model is an efficient continual learner. arXiv preprint arXim:2210. 031114, 2022.
Ruilin Tong, Yuhang Liu, Javen Qinfeng Shi, and Dong Gong. Coresct selection via reducible loss continual learning. In The Thirteenth International Conference on Learning Representations, 2025. URL
16

---

Published in Transactions on Machine Learning Research (12/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report T2S-1110011-001, California Institute of Technology, 2011.
Fu-Yun Wang, Wa-Wei Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, and Peilin Zhao. Beef: Bi- compatible class-incremental learning via eveegy-bbseed expansion and fusion. In The Eleventh International Conference on Learning Representations, 2022a.
Fu- Yum Wang, Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In European conference computer vision, pp- 398 -11.. Springer, 2022b.
Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong, Self-expansion of pre-traines models with mixture of adapters for continual learning. arXiv preprint arXiv:2403. 1888882403.8886, 2024.
Runqi Wang, Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu, Shaohui Lin, Songcen Xu, Jinhu Lü, and Baochang Zhang. Attriclip: A non-incremental learner for incremental knowledge learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3654 3663, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through aligmment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35: 5682 5695, 2022c.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahiii, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference Computer Vision, pp. 631 648. Springer, 2022d.
Zifeng Wang, Zizhao Zhang, Yuen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual lcarning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp. 139 149, 2022e.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 374 382, 2019.
Qingsen Yan, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. Learning bayesian sparse networks with full experience replay for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 109 118, 2022.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2021.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guided context. optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757 6767, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the TEEE/CO /CF Conference on Computer Vision and Pottern Recognition, pp. 23219 23230, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Tn International conference on machine learning, pp. 3987-3995. PMLR, 2017.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free free clip-adapter for better vision- lasounlanguuae modeling. arXiv preprint 1rXiv:211.039900 2021.
Wenxuan Zhang, Paul Janson, Rahaf Aljundi, and Mohamed Elhoseiny. Overcoming generic knowledge loss with selective parameter update. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2024.
Pottern Recognition, pp.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. arXiv preprint arXiv:2303066628
2023.
Da-We Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning, arXiv preprint Xiv: 13218, 2022a.
Wei Zhou, Qi- Wang, Zhi-Hong Qi, Han-Jia Ye, De- Dh-Chuan Zhan, and Ziwei Liu. Deep ilass-incremental learning: A survey. arXiv preprint arXiv:2302.066, 036448 2023a.
Da- Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-increments learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint XxXiv23303.07338 2023b.
Da-Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without. forgetting for vision-language models. arXiv preprint ar 2rxiw 2023c.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816 16825, 2022b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lision-language models. International Journal of Compuler Vision, 130(9):2337-29)) 2348, 2022c.
Zhen Zhu, Weijie Lyu, Yao Xiao, and Derek Hoiem. Continual learning in open-vocabulary classification with complementary memory systems. arXiv preprint 230.22307.01430, 2023.
Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classification. In European Conference on Computer Vision, pp. 269 285. Springer, 2024.
18

---

Published in Transactions on Machine Learning Research (12/2025)
A More Details of the Proposed Method
A.1 Algorithm Details
We illustrate the algorithm details of TPPT-VT in Algorithm 1 The training pipeline of TPPT-V follows the same structure TPPT-VT, except that the textual features are obtained with the frozen text encoder without textual prompts, and the training objectives for TPPT-V are detailed in Equation
Algorithm 1: Training process of the proposed TPPT-VT
Input: Pre-trained image encoder fo: pro-trained text encoder Ie: number of tasks T, training epochs E, training set {{(x,, v/) visual prompts Po, linear layer for obtaining weights for visual prompts IN(.), textual prompts P.
Initialize: P., LN,,),,,,
for t= 1. 1........
......... T do for e e= 1,......
do
Draw a mini-batel for (x, E B do
[(x] v.)
Generate query feature ((x) and visual prompts aggregation weights via = LN(((x))
Aggregate the visual prompts P. for given x via Equation 2
Select the textual prompts p² corresponding to class y
Obtain prompted visual feature å and textual feature
Calculate per sample loss via Equation
Update P,,LN() Pt by backpropagation
A.2 More Discussions with Related Works
DIKT Tang et al. 2024) and CPP Li et al. 2024] tune prompts under primarily intra-visual supervision (calibration in DIKI; prototype steering and routing in CPP), which can leave cross-modal alignment under- constrained. TPPT anchors learning to CLIP text prototypes
sreed/leernable) and optimizes a cross-modal
TPCL with text-side diversity, stabilizing the image- text space. CLAPACLIP (Jha al. 2024 (adapters with probabilistic logit mixtures) and LoRA/MoF-Adapter variants (Liang Li 2024 Yu et al. 2024 struggle to maintain vision-text aligument across incremental learning stages, leading to suboptimal knowledge retention. SPU (Zhang et al.] 2024) selects subset of important MLP layers via gradient- gradient-based scoring and updates them directly, which introduces greater computational overhead than optimizing only a small set of prompt tokens. In contrast, TPPT keeps the backbone frozen and anchors lcarning to CLIP text prototypes (fixed and learnable), optimizing a cross-modal TPCL. with text-side diversity to stabilize the image text. embedding space, and TPPT outperforms these methods (Table 2 and Appendix Table I0].
TreeProbe (Zhu et al. 2023) pairs fixed CLIP zero-shot head with an exemplar probe and uses tree-style gating to choose between stored exemplars and pre-trained knowledge for opcn-vocabulary increments. AnytimeCL (Zhu et al. 2024) emphasizes update scheduling and memory compression to support single- sample, streaming updates. CGIL (Frascaroli et al. 2024) performs generative latent replay by training a VAE in CLIP's feature space to hesize past examples. These directions prioritize buffer usage/ /qualit,, scheduling/ compression, or
to our method, which targets representation learning via textual anchors and cross-modal alignment.
B Additional Experimental Details
We provide additional details of our experiments in this appendix section.
19

---

Published in Transactions on Machine Learning Research (12/2025)
B.1 Datasets
For thorough evaluation of our methodologies, compare the performance with previous methods across a variety of datasets. This includes three popular CIL datasets (Rebuffi et al. 2017 Wang et al. 2022c d 2009), Tiny MaagllmaeNNe as well as two widely used datasets used for prompt tuning research (Zhou et al. 2022c Yao et al.] 2023 Khattak et al.) 2023a bj:
Smith et al. )023): CIFARIO0 (Krizhevsky et al. 2009), ImageNet-R (Deng et al.
(L & Yang 2015). CUB200 (Wah et al.) 2011), and DomainNet (Peng et al., 2019
FGVCAircraft (Maji et al.) 2013) and Stanford Cars (Krause et al.) 2013). Each datasct is split into tasks to formulate a CIL setup. We hereby provide detailed description of these datasels.
CIFAR100 consists of 60,000 32x32 color images in 100 classes, with 600 images per class. There are 500 training images and 100 testing images per class. Given that classes across different tasks may belong to the same superclass, there is a potential overlap in similarities between lasks.
Image/Net-R] is a diverse dataset featuring renditions of ImageNet classes across various artistic and real- world domains. It includes images from categorics such as art, cartoons, deviantart, graffiti, embroidery, graphics origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, lattoos, toys, and video games. The dataset encompasses 200 classes, comprising a total of 24,000 training images and 6,000 testing images.
LinyLmageNett is a downsampled subset of ImageNet containing 200 classes, each resized to 64 x 64 pixels. It is widely used as a challenging benchmark for evaluating continual learning algorithms due to its larger class set and greater intra-class variability compared to CIFAR100.
CUB200 is designed for fine-graine grained visual categorization. It contains 11,788 images of 200 bird species, cach with detailed annotations. This datasct is particularly challenging due to the subtle differences between species.
FGVCAircraft is a specialized dataset for fine-grained visual categorization, focusing on aircraft models. It includes a variety of images depicting different aircraft models, presenting challenges in model identification. We randomly select 100 classes from this dataset.
Stanford Cars contains a collection of 16,185 images spread across 196 car classes. Each image is annotated with details such as the make, model, and year of the car. This datasct is primarily used for fine-grained car recognition tasks, especially useful for distinguishing between visually similar car models. Similar to FGVCAircraft, we use a random sample of 100 classes for model training and testing.
Nom is a large-scale multi-domain image dataset with roughly 0.6M images spanning 345 categories across six visual domains (clipart, infograph, painting, quickdraw, real, sketch). It is widely used for domain adaptation and continual learning benchmarks to assess scalability across many classes and heterogencous domains.
B.2 Compared Methods
In our experiments, we conduct a comprehensive comparison of our methods against a variety of previous dethods. Firstly, we assess the zero-shot performance of the pre-trained CLIP model (ZS-CLIP) [Raffrrd
et al. 2021 This is followed by comparisons with two widely recognized prompt tuning methods: CoOp [Zhou et al. 2022c) and VPT-CL (our implementation of VPT (Jia et al.) 2022 ) adapted to the CIL scenario.
Further, we compare against several popular CT. methods: T.2P (Wang et al. 2022e), DualPrompt (Wang]
et al. 2022d), CODA-P (Smith et al. 2023), AttriCLIP ( (Wang et al. 2023). ZSCL (Zheng et al.
2023
MoE-Adapter Yu ct al. 2024). InfLoRA (Liang & Li, 2024), SPU (Zhang et al.] 2024). DIKI (Tang et al.,
2024 CPP [Li et al. 2024), CLAPACLIP (Jha et al. 2024 and PROOF (Zhou et al. 2023c to provide a
comprehensive evaluation of our mothods.
ZS-CLIP. We use the pre-trained frozen image encoder and lext encoder to obtain visual features and textual features, and use textual features as a classifier (Equation [ to make predictions. Specifically, textual
features are obtained through hand-crafted prompt templates as suggested by OpenAI P with class names.
hhtps///gghubb.cm//eenn///CC
20

---

Published in Transactions on Machine Learning Research (12/2025)
CoOp. CoOp introduces method of prepending learnable prompt tokens to the text encoder of the CLIP model. This strategy is designed to adapt the pre-trained model for various downstream tasks. In our experiments, we adapt the CoOp design to fit the CIL scenario, which continually trains the textual prompts over a sequence of tasks.
VPT-CL. VPT is a popular prompt tuning method designed specifically for the Vision Transformer (ViT) model. In our study, we adapt VPT for use with the pre-trained CLIP model in a CIL context, and we name it VPT-CL. Specifically, discard the classification head of VPT and make predictions using textual features in ZS-CLIP, and adapt this implementation to the CIL scenario that continually trains the visual prompts.
L2P. L2P innovales by introdugigg pools of prompls specifically for ViT-based Clase-Incrementtal Learning. To remain consistent with their approach, we exclude the text encoder the pre-trained CLIP model in our implementation.
DualPrompt. The DualPrompt method integrates both global and task-specific visual prompts for CIL applications. In line with the original design, the text encoder from the CLIP model is not utilized in our adaptation.
CODA-P. CODA-P presents a novel approach by decomposing visual prompts into individual prompt components and learning instance- wise prompts. Consistent with their methodology, we also omit the text encoder from the CLIP model our implementation.
AttriCLIP. AttriCLIP enhances the textual branch of the CLIP model through the learning of text attribute prompts. These prompts are chosen from the proposed attribute bank to encode broader knowledge of textual information.
ZSCL. ZSCL tackles continual learning by fully fine-tuning the CLIP model on sequential tasks. To mitigate forgetting, it introduces an auxiliary reference dataset and distills knowledge from the pro-trained model back to the fine-tuned model, preserving prior knowledge.
MoE-Adapter. MoE- MoB-Adapter integrates mixture-of-experts (Mol)) mechanism into adapter layers enhance adaptability in continual learning. A router dynamically seleets multiple experts per layer based inpuls, enabling elficient adaptation while miligaling catastrophic forgetting.
InfLoRA. InfLoRA extends Low-Rank Adaptation (LoRA) for continual learning by incrementally introducing new LoRA modules per task while keeping the base model frozen. By ensuring the orthogonality of LoRA updates, it efficiently accommodates new lasks while mitigating catastrophic forgetting.
SPU. SPU updates a small, selected subset of backbone weights per task (typically in MLP layers) while freezing the rest. A sparsity mask (e.g., top-k by importance/ graadients)) identifies the parameters to update, limiting growth and preserving pre-trained knowledge; gradients computed for full layers, but only the selected weights are updated.
/
DIKI. DIKT is a parameter-efficienl CL method for CLIP that adds lightweight. residual modules and uses domain-aware colibration/regularizatee to reduce interference and retain pre-trained knowledge.
CPP. CPP maintains and steers image class mean prototypes with prompt tuning, and uses query-ko key routing to select (ask-specific prompts at inference, operating entirely in the visual space.
CLAP4CLIP. CLAP4CLIP proposes probabilistic fine-tuning of CLIP with adapters plus mixture/logit calibration across tasks; improves stability via uncertainy--waree combination of heads.
PROOF. Different from recent trends that mainly learn prompt tokens, PROOF focuses on learning projection layers. This method involves fusing the projections of visual and textual features and dynamically expanding these projections new tasks are introduced during training.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Forgetting measurements (lower is better) different methods. All methods are trained with identical exemplar sizes, utilizing the same pre-trained weights, and employing the CLIP model with a ViT-B/16 backbone. The best two forgetting measurements are marked in bold.
Forgetting
Methods
CUB
Aircraft
Cars
CoOp
18.47
14.01
4.76
VPT
18.08
32.73
3.78
CODA-P
18.67
31.44
5.38
PROOF
14.38
26.23
5.78
TPPT-V (Ours)
13.64
27.61
3.28
TPPT-VT (Ours)
13.11
25.17
3.34
Table 6: Average and final accuracy of different methods with different bulfer sizes. All methods are trained with the same pre-trained weights and employ the CLIP model with a ViT-B/16 backbone. A denotes the average test accuracy over incremental stages, AT denotes the test accuracy at the last incremental stage. The best and secondary performances are indicated in red and blue, respectively.
CIFAR100
ImageNet-R
CUB
Aircraft
Cars
Methods
Buffer size
À
Ar
Á
Ar
A
AT
A
Ar
A
Ar
CoOp
41.80
33.67
48.80
43.40
49.65
43.72
12.70
11.49
45.48
42.37
VPT-CL
65.63
51.96
67.25
69.53
73.70
64.38
33.60
21.84
90.59
87.58
CODA-P
74.20
54.63
60.35
46.28
70.37
57.34
28.48
21.21
89.46
85.05
5/class
PROOF
73.55
53.00
74.38
66.40
74.62
64.84
44.11
36.45
88.89
83.46
TPPT-V (Ours)
76.51
62.91
79.84
76.50
77.48
68.24
38.10
29.91
94.20
91.90
TPPT-VT (Ours)
78.74
70.18
74.78
70.28
79.29
70.65
41.98
31.94
90.83
90.86
CoOp
54.38
53.20
56.61
53.95
60.98
55.89
20.97
19.14
67.48
62.09
VPT-CL
72.70
62.74
80.25
76.92
77.35
67.18
38.55
26.19
92.02
89.53
CODA-P
79.90
65.66
68.41
56.48
77.54
67.47
36.90
33.72
93.33
89.79
10/class
PROOF
76.23
65.17
74.90
67.70
74.73
65.39
44.83
37.14
89.15
83.22
TPPT-V (Ours)
82.41
72.42
83.24
78.87
82.48
74.72
49.84
42.78
94.77
92.77
TPPT-VT (Ours)
78.53
70.20
79.45
74.70
82.17
75.49
53.04
45.03
95.90
94.49
CoOp
60.33
54.30
61.61
57.07
63.59
58.58
24.50
23.61
45.15
76.67
VPT-CL
76.92
67.06
69.42
76.40
79.18
69.40
40.64
28.53
93.06
89.73
CODA-P
82.78
70.12
72.50
61.50
79.69
69.55
44.52
42.75
94.59
92.07
15/class
PROOF
81.77
72.58
75.40
69.18
75.26
65.52
45.25
37.59
89.37
83.70
TPPT-V (Ours)
84.85
76.68
85.28
79.17
83.81
76.12
52.13
45.72
96.53
95.12
TPPT-VT (Ours)
82.21
75.27
82.01
76.62
84.04
76.59
57.95
50.50
95.70
94.11
CoOp
83.37
73.36
82.40
76.20
77.34
68.70
44.26
39.87
89.73
84.91
VPT-CL
87.11
78.21
84.60
78.97
80.81
69.89
35.52
25.50
93.71
91.49
CODA-P
83.53
72.20
74.45
65.88
83.36
76.12
54.51
45.06
96.16
92.17
20/class
PROOF
86.70
79.05
85.34
80.10
84.93
79.43
61.00
53.59
93.26
89.84
TPPT-V (Ours)
88.28
80.81
85.62
80.75
85.77
80.28
63.47
54.85
95.26
94.57
TPPT-VT (Ours)
87.20
78.98
84.87
81.02
86.60
81.55
69.34
61.99
96.83
95.37
22

---

Published in Transactions on Machine Learning Research (12/2025)
CO0A- VTVEAV
80 (0
Fiull Accuracy
Aireraft
CUR
Cars
Figure 10: Experiment results of different methods on 20-task split incremental setting.
Accuracy (%)
95
90
AMdnCLIP
TPPT-V
TPPT-VT
10
20
30
40
50
60
70
80
90
100
Number of Classos
Figure 11: Comparisons to AttriCLIP CIFAR-100 using ViT-L/14 with pre-trained weight from Ope- Radford et al. 2021
C More Experimental Results and Ablation Studies
Additional comparison with AttriCLIP (Wang et al., 2023). For fair comparison, compare our methods with AttriCLIP reported results under the same experiment settings (Wang et al. 2023 on CIFAR-100 in Fig. 11 The backbone model employed was CLIP with ViT-L/14 architecture, utilizing pre-trained weights from OpenAI. Throughout the incremental learning process, our approach demonstrated superior performance, consistently outperforming AttriCLIP.
Additional comparison with SPU (Zhang al., 2024). In Table 10 we reproduce SPU under our experimental setup, using the same memory bulfer construction method (iCaRL, consistent with baseline implementations), task splits, and training budget in the main experiments. Unlike TPPT, which applies parameter-efficiett (PEFT) soft prompts, SPU performs sparse parameter updates directly on the MLP layer, which requires gradient computation over the entire linear layer rather than only few trainable prompt tokens as in TPPT. Consequently, SPU can potentially allow more adaptation than PEFT modules, as reflected in its higher performance on CIFAR100 (a simpler, low-resolution dataset). fine-grained datasets with more diverse and complex representations, however, TPPT (especially TPPT-VT) shows significant gains over SPU, suggesting that learning with textual anchors more effective for continual learning of complex semantics. Morcover, regularized textual prototypes further boost performance to stato-of-tho-art levels (notably on Aircraft and Cars).
Comparisons on larger-scale DomainNet (Peng et al. 2019). We evaluate on DomainNet with 345 classes, splitting the label set into sequences of 5, 15, 30, and 69 tasks. As shown in Table 11 TPPT consistently outperforms prior prompt-based continual learning methods across all task sequence lengths,
Table 7: Upper-bound performance of our methods.
CIFAR100
Image.Net-R
CUB
Aircraft
Cars
TPPT-V
87.29
85.05
79.64
59.35
95.88
TPPT-VT
86.93
85.23
79.11
63.92
96.02
23

---

Published in Transactions on Machine Learning Research (12/2025)
Table Final Accuracies over 5 independent runs with varying order of seen classes
Methods
CIFAR100
ImageNet-R
CUB
Aircraft
Cars
TPPT-V 79.57 + 0.64
80.65 + 0.29
79.98 + 0.38
55.22 + 0.66
95.25 + 0.81
TPPT-VT
78.11 + 0.34
80.64 = 0.67
80.33 = 0.82
61.52 = 0.53
94.61 = 0.59
Table 9: Average and final accuracy of different methods using OpenAI (Radford et al. 2021 pre-trained weight. All methods are trained with identical exemplar sizes, utilizing the same pre-trained weights, and employing the CLIP model with a ViT-B/16 backbone. A demotes the average of test accuracy over incremental stages, Ar denotes the test accuracy at the last incremental stage. The best and secondary performances are indicated in red and bluc, respoctively.
CIFAR100
ImageNet-R.
CUB
Aircraft.
Cars
Methods
Ā
Ar
Á
Ar
A
Ar
A
Ar
A
AT
ZS-CLIP
74.47
65.92
79.34
72.37
74.77
64.67
33.71
23.01
83.78
76.11
CoOp
81.27
70.30
81.91
76.23
76.22
68.21
46.09
38.87
81.12
74.87
VPT-CL
83.95
75.74
85.27
79.93
77.82
67.22
50.39
38.25
88.31
82.37
L2P
79.35
69.82
76.84
68.95
77.00
68.16
55.21
43.79
82.49
77.48
DualPrompt
81.49
74.69
81.49
75.03
82.15
75.93
55.61
44.72
82.99
78.07
CODA-P
83.88
73.63
76.05
65.65
80.59
71.16
47.50
48.66
90.81
86.78
PROOF
84.97
77.30
83.66
78.31
83.83
78.28
57.09
50.44
88.29
82.66
TPPT-V (Ours)
83.88
77.85
86.12
80.85
85.32
77.01
62.50
52.75
92.06
87.97
TPPT-VT (Ours)
85.72
77.75
85.90
80.10
84.81
78.33
65.17
61.75
93.28
90.57
and the relative ad advantage is maintained as the number of tasks increases. Average and final accuracies remain stable rather than degrading with longer sequences, indicating that the approach scales larger class numbers and longer task sequences.
Discussions on pre-trained knowledge retention. The retention of pre-trained knowledge during and after continual learning has recently attracted growing attention (Zheng et al. 2023). Following prior work Zheng et al., 2023), we assess general knowledge by measuring zero-shot accuracy on unseon tasks and report these results across the CIFAR- 100 continual learning sequence (Table 12). Results show: (a) As continual learning progresses, unscen-task accuracy generally increases because the number of unseen tasks decreases, making classification easier as the number of classes shrinks. (b) For CODA-P, a prompt-tuning CL method related to incremental prompt framework, zero-shot performance relative ZS-CLIP (frozen CLIP), indicating severe degradation of general knowledge. (c) For TPPT-V, our static textual--rototype anchors guide CLIP's representation lcarning with learnable visual prompts, keeping visual representation clustered around pretrained text protolypes and thus preserving the integrity of the embedding space. We even observe a slight improvement over frozen CLIP, suggesting that visual prompts learned from carlier tasks help the foundation model generalize unseen tasks. (d) For TPPT-VT, we
unseen tasks drops significantly
Table 10: Comparisons with SPU Zhang et al.
2024
CIFAR100
CUB
Aircraft
Stanford Cars
TinylmageNe
Methods
A
Ar
A
Ar
A
Ar
A
Ar
A
Ar
ZS-CLIP
74.47
65.92
74.77
64.67
33.71
23.01
83.78
76.11
69.55
65.59
SPU
86.22
81.31
80.26
71.98
52.40
42.04
88.12
82.22
79.68
74.89
TPPT-V (Ours)
83.88
77.85
85.32
77.01
62.50
52.75
92.06
87.97
83.18
77.51
TPPT-VT (Ours)
85.72
77.75
84.81
78.33
65.17
61.75
93.28
90.57
82.41
78.08
24

---

Published in Transactions on Machine Learning Research (12/2025)
Accunss
Accomnsy
Accunmes
MCCP UI
20001
60
18
Namber
Nanber
(a) C1FAR100
ImageNet-RR
CUB
Autrrnay (80
10
Nomber Uloses
Nunter Closes
(d) Aircraft
(e) Stanford Cars
Figure 12: Porformance of different methods across incremental stages using pro-trained weights from OpenAI. All methods are trained with the same exemplar size using the same pre-trained weight and backbone model of CLIP with ViT-B/16.
Table 11: Comparisons on DomainNet.
Net
Methods
5 Tasks
15 Tasks
30 Tasks
69 Tasks
A
Ar
Ä
Ar
A
Ar
A
Ar
ZS-CLIP
63.94
56.66
63.31
56.66
63.66
56.66
63.28
56.66
CoOp
57.45
46.20
55.97
49.24
48.45
47.96
46.38
50.07
VPT-CL
68.70
59.42
70.42
62.78
68.74
61.24
71.84
62.49
CODA-P
68.04
59.09
70.68
61.94
69.72
61.75
70.45
61.25
TPPT-V
72.70
63.73
73.28
63.80
72.81
63.20
74.24
64.69
TPPT-VT
73.70
65.34
73.19
64.08
72.81
62.90
73.78
63.63
unfreeze textual prototypes with learnable text prompts to capture more representative linguistic semantics beyond pretrained features. This initially reduces unseen accuracy relative to TPPT-V (especially in the first few tasks) bocause the text embodding space being adjusted learn more representative prototypes. Notably, after Task 6, unseen accuracy slightly surpasses ZS-CLIP, indicating that TPPT-VT benefits from learning generalizable multimodal semantics over few continual learning stages.
Experiment results on Upper-bound performance. In complement to the incremental Icarning performance in Table 1 and Fig. 4 we further report the Upper-bound performance of our methods in Table 7, measuring the learning capacity of the model under a non-increnne non-incremental setting. Notably, for fine-grained datasets such as CUB, Aircraft, and Cars, the performance gap between the final accuracy reported in Table and the bouer- performance is marginal, highlighting the effectiveness of the introduction of textual prototypes, which greatly stabilizes the training during the incremental phases.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: Unseen-tase zero-shot accuracy across tasks. Deltas are w.r.t. ZS-CLIP at each step (+ improvement, degradation).
Tasks Seen
Methods
2
3
5
6
7
8
9
ZS-CLIP
66.61
68.14
68.41
69:12
71.44
73.85
75.90
80.15
85.00
CODA-P
28:30 (:38.1))
41.10
(:27.04)
47.33
(:21.08)
47.90 (2)
52.30
(-19.34)
55.90
(-18,05)
62.00 (-13.90)
70.50 (-9.63)
82.60 (-2.40)
TPPT-V
69.49 (+2.88)
72.58
(4-4.44)
73.06 (46625)
72.57 (-0.46)
73.44
(+2.00)
77.05 (43.60)
80.47 (44.87)
82.35
(42.20)
87.60 (4-2.00)
TPPT-VT
45.72 (:20,89)
58.08
(-10.06)
64.44
(-3.97)
69.90
(-0.78)
69.88
(-1.60)
75.45 (+1.00)
78.70 (+2.90)
81.35 (+1:20)
90.40 (+5.900
Studies on Forgetting. In Table 1 we present the average and final accuracies of our proposed methods, alongside comparisons with previous works. Additionally, we evaluate the average forgetting of our methods in Table 5 following (Wang et al. 2022eld /mmihh et al., 2023). We formally denote the forgetting metric for Ar,)) where Ar, represents the accuracy on task I after training on task We then evaluate average of forgetting over tasks F = 7 Ei= Fr. The results demonstrate that our methods not only consistently outperform the previous approaches but also exhibit low levels of forgetting, thus effectively preventing catastrophic forgetting. Benefited from our design of learning with textual prototypes, our method is inherently less prone to forgetting, Enriching textual prototypes with textual diversity not only enhances the stability and efficacy of the training process but also slightly improves the forgetting measure. While CoOp achieves lower forgetting on the Aircraft dataset, it is important to note the substantial difference in prediction accuracy favoring our methods. Given this considerable performance gap, our proposed mothods stand out as superior terms of overall efficacy.
a given task as: Fr = - (1..
Analyses on reply buffer. To provide comprehensive understanding, we explore the impact of different. buffer sizes on performance. These additional evaluations, including comparisons with previous works, are detailed in Table 6 The results from these experiments indicate that our proposed methods consistently outperform or are comparable to previous methods, demonstrating robustness irrespective of the chosen buffer size. A notable aspect of our methods is the effectiveness even with a limited number of exemplars. For instance, when using only 5 exemplars per task, TPPT-VT achieves an average accuracy of 79.29 on the CUB dataset. This performance is on par with that of previous works using a larger number of exemplars, which attain an average accuracy of 79.69 using 15 exemplars per task. Such results show the efficiency and competitiveness of our methods even under constrained exemplar conditions.
Experiment results with larger number of tasks. Evaluation results in Table 1 and Fig. were conducted under our default experimental setup, where each dataset is divided into 10 tasks for incremental learning. In Fig. 10 we extend our cvaluation to a more challenging experiment sctting by splitting cach dataset into 20 tasks, which increases the complexity of the learning process and makes the model more prone to forgetting. Despite the increased complexity, our methods, which leverage learning with textual prototypes and the diversity regularization of these prototypes, consistently outperform previous methods in the 20-task split setting.
Experiment results with varying class orders. In Table g the average final accuracy and standard deviation. Our method is robust to different class orderings, showing its robust performance across varied learning scenarios.
we perform 5 independent runs and report
Computational efficiency of our methods. In addition to the discussion of our parameters Sec. we hereby discuss the computation efficiency of our methods. In Table 13 we report per-image training/infert /inference time, peak GPU memory, and estimated FLOPs (GFLOPs/image and data pipelines, comparing TPPT with related contimual prompt-tuning methods. All experiments are run using one NVIDIA RTX3090 GPU. CODA-P uses similar design of incremental prompts to us, takes approximately Sms for training and 5ms for inference per image. TPPT-V takes a comparable or less amount of time, as it does not require additional learnable attention vectors, yet it delivers a significant performance boost of 11% in final accuracy averaged across all datascts compared to CODA-P. TPPT-VT additionally learns textual prompts and further increases final accuracy on the Aircraft dataset by 13%, with only an additional 1.5ms required for both training and inference compared to TPPT-V.
under identical hardware, batch size,
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 13: Detailed computation costs. We report per-image train/inference time (ms) and peak GPU memory
usage (MiB), and estimated FLOPS (GFLOPs/imago).
Train
Inference
Methods
FLOPs
Time
Memory
Time
Memory
ZS-CLIP
2.9
1210.17
33.71
CoOp
3.8
2171.6
3.2
1226.4
38.7
VPT
6.2
5928.0
2.9
1161.6
37.1
L2P
6.9
5963.3
2.9
1472.2
68.3
DualPrompt
7.9
5775.1
2.9
1578.4
68.8
CODA-P
8.2
7974.3
5.4
1653.7
68.1
TPPT-V (Ours)
8.2
7518.1
5.4
1335.6
68.1
TPPT-VT (Ours)
9.7
8821.0
5.9
1344.1
72.7
Table 14: Experiment results on SigLIP
(Zhai et al.
2023
CTFAR100
CUB
Aircraft
Methods
Ā
AT
A
AT
A
Ar
ZeroShot.
79.07
70.99
78.71
67.43
51.56
40.74
CoOp
77.97
69.44
75.97
64.29
48.06
39.63
VPT
85.99
77.90
80.23
69.80
51.51
46.28
CODA-P
86.69
79.42
86.11
80.36
71.33
69.43
TPPT-V (Ours
89.16
81.71
88.18
82.19
72.00
69.91
TPPT-VT (Ours)
88.99
81.64
87.90
82.40
74.81
71.08
Experiment results using OpenAl pre-trained CLIP weight. our main paper, we mainly report the performance using pre-trained weights from LAION-400M (Ilharco al. 2021). To demonstrate the robustness of our proposed methods, we further evaluate our performance using pre-trained weights from OpenAI (Radford et al. 2021 in Table g and illustrate the test accuracy incremental stages in Fig. 12 Our experiment results demonsttate that our proposed methods are insensitive to the selection of pre- pre-trained weights, consistently outperforming previous methods.
Specifically, TPPT-V, which employs static textual prototypes, demonstrates performance that is comparable or surpasses that of previous methods across evaluated datasets. This underscores the simplicity and effectiveness of our design. Notably, TPPT-VT that learns textual prompts to create regularized textual prototypes with diversity, leads superior performance across all metrics when compared to previous methods. In comparing beneficial for fine-grained datasets such as CUB, Aircraft, and Stanford Cars. This finding highlights the effectiveness of learning regularized textual prototypes. Such an approach is beneficial in preserving the maximum information of the datascts and promoting uniformity within the embedding space.
two proposed methods, observe that learning textual prototypes is particularly
Generalization to other VLMs. To further validate effectiveness, we evaluate TPPT on SigLIP (Zaai et al. 2023), whose pretraining differs from CLIP and yields stronger zero-shot performance. Despite these differences, TPPT, guided by stable textual anchors, consistently improves the SigLIP backbone across diverse downstream tasks Ccoarse-graines CIFAR100 and fine-grained CUB and Aircraft), achieving state-of-the-art results relative to other continual prompt tuning methods.
27


Reviews:
[{"review_id": "E9RijGE4qY", "reviewer": "Reviewer_YX6r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a continual learning (CL) method for the CLIP model, named **Textual Prototype-guided Prompt Tuning (TPPT)**. The method aims to adapt CLIP incrementally to new tasks while mitigating catastrophic forgetting. The main idea is to use CLIP’s textual prototypes (class text embeddings) as stable anchors to guide incremental prompt tuning.\n\nTwo variants are presented:\n- **TPPT-V:** Learns visual prompts guided by fixed textual prototypes, with a *textual prototypical contrastive loss* (LTPCL) that reduces representation drift.  \n- **TPPT-VT:** Jointly learns visual and textual prompts and introduces a *textual diversity regularization* (LDIV) to avoid feature collapse and improve uniformity.\n\nThe approach is conceptually simple, essentially prompt tuning with additional prototype-guided and diversity regularizations. Experiments on six datasets (CIFAR-100, TinyImageNet, ImageNet-R, CUB, Aircraft, Cars) show consistent improvements over previous prompt-based CL methods.\n\n**Strengths**\n- Straightforward and well-motivated design that leverages CLIP’s intrinsic textual stability.  \n- Strong empirical performance and extensive ablation studies (representation drift, diversity, prompt number).  \n- Clear writing and reproducible experimental setup.  \n\n**Weaknesses**\n- Limited conceptual novelty, the method mainly combines known techniques (prompt tuning, prototype contrastive loss etc.).  \n- Performance appears sensitive to hyperparameters such as α, prompt depth, and number of prompts per task.  \n- Experiments are limited to CLIP ViT-B/16; generalization to other architectures is unclear.\n- Despite good efficiency claims, the method still relies on replay buffers, which raises questions about scalability and fairness when compared to replay-free continual learning methods", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical results overall support the paper’s core claims that textual prototypes can anchor CLIP’s visual prompts to reduce representation drift, and that learning textual prompts with a diversity regularization can improve performance on fine-grained CL datasets. \n\nHowever, some claims could be more rigorously supported:\n- The paper frequently asserts that the method is “general” or “applicable to any model,” but all experiments are confined to CLIP ViT-B/16.  \n- The ablations, while extensive, still show sensitivity to hyperparameters such as the prompt depth and diversity coefficient α, which may limit reproducibility.  \n- The diversity regularization is motivated heuristically, with no clear theoretical backing for how it prevents collapse beyond empirical trends.  \n- Metrics focus on average accuracy; key continual learning measures like forward/backward transfer or forgetting are not explicitly reported.", "audience_yn": "Yes", "audience_explanation": "CL for VLMs is a timely and active research area. The paper fits well within TMLR’s readership. While the conceptual novelty is moderate, the work offers a clean empirical contribution: it demonstrates how textual prototypes can stabilize multimodal CL and improve fine-grained recognition without large computational cost.", "requested_changes": "- Explicitly state that the text prototype set at each incremental stage includes *only the classes seen so far*, to clarify compliance with the standard class-incremental protocol.  \n- Provide quantitative results for additional CL metrics (e.g., forgetting rate, backward transfer) to substantiate the claim of “reduced forgetting.”  \n- Provide a more intuitive explanation for how the diversity regularization improves stability beyond enforcing uniformity.\n- Evaluate the method on other VLMs backbone to test generality.  \n- Clarify the computational overhead comparison against other methods, including training and inference cost.", "broader_impact_concerns": "No major ethical issues identified. The work is methodological and low-risk. A brief note acknowledging potential bias inherited from CLIP pretraining and data reuse via replay buffers would suffice.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zzbXfZCYTY", "reviewer": "Reviewer_fq2A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary \nThis paper introduces TPPT (Textual Prototypical Prompt Tuning), a continual learning (CL) approach for CLIP that incrementally tunes prompts rather than the entire model. The method leverages textual prototypical contrastive loss alongside cross-entropy loss, aiming to increase diversity and reduce representational drift, thereby mitigating catastrophic forgetting. Two variants are proposed:\n\nTPPT-V: Fixes text features and uses contrastive loss.\nTPPT-VT: Extends TPPT-V by learning textual prompts and applying diversity regularization to prevent embedding collapse and improve retention.\n\nThe authors conduct comprehensive experiments on CIFAR100, ImageNet-R, etc demonstrating improvements over state-of-the-art methods,\n\n### Strengths\n\n- TPPT-VT excels on fine-grained continual learning benchmarks.\n- Both variants are useful, with effectiveness depending on dataset characteristics.\n- The method is conceptually simple yet effective, reducing the complexity of previous approaches.\n- Strong ablation studies validate the contributions of the proposed losses.\n\n### Weakness\n\n- The comparison lacks a baseline where the CLIP model is fine-tuned in an i.i.d. setting to see the upper bound of performance on that dataset\n- TPPT results for CIFAR100 differ between Table 1 and Table 2, with no explanation provided.\n- The main paper omits details about the buffer size used for reported results.\n- In the fine-tuning-based and adapter/LoRA-based method , sparse parameter fine-tuning methods (e.g., SPU[1]) are not compared, despite their relevance to the setting. \n- The paper reports lower values for ZSCL  on CIFAR100 compared to their original paper (e.g., ZSCL’s last accuracy is 80.1).\n- The paper does not address/discuss the zero shot performance degradation of CLIP’s general knowledge after continual finetuning as discussed in ZSCL[2]. \n\n[1 ]Zhang, W., Janson, P., Aljundi, R., & Elhoseiny, M. (2024). Overcoming generic knowledge loss with selective parameter update. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024.\n\n[2] Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., & You, Y. (2023). Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision 2023", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper effectively demonstrates the importance and relevance of the two proposed losses through controlled ablation studies. Additionally, the claims regarding representational drift and embedding diversity are clearly substantiated with supporting visualizations.", "audience_yn": "Yes", "audience_explanation": "The paper addresses catastrophic forgetting in CLIP, a widely used foundation model, making it of interest to researchers in continual learning, prompt tuning, and efficient fine-tuning.", "requested_changes": "See the weaknesses\n\nMinor: \n\n- The phrase \"By leveraging the powerful language modality\" in the prompt selection section is misleading. The method does not directly utilize language modality for prompt selection as far as I understand; instead, it leverages powerful pre-trained features for prompt selection.\n\n- The caption in Fig 1 (d) is incorrectly labeled as (b).", "broader_impact_concerns": "The broader impact statement is generally sufficient", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vQ9ID8DS89", "reviewer": "Reviewer_vrhc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary Of Contributions\nThe paper introduces Textual Prototype-guided Prompt Tuning (TPPT) for class-incremental learning (CIL) with CLIP. Two variants are proposed: TPPT-V, which uses fixed textual prototypes as anchors for visual prompt learning, and TPPT-VT, which additionally learns textual prompts with a diversity regularizer to avoid collapse. A new loss, Textual Prototypical Contrastive Loss (TPCL), is used to align visual prompts with textual prototypes while separating them from others. The methods are evaluated across multiple datasets, with reported gains over existing prompt-based baselines.\n\nKey Strengths\n- Clear and easy-to-follow writing.\n- A simple, conceptually intuitive idea: using textual prototypes as anchors.\n- Promising results, with evidence of robustness across different CLIP initializations.\n- Ablation studies on drift, diversity, and parameter counts.\n\nKey Weaknesses\n- Incremental novelty: TPCL is a straightforward adaptation of prior contrastive/prototypical losses, and TPPT overlaps conceptually with prior work (e.g., DIKI [A], CPP [B]).\n- Missing comparisons with several strong baselines (e.g., DIKI [A], LoRA-based methods, MoE-Adapter, CLAP4CLIP [C], CPP [B], TreeProbe [D], AnytimeCL [E], CGIL [F]).\n- Claims of efficiency are not supported: no FLOP, runtime, GPU memory, or inference-time comparisons are provided. Parameter counts alone are insufficient.\n- Experiments limited in scale: tested only up to ~200 classes, not on larger benchmarks like ImageNet-1k.\n- Method relies on replay buffers, but this is not clearly described in the main paper.\n- Technical clarity issues (e.g., missing details on aggregation weights, unclear symbols in equations, small and hard-to-read figures).\n\n[A] Tang, Longxiang, et al. \"Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models.\" ECCV2024\n\n[B] Li, Z., et al. \"Steering prototypes with prompt-tuning for rehearsal-free continual learning.\" WACV 2024.\n\n[C] Saurav Jha, et al. \"CLAP4CLIP: Continual learning with probabilistic finetuning for vision-language models.\" NeurIPS 2024\n\n[D] Zhu, Z., et al. \"Continual Learning in Open-vocabulary Classification with Complementary Memory Systems.\" TMLR 2024.\n\n[E] Zhu, Z., et al.  \"Anytime Continual Learning for Open Vocabulary Classification.\" ECCV 2024.\n\n[F] Frascaroli, Emanuele, et al. \"Clip with generative latent replay: a strong baseline for incremental learning.\" BMVC 2024", "claims_and_evidence_yn": "No", "claims_explanation": "The paper provides empirical improvements over prompt-based baselines and includes ablation studies that support some of the claims. However, the absence of critical runtime/efficiency analysis, limited benchmark coverage, and missing comparisons with stronger recent methods weaken the evidence. Furthermore, some design choices (e.g., handling of forgetting, scalability of objectives) are not sufficiently justified.", "audience_yn": "Yes", "audience_explanation": "Continual learning with large vision-language models like CLIP is an active and important area. The idea of using textual prototypes as anchors is simple and interpretable, and the reported results are promising. Even if the novelty is limited, the paper would likely interest researchers working on parameter-efficient continual learning.", "requested_changes": "*Critical*:\n- Include comparisons with closely related methods (e.g., DIKI, CPP, LoRA-based continual learning, CLAP4CLIP, MoE-Adapter, TreeProbe, AnytimeCL, CGIL).\n- Provide efficiency analyses beyond parameter counts: report training time, inference time, FLOPs, and GPU memory usage.\n- Clarify novelty relative to prior prototype-anchored prompt tuning methods (CPP, prototypical contrastive loss).\n- Provide more experimental evidence on larger-scale datasets (e.g., ImageNet-1k or MCIL benchmarks).\n\n*Strengthening* (not critical but recommended):\n- Improve clarity of exposition: define all terms and variables (e.g., N in Eq. 3), expand details on aggregation weights, and make figures/tables more readable.\n- Explicitly describe the reliance on replay buffers in the main paper, not just the supplementary materials.\n- Discuss the scalability of TPPT-VT with respect to memory growth when the number of classes becomes large.\n- Reposition title/claims more narrowly around CIL, since other continual learning settings are not explored.", "broader_impact_concerns": "No significant ethical concerns were identified. The method introduces no obvious risks beyond those typically associated with continual learning research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)