LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Authors: Mao-Lin Luo
Zi-Hao Zhou
Tong Wei
Min-Ling Zhang
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Vision-Language Model
Primary area: general_machine_learning->supervised_learning

Abstract: Continual learning with vision-language models like CLIP offers a pathway toward scalable machine learning systems by leveraging its transferable representations. Existing CLIP-based methods adapt the pre-trained image encoder by adding multiple sets of learnable parameters, with each task using a partial set of parameters. This requires selecting the expected parameters for input images during inference, which is prone to error that degrades performance. To address this problem, we introduce LADA (**L**abel-specific **ADA**pter). Instead of partitioning parameters across tasks, LADA appends lightweight, label-specific memory units to the frozen CLIP image encoder, enabling discriminative feature generation by aggregating task-agnostic knowledge. To prevent catastrophic forgetting, LADA employs feature distillation for seen classes, preventing their features from being interfered with by new classes. Positioned after the image encoder, LADA prevents gradient flow to the frozen CLIP parameters, ensuring efficient training. Extensive results show that LADA achieves state-of-the-art performance in continual learning settings.  The implementation code is available at [https://github.com/MaolinLuo/LADA](https://github.com/MaolinLuo/LADA).

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Mao-Lin Luo 12 Zi-Hao Zhou 12
Tong Wei Min-Ling Zhang 12
Abstract
fine-tuning (Gao al., 2024; Wortsman al., 2022a; Zhang et al., 2022; Wei et al., 2024; Gan & Wei, 2024; Shi et al., 2024). However, adapting CLIP representations using only task-specific data can severely impair the "general" knowl- edge encoded in the pre-trained CLIP parameters (Iuo et al., 2023). This challenge is particularly problematic in contin- ual learning settings, where the model must not only learn incrementally from series of tasks, but also retain its pre- viously acquired knowledge. As the model adapts to new tasks, the performance of previously learned knowledge often declines, a phenomenon known as catastrophic for- getting (French, 1999; McCloskey & Cohen, 1989). This issue highlights the trade-off between memory stability and learning plasticity: too much focus on the former can interfere with the latter, and vice versa (Wang et al., 2024b).
Continual learning with vision-language models like CLIP offers a pathway toward scalable ma- chine learning systems by leveraging its transfer- able representations. Existing CLIP-based meth- ods adapt the pre- pre-trained image encoder by adding multiple sets of learnable parameters, with each task using partial set of parameters. This re- quires selecting the expected parameters for input images during inference, which is prone to er- ror that degrades performance. To address this problem, we introduce LADA (Label-specific ADApter). Instead of partitioning parameters across tasks, LADA appends lightweight, label- specific memory units to the frozen CLIP image encoder, enabling discriminative feature gener- ation by aggregating aggostin knowledge. To prevent catastrophic forgetting, LADA em- ploys feature distillation for seen classes, pre- venting their features from being interfered with by new classes. Positioned after the image en- coder, LADA prevents gradient flow to the frozen CLIP parameters, ensuring efficient training. Ex- tensive results show that LADA achieves state- of-the-art performance in continual learning set- tings. The implementation code is available http://ggttubbccom/Mallinnoo/o/
Stability in continual learning is closely related to catas- trophic forgetting and can be divided into two aspects: for- getting newly learned tasks and forgetting pre-trained gen- eral knowledge. This dual forgetting phenomenon is charac- terized by forward forgetting, forgeettngg where the general knowledge in the pre-trained model degrades when making predictions on unseen tasks, and backward forgetting, where previously learned knowledge of seen tasks is lost (Tang et al., 2024). Although backward forgetting has been effectively mitigated through techniques such as regularization (Ahn et al., 2019; Zenke et al., 2017), prototype augmentation (Zhu et al., 2021; McDonnell et al., 2023; Zhang et al., 2023; Huang et al., 2024), and replay (Rolnick et al., 2019; Rebuffi et al., 2017), addressing forward forgetting remains significant challenge due to the unavailability of pre-training data. To mitigate forward forgetting. ZSCL (Zheng et al., 2023) dis- tills knowledge from a vanilla zero-shot CLIP model as a teacher into a fine-tuned CLIP model, using regulariza- tion to reduce forward forgetting. However, this approach still struggles to preserve the pre-trained knowledge due to potential updates pre-trained parameters. Furthermore, many methods (Yu et al., 2024; Zheng al., 2023) rely on external reference datasets, such as ImageNet and Concep- tual Captions, to maintain generalization on general tasks. However, this is often impractical in real-world scenarios (Tang et al., 2024; Xu et al., 2024).
1. Introduction
Pre-trained vision-language models, such as CLIP (Radford al., 2021). have become natural continual learners due to their ability to transfer representations across diverse tasks. Recently, several fine-tuning approaches for CLIP have been proposed to improve performance in downstream tasks, in- cluding full parameter line-tuning and parameter-efffcient
'School of Computer Science and Engineering, Southeast University, Nanjing 210096, China "Key Laboratory of Com- puter Network and Information Integration (Southeast University), Ministry of Education, China. Correspondence to: Tong Wei secittseu.edu.cn>.
For plasticity, full-parameter tuning methods, such as ZSCL (Zheng et al., 2023), distill features from vanilla CLIP to mitigate catastrophic forgetting. However, such
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
label-specifie memory
Jnbel-peciific features
Adapter
Adapter Selector
Adapter
Prompt Selector
Adapter 3
Transformer Image Encader
Transformer
Transformer
Image Encoder
Image Encoder
(a) Prompt-bane based Methods
(b) MoE-Adapiers
(c) Label-Specific Adapter (ours)
Figure 1. Comparison of CLIP tuning paradigms in continual learning. Our label-specific adapter leverages Icarned memory of all seen tasks and CLIP representations to generate label-specific features, climinating the need for paramcter selection.
approaches impede the model's ability to effectively learn new tasks, leading to suboptimal trade-off. Other prompt-based methods, such as L2P (Wang et al., 2022c), DualPrompt (Wang et al., 2022b) and S-Prompts (Wang et al., 2022a) expand the prompt pool as new tasks are learned, but their performance is limited by the constrained length of prompt tokens (Zheng al., 2023; Tang et al., 2024). MoE-Adee MoE-Adapters (Yu et al., 2024) inserts mixture of adapters into the image encoder to adapt CLIP representa- tions by activating a few adapters for each task. Although effective, the number of adapters must be predefined, re- quiring prior knowledge of the total number of tasks. More importantly, since only a partial set of parameters is ac- tivated during training for each task, both prompt-based methods and MoE- AdE-Adapters involve an auxiliary parameter selection step to extract image features using the expected prompts or adapters (as illustrated in Figure 1). This se- lection process can lead to misassignments, significantly reducing classification performance.
representation into label-specific features, eliminating the need for parameter selection seen in previous mainstream methods.
Our method is scalable and efficient, adding only small sets of learnable memory units for novel tasks, with- out requiring gradient propagation to the frozen CLIP image encoder.
We achieve tfate-of-the-ar results in both 16-shot and full-shot continual learning settings, surpassing previ- ous methods in Transfer, Average, and Last accuracy.
2. Related Works
Continual Learning Settings. Early continual learning research mainly considers Task-Incremental Learning (TIL) (Hsu, 2018), where models sequentially learn tasks and rely on task identities at inference time. Later, Class Ltass-Incremental Learning (CIL) (Rebuffi et al., 2017) removes this reliance, compelling models distinguish among all encountered classes without explicit task cues. Both TIL and CIL gen- erally assume that all tasks originate from single dataset (Yan et al., 2021; Zhu et al., 2021; Douillard et al., 2022), thus confining themselves to a single- -ogainn scenario. In contrast, Multi-domain Fackemanceetntal I Learning (MTIL.) (Zheng et al., 2023) introduces tasks drawn from diverse sources that each demand specialized domain knowledge from animal species to aircraft series, reflecting the complex- ity of real-world applications. Recently, Cross-domain Task- Agnostic Incremental Learning (X-TAIL) (Xu al., 2024) has taken one step further by discarding task or domain identity, pushing the setting closer to practical scenarios.
To address these problems, we introduce novel label- specific CLIP adapter, LADA, which appends compact set of learnable label-specific vectors after the CLIP image encoder. LADA handles new tasks by sequentially adding few label-specific vectors for each class, which act as cached memory units for seen classes. These vectors generate label- specific features by calculating their inner product with the CLIP representation. Memory units are learned by en- couraging higher activations of the calculated label-specific features at positions corresponding the ground truth label. LADA freezes the label-specific memory units of previ- ous tasks and only updates the memory for the new task, using both new task samples and the distilled features of previous tasks, reducing the risk of catastrophic forgetting. The design of LADA condenses discriminative information from all tasks, eliminating the need for parameter selection. Moreover, LADA is efficient for training as it does not re- quire gradient propagation to the CLIP image encoder. In summary, our contributions are as follows:
Continual Learning Methods. Prevailing continual learn- ing methods can be broadly classified into three cate- gories, including replay-based, regularization-base architecture-bate
and approaches based on surveys (De Lange et al., 2021; Wang et al., 2024b; Roth et al., 2024). Replay- based methods (Rebufli et al., 2017: Isele & Cosgun, 2018; Lavda et al., 2018; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Rolnick et al., 2019) store a subset of previous task data, which is replayed alongside new task data during train-
We introduce LADA, a lightweight CLIP adapter that condenses aask-agnostic knowledge to transform CLIP
2

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
ing mitigate catastrophic forgetting. These methods, such as Experience Replay (Isele & Cosgun, 2018; Rolnick et al., 2019) and Generative Replay (Lavda et al., 2018; Shin et al., 2017), aim to maintain the performance on prior tasks by revisiting old examples or generating pseudo-samples us- ing a generative model. While effective, the need to store or generate exemplars raises concerns regarding memory efficiency and scalability. Regularization based methods (Zheng et al., 2023; Ahn et al., 2019; Kirkpatrick et al., 2017; Jha et al., 2024; Zenke et al., 2017) introduce additional regularization terms in the loss function to mitigate catas- trophic forgetting, balancing between old and new tasks. One such method, ZSCL (Zheng et al., 2023), regularizes the model parameters by penalizing shifts in the parameter space, thus preserving the robustness of pre-trained mod- els even without direct access the original training data. Architecture- based methods (Wang et al., 2024a; Lu et al., 2024; Li et al., 2019; Houlsby et al., 2019; Yoon et al., 2018) enhance continual learning by expanding the model architecture, often by adding new parameters or components dedicated to specific tasks. prompt-based methods, such as L.2 (Wang et al., 2022c) and DualPrompt (Wang et al., 2022b), expand the prompt pool as new tasks are learned, re- lying on auxiliary losses. CODA-Prompt Prompt (Smith et al., 2023) proposes end-to-end prompt selection methods to increase plasticity. APG (Tang et al., 2023) introduces a prompt generator to reduce the gap between pretraining and future tasks, while EvoPrompt (Kurniawan et al., 2024) proposes an adaptive and continuous prompting approach to alleviate issues of selection mismatches and limited prompt share- ability. MoF-Adapters (Yu et al., 2024) insert a mixture of adapters into the image encoder, activating a few for each task. However, these methods typically involve an auxiliary parameter selection step during inference to extract image features using the expected prompts or adapters, which can lead to misassignments and degrade classification perfor- mance. Classifier- based methods, such as RAIL (Xu et al., 2024), extend the classifier dimension while keeping feature representations fixed. It relies on the vanilla zero-shot CLIP to distinguish whether task has been learned, which can lead to error propagation and degrade the performance.
C =( denotes the classnames of the task Thk
During inference at all steps, the learner attempts to classify input images from any task without the task-identity hint. Formally, the ground-truth label of the test image belongs to C= Cr U Cv. where C2, = U O³ is the union of seen class labels from all previous learning steps and Cy is the set of unseen class labels.
CLIP Model. The CLIP (Radford et al., 2021) model con- tains an image encoder Jr and a text encoder Jr. The training of the CLIP model is based on contrastive learn- ing, which aligns image and text features in shared latent space. This training paradigm enables CLIP to perform zero-shot classification by aligning image and text repre- sentations effectively. The zero-shot inference process of the CLIP model for image classification is as follows. First, each class j E [M*] of the task T is transformed into a sentence 1ÿ using a template "a photo of {c))." Then, the text encoder fr processes 12 to a text feature ty., given by fr (L)). Similarly, the image encoder fr processes the input image v, generating an image embedding i. In the following text, superscript numbers are used as task indices, while subscript numbers represent category indices. If a third index is required, it is enclosed in parentheses.
Text Encoder Fine-tuning Framework for Continual Learning. Given that CLIP is an efficient continual learner, it can achieve strong classification performance with only single classifier. Moreover, due to CLIP's strong text- image alignment capabilities, straightforward yet effective approach is to fine-tune the text encoder and extract text features from class names as the classifier. Specifically, we fine-tune the text encoder to extract optimized text features for the current task and concatenate them with the text fea- tures of previous classes to obtain the final classifier. We employ a simple mechanism to prevent catastrophic forget- ting. For the current task T the text features from the previous h - 1 tasks are kept frozen, while only the text features for 7 are updated by gradient optimization.
The objective of continual learning is to minimize the error not only in the current task Th but also across all previously learned - tasks. The optimization objective for fine- tuning the text encoder is formulated follows:
3. Method
3.1. Preliminary
k M²
L(t: = Σ
(t;k,,
(1)
Problem Setting. Cross-domain Task-Agnostic Incremental Learning (X-TAIL) (Xu et al., 2024) is defined as the sce- nario where given a pre-trained vision-languag model, the learner is required to incrementally learn K different tasks (T',7 _ T J. Each task T = =D* C(*) is available only during the k-th learning step, where D = U Dj with M¹ denotes the number of classes for task T and Dj denotes the samples of the j-th class for task Tk
The function L(t; () represents the classification loss for the j-th class of task T and is defined as follows:
((t(o))))))
E B
-log Σ Σ (f((t)))))) ne[k] m
(2)
.
3

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
For the current task k, the estimated loss k(t;k,k, is calculated as follows:
& + RM* xλ1 for task T from the original CLIP
image feature space:
elfi(m),t))
1 D
log Σ Σ e(ti(w)ea)))
go (t) = W i..,.... W W
M
(5)
VED
nE(k] me[[Wm]
Given all above =p, ,...... spt, we construct final feature rep- resentation by condensing the information across all the tasks (p()) = (i). p (i)]. It is noteworthy that as the number of tasks increases, the features obtained through the aforementioned method can naturally expand. This en- hances learning plasticity while ensuring memory stability.
For classes of previous tasks, the original images are longer accessible. To address this, we distill À cluster cen- ters, denoted as p)} (1)..) (2)), which represent distillation image prototypes in Dj. to compute the esti- mated loss C(t: k, i) as follows:
log (l.)))
Training of Label-Specific CLIP Adapter. Although the initialization process condenses and represents information from all tasks, does not necessarily guarantee that the extracted features exhibit optimal classification properties. Therefore, further fine-tune W to enhance the quality of the extracted features. To mitigate catastrophic forgetting, we adopt a simple mechanism by freezing W ...... w during fine-tuning on the current task T allowing only W to be updated.
1 A
(4)
Σ ().m)
(E( ME(M")
The total training loss is the sum of the current task loss in Eq. (3) and the distillation loss in Eq. (4), enabling joint optimization of both current task and past tasks.
Our text encoder fine-tune framework generally preserves the alignment between image and text features, achieving good classification performance, and improving training efficiency by avoiding backpropagation through the image encoder. However, it still struggles with the limited adapt- ability of image features, which poses a significant challenge in dynamically scaling the training parameters. To address this, we propose scalable label-specific CLIP adapter to enhance the adaptation capability of image features.
To enhance the classification performance of the extracted features, we first consider a fixed classifier h RB + RM where M = Ex= M' that maps above constructed feature to the corresponding classification logits space, for any i E [k] and j E [M*]
(h) (0)) = o oW)
(6)
where é o= exx - z)) is employed to convert the in- ner product into non-negative values, with 33 modulating the sharpness of the transformation and 1 is a column vector of all ones. Intuitively, the fixed classifier can be regarded as a nearest-neighbor classifier, which estimates the likelihood that a sample belongs to certain class by measuring the dis- tance between the sample's features and the representative features of the corresponding class.
3.2. Scalable Speel- CLIP Adapter
Constructing Lpbel-Specific Features. During the training stage, previous prompt- based methods and MdE-Adapters tend to learn features for specific tasks by selectively activat- ing a partial set of parameters. However, these features are not well aligned with previously learned tasks, necessitating the selective activation of parameters during the inference stage as well, which can lead to suboptimal performance. To address the above issues, we propose LADA, which condenses all task information into a unified representation space during the training stage. This approach eliminates the need for selective parameter activation during inference and helps prevent error propagation. Specifically, LADA aims to generate distinguishing features that capture the specific features of each task to facilitate its discrimination process. To achieve this, LADA employs clustering tech- niques that have been widely used as standalone tools to gain insights into the properties of data. Given the train- ing set DK = - D of the task T*, we extract all im- age features of DI and apply k-means clustering to obtain t cluster centers W E [ = (wf(1) (1)..... w (Ar)]- The retained cluster centers characterize the underlying structure of the corresponding class feature space. There- fore, we can construct the -aheifspecific features mapping
For the current task, we can directly minimize the following expected classification cross-entropy loss for each class j E [M to enhance the classification properties of the extracted features:
1 [D]
(ho))(fr(())
log
Σ
((ho)))))
VEDE
RETA mE(M")
Although the frozen parameters W W....., W remain un- changed, ensuring the stability of (h o pp) (i)} for E 11. which prevents catastrophic forgetting in previous tasks. However, the introduction of the new parameter T for the current task may lead to the misclassification of the old task samples into current task classes. Therefore, when fine- tuninunnng W it is crucial to maintain a clear distinc- tion between samples from previous tasks and the new task classes. By utilizing the distilled X2 cluster centers, denoted
4

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Comparison different continual learning methods on X-TAIL 16-shot each terms of Transfer. Average, Last scores (%). The best results are highlighted with bold style.
Caltech101
Aircraft
EuroSAT
Flowers
MNIST
Sun397
DTD
Food
Pets
Cars
Method
Average
Zero-shot
23.8
74.3
36.4
37.4
64.1
83.4
43.9
65.5
60.8
57.7
Transfer
LwF & Hoiem, 2017)
66.6
26.9
19.5
51.0
78.4
26.6
68.9
35.5
47.7
WiSE-FT (Wortsman et al., 2022h)
70.1
31.9
25.3
56.3
79.8
29.9
74.9
45.6
56.8
52.3
ZSCL (Zheng et al., 2023)
73.3
32.6
36.8
62.1
83.8
42.1
83.6
56.5
60.2
59.0
MoE-Adapters (Yu et al., 2024)
71.0
34.9
19.2
63.0
86.6
20.0
87.2
63.7
58.6
56.0
Ours
75.0
36.1
35.9
66.3
83.7
42.1
88.0
65.3
61.4
61.5
Average
LwF & Hoiem, 2017)
24.7
79.7
38.3
36.9
63.9
81.0
36.5
71.9
42.7
56.7
53.2
WiSE-FT (Wortsman el al., 2022b)
27.1
76.5
40.9
31.3
68.7
81.6
31.4
74.7
51.7
58.4
54.2
ZSCL (Zheng et al., 2023)
36.0
75.0
40.7
40.5
71.0
85.3
46.3
83.3
60.7
61.5
60.0
MoE- MoE-Adapters (Yu et al., 2024)
43.6
77.9
52.1
34.7
75.9
86.3
45.2
87.4
66.6
60.2
63.0
Primal-RA (Xu et al., 2024)
44.6
89.5
56.1
69.0
83.8
85.0
63.2
88.9
68.6
62.2
71.1
Dual-RAlL (Xu et al., 2024)
45.4
89.4
56.4
69.6
84.0
85.0
63.5
88.8
68.8
62.3
71.3
Ours
49.1
91.0
61.3
71.6
84.4
85.0
62.8
89.7
69.2
62.9
72.7
Last
LwF (Li & Hoiem, 2017)
20.9
83.1
47.5
38.2
75.55
84.7
50.1
78.0
75.8
74.6
62.8
WiSE-FT (Wortsman et al., 2022h)
21.8
76.8
42.9
20.8
77.5
84.9
30.7
76.6
75.8
72.5
58.0
ZSCL (Zheng al., 2023)
33.1
75.3
43.5
35.2
74.6
87.4
50.4
84.2
77.3
73.4
63.4
MoE-Adapters (Yu al., 2024)
43.2
78.7
57.6
32.8
79.4
86.0
86.7
87.8
78.2
74.2
70.5
Primal-RAIL (Xu et al., 2024)
44.2
94.6
66.8
85.9
96.3
86.8
91.6
91.5
80.6
75.4
81.4
Dual-RAIL (Xu et al., 2024)
45.5
94.6
68.6
87.7
97.2
86.9
92.8
91.4
81.9
75.9
82.3
Ours
49.6
93.7
69.3
86.9
96.7
86.9
93.8
93.7
84.6
76.0
83.1
as p (A2)) we minimize the following expected classification loss for each previous task < k and class j E [M*]
where is Gaussian noise with the same dimension as the prototype and the scale v Tr (x(())) controls the uncertainty
of the augmented prototypes.
1
(ho)))p3(
During the training stage, jointly optimize the text encoder and the LADA feature extraction module. Specifically, the cross-entropy loss is computed by summing the logits produced by the text encoder, which are derived from Eq. (3) and Eq. (4), and the logits produced by LADA, which are derived from Eq. (7) and Eq. (10). By integrating these two sets logits, the model is encour- aged to leverage both textual information and visual feature adaptations comprehensively. This training approach facili- tates an effective end-to-end learning process through their simultancous optimization.
Overall Framework.
log
A2
(1)))
ME mE[""
Distribution-Preserve Training, For previous tasks, dis- tilling only λ cluster centers p insufficient to preserve sufficient information of Dj. To better capture the underly- ing distribution, we fit Dj using Gaussian Mixture Model:
(T. E 20)2 GMM(D)).
(9)
Given the estimated parameters, we calculate the classifi- cation loss for task E and E [MC] using the augmented features:
During the inference stage, for all seen classes Cz. we uti- lize their text features as classification features, while for unseen classes Cv, the features are extracted using the vanilla text encoder and also serve classification features. These classification features concatenated for the final classification. If the predicted class belongs to the unseen classes Co. the text-visual classification result is used di- rectly. Otherwise, for classes in Ct., the final prediction is obtained by applying linear weighting between the logits produced by LADA and the corresponding text features.
da
(hop)
(1() log
(10)
Σ RE|H mE[M"]
where स represents the mixture weight of each component and p augmented prototypes defined follows:
Tr (0)
(1) -(1)=p((1) +
(1))
5

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Comparison different continual learning methods on X-TAIL full-shot for each task in terms Transfer. Average, Last scores (%). The best results are highlighted with bold style.
Caltech101
Aircraft
EuroSAT
Flowers
MNIST
Sun397
DTD
Food
Pets
Cars
Method
Average
Zero-shot
23.8
74.3
36.4
37.4
64.1
83.4
43.9
87.8
65.5
60.8
57.7
Transfer
LwF & Hoiem, 2017)
62.4
27.8
10.7
52.0
76.0
25.4
68.3
30.4
54.5
45.3
WiSE-FT (Wortsman et al., 2022h)
59.9
25.4
10.2
43.9
67.9
29.4
57.4
24.1
50.3
40.9
ZSCL (Zheng et al., 2023)
71.3
33.8
33.0
66.2
85,2
40.2
81.9
57.3
62.5
59.0
MoE-Adapters (Yu et al., 2024)
69.5
30.8
19.0
60.3
85.6
43.7
85.6
55.5
57.3
56.4
Ours
75.2
36.1
36.7
65.6
83.9
45.2
88.0
65.3
61.1
61.9
Average
LwF & Hoiem, 2017)
25.8
81.2
48.1
57.0
75.1
54.9
74.5
35.5
57.1
54.2
WiSE-FT (Wortsman el al., 2022b)
14.9
79.8
45.3
17.1
55.7
70.1
52.0
68.0
35.6
53.3
49.2
ZSCL (Zheng et al., 2023)
39.7
80.8
52.9
40.8
79.3
88.0
51.4
85.5
62.9
64.1
64.5
MoE- AdE-Adapeers (Yu et al., 2024)
52.4
79.4
57.7
42.7
81.1
86.6
64.8
86.7
61.3
59.0
67.2
Primal RAIL (Xu et al., 2024)
48.8
89.6
59.0
74.4
84.0
86.
65.6
89.5
68.9
62.3
72.8
Ours
53.9
93.6
66.6
78.0
85.3
86.7
65.2
89.9
69.7
62.7
75.2
Last
LwF (Li & Hoiem, 2017)
9.6
77.1
55.3
38.7
60.5
83.1
99.5
85.9
49.6
80.0
63.9
WiSE-FT (Wortsman et al., 2022h)
18.1
84.9
53.4
27.0
69.6
88.0
88.4
91.5
76.7
80.2
67.8
7SCL (Zheng et al., 2023)
33.8
80.4
60.2
31.1
85.8
91.3
80.4
93.7
84.9
79.0
72.1
MoE-Adapters (Yu al., 2024)
51.9
79.0
64.2
51.5
95.1
87.6
96.4
89.1
84.4
74.0
77.3
Primal-RAIL (Xu et al., 2024)
45.8
94.1
70.7
94.2
96.5
89.0
98.1
93.5
82.0
76.5
84.0
Ours
55.5
96.2
75.8
95.8
98.4
89.6
98.8
94.5
87.3
77.2
86.9
4. Experiments
defining the Transfer metric. The Average metric represents the mean accuracy across all time steps, offering holistic measure of stability and plasticity. Detailed definitions of these metrics are provided in Appendix A.
Benchmarks. We conduct experiments on the recently pro- posed X-TAIL (Xu et al., 2024) benchmark which consists of 10 image classification datasets: Aircraft (Maji et al., 2013), Caltech101 (Fei-Fei et al., 2004), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nils- back & Zisserman, 2008), Food (Bossard et al., 2014), MNIST (Deng, 2012), OxfordPet (Parkhi al., 2012), Stan- fordCars (Krause al., 2013), and SUN397 (Xiao al., 2010). Each dataset is treated task, and the benchmark includes total of 1,100 classes across the 10 tasks.
Implementation Details. We adopt the CLIP (Radford et al., 2021) model with ViT-B/16 (Dosovitskíy al.. 2021) image encoder. The training process is carried out using the Adam (Loshchilov & Hutter, 2019) optimizer, with a learning rate of 0.001 and a batch size of 64 across all tasks. For the primary experiments, we set the hyperparam- eters as A = 16 and X 4. All experiments of LADA are conducted on a single NVIDIA 4090 GPU. Additional im- plementation details for the fine-tuning of the text encoder are provided in Appendix B.
In addition to the 16-shot setting proposed by (Xu et al., 2024), in which 16 training samples per class were selected for each task, we also evaluate the benchmark under full- shot setting. This more realistic scenario maintains the original dataset distribution, with varying numbers of train- comprehensive
4.1. Main Results
ing samples across tasks, providing and challenging evaluation for continual learning methods.
We evaluate our method on both 16-shot and full-shot set- tings. The learning order is set alphabetically: Aireraft, Caltech101, DTD, EuroSAT, Flowers, Food, MNIST, Ox- fordPet, StanfordCars, and SUN397. Additional experi- ments in random order are provided in Appendix C. The performance averaged across the 10 tasks for our method and other baseline approaches the X-TAIL. setting are pre- sented in the Average column of both Table and Table 2. The Zero-shot indicates the zero-shot performance of the pre-trained CLIP model on each task.
Evaluation Metrics. To evaluate both stability and plastic- ity as discussed in Section 1, we employ the Transfer, Av- erage, and Last metrics from Zheng al. (2023). The Last metric assesses model performance after continual training, capturing plasticity and backward forgetting. To quantify forward forgetting, calculate the model's average accu- racy tasks I + 1, k +2, 2,... K after training on task k.
6

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Ablation study of LADA in 16-shot and full-shot settings.
Table 4. Impact label-specific dimension A and prototype num- ber A2 on TAIL full-shot setting. The asterisk (*) indicates the number of samples in some classes is fewer than 32. Time measured in seconds per batch (s/batch), Memory is measured in gigabytes (GB), and Params is measured in millions (M).
Settings
BF
DPT LADA
Transfer Average
Last
59.4
70.9
82.1
V
59.9
71.6
82.6
16-shot
61.2
72.3
83.0
V
61.5
72.7
83.1
A1
X
Transfer Average
Last
Time
Memory Params
59.6
73.2
85.6
8
60.9
74.7
86.7
0.280
17.84
4.51
V
60.3
74.4
86.3
8
4
61.7
75.1
86.7
0.287
18.14
4.51
Full-shot
61.1
74.9
86.8
8
16
62.2
75.1
86.7
0.318
19.05
4.51
V
61.9
75.2
86.9
16
61.4
75.1
86.9
0.281
18.01
9,01
16
61.9
75.2
86.9
0.289
18.51
9.01
16
62.3
74.8
86.9
0.330
20.62
9.01
In the 16-shot setting, our method outperforms the pre- vious best approach with 2.5% increase in Transfer ac- euracy, 1.4% increase in Average accuracy, and 0.8% increase in Last accuracy. In the full-shot setting, which represents more realistic and challenging scenario, our method achieves even greater gains over the baseline, with improvements of 2.9% in Transfer accuracy, 2.4% in Aver- age accuracy, and 2.9% in Last accuracy. These results indi- cate that our approach enhances both stability and plasticity, effectively learning knowledge from new tasks and preserv- ing them while retaining pre-trained knowledge. This is further supported by Figure 2, which provides a clear vi- sualization of the accuracy changes across all tasks over all learning steps. Notably, on datasets such as Flowers, Food, and SUN397, our method surpasses the vanilla zero- shot CLIP model in Transfer accuracy, indicating that it not only mitigates forward forgetting, but also enhances the retention of pre-trained knowledge. We further analyze the underlying reasons for this improvement in Section 4.5.
32
60.9
74.7
86.9
0.282
18.42
17.51
32
61.2
74.6
86.9
0.297
18.97
17.51
32
16
61.9
74.0
86.9
0.358
23.13
17.51
semantic drift introduced during text encoder fine-tuning. The DPT module further strengthens the representations of image prototypes by generating underlying distributions, leading to overall performance gains. This ablation study highlights the effectiveness of LADA in improving both stability and plasticity.
4.3. Analysis of LADA Dimension and Prototypes
We analyze the impact of different LADA dimensions and the number of distilled prototypes per class on both perfor- mance and computational cost in the full-shot setting, with results presented in Table 4.
Performance Impact. Adjusting the LADA dimension has minimal effect on performance metries, demonstrating the model's robustness to this parameter. Distilling a single feature as prototype per class, combined with DPT, is suf- ficient to prevent backward forgetting, reflected in the stable Last metric. However, increasing the number of pro- totypes improves distribution estimation for each class in previous tasks, reducing the risk of semantic feature space shifts caused by image prototype augmentation, which helps preserve pre- me-trained knowledge and prevents degradation in Transfer performance.
We do not present the performance of RAIL on the Transfer metric, as their reported results directly use the zero-shot accuracy of the vanilla CLIP model, which does not reflect the extent of forward forgetting on model after continual learning. Additionally, since Dual-RAlL needs store all features, it is not possible to complete training under the full-shot setting because of computational overhead and GPU memory constraint.
4.2. A Ablation Study
Computational Cost. We evaluate computational costs by measuring the time cost per batch for the final task (which involves the most sampled image prototypes), peak memory usage during continual learning across all K tasks, and the total parameter count of LADA. Since previously trained portions of LADA remain frozen, only 1/K of its total parameters are updated on average per task. Our results show that, under the same prototype settings, inereasing the LADA dimension has minimal impact on both time and memory costs. This is because LADA adjusts features after the backbone outputs, eliminating the need for back- propagation through the image encoder. When the LADA dimension is fixed, increasing the number of prototypes per
We analyze the contributions of our basic text encoder fine-tuning framework (BF), distribution-preservdd training (DPT), and LADA under both 16-shot and full-shot settings, with consistent findings presented in Table 3. The BF setup, which fine-tunes only the text encoder, achieves competi- tive baseline performance. However, does not effectively address plasticity and stability due to the absence of adapta- tion of image features and learned distribution preservation. Incorporating LADA improves both the Average and Last metrics by facilitating the learning of better -specficc feature representations. Additionally, LADA enhances the Transfer metric by mitigating forward forgetting, reducing
7

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Aircraft
Caltech101
DTD
EuroSAT
Flowers
96
50
88
60
75
90
80
50
75
30
72
30
25
910
Food
MNIST
OxfordPet
Cars
SUN397
80
80
90
80
72
87
60
64
60
40
56
1
2
3
5
6
910
0
1
910
2
5
6
7
910
2 3
4
910
0
910
ZSCL
AoE-Adapters
RAIL
LADA
Figure 2. Accuracy (%) changes all tasks all learning steps in the full-shot setting.
2.5
120
35
102
Impact of DPT. As shown in Table 3. incorporating DPT leads to moderate improvement in Transfer. This is be- cause DPT enhances the model's 's ability to better preserve the original feature distribution. Furthermore, shown in Table 4, increasing the number of image prototypes further improves Transfer performance. A higher number of pro- totypes enables more accurate estimation of embeddings from previous tasks, reducing interference with pre-traine pre-trained knowledge and mitigating feature space degradation.
3.0
Seledtr
28
59
(a) 16-shot setting
(b) Full-shot setting
Figure Comparison whether to use zero-shot CLIP as selec- tor to distinguish between seen and unseen classes. Without the selector, LADA performs better in the continual learning process it utilizes the learned knowledge to improve average task recall.
4.5. Why Transfer in Some Tasks Surpasses Zero-Shot CLIP Performance
In this section, we analyze why the Transfer metric in some tasks surpasses zero-shot CLIP performance. Figure 3 presents comparative analysis of whether to use vanilla zero-shot CLIP as a selector to distinguish between seen and unseen classes. The evaluation considers two key perfor- mance metrics per training step: task recall which measures the proportion of relevant samples successfully assigned to their respective tasks, and accuracy difference which measures the difference in classification accuracy between LADA and LADA with the selector. LADA achieves higher accuracy in continual learning without relying on vanilla zero-shot CLIP as selector. In contrast, the RAIL method (Xu et al., 2024) employs CLIP as a selector, restricting the trained model classifying only the seen classes.
class leads to moderate growth in time and memory costs. However, distilling a small number prototypes suffi- cient for strong performance, making this a cost-effective strategy for mitigating forgetting.
Moreover, the slight increase in time and memory costs with respect to parameters of LADA indicates that it can scale efficiently to accommodate more tasks continual learning, with only a gradual and accessible increase in resource consumption.
4.4. Impacts on Forward Forgetting
In this subsection, we analyze the impact of LADA and DPT on improving Transfer performance.
Our unified and straightforward method LADA eliminates the two-step process required by RAIL, reducing error prop- agation and fully leveraging learned knowledge to distin- guish between seen and unseen classes. Notably, our results provide insight into why some task transfers outperform zero-shot CLIP: Since task recall constrains the upper bound on classification accuracy, our method enhances differen- tiation between seen and unseen classes based on learned
Impact of LADA. As shown in Table 3, LADA achieves significant improvement in Transfer without the DPT mod- ule. By effectively capturing label-specific features, LADA mitigates semantic drift that typically arises from fine-tuning tunittning the text encoder. This reduction in semantic drift helps pre- serve previously learned knowledge, thereby minimizing forward forgetting and enhancing stability.
8

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
knowledge, thus improving task recall for unseen classes and ultimately improving their classification accuracy. Our experiments show that datasets such Flowers, Food, and Sun397, our method outperforms vanilla zero-shot CLIP on Transfer metric, as evidenced by Table and Table 2. In real-world continual learning scenarios, it is vital to leverage learned knowledge to enhance generalization performance. Our method effectively achieves this by providing reliable and efficient classification without the complexity of sepa- rate classification stages.
2014.
Chen, S., Ge. C., Tong, Z., Wang. J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in neural information processing systems (NeurIPS), 2022.
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2014.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence (TPAMI). 2021.
5. Conclusion
This paper introduces LADA, a novel CLIP adapter that transforms CLIP features into label-specific features by learning a compact set of vectors for each class. LADA maintains high discriminability between seen and new classes by training on both current task samples and dis- tilled representations from previous tasks. Unlike previous CLIP-based approaches, our method condenses all task in- formation into a unified representation space, climinating the need for auxiliary prompt selection, adapter selection, or reliance on vanilla zero-shot CLIP as an unseen class selector. Extensive empirical results show that method significantly enhances stability and plasticity in continual learning settings. Moreover, its efficiency enables seam- less scaling to more tasks and larger training sets while maintaining a moderate computational cost.
Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2012.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021.
Douillard, A., Ramé, A., Couairon, G., and Cord, M. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Acknowledgements
This work was supported by the National Science Foun- dation of China (62206049, 62225602), and the Big Data Computing Center of Southeast University. We would like to thank anonymous reviewers for their constructive sugges- tions.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In Conference on computer vision and pattern recognition workshop, 2004.
Impact Statement
French, R. M. Catastrophic forgetting in connectionist net- works. Trends cognitive sciences, 1999.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gan, K. and Wei, T. Erasing the bias: fine-tuning foundation models for semi-supervised learning. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
References
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y.. Li, H., and Qiao, Y. Clip-adapter: Better vision vision-languee language models with feature adapters. International journal of computer vision (IJCV), 2024.
Ahn, H., Cha. S., Lee, D., and Moon, T. Uncertainty-basee continual learning with adaptive regularization. Advances in neural information processing systems (NeurIPS), 2019.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of selected topics in applied earth observations and remote sensing, 2019.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- -010 mining discriminative components with random forests. In European conference of computer vision (ECCV),

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nip. In International conference on machine learning (ICML), 2019.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations (ICLR), 2019.
Lu, H., Zhao, C., Xue, J., Yao, L.., Moore, K., and Gong, D. Adaptive rank, reduced forgetting: Knowl- edge retention in continual learning vision-language mod- els with dynamic rank-selective lora. arXiv preprint arXiv:2412.01004 2024.
Hsu, Y. Re-evaluating continual learning scenarios: A cate- gorization and case for strong baselines. arXiv preprint 1rX1v:1810.12488, 2018.
Luo, Y., Yang. Z.., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning arXiv preprint arXiv:2308.08747 2023.
Huang, L., Cao, X., Lu, H., and Liu, X. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. In European Conference on Com- puter Vision (ECCV), 2024.
Maji, S., Rahtu, E.,, Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151/ 2013.
Isele, D. and Cosgun, A. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2018.
McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning prob- lem. In Psychology of learning and motivation. Elsevier, 1989.
Jha, S., Gong, D., Zhao, H., and Yao, L. Npel: Neu- ral processes for uncertainty-aware continual learning- Advances in Neural Information Processing Systems (NeurIPS), 2024.
McDonnell, M. D., Gong, D., Parvaneh, A., Abbasnejad, E., and Van den Hengel, A. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barne Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, 2008
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine- treeggraied categorization. In Proceed- ings of the IEEE international conference on computer vision workshops. 2013.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C.
Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition (CVPR), 2012.
Kurniawan, M. R., Song, X., Ma, Z., He, Y., Gong, Y., Qi. Y.. and Wei, X. Evolving parameterized prompt memory for continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning (ICML), 2021.
Lavda, F., Ramapuram, J., Gregorova, M., and Kalousis, A. Continual classification learning using generative models. arXiv preprint arXiv: /0 2018
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on computer vi- sion and pattern recognition (CVPR), 2017.
Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong. C. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International con- ference on machine learning (ICML), 2019.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learn- ing. Advances neural information processing systems (NeurIPS), 2019.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE Iransactions on paltern analysis and machine intelligence (TPAMI), 2017.
Roth, K., Udandarao, V., Dziadzio, S., Prabhu, A., Cherti, M., Vinyals, O.,, Hénaff, O., Albanie, S., Bethge, M., and Akata, Z. A practitioner's e's guide to continual multimodal pretraining. arXiv preprint arXiv:2408. /4471, 2024.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
10

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Shi, J.-X., Wei, T., Zhou, Z., Shao, J.-J., Han, X.-Y., and Li, Y.-F. Long-tail learning with foundation model: heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning (ICML), 2022a.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural informa- tion processing systems (NeurIPS), 2017.
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P.,
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tum tuning of zero-shot models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition (CVPR), 2022b.
Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2023.
Xiao, J., Hays, J.. Ehinger. K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. IEEE computer society conference on computer vision and pattern recognition, 2010.
Tang, L., Tian, Z., Li, K., He, C., Zhou, H., Zhao, H., Li, X., and Jia, Mind the interference: Retaining pre-traing traitea knowledge in parameter efficient continual learning of vision-language models. In European conference on com-
puter vision (ECCV), 2024.
Xu, Y., Chen, Y., Nie, J., Wang. Y., Zhuang, H., and Oku- mura, M. Advancing cross-domain discriminability in continual learning of vision- lision-language models. Advances neural information processing systems (NeurIPS), 2024.
Tang, Y.-M., Peng, Y.-X., and Zheng, W.-S. When prompt- based incremental learning does not meet strong pretrain- ing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), 2023.
Wang, L., Xie, J., Zhang, X., Huang, M., Su, H., and Zhu, Hierarchical decomposition of prompt-based based continual learning: Rethinking obscured ootioaattmllty....m Advances in Neural Information Processing Systems (NeurIPS). 2024a.
Yan, S., Xie, J., and He, X. Der: Dynamically expandable representation for class incremental learning. Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2021.
Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. In Inter- national Conference on Learning Representations (ICLR), 2018.
Wang, L.. Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024b.
Wang, Y. Huang, Z., and Hong, X. S-prompts learning with pre-trained transformers: occam's 's razor for domain incremental learning. Advances in neural information processing systems (NeurIPS), 2022a.
Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., and He, Y. Boosting continual learning of vision-language models via mixture-of-expeer adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision (ECCV), 2022b.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International conference on machine learning (ICML), 2017.
Zhang, G., Wang. L., Kang, G., Chen, L., and Wei, Y. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H.,, Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition (CVPR), 2022c.
Wei, T., Li, H.-T., Li, C., Shi. J.-X., Li, Y.-F., and Zhang, M.-L. Visian-language models are strong noisy label detectors. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Zhang, R., Zhang, W., Fang, R., Gao, Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training- free adaption of clip for few-shot classification. In European conference on computer vision (ECCV), 2022.

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., and You, Y. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2023.
Zhu, F., Zhang, X.-Y., Wang, C.,, Yin, F., and Liu, C.-L. Pro- totype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
12

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
A. Evaluation Metrics
define the Transfer, Average, and Last metrics to evaluate model performance under continual learning scenarios. Let represent the accuracy of the model on task after training on task j. The Transfer, Average, and Last metrics for task A are computed as follows:
k-1 h:
Transfer
Σ
()
k 2.3....... .,
(12)
j=1
K
Average;
Σ
k
(j)
k= 1.1,,,..,, K,
(13)
K
f=1
(k) Last, oz
k= L..,..,KKKK
(14)
Here, K denotes the total number of tasks. The Transfer metric evaluates the model's ability to retain zero-shot capability by calculating its average accuracy on future tasks (j 1. + 2...., K) after training task j. The Last metric measures the model's final performance on each task after completing all training, thereby quantifying the extent of backward forgetting. Finally, the Average metric represents the mean accuracy across all time steps, providing a comprehensive view of the model's performance throughout the entire continual learning process.
B. Additional Implementation Details
To mitigate disruptions to the text encoder and enhance training efficiency, adopt the parameter-efficient fine-tuning method, AdaptFormer (Chen et al., 2022). This approach allows for the fine-tuning of only small subset of parameters, while keeping the text encoder backbone frozen. The AdaptFormer operates as follows:
X ((ReLU (LN (X) W down)) W up + x)-ss, -S,
(15)
where X represents the output of the multi-head head attention in transformer, W down and W., up are learnable projection matrices, and s a learnable scaling factor that controls the impact of each layer on the backbone. AdaptFormer adopts parallel fine-tuning with residual connections from the transformer feedforward network output, enabling minimal interference with the original feature spaces while effectively adapting the current task.
C. Comparison of different methods on X-TAIL with order II.
In this section, we compare different methods within the X-TAIL 16-shot setting and full-shot setting using random order: StanfordCars, Aircraft, OxfordPet, Food, SUN397, MNIST, Flowers, DTD, Caltech101. and EuroSAT. As shown in Table 5 and Table 6, our method consistently outperforms previous approaches across all metrics, further supporting the conclusions drawn Section 4.1.
D. Aditionaa X-TAIL Results
In this section, we present the per-training-step accuracies for both the 16-shot and full-shot settings, under Order-1 and Order-Il, in Tables 7 to 10. These results demonstrate strong performance in terms of both learning plasticity and memory stability.
13

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Comparison of different continual learning methods X-TAIL 16-shot for each task with order-Il in terms of Transfer, Averuge, and Last scores (%). The best results are highlighted with bold style.
Aircraft
SUNN997
MNIST
Flowers
Caltech 101
EuroSAT
Cars
Pets
Food
DTD
Method
Average
Zero-shot
65.5
23.8
87.8
83.4
60.8
43.9
64.1
36.4
74.3
37.4
57.7
Transfer
LwF & Hoiem, 2017)
20.0
74.1
79.6
58.1
48.9
27.7
64.4
15.1
46.9
WiSE-FT (Wortsman el al., 2022b)
21.3
79.5
83.3
61.0
39.9
56.5
29.6
68.0
20.8
51.1
ZSCL (Zheng et al., 2023)
23.0
84.3
87.2
63.0
42.1
65.2
34,6
71.4
40.9
56.9
MoE-Adapters (Yu et al., 2024)
17.1
87.2
87.5
58.4
12.6
65.5
35,9
70.0
17.9
50.2
Ours
23.8
87.8
84.3
61.3
42.2
65.7
37.2
71.9
36.0
56.7
Average
LwF & Hoiem, 2017)
49.0
27.4
69.7
83.0
65.7
42.2
63.5
68.5
17.5
52.0
WiSE-FT (Wortsman et al., 2022b)
57.9
29.6
77.8
85.4
68.0
51.6
69.3
35.5
71.0
23.0
56.9
7SCL. (Zheng et al., 2023)
74.4
36.4
86.7
88.7
68.9
50.0
75.1
40.1
72.5
43.7
63.6
MoF-Adapters (Yu et al., 2024)
74.4
38.6
87.7
87.3
67.9
50.6
76.5
43.7
72.3
18.8
61.8
Primal-RAlL, (Xu al., 2024)
81.1
43.0
91.1
86.6
69.0
59.2
76.7
45.6
78.4
42.3
67.3
Dual-RAIL (Xu al., 2024)
82.3
85.6
59.5
77.1
78.4
67.5
Ours
84.7
46.2
92.2
86.4
70.0
68.0
77.4
47.2
75.9
41.2
68.9
Last
LwF (Li & Hoiem, 2017)
29.6
17.5
63.0
83.8
67.7
44.9
79.3
44.8
84.6
39.0
55.4
WiSE-FT (Wortsman et al., 2022b)
46.1
23.5
71.3
85.7
70.2
59.1
85.5
47.9
82.4
42.8
61.5
ZSCL (Zheng al., 2023)
71.7
35.3
86.5
89.2
71.8
52.3
89.8
52.0
77.1
68.4
69.4
MoE-Adapters
al., 2024)
75.1
87.9
74.1
89.7
92.6
61.2
81.0
27.4
71.7
Hhimal-RAlL (Xu et al., 2024)
80.8
45.0
92.3
86.7
75.1
91.8
96.4
68.2
94.6
86.9
81.8
Dual-RAW RaI- (Xu et al., 2024)
82.3
46.3
92.2
86.9
75.7
92.6
97.4
69.0
94.7
88.3
82.5
Ours
84.8
49.2
93.6
87.6
76.6
93.9
97.4
70.7
91.7
87.4
83.3
Table Comparison of different continual learning methods on X-TAIL full-shot each task with order- in terms of Transfer, Averuge, and Last scores (%). The best results are highlighted with bold style.
Caltech101
Aircraft
SUN397
MNIST
Flowers
EuroSAT
Cars
Pets
Food
DTD
Method
Average
Zero-shot
65.5
23.8
87.8
83.4
60.8
43.9
64.1
36.4
74.3
37.4
57.7
Transfer
LwF (Li & Hoiem, 2017)
16.7
78.5
76.0
59.7
41.3
46.6
27.3
63.3
10.4
46.6
WiSE-FT (Wortsman et al., 2022h)
20.3
77.9
75.7
55.6
39.6
45.0
25.4
58.9
8.3
45.2
ZSCL (Zheng al., 2023)
21.7
83.2
85.6
63.0
39.3
61.8
34.3
72.2
26.4
54.2
MoE-Adapters (Yu al., 2024)
17.5
87.1
86.8
58.2
44.2
63.4
33.9
67.9
15.3
52.7
Ours
23.8
87.8
84.3
61.4
41.6
65.7
34.5
71.0
28.9
55.4
Average
LwF (Li & Hoiem, 2017)
38.6
21.6
73.2
75.9
67.7
69.4
62.2
36.8
68.0
15.1
52.9
WiSE-FT (Wortsman et al., 2022b)
47.1
30.9
77.9
76.6
65.0
59.0
58.7
36.1
65.4
9.8
52.7
ZSCL (Zheng et al., 2023)
77.8
43.1
90.3
89.5
71.8
61.3
73.6
42.9
74.2
26.8
65.1
MoE-Adapters (Yu et al., 2024)
84.2
47.4
89.0
88.0
65.2
70.7
76.3
43.4
71.5
14.8
65.1
Primal-RAIL (Xu et al., 2024)
83.0
45.7
92.1
70.0
61.6
76.8
46.4
78.3
68.4
Ours
87.0
49.3
93.1
88.0
70.9
66.8
79.0
46.8
76.0
35.6
69.2
Last
LwF (Li & Hoiem, 2017)
13.4
6.9
58.2
74.5
71.2
99.4
76.3
53.2
85.4
57.0
59.6
WiSE-FT (Wortsman et al., 2022b)
19.3
2.0
70.5
77.6
67.9
72.1
66.2
52.7
90.6
22.7
54.2
ZSCL (Zheng al., 2023)
75.6
31.7
90.4
90.7
77.0
75.4
88.3
60.5
82.1
29.6
70.1
MoE-Adapters (Yu al., 2024)
84.1
50.6
88.9
88.1
68.7
97.2
95.5
65.6
86.2
10.3
73.5
Primal-RAll (Xu et al., 2024)
81.9
46.1
93.3
89.0
76.6
98.2
96.6
70.6
94.1
94.1
84.1
Ours
87.0
55.4
94.5
89.6
77.7
98.8
99.0
75.8
95.8
95.8
86.9
14

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Accuracy of Ours X-TAIL. 16-shot with order-I. row represents the performance on every dataset of model trained after the corresponding task.
Transfer
Average
Last metries are shown.
Caltech101
Aircraft
EuroSAT
Flowers
MNIST
SUN397
DTD
Food
Pets
Cars
Transfer
75.0
36.1
35.9
66.3
83.7
42.1
88.0
65.3
61.4
61.5
Aircraft
48.6
75.0
36.4
37.4
64.1
83.4
43.9
87.8
65.5
61.1
10
49.1
91.7
35.7
37.1
67.2
83.9
44.0
88.0
65.3
61.4
DTD
49.1
92.5
66.7
33.1
67.0
83.7
44.5
88.0
65.3
61.2
EuroSAT
49.1
66.7
67.0
83.7
40.1
65.3
Flowers
49.1
92.7
66.8
86.9
96.3
83.7
40.1
88.0
65.3
61.5
Food
49.1
92.7
67.8
86.9
96.4
86.1
40.1
88.0
65.3
61.5
MNIST
49.1
92.9
67.8
86.9
96.4
86.2
93.8
88.0
65.3
61.5
Pets
49.1
93.0
67.9
86.9
96.4
86.2
93.8
65.3
61.5
Cars
49.1
93.2
67.9
86.9
96.4
86.2
93.8
93.5
84.6
61.6
SUN397
49.6
93.7
69.3
86.9
96.7
86.9
93.8
93.7
84.6
76.0
83.
Average
49.1
91.0
61.3
71.6
84.4
85.0
62.8
89.7
69.2
62,9
72.7
Table Accuracy on the X-TAIL full-shot with order-I, Each row represents the performance on every dataset model
trained after the corresponding task.
Transfer
Average
and
metries are shown.
Caltech101
Aircraft
EuroSAT
Flowers
MNIST
SUN397
DTD
Food
Pet
Cars
Transfer
75.2
36.1
36.7
65.6
83,9
45.2
88.0
65.3
61.1
61.9
Aircraft
51.7
75.2
36.4
37.4
64.1
83.4
43.9
87.8
65.5
61.1
1a
53.9
95.2
35.7
37.3
66.3
84.2
44.0
88.0
65.3
61.2
DTD
53.9
95.3
73.2
35.5
65.9
84.0
45.9
88.0
65.3
60.9
EuroSAT
53.9
73.2
65.9
84.0
46.4
88.0
65.3
61.1
Flowers
54.0
95.7
73.7
95.8
98.3
84.0
45.6
88.0
65.3
61.2
Food
54.0
95.7
74.6
95.7
98.4
89.4
45.6
88.0
65.3
61.1
MNIST
54.0
95.7
74.6
95.7
98.4
89.4
84.7
88.0
65.3
61.1
Pets
54.0
95.7
74.5
95.7
98.4
89.4
98.4
94.4
65.3
61.0
Cars
54.0
96.1
74.5
95.7
98.4
89.4
98.8
94.4
87.3
60.9
SUN397
55.5
96.2
75.8
95.8
98.4
89.6
98.8
94.5
87.3
77.2
86.9
Average
53.9
93.6
66.6
78.0
85.3
86.7
65.2
89.9
69.7
62,7
75.2
15

---

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Table Accuracy Ours on the X-TAIL. 16-shot with order-II, Each row represents the performance on every dataset model
trained after the corresponding task.
Transfer
Average
and metries are shown.
Caltech101
Aircraft
SUN397
MNIST
Flowers
EunSAT
Cars
Pets
Food
DTD
Transfer
23.8
87.8
84.3
61.3
42.2
65.7
37.2
71.9
36.0
56.7
Cars
84.7
23.8
87.8
84.3
61.1
43.9
65.7
36.4
72.5
37.4
Aircraft
84.7
48
87.8
84.3
61.4
43.9
65.7
36.4
73.1
37.4
Pets
84.7
48.4
93.2
84.3
61.4
41.0
65.7
36.4
72.8
37.4
Food
84.7
93.2
61.4
41.0
65.7
73.3
SUN397
84.7
48.6
93.3
87.1
75.3
41.0
65.7
37.6
70.3
35.3
MNIST
84.7
48.6
93.3
87.1
75.3
93.8
65.7
37.6
70.3
35.5
Flowers
84.7
48.7
93.3
87.1
75.4
93.8
92.0
38.1
71.4
35.5
DTD
84.7
48.7
93.3
87.4
75.5
93.8
92.7
70.4
71.8
34.2
C0
84.8
49.2
93.6
87.6
76.1
93.8
97.4
70.6
91.6
34.0
EuroSAT
84.8
49.2
93.6
87.6
76.6
93.9
97.4
70.7
91.7
87.4
83.3
Average
84.7
46.2
92.2
86.4
70.0
68.0
77.4
47.2
75.9
41.2
68.9
Table Accuracy of Ours on the X-TAIL full-shot with order-II. row represents the performance on every dataset of model
trained after the corresponding task.
Transfer
Average
and
metries are shown.
Caltech101
Aircraft
SUN397
MNIST
Flowers
EuroSAT
Cars
Pets
Food
DTD
Transfer
23.8
87.8
84.3
61.4
41.6
65.7
34.5
71.0
28.9
55.4
Cars
86.9
23.8
87.8
84.3
61.2
43.9
65.7
36.4
72.4
37.4
Aircraft
86.9
51.0
87.8
84.3
61.5
43.9
65.7
36.4
73.4
37.4
Pets
86.9
51.0
94.3
84.4
61.5
40.1
65.7
36.3
73.0
37.4
Food
86.9
51.0
94.3
89.4
61.5
65.7
35.8
73.2
36.7
SUN397
87.0
51.4
94.4
89.5
77.0
39.8
65.7
32.7
69.1
22.4
MNIST
87.0
51.4
94.4
89.5
77.0
78.2
65.7
32.8
69.1
22,4
Flowers
87.0
51.5
94.4
89.5
77.1
88.8
98.7
31.0
69.3
22.4
DTD
87.0
51.5
94.4
89.6
77.1
95.4
98.7
75.5
68.6
22.2
11
87.0
55.4
94.5
89.6
77.4
98.8
99.0
75.7
95.7
21.9
EuroSAT
87.0
55,4
94.5
89.6
77.7
98.8
99.0
75.8
95.8
95.8
86.9
Average
87.0
49.3
93.1
88.0
70.9
66.8
79.0
76.0
35.6
69.2
16


Reviews:
[{"review_id": "3Mnyxi77me", "reviewer": "Reviewer_4dD8", "summary": "The paper proposes a task incremental learning approach for CLIP encoders. In contrast to using adapters for the image encoder, it learns task (or class) specific memories represented by learnable vectors. The classification is performed via dot products of these task specific memories and CLIP image embeddings. The training mechanism attempts to prevent forgetting by multiple techniques including freezing memories of old tasks, separating samples from old tasks from the new ones, suitable modeling of distribution of clusters of past memories etc. The method also performs fine tuning for the text encoder for the new classes with mechanism to prevent forgetting.\n\nThe method was tested on multiple standard datasets, compared against relevant baselines and evaluated in different setting to illustrate capability of learning new classes as well as retaining classification performances on the old ones.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4dD8,\n\nThank you for your detailed review. We address your concerns one by one in the followings.\n\n\n\n> **W1** It should have been clearly mentioned that finetuning text encoder is part of the proposed approach.\n\n**A1** We sincerely apologize for the lack of clarity and fully agree with your assessment that the role of text encoder finetuning should have been explicitly stated early in the paper. We will make the following modifications in the next version.\n\n1. **Introduction (Section 1):**  \n   \n   - We will add a paragraph in the introduction explicitly stating that our proposed approach jointly optimizes both the image features and text encoders. Specifically, we will emphasize that we first establish a text encoder fine-tuning approach for continual learning by combining parameter freezing and distillation techniques.\n\n2. **Methodology (Section 3):**  \n   \n   - We will explicitly outline the full pipeline, including text encoder fine-tuning for continual learning, before introducing implementation details.  Also, an [overall framework](https://anonymous.4open.science/r/ICML25-LADA-4B78/README.md) will be added, illustrating both text encoder fine-tuning module and Label Specific CLIP Adapter module.\n   - The section titled **\"A Simple Baseline via Text Encoder Fine-tuning\"**  will be renamed to **\"Text Encoder Fine-tuning Framework for Continual Learning\"** to clarify its role as an ablated variant of our approach. We will add a transition sentence:  \"To address continual learning in X-TAIL senario, we first introduce a baseline framework with text encoder finetuning only.\" \n     \n     \n\n> **W2** The finetuning of text encoder does not clearly mention the full form of the loss function used. If I understood correctly, Eq. 3 and 4 are used for current and past tasks respectively and the total loss is a summation of the two.\n\n**A2** We apologize for the confusion. In the revised manuscript, we will add the following sentence in Section 3.2 after Equation 4:  **\"The total training loss is the sum of the current task loss (Eq. 3) and the distillation loss (Eq. 4), enabling joint optimization of both current and past task objectives.\"** \n\n\n\n\n> **W3** Is Eq. 8 or 10 used for previous task? Here too, the writeup does not clearly state the full loss function.\n\n**A3** We will clarify that  total loss combines Eq. 7 (current task) and **Eq. 10** (Distribution-Preserved Training loss for **previous task**) in next version.  \n\n\n\n> **W4** The introduction is unnecessarily long and not well written.\n\n**A4** Thank you for the constructive feedback. We will revise the introduction by:  \n\n1. Merge paragraphs 2-3 to concisely present the dual challenges of *memory stability* (preserving past knowledge) and *learning plasticity* (adapting to new tasks) in continual learning.  \n2. Remove detailed discussions of prior methods (e.g., regularization-based approaches) and shift them to the Related Work section.  \n\n \n\n> **W5** An analysis or ablation on why GMM modeling was needed would be good to justify this choice.\n\n**A5** We acknowledge that the the choice of GMM should be explicitly stated. We will add the following discussions in the next version of paper.\n\n**Choice of GMM**: Distilling only cluster centers loses fine-grained distribution information. GMM models the **full feature distribution** of past tasks, which controls the influence of prototypes by incorporating their contribution weights $\\pi$ into the loss function Eq. 10.\n\n**Impact of GMM**: Table 3 shows DPT based on GMM boosts Transfer and Average performance.  Table 4 shows increasing the number of image prototypes $\\lambda_2$ further improves Transfer performance.\n\n\n\n> **W6** Is \"CLIP Adapter\" the right term for the technique proposed? Would it not give the impression that the method is using adapters for image encoder before one reads the intro?\n\n**A6** Thank you for raising this concern. The term \"CLIP-Adapter\" has been recognized in vision-language research [1] to denote lightweight feature adaptation after the frozen CLIP backbone extracts representations, not input-level modifications. This naming convention emphasizes post-encoder refinement for downstream tasks, aligning with its technical definition and community usage. We will clarify this distinction explicitly in the revised manuscript to avoid potential misinterpretation.\n\n[1] Gao, Peng, et al. Clip-adapter: Better vision-language models with feature adapters. *International Journal of Computer Vision* 132.2 (2024): 581-595.\n\n\n\n> **W7** Will rely on other reviewers for novelty assessment.\n\n**A7** Please refer to **A1** and **Summary of  contributions** of **reviewer vJJv**.\n\n\n\nWe hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!", "claims_and_evidence": "---", "methods_and_evaluation_criteria": "I found the method distribution is sloppy and confusing.\n\n1. It is not mentioned in the introduction or at the beginning of the method section that the proposed method also finetunes the text encoder. Line 121 starts a new section titled \"A simple baseline\" which confused me as whether this is a baseline method that was compared against or part of the overall approach -- I think it is the latter. It should have been clearly mentioned that finetuning text encoder is part of the proposed approach.\n\n2. The finetuning of text encoder does not clearly mention the full form of the loos function used. If I understood correctly, Eqns 3 and 4 are used for current and past tasks respectively and the total loss is a summation of the two. But it is not clear in the text.\n\n3. Similarly, Line 206 suddenly, and without much justification, introduces GMM modeling of the p^i_j vectors. This again would confuse the reader, is Eqn 8 or 10 used for previous task?  Here too, the writeup does not clearly state the full loss function.\n\n4. The introduction is unnecessarily long and not well written.", "theoretical_claims": "---", "experimental_designs_or_analyses": "Sufficient experimentation (with analysis) was performed to show the strength of the method.  Evaluation in 3 settings (transfer, last and avg) as well as illustration of performance consistency over learning steps in Fig 2 are both good attempts to convince the reader of the merits of this approach.\nAn analysis or ablation on why GMM modeling was needed would be good to justify this choice.", "relation_to_broader_scientific_literature": "Will rely on other reviewers for novelty assessment.", "essential_references_not_discussed": "---", "other_strengths_and_weaknesses": "---", "other_comments_or_suggestions": "Is \"CLIP Adapter\" the right term for the technique proposed? Would it not give the impression that the method is using adapters for image encoder before one reads the intro?\n\nSomething like Task Memories for Scalable Continual Learning with CLIP Encoders might be more reflective of the method? Just a thought.", "questions_for_authors": "---", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIttz8QkP7", "reviewer": "Reviewer_EaVq", "summary": "Instead of partitioning parameters across tasks, this paper proposed LADA appended lightweight, labelspecific memory units to the frozen CLIP image encoder, enabling discriminative feature generation by aggregating task-agnostic knowledge. The method achieves state-of-the-art performance in continual learning settings on several datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EaVq,\n\nThank you for your detailed review. We address your concerns one by one in the followings.\n\n> **W1**: **Essential references not discussed of [1]** and the second point in Other Strengths And Weaknesses of **comparisons with RanPAC [1]**\n\n**A1**: We acknowledge the reviewer’s comments regarding the article and address the concerns about the omission of RanPAC [1] as an essential reference, as well as the need for a comparison between RanPAC and our proposed method.\n\n1. Core Idea of RanPAC\n   \n   RanPAC [1] projects features into a higher-dimensional space, which has been shown to benefit continual learning. RanPAC [1] tackles multicollinearity in CLIP’s original feature dimensions by employing **random Gaussian projection matrices** and **nonlinear activation functions** (e.g., ReLU). Specifically, it extracts CLIP features using a frozen encoder, projects them via a Gaussian matrix $W$, applies nonlinear transformations, and uses ridge regression on 0-1 encoded labels to train classifiers. This approach leverages second-order statistics to reduce feature redundancy and enhance classifier accuracy.\n\n2. Key Differences and Advantages of LADA\n   \n   - **Feature Optimization**\n   \n     RanPAC optimizes features using static **random Gaussian projection matrices** to reduce multicollinearity in CLIP features, but it lacks adaptability to task-specific nuances, often failing to capture fine-grained or diverse class characteristics effectively. In contrast, LADA introduces a **label specific memory** dynamically amplifying CLIP feature most relevant to each class. This process condenses features into sparse, high-dimensional representations tailored to individual tasks, enhancing discriminability without modifying the frozen CLIP encoder. By leveraging this class-specific adaptability, LADA achieves superior plasticity and stability.\n   \n   - **Cross-Task Adaptation**\n   \n     RanPAC’s dependence on **fine-tuning the CLIP encoder for the first task** incurs substantial computational overhead and risks eroding CLIP’s pre-trained general knowledge, making it impractical for continual learning across diverse tasks. In contrast, LADA takes a fundamentally different approach by **freezing the CLIP encoder entirely** and training lightweight a **label-specific adapter**. \n\n3. Empirical Validation\n   \n   To compare LADA with RanPAC, we integrate RanPAC into our framework (denoted as BF+RanPAC+DPT, where BF refers to baseline fine-tuning and DPT to Distribution-Preserved Training) and evaluate it against our method (BF+LADA+DPT). In the X-TAIL 16-shot setting, the results demonstrate that BF+LADA+DPT outperforms BF+RanPAC+DPT by **1.4% in Transfer**, **1.4% in Average**, and **1.0% in Last** metrics, averaged across the 10 tasks.\n\n||Aircraft|Caltech101|DTD|EuroSAT|Flowers|Food101|Mnist|Pets|Cars|Sun397   |***Average***|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| **Transfer**|\n|BF+**RanPAC**+DPT|--|74.4|35.4|34.6|63.4|83.3|36.9|87.6|64.7|60.8|60.1|\n|Ours|--|**75.0**|**36.1**|**35.9**|**66.3**|**83.7**|**42.1**|**88.0**|**65.3**|**61.4**|**61.5**|\n|**Average**|\n|BF+**RanPAC**+DPT|47.0|86.7|61.0|71.4|83.3|84.3|59.5|89.4|68.6|62.2|71.3|\n|Ours|**49.1**|**91.0**|**61.3**|**71.6**|**84.4**|**85.0**|**62.8**|**89.7**|**69.2**|**62.9**|**72.7**|\n|**Last**|\n|BF+**RanPAC**+DPT|47.1|90.3|68.3|87.2|96.5|85.4|93.4|**93.7**|84.2|74.4 |82.1|\n|Ours|**49.6**|**93.7**|**69.3**|**86.9**|**96.7**|**86.9**|**93.8**|**93.7**|**84.6**|**76.0**|**83.1**|\n\n[1] McDonnell, Mark D., et al. \"Ranpac: Random projections and pre-trained models for continual learning.\" NeurIPS,2023.\n\n> **W2**: It is recommended to add a flowchart of the LADA method or end-to-end loss function in the Overall Framework section\n\n**A2**: We appreciate the reviewer’s suggestion. The overall framework and end-to-end loss function of LADA are available in an [anonymized  link](https://anonymous.4open.science/r/ICML25-LADA-4B78/README.md). We will include them in the revised manuscript. \n\n> **W3**: Experiments on some important datasets are necessary, such as CIFAR100.\n\n**A3**: Thank you for your suggestion. CIFAR-100 is indeed an important dataset. However, due to overlapping class names with other datasets, it was not included in the original X-TAIL dataset to ensure accurate evaluation. Nevertheless, we have conducted additional experiments on **Inclusion of CIFAR-100 in the X-TAIL** to further demonstrate the robustness of our approach. Under the 16-shot setting, our method still achieves state-of-the-art results, with improvements of **4.3%**, **2.0%**, and **1.0%** on the Transfer, Average, and Last metrics, respectively. Please refer to the [anonymized  link](https://anonymous.4open.science/r/ICML25-LADA-4B78/README.md) for detailed experimental results.\n\nWe hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "It makes sense in general", "theoretical_claims": "Most proofs were checked.", "experimental_designs_or_analyses": "Most experiments were checked. Please refer to Other Strengths and Weaknesses.", "relation_to_broader_scientific_literature": "The method appends lightweight, label-specific memory units to the frozen CLIP image encoder, enabling discriminative feature generation by aggregating task-agnostic knowledge.", "essential_references_not_discussed": "In [1], RANPAC also projects features into a higher-dimensional feature space, which has been proven to be beneficial for continual learning. It has not been demonstrated that the effect of LADA is better than that of nonlinear random projection [1]. \n\n[1]McDonnell, Mark D., et al. \"Ranpac: Random projections and pre-trained models for continual learning.\" NeurIPS,2023.", "other_strengths_and_weaknesses": "1. It is recommended to add a flowchart of the LADA method or end-to-end loss function in the Overall Framework section, which will make the method better understood.\n2. LADA projects features into a higher-dimensional feature space, which has been proven to be beneficial for continual learning [1]. However, in the ablation experiment, it has not been demonstrated that the effect of LADA is better than that of nonlinear random projection [1]. I think [1] should be included in the reference.\n3. Experiments on some important datasets are necessary, such as CIFAR100. It is recommended to supplement.\n\n[1]McDonnell, Mark D., et al. \"Ranpac: Random projections and pre-trained models for continual learning.\" NeurIPS,2023.", "other_comments_or_suggestions": "Please refer to Other Strengths and Weaknesses.", "questions_for_authors": "Please refer to Other Strengths and Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Vdb91D6mY", "reviewer": "Reviewer_vJJv", "summary": "This paper primarily proposes an adapter-based method initialized with class cluster centers for CLIP-based Cross-Domain Task-Agnostic Incremental Learning(X-TAIL). This approach enables class discrimination within a unified feature space using class-specific parameters without requiring task-specific parameter selection. \nAdditionally, it models old class features using the Gaussian Mixture Model (GMM) and gets old class features by sampling from the GMM distributions. \nThe approach achieves state-of-the-art performance in X-TAIL.\n\n##update after rebuttal \nAfter carefully reviewing the authors' responses and the other reviewers' comments, I appreciate that my main concerns have been adequately addressed. I am now inclined to recommend acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vJJv,\n\nThank you for your detailed review. We address your concerns one by one in the followings.\n\n> **W1 Other Strengths And Weaknesses:** Label-Specific CLIP Adapter bears a strong resemblance to the CLIP-based few-shot learning method Tip-Adapter [5]. **Questions For Authors:** Similarity to Tip-Adapter.\n\n**A1** We agree with the reviewer that the adapter module in LADA shares similarities with Tip-Adapter. **However, our key contribution lies in generalizing this module for continual learning tasks—a scenario where Tip-Adapter fails to perform effectively.**\n\nWe demonstrate that the adapter (which we term *label-specific memory*) can naturally capture task-specific data distributions and mitigate forgetting. To maintain scalability and simplicity, we initialize the memory cache using k-means clustering and adopt Tip-Adapter’s output formulation. In our view, adapting Tip-Adapter to continual learning is nontrivial: LADA introduces significant improvements to address both backward and forward forgetting.\n\nMoreover, LADA seamlessly extends to few-shot learning, matching Tip-Adapter’s original setting. Under the 16-shot benchmark (using CLIP ViT-B/16, following [6]), LADA outperforms Tip-Adapter-F by **1.6%** (see table below). Notably, it achieves this with only **1/4** of the feature embeddings required by Tip-Adapter, underscoring its efficiency.\n\n||Aircraft|Caltech101|DTD|EuroSAT|Flowers|Food|Pets|Cars|Sun397|Average|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Tip-Adapter-F|44.6|95.7|70.8|85.9|96.2|86.8|92.6|82.3|76.0|81.2|\n|Ours|**49.6**|**96.4**|**71.5**|**86.8**|**97.4**|**87.7**|**94.0**|**84.8**|**76.8**|**82.8**|\n\nThe performance results for Tip-Adapter-F are sourced from recent work [6].\n\n**Summary of  contributions**\n\n**Contribution 1:** We design a text encoder fine-tuning framework which enhances the classification ability and naturally obtains the transfer ability of CLIP model.\n\n**Contribution 2:** We propose LADA, a lightweight CLIP adapter that condenses task-agnostic knowledge to transform CLIP representation into label-specific features, eliminating the need for parameter selection as seen in previous mainstream methods.\n\n**Contribution 3:** We propose Distribution-Preserved Training which controls the influence of prototypes by incorporating their contribution weights into the loss function. We not only addresses backward forgetting but also tackles forward forgetting, a challenge overlooked in prior methods.\n\n> **W2 Essential References Not Discussed:** There are several essential references related to Distribution-Preserved Training that are either briefly mentioned or not cited in the paper.\n\n**A2** Thanks for the reviewer highlighting these relevant papers and we will include citations and discussions. Our approach aligns with a standard Gaussian distribution when $\\lambda_2=1$, similar to the prototype augmentation method in [1]. When $\\lambda_2 > 1$, our loss function (Equation 10) leverages the mixture weights $\\pi$ from the GMM to perform more sophisticated augmentation. As shown in Table 4, when $\\lambda_2=1$, the *Transfer* performance is lower compared to when $\\lambda_2 > 1$. This indicates that the effectiveness of DPT while prior methods in [1-4] do not address *forward forgetting* in pre-trained models, our approach explicitly tackles this issue.\n\n> **W3 Other Comments Or Suggestions:** The improvement of BF achieved by the presented modules appears to be limited. \n\n**A3** In Tab.2 and Tab.3, our basic fine-tuning framework (BF) already outperforms several baseline methods, demonstrating the challenges prior approaches face in task-agnostic settings. In the full-shot setting, these modules improve *Transfer* by an average of **2.3%**, *Average* by **2.0%**, and *Last* by **1.3%** compared to the BF framework alone.\n\n> **W4 Other Comments Or Suggestions:** It is unclear whether this linear weighting is a hyperparameter.\n\n**A4** We regret the confusion. The linear weighting is a fixed hyperparameter (set to 1.0 in experiments), not learned, balancing LADA logits and text features equally. This will be clarified in the revision.\n\nWe hope this rebuttal addresses your concerns. Please let us know if further refinements are needed!\n\n[1] Zhu, Fei, et al. Prototype augmentation and self-supervision for incremental learning. *CVPR* 2021.\n\n[2] Tang, Yu-Ming, et al. When prompt-based incremental learning does not meet strong pretraining. *ICCV* 2023\n\n[3] Zhang, Gengwei, et al. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. *ICCV* 2023.\n\n[4] Huang, Linlan, et al. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. *ECCV* 2024.\n\n[5] Zhang, Renrui, et al. Tip-adapter: Training-free adaption of clip for few-shot classification. *ECCV* 2022.\n\n[6] Zanella, Maxime, et al. Low-rank few-shot adaptation of vision-language models. *CVPR* 2024.", "claims_and_evidence": "Yes, their claims are clear and convincing", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem", "theoretical_claims": "The paper does not include formal theoretical Claims or proofs.", "experimental_designs_or_analyses": "The experimental design and analysis follow the setup established in previous works introducing this problem. The study utilizes ten different domain datasets as ten sequential tasks for continual learning. This experimental design is well-founded and appropriate.", "relation_to_broader_scientific_literature": "This work builds upon prior research in incremental learning and pre-trained model fine-tuning. Unlike prompt tuning in incremental learning areas, there is no need to select additional parameters. Class features are trained using kmeans cluster center initialization instead of random initialization of the vanilla adapter", "essential_references_not_discussed": "There are several essential references related to Distribution-Preserved Training that are either briefly mentioned or not cited in the paper.\n[1] proposes a method that augments prototypes using Gaussian noise. While it is mentioned in the paper, its similarity to the proposed approach, where GMM sampling is used to augment prototypes, warrants a more detailed discussion. [2], [3], and [4] all adopt the strategy of sampling old class features from a class-specific Gaussian distribution, yet they are not cited. The key distinction of this work is the use of a Gaussian Mixture Model (GMM) instead of a simple Gaussian distribution, which should be explicitly discussed.\n\nAdditionally, the method [5], which is highly relevant to the Training of Label-Specific CLIP Adapter, is only cited but not discussed. Although [5] focuses on a CLIP-based few-shot adapter, its overall structure, initialization strategy, training approach, non-linearity design, and final feature extraction process share significant similarities with the proposed method. A more detailed discussion is needed to clarify these connections.\n\n[1] Zhu, F., Zhang, X. Y., Wang, C., Yin, F., & Liu, C. L. (2021). Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5871-5880).\n\n[2] Tang, Y. M., Peng, Y. X., & Zheng, W. S. (2023). When prompt-based incremental learning does not meet strong pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 1706-1716).\n\n[3] Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19148-19158).\n\n[4] Huang, L., Cao, X., Lu, H., & Liu, X. (2024, September). Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. In European Conference on Computer Vision (pp. 214-231). Cham: Springer Nature Switzerland.\n\n[5] Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., ... & Li, H. (2022, October). Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision (pp. 493-510). Cham: Springer Nature Switzerland.", "other_strengths_and_weaknesses": "The novelty of this paper is questionable. The key module, Label-Specific CLIP Adapter, bears a strong resemblance to the CLIP-based few-shot learning method Tip-adapter [5]. Tip-adapter initializes the adapter by using multiple features from same-class images to set the corresponding class-specific weights in the adapter. It also explores fine-tuning this initialized adapter to further improve performance. The adapter structures are highly similar, and the output formulation of Tip-adapter is identical to Equation (6) in this paper. Additionally, both methods employ the same technique to convert inner products into non-negative values. The overall framework follows a similar strategy, where the final classification result is obtained by weighting the adapter’s output and the text encoder’s output.\nThe primary difference lies in the experimental setup: this paper focuses on full-shot datasets, allowing the use of K-means to obtain class sub-centers for initialization, whereas Tip-adapter, designed for few-shot scenarios, directly initializes the adapter using a limited number of sample features. Additionally, this paper incorporates a continual learning setting, introducing the freezing of old class classifiers—a widely used technique in the continual learning domain. However, this adaptation is a general strategy rather than a novel methodological contribution.", "other_comments_or_suggestions": "In the ablation study results, the improvement of BF achieved by the presented modules appears to be limited. Further analysis is needed to clarify and justify the effectiveness of these modules. \nAdditionally, on page 5, in the \"Overall Framework\" section, the authors mention that \"the final prediction is obtained by applying a linear weighting between the logits produced by LADA and the corresponding text features.\" It is unclear whether this linear weighting is a hyperparameter and how it is determined. Further clarification on this aspect would be beneficial.", "questions_for_authors": "Similarity to Tip-adapter: The proposed Label-Specific CLIP Adapter appears to be highly similar to Tip-adapter, with key similarities in initialization, structure, and final classification formulation. The primary differences seem to be (1) using K-means clustering for adapter initialization instead of directly leveraging a few-shot feature set, and (2) incorporating a continual learning setting with frozen classifiers for old classes. Given these similarities, could the authors clarify the novelty of their approach beyond these modifications?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSpAoew14o", "reviewer": "Reviewer_oKy3", "summary": "This paper presents LADA (Label-specific ADApter), an approach for continual learning with vision-language models like CLIP. LADA enhances scalability and performance by generating discriminative, label-specific features. Unlike existing methods that partition parameters across tasks, LADA appends lightweight memory units to the frozen CLIP image encoder, enabling task-agnostic knowledge aggregation while preventing catastrophic forgetting through feature distillation. LADA ensures efficient training without updating the frozen parameters of CLIP. Experiment results show that LADA achieves state-of-the-art performance in continual learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oKy3,\n\nThank you for your detailed review. We address your concerns one by one in the followings.\n\n> **W1 Claims And Evidence:** the evidence provided in Table 4 focuses on ablation without comparisons to other methods. **Experimental Designs Or Analyses:**  computational efficiency would further support the paper’s claims.\n\n**A1** We conducte a computational efficiency experiments. We compare our method with other methods using a consistent input batch size of 64 in the full-shot setting. Time comparisons for MoE-Adapters and ours were conducted on a single NVIDIA 4090 GPU. The results are summarized in the table below:\n\n|Method|Train Params (M)|Max GPU Memory (GB)|Time (s/batch)|\n|-|-|-|-|\n|LWF|149.6|31.42|--|\n|ZSCL|149.6| 25.67|--|\n|MoE-Adapters|59.8|21.83|0.337|\n|Ours| **11.2**|**18.51**|**0.289**|\n\n> **W2 Essential References Not Discussed:** \n> \n> 1. The related work should discuss CL methods based on CLIP that utilize PEFT techniques.\n> 2. The authors should discuss related works such as CODA-P, APG, and EvoPrompt. \n\n**A2** We appreciate this suggestion and will add a dedicated subsection titled **\"Continual Learning Methods Based on PEFT\"** to the related work section of the revised manuscript:\n\nParameter-efficient fine-tuning (PEFT) has been widely adopted in continual learning to enhance representation learning for specific tasks. Prompt-based methods, such as L2P [1], expand the prompt pool as new tasks are learned, relying on auxiliary losses. **CODA-Prompt** [2] propose a end to end prompt selection methods to increase plasticity. **APG** [3] introduces a prompt generator to reduce the gap between pretraining and future tasks, while **EvoPrompt** [4] proposes an adaptive and continuous prompting approach to alleviate issues of selection mismatches and limited prompt shareability. MoE-Adapters [5] insert a mixture of adapters into the image encoder, activating a few for each task. However, these methods typically involve an auxiliary parameter selection step during inference to extract image features using the expected prompts or adapters, which can lead to misassignments and degrade classification performance. \n\n> **W3 Essential References Not Discussed:** The paper lacks discussion and comparison with prior CL methods that leverage prototypes for pseudo-feature replay, such as HiDe-Prompt and CPP. \n\n**A3** We are grateful for the reviewer’s suggestion. We will incorporate a discussion on these approaches in the related work of the revised manuscript:\n\nPrototype-based methods leverage pseudo-feature replay to mitigate forgetting in continual learning. **HiDe-Prompt** [6] optimizes hierarchical components with task-specific prompts using a single prototype per class, while **Contrastive Prototypical Prompt** [7] employs task-specific prompt-tuning with contrastive learning and multiple prototypes per class to address semantic drift and prototype interference. However, as highlighted in our introduction, while backward forgetting has been effectively addressed, these approaches often overlook the impact of pseudo-feature replay on the model’s original generalization ability, leading to forward forgetting. \n\n> **W4** what does the **i** represent in Eq. (5)?\n\n**A4** We apologize for the oversight. The notation **i** represents the image embedding, as defined in line 115 of the original manuscript. We will explicitly clarify this directly at Eq. (5) in the revised version.\n\n> **W5** More explanation is needed for the statement “a linear weighting between the logits produced by LADA and the corresponding text features”. Clarifying this process is crucial for understanding how final predictions for seen classes are made.\n\n**A5** We regret the confusion. The linear weighting is a fixed hyperparameter (set to 1.0 in experiments), not learned, balancing LADA logits and text features equally. This will be clarified in the revision.\n\n> **W6 Questions For Authors:** My primary concern is the lack of discussion on related works. \n\n**A6**  We have rewritten the related work in **A2** and **A3** . We commit to integrating these revisions into the manuscript.\n\n[1] Wang, Zifeng, et al. Learning to prompt for continual learning. *CVPR*. 2022.\n\n[2] Smith, James Seale, et al. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. *CVPR* 2023.\n\n[3] Tang, Yu-Ming, et al. When prompt-based incremental learning does not meet strong pretraining. *ICCV* 2023.\n\n[4] Kurniawan, Muhammad Rifki, et al. Evolving parameterized prompt memory for continual learning. *AAAI* 2024.\n\n[5] Yu, Jiazuo, et al. Boosting continual learning of vision-language models via mixture-of-experts adapters. *CVPR* 2024.\n\n[6] Wang, Liyuan, et al. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. *NeurIPS* 2023.\n\n[7]  Li, Zhuowei, et al. Steering prototypes with prompt-tuning for rehearsal-free continual learning. *WACV* 2024.", "claims_and_evidence": "The authors claim that “Our method is scalable and efficient, adding only small sets of learnable memory units for novel tasks, without requiring gradient propagation to the frozen CLIP image encoder.” However, the evidence provided in Table 4 primarily focuses on ablation studies without direct comparisons to other methods. To better support this claim, the authors are encouraged to include efficiency metrics of existing methods for a more comprehensive evaluation.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There’s no theoretical claims.", "experimental_designs_or_analyses": "The experimental design is well-structured, and the analyses are generally strong. However, additional comparisons in ablation studies, particularly in terms of computational efficiency, would further support the paper’s claims.", "relation_to_broader_scientific_literature": "This work eliminates the need for auxiliary prompt or adapter selection, offering a unified and scalable solution for continual learning scenarios.", "essential_references_not_discussed": "1. The related work section should include a dedicated subsection discussing continual learning (CL) methods based on CLIP that utilize parameter-efficient fine-tuning (PEFT) techniques, such as prompting, adapters, and LoRA.\n\n2. While many early CL methods using PEFT involve a selection process for determining the appropriate prompt or adapter for a given task sample, recent efforts aim to avoid this selection step. The authors should discuss related works such as CODA-P (CVPR’23), APG (ICCV’23), and EvoPrompt (AAAI’24).\n\n 3. The paper lacks discussion and comparison with prior CL methods that leverage prototypes for pseudo-feature replay. Some methods, such as HiDe-Prompt (NeurIPS’23), employ a single prototype per class, while others, such as CPP [1], utilize multiple prototypes per class. A discussion on these approaches would enhance the contextualization of LADA within the broader CL landscape.\n\n[1] Steering prototype with prompt-tuning for rehearsal-free continual learning. WACV 2024", "other_strengths_and_weaknesses": "Other stengths.\n\n1. The paper is generally easy to follow.\n\n2. The proposed method demonstrates superior performance in Cross-domain TaskAgnostic Incremental Learning setting.\n\nOther weaknesses.\n\n1. Some notations are not clearly introduced. For example, what does the $\\mathbf{i}$ represent in Eq. (5)?\n\n2. More explaination is needed for the statement “a linear weighting between the logits produced by LADA and the corresponding text features”. Clarifying this process is crucial for understanding how final predictions for seen classes are made.", "other_comments_or_suggestions": "Missing “:” at the end of the line above Eq. (9).", "questions_for_authors": "My primary concern is the lack of discussion on related works. Please refer to the section “Essential References Not Discussed” for further details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes LADA (Label-specific ADApter), a method for continual learning with CLIP-based models that avoids task-specific parameter partitioning. Instead, LADA appends lightweight, label-specific memory units to the frozen CLIP image encoder, enabling discriminative feature generation while preventing catastrophic forgetting through Distribution-Preserved Training (DPT). The method achieves state-of-the-art performance in Cross-domain Task-Agnostic Incremental Learning (X-TAIL) by combining task-agnostic knowledge aggregation with efficient training (no updates to CLIP’s frozen parameters).

LADA presents a practically effective solution for CLIP-based continual learning, with strong empirical results and scalability advantages. While the initial writing needed refinement, the rebuttal convincingly addressed critiques. The paper’s real-world applicability (minimal overhead, no CLIP fine-tuning) and unified approach (no task-specific selection) align with ICML’s emphasis on scalable ML. Post-rebuttal revisions solidify its contribution.
Decision: Accept (poster)