Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Authors: Yicheng Xu
Yuxin Chen
Jiahao Nie
Yusong Wang
Huiping Zhuang
Manabu Okumura
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Incremental learning
Analytic learning
Cross-domain
Vision-Language model
Primary area: machine_vision

Abstract: Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose Regression-based Analytic Incremental Learning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data.
Additionally, we introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint.
We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning.

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Yicheng Xu
Yuxin Chen 2+ Jiahao Nie
Yusong Wang'
Huiping Zhuang Manabu Okumura'
Institute of Science Tokyo University of California Berkeley
3 Nanyang Technological University "South China University Technology Greater Bay Area Institute for Innovation, Hunan University, China
yxu0400e ntu. atu.eddu.ss sg, yuxincâberkeley. eedu .ddu
Abstract
Continual learning (CL.) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL. which only focuses previously encountered classes. During the CL VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely domain-ident iddetity hints to classify images across dif- ferent domains. In this study, we propose Regression-bast Analytic Incremental Learning (RAIL), which utilizes recursive ridge eegression-based adapter learn from sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features Cooperating with a training-free fusion module, RAIL. absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data. Addition- ally, we introduce Cross-domain Task-Agnostic Incremental Learning (X- -X-AIL) setting. this setting, CL learner is required incrementally learn from multiple domains and classify test images from both seen and unseen domains without any nomain-identig hint. We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of- the-art performance both X-TAIL and existing Multi-domal Task Incremental Learning settings. The code is released at wtfhttps://ggt thub. wwt.coommm/a/a wwmmminggan11997////// Regression based- Anal www.eeeeeeaeal-eerrrriin
dimerr-oiienii space.
Introduction
is a crucial area in machine learning, which requires a learner to incrementally learn new data instead of training from scratch. The main challenge in CL is known as catastrophic forgetting [4) where lcarning new knowledge results in the forgetting of the old one. To this end, various CL approaches [5] 7
Continual learning (CL) I
have been proposed to solve the forgetting issue. As a typical CL setting, Intaemnanceemltalll Learning (CIL) (Fig. (a)) aims to achieve robust discriminability on all seen classes. Despite the advancements, existing approaches mainly focus classifying images only from seen classes, thereby limiting the model's generalizability.
6
3 g
Consequently, Zheng et al. [10 proposed Multi-domain Task-Incremental Learning (MTIL), which cooperates CL with the zero-shot ability of Language Models (VLMs) 11 as CLIP 13). This integration equips models with the ability to classify domains they have not
13 [d] such
'Equal contribution
Corresponding author: hpzhuang scut.edu.en
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Learning Sequence
Traditional
(b)
X-TAIL
Dataset
photo of photo of Chee
photo
2NC1
photo of
photo Ieis
photo of 9
Dataset
25C1
RAIL
pãoto of Pickup
A. photo of Pickup
photo of Pickup
Figure Comparison of different CL settings. (a) In CIL, models classify images within all previously encountered classes. (b) In MTIL, models classify images from both seen and unseen domains based on the given domain-identities. (c) In X-TAIL, models classify images from both seen and unseen domains without any domain-identity hint.
yet encountered, enhancing their generalizability across multiple domains (Fig. I (b)). Several methods [10 15] have been specifically designed for MTIL, in which the model required to retain both the incrementally learned knowledge during CL and the zero-shot ability of VLMs. However, these methods require a domain-identity hint to indicate the specific domain of the test image, which often not applicable in real- worldworld scenarios 77. Additionally, the of reference datasets during training is necessary to maintain pre-trained VLMs' zero-shot performance.
To address the aforementioned limitations, we introduce Regression regression-based Analytic Incremental Learning (RAIL), novel approach that incrementally learns new knowledge and performs effectively on both seen and unseen domains. Specifically, we leverage non-linear projection functions from both primal and dual perspectives to enhance the expressiveness fcatures extracted by the pre-trained CLIP. It endows the learner with the ability to classify images in a cross-domain label set without any domain -itttttt hint. In the incremental learning process, RAIL utilizes a ridge regression based adapter that updates the parameters recursively. This is identical learning all encountered domains at once, achieving absolute memorization on learned domains. Additionally, we freeze the pre-trained CLIP and design training-fre free fusion module to determine whether the test data belongs to seen or unseen domains. This strategy absolutely preserves CLIP's zero-shot ability on unseen domains, meeting practical requirements for models deployed in dynamic environments.
To demonstrate the effectiveness of our method, we propose Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting as illustrated in Fig. (c). Particularly. X-TAIL requires CL methods to incrementally transfer a pre-trained VLM to multiple domains while evaluating the model's performance
both seen and unseen domains. Moreover, domain hints are forbidden in X-TAIL, making it more realistic and challenging B]. As a result, effective CL methods must classify test image into the correct domain and class simultaneously.
Our contributions are summarized follows:
We propose a new CL method RAIL to incrementally adapt the pre-trained VLM to multiple domains without forgetting both pre-trained and incrementally learned knowledge.
To meet the practical scenario where CL methods need to sequentially learn data from new domains and classify images across these domains, we propose a new setting X-TAIL to evaluate the preservation of VLM's zero-shot ability and the adaptability to new domains.
We theoretically prove the RAIL's absolute memorization on incrementally learned domains and demonstrate that the zero-shot ability of the pre-trained VLM unseen domains is absolutely preserved.
We empirically show that the proposed method achieves state-of-the-art performances on both existing MTIL and the novel X-TAIL settings.

---

2
Related work
Early CL methods focused on Task-Incremental Learning (TIL) [16), where task-id is given during testing. Subsequently, a more practical and challenging setting of Class-Incremental Learning (CIL) 7 was proposed, where the access to the task-id is forbidden at inference time. Methods for CIL must therefore distinguish between all classes encountered in learned tasks. More recently, Zheng et al. [T0] proposed Multi-Domaii Task-Incremen
Learning (MTIL), which is especially designed to evaluate CL methods with pre-trained VLMs. In MTIL, a pre-trained VLM continually tasks. The performance both seen and unseen tasks measure the retention of both incrementally acquired and pre-trained knowledge. However, it still requires the task-id to create specific domain label space inference time. Apart from them, X-TAIL combines the challenges of both CIL and MTIL, in which the model learns new classes from various incoming domains and distinguishes between both seen and unseen classes without any domain-identity.
adapts to
Prevailing continual learning methods include replay-based, distillation- based, regularization-bann and architecture-basee approaches [3]. Replay-based methods such as iCaRL [7] typically store small portion of the previous task data as exemplars. The model is then trained jointly on new task data and the saved exemplars to preserve the previous knowledge. Distillation- based methods such as LwF [6] use either weight or function regularization to transfer knowledge from the previous model to the current model for knowledge distillation. Regularization- based methods such as ZSCL [IO] penalize the shift of either model parameter or feature space by adding a regularization term to the cross-entropy loss function. To preserve the robustness of the strong pre-trained model without access to the pre-trained dataset, 7SCL utilizes large-scale reference datasets to regularize the parameter space. Architecture-basee methods [17 18. 19] expand the model by constructing task-specific parameters to avoid inter-task interference. For example, MoE-Adapters [15] cooperates the pre- trained CLIP with mixture of experts (MoE) [20] learn from different domains. By leveraging a reference dataset to initialize a task-id indicator, it enables the model to distinguish unseen tasks from seen ones.
The aforementioned methods either neglect the forgetting issue of pre-trained knowledge or require multiple iterations and large-scale reference datasets for training, making it challenging to efficiently adapt to new data in continual learning scenarios. By contrast, RAIL employs an analytical solution that achieves the optimum in single epoch without additional reference data, ensuring its efficiency.
3 Cross-domain task-agnostic incremental learning
3.1 Problem setting
We define Cross-domain Task-Agnostic Incremental Learning (X-TAIL) follows. Given pre-trained VLM, the learner is required to incrementally transfer it to N different domains (DO), Dl,),..,,
Each domain (n)) = {(x,"" (m) ,, | is available only during the n-th learning step. The class labels (n) E Glad from the incrementally Icarned domain (()) are added to the set of seen class labels. During inference at all steps. the learner attempts to classify input images from any domain without the domain-identity hint. In other words, the ground-truth label of the test image belongs to CN Cr. U Ct. where Cr. = U._ c(f) is the union of seen class labels from all previous learning steps and CU is the set of unseen class labels.
Similar to the task-id in TIL, the Nomain-identiv hint allows the learner to classify input data within the label space of specific domain during evaluation. Essentially, the learner knows the domain of test images, which is far from real- holld application scenarios. For instance, the learner is supposed to predict an image as the class of husky from all possible classes CN =/car, bus, .... churros, donuts, ..., husky, beagle, bulldog, .../. However, if the domain-identity hint is given, the learner only needs to predict from a limited subset Cang =(husky, beagle, bulldog, .../. which is simpler but less realistic compared to practical applications. Therefore, we extend our setting to a task-agnostic scenario. Specifically, the learner predicts images from CN. the union of any potential class labels, without any domain hint.
3

---

3.2 Datasets
In X-TAIL's cross-domain setting, the learner should encompass as extensive data distributions
as possible. Following previous work [10] [3] 21 13 22 23], we select 10 different image-
classification datasets from different domains for our setting: Aireraft [24]. Caltech101 [25].
DTD (26). EuroSAT (27]. Flowers (28), Food [29]. MNIST 30), OxfordPet 31, StanfordCars 32
and SUN397 [R Specifically, to prevent the redundancy of learning overlapping classes and to
maintain the integrity of the setting, CIFAR-100 R was excluded because it includes many classes
that overlap with those in other domains. As a result, CL methods under X-TAIL should discriminate
images from total of 1, 100 classes across all domains.
3.3 Evaluation metrics
We adopt the evaluation metries from [10 for our setting. As illustrated in Fig. 2 each column represents the perfor- mance on a specific domain after each learning step, while the rows correspond to the learning sequence. In traditional CL settings, only the results in the lower diagonal where the learner has been exposed to the exemplars from the test domains are measured. Nevertheless, X-TAIL extends this evaluation to cover the entire matrix, recording performances across both learned and unlearned domains. The "Average" "
Evaluation Domain ID
Transfer
kg
Learning Step
the orange blocks, indicates the average accuracy of all learning steps across all domains. The gray and green blocks under the diagonal show the classification performance on these domains after the model has learned these domains. Specifically, the green blocks represent the model's 's last performance on these domains after learning all domains. The "Last" metric, which the average of the green blocks, reflects the learner's s adaptability to new domains. Additionally, the blue blocks in these domains before learning
metric, averaged
Last
pep
Average
Figure 2: Metrics for X-TAIL setting.
the upper-right matrix indicate the model's zero-shot performance these domains. The average of these blocks, referred to as the "Transfer" metric, measures the extent which the zero- shot ability is preserved throughout incremental learning.
Approach
4.1
Motivation
While CLIP demonstrates the generalizability of zero-shot classification, still struggles in certain unfamiliar domains [21]. Leveraging CLIP's robust feature extraction capabilities, linear probe offers straightforward approach transfer the CLIP to these domains [33. Among various linear solutions, Ridge Regression (RR) provides an effective classification strategy by mapping the image features onto one-hot-label targets (35). Given a pre-trained CLIP image encoder fi and a dataset D {XX Y)}. where X is the tensor of training images and Y is the matrix of corresponding one-hot labels, the predicted logits and the optimization problem are defined as:
y = W,W, arg min Y -
X. F + >I
W
where X. = fi (X) denotes the CLIIP extracted features, W is the classifier parameter, and À is the regularization parameter.
In the context of X-TAIL, the classifier needs to distinguish a wide range of classes from different domains. However, the extracted CLIP features of images from different domains suffer from certain cross-domain correlations, leading to limited domain discriminability. Based on Cover's 's theorem [6),, one promising approach (37 38 39 to enhance the linear separability of features is project the features into a higher- dimees--immnsionall space via some non- non-linear projection function. We explore this non-linear projection function from two perspectives:
Primal form ridge regression. Following (38, we use Randomly-initialzzer Hidden Layer (RHL.)
to project raw features to a higher dimensional space. By explicitly defining the projection function

---

as ((),, the classifier parameter is determined as follows:
W
+
+ + AI) Y
where Φ (X.)). In this way, o(-) is fixed throughout the training process.
Dual form ridge regression [40]. Instead of manually designing the projection function, we utilize the Kernel method [41] to implicitly define o6-) based the inner-product nature of dual form ridge regression. Depending on the choice kernel function, this approach allows for an infinite projection dimension, which is unachievable through any explicit definition. The dual form solution is defined as:
a (K -
(3)
where = IC (X, X) denotes the covariance kernel matrix, and IC (:,-) can be any positive-definite kernel functtin. The classification logits are derived as ÿ (fr (Nest) x a. Throughout the paper, we use the Radial Basis Function (RBF) kernel 42 by default. The choice between primal and dual form ridge regression depends on whether the system over-determined (more equations than unknowns) or under-determined (more unknowns than equations) 43
Details on the relationships
between primal and dual ridge regression can be found in Appendix
100
1.0
In-domain Accuracy
Linear
Primal
Dual
0.
Linear Primal
80
002
Dusl
3
10
Figure 3: Pearson correlation coefficients (CCs) for Figure 4: Comparison of in no-mmmiin accuracy 10 pairs of Nomain-prototype
(%) on each domain with three classifiers.
To empirically verify whether these non-linear projections enhances the separability of CLIP features of images from different domains, we trained three types of classifiers (denoted as Linear, Primal and Dual, respectively) on 10 domains introduced in Sec. 3.2 jointly. To compare the standard linear regression form with aforementioned two approaches, we take the averaged weight vectors as the domain-prototypes and then calculate the inter-domain Pearson correlation coefficients (CCs) between 10 pairs of domain domain-prototypess As shown in Fig. 3 the linear regression classifier exhibits high cross-domain correlations. By contrast, the RHL the primal form significantly reduces these correlations. The implicit projection provided by the kernel trick in the dual form enables better disentangling of different domains. We further evaluate the in-domain accuracy, which represents the rate of correctly classifying images into the appropriate domains. Fig. 4 shows that the in- domain accuracy is negatively correlated cross- cross-domain correlations. Both primal and dual forms demonstrate certain improvements through the projection designs, allowing for accurate classification of images into their respective domains without domain identity hint.
4.2 Regression-baste analytic incremental learning
Based on the projection approaches introduced above, propose the Regression- Analytic Incremental Learning (RAIL) method, which incorporates a ridge regression- based adapter and a training-free fusion module. The adapter progressively adapts the pre-trained CLIP to new domains, while the training-free fusion module preserves CLIP's zero-shot ability on unseen domains. An overview of RAIL is illustrated in Fig. 5] The pseudo-codes of both training and testing algorithms are provided in Appendix A
4.2.1 RAIL-Adapter
In the context of CL, in which data arrives progressively, we extend both primal and dual ridge regres- sion solutions to incremental learning manner. Our solutions identical to that obtained by joint training, which achieves absolute non-forgetting of learned knowledge. Let ((n) = (X("), Y (n)) represent the nith training set and (1m) (1m)) Y (1m)) represent the union of the training sets
5

---

(a) Inference Stage
(b) Training Stage Primal
n-f step
RAII-Adapter
photo of (object)
Sgc
Dataset
giirt
6-
CLIP
REU
Pre-trained CLIP
Imaage Encoder
Text Encoder
plane
slep
M,
Tataset
RETI
+
plant
Object
Updated RAIL RAdptte
(c) Training Stage Dual
n-f step
RAlI-Adapter
Dataget
CLIP
plane
step
M
Adapter
RAIL-
Tattseet
11
plant
Object
Updated RAIL-Adapter
Figure 5: RAIL Overview. (a) During inference, the fusion module utilizes the Zero-shot logits to identify whether test image is aligned with seen unseen classes. If classified seen class, the Fusion logits combine the RAIL-Adapter logits and the Zero-shot logits; otherwise solely rely on the Zero-shot logits. (b) Primal: at the n-th learning step, features X. extracted by CLIP's image encoder are projected to higher dimensional via RHI. and then update the parameter W and memory M, by Theorem T (c) Dual: features extracted by CLIP's image encoder update the kernel K, parameter a, and memory Ma by Theorem
from the first domains. At the n-th learning step, the optimization target for the joint training is expressed as
arg min (Im)) (1:n) w()) I A W where (lm)) & (fr (x(I)) The objective is to obtain W(n) that satisfies Eqn. without accessing data from the previous n 1 domains. For primal form, we propose to solve ((n) recursively using W(n-1) and memory matrix M("). The solution summarized as in Theorem
w(n) F'
Theorem The parameter calculated by
W(n) W(n-1)
-M - (n)) (n) ((-11)
(
(5)
an optimal solution to the optimization problem of joint training on all n domains in Eqn. M is obtained by
where
M - M =
M(n-1)
(I (I- ( M
((n) (0(")(n
(6)
Similarly, the dual parameter (n) satisfying Eqn. 4 can be obtained based on (n-1), an updating kernel K(*), and memory matrix M(n). We denote the matrix ((n) the concatenated one-hot label matrices of all domains. The solution is defined in Theorem 2
Theorem The parameter calculated by
-1 (a) (K(m) + AI)
((n)
an optimal solution to the optimization problem of joint training on all n domains in Eqn.
where
K (n-
(x!" (m) ,M ((m-1))
((n-1) 0
K
((1)
(8)
K (x(x)) M
K. [x(!" x(1))
Y (m)

---

An alternative solution is to leverage CLIP's zero-shot ability to indicate the domain of the test image. Despite CLIP's strong generalization ability across domains, certain cross- errors (i.e. misclassification into an incorrect domain) persist and do not diminish during incremental learning process. Therefore, the task of classifying images across seen domains is delegated to the RAIL- Adapter, which significantly reduces these errors, as discussed in Sec. 4.1 Consequently, CLIP's zero-shot ability is only leveraged to distinguish classes in unseen domains (i.e., or OOD) from those in seen ones (i.e., In-Distribution or ID) maintain its performance on unseen domains. We summarize this approach as RAIL-Fusion, which combines both CLIP's zero-shot and the memory matrix is given by M
( x!"IT]"
d
At each incremental learning step, the kernel matrix K updates recursively along the main diagonal, preserving the correlations among class-prototyye
from all domains. During testing, the kernel covariance between the feature of test image extracted by CLIP and memory matrix M2 is calculated to obtain the classification logits y = KC (fr (Xies)) Ma) a. Specifically, M, dynamically updates according to the data stream via concatenated plass-proootypes. These class-prototypes can be the feature embeddings X. extracted by CLIP, K-means centroids, or Gaussian Mixture Model means. We use raw feature embeddings X. by default, which is sufficient to validate our method. The complete proofs for both theorems are provided in Appendix C
4.2.2 RAIL-Fusion
Next, we introduce the fusion strategy, which leverages the refined knowledge on seen domains from the RAIL--daater while preserving CLIP's pre-trained knowledge on unseen domains. To distinguish data from different domains without any domain-identity hint, a common approach [18] is compute domain centers from glass-prototyper The test image is first assigned to a specific domain based on the distances these domain centers and then classified by domain-specific classifier. However, this method fails when statistics for unseen domains are unavailable.
logits and RAlL-Adapter logits for prediction, regardless of whether the domain of the test image is seen or unseen.
Specifically, CLIP first makes rough prediction based on its zero-shot logits, i.e., the similarity scores between image embeddings and language embeddings from the cross- dossin label set Cy:
Ya = Softmax (fx (Xies) fr (Tokenizer (iP, (P,Cx)))T)
(9)
where Y2s represents the zero-shot logits, P denotes the pre-defined prompt template, and fr and fr the CLIP text encoder and image encoder, respectively. The result determines whether the test image aligns with the seen classes (ID) that have been encountered during the incremental learning or with the unseen classes (OOD). If classified ID, the RAIL-adapter refines the rough prediction using its incrementally learned knowledge. If classified as OOD, the rough prediction is taken as the final prediction, fully relying on CLIP's s zero-shot ability. Notably, our fusion strategy guarantees that OOD images correctly classified by CLIP's zero-shot prediction will never be misclassified as ID, thereby absolutely preserving CLIP's zero-shot ability on unseen domains. In addition, to prevent the forgetting of the pre- knowledge of CLIP on domains with good zero-shot performance, we combine the zero-shot logits and AAlL-Adapeer logits with a weighted sum:
yn (1 3) Yna B8
(10)
where Yau denotes the RAIL-Adapter logits and /3 is the fusion ratio that adjusts the influence of zero-shot prediction on seen domains. The ablation study on 3 is presented in Appendix E.2
5 Experiments
We evaluate RAIL method under both X-TAIL and MTIL settings, as mentioned in Sec. 3.2 learning order is set alphabetically: Aircraft, Caltech101, DTD, EuroSAT, Flowers, Food, M OxfordPet, StanfordCars, and SUN397. Additional experiments with a random order are provided in Tab. 3 in Appendix G To ensure compatibility with different domains, we follow the common practice of sampling T6-shot training set for each domain, while using the original test set for 46). The implementation details are provided in Appendix D
The
evaluation [1 44 45 22
7

---

Caltech101
DTD
EumSAT
Flowers
MNIST
70
Primal
90
60
80
80
ZSCL
50
MoE
80
40
40
40
20
30
1
910
Figure 6: Accuracy (%) on five domains changes over all learning steps.
5.1 Comparison results
Cross-domain task-agnostic incremental learning. The performances averaged on 10 domains of RAIL and other baseline methods in the X-TAIL setting are presented in the Average column of Tab. [ Specific performances on each domain are provided in the columns named after each respective domain. We evaluate RAIL in both primal and dual forms. Zero-shot indicates the zero- shot performance of the pre-trained CLIP model on each domain. Fine-tune denotes the performance of fine-tuning both CLIP image and text encoders with joint dataset of all 10 domains, serving strong baseline for comparison.
Specifically, the primal-R RAAL outperforms the previous best one with a 6.4% improvement in "Trans- fer" accuracy, achieves an additional 7.7% in "Average" accuracy, and gains an 8.6% improvement in "Last" accuracy. The dual-RAll further surpasses the primal one by 1.2% in "Average" accuracy and 3.3% in "Last" accuracy, while maintaining consistent "Trunsfer" accuracy due to the same fusion strategy. These results indicate that RAIL has more stable transfer performance and is more robust to catastrophic forgetting, effectively preserving both knowledge from new domains and pre-trained knowledge. We repeat the experiments with random order and present the results in Tab, RAIL consistently outperforms the baselines, reaffirming the previous conclusions.
We illustrate how accuracy changes on several example domains in Fig. 6 We observe that the accuracy of RAIL remains consistent with the zero-shot results before learning the corresponding domain. Furthermore, RAIL exhibits strong cross-domain discriminative capabilities. For example, once DTD is learned, learning further new domains does not affect the accuracy on DTD. Accuracy on certain domains, like Caltech101 even improves due to the fusion module's 's ability to reduce OOD errors by learning from more domains.
Multi-domain lask-incremental learning. We follow the setting in [1] to evaluate our methods on the few-shot MTIL, comparing it against the performance of baselines reported in [15). In this context, RAIL is reduced to the structure of multiple domain-specific classifiers trained on each domain separately. The domain-ided -inantt guides the test image to the corresponding classifier for within- domann prediction. The comparison results are shown in Tab. 2 RAIL consistently outperforms previous state-of-ther oh-tth-art approaches on all three metrics. These results demonstrate that the proposed non-linear projections significantly improve the separability of features extracted by CLIP. Consequently, the ridge regression-basse classifier can effectively adapt the pre-trained model to new domains.
5.2 Discussion
Regression targets. For VLMs, aside from using one-hot labels the regression targets Y in Eqn. 4 the text embeddings generated from class labels is also a viable option [39]. We compare the "Last" accuracy of 10 domains using the dual RAll-Adapter with these two different regression targets as shown in Fig. 7al The results indicate that training with one-hot labels surpasses its counterpart with text embeddings by an average 3.8%. We emphasize that using text embeddings as targets is suboptimal compared to uniformly distributed one-hot labels. This effect is particularly notable in domains such as Aircraft, where the "Lust" accuracy with one-hot label targets outperforms that with text embedding targets by 7.5%. We argue that insufficiently semantic class names (e.g.. "707-320") result in text embeddings that are not well-dispersed in the feature space, thus compromising the classification performance.
8

---

Table 1: Comparison of different CL methods on X-TAIL for each domain in terms of "Transfer", "Average", and "Last" scores (%). The best results are highlighted with bold style.
Caltech101
Aircraft
EuroSAT
Flowers
Food101
MNIST
Sun397
DTD
Pets
Cars
Method
Average
Zero-shot
23.5
76.8
37.3
36.7
63.6
84.0
46.7
86.7
66.1
63.7
58.5
Fine-tune
39.6
93.3
68.2
89.2
95.4
85.5
95.1
84.4
77.4
72.4
80.1
Transfer
LwF 6
66.6
26.9
19.5
51.0
78.4
26.6
68.9
35.5
56.1
47.7
WiSE-FT
70.1
31.9
25.3
56.3
79.8
29.9
74.9
45.6
56.8
52.3
iCaRL
71.7
35.0
43.0
63.4
86.9
43.9
87.8
63.7
60.0
61.7
ZSCL [0]
73.3
32.6
36.8
62.1
83.8
42.1
83.6
56.5
60.2
59.0
MoE-Adapter
15
71.0
34.9
19.2
63.0
86.6
20.0
87.2
63.7
58.6
56.0
Primal-RAll
76.8
37.3
36.7
63.6
84.0
46.7
86.7
66.1
63.7
62.4
Dual-RAIL
76.8
37.3
63.6
84.0
46.7
66.1
62.4
Average
LwF
24.7
79.7
38.3
36.9
63.9
81.0
36.5
71.9
42.7
53.2
WiSE-FT
27.1
76.5
40.9
31.3
68.7
81.6
31.4
74.7
51.7
58.4
54.2
iCaRL.
25.4
72.1
37.5
51.6
65.1
87.1
59.1
88.0
63.7
60.1
61.0
ZSCL
36.0
75.0
40.7
40.5
71.0
85.3
46.3
83.3
60.7
61.5
60.0
MoE-Adapter
43.6
77.9
52.1
75.9
45.2
66.6
63.0
Primal-RAIL
42.4
89.8
55.7
68.5
84.0
83.3
65.3
85.8
67.9
64.5
70.7
Dual-RAlL
45.3
89.9
57.6
68.7
83.9
85.5
65.2
88.4
69.4
65.0
71.9
Last
LwF
20.9
83.1
47.5
38.2
75.5
84.7
50.1
78.0
75.8
74.6
62.8
WiSE-FT
21.8
76.8
42.9
20.8
77.5
84.9
30.7
76.6
75.8
72.5
58.0
iCaRL
25.5
72.1
38.9
55.4
65.5
87.3
81.9
88.6
63.6
61.5
64.0
ZSCL
33.1
75.3
43.5
35.2
74.6
87.4
50.4
84.2
77.3
73.4
63.4
MoE-Adapter
43.2
78.7
57.6
32.8
79.4
86.0
86.7
87.8
78.2
74.2
70.5
Primal-RAN RAIL
41.7
94.0
66.0
86.4
97.2
82.4
93.1
83.6
75.0
71.3
79.1
Dual-RAlL
45.3
94.2
69.0
87.0
97.2
87.2
93.0
92.4
82.5
76.3
82.4
100
LastAcuursoy (%)
80
Text Embeddinge Labels
Muli-Casssier
40
RAIl-Fusion
Atrontt
Cattes101
DTD
EUDSAT
Flowers
Food
Moist
Pata
Sun3W7
Aircrat
Cateeh101
DTD
EueSAT
Flowere
Food
Mrial
Pats
Came
(a) Comparison different regression targets.
(b) Comparison different fusion stralegies.
Figure 7: Each bar indicates the "Last" accuracy (%) on each domain after the last learning step.
Fusion strategies. The CL setting associated with multiple domains is typically decomposed into two stages: domain-identity inference and in-domain prediction [19]. As discussed in Sec. 4.2.2 an intuitive strategy for distinguishing classes from different domains in X-TAIL is to utilize CLIP's zero-shot prediction as a domain indicator, which then cooperates with multiple domain-specific classifiers to perform classification within each distinct domain. We evaluate the effectiveness of the RAIL-Fusion against this strategy by comparing the "Last" accuracy across 10 domains using the dual RAIL-Adapter. The "Transfer" accuracy remains consistent between these two strategies.
As shown Fig. 7b the RAIL-Fusion strategy outperforms the multi-classifier approach in most domains by an average of 7.5%. This improvement is due to RAIL-Fusion'' design, which focuses on distinguishing unseen classes (OOD domain) from seen classes (ID domain), rather than identifying specific domains (Sec. 4..22
This OOD detection design incrementally reduces errors as the number

---

Table Comparison with state-of of the-thh-art methods 5-shot MTIL setting terms of "Transfer". "Average". and "Last" scores (%). The best results are highlighted with bold style.
Caltech101
CIFARI00
Aircraft
EuroSAT AT
Flower
Food101
MNIST
Sun397
DTD
Pets
Cars
Method
Average
Zero-shot Fine-tune
24.3 30.6
88.4
68.2
44.6
54.9
71.0
88.5
59.6
89.0
64.7
65.2
65.3
93.5
76.8
65.1
91.7
92.9
83.3
96.6
84.9
65.4
71.3
77.5
Transfer
LwF
72.1
49.2
35.9
44.5
41.1
66.6
50.5
69.0
19.0
51.7
50.0
LwF-VR
82.2
62.5
40.1
56.3
80.0
77.6
60.8
WISE-FT
77.6
60.0
41.3
39.4
53.0
76.6
58.1
75.5
37.3
58.2
57.7
ZSCL
84.0
68.1
44.8
46.8
63.6
84.9
61.4
81.4
55.5
62.2
65.3
MoF
87.9
68.2
44.1
48.1
64.7
88.8
69.0
89.1
64.5
65.1
68.9
Primal-RAIL
88.4
68.2
44.6
54.9
71.0
88.5
59.6
89.0
64.7
65.2
69.4
Dull-RAll
88.4
68.2
44.6
54.9
71.0
88.5
59.6
89.0
64.7
65.2
69.4
Average
LwF
23.5
77.4
43.5
41.7
43.5
52.2
54.6
63.4
68.0
21.3
52.6
49.2
LwF-VR
24.9
89.1
64.2
53.4
54.3
70.8
79.2
66.5
79.2
44.1
61.6
62.5
WiSE-FT
32.0
87.7
61.0
55.8
68.1
69.3
76.8
71.5
77.6
42.0
59.3
63.7
ZSCL
28.2
88.6
66.5
53.5
56.3
73.4
83.1
56.4
82.4
57.5
62.9
64.4
MoE
30.0
89.6
73.9
58.7
69.3
79.3
76.5
89.1
65.3
65.8
71.4
Primal-RAIL
32.9
94.5
69.9
58.1
71.8
84.4
88.5
70.4
89.0
66.1
65.7
71.9
Dual-RAIL
36.0
94.2
70.9
58.8
70.6
84.3
88.5
70.3
89.7
66.5
65.8
72.3
Last
LwF
22.1
58.2
17.9
32.1
28.1
66.7
46.0
84.3
64.1
31.5
60.1
46.5
LwE-VR
22.9
89.9
59.3
57.1
57.6
79.2
78.3
77.7
83.6
60.1
69.8
66.9
WiSE-FT
30.8
88.9
59.6
60.3
80.9
81.7
77.1
94.9
83.2
62.8
70.0
71.9
ZSCL
26.8
88.5
63.7
55.7
60.2
82.1
82.6
58.6
85.9
66.7
70.4
67.4
MoE
30.1
89.3
74.9
64.0
82.3
89.4
87.1
89.0
89.1
69.5
72.5
76.1
Primal-RAIL
32.9
95.1
70.3
63.2
81.5
95.6
88.5
89.7
89.0
72.5
71.0
77.2
Dual-RAIL
36.0
94.8
71.5
64.1
79.5
95.3
88.5
89.4
91.5
74.6
71.3
77.9
of ID domains grows, making it particularly effective dynamically adapting domains. By contrast, using domain indicator maintains a consistent level of cross-domain errors associated for each domain, leading to етгог propagation in the final prediction. This issue especially critical for domains with low in-domain accuracy, where any misalignment between the domain indicator and the correct domain can significantly impact performance, These cross-domain errors mitigated in the R RAll--d thanks to its non-linear projection design.
Conclusion
In this work, we introduce the Cross-domain Task-Agnostic Incremental Learning (X-TAIL) to evaluate the preservation of pre- pre-trained knowledge and cross-domain discriminative ability in con- tinual learning context. We introduce a novel CL approach, Regression- Analytic Incremental Learning (RAIL), to improve the performance of pre-trained lision-language models progressively incoming domains, while maintaining zero-shot ability on unseen domains. We theoretically prove the absolute memorization on learned knowledge and show that the fusion module inherently avoids the forgetting VLM's zero-shot ability. Comprehensive experiments on both existing and proposed settings empirically demonstrate the superiority of our method.
7 Acknowledgement
This research was supported by the National Natural Science Foundation of China (62306117) and the Guangzhou Basic and Applied Basic Research Foundation (2024A0413681).
10

---

References
[1] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con- tinual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028-1000 2020.
[2] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
44(7):3366-338) 2021.
[3] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[4] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Elsevier, 1989.
[5] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(1))3521-3526 2017.
et al.
[6] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE Transactions Pattern Analysis and Machine Intelligence, 40(12):2935-22)) 2947, 2017.
[7] Sylvestre-Allvse Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920-15930, 2020.
[9] Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for learning world models online. arXiv preprint 2rXi::2401.13034. 2024.
[10] Zangwei Zheng. Mingyuan Ma, Kai Wang. Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19125-19136, October 2023.
[11] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-970s 2021.
[12] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- sosoon-nn--laagaee and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021.
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 9748-8763. PMLR, 2021.
[14] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- language understanding and generation. In International Conference Machine Learning, pages 12888-12900. PMLR, 2022.
[15) Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. arXiv preprint arXiv:2403.1 11549, 2024.
[16] Yen- -hnaC Hsu, Yen- Yen-Cheng Liu. Anita Ramasamy, and Zsolt Kira. Re-evaluating con- tinual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv: 1810. / /2488, 2018.
[17] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67-82, 2018.

---

[18] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre- trainrined trans- formers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695. 2022.
[19] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchi- cal decomposition of prompt-basee continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2024.
[20] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Newral computation, 3(1):79-77, 1991.
[21] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- -anguug models with feature adapters. International Journal of Computer Vision, (2(2):581-555) 2024.
[22] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))2337-2488, July 2022.
[23] Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision-language models via augmentation, weighting, and transportation. arXiv preprint wwXiv:247.006003 2024.
[24] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft, 2013.
[25] Xinxing Xu, Joey Tianyi Zhou, IvorW. Tsang, Zheng Qin, Rick Siow Mong Goh, and Yong Liu. Simple and efficient learning using privileged information, 2016.
[26] Ahmed Ben Saad, Youssef Tamaazousti, Josselin Kherroubi, and Alexis He. Where is the fake? patch-wise supervised gans for texture inpainting. In 2020 IEEE International Conference Image Processing (ICIP). IEEE, October 2020.
[27] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[28] Alexander Mamaev.
https://universet http:// com/ joseph-nelson/ waalssonnfowwwrrg flowers apr 2020. visited on 2024-05-02.
Flowers dataset.
[29] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[30] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29())141-142 2012.
[31] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3498-3505. IEEE, 2012.
[32] Afshin Dehghan, Syed Zain Masood, Guang Shu, and Enrique. G. Ortiz. View independent vehicle make, model and color recognition using convolutional neural network, 2017.
[33] Jianxiong Xiao, James Hays, Krista A Fhinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pages 3485-3492, 2010.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Kar-Ann Toh and How-Lung Eng. Between classification-erro approximation and weighted least-squares learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4):658-669) 2008.
[36] Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, pages 326-334, 1965.
[37] Guang-Bit Huang, Qin Yu Zhu, and Thee-Kheom, Siew. Extreme learning machine: theory and applications. Neurocomputing, 500- 2006.
12

---

[38] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie. Kar-Ann Toh, and Zhiping Lin.
Acil: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35:1 1602-11614, 2022.
[39] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2023.
[40] Craig Saunders, Alexander Gammerman, and Volodya Vovk. Ridge regression learning algo- rithm in dual variables. In Proceedings of the Fifteenth International Conference Machine Learning, page 515-521, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.
[41] Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods in machine learning. The Annals of Statistics, 36(3):117) 1220, 2008.
[42] Shunjie Han, Cao Qubo, and Han Meng. Parameter selection in svm with rbf kernel function. In World Automation Congress 2012, pages 1-4. IEEE, 2012.
[43] James E Gentle. Numerical linear algebra for applications in statistics. Springer Science & Business Media, 2012.
[44] Renrui Zhang. Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 2021.
[45] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[46] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[47] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong. al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
13

---

Appendix
A
Algorithm details
In this section, we summarize the training and testing procedures of RAIL in Algorithm | and respectively.
Algorithm RAIL. training
Require: domains (D(),... ..... D pre-trained CLIP model {fi, fr} Initialize: Ridge regression parameter X, random projection function @ (+), kernel function K. (+:
Dataset learning
for training batches in ( do
Extract features x = fr (x(") if Primal: then
Initialize the memory matrix M, using Eqn. 15
Obtain the primal parameter ( using Eqn. 2
end if
Dual: then
Initialize the memory matrix M label matrix C()) & kernel ( by Theorem
Obtain the dual parameter a (!) using Equ.
end if end for
Incremental learning
for (()) in (D(2).... D(N) } do
for training batches in ((1) do
Extract features (m) = fi (X)
if Primal: then
Update the memory matrix M.") (") using Eqn 6
Update the primal parameter ((a) using Equ. end if
if Dual: then
Update the memory matrix M( label matrix C()) kernel K()) by Theorem 2
Update the dual parameter (n) using Eqn.
end if end for end for
Algorithm 2 RAIL testing
Require: Test dataset Diest, CLIP model (fi,fr), trained RAIL-Adapter Rad (+), cross-domain
label CN
for Xeest E Dies do
Obtain the rough prediction yz argmax y.. by Eqn.
if yo E Cr. then
Obtain RAlL-adapter logits ÿad = Rad (/r (Xess)
Obtain predicted class based fusion logits y² argmax f fz by Eqn. 10
else
Regard vo as the final prediction
end
end for
14

---

B Connection between primal & dual ridge regression
In this section, we introduce the connection between primal and dual forms of ridge regression.
Based on the identity (P-1 + B
R
+ +R) the solution of ridge BBBBBBBBBB R- PB (BPB
regression is given by:
W
Φ + (I = + ( + +
(11)
where the former solution based on the outer-product of data and the latter one is based on the inner- product of data. Parameter W can thus be rewritten as: W = o a with a = ( + + -Y . In this way, the solution W is interpreted lie in the span of the sample-cases, even if the dimensionality of the projected features T is larger than the number of samples.
Utilizing the kernel method, we never actually require access the explicit features D which could be of indefinite dimensions. We obtain the prediction with given data x by projecting it onto the solution W,
y = (x) W = @ (x) + ($ + AI Y
K. Xx X)
(12)
where IC (x, xj) = o (x,) Φ (x,). What we require here is the choice of kernel function K. (---) instead of explicitly defining the projection function.
C Proof of Theorems
In this section, we provide two mathematical proofs for both Theorem and Theorem
2
First, prove the Theorem T from the solution of joint training datasets with primal ridge regression:
w
@ (1:n) (1m) + AI)
-1
T (1:n) ((1m)
(13)
By decoupling the n-th data from previous datasets, the (")) can be written as:
( (1))mm11)) (g)(r))
(1=--1) 0
W(n)
[0(l:n11TT
[[n]T]
+ AI
[ (1:n 1)T
(n)
y(m)
= (Im (1m-1) + + (x)
(()))((()))) ((())) -1
[ (1:n ((mm-)
((n)Ty("))
(14)
We introduce the memory matrix as the following definition:
c (1x)T (1:x) + AI) -1
(n)
(15)
which is the matrix inversion term of Eqn 13
Noticing that M (-1) = (Φ (1m--1) (1m-1) 1) + AI) -1 by Woodbury matrix identity where
(A + UC A-1 A (C-1 VA VA -U) -1 VA-1 -1 and treating Mfn-1) as A A-1 the
memory at n-th step can be further defined recursive solution:
M ( (p (I+ t
(p(m) (00) (
(16)
Thus, the parameter ("n) is derived as
w(n) = M (n) (1.m (1:n-1) M ((m)
(17)
Denotes the left submatrix M (1 H. By substituting Eqn 16 into 17
((n-1) M (n))
(m) (1)) (())) (m))T) -1
$ (()) ((n-1)
(18)
15

---

Based on the identity of (I+P) = - (I+P) P, it is further derived as:
H W(n M (n)T ( W(n-1)
(19)
Thus.
W
w(n-1)
M (
M p())
(20)
The Theorem is proved.
Next, we prove the Theorem 2 as follows. We use the few-shot features X as the class- plastpprtostpes as default. The solution joint training on n datasets with dual form ridge regression shown as:
(n)) (KC
( (1:m)
((()))
+ AI)
(l:)).
(21)
Define the memory matrix Ma as the concatenation of class-prototypes from learned domains, the memory matrix n-th step obtained by:
[M
x!")
(22)
The kernel matrix at n-th step for (") can be partitioned as:
(m)
X x (1:n)
(1:m)
K x((ln-11 xx x(()-1))
K. (x!"), (M((-1))
K. (((m) M(n-1))
K. (x!x) x
(23)
K(n-1)
10 (1)) (n) M
K. (x!"),
(n) M
(n) (m)
indicates that the kernel matrix updates recursively along the diagonal by kernel matrices of the intra-domain covariance within x! (n) and the inter-domain covariance between x(") and the memory M/--1 The kernel matrix K therefore memorizes all the covariance information of learned domains. We further denote (1m) as C(s), which is updated by:
C (n-1) 1)
(
(24)
Yn))
where the matrix (n) consists of one-hot labels that are disjoint with those in previous n - domains. In this way, the parameter (x) is solved by updating ((n) and C(*), resulting in an identical solution the one of joint learning in Eqn. 21 Thus, the Theorem is proved.
D Implementation details
In this section, we introduce the implementation details, including model configuration, hardware setup, and hyperparamter selection. We use the pre-trained CLIP [I3] model the ViT-B/16 image encoder 488 All the results are conducted on Ubuntu 20.04 with Intel Core i9-13900K CPU with a single RTX 4090Ti GPU by the average regularization parameter À over the range 10-6 10-3 .... 1 and the RBF kernel bandwidth over the range 10 10 ..... 10. The optimal values are determined by minimizing the regression error on the validation set of the first domain, without access future domains. These parameters are then fixed for all subsequent learning steps. We use the simplest prompt template "A photo of a {)." for generalization across different domains.
runs. We conduct grid search for the
16

---

79
Average Last Accuracy (%)
78
77
76
75
Hidden Layer Dimension
Figure 8: RHL dimension vs. "Last" accuracy (%) averaged on 10 domains.
82
80
78
Accuracy (%)
76
Primal Average
Dual Average
74
Primal Last
Duel Last
72
0.5
0.6
0.7
0.8
0.9
1.0
Fusion Ratio
Figure 9: Fusion ratio vs. "Average" and "Last" accuracy (%) averaged on 10 domains.
E
Ablation studies
In this section, we conduct two ablation studies to observe the average performance on 10 domains w.E.t the hidden dimension of RHL and the fusion ratio, respectively.
E.1 RHL dimension
We first ablate the hidden dimension of RHL in primal RAIL over the "Last" accuracy in Fig. It is evident that increase in the RHL dimension correlates with improved adapter's accuracy. Specifically, increasing the dimension from 1k 10k leads to notable improvements (from 73.6% to 79.0%). However, beyond the 10k threshold, the gain in accuracy becomes saturated. The dimensions of 15k and 20k result the same performance of 79.1 10. Considering the computational cost associated with higher dimensions, we set the RHL dimension 15k default in the experiments.
E.2 Fusion ratio
Additionally, we conduct ablation study of fusion ratio in terms of "Average" and "Last" accuracy. The "Transfer" accuracy is not considered here since 3 does not influence it. From Fig. 9 we observe that the best ratio 0.8 for both "Average" and "Last" scores. Note that when equals to 1, the performance on seen domains fully relies on the RAlL-Adapter. The "Last" accuracy with 0.8 surpasses one pure RAIL-Adapter performance (3 1) by 1.9% and 1.3% in primal and dual RAIL,, respectively. This result verifies our claim that the cooperation with CLIP's generalization ability can preserve the performance on confident domains (already having good zero-shot performance) and avoid overfitting limited domain exemplars.
17

---

424
223
610
467
822
635
42
624
90
564
267
685
988
467
663
61
58.
347
635
648
46
622
641
627
66
467
Notict
621
60.
624
467
Bost22
612
124
805
610
624
96.4
62.2
60.7
924
64.
16.3
6-44
RL-
12.2
87.
624
66.
98.
612
80.
644
+13
16:2
824
61:
1:11
41-
823
602
822
825
batnn Nower
cartant cara
Bottl
Luar
(8) Primal-RAIL
(b) Dual-RAll
Figure 10: Accuracy (%) of Primal-RAIL. and Dual-RAIL. in the X-TAIL setting with order-I. Each row denotes the performance on every domain after learning on one domain.
F Detailed performance of RAIL with order-I
In this section, we visualize the performance on each domain after every learning step in Fig. 10 The upper-diagonal represents the performance before learning on the corresponding domain, which remains consistent with the zero-shot performance thanks to RAIL's absolute memorization of the zero-shot ability on unseen domains. On the other hand, the performance after learning a specific domain (lower-diagonal) does not degrades. Performance learning step (e.g., Caltech-101) benefiting from our RAIL fusion module design, which reduces OOD errors more domains are learned, thereby enhancing overall accuracy.
some domains even improves with the
G Comparison of different methods on X-TAIL with order II.
In this section, we compare different methods in X-TAIL setting with random order: StanfordCars, Aireraft, OxfordPet, Food, SUN397, MNIST, Flowers, DTD, Caltech101, EuroSAT. As shown in Tab. 3 our method again outperforms previous methods on all metrics, reinforcing the conclusions presented in Sec. 5.1
H Limitation
RAIL exhibits its superior performance and efficiency for transferring pre-trained VLMs to various domains. limitation in this approach that the pre-trained VLM remains fixed, preventing any improvement in its feature extraction capability throughout the incremental learning process. A promising direction for future work to adjust the pre-trained encoder according to new data with low computational cost, thus further boosting RAIL's performance while maintain its efficiency. Beside, extending RAIL to encompass additional downstream tasks VLMs, such as image segmentation, can broaden its applicability and enhance its utility in more complex visual understanding scenarios.
18

---

Table 3: Comparison of different CL methods X-TAIL for each domain with order II in terms of "Transfer", "Average", and "Last" scores (%) The best results are highlighted with bold style.
Caltech 101
Aircraft
SUN397
MNIST
Flower
EuroSAT
Cars
Pets
Food
DTD
Method
Average
Zero-shot
66.1
23.5
86.7
84.0
63.7
46.7
63.6
37.3
76.8
36.7
58.5
Fine-tune
77.4
39.6
84.4
85.5
72.4
95.1
95.4
68.2
93.3
89.2
80.1
Transfer
LWF
20.0
74.1
79.6
58.1
34.1
48.9
27.7
64.4
15.1
46.9
WiSE-FT
21.3
79.5
83.3
61.0
39.9
56.5
29.6
68.0
20.8
51.1
ZSCL
23.0
87.2
42.1
34.6
40.9
MoE-Adapter
17.1
87.2
87.5
58.4
12.6
65.5
35.9
70.0
17.9
50.2
Primal-RAN RAla
23.5
86.7
84.0
63.7
46.7
63.5
37.3
76.8
36.7
57.7
Dual-RAll
23.5
86.7
84.0
63.7
46.7
63.5
37.3
76.8
36.7
57.7
Average
LwF
49.0
27.4
69.7
83.0
65.7
42.2
63.5
33.1
68.5
17.5
52.0
WISE-FT
57.9
29.6
85.4
51.6
35.5
23.0
56,9
ZSCL
74.4
36.4
86.7
88.7
68.9
50.0
75.1
40.1
72.5
43.7
63.6
MoE-Adapter
74.4
38.6
87.7
87.3
67.9
50.6
76.5
43.7
72.3
18.8
61.8
Primal-RAII.
77.9
40.4
85.6
83.3
68.3
62.2
76.6
45.8
80.4
41.7
66.2
Dual-RAll
82.3
43.4
90.4
86.0
71.0
62.8
76.7
46.7
80.3
41.7
68.1
Last
LWF
17.5
83.8
44.9
44.8
39.0
WiSE-FT
46.1
23.5
71.3
85.7
70.2
59.1
85.5
47.9
82.4
42.8
61.5
ZSCL
71.7
35.3
86.5
89.2
71.8
52.3
89.8
52.0
77.1
68.4
69.4
MoB-Adapter
75.1
41.1
87.9
87.1
74.1
89.7
92.6
61.2
81.0
27.4
71.7
Primal-RAIL.
77.7
41.9
86.1
83.3
71.8
91.6
97.3
66.4
94.8
86.9
79.8
Dual-RAIL
82.3
45.8
87.1
93.1
69.1
94.4
87.1
82.5
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarize our contributions in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide a complete proof of our theorem in Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our method is mainly based on a pre-trained CLIP and close-form solutions. All required equations have been provided in Sec.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
htt.g/ghhhh..caaagggggaaga999991 Regression-based - www.--nnceeeeal-eeeeee Leerninn
Justification:
We release the code at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Btaia///guiddes/oeeSubbbee ionneooooii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe our implementation details Sec. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our method is based on analytic learning, whose closed-fome solutions are not influenced by random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computer resources in Sec. D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uurrr...copppoa.cttcccaaa
Answer: [Yes]
Justification:
We preserve the anonymity with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper does not bring any societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper is based on open apen-source datasets and pre-trainn pre-trained models. There is no potential risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "vLn0NA5p4Z", "reviewer": "Reviewer_ThWm", "summary": "This paper proposes a novel setting called Cross-domain Task-Agnostic Incremental Learning (X-TAIL), in which the model is required to incrementally learn from multiple domains and test images from both seen and unseen domains without any domain identity. Additionally, the authors introduce two Regression-based Analytic Incremental Learning (RAIL) methods (primal form and dual form) and validate the effectiveness of these methods both theoretically and experimentally.", "questions": "See Weakness", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and includes clear and accurate figures and equations.\n2. The method begins with an introduction to primal and dual form ridge regression, analyzing whether these non-linear projections enhance the separability of CLIP features in images from different domains. This analysis motivates the design of the RAIL-adapter, making the overall approach easier to understand.\n3. It is intriguing to explore the regression-based analytic IL, leveraging non-linear projection functions from both primal and dual perspectives to enhance the expressiveness of features extracted by the pre-trained CLIP.", "weaknesses": "1. There is little difference in performance between the primal and dual ridge regression methods on most datasets, as shown in Figure 6. Could you provide more analysis on this?\n2. Could you compare the parameters of primal and dual ridge regression methods? What are the advantages and disadvantages of each?\n3. Please provide more explanation of Figure 2. For example, what do the different colored blocks represent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your overall supportive review of our work. Moreover, we are glad that you found this paper is well-written with good soundness and contribution to the NeurIPS community. Please see our responses addressing your specific concerns below:\n***\n>***W1** There is little difference in performance between the primal and dual ridge regression methods on most datasets, as shown in Figure 6. Could you provide more analysis on this?*\n\n**A1** Thank you for pointing out that the differences in performance between the primal and dual forms of ridge regression might appear minimal in Figure 6. This issue primarily stems from the **scale of the axes** used in the plot, which does not sufficiently emphasize small variations in performance metrics. We understand that this could potentially obscure meaningful distinctions in the comparative evaluation. To provide a clearer picture of the performance differences, **we kindly refer you to Table 1, where the last performance of both methods are detailed.** The results are also copied below. Here, you can observe that the dual method outperforms the primal method by 3.3% on average in the last performance.\n\n|             | Aircraft | Caltech101 | DTD  | EuroSAT | Flowers | Food101 | MNIST | Pets  | Cars  | Sun397 | Average |\n|-------------|:--------:|:----------:|:----:|:-------:|:-------:|:-------:|:-----:|:-----:|:-----:|:------:|:-------:|\n| Primal-RAIL |   41.7   |     94.0   | 66.0 |   86.4  |   **97.2**  |   82.4  |  **93.1** |  83.6 |  75.0 |  71.3  |   79.1  |\n| Dual-RAIL   |   **45.3**   |     **94.2**   | **69.0** |   **87.0**  |   **97.2**  |   87.2  |  93.0 |  **92.4** |  **82.5** |  **76.3**  |   **82.4**  |\n\n***\n>***W2** Could you compare the parameters of primal and dual ridge regression methods? What are the advantages and disadvantages of each?*\n\n**A2 Parameter Differences**:\n- **Primal Form**: The parameter matrix **W** in primal ridge regression is determined by the dimensionality $d$ of the projected features, which is a $d \\times c$ matrix where $c$ represents the number of classes. The dimension $d$ remains fixed, meaning that the complexity associated with the solution of parameter matrix **W** does not increase as the amount of data grows.\n- **Dual Form**: The parameter matrix **α** in dual ridge regression is a $N \\times c$ matrix, where $N$ is the number of data points involved in constructing the kernel matrix. As training progresses and more data is incorporated, *i.e.*, $N$ increases, the computational demands of the method get impacted.\n\n**Advantages and Disadvantages**:\n- **Primal Ridge Regression**:\n    - **Advantages**:\n        - Since the dimensionality of features $d$ is fixed, the computational complexity, mainly determined by matrix inversion, does not increase as data accumulates.\n    - **Disadvantages**:\n        - Requires explicit specification of an appropriate projection function,  which can be challenging to design optimally.\n        - A sufficiently large $d$ is necessary to ensure that the features are expressive enough for accurate classification, which can be computationally expensive.\n- **Dual Ridge Regression**:\n    - **Advantages**:\n        - Leveraging the advantage of the kernel trick, the projection function can be implicitly defined based on the choice of the kernel function, avoiding direct computation and storage of high-dimensional features. It allows for the selection of appropriate kernel functions based on different tasks, enhancing the adaptability of the method.\n        - The method is computationally efficient when the amount of data is small, especially in the few-shot case.\n    - **Disadvantages**:\n        - As the amount of data increases, the computational complexity of kernel matrix ($N \\times c$) inversion becomes significant, potentially making the method computational expensive for very large incoming dataset.\n\n***\n>***W3** Please provide more explanation of Figure 2. For example, what do the different colored blocks represent?*\n\n**A3** Blocks in Figure 2 represents the classification performance of the model across all domains after learning each specific domain. \n- The **blue blocks** in the upper-right matrix indicate the model's zero-shot performance on these domains before learning these domains. These blocks are utilized to evaluate the ability of continual learning methods to *“transfer”* zero-shot capabilities of the VLM. \n- The **gray and green blocks** under the diagonal show the classification performance on these domains after the model has learned these domains. Specifically, the **green blocks** represent the model's “*last”* performance on these domains after learning all domains. These blocks evaluate the adaptability of continual learning methods to new domains and their ability to retain the newly acquired knowledge throughout the continual learning process.\n- The **orange blocks** indicate the *“average”* performance across all time stamps for each domain.\n***\nWe will update the explanations mentioned above into the caption of Figure 2 and the corresponding subsection to clarify the metrics and facilitate easier understanding. In light of these clarifications, would you consider increasing your score for our paper? Otherwise, could you let us know any additional changes you would like to see in order for this work to be accepted?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1aHXgYx17", "reviewer": "Reviewer_Tz45", "summary": "Continual Learning (CL) with Vision-Language Models (VLMs) has a challenge that the model must not forget both previously learned knowledge and VLM pre-trained knowledge. Existing methods realize this by using large-scale reference data or domain identity hints, which is not practical. This paper proposed RAIL, which uses a recursive ridge-regression-based adapter to address the previously seen knowledge while using the zero-shot ability of VLMs for unseen classes. In addition, This paper proposes a novel task X-TAIL that evaluates the model on seen and unseen domains without any domain-identifier hints. The proposed method is empirically evaluated on MTIL and X-TAIL and shows state-of-the-art performance.", "questions": "+ Why the average Last score of Dual-RAIL is higher than that of fine-tune? Are fine-tune results not the performance in the so-called ideal situation? \n+ How are the existing methods evaluated? How were the methods before the advent of CLIP (e.g. LwF, iCaRL) applied for CLIP? How were the MTIL methods applied for X-TAIL?", "limitations": "+ As I mentioned in Weaknesses, the proposed method based on ridge regression requires high computational complexity, so it cannot be applicable when the amount of data is very large.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ This paper tackled a novel and practical problem, X-TAIL. It is certainly unrealistic to assume that we always have domain information, and it is more natural to consider all classes.\n+ The proposed method seemed to have some technical novelty. It is worth commending that the proposed method is not merely an application of adapter methods such as CLIP-Adapter and Tip-Adapter, but has a recursive structure that makes it the method with affinity for CL.\n+ The proposed method showed higher performance on both MTIL and X-TAIL than existing methods.", "weaknesses": "+ I feel some parts of the paper were difficult to understand.\n  + The explanation of the proposed method in the introduction was so unclear that I could not understand the method at all from the introduction section. The author should modify the manuscripts to prepare some belief figures to support the reader’s understanding.\n  + I couldn’t understand how to identify ID and OOD classes in the proposed method until I read the appendix. The main manuscript should be self-contained as we can understand it without reading the appendix.\n+ The proposed method was evaluated on the few-shot setting in the MTIL evaluation, but it was not evaluated on the normal MTIL.\n  + The proposed method should be evaluated on the full data setting following the previous works[9, 14].\n+ Since the proposed method is based on ridge regression, there is concern about computational complexity when the amount of data increases. It is necessary to compare and discuss with existing methods, taking into account not only the recognition performance but also the computational efficiency.\n+ Typos:\n  + eq(1) ||Y-X_e||^2_F -> ||Y-X_eW||^2_F", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for detailed feedback and valuable suggestions.\n***\n>***W1** Concerns of paper understanding.*\n\n**A1** Thank you for bringing up your confusions. Based on your suggestion, we decide to make following revisions in the final version.\n- We will update the introduction with clearer explanation of both primal and dual methods. We will include a detailed description of how the CLIP extracted features are projected into higher dimensional space and then classified with ridge regression.\n- We agree that Algorithm 2 in the Appendix provides a clearer explanation of the process for identifying ID and OOD classes. We will move this algorithm to the main manuscript to improve paper's self-containment.\n***\n>***W2** Evaluation on full data MTIL.*\n\n**A2** We have added experiments of comparing the proposed method with SOTA methods ZSCL[1] and Moe-Adapter[2] under the **full data MTIL setting**, reported **Experiment 1 in the general response.** Our methods consistently surpass existing methods under this setting. We will include the experiments in the final version.\n***\n>***W3&Limitation** Concerns for large data.*\n\n**A3** Thank you for pointing out the concerns regarding the computational complexity of our method. Our method requires **only forward pass** and matrix computations for parameter updates, **without back-propagation or gradient updates**. Since the backward pass typically accounts for approximately 70% of the entire foward-backward training time[3], and our method trains on every dataset over **only one epoch**, it is faster than back-propagation-based methods. Regarding the impact of the amount of data, we provide the following analysis:\n\n**Primal Ridge Regression**: As detailed in Theorem 1 of our manuscript, the computational complexity for updating parameters mainly relies on the matrix inversion operation in Eqn. 6. Assuming $N$ is the amount of training data and $d$ is the dimension of features, the size of $\\mathbf{Φ}^{(n)}$ is $N \\times d$, and the size of $\\mathbf{M}_p^{(n)}$ is $d \\times d$. The large size of the $\\mathbf{Φ}^{(n)}\\mathbf{M}_p^{(n−1)}\\mathbf{Φ}^{(n)T}$ matrix, which is affected by data amount, can be managed by using mini-batches. By breaking down the $\\mathbf{Φ}^{(n)}$ into manageable batches and updating $\\mathbf{M}_p^{(n)}$ iteratively, our primal method prevents computational complexity from escalating with large data amount.\n\n**Dual Ridge Regression**: We acknowledge that the limitation of high computational complexity in large data scenarios is inherent to kernel-based methods (*e.g.*, kernel SVM), including our dual method. As stated in Theorem 2, the complexity mainly lies in inverting the $\\mathbf{K}^{(n)} + \\lambda \\mathbf{I}$ matrix, where the size of $\\mathbf{K}^{(n)}$ is $N \\times N$, determined by the amount of data involved in constructing the kernel, making it more suited for few-shot settings. However, existing techniques, such as Nyström approximation[4], can mitigate the computational complexity of kernel-based methods under large data scenario, enhancing the scalability of our method.\n\nFurthermore, to justify the computational efficiency of our method, we added experiments on the comparison of computational times between our proposed RAIL methods and the SOTA methods ZSCL[1] and Moe-Adapter[2] in the X-TAIL setting, reported in **Experiment 2 in the general response**. Our methods are significantly faster than the SOTA methods.\n***\n>***W4** Typos.*\n\n**A4** We thank the reviewer for pointing out the typo and we will fix it in the final version.\n***\n>***Q1** Concerns for finetune.*\n\n**A5** We chose finetune as a baseline as it represents a classic method for transferring CLIP to downstream tasks. However, it is important to clarify that finetune is **NOT** the upper bound or the ideal situation for our method.\n\nFirst, our method differs fundamentally from finetune in terms of optimization. We use ridge regression to achieve a closed-form solution for parameter updates, while finetune relies on back-propagation. The trainable parameters in finetune are those of CLIP encoders, while our method freezes the encoders and only trains additional adapter parameters instead. Notably, our method is proved to be an optimal solution to the optimization problem of joint training, as evidenced by Theorems 1 and 2, making our method equivalent to its own continual learning upper bound.\n\nSecond, in the few-shot setting, finetuning the whole encoders of CLIP could potentially lead to overfitting. By contrast, our method freezes the encoders and only trains the adapter via ridge regression, which can overcome overfitting with appropriate regularization parameter, mitigating this risk. Thus, it is reasonable for our method to outperform finetune.\n***\n>***Q2** How are existing methods evaluated?*\n\n**A6** We replicated existing methods with open-source codes from ZSCL[1] and Moe-Adapter[2] for evaluation. For methods before the advent of CLIP, we adopted implementations from ZSCL's open-source repository. For instance, applying iCaRL to CLIP involves maintaining an exemplar set and updating representations, consistent with the original iCaRL paper. In the X-TAIL setting, we eliminated the specification of domain identity for all methods to ensure a fair comparison between baselines and our method.\n***\nGiven these clarifications, would you consider raising your score for our paper?\n***\n***Reference***\n\n[1] Zheng, Zangwei, et al. \"Preventing zero-shot transfer degradation in continual learning of vision-language models.\" *ICCV*. 2023.\n\n[2] Yu, Jiazuo, et al. \"Boosting continual learning of vision-language models via mixture-of-experts adapters.\" *CVPR*. 2024.\n\n[3] Huo, Z., Gu, B. and Huang, H. \"Decoupled parallel backpropagation with convergence guarantee.\" *ICML*. PMLR, 2018.\n\n[4] Chen, Y. and Yang, Y.. \"Fast statistical leverage score approximation in kernel ridge regression.\" *International Conference on Artificial Intelligence and Statistics*. PMLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FwoeunSXL8", "reviewer": "Reviewer_rqEC", "summary": "This paper proposes a Regression-based Analytic Incremental Learning (RAIL). It utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Additionally, the paper introduces Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. The paper theoretically proves RAIL’s absolute memorization on incrementally learned domains. Experiment results affirm RAIL’s state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings.", "questions": "Please refer to the Weakness.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed RAIL adopts traditional machine learning techniques (e.g., Primal and Dual forms) for dealing with continual learning is novel.\n2. The idea of RAIL’s absolute memorization based on analytic techniques is very appealing. Upon a good pre-trained network, the forgetting problem no longer exists, and this is very rare in the continual learning community.\n3. The paper is easy to follow overall.\n4. It was difficult to handle CIL problems in multi-modal CIL. This paper extends the continual learning scenario from task incremental to class incremental without task ID. This is a good contribution.\n5. Experiments and settings are overall well formulated.", "weaknesses": "1. Could you explain the main difference between primal and dual forms in the CIL problem? It could be quite diffcult in this community to understand.\n2. Experiemnts seems a little bit thin in the manucript. Perhaps try to move some of the experiments in the appendix to the main content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback on our work, and further thank the reviewer for finding our work novel in adopting traditional machine learning techniques for continual learning and appealing in its approach to absolute memorization based on analytic techniques. Below, we address the reviewer's concerns in turn:\n***\n>***W1** Could you explain the main difference between primal and dual forms in the CIL problem? It could be quite difficult in this community to understand.*\n\n**A1** In the proposed method, the primary distinction between the primal and dual forms lies in the differences between primal ridge regression and dual ridge regression: the former **explicitly** projects CLIP extracted features into a higher dimensional space, while the latter does so **implicitly** (refer to Appendix B). For tackling the CIL problem, we extended both forms of ridge regression to iterative solutions by Theorem 1 and Theorem 2, respectively. We summarize the difference of two methods as follows.\n\n- **Primal Ridge Regression**:\n    - **Advantages**:\n        - Since the dimensionality $d$ is fixed, the computational complexity, mainly determined by matrix inversion, does not increase as data accumulates.\n    - **Disadvantages**:\n        - Requires explicit specification of an appropriate projection function,  which can be challenging to design optimally.\n        - A sufficiently large $d$ is necessary to ensure that the features are expressive enough for accurate classification, which can be computationally expensive.\n- **Dual Ridge Regression**:\n    - **Advantages**:\n        - Leveraging the advantage of the kernel trick, the projection function can be implicitly defined based on the choice of the kernel function, avoiding direct computation and storage of high-dimensional features. It allows for the selection of appropriate kernel functions based on different tasks, enhancing the adaptability of the method.\n        - The method is computationally efficient when the amount of data is small, especially in the few-shot case.\n    - **Disadvantages**:\n        - As the amount of data increases, the computational complexity of kernel matrix ($N \\times c$) inversion becomes significant, potentially making the method computational expensive for very large incoming dataset.\n***\n>***W2** Experiemnts seems a little bit thin in the manucript. Perhaps try to move some of the experiments in the appendix to the main content.*\n\n**A2** We agree with your suggestion to enrich the Experiment section of the main manuscript. We will move a portion of the experiments in the Appendix into the main content for the final version.\n***\nBased on these additional clarifications, we hope you can keep your support of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjPbvgtaCe", "reviewer": "Reviewer_ZU8x", "summary": "The paper points out that current VLM-based incremental learning tasks face the issue of text being limited to the corresponding task. It aims to propose a method that can achieve better incremental classification performance on a broader range of texts. \n\nSpecifically, the paper proposes Regression-based Analytic IncrementalLearning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner.", "questions": "1. Additionally, during the fine-tuning stage, is each task fine-tuned using only the text corresponding to that task to ensure no label leakage occurs during the training of each task?\n2. Why is the setting for Order2 different from existing methods? I am concerned about the actual performance of this method.\n3. How does Formula 1 compute one-hot with embedding, given that $Y$ and $X_e$ are of different dimensions? Could you clarify the content of the formula clearly?", "limitations": "I am primarily concerned about the application scenarios and practical significance of the proposed setting, as well as the insufficient experimental validation.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work introduces a Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting, which evaluates the performance on a broader range of texts about VLM's continual learning ability. \n2. The paper proposes a framework RAIL, which can incrementally transfer a pre-trained VLM to multiple domains while evaluating the model's performance on both seen and unseen domains.", "weaknesses": "1. Compared to existing methods, one dataset is missing. Why was CIFAR-100 removed from the forgetting benchmark? Is it because the method performs poorly on this dataset?\n2. Determining whether a sample is OOD relies on learned class labels, which inevitably use label (text) information. I am concerned that this could lead to label leakage.\n3. Figure 2d's graph is consistent with ZSCL, only styled differently, which I find unnecessary. It may not highlight the novelty of the X-TAIL setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback, which we believe will improve our final manuscript. Please see our responses to the helpful points raised in your review below:\n***\n\n>***W1** Compared to existing methods, one dataset is missing. Why was CIFAR-100 removed from the forgetting benchmark? Is it because the method performs poorly on this dataset?*\n\n**A1** We appreciate your concern regarding the exclusion of CIFAR-100 from the X-TAIL setting. \nWe have added experiments of including CIFAR-100 dataset in the X-TAIL setting. Due to the space limitation, please refer to the **Experiment 3 in the general response**. Our methods surpass existing SOTA methods on CIFAR-100 across all metrics.\n\nHowever, the primary purpose of X-TAIL is to evaluate a model's cross-domain discriminative capabilities while transferring continually to distinct domains. **CIFAR-100 was excluded because it includes many classes that overlap with those in other domains**; for instance, the 'forest' class in CIFAR-100 overlaps with 'broadleaf forest' in EuroSAT. During testing, if an image from EuroSAT labeled as 'broadleaf forest' is classified simply as 'forest,' this would be factually correct but considered incorrect in the evaluation metrics. To prevent the redundancy of learning overlapping classes and to maintain the integrity of the setting, CIFAR-100 was not included in X-TAIL. We will include a detailed explanation in our manuscript for clarification.\n***\n>***W2** Determining whether a sample is OOD relies on learned class labels, which inevitably use label (text) information. I am concerned that this could lead to label leakage.*\n\n**A2** We would like to respectfully clarify that our method does **NOT** lead to label leakage. We determine whether a sample is OOD (*i.e.*, belonging to unlearned domain) during the testing process by checking if the CLIP zero-shot predicted class belongs to the set of classes learned by the adapter. If the zero-shot result of CLIP falls within the recorded set, the test image is classified as ID; otherwise, it is classified as OOD. This process does not involve any exposure to the labels or domains of the test images, **thus does NOT lead to label leakage.**\n\nWe will emphasize this aspect more explicitly in the final version to ensure there is no confusion.\n***\n>***W3** Figure 2d's graph is consistent with ZSCL, only styled differently, which I find unnecessary. It may not highlight the novelty of the X-TAIL setting.*\n\n**A3** We acknowledge that the metric presented in this figure is not a novelty of our work but rather an essential aspect of understanding the metrics involved in the X-TAIL setting. The styling similarity to ZSCL is intended solely for clarity and consistency in presentation, not as a highlight of novelty. Our intent is to facilitate reader understanding of the setting's metrics, as evidenced by the confusion expressed by Reviewer ThWm. We will add another citation to ZSCL in the caption of Figure 2 in the final version for further clarification.\n***\n>***Q1** Additionally, during the fine-tuning stage, is each task fine-tuned using only the text corresponding to that task to ensure no label leakage occurs during the training of each task?*\n\n**A4** During the fine-tuning stage, we ensure that labels (texts) from future tasks are not accessible to current task, thereby preventing any label leakage problem.\n***\n>***Q2** Why is the setting for Order2 different from existing methods? I am concerned about the actual performance of this method.*\n\n**A5** The order 2 of X-TAIL is randomly shuffled. In response to your concerns, we have conducted additional experiments as outlined in **Experiment 4 in the general response**, following the Order 2 sequence in ZSCL paper[1]. Our methods consistently surpass existing methods, similar to the results observed in the two orders discussed in the manuscript. This demonstrates the robustness of our methods to variations in task order.\n***\n>***Q3** How does Formula 1 compute one-hot with embedding, given that Y and Xe are of different dimensions? Could you clarify the content of the formula clearly?*\n\n**A6** We apologize for the confusion regarding Formula 1. This was indeed a typo in our manuscript. $\\mathbf{X}_e$ should be multiplied by a weight matrix $\\mathbf{W}$ to map it to the same dimension as $\\mathbf{Y}$. We will correct this formula according to your observation and the suggestion provided by Reviewer Tz45. \n***\n>***Limitation** I am primarily concerned about the application scenarios and practical significance of the proposed setting, as well as the insufficient experimental validation.*\n\n**A7** The X-TAIL setting primarily addresses a significant limitation in the existing continual learning scenarios for Vision-Language Models, specifically within the MTIL setting where specifying the domain information of test images is necessary. The contribution of our setting is also acknowledged by Reviewer Tz45. To further validate the effectiveness of our method, we have conducted **additional experiments in the general response**. If you have any questions about our setting, we would be glad to provide additional clarification during the discussion.\n***\nBased on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?\n***\n***Reference***\n\n[1] Zheng, Zangwei, et al. \"Preventing zero-shot transfer degradation in continual learning of vision-language models.\" *ICCV*. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate for all reviewers' insightful and valuable comments. We thank Reviewer ThWm and Reviewer rqEC, who agree that our work is **well-written** and **easy to follow**. We are pleased that Reviewer Tz45 finds **our setting more realistic**, recognizing our efforts to address the requirement for domain information in the existing setting. We are grateful for the recognition from Reviewer ThWm, Reviewer Tz45 and Reviewer rqEC of the **novelty** of our approach and its effectiveness in **addressing the problem of forgetting in continual learning through a recursive solution**.

We have provided detailed, point-by-point responses to address all comments and concerns raised by the reviewers. Additionally, we have conducted the following experiments to further validate our method and address the issues raised.

**1. Full-Shot MTIL Setting**. Following the previous works[1,2], we additionally evaluated our methods on full data MTIL setting. We compared our methods with SOTA methods, ZSCL[1] and MoE-Adapter[2].

||Aircraft|Caltech101|Cifar100|DTD|EuroSAT|Flowers|Food101|Mnist|Pets|Cars|Sun397|***Average***|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|**Transfer**||||||||||||
|ZSCL|--|86.0|67.4|**45.4**|50.4|69.1|87.6|**61.8**|86.8|60.1|**66.8**|68.1|
|MoE-Adapter|--|87.9|**68.2**|44.4|49.9|70.7|**88.7**|59.7|64.5|64.5|65.5|68.9|
|Primal-RAIL|--|**88.4**|**68.2**|44.6|**54.9**|**71.0**|88.5|59.6|**89.0**|**64.7**|65.2|**69.4**|
|Dual-RAIL|--|**88.4**|**68.2**|44.6|**54.9**|**71.0**|88.5|59.6|**89.0**|**64.7**|65.2|**69.4**|
|**Average**||||||||||||
|ZSCL|45.1|92.0|80.1|64.3|79.5|81.6|**89.6**|**75.2**|88.9|64.7|**68.0**|75.4|
|MoE-Adapter|50.2|91.9|**83.1**|69.4|78.9|84.0|89.1|73.7|89.3|67.7|66.9|76.7|
|Primal-RAIL|51.9|95.8|80.1|70.3|81.1|86.1|89.0|73.9|**90.2**|68.4|66.4|77.6|
|Dual-RAIL|**52.5**|**96.0**|80.6|**70.4**|**81.3**|**86.3**|89.1|73.9|**90.2**|**68.5**|66.5|**77.8**|
|**Last**||||||||||||
|ZSCL|40.6|92.2|81.3|70.5|94.8|90.5|**91.9**|98.7|**93.9**|85.3|**80.2**|83.6|
|MoE-Adapter|49.8|92.2|**86.1**|78.1|95.7|94.3|89.5|98.1|89.9|81.6|80.0|85.0|
|Primal-RAIL|51.9|96.5|82.8|80.0|96.0|98.7|89.7|**98.8**|93.3|84.8|78.7|86.5|
|Dual-RAIL|**52.5**|**96.8**|83.3|**80.1**|**96.4**|**99.0**|89.9|**98.8**|93.5|**85.5**|79.2|**86.8**|
***
**2. Computational Efficiency**. We added experiments on the computational efficiency by comparing the computational times of our proposed RAIL methods with SOTA methods, ZSCL[1] and MoE-Adapter[2], in the X-TAIL setting. Our methods are significantly **faster** due to their **one-epoch** nature. (Hardware: i9-13900K & RTX 4090)

||Real time| 
|:-------------:|:--------:|
|ZSCL| 514m40.163s|
|Moe-Adapter|47m2.319s|
| Primal-RAIL|**4m0.071s** |
| Dual-RAIL|4m13.200s|
***
**3. Inclusion of CIFAR-100 in X-TAIL setting**. We compared our methods with SOTA methods, ZSCL[1] and MoE-Adapter[2].

||Aircraft |Caltech101|Cifar100|DTD| EuroSAT | Flowers | Food101|Mnist | Pets | Cars | Sun397 |***Average***|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|**Transfer**|||||||||||||
|ZSCL|--|63.7|37.2|32.1|15.8|60.1|82.9|32.1|82.6| 53.3|53.6|51.3|
|MoE-Adapter|--|64.2|35.9| 32.9|17.3|60.6|**86.6**|23.0|**87.2**|63.7|57.1|52.9|
|Primal-RAIL|--|**69.7**|**37.3**|**36.5**|**36.6**|**60.7**|84.0|**46.6**| 86.7 |**66.1**|**62.5**|**58.7**|
|Dual-RAIL|--|**69.7**|**37.3**|**36.5**|**36.6**|**60.7**|84.0|**46.6**| 86.7 |**66.1**|**62.5**|**58.7**|
|**Average**|||||||||||||
|ZSCL|33.4|57.9|41.0|37.7|20.3|68.1|84.0|36.1|82.0|57.7|55.2|52.1|
|MoE-Adapter|42.4|66.4|55.3|49.0|38.3|74.9|**86.2**|46.7|87.4|66.2|58.4|61.0|
|Primal-RAIL|42.4|88.5 |57.1|55.7|64.7|80.7|83.0|62.9|84.8|68.7|63.7|68.4|
|Dual-RAIL|**45.0**|**88.8**|**57.8**|**56.8**|**66.2**|**81.0**|85.2|**63.4**|**87.8**|**68.9**|**64.7**|**69.6**|
|**Last**|||||||||||||
|ZSCL|31.4|59.6|43.9|39.7|28.4|71.6|86.4|40.7|82.6|77.0|70.8|57.5|
|MoE-Adapter|41.8|66.2|59.5|53.7|45.9|84.3|85.8|86.8|87.7|76.2|71.7|69.1|
|Primal-RAIL|41.9|94.0|73.7|67.8|84.4|97.0|83.4|92.6|86.9|75.7|71.4|79.0|
|Dual-RAIL|**45.2**|**94.4**|**74.7**|**70.7**|**87.3**|**97.9**|**86.5**|**92.8**|**91.9**|**81.7**|**76.7**|**81.8**|
***
**4. X-TAIL setting with another order**. We evaluated our methods in the X-TAIL setting with the same order as MTIL order 2, consistent with previous works[1,2]. Our methods are compared with SOTA methods, ZSCL[1] and MoE-Adapter[2].

||Cars|Food101|Mnist|Pets|Flowers|Sun397|Aircraft|Caltech101|DTD|EuroSAT|***Average***|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|**Transfer**|||||||||||||
|ZSCL|--|**87.7**|37.3|84.0|**63.6**|63.1|19.7|71.1|34.3|**39.5**|55.6|
|MoE-Adapter|--|83.7|20.6|87.2|62.2|58.4|18.0|70.1|35.6|19.9|50.6|
|Primal-RAIL|--|84.0| **46.7** | **86.7** | 63.5 | **63.7** | **23.5** | **76.8** | **37.3** | 36.7 | **57.7**|
|Dual-RAIL|--|84.0| **46.7** | **86.7** | 63.5 | **63.7** | **23.5** | **76.8** | **37.3** | 36.7 | **57.7** |
|**Average**|||||||||||||
|ZSCL|75.2|**88.9**|48.6|86.8|77.6|67.9|27.7|72.7|38.5|**42.3**|62.6|
|MoE-Adapter|75.4|85.3|64.0|87.4|80.2|65.6|27.6|72.6|40.9|22.2|62.1|
|Primal-RAIL | 78.0 | 83.2 | 65.1 | 88.4 | 83.6 | 67.5 | 31.0 | 80.9 | 43.1 | 41.7 | 66.3 |
|Dual-RAIL | **81.8** | 85.8 | **65.5** | **89.5** | **83.9** |**69.5** | **31.6**| **81.0** |  **43.9** | 41.8| **67.4** |
|**Last**|||||||||||||
|ZSCL|71.9|**88.8**|50.6|88.9|87.2|71.1|38.6|76.1|55.6|67.7|69.7|
|MoE-Adapter|75.1|86.0|79.2|87.4|88.7|72.7|42.2|78.4|61.6|51.3|72.3|
|Primal-RAIL|77.8|83.3|91.4|86.3|97.0|71.6|42.1|94.8|66.3|86.9|79.8|
|Dual-RAIL|**81.8**|86.3|**92.8**|**91.3**|**97.6**|**75.6**|**45.3**|**95.1**|**70.7**|**87.5**|**82.4**|
***
***Reference***

[1] Zheng, Zangwei, et al. "Preventing zero-shot transfer degradation in continual learning of vision-language models." *ICCV*. 2023.

[2] Yu, Jiazuo, et al. "Boosting continual learning of vision-language models via mixture-of-experts adapters." *CVPR*. 2024.

Meta-review: Reviewers agree on acceptance based on the practical problem setting, technical novelty, and state-of-the-art performance. The authors are encouraged to include discussions and results in the rebuttal to the final version.
Decision: Accept (poster)