Title: Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Continual Learning

Incremental learning

Analytic learning

Cross-domain

Vision-Language model
Abstract: Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose Regression-based Analytic Incremental Learning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data.
Additionally, we introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint.
We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning.

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Abstract

Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose R egression-based A nalytic I ncremental L earning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data. Additionally, we introduce Cross -domain T askA gnostic I ncremental L earning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint. We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-ofthe-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/ Regression-based-Analytic-Incremental-Learning .

1 Introduction

Continual learning (CL) [1, 2, 3] is a crucial area in machine learning, which requires a learner to incrementally learn new data instead of training from scratch. The main challenge in CL is known as catastrophic forgetting [4], where learning new knowledge results in the forgetting of the old one. To this end, various CL approaches [5, 6, 7, 8, 3, 9] have been proposed to solve the forgetting issue. As a typical CL setting, Class-Incremental Learning (CIL) (Fig. 1 (a)) aims to achieve robust discriminability on all seen classes. Despite the advancements, existing approaches mainly focus on classifying images only from seen classes, thereby limiting the model's generalizability.

Consequently, Zheng et al. [10] proposed Multi-domain Task-Incremental Learning (MTIL), which cooperates CL with the zero-shot ability of Vision-Language Models (VLMs) [11, 12, 13, 14] such as CLIP [13]. This integration equips models with the ability to classify domains they have not

Learning

Sequence

(a) Traditional CIL

(b) MTIL

Dataset

n

Dataset

i

Dataset 1

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

(c) X-TAIL

A photo of a

Pickup

A photo of a

Pickup

A photo of a

A photo of a

A photo of a

A photo of a

Cheetah

A photo of a

Cheetah

A photo of a

Cheetah

A photo of a

Pickup

CIL

CIL

ZSCL

ZSCL

ZSCL

RAIL

RAIL

RAIL

CIL

一

1

yet encountered, enhancing their generalizability across multiple domains (Fig. 1 (b)). Several methods [10, 15] have been specifically designed for MTIL, in which the model is required to retain both the incrementally learned knowledge during CL and the zero-shot ability of VLMs. However, these methods require a domain-identity hint to indicate the specific domain of the test image, which is often not applicable in real-world scenarios [7]. Additionally, the use of reference datasets during training is necessary to maintain pre-trained VLMs' zero-shot performance.

To address the aforementioned limitations, we introduce R egression-based A nalytic I ncremental L earning (RAIL), a novel approach that incrementally learns new knowledge and performs effectively on both seen and unseen domains. Specifically, we leverage non-linear projection functions from both primal and dual perspectives to enhance the expressiveness of features extracted by the pre-trained CLIP. It endows the learner with the ability to classify images in a cross-domain label set without any domain-identity hint. In the incremental learning process, RAIL utilizes a ridge regression-based adapter that updates the parameters recursively. This is identical to learning on all encountered domains at once, achieving absolute memorization on learned domains. Additionally, we freeze the pre-trained CLIP and design a training-free fusion module to determine whether the test data belongs to seen or unseen domains. This strategy absolutely preserves CLIP's zero-shot ability on unseen domains, meeting practical requirements for models deployed in dynamic environments.

To demonstrate the effectiveness of our method, we propose Cross -domain T askA gnostic I ncremental L earning (X-TAIL) setting as illustrated in Fig. 1 (c). Particularly, X-TAIL requires CL methods to incrementally transfer a pre-trained VLM to multiple domains while evaluating the model's performance on both seen and unseen domains. Moreover, domain hints are forbidden in X-TAIL, making it more realistic and challenging [3]. As a result, effective CL methods must classify a test image into the correct domain and class simultaneously.

Our contributions are summarized as follows:

We propose a new CL method RAIL to incrementally adapt the pre-trained VLM to multiple domains without forgetting both pre-trained and incrementally learned knowledge.

To meet the practical scenario where CL methods need to sequentially learn data from new domains and classify images across these domains, we propose a new setting X-TAIL to evaluate the preservation of VLM's zero-shot ability and the adaptability to new domains.

We theoretically prove the RAIL's absolute memorization on incrementally learned domains and demonstrate that the zero-shot ability of the pre-trained VLM on unseen domains is absolutely preserved.

We empirically show that the proposed method achieves state-of-the-art performances on both existing MTIL and the novel X-TAIL settings.

2 Related work

Early CL methods focused on Task-Incremental Learning (TIL) [16], where a task-id is given during testing. Subsequently, a more practical and challenging setting of Class-Incremental Learning (CIL) [7] was proposed, where the access to the task-id is forbidden at inference time. Methods for CIL must therefore distinguish between all classes encountered in learned tasks. More recently, Zheng et al. [10] proposed Multi-Domain Task-Incremental Learning (MTIL), which is especially designed to evaluate CL methods with pre-trained VLMs. In MTIL, a pre-trained VLM continually adapts to multi-domain tasks. The performance on both seen and unseen tasks measure the retention of both incrementally acquired and pre-trained knowledge. However, it still requires the task-id to create specific domain label space at inference time. Apart from them, X-TAIL combines the challenges of both CIL and MTIL, in which the model learns new classes from various incoming domains and distinguishes between both seen and unseen classes without any domain-identity.

Prevailing continual learning methods include replay-based, distillation-based, regularization-based, and architecture-based approaches [3]. Replay-based methods such as iCaRL [7] typically store a small portion of the previous task data as exemplars. The model is then trained jointly on new task data and the saved exemplars to preserve the previous knowledge. Distillation-based methods such as LwF [6] use either weight or function regularization to transfer knowledge from the previous model to the current model for knowledge distillation. Regularization-based methods such as ZSCL [10] penalize the shift of either model parameter or feature space by adding a regularization term to the cross-entropy loss function. To preserve the robustness of the strong pre-trained model without access to the pre-trained dataset, ZSCL utilizes large-scale reference datasets to regularize the parameter space. Architecture-based methods [17, 18, 19] expand the model by constructing task-specific parameters to avoid inter-task interference. For example, MoE-Adapters [15] cooperates the pretrained CLIP with mixture of experts (MoE) [20] to learn from different domains. By leveraging a reference dataset to initialize a task-id indicator, it enables the model to distinguish unseen tasks from seen ones.

The aforementioned methods either neglect the forgetting issue of pre-trained knowledge or require multiple iterations and large-scale reference datasets for training, making it challenging to efficiently adapt to new data in continual learning scenarios. By contrast, RAIL employs an analytical solution that achieves the optimum in a single epoch without additional reference data, ensuring its efficiency.

3 Cross-domain task-agnostic incremental learning

3.1 Problem setting

We define Cross-domain Task-Agnostic Incremental Learning (X-TAIL) as follows. Given a pre-trained VLM, the learner is required to incrementally transfer it to N different domains { D (1) , D (2) , ..., D ( N ) } . Each domain D ( n ) = { ( x ( n ) j , y ( n ) j ) } | D ( n ) | j =1 is available only during the n -th learning step. The class labels y ( n ) j ∈ C ( n ) label from the incrementally learned domain D ( n ) are added to the set of seen class labels. During inference at all steps, the learner attempts to classify input images from any domain without the domain-identity hint. In other words, the ground-truth label of the test image belongs to C N = C L ∪ C U , where C L = ⋃ n i =0 C ( i ) label is the union of seen class labels from all previous learning steps and C U is the set of unseen class labels.

Similar to the task-id in TIL, the domain-identity hint allows the learner to classify input data within the label space of a specific domain during evaluation. Essentially, the learner knows the domain of test images, which is far from real-world application scenarios. For instance, the learner is supposed to predict an image as the class of husky from all possible classes C N = {car, bus, ..., churros, donuts, ..., husky, beagle, bulldog, ...} . However, if the domain-identity hint is given, the learner only needs to predict from a limited subset C dog = {husky, beagle, bulldog, ...} , which is simpler but less realistic compared to practical applications. Therefore, we extend our setting to a task-agnostic scenario. Specifically, the learner predicts images from C N , the union of any potential class labels, without any domain hint.

3.2 Datasets

In X-TAIL's cross-domain setting, the learner should encompass as extensive data distributions as possible. Following previous work [10, 15, 21, 13, 22, 23], we select 10 different imageclassification datasets from different domains for our setting: Aircraft [24], Caltech101 [25], DTD [26], EuroSAT [27], Flowers [28], Food [29], MNIST [30], OxfordPet [31], StanfordCars [32], and SUN397 [33]. Specifically, to prevent the redundancy of learning overlapping classes and to maintain the integrity of the setting, CIFAR-100 [34] was excluded because it includes many classes that overlap with those in other domains. As a result, CL methods under X-TAIL should discriminate images from a total of 1 , 100 classes across all domains.

3.3 Evaluation metrics

We adopt the evaluation metrics from [10] for our setting. As illustrated in Fig. 2, each column represents the performance on a specific domain after each learning step, while the rows correspond to the learning sequence. In traditional CL settings, only the results in the lower diagonal where the learner has been exposed to the exemplars from the test domains are measured. Nevertheless, X-TAIL extends this evaluation to cover the entire matrix, recording performances across both learned and unlearned domains. The 'Average' metric, averaged on the orange blocks, indicates the average accuracy of all learning steps across all domains. The gray and green blocks under the diagonal show the classification performance on these domains after the model has learned these domains. Specifically, the green blocks represent the model's last performance on these domains after learning all domains. The 'Last' metric, which is the average of the

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

D₁

D₁

D₂

D₃

D₄

D₂

D₃

Dn₋₁

Dn₋₁

Dn

Dn

Evaluation Domain ID

Learning Step

avg.

avg.

avg.

avg.

avg.

Transfer

Last

Average

green blocks, reflects the learner's adaptability to new domains. Additionally, the blue blocks in the upper-right matrix indicate the model's zero-shot performance on these domains before learning these domains. The average of these blocks, referred to as the 'Transfer' metric, measures the extent to which the zero-shot ability is preserved throughout incremental learning.

4 Approach

4.1 Motivation

While CLIP demonstrates the generalizability of zero-shot classification, it still struggles in certain unfamiliar domains [21]. Leveraging CLIP's robust feature extraction capabilities, linear probe offers a straightforward approach to transfer the CLIP to these domains [13]. Among various linear solutions, Ridge Regression (RR) provides an effective classification strategy by mapping the image features onto one-hot-label targets [35]. Given a pre-trained CLIP image encoder f I and a dataset D = { ( X , Y ) } , where X is the tensor of training images and Y is the matrix of corresponding one-hot labels, the predicted logits and the optimization problem are defined as:

where X e = f I ( X ) denotes the CLIP extracted features, W is the classifier parameter, and λ is the regularization parameter.

In the context of X-TAIL, the classifier needs to distinguish a wide range of classes from different domains. However, the extracted CLIP features of images from different domains suffer from certain cross-domain correlations, leading to limited domain discriminability. Based on Cover's theorem [36], one promising approach [37, 38, 39] to enhance the linear separability of features is to project the features into a higher-dimensional space via some non-linear projection function. We explore this non-linear projection function from two perspectives:

Primal form ridge regression. Following [38], we use a Randomly-initialized Hidden Layer (RHL) to project raw features to a higher dimensional space. By explicitly defining the projection function

as ϕ ( · ) , the classifier parameter is determined as follows:

where Φ = ϕ ( X e ) . In this way, ϕ ( · ) is fixed throughout the training process.

Dual form ridge regression [40]. Instead of manually designing the projection function, we utilize the Kernel method [41] to implicitly define ϕ ( · ) based on the inner-product nature of dual form ridge regression. Depending on the choice of kernel function, this approach allows for an infinite projection dimension, which is unachievable through any explicit definition. The dual form solution is defined as:

where K = K ( X , X ) denotes the covariance kernel matrix, and K ( · , · ) can be any positive-definite kernel function. The classification logits are derived as ˆ y = K ( f I ( x test ) , X ) α . Throughout the paper, we use the Radial Basis Function (RBF) kernel [42] by default. The choice between primal and dual form ridge regression depends on whether the system is over-determined (more equations than unknowns) or under-determined (more unknowns than equations) [43]. Details on the relationships between primal and dual ridge regression can be found in Appendix B.

1

2

3

4

5

6

7

8

9 10

1

2

3

4

5

6

7

8

9

10

Linear

1

2

3

4

5

6

7

8

9 10

Primal

1

2

3

4

5

6

7

8

9 10

Dual

0.2

0.4

0.6

0.8

1.0

1

2

3

4

5

6

7

8

9

10

90

95

100

In-domain Accuracy (%)

Linear

Primal

Dual

To empirically verify whether these non-linear projections enhances the separability of CLIP features of images from different domains, we trained three types of classifiers (denoted as Linear, Primal and Dual, respectively) on 10 domains introduced in Sec. 3.2 jointly. To compare the standard linear regression form with aforementioned two approaches, we take the averaged weight vectors as the domain-prototypes and then calculate the inter-domain Pearson correlation coefficients (CCs) between 10 pairs of domain-prototypes. As shown in Fig. 3, the linear regression classifier exhibits high cross-domain correlations. By contrast, the RHL in the primal form significantly reduces these correlations. The implicit projection provided by the kernel trick in the dual form enables better disentangling of different domains. We further evaluate the in-domain accuracy, which represents the rate of correctly classifying images into the appropriate domains. Fig. 4 shows that the indomain accuracy is negatively correlated to cross-domain correlations. Both primal and dual forms demonstrate certain improvements through the projection designs, allowing for accurate classification of images into their respective domains without domain identity hint.

4.2 Regression-based analytic incremental learning

Based on the projection approaches introduced above, we propose the Regression-based Analytic Incremental Learning (RAIL) method, which incorporates a ridge regression-based adapter and a training-free fusion module. The adapter progressively adapts the pre-trained CLIP to new domains, while the training-free fusion module preserves CLIP's zero-shot ability on unseen domains. An overview of RAIL is illustrated in Fig. 5. The pseudo-codes of both training and testing algorithms are provided in Appendix A.

4.2.1 RAIL-Adapter

In the context of CL, in which data arrives progressively, we extend both primal and dual ridge regression solutions to an incremental learning manner. Our solutions are identical to that obtained by joint training, which achieves absolute non-forgetting of learned knowledge. Let D ( n ) = { X ( n ) , Y ( n ) } represent the n -th training set and D (1: n ) = { X (1: n ) , Y (1: n ) } represent the union of the training sets

Pre-trained

CLIP

Seen Classes

(a) Inference Stage

Text

Encoder

Image

Encoder

plane

car

girl

...

A photo of

a

{object}

I₁

I₂

T₁

T₂

T₃

T₄

T₅

T₆

T

k

T

i

...

...

T₁

T₂

T₃

T₄

T₅

T₆

T

k

T

i

...

...

...

...

...

...

...

...

Object i

Object 3

...

Mask

=

RAIL-

Adapter

n-1

step

n

step

(c) Training Stage - Dual

plane

Dataset

n-1

plant

Dataset

n

CLIP

CLIP

RHL

RHL

(b) Training Stage - Primal

/u1D6BD

W

/uni207F⁻¹

RAIL-Adapter

Updated RAIL-Adapter

/u1D6BD

X

ₑ

X

ₑ

X

ₑ

X

ₑ

W

/uni207F

M

p

/uni207F⁻¹

n-1

step

n

step

plane

Dataset

n-1

plant

Dataset

n

CLIP

CLIP

K

/uni207F⁻¹

/u1D736

/uni207F⁻¹

RAIL-Adapter

Updated RAIL-Adapter

K

/uni207F

/u1D736

/uni207F

M

d

/uni207F⁻¹

from the first n domains. At the n -th learning step, the optimization target for the joint training is expressed as

where Φ (1: n ) = ϕ ( f I ( X (1: n ) )) . The objective is to obtain W ( n ) that satisfies Eqn. 4 without accessing data from the previous n -1 domains. For primal form, we propose to solve W ( n ) recursively using W ( n -1) and a memory matrix M ( n ) p . The solution is summarized as in Theorem 1.

Theorem 1 The parameter calculated by

is an optimal solution to the optimization problem of joint training on all n domains in Eqn. 4, where M ( n ) p is obtained by

Similarly, the dual parameter α ( n ) satisfying Eqn. 4 can be obtained based on α ( n -1) , an updating kernel K ( n ) , and a memory matrix M ( n ) d . We denote the matrix C ( n ) as the concatenated one-hot label matrices of all n domains. The solution is defined in Theorem 2.

Theorem 2 The parameter calculated by

is an optimal solution to the optimization problem of joint training on all n domains in Eqn. 4, where

At each incremental learning step, the kernel matrix K updates recursively along the main diagonal, preserving the correlations among class-prototypes from all domains. During testing, the kernel covariance between the feature of test image extracted by CLIP and memory matrix M d is calculated to obtain the classification logits ˆ y = K ( f I ( x test ) , M d ) α . Specifically, M d dynamically updates according to the data stream via concatenated class-prototypes. These class-prototypes can be the feature embeddings X e extracted by CLIP, K-means centroids, or Gaussian Mixture Model means. We use raw feature embeddings X e by default, which is sufficient to validate our method. The complete proofs for both theorems are provided in Appendix C.

4.2.2 RAIL-Fusion

Next, we introduce the fusion strategy, which leverages the refined knowledge on seen domains from the RAIL-Adapter while preserving CLIP's pre-trained knowledge on unseen domains. To distinguish data from different domains without any domain-identity hint, a common approach [18] is to compute domain centers from class-prototypes. The test image is first assigned to a specific domain based on the distances to these domain centers and then classified by a domain-specific classifier. However, this method fails when statistics for unseen domains are unavailable.

An alternative solution is to leverage CLIP's zero-shot ability to indicate the domain of the test image. Despite CLIP's strong generalization ability across domains, certain cross-domain errors ( i.e., misclassification into an incorrect domain) persist and do not diminish during incremental learning process. Therefore, the task of classifying images across seen domains is delegated to the RAILAdapter, which significantly reduces these errors, as discussed in Sec. 4.1. Consequently, CLIP's zero-shot ability is only leveraged to distinguish classes in unseen domains ( i.e. , Out-Of-Distribution or OOD) from those in seen ones ( i.e. , In-Distribution or ID) to maintain its performance on unseen domains. We summarize this approach as RAIL-Fusion, which combines both CLIP's zero-shot logits and RAIL-Adapter logits for prediction, regardless of whether the domain of the test image is seen or unseen.

Specifically, CLIP first makes a rough prediction based on its zero-shot logits, i.e., the similarity scores between image embeddings and language embeddings from the cross-domain label set C N :

where ˆ y zs represents the zero-shot logits, P denotes the pre-defined prompt template, and f T and f I are the CLIP text encoder and image encoder, respectively. The result determines whether the test image aligns with the seen classes (ID) that have been encountered during the incremental learning or with the unseen classes (OOD). If classified as ID, the RAIL-adapter refines the rough prediction using its incrementally learned knowledge. If classified as OOD, the rough prediction is taken as the final prediction, fully relying on CLIP's zero-shot ability. Notably, our fusion strategy guarantees that OOD images correctly classified by CLIP's zero-shot prediction will never be misclassified as ID, thereby absolutely preserving CLIP's zero-shot ability on unseen domains. In addition, to prevent the forgetting of the pre-trained knowledge of CLIP on domains with good zero-shot performance, we combine the zero-shot logits and RAIL-Adapter logits with a weighted sum:

where ˆ y ad denotes the RAIL-Adapter logits and β is the fusion ratio that adjusts the influence of zero-shot prediction on seen domains. The ablation study on β is presented in Appendix E.2.

5 Experiments

We evaluate RAIL method under both X-TAIL and MTIL settings, as mentioned in Sec. 3.2. The learning order is set alphabetically: Aircraft, Caltech101, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397. Additional experiments with a random order are provided in Tab. 3 in Appendix G. To ensure compatibility with different domains, we follow the common practice of sampling a 16-shot training set for each domain, while using the original test set for evaluation [21, 44, 45, 22, 46]. The implementation details are provided in Appendix D.

0 1 2 3 4 5 6 7 8 9 10

70

80

90

Caltech101

0 1 2 3 4 5 6 7 8 9 10

30

40

50

60

70

DTD

0 1 2 3 4 5 6 7 8 9 10

20

40

60

80

EuroSAT

0 1 2 3 4 5 6 7 8 9 10

60

70

80

90

Flowers

0 1 2 3 4 5 6 7 8 9 10

20

40

60

80

MNIST

Primal

Dual

ZSCL

MoE

5.1 Comparison results

Cross-domain task-agnostic incremental learning. The performances averaged on 10 domains of RAIL and other baseline methods in the X-TAIL setting are presented in the Average column of Tab. 1. Specific performances on each domain are provided in the columns named after each respective domain. We evaluate RAIL in both primal and dual forms. Zero-shot indicates the zeroshot performance of the pre-trained CLIP model on each domain. Fine-tune denotes the performance of fine-tuning both CLIP image and text encoders with a joint dataset of all 10 domains, serving as a strong baseline for comparison.

Specifically, the primal-RAIL outperforms the previous best one with a 6 . 4% improvement in 'Transfer' accuracy, achieves an additional 7 . 7% in 'Average' accuracy, and gains an 8 . 6% improvement in 'Last' accuracy. The dual-RAIL further surpasses the primal one by 1 . 2% in 'Average' accuracy and 3 . 3% in 'Last' accuracy, while maintaining consistent 'Transfer' accuracy due to the same fusion strategy. These results indicate that RAIL has more stable transfer performance and is more robust to catastrophic forgetting, effectively preserving both knowledge from new domains and pre-trained knowledge. We repeat the experiments with a random order and present the results in Tab. 3. RAIL consistently outperforms the baselines, reaffirming the previous conclusions.

We illustrate how accuracy changes on several example domains in Fig. 6. We observe that the accuracy of RAIL remains consistent with the zero-shot results before learning the corresponding domain. Furthermore, RAIL exhibits strong cross-domain discriminative capabilities. For example, once DTD is learned, learning further new domains does not affect the accuracy on DTD. Accuracy on certain domains, like Caltech101, even improves due to the fusion module's ability to reduce OOD errors by learning from more domains.

Multi-domain task-incremental learning. We follow the setting in [15] to evaluate our methods on the few-shot MTIL, comparing it against the performance of baselines reported in [15]. In this context, RAIL is reduced to the structure of multiple domain-specific classifiers trained on each domain separately. The domain-identity guides the test image to the corresponding classifier for within-domain prediction. The comparison results are shown in Tab. 2. RAIL consistently outperforms previous state-of-the-art approaches on all three metrics. These results demonstrate that the proposed non-linear projections significantly improve the separability of features extracted by CLIP. Consequently, the ridge regression-based classifier can effectively adapt the pre-trained model to new domains.

5.2 Discussion

Regression targets. For VLMs, aside from using one-hot labels as the regression targets Y in Eqn. 4, the text embeddings generated from class labels is also a viable option [39]. We compare the 'Last' accuracy of 10 domains using the dual RAIL-Adapter with these two different regression targets as shown in Fig. 7a. The results indicate that training with one-hot labels surpasses its counterpart with text embeddings by an average of 3 . 8% . We emphasize that using text embeddings as targets is suboptimal compared to uniformly distributed one-hot labels. This effect is particularly notable in domains such as Aircraft, where the 'Last' accuracy with one-hot label targets outperforms that with text embedding targets by 7 . 5% . We argue that insufficiently semantic class names ( e.g. , '707-320') result in text embeddings that are not well-dispersed in the feature space, thus compromising the classification performance.

Aircraft

Caltech101

DTD

EuroSAT

Flowers

Food101

Mnist

Pets

Cars

Sun397

40

60

80

100

Text Embeddings

One-hot Labels

(a) Comparison of different regression targets.

Aircraft

Caltech101

DTD

EuroSAT

Flowers

Food101

Mnist

Pets

Cars

Sun397

40

60

80

100

Multi-Classifier

RAIL-Fusion

(b) Comparison of different fusion strategies.

Figure 7: Each bar indicates the 'Last' accuracy (%) on each domain after the last learning step.

Fusion strategies. The CL setting associated with multiple domains is typically decomposed into two stages: domain-identity inference and in-domain prediction [19]. As discussed in Sec. 4.2.2, an intuitive strategy for distinguishing classes from different domains in X-TAIL is to utilize CLIP's zero-shot prediction as a domain indicator, which then cooperates with multiple domain-specific classifiers to perform classification within each distinct domain. We evaluate the effectiveness of the RAIL-Fusion against this strategy by comparing the 'Last' accuracy across 10 domains using the dual RAIL-Adapter. The 'Transfer' accuracy remains consistent between these two strategies.

As shown in Fig. 7b, the RAIL-Fusion strategy outperforms the multi-classifier approach in most domains by an average of 7 . 5% . This improvement is due to RAIL-Fusion's design, which focuses on distinguishing unseen classes (OOD domain) from seen classes (ID domain), rather than identifying specific domains (Sec. 4.2.2). This OOD detection design incrementally reduces errors as the number

of ID domains grows, making it particularly effective in dynamically adapting to new domains. By contrast, using a domain indicator maintains a consistent level of cross-domain errors associated for each domain, leading to error propagation in the final prediction. This issue is especially critical for domains with low in-domain accuracy, where any misalignment between the domain indicator and the correct domain can significantly impact performance. These cross-domain errors are mitigated in the RAIL-Adapter thanks to its non-linear projection design.

6 Conclusion

In this work, we introduce the Cross-domain Task-Agnostic Incremental Learning (X-TAIL) to evaluate the preservation of pre-trained knowledge and cross-domain discriminative ability in a continual learning context. We introduce a novel CL approach, Regression-based Analytic Incremental Learning (RAIL), to improve the performance of pre-trained vision-language models on progressively incoming domains, while maintaining its zero-shot ability on unseen domains. We theoretically prove the absolute memorization on learned knowledge and show that the fusion module inherently avoids the forgetting of VLM's zero-shot ability. Comprehensive experiments on both existing and proposed settings empirically demonstrate the superiority of our method.

References

Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences , 24(12):1028-1040, 2020.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(7):3366-3385, 2021.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation , volume 24, pages 109-165. Elsevier, 1989.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(12):2935-2947, 2017.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems , 33:15920-15930, 2020.

Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for learning world models online. arXiv preprint arXiv:2401.13034 , 2024.

Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 19125-19136, October 2023.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems , 34:9694-9705, 2021.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. arXiv preprint arXiv:2403.11549 , 2024.

Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:1810.12488 , 2018.

Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 67-82, 2018.

Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems , 35:5682-5695, 2022.

Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems , 36, 2024.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 132(2):581-595, 2024.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, July 2022.

Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision-language models via augmentation, weighting, and transportation. arXiv preprint arXiv:2407.04603 , 2024.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft, 2013.

Xinxing Xu, Joey Tianyi Zhou, IvorW. Tsang, Zheng Qin, Rick Siow Mong Goh, and Yong Liu. Simple and efficient learning using privileged information, 2016.

Ahmed Ben Saad, Youssef Tamaazousti, Josselin Kherroubi, and Alexis He. Where is the fake? patch-wise supervised gans for texture inpainting. In 2020 IEEE International Conference on Image Processing (ICIP) . IEEE, October 2020.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

Alexander Mamaev. Flowers dataset. https://universe.roboflow.com/ joseph-nelson/flowers , apr 2020. visited on 2024-05-02.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3498-3505. IEEE, 2012.

Afshin Dehghan, Syed Zain Masood, Guang Shu, and Enrique. G. Ortiz. View independent vehicle make, model and color recognition using convolutional neural network, 2017.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485-3492, 2010.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Kar-Ann Toh and How-Lung Eng. Between classification-error approximation and weighted least-squares learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 30(4):658-669, 2008.

Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers , pages 326-334, 1965.

Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing , 70(1-3):489-501, 2006.

Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems , 35:11602-11614, 2022.

Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems , 36, 2023.

Craig Saunders, Alexander Gammerman, and Volodya Vovk. Ridge regression learning algorithm in dual variables. In Proceedings of the Fifteenth International Conference on Machine Learning , page 515-521, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.

Thomas Hofmann, Bernhard Schölkopf, and Alexander J. Smola. Kernel methods in machine learning. The Annals of Statistics , 36(3):1171 - 1220, 2008.

Shunjie Han, Cao Qubo, and Han Meng. Parameter selection in svm with rbf kernel function. In World Automation Congress 2012 , pages 1-4. IEEE, 2012.

James E Gentle. Numerical linear algebra for applications in statistics . Springer Science & Business Media, 2012.

Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 , 2021.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Appendix

A Algorithm details

In this section, we summarize the training and testing procedures of RAIL in Algorithm 1 and 2, respectively.

B Connection between primal & dual ridge regression

In this section, we introduce the connection between primal and dual forms of ridge regression.

Based on the identity ( P -1 + B ⊤ R -1 B ) -1 B ⊤ R -1 = PB ⊤ ( BPB ⊤ + R ) -1 , the solution of ridge regression is given by:

where the former solution is based on the outer-product of data and the latter one is based on the innerproduct of data. Parameter W can thus be rewritten as: W = Φ ⊤ α with α = ( ΦΦ ⊤ + λ I n ) -1 Y . In this way, the solution W is interpreted to lie in the span of the sample-cases, even if the dimensionality of the projected features Φ is larger than the number of samples.

Utilizing the kernel method, we never actually require access to the explicit features Φ , which could be of indefinite dimensions. We obtain the prediction with given data x by projecting it onto the solution W ,

where K ( x i , x j ) = ϕ ( x i ) ⊤ ϕ ( x j ) . What we require here is the choice of kernel function K ( · , · ) instead of explicitly defining the projection function.

C Proof of Theorems

In this section, we provide two mathematical proofs for both Theorem 1 and Theorem 2.

First, we prove the Theorem 1 from the solution of joint training on n datasets with primal ridge regression:

By decoupling the n -th data from previous datasets, the W ( n ) can be written as:

We introduce the memory matrix as in the following definition:

which is the matrix inversion term of Eqn 13.

Noticing that M ( n -1) p = ( Φ (1: n -1) ⊤ Φ (1: n -1) + λ I ) -1 , by Woodbury matrix identity where ( A + UCV ) -1 = A -1 -A -1 U ( C -1 + VA -1 U ) -1 VA -1 and treating M ( n -1) p as A -1 , the memory at n -th step can be further defined as a recursive solution:

Thus, the parameter W ( n ) is derived as

Denotes the left submatrix M ( n ) p Φ (1: n -1) ⊤ Y (1: n -1) as H . By substituting Eqn 16 into 17,

Based on the identity of ( I + P ) -1 = I -( I + P ) -1 P , it is further derived as:

Thus,

The Theorem 1 is proved.

Next, we prove the Theorem 2 as follows. We use the few-shot features X e as the class-prototypes as default. The solution of joint training on n datasets with dual form ridge regression is shown as:

Define the memory matrix M d as the concatenation of class-prototypes from learned domains, the memory matrix at n -th step is obtained by:

The kernel matrix at n -th step for α ( n ) can be partitioned as:

It indicates that the kernel matrix updates recursively along the diagonal by kernel matrices of the intra-domain covariance within X ( n ) e and the inter-domain covariance between X ( n ) e and the memory M ( n -1) d . The kernel matrix K therefore memorizes all the covariance information of learned domains. We further denote Y (1: n ) as C ( n ) , which is updated by:

where the matrix Y ( n ) consists of one-hot labels that are disjoint with those in previous n -1 domains. In this way, the parameter α ( n ) is solved by updating K ( n ) and C ( n ) , resulting in an identical solution to the one of joint learning in Eqn. 21. Thus, the Theorem 2 is proved.

D Implementation details

In this section, we introduce the implementation details, including model configuration, hardware setup, and hyperparamter selection. We use the pre-trained CLIP [13] model of the ViT-B/16 image encoder [48]. All the results are conducted on Ubuntu 20.04 with Intel Core i9-13900K CPU with a single RTX 4090Ti GPU by the average of 3 runs. We conduct a grid search for the regularization parameter λ over the range 10 -6 , 10 -5 , ..., 1 and the RBF kernel bandwidth over the range 10 -6 , 10 -5 , ..., 10 . The optimal values are determined by minimizing the regression error on the validation set of the first domain, without access to future domains. These parameters are then fixed for all subsequent learning steps. We use the simplest prompt template 'A photo of a {}. ' for generalization across different domains.

1k 2k

5k

10k

15k

20k

Hidden Layer Dimension

74

75

76

77

78

79

Average Last Accuracy (%)

0.5

0.6

0.7

0.8

0.9

1.0

Fusion Ratio

68

70

72

74

76

78

80

82

Accuracy (%)

Primal Average

Primal Last

Dual Average

Dual Last

E Ablation studies

In this section, we conduct two ablation studies to observe the average performance on 10 domains w.r.t. the hidden dimension of RHL and the fusion ratio, respectively.

E.1 RHL dimension

We first ablate the hidden dimension of RHL in primal RAIL over the 'Last' accuracy in Fig. 8. It is evident that an increase in the RHL dimension correlates with improved adapter's accuracy. Specifically, increasing the dimension from 1k to 10k leads to notable improvements (from 73.6% to 79.0%). However, beyond the 10k threshold, the gain in accuracy becomes saturated. The dimensions of 15k and 20k result the same performance of 79.1%. Considering the computational cost associated with higher dimensions, we set the RHL dimension to 15k as default in the experiments.

E.2 Fusion ratio

Additionally, we conduct an ablation study of fusion ratio β in terms of 'Average' and 'Last' accuracy. The 'Transfer' accuracy is not considered here since β does not influence it. From Fig. 9, we observe that the best ratio is 0 . 8 for both 'Average' and 'Last' scores. Note that when β equals to 1 , the performance on seen domains fully relies on the RAIL-Adapter. The 'Last' accuracy with β = 0 . 8 surpasses the one of pure RAIL-Adapter performance ( β = 1 ) by 1.9% and 1.3% in the primal and dual RAIL, respectively. This result verifies our claim that the cooperation with CLIP's generalization ability can preserve the performance on its confident domains (already having good zero-shot performance) and avoid overfitting on limited domain exemplars.

(a) Primal-RAIL

(b) Dual-RAIL

-100

-100

aircraft

43.0

76.8

37.3

36.7

63.5

84.0

46.7

86.7

66.1

63.7

aircraft

45.3

76.8

37.3

36.7

63.5

84.0

46.7

86.7

66.1

63.7

caltech101

42.6

88.4

37.3

36.7

63.5

84.0

46.7

86.7

66.1

63.7

caltech101

45.5

88.3

37.3

36.7

63.5

84.0

46.7

86.7

66.1

63.7

80

80

pap

42.8

90.1

56.1

36.7

63.5

84.0

46.7

86.7

66.1

63.7

pap

45.3

90.1

58.3

36.7

63.5

84.0

46.7

86.7

66.1

63.7

eurosat-

42.7

90.2

56.9

77.2

63.5

84.0

46.7

86.7

66.1

63.7

45.2

90.1

58.6

77.4

63.5

84.0

46.7

86.7

66.1

63.7

60

60

oxford_flowers

42.5

91.1

58.1

77.3

97.8

84.0

46.7

86.7

66.1

63.7

oxford_flowers

45.1

91.2

60.6

77.6

97.6

84.0

46.7

86.7

66.1

63.7

food101

42.4

91.1

61.1

77.4

97.6

82.9

46.7

86.7

66.1

63.7

food101

45.2

91.2

63.7

77.6

97.5

86.9

46.7

86.7

66.1

63.7

40

40

mnist -

42.4

91.1

61.2

85.6

97.7

82.8

93.2

86.7

66.1

63.7

mnist -

45.3

91.3

63.8

86.0

97.5

86.9

92.9

86.7

66.1

63.7

oxford_pets

41.9

91.8

61.8

85.7

97.6

82.7

93.5

83.8

66.1

63.7

oxford_pets

45.2

92.0

63.9

86.0

97.5

86.9

93.1

92.3

66.1

63.7

20

20

stanford_cars-

42.2

93.3

60.9

85.4

97.7

82.1

93.5

83.5

75.4

63.7

stanford_cars-

45.3

93.5

63.8

86.0

97.4

86.9

93.1

92.2

82.4

63.7

sun397

41.7

94.0

66.0

86.4

97.2

82.4

93.1

83.6

75.0

71.3

sun397

45.3

94.2

69.0

87.0

97.2

87.2

93.0

92.4

82.5

76.3

S

O

F Detailed performance of RAIL with order-I

In this section, we visualize the performance on each domain after every learning step in Fig. 10. The upper-diagonal represents the performance before learning on the corresponding domain, which remains consistent with the zero-shot performance thanks to RAIL's absolute memorization of the zero-shot ability on unseen domains. On the other hand, the performance after learning a specific domain (lower-diagonal) does not degrades. Performance on some domains even improves with the learning step (e.g., Caltech-101), benefiting from our RAIL fusion module design, which reduces OOD errors as more domains are learned, thereby enhancing overall accuracy.

G Comparison of different methods on X-TAIL with order II.

In this section, we compare different methods in X-TAIL setting with a random order: StanfordCars, Aircraft, OxfordPet, Food, SUN397, MNIST, Flowers, DTD, Caltech101, EuroSAT. As shown in Tab. 3, our method again outperforms previous methods on all metrics, reinforcing the conclusions presented in Sec. 5.1.

H Limitation

RAIL exhibits its superior performance and efficiency for transferring pre-trained VLMs to various domains. A limitation in this approach is that the pre-trained VLM remains fixed, preventing any improvement in its feature extraction capability throughout the incremental learning process. A promising direction for future work is to adjust the pre-trained encoder according to new data with low computational cost, thus further boosting RAIL's performance while maintain its efficiency. Beside, extending RAIL to encompass additional downstream tasks of VLMs, such as image segmentation, can broaden its applicability and enhance its utility in more complex visual understanding scenarios.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarize our contributions in both the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Sec. H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a complete proof of our theorem in Sec. C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our method is mainly based on a pre-trained CLIP and close-form solutions. All required equations have been provided in Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release the code at https://github.com/linghan1997/ Regression-based-Analytic-Incremental-Learning .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe our implementation details in Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our method is based on analytic learning, whose closed-form solutions are not influenced by random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the computer resources in Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We preserve the anonymity with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper does not bring any societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper is based on open-source datasets and pre-trained models. There is no potential risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points out that current VLM-based incremental learning tasks face the issue of text being limited to the corresponding task. It aims to propose a method that can achieve better incremental classification performance on a broader range of texts. \n\nSpecifically, the paper proposes Regression-based Analytic IncrementalLearning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner.\n\nSTRENGTHS:\n1. This work introduces a Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting, which evaluates the performance on a broader range of texts about VLM's continual learning ability. \n2. The paper proposes a framework RAIL, which can incrementally transfer a pre-trained VLM to multiple domains while evaluating the model's performance on both seen and unseen domains.\n\nWEAKNESSES:\n1. Compared to existing methods, one dataset is missing. Why was CIFAR-100 removed from the forgetting benchmark? Is it because the method performs poorly on this dataset?\n2. Determining whether a sample is OOD relies on learned class labels, which inevitably use label (text) information. I am concerned that this could lead to label leakage.\n3. Figure 2d's graph is consistent with ZSCL, only styled differently, which I find unnecessary. It may not highlight the novelty of the X-TAIL setting.\n\nQUESTIONS:\n1. Additionally, during the fine-tuning stage, is each task fine-tuned using only the text corresponding to that task to ensure no label leakage occurs during the training of each task?\n2. Why is the setting for Order2 different from existing methods? I am concerned about the actual performance of this method.\n3. How does Formula 1 compute one-hot with embedding, given that $Y$ and $X_e$ are of different dimensions? Could you clarify the content of the formula clearly?\n\nLIMITATIONS:\nI am primarily concerned about the application scenarios and practical significance of the proposed setting, as well as the insufficient experimental validation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper proposes a Regression-based Analytic Incremental Learning (RAIL). It utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Additionally, the paper introduces Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. The paper theoretically proves RAIL’s absolute memorization on incrementally learned domains. Experiment results affirm RAIL’s state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings.

STRENGTHS:
1. The proposed RAIL adopts traditional machine learning techniques (e.g., Primal and Dual forms) for dealing with continual learning is novel.
2. The idea of RAIL’s absolute memorization based on analytic techniques is very appealing. Upon a good pre-trained network, the forgetting problem no longer exists, and this is very rare in the continual learning community.
3. The paper is easy to follow overall.
4. It was difficult to handle CIL problems in multi-modal CIL. This paper extends the continual learning scenario from task incremental to class incremental without task ID. This is a good contribution.
5. Experiments and settings are overall well formulated.

WEAKNESSES:
1. Could you explain the main difference between primal and dual forms in the CIL problem? It could be quite diffcult in this community to understand.
2. Experiemnts seems a little bit thin in the manucript. Perhaps try to move some of the experiments in the appendix to the main content.

QUESTIONS:
Please refer to the Weakness.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nContinual Learning (CL) with Vision-Language Models (VLMs) has a challenge that the model must not forget both previously learned knowledge and VLM pre-trained knowledge. Existing methods realize this by using large-scale reference data or domain identity hints, which is not practical. This paper proposed RAIL, which uses a recursive ridge-regression-based adapter to address the previously seen knowledge while using the zero-shot ability of VLMs for unseen classes. In addition, This paper proposes a novel task X-TAIL that evaluates the model on seen and unseen domains without any domain-identifier hints. The proposed method is empirically evaluated on MTIL and X-TAIL and shows state-of-the-art performance.\n\nSTRENGTHS:\n+ This paper tackled a novel and practical problem, X-TAIL. It is certainly unrealistic to assume that we always have domain information, and it is more natural to consider all classes.\n+ The proposed method seemed to have some technical novelty. It is worth commending that the proposed method is not merely an application of adapter methods such as CLIP-Adapter and Tip-Adapter, but has a recursive structure that makes it the method with affinity for CL.\n+ The proposed method showed higher performance on both MTIL and X-TAIL than existing methods.\n\nWEAKNESSES:\n+ I feel some parts of the paper were difficult to understand.\n  + The explanation of the proposed method in the introduction was so unclear that I could not understand the method at all from the introduction section. The author should modify the manuscripts to prepare some belief figures to support the reader’s understanding.\n  + I couldn’t understand how to identify ID and OOD classes in the proposed method until I read the appendix. The main manuscript should be self-contained as we can understand it without reading the appendix.\n+ The proposed method was evaluated on the few-shot setting in the MTIL evaluation, but it was not evaluated on the normal MTIL.\n  + The proposed method should be evaluated on the full data setting following the previous works[9, 14].\n+ Since the proposed method is based on ridge regression, there is concern about computational complexity when the amount of data increases. It is necessary to compare and discuss with existing methods, taking into account not only the recognition performance but also the computational efficiency.\n+ Typos:\n  + eq(1) ||Y-X_e||^2_F -> ||Y-X_eW||^2_F\n\nQUESTIONS:\n+ Why the average Last score of Dual-RAIL is higher than that of fine-tune? Are fine-tune results not the performance in the so-called ideal situation? \n+ How are the existing methods evaluated? How were the methods before the advent of CLIP (e.g. LwF, iCaRL) applied for CLIP? How were the MTIL methods applied for X-TAIL?\n\nLIMITATIONS:\n+ As I mentioned in Weaknesses, the proposed method based on ridge regression requires high computational complexity, so it cannot be applicable when the amount of data is very large.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper proposes a novel setting called Cross-domain Task-Agnostic Incremental Learning (X-TAIL), in which the model is required to incrementally learn from multiple domains and test images from both seen and unseen domains without any domain identity. Additionally, the authors introduce two Regression-based Analytic Incremental Learning (RAIL) methods (primal form and dual form) and validate the effectiveness of these methods both theoretically and experimentally.

STRENGTHS:
1. This paper is well-written and includes clear and accurate figures and equations.
2. The method begins with an introduction to primal and dual form ridge regression, analyzing whether these non-linear projections enhance the separability of CLIP features in images from different domains. This analysis motivates the design of the RAIL-adapter, making the overall approach easier to understand.
3. It is intriguing to explore the regression-based analytic IL, leveraging non-linear projection functions from both primal and dual perspectives to enhance the expressiveness of features extracted by the pre-trained CLIP.

WEAKNESSES:
1. There is little difference in performance between the primal and dual ridge regression methods on most datasets, as shown in Figure 6. Could you provide more analysis on this?
2. Could you compare the parameters of primal and dual ridge regression methods? What are the advantages and disadvantages of each?
3. Please provide more explanation of Figure 2. For example, what do the different colored blocks represent?

QUESTIONS:
See Weakness

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15