Title: UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: model calibration

test-time adaptation

CLIP

multi-domain
Abstract: Pre-trained vision-language models (e.g., CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data to adapt to downstream tasks, which could be costly. In this work, we aim to  leverage unlabeled data that naturally spans multiple domains to enhance the transferability of vision-language models.  Under this unsupervised multi-domain setting, we have identified inherent model bias within CLIP, notably  in its visual and text encoders. Specifically, we observe that CLIP’s visual encoder tends to prioritize  encoding domain over discriminative category information, meanwhile its text encoder exhibits a preference for domain-relevant classes. To mitigate this model bias, we propose a training-free and label-free feature calibration method, Unsupervised Multi-domain Feature Calibration (UMFC). UMFC estimates image-level biases from domain-specific features and text-level biases from the direction of domain transition. These biases are subsequently   subtracted from original image and text features separately, to render them domain-invariant. We evaluate our method on multiple settings including transductive learning and test-time adaptation. Extensive experiments show that our method outperforms CLIP and performs on par with the state-of-the-arts that need additional annotations or optimization.
Our code is available at https://github.com/GIT-LJc/UMFC.

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

Abstract

Pre-trained vision-language models ( e . g ., CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data to adapt to downstream tasks, which could be costly. In this work, we aim to leverage unlabeled data that naturally spans multiple domains to enhance the transferability of vision-language models. Under this unsupervised multi-domain setting, we have identified inherent model bias within CLIP, notably in its visual and text encoders. Specifically, we observe that CLIP's visual encoder tends to prioritize encoding domain over discriminative category information, meanwhile its text encoder exhibits a preference for domain-relevant classes. To mitigate this model bias, we propose a training-free and label-free feature calibration method, Unsupervised Multi-domain Feature Calibration (UMFC). UMFC estimates imagelevel biases from domain-specific features and text-level biases from the direction of domain transition. These biases are subsequently subtracted from original image and text features separately, to render them domain-invariant. We evaluate our method on multiple settings including transductive learning and test-time adaptation. Extensive experiments show that our method outperforms CLIP and performs on par with the state-of-the-arts that need additional annotations or optimization. Our code is available at https://github.com/GIT-LJc/UMFC .

1 Introduction

Recently, Vision-Language Foundation Models (VLFMs) such as CLIP [30], BLIP [24], Flamingo [1] and ALIGN [20] have demonstrated remarkable performance across various downstream tasks. These VLFMs formulate the training objective as contrastive learning, leveraging millions of imagetext pairs to establish a shared embedding space. Equipped with a wide range of visual and text representations, VLFMs exhibit the capability to tackle downstream tasks in a zero-shot manner.

Despite VLFMs being exposed to abundant examples, they may still encounter examples with new variations in downstream tasks. To address the problem of distribution shift between the pre-training and downstream domains, a natural approach involves fine-tuning VLFMs on various target tasks, such as prompt engineering [39, 38] and adapter learning [11, 37]. However, these methods generally require labeled samples for fine-tuning, which is prohibitively expensive to be satisfied in reality. Conversely, abundant unlabeled data are often available for downstream tasks. Notably, in practical scenarios, the unlabeled data typically contains multiple domains, which exacerbates the adaptation difficulty of VLFMs. Therefore, in this paper, we aim to improve the adaptation performance of VLFMs directly on unlabeled data that naturally spans multiple domains.

infograph

painting quickdraw

real

sketch

0

10

20

30

40

50

60

70

80

accuracy

(a)

80

60

40

20

0

20

40

60

80

60

40

20

0

20

40

60

infograph

painting

quickdraw

real

sketch

(b)

squiggle

line

circle

paintcan paintbrush

picture

10

1

10 0

10 1

%

quickdraw

painting

(c)

In this unsupervised multi-domain setting, we observe that CLIP cannot perform well when unlabeled data are drawn from mixed distributions. As shown in Figure 1(a), even within the same class space, the accuracy of CLIP varies significantly across different domains. While CLIP performs exceptionally well for images from common distributions encountered during pre-training, e . g ., achieving 83 . 0% accuracy on real domain, it struggles with rare distributions encountered during pretraining, e . g ., only achieving 14 . 2% accuracy on quickdraw domain. Above observations highlight that CLIP exhibits model biases that lead to incorrect predictions in specific scenarios. This raises a fundamental question: where do these biases originate ?

We point out that these model biases stem from deficiencies in the visual encoder and textual encoder. On the side of visual encoder, we observe that CLIP' visual encoder prioritizes encoding domain information over discriminative category information. As shown in Figure 1(b), features from the same domain clearly cluster together, whereas a notable gap separates features from different domains. This phenomenon indicates that CLIP's visual encoder exhibits a higher sensitivity to domain information over category information. When the domain of downstream tasks shifts from pre-trained tasks, the mismatch in domain-specific information encoded in image features could adversely affect classification accuracy. On the side of textual encoder, we observe that CLIP demonstrates varying category preferences across different domains. Specifically, CLIP tends to classify images into categories whose name are closely related to corresponding domain. As shown in Figure 1(c), within 'quickdraw' domain, a large portion of samples ( ∼ 30% ) are classified as 'squiggle' or 'line' categories. Conversely, within 'painting' domain, CLIP favors categories like 'paintcan' and 'paintbrush'. This observation shows that the class embeddings encoded by CLIP's textual encoder inherently carry domain-specific information, misleading the model to prioritize categories highly associated with respective domains. Due to the combined effects of visual and textual encoder biases, CLIP's performances vary significantly across different domains.

In this paper, we aim to calibrate CLIP to mitigate the model biases. Initially, we analyze the model biases from a probabilistic standpoint. The factors influencing p ( y | x ) , affected by domain variable z , can be decoupled into two parts: the sample distribution conditioned on classes p ( x | y, z ) and the class distribution p ( y | z ) . If the two probability distributions are independent of z , domain shifts will not affect the predictions. To this end, we propose Unsupervised Multi-domain Feature Calibration (UMFC), a simple yet efficient framework for calibrating CLIP to generalize to various downstream tasks using multi-domain unlabeled data. UMFC jointly calibrates CLIP through two training-free modules: Image Feature Calibration module (IFC) and Text Feature Calibration module (TFC). Firstly , IFC focuses on calibrating CLIP's image encoder to prioritize category-level over domain-level information, thereby reducing prediction error caused by domain shifts. Specifically, we calculate the average image features for each domain i , denoted as µ i , and posit that the prediction of µ i reflects the inherent bias of CLIP's image encoder within that domain. By subtracting this domain-specific bias from original predictions, we can derive domain-agnostic predictions. Secondly , TFC focuses on calibrating CLIP's text encoder to remove its preference towards domain-related class names. As observed by [7, 28, 8], a 'global direction' exists in CLIP, representing the shift from training distribution to unseen distribution, shared across image and text embedding spaces. Motivated by this observation, we utilize the shift direction between different-domain images to estimate the text shift direction. Then, we subtract this shift vector to counteract the text encoder's bias towards domain-related categories. By combing IFC and TFC, we can calibrate the features on both CLIP's image and text encoders, thereby alleviating the model bias in downstream tasks.

We validate the efficacy of UMFC on three downstream tasks: unsupervised calibration, transductive learning, and test-time adaptation, demonstrating consistent gains over CLIP. UMFC presents a lowcost solution for classification, unlocking the potential of CLIP-like models for practical scenarios characterized by abundant images across multiple domains but scarce labels.

2 Related Work

Few-Shot Learning. Few-shot learning (FSL) [17, 16, 18, 10, 34] aims to learn a good model on a novel task with few labeled samples. Traditional FSL methods [17, 16, 18, 10, 34] often rely on abundant related training tasks for pre-training and design specialized algorithms to transfer cross-task knowledge, facilitating task adaptation. Additionally, some semi-supervised learning (SSL) methods [26, 14, 15] tackle the scarcity of labeled samples by assuming access to extensive unlabeled data. While both FSL and SSL methods have achieved encouraging results, their generalization abilities are limited. Recently, the CLIP model is proposed, which is a vision-language foundation model that learns a shared vision-language embedding space. As pre-trained on large-scale data, CLIP exhibits impressive zero-shot ability across various downstream tasks. Based on CLIP model, many works focus on improving its performance further on downstream tasks with few labeled data. For example, CoOp [39], CoCoOp [38], MaPLe [21] and PromptSRC [22] leverage few labeled samples to learn the prompt in the continual input embedding space, offering a parameter-efficient way of fine-tuning foundation models. Similarly, CLIP-Adapter [11] and Tip-Adapter-F [37] introduce a lightweight adapter module to produce adaptive multi-modal features. However, these methods require extra labeled data and the process of tuning pre-trained parameters for CLIP, which is cost-expensive. Differently, in this work, we focus on utilizing unlabeled data to enhance CLIP's performance in a training-free manner.

Domain Adaptation / Domain Generalization. Recently, several methods [23, 3, 8, 31, 2, 33, 5, 19, 7] exploit a large-scale pre-trained model ( e . g ., CLIP) to address the domain adaptation and generalization problems. For example, RISE [19] leverages CLIP as a teacher to regularize the student's learned representation through images. The work [3] utilizes domain-invariant and domainspecific prompts for multi-source unsupervised domain adaptation. Other works [8, 7, 5] focus on utilizing the transferability between visual and textual modalities to guide domain information transfer. PromptStyler [5] attempts to simulate various distribution shifts to explore diverse styles in a joint vision-language space. PODA [8] and LADS [7] generate samples in the style of the target domain, and adapt to the target domain based on these samples. However, these language-guided methods require the prior of target-domain names, which may not be satisfied in reality. In contrast, our UMFC method does not require any extra information about the target domain.

Test-Time Adaptation. Test-time adaptation aims to adapt a pre-trained model to test tasks, where distribution of test data differs from that of pre-training data. TPT [32] proposes a test-time prompt tuning strategy, which extends traditional TTA methods to vision-language models. Building upon TPT, DiffTPT [9] utilizes pre-trained diffusion models to augment the diversity of test data samples used in prompt tuning. SwapPrompt [27] employs a dual prompts paradigm to enhance the swapped prediction mechanism. However, these prompt learning methods are computationally expensive and time-consuming. Different from above methods, our UMFC only needs to calibrate features in a training-free way, making it more efficient for test-time adaptation.

3 CLIP and Model Biases

In this section, we first describe the backgrounds of CLIP and then analyze its bias issue in downstream tasks. We attribute the cause of bias to the visual encoder bias and text encoder bias.

3.1 Contrastive Language-Image Pre-training (CLIP)

CLIP [30] consists of two parallel encoders: a visual encoder that maps image inputs into image features, and a text encoder that maps text inputs into text features. The model is trained with a contrastive loss that maximizes similarity between positive image-text pairs while minimizing similarity between negative pairs. This process ensures alignment between the features of images and their corresponding textual descriptions within the feature space. Trained on a vast collection

of image-text pairs, CLIP benefits from general visual representations, endowing it with powerful zero-shot capabilities.

Formally, we denote a CLIP model as { F, T } , with F and T being the visual and text encoders. Considering a K -class classification problem, we use Y = { y 1 , . . . , y K } to represent the class space, where y k denotes the class name of k th class. In the zero-shot inference phase, CLIP uses hand-crafted prompts p (such as 'a photo of a {}') to covert each class name y i to category-specific text description { p; y i } . Then, we feed these class descriptions to the text encoder to get the text features { t 1 , . . . , t K } , where t i = T (p; y i ) . Meanwhile, given a test image x , the visual encoder is used to compute its visual feature, denoted as f = F ( x ) . The prediction probability on x can be then computed as:

where sim denotes the cosine similarity and τ is the temperature of the softmax function.

3.2 Model Bias in CLIP

As shown in Figure 1(a), the accuracy of CLIP varies significantly across different domains within the same class space, e . g ., 83 . 0% in real domain and 14 . 2% in quickdraw domain. This phenomenon indicates that CLIP favors domains commonly encountered during pre-training (such as natural images). We attribute this model bias to visual encoder bias and text encoder bias.

Visual Encoder Bias. We observe that CLIP' visual encoder prioritizes encoding domain information over discriminative category information, as shown in Figure 1(b). Due to the abundance of natural images in pre-training, CLIP's visual encoder and text encoder are well-aligned in the natural image domain. Consequently, when the style of input images largely deviates from natural domains (such as quickdraw and infograph style), the visual feature gap across different domain (as depicted in Figure 1(b)) hinders the text encoder's capacity to process these shifted images effectively, leading a significant drop in classification accuracy.

Text Encoder Bias. We observe that CLIP exhibits a preference for domain-related categories in specific domains, as shown in Figure 1(c). In domains characterized by distinct styles, certain category names may carry domain-specific information. For instance, in quickdraw domain, where most images consist of lines and squiggle, CLIP demonstrates a severe bias towards the 'line' and 'squiggle' categories. This leads to a large number of quickdraw samples being incorrectly classified into the two categories. Conversely, in the painting domain, as all images inherently possess painting features, CLIP shows a strong preference for categories related to the concept of painting , such as 'paintcan' and 'paintbrush'.

4 Unsupervised Multi-domain Feature Calibration

In this section, we continue to analyze the model biases issue from a probabilistic view. Then we give a detailed introduction of our method aimed at alleviating CLIP's biases.

4.1 Analyze Model Biases in a Probabilistic View

We start from the posterior probability p ( y i | x ) , as introduced in Equation 1. Recall that our goal is to maximize the posterior probability p ( y i | x ) for each image-label pair ( x, y i ) of any domain. A natural question arises: How different domains affect the posterior probability? We answer this question from a probabilistic view. Based on the Bayes' Theorem, we can drive that:

where z is a latent variable that denotes the domain labels. We can further decompose the joint probability p ( y i , x, z ) in Equation 2 as p ( y i , x, z ) = p ( x | y i , z ) p ( y i | z ) p ( z ) . Assume that the probability distribution of domains p ( z ) is uniform, thus maximizing the posterior probability

p ( y i | x ) is equal to maximize the summation ∑ z p ( x | y i , z ) p ( y i | z ) , as

Equation 3 shows that the domains affect posterior probability distribution p ( y | x ) by disturbing two terms: the sample distribution conditioned on classes p ( x | y, z ) and the class distribution p ( y | z ) .

4.2 UMFC: Unsupervised Multi-domain Feature Calibration

As analyzed in Section 3.2, CLIP exhibits both visual and text encoder bias, impacting its generalization ability to downstream tasks. As shown in Equation 3, these biases stem from the probability distributions p ( x | y, z ) and p ( y | z ) , which are disturbed by domain information. To mitigate the model biases, an intuitive idea is to make the two probability distributions p ( x | y ) and p ( y ) independent of domain variable z by calibrating image and text features. To this end, we propose training-free UMFC approach, consisting of an Image Feature Calibration (IFC) module to alleviate visual encoder bias and Text Feature Calibration (TFC) module to alleviate text encoder bias, facilitating the transfer of CLIP to downstream tasks. Algorithms are provided in the Appendix C.

Image Feature Calibration Module. On the side of visual encoder, we focus on making the conditional probability p ( x | y ) independent of domain variable z , which is equivalent to achieving p ( x | y, z ) = p ( x | y ) . A straightforward method is to align image feature distributions given a class y across different domains. Unfortunately, as only mixed unlabeled data is provided, we cannot access to class labels and domain labels of images. However, we empirically observe that CLIP's visual encoder prioritizes encoding domain information over discriminative category information. Thus, we can distinguish image features from different domains through a simple clustering algorithm. Formally, we assume that there are M clusters { c 1 , ..., c M } after applying a clustering algorithm. Each cluster is assumed to follow a Gaussian distribution c i ∼ N ( µ i , Σ i ) , where mean vector µ i = ( 1 | c i | ∑ f ∈ c i f ) represents the average of image features from cluster c i . Due to the model biases, the pseudo labels produced by zero-shot CLIP are not reliable. Therefore, we directly align the margin image feature distribution of each cluster by subtracting the corresponding mean vector 2 . Specifically, for each visual feature f belonging to the cluster c i , we calibrate it as follows:

Text Feature Calibration Module. On the side of text encoder, we focus on making the class probability p ( y ) independent of domain variable z , which is equivalent to achieving p ( y | z ) = p ( y ) . Previous works [35] have found that CLIP's performance is sensitive to class name y . For example, replacing category names with synonyms or near-synonymous terms can significantly impact CLIP's prediction results. Due to the influence of domain information in image features, text encoder bias can cause CLIP to categorize domain-specific images into categories whose names are semantically similar to that domain. We further observe that such sensitivity to class names varies across different domains, as shown in Figure 1(c). This observation inspires us to calibrate text features by removing domain-specific information. However, a challenge arises as domain labels are unavailable. To address this issue, we attempt to extract domain information from unlabeled images, and then transfer this domain information to the text embedding space to estimate the text bias.

Observation 4.1 Cross-Modality Transition Direction . The underlying assumption behind using images to simulate the corresponding shifts in texts is that the transition direction from domain i to domain j is consistent across both the image embedding and text embedding spaces [7, 28, 8], which can be formulated as:

where ( x i , y i ) and ( x j , y j ) represent training samples from domain i and domain j respectively, p i and p j denote the domain-specific text prompts for domain i and j respectively. For example, the text prompt of 'quickdraw' domain can be 'a quickdraw image of a [class]'.

Inspired by Observation 4.1, we estimate the text-level domain transition direction using differentdomain images. By clustering image features, we calculate the average feature µ i of unlabeled images from each domain i , representing the domain-specific feature for that domain. Also, we calculate the average feature of all unlabeled images µ avg , representing the domain-invariant feature since it encompasses various domain distributions. Following Equation 5, the transition information ̂ t i of domain i can be computed as ̂ t i = µ i -µ avg . By subtracting this domain transition vector, we suppress the preference for specific class names in the original text features. To ensure that the calibrated text features effectively eliminate biases towards a wide range of domains, we integrate the text features calibrated on each domain. Specifically, for the text feature t j of class j , we calibrate it as follows:

where M is the number of clusters.

Inference. After calibration with IFC and TFC modules, we obtain the final prediction results. Specifically, this calibration modifies the prediction probability on test image x as follows:

5 Experiments

5.1 Experimental Setting

Datasets. Our UMFC is training free , which only calibrates the image and text features using Equation 4 and 6. To analyze model's generalization capability, we use two large-scale datasets for evaluation: 1) DomainNet [29] consists of 569,010 images with 345 categories from six domains: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), Sketch (S). 2) ImageNet Variants composed of several datasets shifted from ImageNet, including ImageNet-A (IN-A) [13], ImageNet-R (IN-R) [12], and ImageNet-Sketch (IN-S) [36]. We form the class space for ImageNet Variants by taking the union of the class sets in IN-A and IN-R. To ensure the reliability of the evaluation results, we randomly sample the test data to construct a balanced test set where both domain and category distributions are uniform.

Evaluation Paradigms. Our approach is a universal feature calibration technique that can be applied across multiple scenarios. In this work, we explore three settings: 1) Unsupervised Calibration (UC) where the unlabeled training set is provided for computing the calibration vectors; 2) Transductive Learning (TL) where the entire unlabeled test set is provided at once, without providing any training data; 3) Test-Time Adaptation (TTA) where the model can be adapted to test samples shifted from training distribution. Different from TL, the test data usually arrives in batches in TTA setting. More details on the experimental setup, please refer to Appendix D.

5.2 Main Results on Unsupervised Calibration.

Implementation Details. We select CLIP [30] as our pre-trained vision-language model. We use CLIP with ViT-B/16 [6] as image encoder, and keep the original transformer as the text encoder. By default, a fixed prompt, 'a photo of a [class]', is employed for all datasets. The images are resized to 224 × 224 . The hyper-parameter M (cluster number) is set to 6 for DomainNet. Remarkably, our method is training-free where both image encoder and text encoder remain frozen throughout the entire pipeline. All experiments are performed on a GeForce RTX 3090 Ti GPU.

Baselines. We compare our method with four groups of methods: (1) CLIP and its variants to show zero-shot predictions: CLIP [30] that uses a fixed prompt "a photo of a [class]"; CLIP-E [30] that uses an ensemble of prompt templates. (2) CLIP-D [30] that utilizes the domain information of test samples and designs domain-specific prompts. (3) Few-shot learning methods: CoOp [39] that

performs prompt tuning using labeled data of downstream tasks; CLIP-Adapter [11] that trains an adapter using task-specific labeled data. (4) Unsupervised Fine-tuning method: MUST [25] that fine-tunes the model using unlabeled multi-domain data.

Analysis. As shown in Table 1, UMFC achieves superior performance over CLIP and competitive performance with few-shot methods, CoOp [39] and CLIP-Adapter [11]. We can observe that: (1) CLIP-D incorporates domain information into text prompts, such as 'a clipart image of a [class]' for test samples from 'clipart' domain. However, creating domain-specific templates is challenging due to unknown and potentially mixed domain sources of test samples. So, we only use CLIP-D as an oracle result. As shown in Table 1, our method can achieve competitive performance with CLIP-D without prior domain labels for test samples. (2) On domains where CLIP performs poorly, e . g ., 'quickdraw', our UMFC significantly outperforms others. Compared to CLIP, UMFC achieves about 5% performance gain on quickdraw domain. In addition, combined with more diverse prompts of CLIP-E, UMFC further improves performance. (3) MUST [25] uses abundant unlabeled data from 6 domains for fine-tuning. Our UMFC achieves better performance than MUST even without any additional training. (4) For CoOp [39] and CLIP-Adapter [11], we fine-tune them using labeled samples from multiple domains. Specifically, we provide k labeled samples per class for each domain, resulting in a total of (6 × k × 345) labeled samples available for training. When the number of labeled samples is 6 × 345 , our method outperforms few-shot fine-tuning methods. While the performance is higher for few-shot methods with a larger number of labeled samples ( i . e ., 24 × 345 ), it is essential to highlight the challenges in obtaining some labeled data for each class and each domain in real-world scenarios. In contrast, our method does not require selecting class-balanced and domain-balanced labeled samples and incurs no additional training overhead.

Table 2 provides 8 × 345 labeled samples from a single domain for CoOp fine-tuning and an equal number of unlabeled samples for UMFC calibration. While CoOp trained on a single domain can improve performance within that domain, its performance declines on other domains. This decline becomes particularly notable when a significant distribution gap between the training and test domains exists, e . g ., CoOp (Q), leading to a large decrease in average performance across multiple domains. In contrast, our method achieves consistent performance improvements on both training and unseen domains with the same amount of training data. Additionally, unlike CoOp, UMFC does not require labeled data or any parameter fine-tuning.

5.3 Main Results on Transductive Learning

In this part, we compare our UMFC with three groups of methods: (1) Zero-Shot CLIP models. (2) Domain Generalization (DG) method: MIRO [2] that trains CLIP on available data to learn great generalization capability. (3) Data-Free DG method: PromptStyler [5] that learns to simulate diverse distributions. The hyper-parameter M (cluster number) is set to 3 for ImageNet-Variants. The compared results are shown in Table 3 and Table 4. As seen, our UMFC can achieve the best average performance on DomainNet and ImageNet-Variants benchmarks, which validates the effectiveness of our approach in transductive learning.

5.4 Main Results on Test-Time Adaptation

Implementation Details. In TTA setting, where we cannot access all data simultaneously, we adopt an incremental clustering approach. By default, the batch size is set to 100. Initially, we apply K-Means clustering algorithm to the first batch of data. For subsequent batches, we use the prototype classification, with cluster centers serving as prototypes, to assign samples in that batch to different clusters. Then, the cluster centers and calibration statics { µ i } M i =1 are updated accordingly. We have developed two strategies for updating the calibration statics: UMFC-Memory that stores the feature information of each batch to calculate the statistical information; UMFC-EMA that uses Exponential Moving Average (EMA) to update statistical information.

Analysis. We mainly compare our UMFC with TPT [32] in test-time adaptation, where TPT finetunes the prompts by minimizing the entropy of the predictions. The comparison results are shown in Table 6 and Table 5. We can observe that: (1) As shown in Table 6, UMFC achieves performance gains across all domains, with particularly noticeable gains in those where CLIP performs poorly. For example, UMFC substantially improves upon CLIP on 'quickdraw' and 'infograph' domains, with

60

40

20

0

20

40

60

60

40

20

0

20

40

60

80

infograph

painting

quickdraw

real

sketch

(a)

real

sketch

painting

0.0

0.1

0.2

0.3

0.4

0.5

probability

uniform

CLIP

Calibration

(b)

an accuracy gain of 5 . 6% . Table 5 shows that the improvement on ImageNet-A is less significant than on ImageNet-R and ImageNet-S. The reason may be attributed to the absence of distinct domain styles in ImageNet-A, limiting the effectiveness of our calibration method that relies on domain information. (2) UMFC-Memory and UMFC-EMA with different statistical update strategies exhibit similar performance. However, UMFC-EMA, which updates features based on the most recent data, notably reduces storage requirements. Moreover, UMFC exhibits significantly higher computational efficiency without training, whereas TPT requires executing optimization steps on 64 different augmented views of each test image. Thus, our method is more suitable for rapid deployment.

5.5 Ablation Study

The effectiveness of IFC. We firstly evaluate the effectiveness of IFC. As shown in Table 7, IFC individually contributes to performance gains of CLIP, about 3% average gains. Furthermore, Figure 1(b) and Figure 2(a) visualize the image features extracted by CLIP with/without IFC respectively. As shown in Figure 1(b), the vanilla CLIP maps different-domain images to different clusters in the feature space. Conversely, Figure 2(a) shows that IFC leads to the merge of image features from different domains, validating its effectiveness of reducing domain-specific information.

The effectiveness of TFC. As shown in Table 7, TFC also individually contributes to performance gains of CLIP. To assess the effectiveness of calibrated text features in eliminating domain bias, we construct a domain classifier. Specifically, we utilize the text features generate by CLIP, with domain prompts '[domain]', as the domain classifer. Then, we use this domain classifier to perform domain classification for the text features before and after calibration ( i . e ., computing the cosine similarity between domain classifier and text features). As shown in Figure 2(b), the original text features exhibit a long-tail phenomenon, with a higher probability of being classified into the 'real' domain. This suggests that the original text features are biased towards the 'real' domain, overlooking their generalization ability to other domains. However, after calibration with TFC, the text features exhibit a near-uniform distribution across domains, indicating that the calibrated text features have largely eliminated domain bias. Consequently, our method enhances performance in other domains without compromising the model's performance in its strongest domain. As shown in Table 7, the combination of IFC and TFC ( i . e ., UMFC) can further bring performance gains, which validates the complementary of image-level calibration and text-level calibration for CLIP.

The impact of cluster number M . Our method involves clustering the unlabeled data to determine their respective clusters. We evaluate our method with respect to the number of clusters M . As shown

in Table 8, our UMFC consistently outperforms vanilla CLIP, even when the number of clusters M does not match the number of domains (6 for DomainNet). More importantly, our method is not sensitive to changes in M . Refer to the Appendix E for more analysis.

The impact of batch size in TTA setting. We present results across various batch sizes during test-time adaptation to confirm that UMFC is robust to batch size variations. When the sample count is initially lower than the number of clusters M , K-Means clustering cannot be directly applied. To address this, we used the first M samples as the initial cluster centers and then proceeded with the same test-time adaptation. As shown in Table 9, even in the extreme case of a batch size of 1, our method still demonstrates consistent improvement.

6 Conclusion

In this work, we point out that the model biases hinder the transfer ability of pre-trained visionlanguage models. We develop UMFC, a simple unsupervised calibration method that mitigates model biases in both visual encoder and text encoder through a training-free manner. We demonstrate the effectiveness of our method across multiple settings, including unsupervised calibration, transductive learning, and test-time adaptation. Without the need for any annotations and training, UMFC improves the zero-shot generalization ability of CLIP.

Limitations and Broader Impacts. Our method requires unlabeled samples from the target domain to establish domain-level calibration vectors. Although our method is much cheaper and more accessible than fine-tuning or few-shot methods [39, 38, 32], the requirement of unlabeled data can still be a limitation for some specific scenarios.

Acknowledgments

This work is partially supported by National Science and Technology Major Project No. 2021ZD0111901, National Natural Science Foundation of China (NSFC): 62376259 and 62306301, National Postdoctoral Program for Innovative Talents under Grant BX20220310.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.

Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. In ECCV , 2022.

Haoran Chen, Zuxuan Wu, and Yu-Gang Jiang. Multi-prompt alignment for multi-source unsupervised domain adaptation. ArXiv , abs/2209.15210, 2022.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR , 2023.

Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In ICCV , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Lisa Dunlap, Clara Mohri, Devin Guillory, Han Zhang, Trevor Darrell, Joseph E. Gonzalez, Aditi Raghunanthan, and Anja Rohrbach. Using language to extend to unseen domains. In ICLR , 2023.

Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Pøda: Prompt-driven zero-shot domain adaptation. In ICCV , 2023.

Chun-Mei Feng, Kai Yu, Yong Liu, Salman A. Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. 2023.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 132:581-595, 2024.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-ofdistribution generalization. In ICCV , 2021.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. In CVPR , 2021.

Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Dual compensation residual networks for class imbalanced learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45:11733-11752, 2023.

Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Triplet adaptation framework for robust semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguan Shan, and Xilin Chen. Task attribute distance for few-shot learning: Theoretical analysis and applications. arXiv , 2024.

Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes. In Advances in Neural Information Processing Systems , 2023.

Minyang Hu, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Scalable modular network: A framework for adaptive learning via agreement routing. In ICLR , 2024.

Zeyi Huang, Andy Zhou, Zijian Lin, Mu Cai, Haohan Wang, and Yong Jae Lee. A sentence speaks a thousand images: Domain generalization through distilling clip with language guidance. In ICCV , 2023.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR , 2023.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV , 2023.

Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen, Fu, and Chen-Nee Chuah. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In ICCV , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Junnan Li, Silvio Savarese, and Steven C. H. Hoi. Masked unsupervised self-training for label-free image classification. In ICLR , 2022.

Jiachen Liang, Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Generalized semi-supervised learning via self-supervised feature adaptation. In NeurIPS , 2023.

Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In NeurIPS , 2023.

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV , 2021.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV , 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal representations for covariate shift. In ICLR , 2022.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS , 2022.

Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. In ICML , 2023.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems , 2017.

Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G. Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. In ICLR , 2023.

Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In NeurIPS , 2019.

Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130:2337-2348, 2022.

A The Prevalence of the Observed Model Bias

Our method is motivated by observed biases in CLIP. Therefore, it is valuable to investigate whether these observations hold when the pre-training distribution and model architecture are altered.

We further verify this on OpenCLIP[4]. OpenCLIP investigates scaling laws for contrastive languageimage pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. The large-scale experiments involve models trained on up to two billion image-text pairs (LAION-2B) and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We used OpenCLIP models trained on different datasets to obtain the corresponding image features, and visualized them using t-SNE. As shown in Figure 3, even when the training data changes, images from different domains still cluster together, demonstrating that our observation is universally applicable.

Additionally, we validated the effectiveness of our method on the OpenCLIP series models in Table 10. It is evident that the zero-shot capabilities of larger-scale models have significantly improved compared to CLIP, and our method still provides further performance gains. This demonstrates that our method is universally applicable to Vision-Language models with different architectures and training data.

ViT-B-16 (3B)

ViT-B-16 (34B)

ViT-B-32 (34B)

ViT-B-16 (80M)

ViT-B-16 (400M)

ViT-B-16 (2B)

(a) The OpenCLIP models are pre-trained with three different dataset (LAION-80M, LAION-400M and LAION-2B).

60

infograph

iefgyaph

peinting

painting

quickdraw

real

quickdraw

meal

20

sketch

29

sketch

29

-20

20

40

iefogaph

painting

-40

0

quickdraw

real

7§

50

25

25

50

75

4

D

-20

0

29

40

-40

20

20

4G

60

sketch

80

dipart

painting

infograph

quickdrax

infograph

painting

real

real

sketch

sketch

-30

20

-29

40

qdeufepu

peinting

quickdraw

reall

-60

sketch

-60

-40

29

0

20

-46

-20

29

60

20

20

60

(b) The OpenCLIP models are pre-trained on LAION-2B with different number of seen samples (3B, 34B) and architectures (ViT-B-16, ViT-B-32).

B Details of Text Feature Calibration Module

Since the domain labels are unavailable, we attempt to extract domain information from images in downstream tasks, and then transfer this domain information to the text embedding space to eliminate

the text bias. Inspired by Observation 4.1, we can estimate the domain transition direction in text embedding space by using images from different domains, as shown in Figure 4.

a quickdraw image of a bird

an infograph image

of a bird

a real image of a bird

a clipart image of a bird

∆𝑡

!

∆𝑡

"

∆𝑡

#

∆𝑓

!

∆𝑓

"

∆𝑓

#

Vision-Language Joint Space

Image Feature

Text Feature

Modality Boundary

C Algorithm

Algorithm 1 summarizes the proposed UMFC method under Test-Time Adaptation (TTA) setting. Algorithm 2 summarizes the proposed UMFC method under Unsupervised Calibration (UC) / Transductive Learning (TL).

Algorithm 1 Algorithm UMFC under TTA

Algorithm 2 Algorithm UMFC under UC / TL

1: Input : unlabeled data { u b } N i =1 , total number of samples N , total number of clusters M , total number of batches B , CLIP's image encoder F ( · ) and text encoder T ( p ; · ) , p is the text template, class names { y i } C i =1 .

▷ Get original text features

▷ Get original image features of unlabeled data

▷ Get the average of { f b } N b =1

▷ Using K-Means to acquire M clusters and their centroids

▷ Assign sample to the nearest clusters

2: t ← T ( p ; { y i } C i =1 )

4: µ avg ← 1 N ∑ N b =1 f b

3: { f b } N b =1 ← F ( { u b } N i =1 )

5: { c j } j M =1 ← K-Means ( { f b } N b =1 )

6: for b ← 1 to N do

7: l ← arg min m ∥ f b -c m ∥ 2

8: ˆ t k ← I ( l = m )( c m -µ avg ) ,

9: f ′ b ← f b -c l ∥ f b -c l ∥ , t ′ b ← 1 M ∑ M m =1 t -ˆ t m ∥ t -ˆ t m ∥ ,

10: end for

11: return { f ′ b } N b =1 , t ′

D Experimental Settings

Our work can be deployed across various scenarios, including Unsupervised Calibration (UC), Test-Time Adaptation (TTA), and Transductive Learning (TL).

Unsupervised Calibration. In the UC scenario, we provide an unsupervised training set and use K-Means to assign cluster labels to the training samples, while also saving the corresponding cluster prototypes. Then, we create an unlabeled training set from a mixed domain by sampling 16 instances from each class across 6 domains in DomainNet. UMFC derives image bias and text bias for different clusters based on the cluster labels. During the testing phase, we first predict the cluster labels of the test samples using the cluster prototypes obtained during training, and then calibrate the predictions using the bias information derived from UMFC.

Test-Time Adaptation. In the TTA scenario, no training data is provided. Test data from mixed domains arrive in batches, with a batch size set to 100. For the first batch, we perform initial clustering using K-Means. For subsequent batches, we assign cluster labels to the samples based on cluster prototypes and continuously update these prototypes. Once the cluster labels are obtained, UMFC calculates the bias information for the current batch, updates the bias information for each cluster based on the new labels, and then calibrates the data for the current batch.

Transductive Learning. In the TL scenario, we provide the entire test set. Similar to UMFC, we gather statistical information and calibrate the predictions for the test data. TL can be viewed as an extreme case of TTA, where the entire test set is treated as a single batch.

E Experimental Analysis

The impact of cluster number M . We evaluate our method with respect to the number of clusters M and demonstrate that our method is not sensitive to the choice of this hyperparameter. As shown in Table 11, For instance, setting M from 2 to 6 all leads to improvements on ImageNet-Variants.

Computation Cost. For analyzing computational cost, we report the training time, inference time and memory of UMFC and other comparison methods under different scenarios. In Unsupervised

▷ Update calibration statics

▷ Calibrate f and t with IFC and TFC

▷ Calibrated image and text features

Calibration scenario, the entire unlabeled training set is provided for training. Corresponding computation cost comparisons are shown in Table 12. Firstly, UMFC incurs minimal training and inference overhead compared to CLIP. This is because UMFC only requires a single forward pass to extract features and then calculate statistics for feature calibration. Secondly, when compared to few-shot fine-tuning methods (CoOp) and unsupervised fine-tuning methods (MUST), UMFC also demonstrates lower consumption of computational resources and time. In Test-time Adaptation scenario, no training data is provided and the test data arrives in batches. Corresponding computation cost comparisons are shown in Table 13. UMFC requires less memory than TPT and shows greater computational efficiency. Specifically, UMFC takes only 296 seconds, whereas TPT requires nearly 197 minutes. This is because TPT requires fine-tuning the text prompt for each test sample and augmenting each test sample 64 times to ensure the reliability of the fine-tuning results, which significantly slows down TPT's inference speed.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: A summary of the paper's contributions is provided at conclusion.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 5.2, 5.4 and supplemental material for implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See https://github.com/GIT-LJc/UMFC

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.2, 5.4 and supplemental material for implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 5.2 and supplemental material for implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See References.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
To boost the transferability of CLIP across downstream domains, the paper proposed Multi-Domain Feature Calibration (UMFC). By mitigating CLIP biases in both visual and text encoders, UMFC significantly improves classification performance over existing methods across 3 downstream tasks.

STRENGTHS:
1. The observation of domain-dependent performance variability in CLIP is interesting. The observation highlights that the accuracy of CLIP can vary across different domains for the same classes. 
2. The paper is well-motivated. The author starts with the bias phenomenon empirically and further presents the motivation from a probabilistic view.
3. The experiment is comprehensive. The efficacy of UMFC is verified on three downstream tasks and extensive experiments demonstrate the consistent gains.

WEAKNESSES:
1. The proposed methods rely on multi-domain data for calibration, which slightly weakens its application scenarios.
2. The comparison in TTA is not enough. As the paper use prototype for calibration, the proposed method be compared with other state-of-the-art prototype-based methods such as T3A [1].
3. The ablation of the number of clusters \textit{M} is insufficient. Since we do not know the number of domains, what would the results be if we set \textit{M} higher than the actual number of domains?

[1] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, 2021.

QUESTIONS:
1. The author set the batch size of TTA to 100. Is the proposed method sensitive to the batch size?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposes a training-free method "Unsupervised Multi-domain Feature Calibration ( UMFC)" to mitigate the model bias problem presented in CLIP models. The paper first observes the bias problem from the visual encoder and the text encoder perspective. Then it mitigates the problem by subtracting the bias term in the visual and text features obtained from the domain data. The experimental results show improvements over existing approaches like CoOp or CLIP-Adapter.

STRENGTHS:
1. The observation is clear. The paper provides a clear observation regarding the inherent biases in CLIP's visual and text encoders, which lead to biased prediction. 
2. The approach is simple and straightforward. The proposed UMFC effectively leverages unlabeled data to mitigate domain biases without involving fine-tuning or extra optimization. 
3. The presentation of the method is clear and its evaluation is clear and easy to follow.

WEAKNESSES:
1. I think the improvements seem minor. In table 1, compared to the really easy approach CLIP-D, UMFC fails to improve it and UMFC + CLIP-E outperforms it by only 0.65% points. What about CLIP-D + CLIP-E? I think you should use this one to agast UMFC + CLIP-E. 
2. In table 2, why did CoOp perform worse than the original CLIP? I believe it's the receipt problem, and I think you should at least perform some hyper-parameter choice to get a reasonable baseline, since finetuning on those in-domain data can easily get performance improvements. 
3. What's the extra computation cost involved in this approach and how it compares to other methods? Since the proposed approach uses test-time adaptation, I believe the authors should tell us the computation cost.

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies the inherent model bias within CLIP, notably in visual and textual encoders. To mitigate the bias, authors propose a feature calibration method, termed Unsupervised Multi-domain Feature Calibration. Experiments in the setting of transductive learning and test-time adaptation show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper is well-structured and easy to follow.\n2. The motivation of this paper is clear.\n\nWEAKNESSES:\n1. The proposed method is inspired by the observation in the TSNE figure (Figure 1(b)) that \"features from the same... different domains\". However, this does not appear to be the case. Features from QuickDraw, Painting, and Infograph are clustered, while features from Real, Clipart, and Sketch are dispersed throughout. This suggests that CLIP models can encode some domain information, but this ability is limited to specific domains. If the observation which inspires the method does not hold, the rationale for the proposed method becomes less convincing.\n\n2. For figure 1(c), it is suggested to show the percentage of predictions rather than the absolute numbers. Each domain in DomainNet has different number of samples in validation set. Comparing the number of predictions is less convincing to show the bias of textual encoders. In addition, only Painting and Quickdraw are shown. It would be interesting to evaluate such bias in other domains.\n\n3. It is important to reveal the process of hyper-parameter selection. The cluster number is set to 6 for DomainNet, which is equal to the number of domains in DomainNet. Does it mean that the proposed method need prior knowledge on the number of domains in the target dataset? Furthermore, it is also valuable to reveal the hyper-parameter in ImageNet-variant experiments.\n\n4. It has been show that pre-training dataset distribution determines the CLIP's robustness [A]. Given that the proposed method is bases on CLIP bias, it would be interesting to know whether the observations persist when the pre-training dataset distribution changes.\n\n5. The compared baselines are out-dated. Please include methods such as MaPLe[B] and PromptSRC[C] and even more recent methods.\n\n6. It is unsure whether the comparison to other methods is fair. In UC setup, full unlabelled training set is provided, while CoOp and CLIP-Adapter are only given roughly two thousands images. \n\n     a) Though labelling images can be expensive, collecting this amount of unlabelled data for clustering is even more expensive. Note that, these data all belong to 345 classes defined in DomainNet with six different styles. \n\n     b) In Table 2, it is also unfair to show the superiority of UMFC by comparing it with CoOp trained on one single domain, given that UMFC is able to see images with different styles. \n\n     c) It can be beneficial to compare the clustering time of UMFC with CoOp training time.\n\n7. In Line 228, it is union or intersection of class sets in ImageNet-A and ImageNet-R?\n\n[A] Data Determines Distributional Robustness in Contrastive Language-Image Pre-training (CLIP)\n\n[B] Maple: Multi-modal prompt learning.\n\n[C] Self-regulating Prompts: Foundational Model Adaptation without Forgetting.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNo clear negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**I am not an expert in this domain. So my review may not be informative.**\n\nThe paper introduces Unsupervised Multi-domain Feature Calibration (UMFC), designed to improve the adaptability of Vision-Language Foundation Models like CLIP to various downstream tasks across multiple domains using unlabeled data. The authors pinpoint model biases within CLIP’s visual and textual encoders that affect its performance across different domains. To mitigate these biases, UMFC employs two calibration modules: the Image Feature Calibration (IFC) and the Text Feature Calibration (TFC). These modules recalibrate the model's focus from domain-specific to category-level information, enhancing its generalization capability. The effectiveness of UMFC is demonstrated through significant performance improvements in unsupervised calibration, transductive learning, and test-time adaptation tasks.\n\nSTRENGTHS:\n1. UMFC addresses the core issue of domain shift by recalibrating both visual and textual features to be domain-agnostic\n2. The framework leverages the typically abundant but underutilized unlabeled data in practical scenarios, offering a cost-effective solution for enhancing model performance without the need for labeled data.\n3. The effectiveness of UMFC is robustly validated across multiple downstream tasks, demonstrating its practical utility and the potential for wide application in real-world scenarios.\n\nWEAKNESSES:\n1. While effective, the calibration process might introduce additional complexity in terms of understanding and implementing the recalibration mechanisms, particularly the calculation and subtraction of domain-specific biases.\n2. There's a risk that overly aggressive calibration could lead to overfitting on the specific domains included in the training set, potentially reducing the model’s ability to generalize to entirely new or unseen domains.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper attempts to address the problem of domain gap in existing VLMs. Based on an unlabelled mixed dataset, this paper first proposes to fit each different domain and remove the domain bias of image features by Gaussian mixture model. Furthermore, the domain bias of text features is similarly removed by the consistency of image and language modalities. The method is validated on several different tasks.

STRENGTHS:
The methodology is simple and easy to understand, and the experiments are relatively extensive.

WEAKNESSES:
The proposed idea is simple and reasonable. My major concern is whether the proposed methodology is a real improvement over CLIP-D (e.g., 61.68 vs. 61.03 in unsupervised calibration), considering the requirement for extra computational overhead and unlabeled dataset.  

0. how does it compare to CLIP-DE (Domain-Specific Ensemble Prompting)? 
1. the Gaussian assumption is a bit too strong for each domain.
2. I would assume the dataset to be unlabelled, as described in L179-L180. However, for text feature calibration, the description in L208 is _'By clustering image features, we can compute the average feature $µ_i$ of unlabelled images in each domain $i$.
representing the domain-specific features of that domain'_ . Does the domain here refer to the clustering in the image feature calibration step?
3. Equation (3) should be $\triangleq$ .
4. More details are needed for each experimental section.

QUESTIONS:
See weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17