UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Authors: Jiachen Liang
RuiBing Hou
Minyang Hu
Hong Chang
Shiguang Shan
Xilin Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: model calibration
test-time adaptation
CLIP
multi-domain
Primary area: machine_vision
TL;DR: This paper proposes UMFC, which calibrate features from multi-domain in a label-free and training-free way.

Abstract: Pre-trained vision-language models (e.g., CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data to adapt to downstream tasks, which could be costly. In this work, we aim to  leverage unlabeled data that naturally spans multiple domains to enhance the transferability of vision-language models.  Under this unsupervised multi-domain setting, we have identified inherent model bias within CLIP, notably  in its visual and text encoders. Specifically, we observe that CLIP’s visual encoder tends to prioritize  encoding domain over discriminative category information, meanwhile its text encoder exhibits a preference for domain-relevant classes. To mitigate this model bias, we propose a training-free and label-free feature calibration method, Unsupervised Multi-domain Feature Calibration (UMFC). UMFC estimates image-level biases from domain-specific features and text-level biases from the direction of domain transition. These biases are subsequently   subtracted from original image and text features separately, to render them domain-invariant. We evaluate our method on multiple settings including transductive learning and test-time adaptation. Extensive experiments show that our method outperforms CLIP and performs on par with the state-of-the-arts that need additional annotations or optimization.
Our code is available at https://github.com/GIT-LJc/UMFC.

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Jiachen Liang!? Ruibing Hou! Minyang Hul.2, Hong Chang Shiguang Shan Xilin Chen 1.2
1 Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
gjiachen.liang, minyang.hu| vipl.ict.ac.cnn (houruibing, changhong, sgshan, xlchen] @ict.ac.cn
Abstract
Pre-trained vision-language models (e.g.. CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data adapt downstream tasks, which could be costly. In this work, we aim to leverage unlabeled data that naturally spans multiple domains to enhance the transferability of vision-language models. Under this unsupervised multi-domain setting, we have identified inherent model bias within CLIP, notably in its visual observe that CLIP's visual encoder tends
and text encoders. Specifically, prioritize encoding domain over discriminative category information, meanwhile encoder exhibits a preference for domain-relevant classes. To mitigate this model bias, we propose a training-free and label-free feature calibration method, Unsupervised Multi-domain Feature Calibration (UMFC). UMFC estimates image- level biases from domain-specifts features and text-level biases from the direction domain transition. These biases are subsequently subtracted from original image and text features separately, to render them domain-invariant We evaluate our method on multiple settings including transductive learning and test-time adaptation. Extensive experiments show that our method outperforms CLIP and performs on par with the state-of-the-arts that need additional annotations or optimization. Our code is available at https: // thub. com/GIT-LJe//Uen
categorr
1 Introduction
Recently, Vision-Language Foundation Models (VLFMs) such as CLIP [30], BLIP [24], Flamingo [1] and ALIGN [20] have demonstrated remarkable performance across various downstream tasks. These VIFMMs formulate the training objective as contrastive learning, leveraging millions of image- text pairs to establish a shared embedding space. Equipped with a wide range of visual and text representations, VLFMs exhibit the capability to tackle downstream tasks in a zero-shot manner.
Despite VLFMs being exposed to abundant examples, they may still encounter examples with new variations in downstream tasks. To address the problem of distribution shift between the pre-training and downstream domains, natural approach involves fine-tuning VLFMs on various target tasks, such as prompt engineering [39, 38] and adapter learning [11, 37]. However, these methods generally require labeled samples for fine-tuning, which prohibitively expensive be satisfied reality. Conversely, abundant unlabeled data are often available for downstream tasks. Notably, in practical scenarios, the unlabeled data typically contains multiple domains, which exacerbates the adaptation difficulty of VLFMs. Therefore, in this paper, aim improve the adaptation performance of VLFMs directly on unlabeled data that naturally spans multiple domains.
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

14-
28
105
18
(a)
(b)
(C)
Figure I: On DomainNet datasct, we visualize (a) The accuracy of CLIP on the six domains. (b) The image features extracted by CLIP's image encoder across different domains. The visualization show that CLIP exhibits inherent model bias. (c) The number of predictions for different classes quickdraw and painting domains.
In this unsupervised multi-domain setting, we observe that CLIP cannot perform well when unlabeled data are drawn from mixed distributions. As shown in Figure (a),, even within the same class space, the accuracy of CLIP varies significantly across different domains. While CLIP performs exceptionally well for images from common distributions encountered during pre-training, e-g., achieving 83.0% accuracy on real domain, it struggles with rare distributions encountered during pre- training, e-g., only achieving 14.2% accuracy on quickdraw domain. Above observations highlight that CLIP exhibits model biases that lead to incorrect predictions in specific scenarios. This raises fundamental question: where do these biases originate?
We point out that these model biases stem from deficiencies in the visual encoder and textual encoder. On the side of visual encoder, we observe that CLIP' visual encoder prioritizes encoding domain information over discriminative category information. As shown in Figure 1(b), features from the same domain clearly cluster together, whereas a notable gap separates features from different domains. This phenomenon indicates that CLIP's visual encoder exhibits higher sensitivity to domain information over category information. When the domain of downstream tasks shifts from pre-trained tasks, the mismatch in domain-specific information encoded in image features could adversely affect classification accuracy. On the side of textual encoder, we observe that CLIP demonstrates varying category preferences across different domains. Specifically, CLIP tends to classify images into categories whose name are closely related to corresponding domain. As shown in Figure 1(c), within "quickdraw" domain, a large portion of samples (~ 30%) classified as "squiggle" " or "line" categories. Conversely, within "painting" domain, CLIP favors categories like "paintean" and "paintbrush". This observation shows that the class embeddings encoded by CLIP's textual encoder inherently carry domain-specific information, misleading the model to prioritize categories highly associated with respective domains. Due the combined effects of visual and textual encoder biases, CLIP's performances vary significantly across different domains.
In this paper, we aim calibrate CLIP to mitigate the model biases. Initially, analyze the model biases from a probabilistic standpoint. The factors influencing p(y|æ)) affected by domain variable 2, can be decoupled into two parts: the sample distribution conditioned classes p(xly, z) and the class distribution ). If the two probability distributions are independent of 2, domain shifts will not affect the predictions. To this end, we propose Unsupervised Multi-domain Feature Calibration (UMFC), a simple yet efficient framework for calibrating CLIP to generalize to various downstream tasks using multi-domain unlabeled data. UMFC jointly calibrates CLIP through two training-free modules: Image Feature Calibration module (IFC) and Text Feature Calibration module (TFC). Firstly, IFC focuses calibrating CLIP's image encoder prioritize category- over domain-level information, thereby reducing prediction error caused by domain shifts. Specifically, we calculate the average image features for each domain i, denoted as μ4, and posit that the prediction of μ4 reflects the inherent bias of CLIP's 's image encoder within that domain. By subtracting this domain-specific bias from original predictions, we can derive domain-agnostic predictions. Secondly, TFC focuses on calibrating CLIP's ' text encoder to remove its preference towards domain- related class names. As observed by [7, 28, 81, a "global direction" exists in CLIP, representing the shift from training distribution to unseen distribution, shared across image and text embedding spaces. Motivated by this observation, we utilize the shift direction between different-domain images to estimate the text shift direction. Then, we subtract this shift vector to counteract the text encoder's bias towards delain-related categories. By combing IFC and TFC, we can calibrate the features on both CLIP's image and text encoders, thereby alleviating the model bias in downstream tasks.
2

---

We validate the efficacy of UMFC on three downstream tasks: unsupervised calibration, transductive learning, and test-time adaptation, demonstrating consistent gains over CLIP. UMPC presents low- cost solution for classification, unlocking the potential of CLIP-like models for practical scenarios characterized by abundant images across multiple domains but scarce labels.
2 Related Work
Few-Shot Learning. Few-shot learning (FSL) [17, 16, 18, 10, 34] aims to learn a good model on
novel task with few labeled samples. Traditional FSL methods [17, 16, 18, 10, 34] often rely on abundant related training tasks for pre-training and design specialized algorithms transfer cross-task knowledge, facilitating task adaptation. Additionally, some semi superrsuee learning (SSL) methods [26, 14, 15] tackle the scarcity of labeled samples by assuming access to extensive unlabeled data. While both FSL and SSL methods have achieved encouraging results, their generalization abilities are limited. Recently, the CLIP model is proposed, which is a vision-language foundation model that learns a shared vision- vision-language embedding space. As pre-trained on large-scale data, CLIP exhibits impressive zero-shot ability across various downstream tasks. Based on CLIP model, many works focus on improving its performance further on downstream tasks with few labeled data. For example, CoOp [39], CoCoOp [38], MaPLe [21] and PromptSRC [22] leverage few labeled samples to learn the prompt in the continual input embedding space, offering sarfameter-efficientt way of fine-tuning foundation models. Similarly,
[11] and Tip-Adapter-F [37] introduce a lightweight adapter module to produce adaptive multi-modal features. However, these methods require extra labeled data and the process of tuning pre-trained parameters for CLIP, which is cost-expensive. Differently, in this work, focus utilizing unlabeled data to enhance CLIP's performance in training-free free manner.
Domain Adaptation / Domain Generalization. Recently, several methods [23, 3, 8, 31, 2, 33, 5, 19, 7] exploit a large-scale pre-trained model (e.g.. CLIP) to address the domain adaptation and generalization problems. For example, RISE [19] leverages CLIP as teacher to regularize the student's 's learned representation through images. The work [3] utilizes fomain-invarianl and domain- specific prompts for multi-source unsupervised domain adaptation. Other works [8, 7, 5] focus on utilizing the transfcrability between visual and textual modalities to guide domain information transfer. PromptStyler [5] attempts to simulate various distribution shifts explore diverse styles in a joint vision- languae space. PODA [8] and LADS [7] generate samples in the style of the target domain, and adapt to the target domain based on these samples. However, these language Language-guide guided methods require the prior of target-domaa names, which may not be satisfied in reality. In contrast, our UMFC method does not require any extra information about the target domain.
Test-Time Adaptation. Test-time adaptation aims to adapt a pre-trained model to test tasks, where distribution of test data differs from that of pre-training data. TPT [32] proposes test-time prompt tuning strategy, which extends traditional TTA methods to langune models. Building upon TPT, Di [9] utilizes pre-trained diffusion models augment the diversity of test data samples used in prompt tuning. SwapPrompt [27] employs a dual prompts paradigm to enhance the swapped prediction mechanism. However, these prompt learning methods are computationally expensive and time-consuming. Different from above methods, our UMFC only needs to calibrate features in a training-free way, making more efficient for test-time adaptation.
3 CLIP and Model Biases
In this section, we first describe the backgrounds of CLIP and then analyze its bias issue in downstream tasks. We attribute the cause of bias to the visual encoder bias and text encoder bias.
3.1 Contrastive Maanguage-Image Pre-training (CLIP)
CLIP [30] consists of two parallel encoders: a visual encoder that maps image inputs into image features, and text encoder that maps text inputs into text features. The model is trained with a contrastive loss that maximizes similarity between positive image-text pairs while minimizing similarity between negative pairs. This process ensures alignment between the features of images and their corresponding textual descriptions within the feature space. Trained on vast collection
3

---

of image-text pairs, CLIP benefits from general visual representations, endowing it with powerful zero-shot capabilities.
Formally, we denote a CLIP model as {F, 7), with and being the visual and text encoders. Considering K-class classification problem, we use y (Ul,-..,9KK. } represent the class space, where Vs denotes the class name of kthh class. In the zero-shot inference phase, CLIP uses hand-crafted prompts (such as "a photo of (}") to covert each class name yi to category-specific text description {p: ).)- Then, features {f,..... txx), where t; = T (p: y.). Meanwhile, given a test image z, the visual encoder is used to compute its visual feature, denoted f F (x)) The prediction probability on be then computed as:
feed these class descriptions to the text encoder get the text
exp(sim(J,4)/1
p(y)(a)=
Ej=1
K
where sim denotes the cosine similarity and is the temperature the softmax function.
3.2 Model Bias in CLIP
As shown in Figure I(a), the accuracy of CLIP varies significantly across different domains within the same class space, e.g., 83.0% in real domain and 14.2% in quickdraw domain. This phenomenon indicates that CLIP favors domains commonly encountered during pre-training (such as natural images). We attribute this model bias to visual encoder bias and text encoder bias.
Visual Encoder Bias. We observe that CLIP" visual encoder prioritizes encoding domain infor- mation over discriminative category information, as shown in Figure 1(b). Due the abundance of natural images in pre-training, CLIP's visual encoder and text encoder are well-aligned in the natural image domain. Consequently, when the style of input images largely deviates from natural domains (such as quickdraw and infograph style), the visual feature gap across different domain (as depicted in Figure 1(b)) hinders the text encoder's capacity process these shifted images effectively, leading a significant drop in classification accuracy.
Text Encoder Bias. We observe that CLIP exhibits preference for domain-related categories in specific domains, as shown in Figure I(c). In domains characterized by distinet styles, certain category names may carry domain-specific information. For instance, in quickdraw domain, where most images consist of lines and squiggle, CLIP demonstrates severe bias towards the "line" and "squiggle" categories. This leads to a large number of quickdraw samples being incorrectly classified into the two categories. Conversely, in the painting domain, all images inherently possess painting features, CLIP shows a strong preference for categories related to the concept of painting, such "paintcan" and "paintbrush".
4 Unsupervised Multi-domain Feature Calibration
In this section, continue to analyze the model biases issue from probabilistic view. Then give a detailed introduction of our method aimed at alleviating CLIP's biases.
4.1 Analyze Model Biases in a Probabilistic View
We start from the posterior probability p (ys
(y)(1z), (z)). introduced in Equation 1. Recall that our goal
to maximize the posterior probability p (v.(z) (z) for each image-label pair (r. (x.y:) of any domain. natural question arises: How different domains affect the posterior probability? We answer this question from a probabilistic view. Based on the Bayes" Theorem, we can drive that:
p (u)l P (1). (x)
E.P(,, 2.) (xx
where = is a latent variable that denotes the domain labels. We can further decompose the joint probability p x)) in Equation as p (yi,, xr, 2) p (xlyn, z) (ua(z) p (=). Assume that the probability distribution of domains (z) is uniform, thus maximizing the posterior probability

---

(Vilz) is equal to maximize the summation Σ P (x 2) p) (3h 2),
maxp (vxlz) = max Σ, (zllss 2) (ya (0lll)))1 (z)
((x)
max Σ
p (alyi, 2) (ulzz).
Equation 3 shows that the domains affect posterior probability distribution p (y|x) by disturbing two terms: the sample distribution conditioned on classes p (zly,2) and the class distribution p (y)2))
4.2 UMFC: Unsupervised Multi-domain Feature Calibration
As analyzed in Section 3.2, CLIP exhibits both visual and text encoder bias, impacting its general- ization ability to downstream tasks. As shown in Equation 3, these biases stem from the probability distributions (I y. 2) and (ylz). which are disturbed by domain information. To mitigate the model biases, an intuitive idea is make the two probability distributions p (aly) and p (y) independent of domain variable by calibrating image and text features. To this end, we propose training-free UMFC approach, consisting of Image Feature Calibration (IFC) module to alleviate visual encoder bias and Text Feature Calibration (TFC) module to alleviate text encoder bias, facilitating the transfer of CLIP to downstream tasks. Algorithms are provided in the Appendix C.
Image Feature Calibration Module. On the side of visual encoder, we focus on making the conditional probability p (xly) independent of domain variable 2, which is equivalent to achieving (zly, 2) (zzly). A straightforward method is to align image feature distributions given a class across different domains. Unfortunately, as only mixed unlabeled data provided, we cannot access to class labels and domain labels of images. However, we empirically observe that CLIP's visual encoder prioritizes encoding domain information over discriminative category information. Thus, we can distinguish image features from different domains through a simple clustering algorithm. Formally, we assume that there are M clusters (CI,..., Cas) } after applying a clustering algorithm. Each cluster is assumed to follow a Gaussian distribution
(N,,, E,), where mean vector ,4 = ( EyE:; f) represents the average of image features from cluster c, Due to the model biases, the pseudo labels produced by zero-shot CLIP are not reliable. Therefore, we directly align the margin image feature distribution of each cluster by subtracting the corresponding mean vector Specifically, for each visual feature f belonging to the cluster c. we calibrate it as follows:
f Mi
J 11l12
Text Feature Calibration Module. On the side of text encoder, we focus on making the class probability p (y) independent of domain variable z, which is equivalent to achieving (v/== (y). Previous works [35] have found that CLIP's performance is sensitive class name y, For example, replacing category names with synonyms or synnoyyyoss terms can sigmificantly impact CLIP's prediction results. Due to the influence of domain information image features, text encoder bias can cause CLIP to categorize domain-specito similar to that domain. We further observe that such sensitivity class names varies across different domains, as shown in Figure (c)). This observation inspires us to calibrate text features by removing Nomain-speciif information. However, a challenge arises as domain labels are unavailable. To address this issue, we attempt to extract domain information from unlabeled images, and then transfer this domain information to the text embedding space to estimate the text bias.
images into categories whose names are semantically
Transition Direction. The underlying assumption behind using images to simulate the corresponding shifts in texts is that the transition direction from domain to domain is consistent across both the image embedding and text embedding spaces [7, 28, 81, which can be formulated as:
Observation 4.1
F F(x) -
T''':')) - T (p²:8*)
(5)
|P F
-
"This operation is based on the assumption of a uniform class distribution in each cluster. We experimentally found that this operation remains effective, even if this assumption does not hold strictly.
5

---

where (a', v') and (z',8) represent training samples from domain and domain j respectively, p' and p² denote the domain-specific text prompts for domain i and j respectively. For example, the text prompt of "quickdraw" domain can be "a quickdraw image of [class]".
Inspired by Observation 4.1, we estimate the text-level domain transition direction using different- domain images. By clustering image features, we calculate the average feature μ4, of unlabeled images from each domain i, representing the domain-specifee feature for that domain. Also, we calculate the average feature of all unlabeled images Havg. representing the domain-invarian feature since it encompasses various domain distributions. Following Equation 5, the transition information of domain can be computed as 8 = 14 Havg- By subtracting this domain transition vector, we suppress the preference for specific class names in the original text features. To ensure that the calibrated text features effectively eliminate biases towards wide range of domains, we integrate the text features calibrated on each domain. Specifically, for the text feature ł, of class j, we calibrate it as follows:
M
M
1=
where M is the number of clusters.
Inference. After calibration with IFC and TFC modules, we obtain the final prediction results. Specifically, this calibration modifies the prediction probability on test image x as follows:
exp (sim (f', (t) /T)
p (vilx)
11=1 exp (sim (f', (2)) /T)
5 Experiments
5.1 Experimental Setting
Datasets. Our UMFC is training free, which only calibrates the image and text features using Equation 4 and 6. To analyze model's generalization capability,
two large-scale datasets for evaluation: 1) DomainNet [29] consists of 569,010 images with 345 categories from six domains: Clipart (C), Infograph (I), Painting (P), Quickdraw (Q), Real (R), Sketch (S). 2) ImageNer Variants composed of several datasets shifted from ImageNet, including ImageNet-A (IN-A) [13], ImageNet-R (IN-R) [12], and ImageNet-Sketto (IN-S) [36]. We form the class space for ImageNet Variants by laking the union of the class sets in IN-A and IN-R. To ensure the reliability of the evaluation results, we randomly sample the test data to construct a balanced test set where both domain and category distributions are uniform.
Evaluation Paradigms. Our approach is universal feature calibration technique that can be applied across multiple scenarios. In this work, we explore three settings: 1) Unsupervised Calibration (UC) where the unlabeled training set is provided for computing the calibration vectors; Transductive Learning (TL) where the entire unlabeled test set is provided at once, without providing any training data; 3) Test-Time Adaptation (TTA) where the model can be adapted test samples shifted from training distribution. Different from TL, the test data usually arrives in batches in TTA setting, More details on the experimental setup, please refer Appendix D.
5.2 Main Results on Unsupervised Calibration.
Implementation Details. We select CLIP [30] as our pre-trained vasion-language model. We use CLIP with ViT-B/16 [6] image encoder, and keep the original transformer the text encoder. By default, fixed prompt, "a photo of a [class]", is employed for all datasets. The images are resized to 224 x 224. The hyper-parameter M (cluster number) is set to 6 for NomainNet. Remarkably, our method is training-free free where both image encoder and text encoder remain frozen throughout the entire pipeline. All experiments are performed
GeForce RTX 3090 Ti GPU.
We compare our method with four groups of methods: (1) CLIP and its variants to show zero-shot predictions: CLIP [30] that uses fixed prompt "a photo of a [class]"; CLIP-E [30] that uses an ensemble of prompt templates. (2) CLIP-D [30] that utilizes the domain information of test samples and designs domain-specift prompts. (3) Few-shot learning methods: CoOp [39] that
Baselines.

---

Table 1: Results DomainNet under multi-domain Unsupervised Calibration. CLIP denotes zero-shot CLIP with a fixed text prompt template "a photo a [class]". CLIP-E uses the ensemble prompt templates designed for Imagenet [39], CLIP-D uses the domain-speccfic templates, i.e., "a [domain] image of [class]". CoOp and ELIP-Adapter are trained on multi-domain labeled data, e.g.. 6 x 1 x 345 denotes the number of labeled data.
Method
P
R
S
Avg
CLIP [30]
71.21
49.47
64.61
14.23
82,98
64,81
57.88
CLIP-R [30]
73.16
54,17
67.02
15.86
84.30
67.49
60.33
Unsupervised
CLIP-D D [30]
73.90
55.84
67.75
17.84
83,26
67.56
61.03
MUST|25]
74.83
56.48
61.80
19:06
82.88
70.31
60.89
UMFC (ours)
73.02
55.30
66.36
19.67
83.54
66.87
60.79
UMFC+ + CLIPEE (ours)
73.84
56.59
67.39
20.03
84.33
67.90
61.68
CoOp (6 x 1 [45)
72.73
53.95
66,80
19.58
82.53
67.27
60.48
Few-Shot
CoOp (6 x 4 x 139)
74.7
54,96
68.29
22.14
82.94
69.48
62.09
CLIP (1.PPer (6
345)
72.67
51.69
67.84
17.82
84.26
65.75
60.00
CLIP CLIP-Adapier (6 x 4 x 345) [1]]
74.35
53,79
69.94
19.71
85,26
66.90
61.66
performs prompt tuning using labeled data of downstream tasks; CLIP-Adapter [11] that trains an adapter using task- specifce labeled data. (4) Unsupervised Fine-tuning method: MUST [25] that fine-tunes the model using unlabeled multi-domain data.
Analysis. As shown in Table 1, UMFC achieves superior performance over CLIP and competitive performance with few-shot methods, CoOp [39] and ELIP-Adapted [11]. We can observe that: (1) CLIP-D incorporates domain information into text prompts, such as "a clipart image of [class]] for test samples from "clipart" domain. However, creating domain-spectin due to unknown and potentially mixed domain sources of test samples. So, we only use CLIP-D as an oracle result. As shown in Table 1, our method can achieve competitive performance with CLIP-D without prior domain labels for test samples. (2) On domains where CLIP performs poorly, e.g., "quicckraww", our UMFC significantly outperforms others. Compared to CLIP, UMFC achieves about 5% performance gain on quickdraw domain. In addition, combined with more diverse prompts of CLIP-E, UMFC further improves performance. (3) MUST [25] uses abundant unlabeled data from domains for fine-tuning. Our UMFC achieves better performance than MUST even without any additional training. (4) For CoOp [39] and CLIP-
templates is challenging
[11]. we fine-tune them using labeled samples from multiple domains. Specifically, we provide & labeled samples per class for each domain, resulting in a total of (6 x k x x345) labeled samples available for training. When the number of labeled samples is 6 x 345, our method outperforms few-shot fine-tuning methods. While the performance is higher for few-shot methods with a larger number of labeled samples (i.e., 24 x 345), it is essential to highlight the challenges in obtaining some labeled data for each class and each domain in real- world scenarios. In contrast, our method does not require selecting class-balanced and domain-balanced labeled samples and incurs no additional training overhead.
Table 2 provides 8 x 345 labeled samples from single domain for CoOp fine-tuning and an equal number of unlabeled samples for UMFC calibration. While CoOp trained on a single domain can improve performance within that domain, its performance declines other domains. This decline becomes particularly notable when a significant distribution gap between the training and test domains exists, e-g., CoOp (Q), leading to large decrease in average performance across multiple domains. In contrast, our method achieves consistent performance improvements on both training and unseen domains with the same amount of training data. Additionally, unlike CoOp, UMFC does not require labeled data or any parameter fine-tuning.
5.3 Main Results on Transductive Learning
In this part, we compare our UMFC with three groups of methods: (1) Zero-Shor CLIP models. (2) Domain Generalization (DG) method: MIRO [2] that trains CLIP on available data learn great generalization capability. (3) Data-Free DG method: PromptStyler [5] that leams to simulate diverse distributions. The hyper-parameter M (cluster number) is set to 3 for ImageNet- Variintts. The compared results are shown in Table 3 and Table 4. As seen, our UMFC can achieve the best average performance on DomainNet and ImageNet-Variin Variants benchmarks, which validates the effectiveness of approach in transductive learning.
7

---

Table 2: Results DomainNet under single-domain Unsupervised Calibration. 8 345 samples (each class samples) from single domain are provided. CoOp (C/Q/I) and UMFC (C/Q/T) denote training samples for CoOp and UMFC from the "Clipart" "iQuiann"""icaaaaaaa "aanekddrww""Innorrrree ''Inogaapph" domains, respectively.
Method
C
P
Q
R
S
Avg
CLIP [30]
71.21
49.47
64.61
14.23
82.98
64.81
57.88
CoOp [39]
74.55
42,66
55,94
13.82
75.00
58.73
53:45
UMFC (C)
73.27
52.96
65.27
16.94
83.60
67.04
59.85
CoUp (Q) 1391
43.97
25.5
32.63
29.07
48.44
38.74
36.39
UMFC (Q)
72.17
63.85
17.47
66.36
58.72
CoOp [39]
60.19
54.28
50.81
11.19
70.73
54.27
50.24
UMFC (I)
72.54
55.21
64.48
16.30
83.31
66.51
59.73
Table Comparison Results on DomainNet under Transductive Leariing
Method
C
[30]
71.21
CLIPEE
73.16
54.17
67.02
15.86
84.30
67.49
60.33
CLIP-D [30]
73.90
55.84
67.75
17.84
83.26
67.56
61.03
MIRO [2]
$4.00
PromptStyler [5]
73.10
50.90
69.20
13.30
85.40
65,30
59.40
UMFC
73.01
55.44
66.89
20.14
83.66
67.51
61.11
Table 4: Comparison Results on ImageNet- Variianss under Transductive Learning.
Table Comparison Results on ImageNet- Variants under Test-Time Adaptation.
Method
IN-A
IN-R
IN.S
Avg
Method
IN-A
IN-R
IN.S
Avg
CL1P
42.13
66.95
74,58
61.22
CLIP
42.13
66.95
74,58
61.22
CLIP-E [3]]
45.42
71.10
77.08
64.53
TPT [32]
47.16
59.95
67.49
58,20
UMFC
43.42
68.86
77.24
63.17
UMFC-Memory
42.76
67.03
75.15
61.65
UMFC CLIP-E
44.77
72.19
78.62
65.19
UMFC-EMA
42.21
67.82
76.26
62.10
5.4
Main Results on Test-Time Adaptation
Implementation Details. TTA setting, where we cannot access all data simultaneously, we adopt an incremental clustering approach. By default, the batch size is set to 100. Initially, we apply K-Means clustering algorithm to the first batch of data. For subsequent batches, we use the prototype classification, with cluster centers serving prototypes, to assign samples in that batch different clusters. Then, the cluster centers and calibration statics { are updated accordingly. We have developed two strategies for updating the calibration statics: UMFC-Memory that stores the feature information of each batch to calculate the statistical information; UMFC-EMA that uses Exponential Moving Average (EMA) to update statistical information.
M
Table Comparison Results on DomainNet under Test-Time Adaptation. UMFC-Memory and UMFC-EMA represent different ways to update the statics vectors for calibration.
Method
C
P
Q
R
Avg
CLIP [30]
71.21
49.47
64.61
14.23
82.98
64.81
57.88
TPT [32]
73.23
52.63
68.00
12.79
84.39
66.68
UMFC-Memory
72.82
55,12
66.82
19.92
83,62
66.82
60.85
UMFC-EMA
72.99
54.94
66.64
18.58
83.54
66.75
60.57
Analysis. We mainly compare our UMFC with TPT [32] in test-time adaptation, where TPT fine- tunes the prompts by minimizing the entropy the predictions. The comparison results are shown in Table 6 and Table 5. We can observe that: (1) As shown in Table 6, UMFC achieves performance gains across all domains, with particularly noticeable gains in those where CLIP performs poorly. For example, UMFC substantially improves upon CLIP on "quickdraw" and "infograph" domains, with
8

---

8.5
Caltoration
2.4
probokatty
-22
3ehth
(a)
(b)
Figure 2: On DomainNet dataset, we visualize (a) The image features extracted by UMFC image encoder across different domains. (b) The classification probabilities of CLIP's text features on different domains.
Table 7: Ablation study on the effects of TFC and IFC under Transductive Learning.
Method
C
P
Q
R
5
Avg
CLIP [30]
71.21
49.47
14.23
57.88
IFC
72.98
55.07
66.65
19.87
83,54
66.97
60.85
TFC
71.44
49.99
65.59
13.9
83.25
64.68
58.14
UMFC
73.01
55.44
66.89
20.14
83.66
67.51
61.11
an accuracy gain of 5.6%. Table 5 shows that the improvement on ImageNet-A is less significant than on ImageNet-R and ImageNet-S. The reason may be attributed to the absence of distinct domain styles in ImageNet-A, limiting the effectiveness of our calibration method that relies on domain information. (2) UMFC-Memory and UMFC-EMA with different statistical update strategies exhibit similar performance. However, UMFC-EMA, which updates features based on the most recent data, notably reduces storage requirements. Moreover, UMFC exhibits significantly higher computational efficiency without training, whereas TPT requires executing optimization steps on different augmented views of each test image. Thus, our method is more suitable for rapid deployment.
5.5 Ablation Study
The effectiveness of IFC. We firstly evaluate the effectiveness of IFC. As shown in Table 7, IFC individually contributes to performance gains of CLIP, about 3% average gains. Furthermore, Figure 1(b) and Figure 2(a) visualize the image features extracted by CLIP with/without IFC respectively. As shown in Figure 1(b), the vanilla CLIP maps different-domee
images to different clusters in the feature space. Conversely, Figure 2(a) shows that IFC leads to the merge of image features from different domains, validating its effectiveness of reducing domain-specifer information.
The effectiveness of TFC. As shown in Table 7, TFC also individually contributes to performance gains of CLIP. To assess the effectiveness of calibrated text features in eliminating domain bias, we construct domain classifier. Specifically, we utilize the text features generate by CLIP, with domain prompts "[domain]", as the domain classifer. Then, we use this domain classifier to perform domain classification for the features before and after calibration (i.e., computing the cosine similarity between domain classifier and text features). As shown in Figure 2(b), the original text features exhibit a long-tai phenomenon, with a higher probability of being classified into the "real" domain. This suggests that the original text features are biased towards the "real" domain, overlooking their generalization ability to other domains. However, after calibration with TFC, the text features exhibit a near-uniform distribution across domains, indicating that the calibrated text features have largely eliminated domain bias. Consequently, our method enhances performance in other domains without compromising the model's performance in its strongest domain. As shown in Table 7, the combination IFC and TFC (i.e., UMFC) can further bring performance gains, which validates the complementary of image-level calibration and text-level calibration for CLIP.
The impact of cluster number M. Our method involves clustering the unlabeled data to determine their respective clusters. We evaluate our method with respect to the number of clusters M. As shown

---

Table 8: The impact of cluster number M DomainNet under Transductive Learning.
Method
P
Q
R
S
Avg
CLIP [30]
71.21
49.47
64.61
14.23
82.98
64.81
57.88
UMFC (M=3)
72.53
54.60
66.31
20.32
83.35
66.86
60.66
UMFC (M=4)
73.55
56.36
67.19
20.62
84.13
67.69
61.59
UMFC (M=6)
73.01
55.44
66.89
20.14
83.66
67.51
61.11
UMFC (M=8)
73.50
56.58
67.53
20.64
84.06
67.92
61.71
UMPC (M=10)
73.63
36.87
67.81
20.23
84.20
67.87
61.77
Table 9: The impact of batch size under Test-Time Adaptation.
Mathed
C
P
Q
R
Avg
CLIP [30]
71.21
64.61
14.23
82.98
57.88
UMFC (bs=1)
72.64
53.74
66.39
18.25
83.34
66.90
60.21
UMFC (bs=10)
72.64
UMFC (bs=16)
72.70
54,80
66.91
19.11
83.78
66.69
60.66
UMPC (bs=32)
73.02
66.73
19.17
83.66
66.97
UMPC (bs=64)
73.23
55,04
66.72
19.15
83.78
66.84
60.79
UMFC (bs=100)
72.82
55,12
66.82
19.92
83.62
66.82
60.85
in Table 8, our UMFC consistently outperforms vanilla CLIP, even when the number of clusters M does not match the number of domains (6 for DomainNet). More importantly, our method is not sensitive changes in M. Refer the Appendix E for more analysis.
The impact of batch size in TTA setting. We present results across various batch sizes during test-time adaptation to confirm that UMFC is robust to batch size variations. When the sample count is initially lower than the number of clusters M, K-Means clustering cannot be directly applied. To address this, we used the first M samples as the initial cluster centers and then proceeded with the same test-time adaptation. As shown in Table even in the extreme case of batch size of 1, our method still demonstrates consistent improvement.
Conclusion
In this work, point that the model biases hinder the transfer ability of pre-trained vision- language models. We develop UMFC, a simple unsupervised calibration method that mitigates model biases in both visual encoder and text encoder through training-frre manner. We demonstrate the effectiveness of our method across multiple settings, including unsupervised calibration, transductive learning, and test-time adaptation. Without the need for any annotations and training, UMFC improves the zero-shot generalization ability of CLIP.
Linitations and Broader Impacts. Our method requires unlabeled samples from the target domain to establish domain-level calibration vectors. Although our method is much cheaper and more accessible than fine-tuning or few-shot methods [39, 38, 32], the requirement of unlabeled data can still be a limitation for some specific scenarios.
Acknowledgments
This work is partially supported by National Science and Technology Major Project No. 2021ZD011 11901, National Natural Science Foundation of China (NSFC): 62376259 and 62306301, National Postdoctoral Program for Innovative Talents under Grant BX20220310.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
10

---

[2] Junbum Cha. Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. In ECCV, 2022.
[3] Haoran Chen, Zuxuan Wu, and Yu-Gang Jiang. Multi-prompt alignment for multi-source unsupervised domain adaptation. ArXiv, (ttp/2209.152111 2022.
[4] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
[5] Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In ICCV, 2023.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[7] Lisa Dunlap, Clara Mohri, Devin Guillory, Han Zhang, Trevor Darrell, Joseph E. Gonzalez, Aditi Raghunanthan, and Anja Rohrbach. Using language to extend to unseen domains. In ICLR, 2023.
[8] Mohammad Fahes, Tuan-Hung Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Poda: Prompt-driven zero-shot domain adaptation. In ICCV, 2023.
[9] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. 2023.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICML, 2017
[11] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132:581-599 2024.
[12] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of- distribution generalization. ICCV, 2021.
[13] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Xiaodong Song. Natural adversarial examples. In CVPR, 2021.
[14] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Dual compensation residual networks for class imbalanced learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45:1 1773-1 1752, 2023.
[15] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Triplet adaptation framework for robust semi-supereised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[16] Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguan Shan, and Xilin Chen. Task attribute distance for few-shot learning: Theoretical analysis and applications. arXiv, 2024.
[17] Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Under- standing few-shot learning: Measuring task relatedness and adaptation difficulty via attributes. In Advances in Neural Information Processing Systems, 2023.
[18] Minyang Hu, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Scalable modular network: A framework for adaptive learning via agreement routing. In ICLR, 2024.
[19] Zeyi Huang, Andy Zhou, Zijian Lin, Mu Cai, Haohan Wang, and Yong Jae Lee. A sentence speaks a thousand images: Domain generalization through distilling clip with language guidance. In ICCV, 2023.
11

---

[20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[21] Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. CVPR, 2023.
[22] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsure
Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023.
[23] Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen, Fu, and Chen-Nee Chuah. Padclip: Pseudo- Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In CCCV,2022
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICMI., 2022.
[25] Junnan Li, Silvio Savarese, and Steven C. H. Hoi. Masked unsupervised self-training for label-free image classification. In ICLR, 2022.
[26] Jiachen Liang, Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Generalized semi-supervised learning via self-supervised feature adaptation. In NeurIPS, 2023.
[27] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In NeurlPS, 2023.
[28] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen- -r, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[29] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[31] Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal representations for covariate shift. In ICLR, 2022.
[32] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurlPS, 2022.
[33] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distributions. In ICML, 2023.
[34] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, 2017.
[35] Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G. Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. In ICLR, 2023.
[36] Haohan Wang, Songwei Ge, Eric Xing, and Zachary Chase Lipton. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019.
[37] Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-frr free adaption of clip for few-shot classification. In ECCV, 2022.
[38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lasionn models. In CVPR, 2022.
[39] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130:2337-2348, 2022.
12

---

A The Prevalence of the Observed Model Bias
Our method is motivated by observed biases in CLIP. Therefore, is valuable investigate whether these observations hold when the pre-training distribution and model architecture are altered.
We further verify this OpenCLIP[4]. OpenCLIP investigates scaling laws for contrastive language- image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. The large-scale experiments involve models trained up two billion image-text pairs (LAION-2) 2B) and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We used OpenCLIP models trained on different datasets to obtain the corresponding image features, and visualized them using t-SNE, As shown in Figure 3, even when the training data changes, images from different domains still cluster together, demonstrating that our observation is universally applicable.
Additionally, we validated the effectiveness of our method on the OpenCLIP series models in Table 10. evident that the zero-shot capabilities of larger-scale models have significantly improved compared to CLIP, and our method still provides further performance gains. This demonstrates that our method universally applicable to Vision- SaiiunLa models with different architectures and training data.
VIT---SE (OIM)
VITH8-16 (000M)
OpenCLIP models trained with three different dataset (LAION 80M, LAION 400M 28).
(b) The Opentui IP models are pre-trained on LAION-28 with different number seen samples [SRt 548) and architectures (VIT8-16, WIT--32).
Figure 3: Visualization of Image Features based on OpenCLIP series.
Table 10: Comparision Results on DominaNet using OpenCLIP.
Arch
Avg
VIT-B-16
78.66 78.93
56.16
71.09
15.25
86.93
72.26
63.39
UMFC
58.29
71.78
21.56
86.36
73.29
65.04
VIT-B-32
70.54
61.59
UMPC
76.99
53.08
68.70
22.61
85.17
71.27
62.97
VIT-H-14
63.21
18.20
79.18
68.31
UMFC
83.74
63.89
76.41
23.73
89.33
79.49
69.43
B Details of Text Feature Calibration Module
Since the domain labels are unavailable, we attempt to extract domain information from images in downstream tasks, and then transfer this domain information to the text embedding space to climinate
13

---

the text bias. Inspired by Observation 4.1, we can estimate the domain transition direction in text embedding space by using images from different domains, as shown Figure 4.
Vision Joint
quicddoo image of
infegraph ufabidd
ollarri Inagrofahlld
of beid
Image Feoture
Text Feature
Modality Beundary
Figure 4: The domain transition direction between texts is similar to that between images.
C
Algorithm
Algorithm summarizes the proposed UMFC method under Time Adaptation (TTA) setting. Algorithm 2 summarizes the proposed UMFC method under Unsupervised Calibration (UC) Transductive Learning (TL).
Algorithm 1 Algorithm UMFC under TTA
B N 1: Input: test data {uo (ub,ih==1 i=1, batch size B, total number of batches N, total number of clusters M, CLIP's image encoder F(-) and text encoder T(p: :), p the text template, class names [ur)-
2: 4- T(p; (
D Get original text features
3: Initialize centroids of M cluster (c as empty set
4: fori 1-to N do
5:
{fo.s +- F((us,))
> Get original image features of i-th batch
6: if [c is then
7:
{c, 11=1 + K-Means( K-Means(ffo.s)l
A
Using K-Means to acquire M centroids
8:
end if
9:
for b - 1 to B do
16, <- arg min [| Cmila
Assign sample to the nearest clusters
11:
end for
cf
B BB I(bb, m) fo.i
Calculate centroids of current batch Update prototypes, where r) the update weight D update the average image feature b Update calibration statics
CK
Ck +1 cg.
- M M &m
14:
Hang
15:
Im + I[[ = m)(c)
Hang)
t M
MM Em=1 ||t-
t-
17:
for be -- to B do
18:
K6: + [
D. Calibrate f and with IFC and TFC
19:
end for
20:
end for
21: return (fr 1
Calibrated image and text features
14

---

Algorithm 2 Algorithm UMFC under UC/ TL
1: Input: unlabeled data [ub]a=1 total number of samples N, total number of clusters M, total number of batches B, CLIP's image encoder F(.) and text encoder T(p :). is the text template, class names {vi 1C
2: t T T(p {uA 10 Ji=
> Get original text features Get original image features of unlabeled data >> Get the average of {fo i==1 P Using K-Means to acquire M clusters and their centroids
3: (f6)-1 t ((16)11)
4: Maeg t Ei=i fo
5: {ci t K--MMens((foobaa)
6: for + 1to do
7: t arg min Il5b Cmll2 m
Assign sample to the nearest clusters P Update calibration statics > Calibrate and with IFC and TFC
8:
ts c II(1 m) mcm Marg),
f + lif-ql + M6 Em=1 1-p"
fn-c
9:
||-
10: end for
11: return (fN) 1'
Calibrated image and text features
D Experimental Settings
Our work can be deployed across various scenarios, including Unsupervised Calibration (UC), Test-Time Adaptation (TTA), and Transductive Learning (TL).
Unsupervised Calibration. In the UC scenario, we provide an unsupervised training set and use K-Means assign cluster labels to the training samples, while also saving the corresponding cluster prototypes. Then, we create an unlabeled training set from a mixed domain by sampling 16 instances from each class across domains in DomainNet. UMFC derives image bias and text bias for different clusters based on the cluster labels. During the testing phase, we first predict the cluster labels of the test samples using the cluster prototypes obtained during training, and then calibrate the predictions using the bias information derived from UMFC.
Test- Time Adaptation. In the TTA scenario, training data is provided. Test data from mixed domains arrive in batches, with a batch size set to 100. For the first batch, we perform initial clustering using K-Means. For subsequent batches, prototypes and continuously update these prototypes. Once the cluster labels are obtained, UMFC calculates the bias information for the current batch, updates the bias information for each cluster based on the new labels, and then calibrates the data for the current batch.
assign cluster labels the samples based on cluster
Transductive Learning. In the TL scenario, we provide the entire test set. Similar to UMFC, gather statistical information and calibrate the predictions for the test data. TL can be viewed as an extreme case of TTA, where the entire test set is treated as single batch.
E Experimental Analysis
The impact of cluster number M. We evaluate method with respect to the number of clusters M and demonstrate that our method is not sensitive to the choice of this hyperparameter. As shown in Table 11, For instance, setting M from 2 to 6 all leads to improvements on ImageNet-Varte Variants.
Table 11: The impact of cluster number M on ImageNet- Variants under Transductive Learning.
Method
IN-A
IN-R
IN-S
Avg
CLIP [30]
42.13
66.95
74.58
61.22
UMFC (M=2)
45.35
71.71
77.37
64.81
UMPC (M=3)
44.77
72.19
78.62
65.19
UMPC (M=6)
45.29
71.33
71.59
64.74
Computation Cost. For analyzing computational cost, we report the training time, inference time and memory of UMFC and other comparison methods under different scenarios. In Unsupervised
15

---

Calibration scenario, the entire unlabeled training set is provided for training. Corresponding computation cost comparisons are shown Table 12. Firstly. UMFC incurs minimal training and inference overhead compared to CL.P. This is because UMFC only requires single forward pass to extract features and then calculate statisties for feature calibration. Secondly, when compared to few-shot fine-tuning methods (CoOp) and unsupervised fine-tuning methods (MUST), UMFC also demonstrates lower consumption of computational resources and time. In Test-time Adaptation scenario, no training data is provided and the test data arrives in batches. Corresponding computation cost comparisons are shown Table 13. UMFC requires less memory than TPT and shows greater computational efficiency. Specifically, UMFC takes only 296 seconds, whereas TPT requires nearly 197 minutes. This is because TPT requires fine-tuning the text prompt for each test sample and augmenting each test sample 64 times to ensure the reliability of the fine-tuning results, which significantly slows down TPT's inference speed.
Table 12: Computation Cost under Transductive Learning.
Method
Training Time
Inference Time
Epoch
Memory
86 seconds
1797M:B
MUST [25]
10 hours (2 GPUs)
92 seconds
30
25944MiBB
CoOp (6*1 shot) [39]
32 minutes
83 seconds
7007MBB
CoOp (6*4 shots) [39]
160 minutes
83 seconds
100
7007M1B
UMFC
57.3 seconds
86 seconds
1887MiB
Table 13: Computation Cost under Test-Time Adaptation.
Method
Inference Time
Memory
TPT UMFC
minutes
6872MiB
296 seconds
1790MiB
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
A summary of the paper's contributions is provided at conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Not Applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5.2, 5.4 and supplemental material for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
See httpp:///githus www.:ilcccmmI---tC...
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct /ttpa//gguiieee/eeeuuuuiiiiiiiii
/CodeSubmic sswonnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 5.2, 5.4 and supplemental material for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.2 and supplemental material for implementation details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt////eeur..p.eeeeee cc cc/public//eeecccccccccccec www.ehhiiiiieeeeeeeeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See References.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "d4kN3i4ub1", "reviewer": "Reviewer_HbLd", "summary": "This paper attempts to address the problem of domain gap in existing VLMs. Based on an unlabelled mixed dataset, this paper first proposes to fit each different domain and remove the domain bias of image features by Gaussian mixture model. Furthermore, the domain bias of text features is similarly removed by the consistency of image and language modalities. The method is validated on several different tasks.", "questions": "See weakness.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The methodology is simple and easy to understand, and the experiments are relatively extensive.", "weaknesses": "The proposed idea is simple and reasonable. My major concern is whether the proposed methodology is a real improvement over CLIP-D (e.g., 61.68 vs. 61.03 in unsupervised calibration), considering the requirement for extra computational overhead and unlabeled dataset.  \n\n0. how does it compare to CLIP-DE (Domain-Specific Ensemble Prompting)? \n1. the Gaussian assumption is a bit too strong for each domain.\n2. I would assume the dataset to be unlabelled, as described in L179-L180. However, for text feature calibration, the description in L208 is _'By clustering image features, we can compute the average feature $µ_i$ of unlabelled images in each domain $i$.\nrepresenting the domain-specific features of that domain'_ . Does the domain here refer to the clustering in the image feature calibration step?\n3. Equation (3) should be $\\triangleq$ .\n4. More details are needed for each experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**My major concern is whether the proposed methodology is a real improvement over CLIP-D (e.g., 61.68 vs. 61.03 in unsupervised calibration), considering the requirement for extra computational overhead and unlabeled dataset.**\n\nThanks for this question, and I would like to clarify this point in the following.\n\n- **Improvement over CLIP-D.** In CLIP-D, we design customized prompts that incorporates the corresponding domain label and name for each test sample. \nCompared to vallina CLIP, CLIP-D shows better performance, which demonstrates the benefit of considering domain information under multi-domain scenarios.\nHowever, our comparison between UMFC and CLIP-D is not intended to show that UMFC performs better than CLIP-D, as CLIP-D utilizes domain labels and names for all test samples, while UMFC does not. \nInstead, this comparison aims to illustrate that UMFC can extract domain information from data **without supervision**, making it more versatile in various situations.\n\n- **Disadvantages of CLIP-D.** Although CLIP-D a is simple and effective method in multi-domain scenarios, it has many disadvantages: 1. it requires domain labels and names for all test samples. 2. it is sensitive to the choice of domain names. See more details in our global response R1.\n\n**W0: How does UMFC compare to CLIP-DE (Domain-Specific Ensemble Prompting)?** \n\nWe build CLIP-DE by replacing the single domain-specific prompt in CLIP-D with an ensemble of prompt template with domain names. As shown in the table below, CLIP-DE performs only marginally better than CLIP-E, and slightly worse than CLIP-D on DomainNet. We analyze this is because the ensemble strategy weakens the domain information utilized in CLIP-D. Furthermore, our UMFC outperforms CLIP-DE, validating its superiority.\n\n| Method     | C | I | P | Q| R | S | Avg |\n|------------|-----------|----------|----------|----------|----------|----------|----------|\n| CLIP-E | 73.16 | 54.17 | 67.02 | 15.86 | 84.30 | 67.49 | 60.33 |\n| CLIP-D | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | 61.03 |\n| CLIP-DE| 73.62 | 54.34 | 67.64 | 16.34 | 84.49 | 67.45 | 60.65 |\n| CLIP-E+UMFC       | 73.84 | 56.59 | 67.39 | 20.03 | 84.33 | 67.9  | **61.68** |  \n\n\n\n**W1: Gaussian assumption is a bit too strong for each domain.**\n\nThanks for your suggestions. In domain generalization, it is commonly assumed that domain features follow a  multivariate Gaussian distribution [1]. Moreover, since our method relies only on the mean values for feature calibration (Equation 4 of main paper), it can generalize to other distributions where the mean effectively captures the overall information.\n\n> [1] Domain Generalization With Adversarial Feature Learning. CVPR 2018\n\n\n**W2: I would assume the dataset to be unlabelled, as described in L179-L180. However, for text feature calibration, the description in L208 is *'By clustering image features, we can compute the average feature µ of unlabelled images in each domain . representing the domain-specific features of that domain'* . Does the domain here refer to the clustering in the image feature calibration step?**\n\nYes, we apologize for this confusion. Here, 'domain' refers to the cluster labels assigned during the clustering stage. We will revise it in the final version.\n\n**W3: Equation (3) should be $\\triangleq$.**\n\nThank you for pointing this! We will revise it in final version.\n\n\n**W4: More details are needed for each experimental section.**\n\n  Thanks for your suggestions. Our work can be deployed across various scenarios, including Unsupervised Calibration (UC), Test-Time Adaptation (TTA), and Transductive Learning (TL).  We will add the following details for the experimental section in final version. \n\n   + In the UC scenario, we provide an unsupervised training set and use K-Means to assign cluster labels to the training samples, while also saving the corresponding cluster prototypes. Then, we create an unlabeled training set from a mixed domain by sampling 16 instances from each class across 6 domains in DomainNet. UMFC derives image bias and text bias for different clusters based on the cluster labels. During the testing phase, we first predict the cluster labels of the test samples using the cluster prototypes obtained during training, and then calibrate the predictions using the bias information derived from UMFC. \n\n   + In the TTA scenario, no training data is provided. Test data from mixed domains arrive in batches, with a batch size set to 100. For the first batch, we perform initial clustering using K-Means. For subsequent batches, we assign cluster labels to the samples based on cluster prototypes and continuously update these prototypes. Once the cluster labels are obtained, UMFC calculates the bias information for the current batch, updates the bias information for each cluster based on the new labels, and then calibrates the data for the current batch.\n\n   + In the TL scenario, we provide the entire test set. Similar to UMFC, we gather statistical information and calibrate the predictions for the test data. TL can be viewed as an extreme case of TTA, where the entire test set is treated as a single batch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wed25xijb7", "reviewer": "Reviewer_Xgmy", "summary": "**I am not an expert in this domain. So my review may not be informative.**\n\nThe paper introduces Unsupervised Multi-domain Feature Calibration (UMFC), designed to improve the adaptability of Vision-Language Foundation Models like CLIP to various downstream tasks across multiple domains using unlabeled data. The authors pinpoint model biases within CLIP’s visual and textual encoders that affect its performance across different domains. To mitigate these biases, UMFC employs two calibration modules: the Image Feature Calibration (IFC) and the Text Feature Calibration (TFC). These modules recalibrate the model's focus from domain-specific to category-level information, enhancing its generalization capability. The effectiveness of UMFC is demonstrated through significant performance improvements in unsupervised calibration, transductive learning, and test-time adaptation tasks.", "questions": "Please see weaknesses.", "limitations": "None.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. UMFC addresses the core issue of domain shift by recalibrating both visual and textual features to be domain-agnostic\n2. The framework leverages the typically abundant but underutilized unlabeled data in practical scenarios, offering a cost-effective solution for enhancing model performance without the need for labeled data.\n3. The effectiveness of UMFC is robustly validated across multiple downstream tasks, demonstrating its practical utility and the potential for wide application in real-world scenarios.", "weaknesses": "1. While effective, the calibration process might introduce additional complexity in terms of understanding and implementing the recalibration mechanisms, particularly the calculation and subtraction of domain-specific biases.\n2. There's a risk that overly aggressive calibration could lead to overfitting on the specific domains included in the training set, potentially reducing the model’s ability to generalize to entirely new or unseen domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: While effective, the calibration process might introduce additional complexity in terms of understanding and implementing the recalibration mechanisms, particularly the calculation and subtraction of domain-specific biases.**\n\nIn fact, our calibration method is straightforward and computationally efficient.\n\n- **Easy to implement:** The calibration process involves three main steps:\n  1. **Assigning cluster label:** (stage1) Unlabeled samples are clustered, and cluster labels are assigned.\n  2. **Collecting statistical information:** (stage2)  Statistics for each cluster are gathered based on the assigned labels.\n  3. **Feature calibration:** (stage3) The original image and text features are calibrated using the collected statistics.\n\n- **Computation efficient:** We utilize the k-means algorithm to cluster image features and calculate the mean feature of each cluster. Therefore, steps 1 and 2 are efficient and can be completed within one minute in our experiments. In step 3, the feature calibration mechanism does not require backpropagation or modification of model parameters, leading to minimal additional computational overhead. For further details, please refer to our response to Weakness 3 for Reviewer jj75.\n   \n\n**W2: There's a risk that overly aggressive calibration could lead to overfitting on the specific domains included in the training set, potentially reducing the model’s ability to generalize to entirely new or unseen domains.**\n\nThanks for this valuable suggestion.  To evaluate UMFC's generalization capability on unseen domains, we follow the unsupervised calibration setting in Section 5.2. We randomly select disjoint domains as unlabeled training or test data. Specifically, we randomly select 3 domains in DomainNet as unlabeled training data (source domains) and use the remaining 3 domains as test data (target domains). The results are shown in below table.\n\nFrom this table, we can observe that, UMFC consistently outperforms vanilla CLIP model across different source domains, demonstrating its robust generalization ability. Additionally, our method can be easily deployed in Test-Time Adaptation (TTA) scenarios. This allows for continuous updates of statistical information as new test samples from emerging domains arrive. This capability ensures that the model maintains its generalization effectiveness over new domains.\n\n\n| Method | Source Domain |     Target Domain    |  Target Domain        |      |\n|:------:|:-------------:|:--------------:|:--------------:|:----:|\n|        |               |       CQS      |       IPR      |  Avg |\n| CLIP   | -             | 71.2/14.2/64.8 | 49.5/64.6/83.0 | 57.9 |\n| UMFC   | CQS           | 73.6/20.5/67.7 | 56.4/66.2/84.2 | 61.4 |\n| UMFC   | IRP           | 73.3/17.7/67.8 | 56.2/67.5/84.3 | 61.1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "67H6weAbMR", "reviewer": "Reviewer_cRCJ", "summary": "This paper identifies the inherent model bias within CLIP, notably in visual and textual encoders. To mitigate the bias, authors propose a feature calibration method, termed Unsupervised Multi-domain Feature Calibration. Experiments in the setting of transductive learning and test-time adaptation show the effectiveness of the proposed method.", "questions": "See weakness.", "limitations": "No clear negative societal impacts.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-structured and easy to follow.\n2. The motivation of this paper is clear.", "weaknesses": "1. The proposed method is inspired by the observation in the TSNE figure (Figure 1(b)) that \"features from the same... different domains\". However, this does not appear to be the case. Features from QuickDraw, Painting, and Infograph are clustered, while features from Real, Clipart, and Sketch are dispersed throughout. This suggests that CLIP models can encode some domain information, but this ability is limited to specific domains. If the observation which inspires the method does not hold, the rationale for the proposed method becomes less convincing.\n\n2. For figure 1(c), it is suggested to show the percentage of predictions rather than the absolute numbers. Each domain in DomainNet has different number of samples in validation set. Comparing the number of predictions is less convincing to show the bias of textual encoders. In addition, only Painting and Quickdraw are shown. It would be interesting to evaluate such bias in other domains.\n\n3. It is important to reveal the process of hyper-parameter selection. The cluster number is set to 6 for DomainNet, which is equal to the number of domains in DomainNet. Does it mean that the proposed method need prior knowledge on the number of domains in the target dataset? Furthermore, it is also valuable to reveal the hyper-parameter in ImageNet-variant experiments.\n\n4. It has been show that pre-training dataset distribution determines the CLIP's robustness [A]. Given that the proposed method is bases on CLIP bias, it would be interesting to know whether the observations persist when the pre-training dataset distribution changes.\n\n5. The compared baselines are out-dated. Please include methods such as MaPLe[B] and PromptSRC[C] and even more recent methods.\n\n6. It is unsure whether the comparison to other methods is fair. In UC setup, full unlabelled training set is provided, while CoOp and CLIP-Adapter are only given roughly two thousands images. \n\n     a) Though labelling images can be expensive, collecting this amount of unlabelled data for clustering is even more expensive. Note that, these data all belong to 345 classes defined in DomainNet with six different styles. \n\n     b) In Table 2, it is also unfair to show the superiority of UMFC by comparing it with CoOp trained on one single domain, given that UMFC is able to see images with different styles. \n\n     c) It can be beneficial to compare the clustering time of UMFC with CoOp training time.\n\n7. In Line 228, it is union or intersection of class sets in ImageNet-A and ImageNet-R?\n\n[A] Data Determines Distributional Robustness in Contrastive Language-Image Pre-training (CLIP)\n\n[B] Maple: Multi-modal prompt learning.\n\n[C] Self-regulating Prompts: Foundational Model Adaptation without Forgetting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: CLIP's ability to encode domain information is limited to specific domains.**\n\nThis is a valuable question. We would like to clarify that the feature bias of CLIP is **not** limited to some specific domains, but is a general issue. \n- **Explanation of Figure 1(b):** We use t-SNE for visualization, which maps high-dimensional data into lower-dimensional space while preserving relative distances. Large distances between I/P/Q  and other domains cause the relative distance among C, R, and S domains to shrink. To mitigate this effect, we visualize only C, R and S domains, As shown in Figure 1(a) of global response PDF, image features from R, C, and S domains are clustered rather than dispersed. \n- **Quantitative results of model bias:** To further illustrate model bias, we calculate the Euclidean distances of image features across 6 domains in DomainNet. \nSee the detailed results and analysis in global response R2.\n- **Model bias in other datasets:** We utilize the image features from ImageNet-Variants, then visualize clustered features with t-SNE (Figure 1(b) of global response PDF) and report the Euclidean distances. These results validate the model bias in CLIP is a general issue. \n\n**W2: For figure 1(c), it is suggested to show the percentage of predictions rather than the absolute numbers.  It would be interesting to evaluate such bias in other domains.**\n\n(1) We have redrawn the figure,  shown in Figure 2 of global response PDF.\n(2) The Table 1 of global response PDF lists the top-5 predicted classes for each domain, shown as \"class name / percentage\". Note that a uniform prediction probability in DomainNet is 1/345 = 0.29%. The table reveals that CLIP favors different classes across domains, which is consistent with conclusion of our paper.\n\n**W3: Does it mean that the proposed method need prior knowledge on the number of domains in the target dataset?**\n\nOur method does not need prior knowledge on the number of domains in the target dataset. See the ablation studies in global response R3.\n\n**W4: Whether the observations persist when the pre-training dataset distribution changes.**\n\nTo answer this question, we use pre-trained models from OpenCLIP[1], which explores scaling laws with LAION dataset. We visualize the image features on DomainNet with different pre-training data distribution. See the details and analysis in global response R2.\n\nMoreother, in Table 2 of global response PDF, we futher verify the effectiveness of our method on OpenCLIP series models. The results demonstrate the universal applicability of our method to various VL models with different architectures and pre-training data.\n\n> [1] Reproducible scaling laws for contrastive language-image learning. CVPR23\n\n**W5: Compared with more recent methods, such as MaPLe[B] and PromptSRC[C].**\n\nWe compare UMFC with MaPLe and PromptSRC under different scenarios. We will add these results in final version.\n- **Domain-balanced Labeled data:** We first consider the domain-balanced data for FSL methods. Specifically, we train MaPLe and PromptSRC with k (k=1 or 4) labeled data per class for each domain on DomainNet.   The Table 3(a) of global response PDF shows that UMFC achieves comparable performance to MaPLe with limited labeled data, but underperforms them when more labeled data is available. \n\n- **Domain-imbalanced Labeled data:** We next consider domain-imbalanced labeled data, as balanced annotations in multi-domain scenarios are expensive. Specifically, we train FSL methods with 1 labeled data per class from a single domain and test them on all domains.\nThe Table 3(b) of global response PDF shows that the choice of training domain significantly impacts the performance of two FSL methods, suggesting that supervised FSL methods may overfit to the training domain.  \n\nIn summary, FSL methods require domain-balanced labeled data for fine-tuning.\nWhen such high-quality labeled data is expensive to acquire in multi-domain scenarios, UMFC offers a cost-effective alternative, as collecting unlabeled data is more economical.\n\n**W6.1: Collecting unlabelled data for clustering is even more expensive than labeled data.**\n\nWe need to highlight that collecting unlabeled data is more cost-efficient than labeled data, especially in multi-domain scenarios.\n- **No Need for Expert Annotation:** Unlabeled data collection avoids the need for expert annotation across domains and categories, unlike labeled data, which requires expert involvement.\n- **Challenges with Labeled Data Quality:** Although FSL methods like CoOp and CLIP-Adapter require only few labeled data, they demand high-quality data. Issues such as category imbalance, domain imbalance (shown in Table 2), and data noise can easily cause models overfitting to fine-tuned data, making labeled data collection costly in multi-domain scenarios.\n\n**W6.2: In Table 2, it is unfair to show the superiority of UMFC by comparing it with CoOp trained on one single domain.**\n\nTable1 compares UMFC with FSL methods with multi-domain training data, and Table2 use single-domain data. These experiments show UMFC as a viable alternative when collecting extensive multi-domain (balanced) labeled data for FSL methods is impractical. UMFC also generalizes well to unseen domains rather than overfitting to training domains. \nSee our response to Weakness 2 for Reviewer Xgmy for detailed results.\n\n**W6.3: It can be beneficial to compare the clustering time of UMFC with CoOp training time.**\n\nWe report the computational cost of UMFC and CoOp on DomainNet. UMFC's clustering time is negligible (2.3 seconds). The entire training process is significantly faster than CoOp.  UMFC does not require gradient back-propagation, leading to lower memory usage. See more details in our response to Weakness 3 for Reviewer jj75.\n   \n**W7: In Line 228, it is union or intersection of class sets in ImageNet-A and ImageNet-R?**\n\nThe class space of ImageNet-Variants is **union** of class sets in ImageNet-A and ImageNet-R.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQgwIzUelS", "reviewer": "Reviewer_jj75", "summary": "This paper proposes a training-free method \"Unsupervised Multi-domain Feature Calibration ( UMFC)\" to mitigate the model bias problem presented in CLIP models. The paper first observes the bias problem from the visual encoder and the text encoder perspective. Then it mitigates the problem by subtracting the bias term in the visual and text features obtained from the domain data. The experimental results show improvements over existing approaches like CoOp or CLIP-Adapter.", "questions": "Please see the weakness part.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The observation is clear. The paper provides a clear observation regarding the inherent biases in CLIP's visual and text encoders, which lead to biased prediction. \n2. The approach is simple and straightforward. The proposed UMFC effectively leverages unlabeled data to mitigate domain biases without involving fine-tuning or extra optimization. \n3. The presentation of the method is clear and its evaluation is clear and easy to follow.", "weaknesses": "1. I think the improvements seem minor. In table 1, compared to the really easy approach CLIP-D, UMFC fails to improve it and UMFC + CLIP-E outperforms it by only 0.65% points. What about CLIP-D + CLIP-E? I think you should use this one to agast UMFC + CLIP-E. \n2. In table 2, why did CoOp perform worse than the original CLIP? I believe it's the receipt problem, and I think you should at least perform some hyper-parameter choice to get a reasonable baseline, since finetuning on those in-domain data can easily get performance improvements. \n3. What's the extra computation cost involved in this approach and how it compares to other methods? Since the proposed approach uses test-time adaptation, I believe the authors should tell us the computation cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: I think the improvements seem minor. In table 1, compared to the really easy approach CLIP-D, UMFC fails to improve it and UMFC + CLIP-E outperforms it by only 0.65% points. What about CLIP-D + CLIP-E? I think you should use this one to agast UMFC + CLIP-E.** \n\nThere are some misunderstandings here, and I would like to clarify them in the following.\n\n- **Minor improvement compared to CLIP-D.** In CLIP-D, we design customized prompts that incorporates corresponding domain label and name for each test sample. CLIP-D shows better performance than vallina CLIP, demonstrating the benefit of integrating domain information in multi-domain scenarios. Notably, our comparison between UMFC and CLIP-D is not meant to imply that UMFC outperforms CLIP-D, as CLIP-D utilizes domain labels and names for all test samples, whereas UMFC operates without such explicit supervision. Rather, the purpose of this comparison is to demonstrate that UMFC can extract domain information from data **without supervision**, highlighting it greater versatility in various situations.\n\n- **Disadvantages of CLIP-D.** The CLIP-D model, while simple and straightforward, has several disadvantages: 1. it requires domain labels and names for all test samples, which is impractical in many real-world scenarios. 2. it is sensitive to the choice of domain names; Our findings show that CLIP-D's performance degrades when synonyms are used instead of the original domain names. Detailed results can be seen the below table.\n\n  |                   | C     | I     | P     | Q     | R     | S     | Avg   |\n  | ----------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n  | CLIP-D            | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | 61.03 |\n  | CLIP-D (Synonyms) | 72.11 | 54.83 | 66.09 | 17.73 | 83.34 | 65.74 | 59.97 |\n\n- **More comparison baseline.** We build  CLIP-DE by replacing the single domain-specific prompt in CLIP-D with an ensemble of prompt template incorporating domain names. The results are shown in below table. As shown, CLIP-DE performs only marginally better than CLIP-E, and slightly worse than CLIP-D on DomainNet. This suggests that CLIP-D may require a more refined prompt design and ensemble strategy to improve performance in multi-domain scenarios.\n\n  |         | C     | I     | P     | Q     | R     | S     | Avg   |\n  | ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n  | CLIP-E  | 73.16 | 54.17 | 67.02 | 15.86 | 84.30 | 67.49 | 60.33 |\n  | CLIP-D  | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | 61.03 |\n  | CLIP-DE | 73.62 | 54.34 | 67.64 | 16.34 | 84.49 | 67.45 | 60.65 |\n\n\n\n**W2: In table 2, why did CoOp perform worse than the original CLIP? I believe it's the receipt problem, and I think you should at least perform some hyper-parameter choice to get a reasonable baseline, since finetuning on those in-domain data can easily get performance improvements.**\n\nThere may be a misunderstanding, and I would like to clarify. In Table 2, we train CoOp using **single-domain** few-shot labeled data. For example, CoOp(Q) represents CoOp model trained using data from Quickdraw domain. As shown in Table 2, CoOp(Q) performs better than vallina CLIP and our UMFC in the Quickdraw domain, but significantly worse in the other 5 domains. This is because single-domain supervised fine-tuning tends to overfit to the specific training domain and cannot generalize well to other new domains.\n\n**W3: What's the extra computation cost involved in this approach and how it compares to other methods? Since the proposed approach uses test-time adaptation, I believe the authors should tell us the computation cost.**\n\nHere we report the computational cost of UMFC and other comparison methods under different scenarios. For analyzing computational cost, we report the training time, inference time and memory.\n\n- **Unsupervised Calibration:** In this scenario, the entire unlabeled training set is provided for training. Corresponding computation cost comparisons are shown in below Table. **Firstly**, UMFC incurs minimal training and inference overhead compared to CLIP.   This is because UMFC only requires a single forward pass to extract features and then calculate statistics for feature calibration. **Secondly**, when compared to few-shot fine-tuning methods like CoOp, UMFC also demonstrates lower consumption of computational resources and time.\n\n\n  | Method           | Training Time            | Inference Time | Epoch | Memory  |\n  | ---------------- | ------------------------ | -------------- | ----- | ------- |\n  | CLIP   | -                        | 86 seconds     | -     | 1797MiB  |\n  | MUST   | 10 hours (2 GPUs)        | 92 seconds     | 30    | 25944MiB |\n  | UMFC             | 2.3 seconds + 55 seconds | 86 seconds     | -     | 1887MiB |\n  | CoOp (6*1 shot)  | 32 minutes               | 83 seconds     | 50    | 7007MiB |\n  | CoOp (6*4 shots) | 160 minutes              | 83 seconds     | 100   | 7007MiB |\n\n- **Test-time Adaptation:** In this scenario, no training data is provided and the test data arrives in batches. Corresponding computation cost comparisons are shown in below Table. As seen, UMFC requires less memory than TPT and shows greater computational efficiency. Specifically, UMFC takes only 296 seconds, whereas TPT requires nearly 197 minutes.  This is because TPT requires fine-tuning the text prompt for each test sample and augmenting each test sample 64 times to ensure the reliability of the fine-tuning results, which significantly slows down TPT's inference speed.\n\n  | TTA  | Inference Time | Memory  |\n  | ---- | -------------- | ------- |\n  | UMFC | 296 seconds    | 1790MiB |\n  | TPT  | 197 minutes    | 6872MiB |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WeNHgSrwXJ", "reviewer": "Reviewer_hYxJ", "summary": "To boost the transferability of CLIP across downstream domains, the paper proposed Multi-Domain Feature Calibration (UMFC). By mitigating CLIP biases in both visual and text encoders, UMFC significantly improves classification performance over existing methods across 3 downstream tasks.", "questions": "1. The author set the batch size of TTA to 100. Is the proposed method sensitive to the batch size?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observation of domain-dependent performance variability in CLIP is interesting. The observation highlights that the accuracy of CLIP can vary across different domains for the same classes. \n2. The paper is well-motivated. The author starts with the bias phenomenon empirically and further presents the motivation from a probabilistic view.\n3. The experiment is comprehensive. The efficacy of UMFC is verified on three downstream tasks and extensive experiments demonstrate the consistent gains.", "weaknesses": "1. The proposed methods rely on multi-domain data for calibration, which slightly weakens its application scenarios.\n2. The comparison in TTA is not enough. As the paper use prototype for calibration, the proposed method be compared with other state-of-the-art prototype-based methods such as T3A [1].\n3. The ablation of the number of clusters \\textit{M} is insufficient. Since we do not know the number of domains, what would the results be if we set \\textit{M} higher than the actual number of domains?\n\n[1] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The proposed methods rely on multi-domain data for calibration, which slightly weakens its application scenarios.**\n\n(1) In fact, our method is not limited to multi-domain scenarios; it is also applicable to single-domain scenarios. In single-domain case, we employ the same calibration process used in UMFC. This involves gathering statistical information from the clustered training set and utilizing this information to calibrate the features. \n\n(2) Following the unsupervised calibration setting in Section 5.2, we conduct experiments on DomainNet, where both the unlabeled training data and test data originate from the same domain. As shown in the Table below, UMFC outperforms vanilla CLIP even in single-domain scenarios, and its  performance is only marginally affected by the number of clusters (M). \n\n|      | C     | I     | P     | Q     | R     | S     | \n| ---- | ----- | ----- | ----- | ----- | ----- | ----- |\n| CLIP | 71.21 | 49.47 | 64.61 | 14.23 | 82.98 | 64.81 |\n| UMFC (M=3)    | 73.90   | 56.69  | 68.12   | 20.43   | 84.70   | 68.04   |\n| UMFC (M=1) | 73.85 | 56.7 | 68.02 | 20.31 | 84.67 | 68.17 |\n\n\n**W2: The comparison in TTA is not enough. As the paper use prototype for calibration, the proposed method be compared with other state-of-the-art prototype-based methods such as T3A [1].**\n\nThanks for pointing out this. In the following, we compare T3A and our UMFC under test-time adaptation setting. To ensure a fair comparison, we integrated T3A into CLIP framework, using CLIP model instead of a pre-trained source domain model.  Additionally, we set the hyperparameter *N* in T3A to 1 (the optimal value in our experiment), which determines the number of supports to restore. We set the batch size to 100 the same as UMFC. The results are shown in the below table.\n\nAs shown in the below table, T3A's performance improved only in the Infograph (I) domain but declined in other domains, particularly in the Quickdraw (Q) domain. We attribute this to  domain bias inherent in CLIP, which is exacerbated by T3A. In contrast, our UMFC leverages the characteristics of CLIP across different domains for calibration, achieving better performance across all domains.\n \n\n|      | C     | I     | P     | Q     | R     | S     | Avg   |\n| ---- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| CLIP | 71.21 | 49.47 | 64.61 | 14.23 | 82.98 | 64.81 | 57.88 |\n| T3A  | 65.56 | 51.55 | 62.02 | 1.80  | 80.49 | 59.53 | 53.49 |\n| UMFC | **72.82** | **55.12** | **66.82** | **19.92** | **83.62** | **66.82** | **60.85**|\n\n\n**W3: The ablation of the number of clusters M is insufficient. Since we do not know the number of domains, what would the results be if we set M higher than the actual number of domains?**\n\nThanks for pointing out this. We conduct ablation studies on cluster number *M* on both DomainNet and ImageNet-Variants (see the table in **Global Response 2**). The results indicate that our method is not sensitive to the value of M, as it still delivers significant improvements even when *M* does not match the actual number of domains (8/10 for DomainNet and 6 for ImageNet-Variants).\n   \n\n**Q1: The author set the batch size of TTA to 100. Is the proposed method sensitive to the batch size?**\n\nIn the table below, we present the results of UMFC with different batch sizes in test-time adaptation scenario. The results validate that our proposed method is not sensitive to the batch size.\n\n\n|               | C     | I     | P     | Q     | R     | S     | Avg   |\n| ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| CLIP          | 71.21 | 49.47 | 64.61 | 14.23 | 82.98 | 64.81 | 57.88 |\n| UMFC (bs=10)  | 72.64 | 54.52 | 66.51 | 18.53 | 83.35 | 67.06 | 60.44 |\n| UMFC (bs=16)  | 72.70 | 54.80 | 66.91 | 19.11 | 83.78 | 66.69 | 60.66 |\n| UMFC (bs=32)  | 73.02 | 55.02 | 66.73 | 19.17 | 83.66 | 66.97 | 60.76 |\n| UMFC (bs=64)  | 73.23 | 55.04 | 66.72 | 19.15 | 83.78 | 66.84 | 60.79 |\n| UMFC (bs=100) | 72.82 | 55.12 | 66.82 | 19.92 | 83.62 | 66.82 | **60.85** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **R1. Disadvantages of CLIP-D.**
CLIP-D serves as a comparison baseline by incorporating the domain names of test samples into its prompts. While CLIP-D demonstrates better performance compared to the CLIP model in Table 1, it has several notable drawbacks:

- **Dependency on Domain Labels and Names**: CLIP-D requires domain labels and names for all test samples, which is impractical in many real-world scenarios. In contrast, our UMFC method estimates the image feature bias from clustered features, without the need for domain labels and names of the test samples.

- **Sensitivity to Domain Name Selection**: CLIP-D's performance is sensitive to the choice of domain names. As shown in the table below, CLIP-D's effectiveness degrades when synonyms are substituted for domain names. 

   |                   | C     | I     | P     | Q     | R     | S     | Avg   |
     | ----------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
     | CLIP-D            | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | 61.03 |
     | CLIP-D (Synonyms) | 72.11 | 54.83 | 66.09 | 17.73 | 83.34 | 65.74 | 59.97 |

In summary, CLIP-D relies on extensive prior knowledge to perform better in the multi-domain scenarios. 
In contrast, UMFC's calibration process does not require domain labels and names of test samples, enabling it to handle a broader range of real-world scenarios.

### **R2. Generality of our observation.**
Here we would like to emphasize that our observation (in Figure 1) is a general issue for CLIP model.

- **Quantitative results of model bias:** To further quantify model bias of CLIP, we calculate the Euclidean distances of image features across different domains in DomainNet and ImageNet-Variants. 
The tables below show that these distances (except the diagonal element) range from 0.2 to 0.6, indicating that CLIP's visual encoder incorporates domain information into image features across different domains and datasets.

  | DomainNet | C    | I    | P    | Q    | R    | S    |
  | -------- | ---- | ---- | ---- | ---- | ---- | ---- |
  | C        | 0.00 | 0.35 | 0.30 | 0.41 | 0.26 | 0.19 |
  | I        | 0.35 | 0.00 | 0.37 | 0.59 | 0.30 | 0.36 |
  | P        | 0.30 | 0.37 | 0.00 | 0.52 | 0.26 | 0.26 |
  | Q        | 0.41 | 0.59 | 0.52 | 0.00 | 0.52 | 0.38 |
  | R        | 0.26 | 0.30 | 0.26 | 0.52 | 0.00 | 0.28 |
  | S        | 0.19 | 0.36 | 0.26 | 0.38 | 0.28 | 0.00 |

   | ImageNet-Variants | IN-A | IN-R | IN-S |
   | ------------ | ---- | ---- | ---- |
   | IN-A         | 0.00 | 0.28 | 0.36 |
   | IN-R         | 0.28 | 0.00 | 0.19 |
   | IN-S         | 0.36 | 0.19 | 0.00 |

- **Different pre-training distribution:** It is an interesting question that whether our observation presist if pre-training data distribution of CLIP changes. To answer this question, we use pre-trained models from OpenCLIP[1], which explores scaling laws with public LAION dataset. We visualize image features from visual encoders on DomainNet using t-SNE, shown in Figure 3 of global response PDF. Figure 3(a) reveals that images from different domains still cluster together despite changes in pre-training data (LAION-80M, LAION-400M and LAION-2B).

### **R3. The ablation of the number of clusters M.**
We conducte additional ablation experiments on the number of clusters M and demonstrate that our method is not sensitive to the choice of this hyperparameter. 
We test different values for the hyperparameter M on DomainNet and ImageNet-Variants. As shown in the table below, our UMFC consistently outperforms vanilla CLIP, even when the number of clusters M does not match the number of domains (6 for DomainNet and 3 for ImageNet-Variants). For instance, setting M from 3 to 10 all leads to improvements on DomainNet.

| DomainNet   | C     | I     | P     | Q     | R     | S     | Avg   |
| ----------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| CLIP        | 71.21 | 49.47 | 64.61 | 14.23 | 82.98 | 64.81 | 57.88 |
| UMFC (M=3)  | 72.53 | 54.60 | 66.31 | 20.32 | 83.35 | 66.86 | 60.66 |
| UMFC (M=4)  | 73.55 | 56.36 | 67.19 | 20.62 | 84.13 | 67.69 | 61.59 |
| UMFC (M=6)  | 73.01 | 55.44 | 66.89 | 20.14 | 83.66 | 67.51 | 61.11 |
| UMFC (M=8)  | 73.50 | 56.58 | 67.53 | 20.64 | 84.06 | 67.92 | 61.71 |
| UMFC (M=10) | 73.63 | 56.87 | 67.81 | 20.23 | 84.20 | 67.87 | 61.77 |

| IN-Variants | IN-A  | IN-R  | IN-S  | Avg   |
| ----------- | ----- | ----- | ----- | ----- |
| CLIP        | 42.13 | 66.95 | 74.58 | 61.22 |
| UMFC (M=2)  | 45.35 | 71.71 | 77.37 | 64.81 |
| UMFC (M=3)  | 44.77 | 72.19 | 78.62 | 65.19 |
| UMFC (M=6)  | 45.29 | 71.33 | 77.59 | 64.74 |

Meta-review: This paper studies how to calibrate the pretrained VLM features for downstream task domains. A key observation is that the pretrained VLM features tend to form clusters according to task domains instead of semantic concepts, which can harm the downstream task performance in differentiating semantic categories. The authors propose a simple mean-shift technique to calibrate the vision features by using unlabeled images from the downstream domain, and adjust the text features in the same direction without using the knowledge of the downstream task vocabulary. This technique relies solely on images and obviates the need for downstream domain name or knowledge of the samples.

This paper receives mixed reviews with scores 6,6,6,4,3. Many reviewers appreciate the clarify and simplicity of the method (HbLd, jj75), the motivation by observing the domain-sensitivity and bias in CLIP (HbLd, jj75, hYxJ), the ability of the method to leverage unlabeled data (Xgmy), and the comprehensive experiments (HbLd, Xgmy, hYxJ). However, the reviewers also noted concerns about the paper including whether there's a real improvement over CLIP-D/E variants (HbLd, jj75), complexity and overfitting (Xgmy), fair comparison with TTA (hYxJ), validity of domain clusters in T-SNE (cRCJ), comparison with MaPLe and PromptSRC (cRCJ). The author's rebuttal was helpful to address the concerns of reviewers HbLd, Xgmy and hYxJ, but not cRCJ. To understand cRCJ's main concern about t-SNE clusters, I have examined the t-SNE visualization and quantitative measures of intra- and inter-cluster distances in the author feedback, and believed that the authors have addressed this adequately. Even though we did not hear back from reviewer jj75, the main question about CLIP-D is similar to HbLd which I think is well-addressed in the author feedback too. Compared to adaptation or FT, the proposed method is simpler because it is training-free, label-free, and can work in the black-box settings without model weight access.

Taken altogether, I'd recommend acceptance of the paper since the reasons to accept outweigh the reasons to reject in my opinion, but I think it'd be best for us to discuss this one since 2 out of 5 reviewers are leaning reject.
Decision: Accept (poster)