UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

Authors: jiachen  liang
ruibing  hou
minyang  hu
hong  chang
shiguang  shan
xilin  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:33:48

Abstract: Pre-trained vision-language models (e.g., CLIP) have shown powerful zero-shot transfer capabilities. But they still struggle with domain shifts and typically require labeled data to adapt to downstream tasks, which could be costly. In this work, we aim to  leverage unlabeled data that naturally spans multiple domains to enhance the transferability of vision-language models.  Under this unsupervised multi-domain setting, we have identified inherent model bias within CLIP, notably  in its visual and text encoders. Specifically, we observe that CLIP’s visual encoder tends to prioritize  encoding domain over discriminative category information, meanwhile its text encoder exhibits a preference for domain-relevant classes. To mitigate this model bias, we propose a training-free and label-free feature calibration method, Unsupervised Multi-domain Feature Calibration (UMFC). UMFC estimates image-level biases from domain-specific features and text-level biases from the direction of domain transition. These biases are subsequently   subtracted from original image and text features separately, to render them domain-invariant. We evaluate our method on multiple settings including transductive learning and test-time adaptation. Extensive experiments show that our method outperforms CLIP and performs on par with the state-of-the-arts that need additional annotations or optimization.
Our code is available at https://github.com/GIT-LJc/UMFC.

Main paper: 
 introduction: 
 Recently, Vision-Language Foundation Models (VLFMs) such as CLIP [30] , BLIP [24] , Flamingo [1] and ALIGN [20] have demonstrated remarkable performance across various downstream tasks. These VLFMs formulate the training objective as contrastive learning, leveraging millions of imagetext pairs to establish a shared embedding space. Equipped with a wide range of visual and text representations, VLFMs exhibit the capability to tackle downstream tasks in a zero-shot manner.Despite VLFMs being exposed to abundant examples, they may still encounter examples with new variations in downstream tasks. To address the problem of distribution shift between the pre-training and downstream domains, a natural approach involves fine-tuning VLFMs on various target tasks, such as prompt engineering [39, 38] and adapter learning [11, 37] . However, these methods generally require labeled samples for fine-tuning, which is prohibitively expensive to be satisfied in reality. Conversely, abundant unlabeled data are often available for downstream tasks. Notably, in practical scenarios, the unlabeled data typically contains multiple domains, which exacerbates the adaptation difficulty of VLFMs. Therefore, in this paper, we aim to improve the adaptation performance of VLFMs directly on unlabeled data that naturally spans multiple domains. In this unsupervised multi-domain setting, we observe that CLIP cannot perform well when unlabeled data are drawn from mixed distributions. As shown in Figure 1 (a), even within the same class space, the accuracy of CLIP varies significantly across different domains. While CLIP performs exceptionally well for images from common distributions encountered during pre-training, e.g., achieving 83.0% accuracy on real domain, it struggles with rare distributions encountered during pretraining, e.g., only achieving 14.2% accuracy on quickdraw domain. Above observations highlight that CLIP exhibits model biases that lead to incorrect predictions in specific scenarios. This raises a fundamental question: where do these biases originate?We point out that these model biases stem from deficiencies in the visual encoder and textual encoder.On the side of visual encoder, we observe that CLIP' visual encoder prioritizes encoding domain information over discriminative category information. As shown in Figure 1 (b), features from the same domain clearly cluster together, whereas a notable gap separates features from different domains. This phenomenon indicates that CLIP's visual encoder exhibits a higher sensitivity to domain information over category information. When the domain of downstream tasks shifts from pre-trained tasks, the mismatch in domain-specific information encoded in image features could adversely affect classification accuracy. On the side of textual encoder, we observe that CLIP demonstrates varying category preferences across different domains. Specifically, CLIP tends to classify images into categories whose name are closely related to corresponding domain. As shown in Figure 1 (c), within "quickdraw" domain, a large portion of samples (∼ 30%) are classified as "squiggle" or "line" categories. Conversely, within "painting" domain, CLIP favors categories like "paintcan" and "paintbrush". This observation shows that the class embeddings encoded by CLIP's textual encoder inherently carry domain-specific information, misleading the model to prioritize categories highly associated with respective domains. Due to the combined effects of visual and textual encoder biases, CLIP's performances vary significantly across different domains.In this paper, we aim to calibrate CLIP to mitigate the model biases. Initially, we analyze the model biases from a probabilistic standpoint. The factors influencing p(y|x), affected by domain variable z, can be decoupled into two parts: the sample distribution conditioned on classes p(x|y, z) and the class distribution p(y|z). If the two probability distributions are independent of z, domain shifts will not affect the predictions. To this end, we propose Unsupervised Multi-domain Feature Calibration (UMFC), a simple yet efficient framework for calibrating CLIP to generalize to various downstream tasks using multi-domain unlabeled data. UMFC jointly calibrates CLIP through two training-free modules: Image Feature Calibration module (IFC) and Text Feature Calibration module (TFC). Firstly, IFC focuses on calibrating CLIP's image encoder to prioritize category-level over domain-level information, thereby reducing prediction error caused by domain shifts. Specifically, we calculate the average image features for each domain i, denoted as µ i , and posit that the prediction of µ i reflects the inherent bias of CLIP's image encoder within that domain. By subtracting this domain-specific bias from original predictions, we can derive domain-agnostic predictions. Secondly, TFC focuses on calibrating CLIP's text encoder to remove its preference towards domain-related class names. As observed by [7, 28, 8] , a "global direction" exists in CLIP, representing the shift from training distribution to unseen distribution, shared across image and text embedding spaces. Motivated by this observation, we utilize the shift direction between different-domain images to estimate the text shift direction. Then, we subtract this shift vector to counteract the text encoder's bias towards domain-related categories. By combing IFC and TFC, we can calibrate the features on both CLIP's image and text encoders, thereby alleviating the model bias in downstream tasks.We validate the efficacy of UMFC on three downstream tasks: unsupervised calibration, transductive learning, and test-time adaptation, demonstrating consistent gains over CLIP. UMFC presents a lowcost solution for classification, unlocking the potential of CLIP-like models for practical scenarios characterized by abundant images across multiple domains but scarce labels. 
related work: 
 Few-Shot Learning. Few-shot learning (FSL) [17, 16, 18, 10, 34] aims to learn a good model on a novel task with few labeled samples. Traditional FSL methods [17, 16, 18, 10, 34] often rely on abundant related training tasks for pre-training and design specialized algorithms to transfer cross-task knowledge, facilitating task adaptation. Additionally, some semi-supervised learning (SSL) methods [26, 14, 15] tackle the scarcity of labeled samples by assuming access to extensive unlabeled data.While both FSL and SSL methods have achieved encouraging results, their generalization abilities are limited. Recently, the CLIP model is proposed, which is a vision-language foundation model that learns a shared vision-language embedding space. As pre-trained on large-scale data, CLIP exhibits impressive zero-shot ability across various downstream tasks. Based on CLIP model, many works focus on improving its performance further on downstream tasks with few labeled data. For example, CoOp [39] , CoCoOp [38] , MaPLe [21] and PromptSRC [22] leverage few labeled samples to learn the prompt in the continual input embedding space, offering a parameter-efficient way of fine-tuning foundation models. Similarly, CLIP-Adapter [11] and Tip-Adapter-F [37] introduce a lightweight adapter module to produce adaptive multi-modal features. However, these methods require extra labeled data and the process of tuning pre-trained parameters for CLIP, which is cost-expensive. Differently, in this work, we focus on utilizing unlabeled data to enhance CLIP's performance in a training-free manner.Domain Adaptation / Domain Generalization. Recently, several methods [23, 3, 8, 31, 2, 33, 5, 19, 7] exploit a large-scale pre-trained model (e.g., CLIP) to address the domain adaptation and generalization problems. For example, RISE [19] leverages CLIP as a teacher to regularize the student's learned representation through images. The work [3] utilizes domain-invariant and domainspecific prompts for multi-source unsupervised domain adaptation. Other works [8, 7, 5] focus on utilizing the transferability between visual and textual modalities to guide domain information transfer. PromptStyler [5] attempts to simulate various distribution shifts to explore diverse styles in a joint vision-language space. PODA [8] and LADS [7] generate samples in the style of the target domain, and adapt to the target domain based on these samples. However, these language-guided methods require the prior of target-domain names, which may not be satisfied in reality. In contrast, our UMFC method does not require any extra information about the target domain.Test-Time Adaptation. Test-time adaptation aims to adapt a pre-trained model to test tasks, where distribution of test data differs from that of pre-training data. TPT [32] proposes a test-time prompt tuning strategy, which extends traditional TTA methods to vision-language models. Building upon TPT, DiffTPT [9] utilizes pre-trained diffusion models to augment the diversity of test data samples used in prompt tuning. SwapPrompt [27] employs a dual prompts paradigm to enhance the swapped prediction mechanism. However, these prompt learning methods are computationally expensive and time-consuming. Different from above methods, our UMFC only needs to calibrate features in a training-free way, making it more efficient for test-time adaptation.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
clip and model biases: 
 In this section, we first describe the backgrounds of CLIP and then analyze its bias issue in downstream tasks. We attribute the cause of bias to the visual encoder bias and text encoder bias.CLIP [30] consists of two parallel encoders: a visual encoder that maps image inputs into image features, and a text encoder that maps text inputs into text features. The model is trained with a contrastive loss that maximizes similarity between positive image-text pairs while minimizing similarity between negative pairs. This process ensures alignment between the features of images and their corresponding textual descriptions within the feature space. Trained on a vast collection of image-text pairs, CLIP benefits from general visual representations, endowing it with powerful zero-shot capabilities.Formally, we denote a CLIP model as {F, T }, with F and T being the visual and text encoders. Considering a K-class classification problem, we use Y = {y 1 , . . . , y K } to represent the class space, where y k denotes the class name of k th class. In the zero-shot inference phase, CLIP uses hand-crafted prompts p (such as "a photo of a {}") to covert each class name y i to category-specific text description {p; y i }. Then, we feed these class descriptions to the text encoder to get the text features {t 1 , . . . , t K }, where t i = T (p; y i ). Meanwhile, given a test image x, the visual encoder is used to compute its visual feature, denoted as f = F (x). The prediction probability on x can be then computed as:EQUATIONwhere sim denotes the cosine similarity and τ is the temperature of the softmax function.As shown in Figure 1 (a), the accuracy of CLIP varies significantly across different domains within the same class space, e.g., 83.0% in real domain and 14.2% in quickdraw domain. This phenomenon indicates that CLIP favors domains commonly encountered during pre-training (such as natural images). We attribute this model bias to visual encoder bias and text encoder bias.• Visual Encoder Bias. We observe that CLIP' visual encoder prioritizes encoding domain information over discriminative category information, as shown in Figure 1 (b). Due to the abundance of natural images in pre-training, CLIP's visual encoder and text encoder are well-aligned in the natural image domain. Consequently, when the style of input images largely deviates from natural domains (such as quickdraw and infograph style), the visual feature gap across different domain (as depicted in Figure 1 (b)) hinders the text encoder's capacity to process these shifted images effectively, leading a significant drop in classification accuracy.• Text Encoder Bias. We observe that CLIP exhibits a preference for domain-related categories in specific domains, as shown in Figure 1(c ). In domains characterized by distinct styles, certain category names may carry domain-specific information. For instance, in quickdraw domain, where most images consist of lines and squiggle, CLIP demonstrates a severe bias towards the "line" and "squiggle" categories. This leads to a large number of quickdraw samples being incorrectly classified into the two categories. Conversely, in the painting domain, as all images inherently possess painting features, CLIP shows a strong preference for categories related to the concept of painting, such as "paintcan" and "paintbrush".Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: See Section 5.2 and supplemental material for implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
unsupervised multi-domain feature calibration: 
 In this section, we continue to analyze the model biases issue from a probabilistic view. Then we give a detailed introduction of our method aimed at alleviating CLIP's biases.We start from the posterior probability p (y i |x), as introduced in Equation 1. Recall that our goal is to maximize the posterior probability p (y i |x) for each image-label pair (x, y i ) of any domain.A natural question arises: How different domains affect the posterior probability? We answer this question from a probabilistic view. Based on the Bayes' Theorem, we can drive that:EQUATION)where z is a latent variable that denotes the domain labels. We can further decompose the joint probability p (y i , x, z) in Equation 2as p (y i , x, z) = p (x|y i , z) p (y i |z) p (z). Assume that the probability distribution of domains p (z) is uniform, thus maximizing the posterior probability p (y i |x) is equal to maximize the summation z p (x|y i , z) p (y i |z), asEQUATIONEquation 3 shows that the domains affect posterior probability distribution p (y|x) by disturbing two terms: the sample distribution conditioned on classes p (x|y, z) and the class distribution p (y|z).As analyzed in Section 3.2, CLIP exhibits both visual and text encoder bias, impacting its generalization ability to downstream tasks. As shown in Equation 3, these biases stem from the probability distributions p (x|y, z) and p (y|z), which are disturbed by domain information. To mitigate the model biases, an intuitive idea is to make the two probability distributions p (x|y) and p (y) independent of domain variable z by calibrating image and text features. To this end, we propose training-free UMFC approach, consisting of an Image Feature Calibration (IFC) module to alleviate visual encoder bias and Text Feature Calibration (TFC) module to alleviate text encoder bias, facilitating the transfer of CLIP to downstream tasks. Algorithms are provided in the Appendix C.Image Feature Calibration Module. On the side of visual encoder, we focus on making the conditional probability p (x|y) independent of domain variable z, which is equivalent to achieving p (x|y, z) = p (x|y). A straightforward method is to align image feature distributions given a class y across different domains. Unfortunately, as only mixed unlabeled data is provided, we cannot access to class labels and domain labels of images. However, we empirically observe that CLIP's visual encoder prioritizes encoding domain information over discriminative category information. Thus, we can distinguish image features from different domains through a simple clustering algorithm. Formally, we assume that there are M clusters {c 1 , ..., c M } after applying a clustering algorithm. Each cluster is assumed to follow a Gaussian distribution c i ∼ N (µ i , Σ i ), where mean vectorµ i = ( 1 |ci| f ∈ci f) represents the average of image features from cluster c i . Due to the model biases, the pseudo labels produced by zero-shot CLIP are not reliable. Therefore, we directly align the margin image feature distribution of each cluster by subtracting the corresponding mean vector 2 . Specifically, for each visual feature f belonging to the cluster c i , we calibrate it as follows:EQUATION)Text Feature Calibration Module. On the side of text encoder, we focus on making the class probability p (y) independent of domain variable z, which is equivalent to achieving p (y|z) = p (y).Previous works [35] have found that CLIP's performance is sensitive to class name y. For example, replacing category names with synonyms or near-synonymous terms can significantly impact CLIP's prediction results. Due to the influence of domain information in image features, text encoder bias can cause CLIP to categorize domain-specific images into categories whose names are semantically similar to that domain. We further observe that such sensitivity to class names varies across different domains, as shown in Figure 1(c ). This observation inspires us to calibrate text features by removing domain-specific information. However, a challenge arises as domain labels are unavailable. To address this issue, we attempt to extract domain information from unlabeled images, and then transfer this domain information to the text embedding space to estimate the text bias.Observation 4.1 Cross-Modality Transition Direction. The underlying assumption behind using images to simulate the corresponding shifts in texts is that the transition direction from domain i to domain j is consistent across both the image embedding and text embedding spaces [7, 28, 8] , which can be formulated as:EQUATIONwhere x i , y i and x j , y j represent training samples from domain i and domain j respectively, p i and p j denote the domain-specific text prompts for domain i and j respectively. For example, the text prompt of "quickdraw" domain can be "a quickdraw image of a [class]".Inspired by Observation 4.1, we estimate the text-level domain transition direction using differentdomain images. By clustering image features, we calculate the average feature µ i of unlabeled images from each domain i, representing the domain-specific feature for that domain. Also, we calculate the average feature of all unlabeled images µ avg , representing the domain-invariant feature since it encompasses various domain distributions. Following Equation 5, the transition information t i of domain i can be computed as t i = µ i -µ avg . By subtracting this domain transition vector, we suppress the preference for specific class names in the original text features. To ensure that the calibrated text features effectively eliminate biases towards a wide range of domains, we integrate the text features calibrated on each domain. Specifically, for the text feature t j of class j, we calibrate it as follows:EQUATION)where M is the number of clusters.Inference. After calibration with IFC and TFC modules, we obtain the final prediction results. Specifically, this calibration modifies the prediction probability on test image x as follows:EQUATION)5 Experiments Baselines. We compare our method with four groups of methods: (1) CLIP and its variants to show zero-shot predictions: CLIP [30] that uses a fixed prompt "a photo of a [class]"; CLIP-E [30] that uses an ensemble of prompt templates. (2) CLIP-D [30] that utilizes the domain information of test samples and designs domain-specific prompts. (3) Few-shot learning methods: CoOp [39] that performs prompt tuning using labeled data of downstream tasks; CLIP-Adapter [11] that trains an adapter using task-specific labeled data. (4) Unsupervised Fine-tuning method: MUST [25] that fine-tunes the model using unlabeled multi-domain data.Analysis. As shown in Table 1 , UMFC achieves superior performance over CLIP and competitive performance with few-shot methods, CoOp [39] and CLIP-Adapter [11] . We can observe that: (1) CLIP-D incorporates domain information into text prompts, such as "a clipart image of a [class]' for test samples from "clipart" domain. However, creating domain-specific templates is challenging due to unknown and potentially mixed domain sources of test samples. So, we only use CLIP-D as an oracle result. As shown in Table 1 , our method can achieve competitive performance with CLIP-D without prior domain labels for test samples.(2) On domains where CLIP performs poorly, e.g., "quickdraw", our UMFC significantly outperforms others. Compared to CLIP, UMFC achieves about 5% performance gain on quickdraw domain. In addition, combined with more diverse prompts of CLIP-E, UMFC further improves performance.(3) MUST [25] uses abundant unlabeled data from 6 domains for fine-tuning. Our UMFC achieves better performance than MUST even without any additional training. (4) For CoOp [39] and CLIP-Adapter [11] , we fine-tune them using labeled samples from multiple domains. Specifically, we provide k labeled samples per class for each domain, resulting in a total of (6×k ×345) labeled samples available for training. When the number of labeled samples is 6 × 345, our method outperforms few-shot fine-tuning methods. While the performance is higher for few-shot methods with a larger number of labeled samples (i.e., 24 × 345), it is essential to highlight the challenges in obtaining some labeled data for each class and each domain in real-world scenarios. In contrast, our method does not require selecting class-balanced and domain-balanced labeled samples and incurs no additional training overhead.Table 2 provides 8 × 345 labeled samples from a single domain for CoOp fine-tuning and an equal number of unlabeled samples for UMFC calibration. While CoOp trained on a single domain can improve performance within that domain, its performance declines on other domains. This decline becomes particularly notable when a significant distribution gap between the training and test domains exists, e.g., CoOp (Q), leading to a large decrease in average performance across multiple domains.In contrast, our method achieves consistent performance improvements on both training and unseen domains with the same amount of training data. Additionally, unlike CoOp, UMFC does not require labeled data or any parameter fine-tuning. 
main results on transductive learning: 
 In this part, we compare our UMFC with three groups of methods: (1) Zero-Shot CLIP models.(2) Domain Generalization (DG) method: MIRO [2] that trains CLIP on available data to learn great generalization capability. (3) Data-Free DG method: PromptStyler [5] that learns to simulate diverse distributions. The hyper-parameter M (cluster number) is set to 3 for ImageNet-Variants. The compared results are shown in Table 3 and Table 4 . As seen, our UMFC can achieve the best average performance on DomainNet and ImageNet-Variants benchmarks, which validates the effectiveness of our approach in transductive learning. Implementation Details. In TTA setting, where we cannot access all data simultaneously, we adopt an incremental clustering approach. By default, the batch size is set to 100. Initially, we apply K-Means clustering algorithm to the first batch of data. For subsequent batches, we use the prototype classification, with cluster centers serving as prototypes, to assign samples in that batch to different clusters. Then, the cluster centers and calibration statics {µ i } M i=1 are updated accordingly. We have developed two strategies for updating the calibration statics: UMFC-Memory that stores the feature information of each batch to calculate the statistical information; UMFC-EMA that uses Exponential Moving Average (EMA) to update statistical information. Analysis. We mainly compare our UMFC with TPT [32] in test-time adaptation, where TPT finetunes the prompts by minimizing the entropy of the predictions. The comparison results are shown in Table 6 and Table 5 . We can observe that: (1) As shown in Table 6 , UMFC achieves performance gains across all domains, with particularly noticeable gains in those where CLIP performs poorly. For example, UMFC substantially improves upon CLIP on "quickdraw" and "infograph" domains, with an accuracy gain of 5.6%. Table 5 shows that the improvement on ImageNet-A is less significant than on ImageNet-R and ImageNet-S. The reason may be attributed to the absence of distinct domain styles in ImageNet-A, limiting the effectiveness of our calibration method that relies on domain information.(2) UMFC-Memory and UMFC-EMA with different statistical update strategies exhibit similar performance. However, UMFC-EMA, which updates features based on the most recent data, notably reduces storage requirements. Moreover, UMFC exhibits significantly higher computational efficiency without training, whereas TPT requires executing optimization steps on 64 different augmented views of each test image. Thus, our method is more suitable for rapid deployment.The effectiveness of IFC. We firstly evaluate the effectiveness of IFC. As shown in Table 7 , IFC individually contributes to performance gains of CLIP, about 3% average gains. Furthermore, Figure 1 (b) and Figure 2 (a) visualize the image features extracted by CLIP with/without IFC respectively. As shown in Figure 1 (b), the vanilla CLIP maps different-domain images to different clusters in the feature space. Conversely, Figure 2 (a) shows that IFC leads to the merge of image features from different domains, validating its effectiveness of reducing domain-specific information.The effectiveness of TFC. As shown in Table 7 , TFC also individually contributes to performance gains of CLIP. To assess the effectiveness of calibrated text features in eliminating domain bias, we construct a domain classifier. Specifically, we utilize the text features generate by CLIP, with domain prompts "[domain]", as the domain classifer. Then, we use this domain classifier to perform domain classification for the text features before and after calibration (i.e., computing the cosine similarity between domain classifier and text features). As shown in Figure 2 (b), the original text features exhibit a long-tail phenomenon, with a higher probability of being classified into the "real" domain. This suggests that the original text features are biased towards the "real" domain, overlooking their generalization ability to other domains. However, after calibration with TFC, the text features exhibit a near-uniform distribution across domains, indicating that the calibrated text features have largely eliminated domain bias. Consequently, our method enhances performance in other domains without compromising the model's performance in its strongest domain. As shown in Table 7 , the combination of IFC and TFC (i.e., UMFC) can further bring performance gains, which validates the complementary of image-level calibration and text-level calibration for CLIP.The impact of cluster number M . Our method involves clustering the unlabeled data to determine their respective clusters. We evaluate our method with respect to the number of clusters M . As shown in Table 8 , our UMFC consistently outperforms vanilla CLIP, even when the number of clusters M does not match the number of domains (6 for DomainNet). More importantly, our method is not sensitive to changes in M . Refer to the Appendix E for more analysis.The impact of batch size in TTA setting. We present results across various batch sizes during test-time adaptation to confirm that UMFC is robust to batch size variations. When the sample count is initially lower than the number of clusters M , K-Means clustering cannot be directly applied. To address this, we used the first M samples as the initial cluster centers and then proceeded with the same test-time adaptation. As shown in Table 9 , even in the extreme case of a batch size of 1, our method still demonstrates consistent improvement. 
conclusion: 
 In this work, we point out that the model biases hinder the transfer ability of pre-trained visionlanguage models. We develop UMFC, a simple unsupervised calibration method that mitigates model biases in both visual encoder and text encoder through a training-free manner. We demonstrate the effectiveness of our method across multiple settings, including unsupervised calibration, transductive learning, and test-time adaptation. Without the need for any annotations and training, UMFC improves the zero-shot generalization ability of CLIP.Limitations and Broader Impacts. Our method requires unlabeled samples from the target domain to establish domain-level calibration vectors. Although our method is much cheaper and more accessible than fine-tuning or few-shot methods [39, 38, 32] , the requirement of unlabeled data can still be a limitation for some specific scenarios.the text bias. Inspired by Observation 4.1, we can estimate the domain transition direction in text embedding space by using images from different domains, as shown in Figure 4 .  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Not Applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See References.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Algorithm 1 summarizes the proposed UMFC method under Test-Time Adaptation (TTA) setting.Algorithm 2 summarizes the proposed UMFC method under Unsupervised Calibration (UC) / Transductive Learning (TL). {y i } C i=1 . 2: t ← T (p; {y i } C i=1 )▷ Get original text features 3: Initialize centroids of M cluster {c j } M j=1 as empty set 4: for i ← 1 to N do 5:{f b,i } B b=1 ← F ({u b,i } B b=1 ) ▷ Get original image features of i-th batch 6: if {c j } M j=1 is ∅ then 7: {c j } M j=1 ← K-Means({f b,i } B b=1 ) ▷ Using K-Means to acquire M centroids 8: end if 9:for b ← 1 to B do EQUATIONprototypes, where η is the update weight 14:µ avg ← 1 M M m=1 c ′ m ▷ update the average image feature 15: tm ← I(l = m)(c ′ m -µ avg ), ▷ Update calibration statics 16: t ′ b ← 1 M M m=1 t-tm ∥t-tm ∥ , 17: for b ← 1 to B do 18: f ′ b,i ← f b,i -c ′ l ∥f b,i -c ′ l ∥ ,▷ Calibrate f and t with IFC and TFC i } C i=1 . 2: t ← T (p; {y i } C i=1 ) ▷ Get original text features 3: {f b } N b=1 ← F ({u b } N i=1 ) ▷ Get original image features of unlabeled data 4: µ avg ← 1 N N b=1 f b ▷ Get the average of {f b } N b=1 5: {c j } M j=1 ← K-Means({f b } N b=1 )▷ Using K-Means to acquire M clusters and their centroids 6: for b ← 1 to N do tk ← I(l = m)(c m -µ avg ), ▷ Update calibration statics 9: f ′ b ← f b -c l ∥f b -c l ∥ , t ′ b ← 1 M M m=1 t-tm ∥t-tm ∥ , ▷ Calibrate fOur work can be deployed across various scenarios, including Unsupervised Calibration (UC), Test-Time Adaptation (TTA), and Transductive Learning (TL).Unsupervised Calibration. In the UC scenario, we provide an unsupervised training set and use K-Means to assign cluster labels to the training samples, while also saving the corresponding cluster prototypes. Then, we create an unlabeled training set from a mixed domain by sampling 16 instances from each class across 6 domains in DomainNet. UMFC derives image bias and text bias for different clusters based on the cluster labels. During the testing phase, we first predict the cluster labels of the test samples using the cluster prototypes obtained during training, and then calibrate the predictions using the bias information derived from UMFC.Test-Time Adaptation. In the TTA scenario, no training data is provided. Test data from mixed domains arrive in batches, with a batch size set to 100. For the first batch, we perform initial clustering using K-Means. For subsequent batches, we assign cluster labels to the samples based on cluster prototypes and continuously update these prototypes. Once the cluster labels are obtained, UMFC calculates the bias information for the current batch, updates the bias information for each cluster based on the new labels, and then calibrates the data for the current batch.Transductive Learning. In the TL scenario, we provide the entire test set. Similar to UMFC, we gather statistical information and calibrate the predictions for the test data. TL can be viewed as an extreme case of TTA, where the entire test set is treated as a single batch.The impact of cluster number M . We evaluate our method with respect to the number of clusters M and demonstrate that our method is not sensitive to the choice of this hyperparameter. As shown in Table 11 , For instance, setting M from 2 to 6 all leads to improvements on ImageNet-Variants. 13 . UMFC requires less memory than TPT and shows greater computational efficiency. Specifically, UMFC takes only 296 seconds, whereas TPT requires nearly 197 minutes. This is because TPT requires fine-tuning the text prompt for each test sample and augmenting each test sample 64 times to ensure the reliability of the fine-tuning results, which significantly slows down TPT's inference speed. Justification: A summary of the paper's contributions is provided at conclusion.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).This operation is based on the assumption of a uniform class distribution in each cluster. We experimentally found that this operation remains effective, even if this assumption does not hold strictly. 


Decision: This paper studies how to calibrate the pretrained VLM features for downstream task domains. A key observation is that the pretrained VLM features tend to form clusters according to task domains instead of semantic concepts, which can harm the downstream task performance in differentiating semantic categories. The authors propose a simple mean-shift technique to calibrate the vision features by using unlabeled images from the downstream domain, and adjust the text features in the same direction without using the knowledge of the downstream task vocabulary. This technique relies solely on images and obviates the need for downstream domain name or knowledge of the samples.

This paper receives mixed reviews with scores 6,6,6,4,3. Many reviewers appreciate the clarify and simplicity of the method (HbLd, jj75), the motivation by observing the domain-sensitivity and bias in CLIP (HbLd, jj75, hYxJ), the ability of the method to leverage unlabeled data (Xgmy), and the comprehensive experiments (HbLd, Xgmy, hYxJ). However, the reviewers also noted concerns about the paper including whether there's a real improvement over CLIP-D/E variants (HbLd, jj75), complexity and overfitting (Xgmy), fair comparison with TTA (hYxJ), validity of domain clusters in T-SNE (cRCJ), comparison with MaPLe and PromptSRC (cRCJ). The author's rebuttal was helpful to address the concerns of reviewers HbLd, Xgmy and hYxJ, but not cRCJ. To understand cRCJ's main concern about t-SNE clusters, I have examined the t-SNE visualization and quantitative measures of intra- and inter-cluster distances in the author feedback, and believed that the authors have addressed this adequately. Even though we did not hear back from reviewer jj75, the main question about CLIP-D is similar to HbLd which I think is well-addressed in the author feedback too. Compared to adaptation or FT, the proposed method is simpler because it is training-free, label-free, and can work in the black-box settings without model weight access.

Taken altogether, I'd recommend acceptance of the paper since the reasons to accept outweigh the reasons to reject in my opinion, but I think it'd be best for us to discuss this one since 2 out of 5 reviewers are leaning reject.

Reviews: review: paper_summary: paper_summary: This paper attempts to address the problem of domain gap in existing VLMs. Based on an unlabelled mixed dataset, this paper first proposes to fit each different domain and remove the domain bias of image features by Gaussian mixture model. Furthermore, the domain bias of text features is similarly removed by the consistency of image and language modalities. The method is validated on several different tasks.
strength_weakness: strength_weakness: The proposed idea is simple and reasonable. My major concern is whether the proposed methodology is a real improvement over CLIP-D (e.g., 61.68 vs. 61.03 in unsupervised calibration), considering the requirement for extra computational overhead and unlabeled dataset.  

0. how does it compare to CLIP-DE (Domain-Specific Ensemble Prompting)? 
1. the Gaussian assumption is a bit too strong for each domain.
2. I would assume the dataset to be unlabelled, as described in L179-L180. However, for text feature calibration, the description in L208 is _'By clustering image features, we can compute the average feature $µ_i$ of unlabelled images in each domain $i$.
representing the domain-specific features of that domain'_ . Does the domain here refer to the clustering in the image feature calibration step?
3. Equation (3) should be $\triangleq$ .
4. More details are needed for each experimental section.
questions: questions: See weakness.
limitations: limitations: N/A
paper_summary: paper_summary: **I am not an expert in this domain. So my review may not be informative.**

The paper introduces Unsupervised Multi-domain Feature Calibration (UMFC), designed to improve the adaptability of Vision-Language Foundation Models like CLIP to various downstream tasks across multiple domains using unlabeled data. The authors pinpoint model biases within CLIP’s visual and textual encoders that affect its performance across different domains. To mitigate these biases, UMFC employs two calibration modules: the Image Feature Calibration (IFC) and the Text Feature Calibration (TFC). These modules recalibrate the model's focus from domain-specific to category-level information, enhancing its generalization capability. The effectiveness of UMFC is demonstrated through significant performance improvements in unsupervised calibration, transductive learning, and test-time adaptation tasks.
strength_weakness: strength_weakness: 1. While effective, the calibration process might introduce additional complexity in terms of understanding and implementing the recalibration mechanisms, particularly the calculation and subtraction of domain-specific biases.
2. There's a risk that overly aggressive calibration could lead to overfitting on the specific domains included in the training set, potentially reducing the model’s ability to generalize to entirely new or unseen domains.
questions: questions: Please see weaknesses.
limitations: limitations: None.
paper_summary: paper_summary: This paper identifies the inherent model bias within CLIP, notably in visual and textual encoders. To mitigate the bias, authors propose a feature calibration method, termed Unsupervised Multi-domain Feature Calibration. Experiments in the setting of transductive learning and test-time adaptation show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The proposed method is inspired by the observation in the TSNE figure (Figure 1(b)) that "features from the same... different domains". However, this does not appear to be the case. Features from QuickDraw, Painting, and Infograph are clustered, while features from Real, Clipart, and Sketch are dispersed throughout. This suggests that CLIP models can encode some domain information, but this ability is limited to specific domains. If the observation which inspires the method does not hold, the rationale for the proposed method becomes less convincing.

2. For figure 1(c), it is suggested to show the percentage of predictions rather than the absolute numbers. Each domain in DomainNet has different number of samples in validation set. Comparing the number of predictions is less convincing to show the bias of textual encoders. In addition, only Painting and Quickdraw are shown. It would be interesting to evaluate such bias in other domains.

3. It is important to reveal the process of hyper-parameter selection. The cluster number is set to 6 for DomainNet, which is equal to the number of domains in DomainNet. Does it mean that the proposed method need prior knowledge on the number of domains in the target dataset? Furthermore, it is also valuable to reveal the hyper-parameter in ImageNet-variant experiments.

4. It has been show that pre-training dataset distribution determines the CLIP's robustness [A]. Given that the proposed method is bases on CLIP bias, it would be interesting to know whether the observations persist when the pre-training dataset distribution changes.

5. The compared baselines are out-dated. Please include methods such as MaPLe[B] and PromptSRC[C] and even more recent methods.

6. It is unsure whether the comparison to other methods is fair. In UC setup, full unlabelled training set is provided, while CoOp and CLIP-Adapter are only given roughly two thousands images. 

     a) Though labelling images can be expensive, collecting this amount of unlabelled data for clustering is even more expensive. Note that, these data all belong to 345 classes defined in DomainNet with six different styles. 

     b) In Table 2, it is also unfair to show the superiority of UMFC by comparing it with CoOp trained on one single domain, given that UMFC is able to see images with different styles. 

     c) It can be beneficial to compare the clustering time of UMFC with CoOp training time.

7. In Line 228, it is union or intersection of class sets in ImageNet-A and ImageNet-R?

[A] Data Determines Distributional Robustness in Contrastive Language-Image Pre-training (CLIP)

[B] Maple: Multi-modal prompt learning.

[C] Self-regulating Prompts: Foundational Model Adaptation without Forgetting.
questions: questions: See weakness.
limitations: limitations: No clear negative societal impacts.
paper_summary: paper_summary: This paper proposes a training-free method "Unsupervised Multi-domain Feature Calibration ( UMFC)" to mitigate the model bias problem presented in CLIP models. The paper first observes the bias problem from the visual encoder and the text encoder perspective. Then it mitigates the problem by subtracting the bias term in the visual and text features obtained from the domain data. The experimental results show improvements over existing approaches like CoOp or CLIP-Adapter.
strength_weakness: strength_weakness: 1. I think the improvements seem minor. In table 1, compared to the really easy approach CLIP-D, UMFC fails to improve it and UMFC + CLIP-E outperforms it by only 0.65% points. What about CLIP-D + CLIP-E? I think you should use this one to agast UMFC + CLIP-E. 
2. In table 2, why did CoOp perform worse than the original CLIP? I believe it's the receipt problem, and I think you should at least perform some hyper-parameter choice to get a reasonable baseline, since finetuning on those in-domain data can easily get performance improvements. 
3. What's the extra computation cost involved in this approach and how it compares to other methods? Since the proposed approach uses test-time adaptation, I believe the authors should tell us the computation cost.
questions: questions: Please see the weakness part.
limitations: limitations: N/A
paper_summary: paper_summary: To boost the transferability of CLIP across downstream domains, the paper proposed Multi-Domain Feature Calibration (UMFC). By mitigating CLIP biases in both visual and text encoders, UMFC significantly improves classification performance over existing methods across 3 downstream tasks.
strength_weakness: strength_weakness: 1. The proposed methods rely on multi-domain data for calibration, which slightly weakens its application scenarios.
2. The comparison in TTA is not enough. As the paper use prototype for calibration, the proposed method be compared with other state-of-the-art prototype-based methods such as T3A [1].
3. The ablation of the number of clusters \textit{M} is insufficient. Since we do not know the number of domains, what would the results be if we set \textit{M} higher than the actual number of domains?

[1] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, 2021.
questions: questions: 1. The author set the batch size of TTA to 100. Is the proposed method sensitive to the batch size?
limitations: limitations: N/A

Comments: title: Response to the comment 4-6
Response to the comment 1-3
Response to the comment
Look forward to your response
Look forward to your response
Look forward to your response
Response to the comment
Response to the comment
comment: **C4. The response to W6.1 is unconvincing. The authors implemented K-means clustering on DomainNet training set, which consists of high-quality images. It’s unclear if clustering on a random web-crawled dataset would yield similar benefits. For a new task, generating high-quality clusters would require experts to curate images from specific classes with different styles, which could be more costly than labeling a few examples per class.**

- **Requirement of a high quality dataset for UMFC**

  Our UMFC does **not** require high quality dataset with all different styles and categories for clustering. **(1)**
  In Section 5.4 of our paper, we consider the test-time adaptation scenario, where the test data arrives in batches.
  In our experiments, the batch size ranges from 1 to 100 (as detailed in our response to Question 1 for reviewer hYxj), significantly smaller than 2070 (6 domains * 345 classes), meaning each batch only contains  a subset of categories and domains.
  Despite this, our UMFC still performs well, showcasing its generalization and robustness.
  **(2)** Besides, our UMFC can also effectively handle single-domain (see our response to reviewer hYxJ's W1) and partial-domain scenarios (see our response to reviewre Xgmy's W2).

- **Experiments on a random web-crawled dataset**
  
  To best of our knowledge, DomainNet is the largest existing multi-domain dataset collected from the web for image classification. We have demonstrated that our method outperforms CLIP and performs comparably to other methods that require additional annotations or optimization in DomainNet.
  Evaluating the performance of UMFC on a more realistic, randomly web-crawled dataset, which would involve significant time and effort, lies beyond the scope of this paper.

- **Data Collection Cost**

  **(1)** Collecting unlabeled data is much easier than collecting labeled data, which is the motivation behind tasks like unsupervised fine-tuning [2] and Multi-target Domain Adaptation [3]. 
  In real-world scenarios like autonomous driving, data sources are complex and vast, making labeling particularly challenging. However, unlabeled data can be efficiently and automatically collected. In contrast, gathering labeled data requires first collecting data from various domains and categories, and then labeling it, making it much more time-consuming than collecting unlabeled data. **(2)** Furthermore, the performance of few-shot fine-tuning methods is more sensitive to the quality of the labeled data, and demond more training time and memory than UMFC (see our response to jj75's W3).

> [2] MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge. ICCV 2023 \
> [3] Multi-Target Domain Adaptation with Collaborative Consistency Learning. CVPR21 

**C5.For computational cost, it seems that the feature extraction time is not considered. May I also know if it is possible to reveal which package you use for clustering?**

We first use CLIP's image encoder to extract all image features (i.e., data), a process that takes only **35.67 seconds**. We utilize the K-Means algorithm from sklearn. The clustering code is provided below, where the shape of data is [32768,512]. If you're interested, you can try running it on a 4090 Ti.
```python
def kmeans(data, n_clusters=3):
    from sklearn.cluster import KMeans
    model = KMeans(n_clusters=n_clusters, max_iter=500)  
    y_pred = model.fit_predict(data)  
    return y_pred
```

**C6.For W6.2, UMFC clusters features from six domains, while CoOp is trained on only one domain. This comparison is not fair.**

We conduct experiments on DomainNet, where the unlabeled training data for UMFC originates from a single domain and the test data contains all six domains.
**For a fair comparison**, we provide UMFC with 8*345 unlabeled samples (each class has 8 samples) from a **single domain** as the unlabeled training data.
Note that, CoOp is trained using the same number of training data, but with labels for supervised fine-tuning.
For clarity, we use CoOp (C/Q/I) and UMFC (C/Q/I) to denote training samples for CoOp and UMFC from the Clipart/Quickdraw/Infograph domains, respectively.

As shown in the table below, with the same amount of training data, our UMFC achieves consistent performance improvements on both training and unseen domains, while the CoOp method suffers significant performance drops on unseen domains. Additionally, unlike CoOp, our UMFC does not require labeled data or any parameter fine-tuning.

||C|I|P|Q|R|S|Avg|
|--------|-----|-----|-----|-----|-----|-----|-----|
|CLIP|71.21|49.47|64.61|14.23|82.98|64.81|57.88|
|CoOp (C)|74.55|42.66|55.94|13.82|75.00|58.73|53.45|
|UMFC (C)|73.27|52.96|65.27|16.94|83.60|67.04|**59.85**|
|CoOp (Q)|43.97|25.50|32.63|29.07|48.44|38.74|36.39|
|UMFC (Q)|72.17|49.65|63.85|17.47|82.84|66.36|**58.72**|
|CoOp (I)|60.19|54.28|50.81|11.19|70.73|54.27|50.24|
|UMFC (I)|72.54|55.21|64.48|16.30|83.31|66.51|**59.73**|
**C1.Even only plotting, Domain-C, R, S, it can still be observed that Clipart and Sketch are not well separated. They are perceptually different domains but CLIP cannot encode them well, which means the observations do not hold.**

For Figure 1(a) of global response PDF, the reviewer thinks Clipart and Sketch are not well separated, thus CLIP cannot encode perceptually different domains. However, this observation and conclusion are both incorrect.

**Firstly**, Figure 1(a) shows that the image features from Clipart, Sketch and Real domains are mainly clustered in the left, middle, and right regions of the t-SNE space, respectively.
**Secondly**, we present the quantitive results about the distance between domains in global response R2. 
As shown in the table, even the closest two domains (Clipart and Sketch) have a distance of 0.19, which is significantly larger than the variation within a single domain (0.002).
The above results demonstrate that CLIP's visual encoder encodes different domains within the representation space.

**C2.Figure 1(c) is used to show that "CLIP tends to classify images into categories whose name are closely related to corresponding domain". However, after showing the results on more domains, this claim is not well-supported, especially on DomainNet-R.**

- **Explanation of DomainNet-R**

  For the Table 1 of global response PDF, there may be some misunderstanding.
  DomainNet [1] contains 6 domains, and DomainNet-R denotes the "Real" domain.
  However, the "Real" domain lacks clear conceptual definition, thus there are no categories closely related to this domain and the text encoder bias may be not observious.
  The results in Table 1 of global response PDF also verify this point, as the percentages of top-5 predictions (0.47%-0.63%) is near by the uniform prediction (0.29%).

- **Text Encoder Bias**

  In Lines 145-146 of our paper, we define text encoder bias as "*CLIP exhibits a preference for domain-related categories in specific domains,*" and empirically, CLIP demonstrates worse zero-shot performance in these domains.
  For example, the percentages of top-1 predictions for classes in the Quickdraw and Infograph domains are 21.9% and 4.91%, respectively, which are significantly higher than the uniform prediction (0.29%).
  Meanwhile, zero-shot CLIP achieves only 14.23% and 49.47% accuracy in these two domains, much lower than its performance in other domains (e.g. accuracy of 82.98% in Real domain).
  We believe that CLIP's category preference is closely related to its poor performance in multi-domain scenarios.

> [1] Moment Matching for Multi-Source Domain Adaptation. ICCV2019


**C3.With the pre-training dataset distribution changes, the features from some domains are not separated from the others. The improvement by the proposed method is also very limited (the performance on some domains even declines).**

- **Feature Visualization**

  For the Figure 3(a) in the global response PDF, the reviewer thinks the domain features are not separated from the others.
  However, the conclusion is incorrect.
  We can observe that the image features from Clipart, Infograph, Painting, Quickdraw, Skecth are well separated.
  Besides, we calculate the Euclidean distances of  image features between different domains using OpenCLIP. 
  As shown in the table below, even the closest two domains (Clipart and Sketch) have a distance of 0.27, which is significantly larger than the variation within a single domain (0.00195).

  | DomainNet | C    | I    | P    | Q    | R    | S    |
  | -------- | ---- | ---- | ---- | ---- | ---- | ---- |
  | C        | 0.00 | 0.36 | 0.37 | 0.53 | 0.32 | 0.27 |
  | I         | 0.36 | 0.00 | 0.44 | 0.67 | 0.36 | 0.43 |
  | P         | 0.37 | 0.44 | 0.00 | 0.66 | 0.29 | 0.37 |
  | Q         | 0.53 | 0.67 | 0.66 | 0.00 | 0.60 | 0.51 |
  | R         | 0.32 | 0.36 | 0.29 | 0.60 | 0.00 | 0.37 |
  | S         | 0.27 | 0.43 | 0.37 | 0.51 | 0.37 | 0.00 |

- **Limited Improvment**

  We think the improve by UMFC is not limited.
  We evaluate our UMFC on a wide range of models, involving different backbones and pre-training data. Across all experiments, our method consistently improved average performance. For instance, in the Quickdraw domain, we improved performance from 15.63% to 22.61%. 
  The only exception was a marginal decline (0.1% ~ 0.6%) in the Real domain, while all other domains showed performance gains.
  We believe that this decline can be easily addressed by adjusting the hyperparameters.
Thank you for the detailed responses. My concerns have been addressed, and I would like to update my score to 6.
Many thanks for the new results. Although, the new results still fall short of the my expectations, requesting further experiments may be unreasonable due to time constraints. The reviewer believes there should be a more suitable strategy for integrating CLIP-D and CLIP-E, and the reviewer still believes that the ensembling approach should achieve results at least as good as those of CLIP-D/CLIP-E.

Nevertheless, I will increase the rating to a 6 - weak accept. As far as I know (may not be complete), previous work on improving CLIP's zero-shot classification ability has typically focused solely on enhancing the template itself. Despite some remaining issues with model evaluation aspect, the focus on domain information in this paper is valuable to the community.
Thanks authors for the rebuttal. However, my concerns are not fully addressed, so that I maintain my score as 3.

1) Even only plotting, Domain-C, R, S, it can still be observed that Clipart and Sketch are not well separated. They are perceptually different domains but CLIP cannot encode them well, which means the observations do not hold.

2) Figure 1(c) is used to show that "CLIP tends to classify images into categories whose name are closely related to corresponding domain". However, after showing the results on more domains, this claim is not well-supported, especially on DomainNet-R. 

3) With the pre-training dataset distribution changes, the features from some domains are not separated from the others. The improvement by the proposed method is also very limited (the performance on some domains even **declines**).

4) The response to W6.1 is not convincing. Authors implements K-means clustering on DomainNet training set. All of these images are from 345 classes with six different styles and high quality. It has not been shown that clustering on a random web-crawled dataset brings the same benefits. Suppose that a new task comes, to generate a high quality cluster, some experts are actually needed to gather images from some specific classes with different styles and high quality. This process is in fact more expensive than labelling few shots examples from each class.

5) For computational cost, it seems that the feature extraction time is not considered. May I also know if it is possible to reveal which package you use for clustering? I personally also clustered features from DomainNet before, but it seems not as fast as 2.3 seconds.

6) For W6.2, UMFL clusters features from six domains, while CoOp is trained on only one domain. This comparison is not fair.
**Could you please do the 6x7 style prompts? By concatenating the two prompts from CLIP-D and CLIP-E.**

Thank you for your response. Based on your suggestion, we concatenated the two sets of prompts from CLIP-D and CLIP-E. For each domain, we created 7 prompts, resulting in a total of 42 prompts (6 domains * 7 prompts) for each class.

```python
prompts=[
  "a {domain} image of a {class}. itap of a {class}.",
  "a {domain} image of a {class}. a bad photo of the {class}.",
  "a {domain} image of a {class}. a origami {class}.",
  "a {domain} image of a {class}. a photo of the large {class}.",
  "a {domain} image of a {class}. a {class} in a video game.",
  "a {domain} image of a {class}. art of the {class}.",
  "a {domain} image of a {class}. a photo of the small {class}."
]
```

In the table below, we still observe that the ensemble strategy underperforms compared to original CLIP-D, suggesting that integrating all 42 prompts across 6 domains may weaken domain-specific information. 

| Method     | C     | I     | P     | Q     | R     | S     | Avg   |
| ---------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| CLIP-D                    | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | **61.03** |
| Ensemble (6 domains * 7 prompts)                     | 72.46 | 54.41 | 66.75 | 15.06 | 84.33 | 66.87 | 59.98 |
Thanks for the response. In my experience, the prompting template matters a lot in CLIP zero-shot classification, even a comma will affect the accuracy. I want to ensure the improvement do come from the proposed idea. Considering the limited time for the author to response, could you please do the 6x7 style prompts? By concatenating the two prompts from CLIP-D and CLIP-E.
Dear Reviewer Xgmy,

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

Thanks, Authors.
Dear Reviewer jj75,

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

Thanks, Authors.
Dear Reviewer cRCJ,

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

Thanks, Authors.
**Actually, I want to see the performance in some extreme but realistic cases (e.g., BS=1).**

Thank you for your response. 
We have added experimental results for the scenario where the batch size is 1. 
Initially, when the number of samples is less than the number of clusters \(M\), K-Means clustering cannot be applied. 
To address this, we used the first \(M\) samples as the initial cluster centers and then continued with the same Test-Time Adaptation. 
As shown in the table below, even in the extreme case of a batch size of 1, our method still demonstrates consistent improvement.


|       | C     | I     | P     | Q     | R     | S     | Avg   |
| ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| CLIP | 71.21 | 49.47 | 64.61 | 14.23 | 82.98 | 64.81 | 57.88 |
| UMFC (bs=1) | 72.64 | 53.74 | 66.39 | 18.25 | 83.34 | 66.90 | 60.21 |
**Could the author supplement more details of CLIP-DE method? As I would expect it to be at least as good as CLIP-D/CLIP-E.**

Thank you for your question. In the following, we provide more details about CLIP-E, CLIP-D and CLIP-DE in DomainNet dataset.

- CLIP-E

  In CLIP-E, we use an ensemble of 7 prompts from the following list. 
  ```python
  prompts = [
      "itap of a {class}.",
      "a bad photo of the {class}.",
      "a origami {class}.",
      "a photo of the large {class}.",
      "a {class} in a video game.",
      "art of the {class}.",
      "a photo of the small {class}.",
  ]
  ```

- CLIP-D

  Different with CLIP-E, CLIP-D utilizes a single prompt but with domain information from the test samples. The prompt used in CLIP-D for each test sample is
  ```python
  prompt = ["a {domain} image of a {class}."]
  ```
  Here, {domain} represents the domain name of the test sample, which belongs to one of the six domains in DomainNet: clipart, infograph, painting, quickdraw, real, or sketch.

- CLIP-DE

  Finally, we build CLIP-DE by replacing the single domain-specific prompt in CLIP-D with an ensemble of 8 prompt templates with domain names. 
  Specifically, these templates include the 7 prompts from CLIP-E and 1 domain-specific prompt from CLIP-D. The prompt list for CLIP-DE is:
  ```python
  prompts = [
      "itap of a {domain} {class}.",
      "a bad photo of the {domain} {class}.",
      "a origami {domain} {class}.",
      "a photo of the large {domain} {class}.",
      "a {domain} in a {class} video game.",
      "art of the {domain} {class}.",
      "a photo of the small {domain} {class}.",
      "a {domain} image of a {class}." 
  ]
  ```

  The table in our response to W0 shows that CLIP-DE outperforms CLIP-E, indicating that domain information is indeed beneficial. However, CLIP-DE performs slightly worse than CLIP-D. We attribute this to the ensemble strategy, which weakens the domain information utilized in CLIP-D. 
  This hypothesis is supported by the some experimental observations.
  As shown in the table below, using the same domain information with different prompt templates can significantly impact zero-shot performance. 
  We think this is due to variations in prompt design, which may reduce the effectiveness of domain-specific cues, leading to suboptimal results in some cases.

  | prompt                                | C     | I     | P     | Q     | R     | S     | Avg   |
  | ------------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
  | "a {domain} of a {class}."            | 73.43 | 55.62 | 67.37 | 17.29 | 82.76 | 66.60 | 60.51 |
  | "a photo of a {class} from {domain}." | 73.31 | 54.88 | 67.04 | 15.81 | 82.76 | 66.51 | 60.05 |
  | "a {domain} image of a {class}."      | 73.90 | 55.84 | 67.75 | 17.84 | 83.26 | 67.56 | 61.03 |
Thanks for the response. Could the author supplement more details of CLIP-DE method? As I would expect it to be at least as good as CLIP-D/CLIP-E.
Thanks for the response.  Actually, I want to see the performance in some extreme but realistic cases (e.g., BS=1).

Overall, I will keep my positive score.

Best