WATT: Weight Average Test Time Adaptation of CLIP

Authors: david  osowiechi
mehrdad  noori
gustavo a vargas
hakim  moslem
yazdanpanah  ali
bahri  milad
cheraghalikhani  sahar
dastani  farzad
beizaee  ismail
ben  ayed
christian  desrosiers
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:02:21

Abstract: Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performances for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a new approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing the overall test performance by aggregating diverse textual cues.
Our findings underscore the effectiveness of WATT across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without the need for additional model transformations or trainable modules. Moreover, compared to other TTA methods, our approach can operate effectively with just a single image. The code is available at: https://github.com/Mehrdad-Noori/WATT.

Main paper: 
 introduction: 
 The integration of vision and language modalities into a unified learning framework, known as Vision Language models (VLM), has shown remarkable effectiveness in a broad range of vision-related tasks [1, 2, 3] . Notably, these models excel in zero-shot generalization scenarios, where they demonstrate proficiency in tasks beyond their original training scope, without requiring additional fine-tuning supervision. Applications of models like CLIP [1] extend across diverse domains including video recognition [4] , audio processing [5] , and medical imaging [6] . These advancements underscore the pivotal role of such methods in shaping the trajectory of future research and applications in machine learning.Despite its powerful capabilities, CLIP, like other traditional deep architectures such as Convolutional Neural Networks (CNNs), experiences performance degradation when confronted with domains it has not been trained on. Current research trends emphasize the importance of domain adaptation mechanisms in the deployment of CLIP [7, 8] . However, a significant challenge remains: swiftly and Template T 0 : "a photo of a {class k}" T 1 : "itap of a {class k}" T 2 : "a bad photo of the {class k}" T 3 : "a origami {class k}" T 4 : "a photo of the large {class k}" T 5 : "a {class k} in a video game" T 6 : "art of the {class k}" T 7 : "a photo of the small {class k}" effectively adapting the model to new domains while preserving its attractive zero-shot capabilities, thus obviating the need for retraining.T 0 T 1 T 2 T 3 T 4 TTo tackle this challenge, we investigate the impact of different text prompt templates, listed in Figure 1a , on model adaptation. Despite these templates being related, as reported in Figure 1b , the cosine similarity between their embeddings varies greatly, suggesting that they encode complementary information about the classes. The variability of information in different templates can also be observed in Figure 1c , where the classification accuracy obtained with these templates on a subset of CIFAR-10 corruptions fluctuates by up to 3%. Given this insight, finding an effective way to leverage the knowledge from different text templates would be useful to yield a better adaptation. This motivates our work proposing Weight Average adaptation during Test-Time (WATT).By strategically averaging the adapted weights derived from multiple text prompt templates, our method aims to harness the complementary strengths of individual templates, resulting in robust and enhanced performance across a wide range of domain shifts. To further illustrate this point, Figure 2 presents the test loss and adaptation error surfaces for three models that are separately adapted using three templates (T 0 , T 1 , T 2 ) under the Gaussian noise corruption of the CIFAR-10-C dataset 2 . The central point in these landscapes, representing the final model obtained by averaging the weights of the separate models, demonstrates a convergence towards lower loss and error, highlighting the potential of weight averaging for test-time adaptation. Moreover, inspired by recent advancements in machine learning utilizing train-time weight averaging techniques [10, 11] , the proposed WATT method can dynamically adjust to new data to tackle unforeseen distribution shifts without relying on class supervision.We outline the main contributions of our work as follows:• We introduce a novel Test-Time Adaptation method for CLIP, which leverages weight averaging across various text templates at test-time. 2 To visualize the loss and error surface, we use weight vectors from models adapted with text templates T 0 , T 1 , and T 2 , denoted as w0, w1, and w2. Following [9] , we define u = w1 -w0 and v = (w2 -w0) -(w 2 -w 0 )•(w 1 -w 0 ) ∥w 1 -w 0 ∥ 2 (w1 -w0). The normalized vectors û = u ∥u∥ and v = v ∥v∥ form an orthonormal basis in the plane of w0, w1, and w2. We create a Cartesian grid in this basis and evaluate the networks at each grid point. A point P with coordinates (x, y) in the plane is given by P = w0 + x • û + y • v. To plot all in the same plane, we used the average of the three templates' text embeddings. 1a ). The central point (cross) shows the model obtained by averaging these weights, demonstrating improved performance.• Our WATT strategy yields highly competitive performances in comparisons to current TTA methods, and could bring improvement using only a single image at test time, a capability not present in previous approaches. • We rigorously evaluate our WATT methodology through comprehensive evaluations across different datasets characterized by diverse types and degrees of domain shifts, encompassing a total of 155 evaluation scenarios. Our experiments demonstrate the robustness and efficacy of WATT compared to existing adaptation methods. 
related work: 
 Test-Time Adaptation (TTA) is crucial in domain adaptation, particularly with unlabeled target domain data and no access to source domain samples. The challenge lies in estimating the target domain's distribution and comparing domain characteristics indirectly. Recent advancements have highlighted the potential and limitations of adapting pre-trained models. A key focus has been on leveraging batch normalization layers for adaptation due to their ability to retain source domain information. Methods such as PTBN [12] and TENT [13] recalibrate batch statistics and optimize affine parameters via entropy minimization, though they often require image augmentations or large batches. MEMO [14] proposes a simple approach that does not require multiple test points; it uses single-test-point data augmentations and minimizes the marginal entropy of the model's average output to enforce consistency and improve robustness. LAME [15] introduces a closed-form optimization strategy that refines model predictions for target images by leveraging the Laplacian of feature maps to encourage clustering, thereby emphasizing feature similarities. In another study, SAR [16] addresses TTA instability by using batch-agnostic norms (e.g., group and layer norms) and a sharpness-aware entropy minimization approach. It filters noisy samples and guides the model to flat minima, improving robustness under mixed domain shifts and small batch sizes.Recently, Test-Time Training (TTT) methodologies have emerged as prominent contenders in TTA [17, 18, 19, 20, 21] . This approach involves training a supplementary sub-branch alongside the primary network in an unsupervised manner, subsequently leveraging it to refine the model. Unlike previous methods, our approach operates on individual image batches, offering a significant advantage in TTA by avoiding the necessity of training additional branches from scratch.In natural language processing, TPT [8] introduced entropy minimization for adapting models like CLIP, albeit with high computational costs due to learning an adapter at the text prompt with multiple transformations. DiffTPT [22] extends this by leveraging pre-trained diffusion models to generate diverse and informative augmented data, combining conventional augmentation methods used in TPT with diffusion-generated data to enhance adaptability. It also introduces a cosine similarity-based filtration technique for improved prediction fidelity. TDA [23] offers a training-free approach with a dynamic adapter, utilizing a lightweight key-value cache and pseudo label refinement, making it computationally efficient. CLIPArTT [24] , fine-tunes normalization layers with minimal disruption to the model's knowledge, enhancing text supervision by introducing pseudo labels. Existing methods often lag behind supervised prompt adaptation techniques in performance. SwapPrompt [25] bridges this gap by leveraging self-supervised contrastive learning, employing a dual prompt paradigm. In comparison, our method combines prompt augmentation and fine-tuning of normalization layers, highlighting its effectiveness in test-time adaptation.Weight Averaging (WA) is a powerful train-time technique for improving deep neural network generalization. Stochastic Weight Averaging (SWA) [26] averages weights of multiple models sampled from different training epochs, aiding smooth optimization trajectory and convergence to points with superior generalization. SWAD [10] refines SWA by densely sampling weights throughout training, enhancing generalization and robustness across tasks. This train-time refinement enhances WA's effectiveness in producing models with improved generalization. The Lookaround [11] optimizer iterates between an "around step" and an "average step", building on SWAD's advancements. In the "around step", independently trained models using various data augmentations explore a broader loss landscape to find flatter minima. In the "average step," weights of these models are then averaged, guiding optimization towards lower loss regions. This method enhances robustness and generalization across tasks, improving upon SWA and SWAD by providing a more effective weight averaging process.In contrast to existing approaches, WATT leverages varied text prompts to adapt vision-language models such as CLIP during testing. Our method also harnesses the benefits of weight averaging while addressing domain shifts without additional model transformations or trainable modules, thereby setting a new precedent in test-time adaptation. 
method: 
 The proposed WATT method, summarized visually in Figure 3 comprises three main components, the first two in the Adaptation Phase and the third in the Evaluation Phase: batch samples in terms of their visual and text features; 2) a weight-averaging strategy using multiple text templates to generate diverse model hypotheses during adaptation; 3) an ensembling technique that boosts performance during evaluation by averaging the embedding of different text templates.While our method can be employed with any TTA framework, in this work, we implement a strategy inspired by the transductive TTA approach of CLIPArTT [24] which effectively incorporates semantic relationships among batch samples.Initially, our process involves executing inference using CLIP, a system comprising a visual encoder f v θ (•) that translates an image x into visual features z v ∈ R D , and a text encoder f t θ (•) which converts text prompts t into text features z t ∈ R D . During inference, we employ pre-defined text prompts assigned to each class within a dataset, such as t 0 k = "a photo of a {class k}". For a new image x i , the likelihood of belonging to class k is then computed using cosine similarity:EQUATIONwhere τ is a softmax temperature parameter set to 0.01 is this work. This prediction is then stored to be used as pseudo labels for the model.Denoting the normalized visual embeddings of the samples within the test batch as Z v ∈ R B×D and the instance-specific text embeddings as Z t ∈ R B×D , we compute an image-to-image similarity matrixS v = Z v (Z v ) ⊤ ∈ [-1, 1]B×B modeling pairwise relationships in terms of image characteristics. Similarly, we construct a text-to-text similarity matrix S t = Z t (Z t ) ⊤ ∈ [-1, 1] B×B , capturing the semantic relationships among text embeddings within the batch. Utilizing the computed pairwise similarity matrices, we generate pseudo labels Q = softmax (S v + S t )/2τ ∈ [0, 1] B×B which are used with cross-entropy in our transductive TTA loss:EQUATIONDrawing a link with the Stochastic Neighbor Embedding (SNE) method for dimensionality reduction [27] , which minimizes the KL divergence between distributions modeling pairwise distances, our TTA loss ensures that the inter-modality (text-to-image) similarities of batch samples are aligned with their intra-modality ones (text-to-text and image-to-image).We explore various text prompt templates suggested in the CLIP paper and detailed in Fig. 1a . As reported in Table 1c , these prompts achieve varying performance across different corruption types of CIFAR-10-C. We formulate prompts of the form t h k = template h(class k), where h ∈ {1, 2, . . . , H}, encompassing a spectrum of textual cues tailored to elicit diverse responses from the model. Two different approaches are investigated for our multi-template weight averaging (MTWA) strategy. The first one denoted as Parallel MTWA (WATT-P), which follows recent optimization approaches like Lookaround [11] , performs the adaptation separately for each text template, starting from the same parameters, and then averages the resulting adapted weights. The second one, called Sequential MTWA (WATT-S), instead considers text templates sequentially without resetting the weights. These two approaches, which we illustrate and compare in Fig. 4 , are detailed below. Parallel MTWA. This approach optimizes the TTA loss in (2) separately for H different models, each utilizing a distinct template. Starting from the same visual encoder parameters θ, these models are updated in parallel for L iterations, resulting in updated parameters θ ′ h , with h ∈ {1, . . . , H}. The parameters are reset after each update, enabling each model to restart the adaptation from the same initial point. Subsequently, we aggregate the weights obtained from these H models by computing their average:θ avg = 1 H H h=1 θ ′ h .We repeat this step M times, and denote the overall process as "(after L iter) × M ". Sequential MTWA. Our Sequential MTWA approach is inspired from the work of [26] , where the averaging of weights across various stages of the training process is employed to mitigate variance and enhance generalization capabilities. Instead of resetting parameters for each model, we update parameters after each template's iteration. To ensure impartiality in the update sequence of templates, a random selection process is implemented, thereby disregarding any predetermined order.At test-time, predictions are computed using Equation 1 through two distinct methodologies. In the first approach, the text features z t 0 are derived from the initial text prompt t 0 k = "a photo of a class k", denoted as single_temp. Conversely, the second method aggregates the text features from all templates by computing their mean, resulting in the prediction z t ens = 1 H H h=1 z t h , denoted as text_avg (see Fig. 3 ). 
experimental setup: 
 Settings. In line with prior TTA methodologies, adjustments are made to all Layer Normalization layers within the visual feature extractor for test-time adaptation. The Adam optimizer is employed with a fixed learning rate of 10 -3 , wheras a smaller learning rate of 10 -4 is chosen for adaptation to the 3D renderings split, as it reflects a more pronounced shift. Throughout our experimentation process, a consistent batch size of 128 is maintained to ensure uniformity and facilitate meaningful comparisons across various scenarios.Datasets. Following [24] , we rigorously evaluate WATT's performance across diverse TTA datasets using established assessment techniques. These datasets simulate intricate domain shifts, providing nuanced insights into our approach's effectiveness. Additionally, we explore WATT's adaptability on the original dataset through zero-shot test-time adaptation. To ensure a thorough examination, we extend our analysis to include various domain generalization datasets, exposing our method to a broad spectrum of image categories for comprehensive evaluation. Our evaluation framework encompasses natural images, common corruptions, simulated shifts, video shifts, texture shifts and style shifts.In our assessment of natural image analysis, we include CIFAR-10, CIFAR-10.1, and CIFAR-100, each comprising 10,000 images and offering varied data distributions. CIFAR-10.1 [28] introduces a natural shift from CIFAR-10, providing a comprehensive evaluation of our model's performance.We also incorporate the CIFAR-10-C and CIFAR-100-C datasets [28] , augmented with 15 distinct corruptions across 5 severity levels, resulting in 75 common corruption scenarios. This comprehensive augmentation assesses the model's resilience effectively.Our investigation also extends to the VisDA-C dataset [29] , challenging models with simulated and video shifts across diverse imagery types. Additionally, we evaluate our method on three datasets mostly used in the field of domain generalization: PACS [30] , VLCS [31] , and OfficeHome [32] datasets, instrumental in understanding texture and style variations. These evaluations effectively demonstrate the generalizability of our method across distinct domain shifts.Benchmarking. We conduct a comparative analysis of WATT against contemporary methods using ViT-B/32 as the backbone. Specifically, we incorporate an adapted version of TENT [13] , customized for CLIP by the authors of [24] , using 10 iterations. We also include SAR [16] and MEMO [14] in the same manner as TENT. Additionally, we compare with TPT [8] , a novel adaptation technique for CLIP that heavily relies on image augmentations, as well as DiffTPT [22] , an extension that incorporates diffusion models. Lastly, we include TDA [23] , which employs a dynamic adapter, and CLIPArTT [24] , a recent approach that utilizes pseudo labels generated through conformal learning. 
results: 
 In this section, we present empirical findings from our WATT method through a series of ablation studies aimed at understanding the impacts of individual components. These studies inform subsequent experiments across diverse datasets. Leveraging insights from ablations, we conduct comprehensive experiments, benchmarking WATT against state-of-the-art techniques across various datasets.In this section, unless otherwise specified, we focus on the Sequential MTWA variant of our method (see Section 3.2) and will use these findings as a reference for the Parallel MTWA method.Comparison of the template used during testing. After updating the model, we proceed to compute the similarity between the image features and the text embeddings, enabling prediction. Typically, text embeddings originate from the text prompt "a photo of a class k". However, by employing multiple templates, we have the flexibility to alter this text embedding through the averaging of all text embeddings from each template. Table 1 conducts a comparative analysis revealing that this averaged text embedding consistently yields superior results across all scenarios. Hence, we adopt this approach for next experiments.Comparison of the number of templates. In Figure 5 , we examine the performance variation relative to the number of utilized templates. In this investigation, we conduct 5 runs wherein the templates are randomly selected from a pool of 8 distinct templates (as outlined in Fig. 1a ). Notably, when the distribution shift is minimal, as observed in CIFAR-10 and CIFAR-10.1, optimal performance is attained using 6 templates, with performance gradually diminishing thereafter. Conversely, in scenarios characterized by substantial corruptions, such as CIFAR-10-C, employing all 8 templates proves advantageous. Consequently, our focus moving forward will be on utilizing all 8 templates in our work. Text Averaging vs Output Averaging vs Weight Averaging. Utilizing the averaging method within a VLM offers several possibilities, including averaging the weights, the outputs or the text embeddings before computing the logits. In Table 3 , a comparison between these approaches is presented. It becomes evident that weight averaging consistently outperforms text embedding averaging across various datasets, showcasing a superiority of approximately 1% even with the less effective weight averaging method. This performance advantage is observed across CIFAR-10, CIFAR-10.1, and CIFAR-10-C, and persists even with larger numbers of classes, such as in CIFAR-100 and CIFAR-100-C. When concentrating on output averaging, the results may be less evident with less effective weight averaging methods. However, they remain valid and even more accurate with superior weight averaging techniques. Therefore, our focus for future experiments will be on weight averaging as the preferred approach.Best moment to do the Weight Averaging. Examining Table 3 , it is evident that the parameters L and M discussed in Section 3 are crucial. Specifically, a large L (e.g., 10) combined with a small M (e.g., 1) is ineffective. Conversely, setting L = 1 and M = 10 yields optimal results for small distribution shift datasets, while L = 2 and M = 5 perform best on highly corrupted datasets. Given that TTA typically encounters substantial distribution shifts, we will use L = 2 and M = 5 in our subsequent experiments.Performance over the number of iterations. In this section, we focus on the method incorporating a Parallel MTWA mechanism and examine the impact of the number of iterations on performance. As illustrated in Figure 6 , the accuracy stabilizes after approximately 20 iterations. Although there is a slight improvement in performance beyond 50 iterations, the difference is marginal. Based on these observations, we have opted to use 50 iterations for our experiments.Model performance across various batch sizes. In our investigation, we delve into the performance implications of TTA methods when operating under small batch sizes, a historical challenge in the field. Performance evaluation with small batches. In existing TTA works that study small batch size scenarios, such as SAR [16] and MEMO [14] , we implement these methods in the context of CLIP. We compare our method with other TTA approaches using a batch size of 1. As shown in Performance evaluation in the presence of natural or no domain shift. In Table 4 , results show consistent performance enhancements with WATT, both with the Parallel and Sequential MTWA strategies, alongside the baseline. On CIFAR-10, performance improves by 2.67% with Parallel MTWA and 2.31% using Sequential MTWA. On CIFAR-10.1, improvements reach 4.53% and 3.73%, and on CIFAR-100, enhancements are 8.70% and 9.06%. While WATT consistently outperforms the baseline, TPT, and CLIPArTT, TENT yields superior results on CIFAR-10. WATT's effectiveness often correlates with the number of classes, showing better performance with more classes, indicating its strength in lower-confidence scenarios.Performance evaluation in the presence of common corruptions. Table 4 shows that both WATT variants consistently outperform alternative methods across various corruptions and class numbers. Notably, WATT with Parallel MTWA improves performance by 16.48% on CIFAR-100 Gaussian Noise and by 17.01% on Glass Blur compared to the baseline, while WATT with Sequential MTWA shows improvements of 17.27% and 17.96% respectively. On common corruptions, the Sequential MTWA variant surpasses Parallel MTWA, with improvements of 0.99% on CIFAR-10 and 0.89% on CIFAR-100. According to the TDA supplementary materials, we selected the weighting factor alpha as 5.0 and the sharpness ratio beta as 2.0, which are stated as optimal. However, these values did not appear to be the best choice for more challenging datasets like CIFAR-10-C or CIFAR-100-C that contain various corruptions. Adjusting these parameters based on the dataset would not be consistent with the principles of a fully TTA method, which might explain their suboptimal performance in our results. Regarding DiffTPT, it generates 64 images per test image, making it challenging to use in a real-world TTA scenario. Similar to TDA, DiffTPT requires carefully chosen parameters to fit the dataset, whereas our method does not require dataset-specific tuning. This highlights the robustness and practicality of our approach in diverse real-world applications.Performance analysis under simulated and video shifts. Results on the 3D (simulated shift) and YT (video shift) splits of VisDA-C demonstrate a significant improvement in accuracy with our proposed WATT method compared to pure CLIP. The Sequential MTWA variant achieves the highest accuracy on both the 3D and YT splits, with scores of 85.36% and 84.69%, respectively, surpassing other adaptation methods including TENT, TPT, and CLIPArTT (see Table 6 ).Performance analysis under texture and style shifts. Results on the OfficeHome, PACS, and VLCS datasets are presented in Table 6 . On average, our proposed WATT method, with Parallel and Sequential MTWA variants, improves performance across the different domains of OfficeHome, PACS, and VLCS compared to other methods. This highlights its robustness in addressing texture and style shifts, which are especially challenging compared to other domain shift variants. 
conclusion: 
 We introduce WATT, a Test-Time Adaptation method tailored for Vision-Language Models. Our approach harnesses Weight Averaging with different text prompts and incorporates text embeddings averaging to bolster prediction accuracy.Through an extensive ablation study, we scrutinized the efficacy of employing varied text prompts and weight averaging. Comparative evaluations across Test-Time Adaptation and Domain Generalization datasets underscored the superiority of our method, particularly in scenarios involving distribution shifts and zero-shot performance enhancements compared to state-of-the-art approaches.Looking forward, investigating the potential of text prompts and weight averaging in classification opens up promising avenues for future exploration. Our methodology, with its focus on template manipulation, suggests potential avenues for extension, such as incorporating alternative class descriptors, yielding valuable insights for future research. Moreover, expanding Test-Time Adaptation to encompass diverse scenarios, including segmentation or object detection with Vision-Language Models, holds significant potential for advancing our comprehension of model adaptability and performance across varied tasks. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper addresses a research field that is not currently utilized for direct practical applications, thereby limiting its immediate social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In our paper, we cite all sources that inspired our work, and we provide links in our git repository to the codes from which we drew inspiration.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We document all aspects of our research and provide a link to our implementation, along with step-by-step explanations for reproducing the results.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our proposed WATT method is implemented in Python using the PyTorch (version 2.0.1) framework. All experiments were conducted on an NVIDIA V100 32 GB GPU. However, due to the effectively and lightweight nature of our method, it can be executed on less powerful GPUs. Specifically, the adaptation with a batch size of 128 using the ViT/B32 backbone requires up to 4 GB of memory, making it feasible to use on a wide range of GPUs. For fairness and consistency, we re-implemented and ran all other methods, including CLIPArTT, TENT, and TPT, in the same environment. Each experiment was performed three times to ensure reliability (three trials per experiment). To facilitate reproducibility, we provide the original implementation and detailed step-by-step instructions in our repository, accessible via this github link.The CLIP paper identifies 80 templates that enhance model robustness and performance. They ultimately conclude that 7 of these templates best summarize their model (see this link). In our work, we use these 7 generic templates and add the common one, "a photo of {}," based on their optimization. These templates are not specifically linked to the content of the images.In our investigation, we use the VisDA-C dataset, which challenges models with two distinct domain shifts: the simulated shift and the video shift. The simulated shift includes 152,397 3D-rendered images across 12 diverse classes, while the video shift comprises 72,372 YouTube video frames spanning the same categories. This dataset addresses the diversity of imagery types applicable to a model, posing a significant challenge.Moreover, we evaluate our proposed method on three other datasets: PACS, VLCS, and OfficeHome. These datasets help understand various domain shifts, including texture and style variations. The PACS dataset consists of 9,991 images across four domains (Art, Cartoons, Photos, Sketches) and seven classes. The VLCS dataset contains 10,729 images across four domains (Caltech101, LabelMe, SUN09, VOC2007) and five classes. Lastly, the OfficeHome dataset includes 15,588 images across four domains (Art, Clipart, Product, Real) and 65 classes. Evaluating across these distinct domain shifts showcases the generalizability of our method. These datasets are more representative of real-world scenarios compared to CIFAR, with complex domain shifts.We conduct a thorough evaluation under consistent conditions using an NVIDIA A6000 GPU within the same Python environment. The Table 7 provided compares the adaptation time, memory usage, and the number of learnable parameters for various TTA methods, including our proposed WATT method.The table clearly demonstrates that WATT-S, a sequential implementation of WATT, maintains competitive adaptation time and memory usage compared to other methods like TENT and ClipArTT, which are efficient but lack the robustness of WATT's method. Additionally, the table highlights that WATT-P, with parallel model training, offers a faster adaptation time than WATT-P with a for-loop implementation, albeit at the cost of higher memory usage. It's important to note that methods like DiffTPT [22] and MEMO [14] , which show significantly higher adaptation times, employ offthe-shelf diffusion models and AugMix augmentation, respectively, resulting in time-consuming processes that may be impractical for real-world scenarios. In contrast, the effectiveness of our WATT-S method makes it better suited for scenarios where a robust, rapid, and resource-efficient adaptation is crucial. In Algorithms 1 and 2, we compare the two variants of WATT: one with Parallel MTWA (WATT-P) and the other with Sequential MTWA (WATT-S). The WATT-P model recalibrates its parameters for each template using the average parameters of m -1, whereas the WATT-S model updates its parameters solely at the start of each new iteration m.Algorithm 1 WATT-P -model f , parameter θ Cross Entropy vs Entropy Minimization. Two unsupervised loss functions were integrated into previous TTA methods: classical entropy minimization and the loss introduced by CLIPArTT [24] , where predictions are utilized as pseudo labels for cross-entropy computation. In Table 8 , a comparison between these two loss functions is presented across the original CIFAR-10 dataset and various corruptions from CIFAR-10-C. It is observed that, for these specific corruptions, entropy minimization generally outperforms with the different templates employed, except for Gaussian Noise. However, upon assessing the weighted average accuracy, computed after 10 iterations for each template, cross-entropy consistently emerges as the superior option. The marginal impact of the weighted average on entropy minimization suggests that, irrespective of the template used, the model updates in a consistent direction to enhance confidence, rendering cross-entropy the preferred choice for subsequent experiments. 1a ) and the weight average.performance alongside the baseline. This pattern persists across different visual encoders, as shown in Tables 10 . Although WATT consistently outperforms the baseline and TPT, TENT and CLIPArTT may occasionally yield superior results depending on the visual encoder used.Performance evaluation in the presence of common corruptions. Table 10 demonstrates a consistent trend where both WATT methods consistently outperform alternative methods across various corruptions and class numbers. Upon closer examination of Table 10 , specifically with ViT-B/16 as the visual encoder, Sequential MTWA exhibits a significant performance advantage, surpassing the baseline by 16 .07% and the leading method in the state-of-the-art by 3.00%. This trend becomes even more pronounced with an increased number of classes, where Sequential MTWA surpasses the baseline and CLIPArTT by 16.94% and 8.87%, respectively.Performance analysis under simulated and video shifts. Our study reveals substantial accuracy improvements on the 3D (simulated shift) and YT (video shift) partitions of VisDA-C when employing different backbones. This enhancement is particularly notable with our proposed WATT method compared to pure CLIP. Notably, the WATT-S variant achieves the highest accuracy across both the 3D and YT partitions, outperforming various adaptation approaches including TENT, TPT, and CLIPArTT. Detailed comparisons can be found in Tables 11 and 12 .Performance analysis under texture and style shifts. Findings across the OfficeHome, PACS, and VLCS datasets are detailed in Tables 11 and 12 . Across these varied domains, our WATT method demonstrates consistent performance enhancements, as evidenced by both its WATT-P and WATT-S variants. These improvements underscore the efficacy of our approach in mitigating the complexities of texture and style shifts, which pose particular challenges compared to other forms of domain shift. This section provides extensive tables with detailed information on the results, which were summarized in the main body of the paper. Justification: Everything is detailed in the section Implementation of the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We have considered all potential harms caused by the research process, societal impacts, and potential harmful consequences, as described in the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes "Weight Average Test Time Adaptation" (WATT) for VLMs like CLIP, aimed at enhancing their ability to handle domain shifts during inference without retraining. The method leverages multiple text prompt templates to adapt model weights at test time and aggregates predictions using weight averaging to improve performance across datasets with domain shifts, such as CIFAR and VisDA. The results show notable improvements in robustness and generalization without the need for model modifications or additional training.

The paper is technically solid and presents a practical approach to improving VLMs under domain shifts without retraining. Despite the limited novelty, the experimental results demonstrate benefits in terms of robustness and generalization, making it a valuable contribution to the test-time adaptation field. While there are some concerns about the breadth of evaluation and computational aspects, the overall impact on the VLMs and TTA subfield justifies its acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a method called Weight Average Test-Time Adaptation (WATT) to improve test-time adaptation (TTA) for the CLIP model. The core idea is to use different text templates to construct multiple text prompts and adapt the model weights using these different prompts. During the evaluation stage, the prompts provide predictions based on the adapted weights and text embedding averaging across multiple prompts. The authors conduct comprehensive experiments that demonstrate the generalization and strong performance of the WATT method.
strength_weakness: strength_weakness: There are no clear weaknesses, but I do have some questions and potential limitations that do not affect my rating.
questions: questions: * Is it possible to combine text prompt augmentation (this method) with TTA strategies that use image augmentation to achieve better results?
Some templates may be incorrect. For example, as shown in Table 2, prompt T0: "a photo of a {class k}" and prompt T2: "a bad photo of the {class k}" refer to the same image, but why is this photo considered bad? If you add distortion or corruption, then it might be appropriate to call it bad. Also, for T6: "art of the {class k}," if you use diffusion to generate an art image of the original image, then the prompt would match.
* Can you measure the similarity score of these text templates in the CLIP text encoder space?
limitations: limitations: Can the authors test whether this method can also improve the latest VLM, such as SigLIP?
paper_summary: paper_summary: This paper proposes a test-time adaptation (TTA) method for CLIP by integrating various textual prompt templates into Weight Average (WA) methods (Ref [11], [22]). Experiments on multiple types of domain shifts show the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The second contribution at the end of Section 1 lacks sufficient evidence. Table 4 only shows that the proposed method can handle small batch sizes without comparison with others. Additionally, there are existing TTA works studying small batch size scenarios. (SAR [Towards Stable Test-Time Adaptation in Dynamic Wild World, 2023], MEMO [MEMO: Test Time Robustness via Adaptation and Augmentation, 2022], etc.). Recent diffusion model-based TTA methods (such as DDA [Back to the Source: Diffusion-Driven Test-Time Adaptation, 2023]) also show robustness for small batch sizes.
- The technical contribution seems not that significant. The core idea is introducing multiple prompt templates into two previous WA methods, to my understanding.
questions: questions: Is the evaluation done online (like the setting in the TENT paper where inference and adaptation happen on each batch in data streams), or does the model adapt with a single batch and then perform inference on the whole test set?
limitations: limitations: The authors mention limitations in the checklist.
paper_summary: paper_summary: The authors introduce a Test-Time Adaptation technique named Weight Average Test-Time Adaptation (WATT) for Vision-Language Models (VLMs) such as CLIP. WATT is the proposed method that improves test-time adaptation by borrowing ideas from different text prompt templates to develop pseudo-labels for model updates and also carrying out weight averaging so as to consolidate learned information from those generated labels. The approach is tested across different datasets, revealing its capacity to boost performance without any further metamorphosis of the model or additional trainable modules.
strength_weakness: strength_weakness: - Limited Scope of Evaluation: State-of-the-art Methods [1, 2] published in recent CVPR/ICCV are missing for comparing in Experimental part and discussing in the Related Work section.
[1] Diverse data augmentation with diffusions for effective test-time prompt tuning. ICCV, 2023.
[2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR, 2024.
- Computational Cost: The paper mentions the efficiency of WATT but does not provide a detailed comparison of computational costs with other TTA methods, which could be crucial for practical deployment.
- Generalization to Other Tasks: The paper focuses on image classification tasks. Extending the approach to other vision tasks like segmentation or detection and evaluating its performance there could provide a more comprehensive understanding of its applicability.
questions: questions: 1. Could you provide a more detailed comparison of the computational cost of WATT compared to other TTA methods?
2. Have you considered evaluating WATT on real-world datasets with more complex domain shifts?
3. How would WATT perform on other vision tasks such as segmentation or object detection?
limitations: limitations: The authors have addressed the limitations and potential negative societal impacts adequately. They highlight the focus on image classification and suggest future work extending the method to other tasks. However, they could discuss more on the computational resource requirements and scalability in practical deployments.

Comments: title: Reply to authors
comment: Thank you for the additional experiments and explanations. The experiments have addressed my main concern, so I have raised my score from 5 to 6. However, I still maintain that the technical contribution is not particularly significant.
We're grateful for your detailed feedback and pleased that our response effectively addressed your concerns. The improvement with SigLip is encouraging, and we sincerely thank you for suggesting that comparison. Your input has been invaluable in strengthening our work.
Thank you for your rebuttal. All my concerns have been addressed. The results reflected in SigLip are very positive, so I have increased my score.
Thank you for your thorough review and constructive feedback throughout this process. We greatly appreciate your positive evaluation and are pleased that our revisions have addressed your concerns.
Thank you for thoroughly addressing my concerns. After reviewing the comments and responses for the other reviewers, I see that their concerns have also been resolved. The authors have provided clear definitions of terms for better understanding, conducted additional experiments to further evaluate the effectiveness of the proposed method, and offered more in-depth analysis of how the proposed method works in various settings.

Overall, the rebuttal enhances my confidence in this paper. With careful consideration, I believe this paper with revision is worthy of NeurIPS and will significantly impact the test-time adaptation field. My final decision is “strong accept.”