WATT: Weight Average Test Time Adaptation of CLIP
Authors: David OSOWIECHI
Mehrdad Noori
Gustavo Adolfo Vargas Hakim
Moslem Yazdanpanah
Ali Bahri
Milad Cheraghalikhani
Sahar Dastani
Farzad Beizaee
Ismail Ben Ayed
Christian Desrosiers
Venue: NeurIPS 2024 poster (2024)
Keywords: Weight Averaging
Test Time Adaptation
CLIP
Text Ensemble
Primary area: online_learning

Abstract: Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performances for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a new approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing the overall test performance by aggregating diverse textual cues.
Our findings underscore the effectiveness of WATT across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without the need for additional model transformations or trainable modules. Moreover, compared to other TTA methods, our approach can operate effectively with just a single image. The code is available at: https://github.com/Mehrdad-Noori/WATT.

WATT: Weight Average Time Adaptation of CLIP
David Osowiechi"
Mehrdad Noori"
Gustavo A. Vargas Hakim
Moslem Yazdanpanah
Ali Bahri
Milad Cheraghalikhani
Sahar Dastani
Farzad Beizaee
Ismail Ben Ayed
Christian Desrosiers
LIVIA, ÉTS Montréal, Canada
International Laboratory on Learning Systems (ILLS)
Abstract
Vision-Language Models (VLMs) such as CLIP have yielded unprecedented perfor- mances for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, new approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing the overall test performance by aggregating diverse textual cues. Our findings underscore the effectiveness of WATT across diverse datasets, including CIFAR-10-C, CIFAR-10.1 CIFAR-100-C; VisDA-C,, and several other challenging datasets, effectively covering wide range domain shifts. Notably, these enhancements are achieved without the need for additional model transformations trainable modules. Moreover, compared other TTA methods, our approach can operate effectively with just a single image. Noori/WATT
The code is available ftaps://githhe
Introduction
The integration of vision and language modalities into unified learning framework. known as Vision Language models (VLM), has shown remarkable effectiveness in broad range of vision-related tasks 32. Notably, these models excel in zero-shot generalization scenarios, where they demonstrate proficiency in tasks beyond their original training scope, without requiring additional fine-tuning supervision. Applications of models like CLIP [1] extend across diverse domains including video recognition 4 audio processing (5), and medical imaging [6]. These advancements underscore the pivotal role of such methods in shaping the trajectory of future research and applications machine
learning.
Despite its powerful capabilities, CLIP, like other traditional deep architectures such Convolutional Neural Networks (CNNs), experiences performance degradation when confronted with domains has not been trained on. Current research trends emphasize the importance of domain adaptation mechanisms in the deployment of CLIP g. However, a significant challenge remains: swiftly and
"qqual contribution.
Correspondence to david..s.oiechh. (@ens.etsml..ca and mehrdad.nooni.. wwt.iooor..eeeeeeeeeeeeo.e @@ensetsmmt.ca
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Aucroge Similaity
Template
0.35
T. "a photo of a {class h3"
0.99
Tt. "itap of a {class k}"
0.90
24: "a bad photo of the (class k)"
0.25
0.77
74. "a origami {class k}"
R
0.54
3.75
0.00
0.74
1.00
0:85
0.58
0.93
0.80
T4, "a photo of the large {class kz"
|889
0.74
0.50
0.75
0:55
1.00
0.58
0.87
0.75
29, "a (class k) in video game"
To "art the {class k)"
+
0:91
3.69
0.84
0:77
0.50
6
1.00
6.89
T "a photo of the small {class ky"
0.95
3.75
0.91
0.77
0.93
0:57
0.55
1.00
0.65
7
(a)
(h)
7
T
T
7
7
7
7
7
WATT
Original
89.80
90.37
90.50
88 28.42
89.93
89.95
90.13
88.54
91.05
Gaussian Noise
60.19
61.01
61.17
58.24
58.84
58.35
59.62
61.13
63.84
Defocus Blur
77.23
77.07
78.00
75.98
76.39
77.45
77.08
75.59
78.94
Snow
76.57
77.36
77.93
75.08
77.45
77.09
77.05
75.57
79.79
JPEG Compression
64.65
65.36
65.24
64.16
64.18
64.36
64.78
65.32
67.36
(c)
Figure 1: (a) The different templates used during our experiments. (b) Matrix of cosine similarity
between each template, averaged over all classes of the CIFAR-10 dataset. (c) Comparison of accuracy (%) using cross-entropy (CE) on CIFAR-10 and some corruptions of CIFAR-10-C datasets using different templates and our weight average strategy.
effectively adapting the model to new domains while preserving its attractive zero-shot capabilities, thus obviating the need for retraining.
To tackle this challenge, we investigate the impact of different text prompt templates, listed in Figure [a| on model adaptation. Despite these templates being related, reported in Figure Tb. the cosine similarity between their embeddings varies greatly, suggesting that they encode complementary information about the classes. The variability of information in different templates can also be observed in Figure Ic, where the classification accuracy obtained with these templates on subset of CIFAR- corruptions fluctuates by up to 3%. Given this insight, finding an effective way to leverage the knowledge from different text templates would be useful to yield better adaptation. This motivates our work proposing Weight Average adaptation during Test-Time (WATT).
By strategically averaging the adapted weights derived from multiple text prompt templates, our method aims to harness the complementary strengths of individual templates, resulting in robust and enhanced performance across wide range of domain shifts. To further illustrate this point, Figure 2 presents the test loss and adaptation error surfaces for three models that are separately adapted using three templates (I T T under the Gaussian noise corruption of the CIFAR-IO- - datase ? The central point in these landscapes, representing the final model obtained by averaging the weights of the separate models, demonstrates a convergence towards lower loss and error. highlighting the potential of weight averaging for test-time adaptation. Moreover, inspired by recent advancements in machine learning utilizing train-time weight averaging techniques [10 the proposed WATT method can dynamically adjust to new data to tackle unforeseen distribution shifts without relying on class supervision.
We outline the main contributions of work as follows:
We introduce novel Test-Time Adaptation method for CLIP, which leverages weight averaging across various text templates at test-time.
"To visualize the loss and error surface, we use weight vectors from models adapted with text templates 7 T and T denoted as wo. W1, and w2. Following (9] we define u = wi wo and (we wo) (wo (og==w))(wr -wa) (wi wa). The normalized vectors ù Tull and à = [il form an orthonormal basis in the -wy-wol?
plane of Wo. WT, and Wz. We create a Cartesian grid in this basis and evaluate the networks at each grid point. A point P with coordinates (z, y) in the plane is given by P = wo + x à + y 6. To plot all in the same plane. we used the average of the three templates' text embeddings.

---

Loss
Error (3-Acc)
1.4355
0.4219
111980
0.0929
0.9934
0.3348
0.113
0.3129
0.6712
0.2988
T1
0.0111
T1
0,2857
Figure 2: Loss and Error surfaces model parameters for the Gaussian noise corruption of the CIFAR-1OC dataset. Points 7 TI, and T represent models adapted with different text templates (please see Fig. [aa. The central point (cross) shows the model obtained by averaging these weights, demonstrating improved performance.
Our WATT strategy yields highly competitive performances in comparisons to current TTA methods, and could bring improvement using only a single image at test time, a capability not present in previous approaches.
We rigorously evaluate our WATT methodology through comprehensive evaluations across different datasets characterized by diverse types and degrees domain shifts, encompassing total of 155 evaluation scenarios. Our experiments demonstrate the robustness and efficacy of WATT compared to existing adaptation methods.
2 Related work
Test-Time Adaptation (TTA) is crucial in domain adaptation, particularly with unlabeled target domain data and no access to source domaín samples. The challenge lies in estimating the target domain's 's distribution and comparing domain characteristics indirectly. Recent advancements have highlighted the potential and limitations of adapting pre-trained models. key focus has been on leveraging batch normalization layers for adaptation due to their ability to retain source domain information. Methods such as PTBN [12] and TENT [3 recalibrate batch statisties and optimize affine parameters via entropy minimization, though they often require image augmentations or large batches. MEMO [4] proposes simple approach that does not require multiple test points; it uses single-test-poing data augmentations and minimizes the marginal entropy of the model's s average output to enforce consistency and improve robustness. LAME [5] introduces closed-form optimization strategy that refines model predictions for target images by leveraging the Laplacian of feature maps to encourage clustering, thereby emphasizing feature similarities. another study, SAR [16] addresses TTA instability by using batch-agnostic norms (c.g.. group and layer norms) and a sharpness-aware entropy minimization approach. filters noisy samples and guides the model to flat minima, improving robustness under mixed domain shifts and small batch sizes.
Recently, Tims-TimeeTT Training (TTT) methodologies have emerged as prominent contenders in TTA 17 18 19
20, 21]. This approach involves training a supplementary sub- branch alongside the primary network in an unsupervised manner, subsequently leveraging it to refine the model. Unlike previous methods, our approach operates on individual image batches, offering significant advantage TTA by avoiding necessity of training additional branches from scratch.
In natural language processing, TPT g introduced entropy minimization for adapting models like CLIP, albeit with high computational costs due learning an adapter the prompt with multiple transformations. DifITPT (22) extends this by leveraging pre-trained diffusion models to generate diverse and informative augmented data, combining conventional augmentation methods used in TPT with diffusion hiffusion-generated data to enhance adaptability. It also introduces cosine similarity- based filtration technique for improved prediction fidelity. TDA (3 offers a training-fres -fre approach with dynamic adapter, utilizing lightweight key-value cache and pseudo label refinement, making it computationally efficient. CLIPArTT (24), fine-tunes normalization layers with minimal disruption to the model's knowledge, enhancing text supervision by introducing pseudo labels. Existing methods often lag behind supervised prompt adaptation techniques in performance. SwapPrompt (5] bridges this gap by leveraging self-supervised contrastive learning, employing dual prompt paradigm. In

---

Test Image
7
Test Image
7
Test image
Tw
A photo Kllass
lap
photo the small (class kg
+
weight iterations times]
averaging every L Iterations (M times)
1 Adapted CLIP (dt)
2nd Adapted CLIP (0*)
H Adapted (#")
upptints of of
vil bistoon calloon-nccoooo anccddar ever Weetyl Horaaoon
Sonn Kooi
Lyra(0))
Lyrx(0
Adaptation Phase
Evaluation Phase
7
72
Adapted CLIP
jolass
Templotes
Rap of
Text Encoder
xF
37
12
Flass
Text E mbbeddng Averaging
the
TV
Text Emboddings
Test Image
Averaged Vision
Encoder
Similarity Scores
Final Prediction
Figure 3: Overview of the proposed WATT method. In the Adaptation Phase, the model adapted using different text templates (7 T ....... T with weight averaging performed periodically. In the Evaluation Phase, the adapted CLIP model uses averaged text embeddings from all templates and the weight averaged model to predict the class of the test image.
comparison, our method combines prompt augmentation and fine-tuning of normalization layers, highlighting its effectiveness in test-time adaptation.
Weight Averaging (WA) is a powerful train-time technique for improving deep neural network generalization. Stochastic Weight Averaging (SWA) (26] averages weights of multiple models sampled from different training epochs, aiding smooth optimization trajectory and convergence to points with superior generalization. SWAD [1O] refines SWA by densely sampling weights throughout training, enhancing generalization and robustness across tasks. This train-time refinement enhances WA's effectiveness in producing models with improved generalization. The Lookaround 11 optimizer iterates between an "around step" and an "average step", building on SWAD's advancements. In the "around step". independently trained models using various data augmentations explore a broader loss landscape to find flatter minima. In the "average step," weights of these models then averaged, guiding optimization towards lower loss regions. This method enhances robustness and generalization across tasks, improving upon SWA and SWAD by providing a more effective weight averaging process.
In contrast existing approaches, WATT leverages varied text prompts to adapt vision-language models such as CLIP during testing. Our method also harnesses the benefits of weight averaging while addressing domain shifts without additional model transformations or trainable modules, thereby setting new precedent in test-time adaptation.
3 Method
The proposed WATT method, summarized visually in Figure comprises three main components, the first two in the Adaptation Phase and the third in the Evaluation Phase: a light-weight transductive TTA strategy that adapts CLIP's visual encoder effectively by considering the similarity between all
Figure3comprises

---

Dataset
single
temp
text avg
CIFAR-
90.87 10.10
91.08 10006
CIFAR-10.1
86.80 10.19
86.85 1:0.18
CIFAR-10-C
72.08
72.66
CIFAR-100
69.79 10.20
70.30 1:0.11
CIFAR-100-C
41.79
42.24
Table I: Accuracy (%) with different text ensembles at test time.
batch samples in terms their visual and text features; 2) a weight-averaging strategy using multiple text templates generate diverse model hypotheses during adaptation: 3) an ensembling technique that boosts performance during evaluation by averaging the embedding of different text templates.
3.1 Transductive TTA loss
While our method can be employed with any TTA framework, in this work, we implement a strategy inspired by the transductive TTA approach of CLIPArTT [24] which effectively incorporates semantic relationships among batch samples.
Initially, our process involves executing inference using CLIP, system comprising visual encoder (-) that translates image x into visual features z" E RD, and text encoder f-(-) which converts text prompts into text features z' E R... During inference, we employ pre-defined text prompts assigned to each class within a dataset, such as to "a photo of {class k}". For new image x,, the likelihood of belonging to class k is then computed using cosine similarity:
exp (cos(z); (T)
Pike E, exp cos(z;) 2/)/T)'
cos(2, = || 1
(1)
where r a softmax temperature parameter set to 0.01 this work. This prediction is then stored to be used pseudo labels for the model.
Denoting the normalized visual embeddings of the samples within the batch as Z" IR BxD and the instance-specific text embeddings as Z' E RB we compute an image-to-image similarity ma- trix S Zz(Z") E [-1, 1] Bx modeling pairwise relationships in terms of image characteristics. Similarly, we construct a text-to-text similarity matrix S Z TZ') [,-1, 111,1 capturing the semantic relationships among text embeddings within the batch. Utilizing the computed pairwise
similarity matrices, we generate pseudo labels Q softmax ((S" / [0, 1] Rx which are used with cross-entropy in our transductive TTA loss:
B Σ
CTTA (0) =
46 log Pij-
Drawing a link with the Stochastic Neighbor Embedding (SNE) method for dimensionality reduction (7), which minimizes the KI. divergence between distributions modeling pairwise distances, our TTA loss ensures that the inter-modality (text-to-image) similarities of batch samples aligned with their intra-modality ones (text-to-text and image-to-image).
3.2 Tempiatempllte Weight Averaging
We explore various text prompt templates suggested in the CLIP paper and detailed in Fig. As reported in Table Ic] these prompts achieve varying performance across different corruption types of CIFAR- We formulate prompts of the form t/ template h (class k), where E {1, ......., H.. encompassing a spectrum textual cues tailored to elicit diverse responses from the model.
Two different approaches are investigated for our multi-templaate weight averaging (MTWA) strategy. The first one denoted as Parallel MTWA (WATT-P), which follows recent optimization approaches like Lookaround [II], performs the adaptation separately for each text template, starting from same parameters, and then averages the resulting adapted weights. The second one, called Sequential MTWA (WATT-S), instead considers text templates sequentially without resetting the weights. These two approaches, which we illustrate and compare in Fig. 4 are detailed below.
5

---

Hereatora (here (-2)
7
Ierations (here L:2)
7
70
WIRTT- -P
Weights
T"
WATT P
Initial Weights
rt
Inttaal
Weights
WATT-
T
T
Weights
7
(here Mx2)
(hore M-2)
Figure 4: Visual comparison of the Parallel (left) and Sequential (right) approaches for multi-template weight averaging during adaptation.
Dataset
CLIP
[SS
=S=2
BS 44
BS=8
BS =
=S=32
BS=64
BS = 128
CIFAR-10
88.74
89.87
89.39
+0.0
89.16
#0.77
88.93
10.16
89.14
10.01
89.51
#0.12
90.16
+0.13
91.05 =0.06
CIFAR-10.1
83.25
84.55
84.32
1015
83.88
10
84.
20.37
84.35 +0.21
84.87
10.1
85.52
20.30
86.98 =0.31
CIFAR-IO-C
59.22
61.26
63.60
63.47
63.94
65.66
68.34
71.21
73.82
Table 2: Accuracy (%) of our method for different batch sizes compared to CLIP
Parallel MTWA. This approach optimizes the TTA loss in (2) separately for H different models, each utilizing a distinct template. Starting from the same visual encoder parameters 8, these models updated in parallel for L iterations, resulting in updated parameters 8% with h E {1...., H). The parameters are reset after each update, enabling each model to restart the adaptation from the same initial point. Subsequently, we aggregate the weights obtained from these H models by computing their average: Bavg Σ# 8%, We repeat this step M times, and denote the overall process as "(after L. iter) x M
Sequential MTWA. Our Sequential MTWA approach is inspired from the work of [26]. where the averaging of weights across various stages of the training process is employed to mitigate variance and enhance generalization capabilities. Instead of resetting parameters for each model, we update parameters after each template's iteration. To ensure impartiality in the update sequence of templates, random selection process is implemented, thereby disregarding any predetermined order.
3.3 Evaluation Phase
At test-time, predictions are computed using Equation [ through two distinct methodologies. In the first approach, the text features zi are derived from the initial text prompt to = "a photo of class k", denoted as single temp. Conversely, the second method aggregates the text features from all templates by computing their mean, resulting in the prediction z = Σ z denoted as text_avg (see Fig. 3h.
Experimental Setup
Settings. line with prior TTA methodologies, adjustments are made to all Layer Normalization layers within the visual feature extractor for test-time adaptation. The Adam optimizer is employed with a fixed learning rate of 10 10-3, wheras a smaller learning rate 10 is chosen for adaptation pronounced shift. Throughout experimentation
to the 3D renderings split, it reflects process, a consistent batch size of 128 is maintained to ensure uniformity and facilitate meaningful comparisons across various scenarios.
Datasets. Following 24 we rigorously evaluate WATT's performance across diverse TTA datasets using established assessment techniques. These datasets simulate intricate domain shifts, providing nuanced insights into our approach's 's effectiveness. Additionally, we explore WATT's adaptability on the original dataset through zero-shot test-time adaptation. To ensure thorough examination,

---

70
90
Test-time accuracy
Test-lime accuracy
60
CIFAR10
CIFAR10.1
50-
80
CIFAR1O-C average
CIFAR100
40
Defocus Blur
Frost
Contrast
CIFARICO-C average
70
1
2
D 10
20
50
80
100
Number templates
Number iterations
Figure 5: Evolution of the accuracy for different
Figure 6: Evolution of accuracy on CIFAR- 100 corruptions with the Parallel MTWA method.
numbers of random template on 5 test-time runs.
extend our analysis to include various domain generalization datasets, exposing our method to a broad spectrum of image categories for comprehensive evaluation. Our evaluation framework encompasses natural images, common corruptions, simulated shifts, video shifts, texture shifts and style shifts.
In our assessment of natural image analysis, we include CIFAR-IO, CIFAR-10.1, and CIFAR-I00, each comprising 10,000 images and offering varied data distributions. CIFAR-10.1 [28 introduces a natural shift from CIFAR- 10. providing a comprehensive evaluation of our model's 's performance. We also incorporate the CIFAR- 10-C and CIFAR- --0--- datasets [28), augmented with 15 distinet corruptions across 5 severity levels, resulting 75 common corruption scenarios. This comprehensive augmentation assesses the model's 'sss resilience effectively.
Our investigation also extends the VisDA-C dataset [99, challenging models with simulated and video shifts across diverse imagery types. Additionally, we evaluate our method on three datasets mostly used the field of domain generalization: PACS [30], VLCS (31,, and OfficeHome [32] datasets, instrumental in understanding texture and style variations. These evaluations effectively demonstrate the generalizability of our method across distinct domain shifts.
Benchmarking. We conduct a comparative analysis of WATT against contemporary methods using VIT-B/32 as the backbone. Specifically, we incorporate adapted version of TENT [33 customized for CLIP by the authors [24), using 10 iterations. We also include SAR [16] and MEMO [1] in the same manner as TENT. Additionally, we compare with TPT [8), novel adaptation technique for CLIP that heavily relies on image augmentations, as well as DiffTPT [22], an extension that incorporates diffusion models. Lastly, include TDA (23), which employs dynamic adapter, and CLIPArTT 24, a recent approach that utilizes pseudo labels generated through conformal learning.
5 Results
In this section, we present empirical findings from our WATT method through series of ablation stud- ies aimed at understanding the impacts of individual components. These studies inform subsequent experiments across diverse datasets. Leveraging insights from ablations, we conduct comprehensive experiments, benchmarking WATT against state-of-the-ae techniques across various datasets.
5.1 Ablation Studies
In this section, unless otherwise specified, focus the Sequential MTWA variant of our method (see Section 3.2) and will use these findings as a reference for the Parallel MTWA method.
Comparison of the template used during testing. After updating the model, we proceed to compute the similarity between the image features and the text embeddings, enabling prediction. Typically, text embeddings originate from the text prompt "a photo of class k". However, by employing multiple templates, we have the flexibility to alter this text embedding through the averaging of all text embeddings from each template. Table conducts comparative analysis revealing that this
7

---

averaged text embedding consistently yields superior results across all scenarios. Hence, we adopt this approach for next experiments.
Comparison of the number of templates. In Figure[5] we examine the performance variation relative the number of utilized templates. In this investigation, we conduct runs wherein the templates are randomly selected from a pool of 8 distinct templates (as outlined in Fig. Ta). Notably, when the distribution shift is minimal, as observed in CIFAR-10 and CIFAR-10.1, optimal performance is attained using 6 templates, with performance gradually diminishing thereafter. Conversely, in scenarios characterized by substantial corruptions, such as CIFAR-IO 10-C, employing all 8 templates proves advantageous. Consequently, our focus moving forward will be on utilizing all 8 templates in our work.
Weight avg. (ours)
Dataset
Text avg.
Output avg.
(after 10 iter) 1
(after iter) 10
(after 2 iter) x5
CIFAR-10
90.58 +0.03
90.90 =0.03
91.08 #0.06
91.39 +0.14
91.05 =0.06
CIFAR-10.1
85.78 +0.25
86.77 =0.08
86.85 10.18
88.02 +0.18
86.98 =0.31
CIFAR-10-C
71.41
72.60
72.66
73.66
73.82
CIFAR-100 100
69.46 10.13
70.32 10.1
70.3 10.11
70.85 1.0.08
70.74 10,20
CIFAR-100-C
41.37
42.68
42.24
45.32
45.57
Table 3: Accuracy (%) obtained with different averaging strategies.
Text Averaging Output Averaging Weight Averaging, Utilizing the averaging method within a VLM offers several possibilities, including averaging the weights, the outputs or the text embeddings before computing the logits. In Table 3 comparison between these approaches is presented. It
becomes evident that weight averaging consistently outperforms text embedding averaging across various datasets, showcasing superiority of approximately 1% even with the less effective weight averaging method. This performance advantage is observed across CIFAR- 10, CIFAR-10.1, and CIFAR-10-C, and persists even with larger numbers of classes, such as in CIFAR- 100 and CIFAR- 100-C. When concentrating on output averaging, the results may be less evident with less effective weight averaging methods. However, they remain valid and even more accurate with superior weight averaging techniques. Therefore, our focus for future experiments will be on weight averaging as the preferred approach.
Best moment to do the Weight Averaging. Examining Table 3 it is evident that the parameters L and M discussed in Section 3 are crucial. Specifically, large L (e.g.. 10) combined with small M (c.g., 1) is ineffective. Conversely, setting L = 1 and M = 10 yields optimal results for small distribution shift datasets, while T, = 2 and M perform best highly corrupted datasets. Given that TTA typically encounters substantial distribution shifts, we will use I. = 2 and M = 5 in our subsequent experiments.
Performance over the number of iterations. In this section, we focus on the method incorporating a Parallel MTWA mechanism and examine the impact of the number of iterations on performance. As illustrated in Figure 6, the accuracy stabilizes after approximately 20 iterations. Although there is slight improvement in performance beyond 50 iterations, the difference is marginal. Based these observations, we have opted to use 50 iterations for our experiments.
Model performance across various batch sizes. In our investigation, we delve into the performance implications of TTA methods when operating under small batch sizes, historical challenge in the field. Table 2 provides insights into this aspect, revealing substantial performance enhancements with increasing batch sizes. Notably, our WATT model showcases remarkable adaptability, demon- strating performance improvements even with single image input contrary to alternative methods. Specifically, we observe enhancements of approximately 1% for CIFAR-10 and CIFAR-I0. and an impressive 2% for CIFAR- when compared to baseline. Moving forward, we maintain batch size of 128 in our experiments, aligning with prevalent practices observed in contemporary state-of-the-un methodologies.
8

---

Dataset
CLIP
TENT
TTT
TDA
DITTPT
SAR
CLIPAITT
WATT-P
WATT-S
CIFAR-10
88.74
91.69 caw
88.06 -uss
84.09 t0.e
83.07 =o1e
89.05 =waz
90.04 =aLs
91.41 +oUr
91,05 =wos
CIFAR-10.1
83.25
87
81.80 4027
78.5 98 +03
76.50
83.65 =004
86.35
87.78 :M
86,98
CIBAR-10-C
59.22
67.56
56.80
48.00
56.77
60.45
71.17
72.83
73,82
CIFAR-100
61.68
69.74
1816
63.78
107s
60.32
Lote
52.80
64,44
69.79
70.38 1014
70.74 1,:30
Gaussian Noise
14,80
14.38 s:ss
14.03 20.10
8.20 10.25
21.40 =:0
15.85 2023
25.32 =014
31.28
32.07 21.23
Shot noise
16.03
17.34
#027
15.25 +0.?
9.58 +0.4
24.17
17.41 =0:25
27.90 =100
33.44 =0.11
34.36 =0LL
Impulse Noise
13.85
10.03
13.01
7.63
1
16.87
14.90
25.62
29,40
gm11
30.33
Defocus blur
36.74
49.05
2our
37.60 20177
25.59 10.4
20.30 2.20
42,00 2003
49.88 gma
52.32 1:23
52.99 1111
Glass blur
14.19
3.71 +oor
16.41 +0002
9.83 +0.5
15.57 =t4e
13.84 =1.05
27.89 =toss
31,20 =0.2
32.15 =0,00
CIFAR- CIFAR-100-C 100-C
Motion blur
36.14
46.62
37.52
28.92
21,00
39.52
47.93
49.72 2:15
50.53 2012
Zoom blur
40.24
51.84
gavs
42.99
1ui
31.08 1076
25.53 Lour
45.40 _inos
52,70 _ou
54.72 1o00
55,30 10:2
Snow
38.95
46.71 #a2
42.35 +4.13
32.94 +a.1
28.83 =a3t
41.85 -4L0s
49.72 =ton
51.79 =om
52.77 #0LL3
Frost
40.56
44,90
43.31
34.84
31.10
42.20
49,63 =6:2
53.04
53.79
Fog
38.00
47.31 goos
38.81
1017
31.13 1
16.60 jevi
40.14 L,n00
48,77 _en
50.78 Loai
51.49 1n21
Brightness
48.18
60.58
50.23
+a.1
42.36 +a.1
38.13 =02>
52.77 =0.0
61.27 =tuB
62.65 #:25
63.57 21.21
Contrast
29.53
45.90
28.09
=000
18.03
7,70
34,40
48.55 =03+
51.34
S2.76
Hlastic transform
26.33
33.09
200
28.12
2015
18.88
1014
21.60 ges
2X
2007
37.45 _/v
39.97 1o0.
40,90 g00
Pixelate
21,98
26.47
20.43
14.59
1::30
22.83 2831
22.91 2war
33.88 2014
39.59
2:10
40.97 2144
JPEG compression
25.91
29.89
28.82
+0.0
17.56 +0.11
31.77
+045
27.20
=0:0
36.07
=003
38.99
=3.66
39,59
Meun
29.43
35.19
30.46
22.08
22.89
31,92
41.51
44.68
45.57
Table 4: Accuracy (%) CIFAR-10 10, CIFAR- 10.1,
LIFAR-10-C. CIFAR-100, and CIFAR- datasets. WATT-P refers to our method with Parallel MTWA and WATT-S to the Sequential MTWA variant of WATT.
5.2 Comparison to SOTA methods
Performance evaluation with small batches. In existing TTA works that study small batch size scenarios, such as SAR [16] and MEMO (14), we implement these methods in the context of CLIP. We compare our method with other TTA approaches using a batch size of 1. As shown in Table 5 WATT-P achieves the highest accuracy across all cases, ouiperforming SAR by 2.23% and MEMO by O. 0.75% on CIFAR- Notably, this improvement achieved without any image augmentation, which is a common practice in previous TTA approaches that deal with small batches.
Dataset
CLIP
TPT
SAR
MEMO
CLIPAITT
WATT-P
CIFAR-I0
88.74
88.29
87.41
89.29
88.76
89.87
CIFAR 10.1
83.25
82.85
82.32
83.80
83.15
84.55
CIFAR-IO-C
59.22
59.03
58.70
61.15
59.18
61.26
Table 5: Comparison of different TTA methods with batch size equal to 1.
Performance evaluation in the presence of natural or no domain shift. In Table 4 results show consistent performance enhancements with WATT, both with the Parallel and Sequential MTWA strategies, alongside the baseline. On CIFAR- 10, performance improves by 2.67% with Parallel MTWA and 2.31 % using Sequential MTWA. On 10., improvements reach 4.53% and 3.73%, and on CIFAR-100 100, enhancements are 8. 8.70% and 9.06%. While WATT consistently outperforms the baseline, TPT, and CLIPArTT, TENT yields superior results on CIFAR-10. 10. WATT's effectiveness often correlates with the number classes, showing better performance with more classes, indicating its strength in lower-confidence scenarios.
Performance evaluation in the presence of common corruptions. Table 4 shows that both WATT variants consistently outperform alternative methods across various corruptions and class numbers. Notably, WATT with Parallel MTWA improves performance by 16.48% on CIFAR-100 Gaussian Noise and by 17.01% % on Glass Blur compared to the baseline, while WATT with Sequential MTWA shows improvements of 17.27% and 17.96% respectively. On common corruptions, the Sequential MTWA variant surpasses Parallel MTWA, with improvements of 0.99% on CIFAR-10 10 and 0.89% on CIFAR- 100. According to the TDA supplementary materials, we selected the weighting factor alpha as 5.0 and the sharpness ratio beta as 2.0, which are stated as optimal. However, these values did not appear to be the best choice for more challenging datasets like CIFAR-10-C or CIFAR- that contain various corruptions. Adjusting these parameters based on the dataset would not be consistent

---

Dataset
Domain
CLIIP
TENT
TPT
CLIPArTT
WATT-P
WATT-S
3D (trainset)
84.43
84,86 40.01
79.35
85.09
85.42 0003
85.36 :0:1
VisDA-C
(valset)
84.45
84.68 40.01
83.57 2004
84.40 =001
84.57 =0.00
84.69 10.01
Mean
84.44
84.77
81.46
84.75
85.00
85.03
Art
73.75
74.03
+0.27
75.76
=0.27
73.84 =0.20
75.65 =0.27
75.76 +:039
Clipart
63.33
63.42 10.04
63.08
63.54
66.23 00.3
65.77 4:.11
OfficeHome
Product
85.32
85.51 10.08
84.07 20.28
85.23 =
85.41 =0.09
85.41
10.01
Real World
87.71
87.74 +0.05
85,89 +:.33
87.61 =0.05
88.22 =0.15
88.37 +:0.5
Mean
77.53
77.68
77.20
77.56
78.88
78.83
Art
96.34
96.65
10.05
95.52
96.57 =0099
96.31 =0.00
96.39 1:000
Cartoon
96.08
96.22 20.0
94.77 +0.20
96.00 =000
96.52 =002
96.62 100002
PACS
Photo
99.34
99.40 +0.00
99.42 1:006
99.28 =0000
99, 99,48 =0.03
99.52 1:000
Sketch
82.85
82,96
20.12
83.22 +0.14
83.93
86.92
86.65 10.12
Mean
93.65
93.81
93.23
93.95
94.81
94.80
Caltech101
99.51
99.51
1:000
99.36
10.06
99.51 00000
99.43 0000
99.51
1:000
VLCS
LabelMe
68.15
67.89
20.13
54.88
+0.12
67.96 0004
66,65 =0.1
68.49
+:0.12
SUN09
68.85
69.27
20.4
67.30
+0.49
68.66
=009
72.61 =0.15
73.13 10.17
VOC2007
84.13
84.42 10.15
76.74 10.28
84.09 00.02
82.30 =0.16
83.41 10.17
Mean
80.16
80.27
74.57
80.06
80.25
81.14
Table Accuracy (%) on different domains of VisDA-C, OfficeHome, PACS and VLCS datasets.
with the principles of a fully TTA method, which might explain their suboptimal performance in results. Regarding DiffTPT, it generates 64 images per test image, making it challenging to use in a real-world TTA scenario. Similar to TDA, DiffTPT requires carefully chosen parameters to fit the dataset, whereas our method does not require dataset-specific tuning. This highlights the robustness and practicality of our approach in diverse real-world applications.
Performance analysis under simulated and video shifts. Results on the 3D (simulated shift) and YT (video shift) splits of VisDA-C demonstrate significant improvement in accuracy with our proposed WATT method compared to pure CLIP. The Sequential MTWA variant achieves the highest accuracy both the 3D and YT splits, with scores of 85.36% and 84.69%, respectively, surpassing other adaptation methods including TENT, TPT, and CLIPAITT (see Table 6
Performance analysis under texture and style shifts. Results on the OfficeHome, PACS, and VLCS datasets are presented in Table 6 On average, our proposed WATT method, with Parallel and Sequential MTWA variants, improves performance across the different domains of OfficeHome, PACS, and VLCS compared other methods. This highlights its robustness addressing texture and style shifts, which are especially challenging compared to other domain shift variants.
Conclusion
We introduce WATT, a Test-Time Adaptation method tailored for Vision-Language Models. Our approach harnesses Weight Averaging with different text prompts and incorporates text embeddings averaging to bolster prediction accuracy.
Through an extensive ablation study, we scrutinized the efficacy of employing varied text prompts and weight averaging. Comparative evaluations across Test-Time Adaptation and Domain Generalization datascts underscored the superiority of our method, particularly in scenarios involving distribution shifts and zero-shot performance enhancements compared to state-of-the-a approaches.
Looking forward, investigating the potential of text prompts and weight averaging in classification opens up promising avenues for future exploration. Our methodology, with focus on template manipulation, suggests potential avenues for extension, such as incorporating alternative class descriptors, yielding valuable insights future research. Moreover, expanding Test-Time Adaptation to encompass diverse scenarios, including segmentation object detection with Vision- Language Models, holds significant potential for advancing our comprehension of model adaptability and performance across varied tasks.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[2] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- -ainnnla representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916 PMLR, 2021.
[3] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yo Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.026443 2023.
[4] Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pages 4088-40..50 Springer, 2022.
[5] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[6] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152-21164 2023.
[7] Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pscudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16155-16165, October 2023.
[8] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 14274-14289, Curran Associates, Inc., 2022.
[9] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
and R. Garnett, editors, Advances in Neural
[10] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22415 2021.
[11] Jiangtao Zhang, Shunyu Liu, Jie Song, Tongtian Zhu, Zhengqi Xu, and Mingli Song. Lookaround optimizer: & steps around, 1 step average. Advances in Neural Information Processing Systems, 36, 2024.
[12] Zachary Nado, Shreyas Padhy, D. Sculley, Alexander Amoor, Balaji Lakshminarayanar and Jasper Snock. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006. 10963 [cs, stat), January 2021. arXiv: 10963.
[13] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. 2021.
[14] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo:
Test time robustness via adaptation and augmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 38629-38642 Curran Associates, Inc., 2022.

---

[15] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-fiee free online test-time adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8344 8353, 2022.
[16] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world, 2023.
[17] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with sulf--uue for generalization under distribution shifts. In International Conference Muchine Leurning (ICML), 2020.
[18] David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikkani,, Ismail Ayed, and Christian Desrosiers. Tuflow: Unsupervised test-time training with normalizing flow. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2125-2126, Los Alamitos, CA, USA, jan 2023. IEEE Computer Society.
[19] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros. Test-time training with masked autoencoders. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[20] Gustavo A Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Ismail Ben Ayed, and Christian Desrosiers, Clust3: Information invariant test-time training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6136-6145, 2023.
[21] David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Nc-ttt: A noise constrastive approach for test-time training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6078-6086, June 2024.
[22] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmen- tation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2704-2707771 2714,, October 2023.
[23] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision- lisgun-la models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14162-1411 June 2024.
[24] Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Light-weight adaptation of clip to new domains at test time. arXiv preprint arXiv:2405.00 00754, 2024.
[25] Xiaosong Ma, Jie ZHANG, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-languee models. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023.
[26] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint 1rXiv:1803.05407 2018.
[27] Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
[28) Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. 2019.
(29] Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018
[30] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
12

---

[31] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference Computer Vision, pages 1657-1664, 2013.
[32] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
13

---

WATT: Weight Average Test-Time Adaption of CLIP Supplementary Material
A
Implementation
Our proposed WATT method is implemented in Python using the PyTorch (version 2.0.1) framework. All experiments were conducted on an NVIDIA V100 32 GB GPU. However, due to the effectively and lightweight nature of our method, it can be executed on less powerful GPUs. Specifically, the adaptation with batch size of 128 using the ViT/B32 backbone requires up to 4 GB of memory, making feasible to use on a wide range of GPUs. For fairness and consistency, we re-implemented and ran all other methods, including CLIPArTT, TENT, and TPT, in the same environment. Each experiment was performed three times to ensure reliability (three trials per experiment). To facilitate reproducibility, provide the original implementation and detailed step-by- instructions in our repository, accessible via this github link
B Template details
The CLIP paper identifies 80 templates that enhance model robustness and performance. They ultimately conclude that of these templates best summarize their model (see this link). our work, use these 7 generic templates and add the common one, "a photo of O,," based their optimization. These templates are not specifically linked to the content of the images.
C Dataset details
In our investigation, shifts: the simulated shift and the video shift. The simulated shift includes 152,397 3D-rendered images across 12 diverse classes, while the video shift comprises 72,372 YouTube video frames spanning the same categories. This dataset addresses the diversity of imagery types applicable to a model, posing significant challenge.
the VisDA-C dataset, which challenges models with two distinct domain
Moreover, we evaluate our proposed method on three other datasets: PACS, VLCS, and OfficeHome. These datasets help understand various domain shifts, including texture and style variations. The PACS dataset consists of 9,991 images across four domains (Art, Cartoons, Photos, Sketches) and seven classes. The VLCS dataset contains 10,729 images across four domains (Caltech101, LabelMe, SUN09, VOC2007) and five classes. Lastly, the OfficeHome dataset includes 15,588 images across four domains (Art, Clipart, Product, Real) and 65 classes. Evaluating across these distinct domain shifts showcases the generalizability of our method. These datasets are more representative of real-world scenarios compared to CIFAR, with complex domain shifts.
D Computational Cost
We conduct a thorough evaluation under consistent conditions using an NVIDIA A6000 GPU within the same Python environment. The Table 7 compares the adaptation time, memory usage, and the number of learnable parameters for various TTA methods, including our proposed WATT method. The table clearly demonstrates that WATT-S, a sequential implementation of WATT, maintains competitive adaptation time and memory usage compared to other methods like TENT and ClipArTT, which are efficient but lack the robustness of WATT's method. Additionally, the table highlights that WATT-P, with parallel model training, offers a faster adaptation time than WATT-P with a for-loop implementation, albeit the cost of higher memory usage. It's important to note that methods like DiffTPT [22] and MEMO [4], which show significantly higher adaptation times, employ off- the-shelf diffusion models and AugMix augmentation, respectively, resulting in time-consuming processes that may be impractical for real-world scenarios. In contrast, the effectiveness of our WATT-S method makes better suited for scenarios where a robust, rapid, and resource-efficient adaptation is crucial.
14

---

Method
Adaptation Time
Memory
Percentage of Learnable Parameters
WATT-S
2.34
1.5
0.026%
WATT-P
23.2 s (23.2/8)
1.5 GB (8 1.5 GB)
0.026% (x8)
TENT
0.28
1.5
0.026%
ClipArTT
0.55
1.7 GB
0.026%
SAR
0,42
1.4 GB
0.026%
MEMO
165
2 GB
0.026%
DifTTPT
8.2 + 0.26 s
8.7 GB + 1.7 GB
0.001 %
Table 7: Comparison of computational cost of different methods.
E Pseudo-code of our both methods
compare the two variants of WATT: one with Parallel MTWA (WATT-P) and the other with Sequential MTWA (WATT-S). The WATT-P model recalibrates its parameters for each template using the average parameters of m 1, whereas the WATT-S model updates its parameters solcly at the start each new iteration m.
In Algorithms]] and 2
Algorithm WATT-P model f. parameter
Algorithm 2 WATT-S model f. parameter
1: for m {1,2,... .M do
1: for m E /1, do
2: Oavg
-h== 0z
2: Davg t- H DH On
3: for E 11,2 H) do
3: f f0nv
4:
{ sews
4:
for {1,2, 2,...... do
5:
for E (1,2.... L} do
5: for E 21, L} do
6: 0 + L-LTTA (template,))
6:
0n + LITA (fo (fe (template,))
7: end for
7: end for
8: end for
8: end for
9:
end for
9:
end for
F Additional Ablation Studies
Cross Entropy vs Entropy Minimization. Two unsupervised loss functions were integrated into previous TTA methods: classical entropy minimization and the loss introduced by CLIPArTT (24). where predictions are utilized as pseudo labels for cross-entropy computation. In Table g com- parison between these loss functions is presented across the original CIFAR-10 dataset and various corruptions from CIFAR-1O-C. It is observed that, for these specific corruptions, entropy minimization generally outperforms with the different templates employed, except for Gaussian Noise. However, upon assessing the weighted average accuracy, computed after 10 iterations for each template, cross-entropy consistently emerges as the superior option. The marginal impact the weighted average on entropy minimization suggests that, irrespective of the template used, the model updates in a consistent direction to enhance confidence, rendering cross-entropy the preferred choice for subsequent experiments.
15

---

Datasel
Loss
7
7
72
70
7
73
7
77
WA
TENT
91.69
91.97
91.69
90.28
91.16
92.11
91.98
89.14
90.60
CIFAR-10
89.8
90.37
90.5
88.42
89.93
89.95
90.13
88.54
91.05
TENT
41.27
37.16
46.39
51.31
39.27
32.51
49.7
42.96
47.08
Gaussian Noise
CE
60.19
61.01
61.17
58.24
58.84
58.35
59.62
61.13
63.84
TENT
77.12
77.13
78.7
76.09
76.85
76.59
77.86
74.31
76.21
Defocus Blur
CE
77.23
77.07
78
75.98
76.39
77.45
77.08
73.59
78.94
TENT
78.29
79.54
80.09
75.39
78.97
78.52
78.78
75.82
77.24
Snow
CE
76.57
77.36
77.93
75.08
77.45
77.09
77.05
75.57
79.79
TENT
62.64
65.83
64.27
59.49
62.78
64.19
62.62
63.39
65.31
JPEG Compression
CE
64.65
65.36
65.24
64.16
64.18
64.36
64.78
65.32
67.36
Table 8: Comparison of accuracy (%) using entropy minimization (TENT) or cross-entropy (CE) on CIFAR-10 10 and some corruptions of LLFAR-10-C datasets with ViT-B/32 encoder on different templates (please see Fig, la and the weight average.
G Experiments with another VLM
We investigate if our TTA method working with other VLMs like SigLip, so we compare if after adaptation with method the performance improves compared to the baseline in Table
Dataset
SigLip
WATT-S
CIFAR-10
66.35
75.02 +0.05
CIFAR-10.1
57.30
65.87 #0.21
CIFAR-10-C
37.52
45.29 20.13
CIFAR-100
33.97
65.87 20.21
C10AR-100-C
14.43
20.05 +0.05
Table 9: Performance comparison of SigLip and WATT-S on different datasets
H Experiments on other Visual Encoders
We replicated the experiments from the main paper using alternative visual encoders, ViT-B/16 and ViT-L/14.
Dataset
Backbone
CLIP
TENT
TPT (BS-32)
CLIPArTT
WATT-P
WATT-S
VIT-B/16
89.25
92.75 10.17
89.80
92.61
92.31 10.10
91.97 10.0
CIFAR-10
ViT-L/14
95.36
96.13
+0.01
95.18 +0.02
95.16 #0.03
95.91 =0.10
95.71 +0.03
VIT-B/16
84.00
88.52
10.33
83.75 10.21
88.72 10.33
87.90
10.11
88.10 10.08
CIFAR-10.1
ViT-L/14
91.20
92.22 +0.25
91.32 +0.12
91.02 #0.02
92.97
=0.13
92.10 +0.33
ViT-B/16
60.15
68.00
59.75
73.22
75.04
76.22
CIFAR-IO-C
VIT-L/14
76.04
79.18
75.01
78.06
80.05
80.06
CIFAR-100
VIT-B/16
64.76
+0.14 78.03
71.73
67.15
+0.23
71.34 #0.07
72.98
=0007
72.85
15.15
VIT-L/14
73.28
10.08
76.85
10.06
79.42 1.008
79.33
10.05
78.85
1.0119
ViT-B/16
32.01
37.90
33.73
40.08
47.86 54.10
48.95
CIFAR-100-C
ViT-L/14
44.59
50.14
47.58
52.52
54.34
Table 10: Accuracy (%) on CIFAR-10, CIFAR-10.J, CIFAR-10-C, CIFAR-100 and CIFAR-100-C datasets with ViT-B/16 and ViT-L/14 as visual encoders.
Performance evaluation in the presence of natural or no domain shift. As indicated in the main results, employing WATT, both with Parallel and Sequential MTWA, consistently enhances
16

---

performance alongside the baseline. This pattern persists across different visual encoders, as shown Tables IO Although WATT consistently outperforms the baseline and TPT. TENT and CLIPArTT may occasionally yield superior results depending on the visual encoder used.
Performance evaluation in the presence of common corruptions. Table T0 demonstrates consistent trend where both WATT methods consistently outperform alternative methods across various corruptions and class numbers. Upon closer examination of Table TO. specifically with ViT-B/16 as the visual encoder, Sequential MTWA exhibits a significant performance advantage, surpassing the baseline by 16.07% and the leading method in the state-of-the-ar by 3.00%. This trend becomes even more pronounced with an increased number of classes, where Sequential MTWA surpasses the baseline and CLIPArTT by 16. 94% and 8.87%, respectively.
Performance analysis under simulated and video shifts. Our study reveals substantial accuracy improvements the 3D (simulated shift) and YT (video shift) partitions of VisDA-C when employing different backbones. This enhancement is particularly notable with our proposed WATT method compared to pure CLIP. Notably, the WATT-S variant achieves the highest accuracy across both the 3D and YT partitions, outperforming various adaptation approaches including TENT, TPT, and CLIPArTT, Detailed comparisons can be found in Tables II and 12
Performance analysis under texture and style shifts. Findings across the OfficeHome, PACS, and VLCS datasets are detailed in Tables [ and 12 Across these varied domains, our WATT method demonstrates consistent performance enhancements, as evidenced by both its WATT-P and WATT-S variants. These improvements underscore the efficacy of our approach in mitigating the complexities of texture and style shifts, which pose particular challenges compared to other forms of domain shift.
Dataset
Domain
CLIP
TENT
TPT
CLIPAITT
WATT-P
WATT-S
3D (trainset)
87.16
87.57 10001
84.04 1003
87.58 10.00
87.61 =001
87.72 10002
VisDA-C
YT (valset)
86.61
86.81 10.00
85,90
86.60
86.66 0.00
86.75 10.04
Mean
86.89
87.19
84.97
87.09
87.14
87.24
Art
79.30
79.26 +0.14
81.97 +0.17
79.34 20.05
80.37 =0.25
80.43 +0000
Clipart
65.15
65.66 10.05
67.01 1:021
65.69 10.11
68.59 0.13
68. 26 1:0.11
OfficeHome
Product
87.34
87.49 40.02
89.00
87.35
=007
88.15 =0.07
88,02 +0008
Real World
89.31
89.50 +0.04
89.66
+0.06
89.29 20.03
90.18 00.03
90.1 14 +0.06
Mean
80.28
80.47
81.91
80.42
81.82
81.71
Art
97.44
97.54
1.002
95.10
10.41
97.64
10.02
97.49
0.008
97.66 1:0.08
Cartoon
97.38
97.37 20.04
91.42
97.37 -002
97.47
=0.04
97.51
10002
PACS
Photo
99.58
99.58
10.00
98.56
10.40
99.58
10000
99.58
0.00
99.58 1:0.0
Sketch
86.06
86.37
1.005
87.23 1:006
86.79
10.04
89.73 0.16
89.56
1:0.14
Mean
95.12
95.22
93.08
95.35
96.07
96.08
Caltech 101
99.43
99.43
10.00
97.62
10.12
99.43 -0.00
99.36 =0.00
99.36 1:000
VLCS
LabelMe
67.75
67.31 :1.14
49.77
10003
67.74 10010
67.55
0.39
68,59
:10.25
SUN09
71.74
71.57
20.15
71.56
+0 86
71.67 20.01
74.75 =0.07
75.16 +-012
VOC2007
84.90
85.10 10.11
71.17 10.70
84.73 10088
82.53 0.10
83.24 1:0.05
Mean
80.96
80.85
72.53
80.89
81.05
81.59
Table 11: Accuracy (%) different domains of VisDA-C, OfficeHome, PACS and VLCS datasets with ViT-B/16 as visual encoder.
17

---

Dataset
Domain
CLIP
TENT
TPT
CLIPATTT
WATT-S
3D (trainset)
91.24
91.40 #0011
90.65 10.00
91.34 +0.00
91.71 1:000
VisDA-C
YT (valset)
85.62
85.77 #0.01
85,41 +0.06
85.61 +0.01
86.80 =0001
Mean
88.43
88.59
88.03
88.48
89.26
Art
82.47
82.61
10.15
86.76 100266
82.35 10.19
84.43 1.0.20
Clipart
72.20
72.51 #0.03
74.76 =0.07
72.41 #0.06
75.43 +0008
OfficeHome
Product
90.94
90.97 =0.02
92.42 =0007
90.94 +0.06
91.88 +0.05
Real World
92.72
92.75 =0.02
92.95 =0.16
92.63 +0.03
94.06 +0002
Mean
84.58
84.71
86.72
84.58
86.45
Art
98.68
98.83
=0.00
94.82
98.76
98,68 1:0000
Cartoon
97.74
97.74 #0.00
95.65 +0.19
97.74 +0.00
97.90 +:0002
PACS
Photo
99.54
99.54
+0.03
99.44 +0003
99.54 +0.00
99.64 1:000
Sketch
93.28
93.51 #0.04
92.72 10.11
93.26 +0002
93.80 10002
Mean
97.31
97.41
95.66
97.33
97.51
C0
99.43
99.43
+0.00
97.86
99.43
99.51 1:000
LabelMe
69.22
69.07 20.12
52.54 10020
69.32 1:015
62.76 10.13
VL.CS
SUN09
68.06
68.23 10:0
69.49
67.89 10.07
10.07
72.21 10.15
VOC2007
83.99
84.08 #0.15
76.16
+0.63
83.89 -0.13
83.02 +0012
Mean
80.18
80.20
74.01
80.13
79.38
Table 12: Accuracy (%) on different domains of VisDA-C, OfficeHome, PACS and VLCS datasets with ViT-L/14 as visual encoder.
I Detailed Experimental Findings
This section provides extensive tables with detailed information on the results, which were summa- rized in the main body of the paper.
Dataset
single dinglettemp
text_ane avg
Dataset
single_teng
text_avg
CIFAR-10
90.87 #0.10
91.08 14.06
CIFAR-100
69.79 =0.20
70.30 4:011
CIFAR-10.1
86.80 20.19
86.85 +:0.88
Gaussian Noise
27.17 20.22
28.08 10.21
Gaussian Noise
61.20 10.05
62.09 11.15
Shot noise
29.69 =0.20
30.47
+0.19
Shot noise
63.16 +0.09
63.51 ::4033
Impulse Noise
25.28 =0.10
26.37 +0.28
Impulse Noise
55.29 40.22
56.04 11.166
Defocus blur
49.83 #0.11
50.52
+0.04
Defocus blur
78.03 +0.12
78.66 ::0.07
Glass blur
27.83 10.03
28.25 1.0.06
Glass blur
62.7 :0.24
63.35 ::0.25
Motion blur
47.77 10.21
47.89 10.21
CIFAR- C
Motion blur
76.33 #0.11
76.96 1:0.14
CIFAR-100-G
Zoom blur
52.90 10.16
53.05 1.010
Zoom blur
78.29 #0.05
79.08 :10.15
Snow
50.31 10.03
50.22 10.17
Snow
78.65 40.21
78.95 14.05
Frost
50.79 +0.07
51.08 +-009
Frost
79.49 20.11
79.95 14.06
Fog
48.70
48.48
Fog
77.21 2003
77.72 14.09
Brightness
61.22 +0.06
61,56 4-0.16
Brightness
86.60 10077
86.98
Contrast
47.87 20.17
47.90 40.14
Contrast
79.22 #0.01
79.62 110.44
Elastic transform
71.17 a0255
71.61 10.09
Elastic transform
37.55 #0.13
37.93 +0.19
Pixelate
67.59 10.08
68.70 10.15
Pixelate
33.81 -0.06
34.56
JPEG compression
66.26 40.00
66.72 10011
JPEG compression
36.09 10.13
37.30 10.18
Mean
72.08
72.66
Mean
41.79
42.24
Table 13: Accuracy (%) on CIFAR-10. 10, CIFAR- 10.1 and CIFAR-10-C datasets with different text ensemble at test time. (WA after 10 iter)
Table 14: Accuracy (%) on CIFAR-100 and CIFAR-100 100-C datasets with different text semble at test time. (WA after 10 iter)
18

---

Weight avg. (ours)
Dataset
Text avg.
Output avg.
(after 10 iter)
(after iter)x 10
(after 2 iter)
CIFAR-100
69.46 +0.13
70.32 10.10
70.3 #0.11
70.85 00008
70.74 20020
Gaussian Noise
27.67
28.58
28.08 =0:11
31.67 10.10
32.07
Shot noise
30.18 10.06
31.05 1:013
30.47 =0.19
34.26 =0228
34.36 =0.11
Impulse Noise
25.79 +0.02
26.86 +0007
26.37 00228
30.12 #0.12
30.33 00.0
Defocus blur
49.51 +0.04
51.04 +0002
50.52 20.14
52.76 20225
52.99 10116
Glass blur
27.88 +0.22
28.72 1:000
28.25 0.06
31.95 2008
32.15 00.3
Motion blur
46.68 40.05
48.30
47.89
50.46 10.10
50.53
CIFAR-100-
Zoom blur
52.05 10.07
53.72 #:011
$3.05 00.0
55.13 00229
55,30 0.22
Snow
49.40 10.18
51.01 10.13
50.22 =0.17
52.60 20:26
52.77 10.15
Frost
49.68 +0.04
51,50 +0.06
51.08 =0,09
53,30 #0,21
53.79 =0,31
Fog
47.36 +0.17
48.67 +0022
48.48 0021
51.35 #0.08
51.49 00.1
Brightness
60.42
61.74
61.56
63.23 00.12
63.57
Contrast
46.86 10.05
48.14 +0.10
47.90 =0.14
52.40 20.23
52.76 00227
Elastic transform
37.00 +0.37
+0.37
38.55 +0.23
37.93 10.99
40.97 20.11
40,90 00.3
Pixelate
33.65 +0.12
34.63 +0.17
34.56 20.4
40.32 20.88
40.97 10.16
JPEG compression
36.38 +0.11
37.67
20023
37.30 10.18
39.35 10.19
39.59 0008
Mean
41.37
42.68
42.24
45.32
45.57
Table 15: Accuracy (%)
CIFAR-100 and CIFAR-1000 100-C datasets with different averaging
Weight avg. (ours)
Dataset
Text avg.
Output avg.
(after 10 iter)
(after iter) 10
(after 2 iter)
CIFAR-10
90.58 40.03
90.90 4:0003
91.08 6000
91.39 20.14
91.05 10.06
CIFAR-10.1
85.78 +0.25
86.77 10.08
86.85 =0.18
88.02 20.18
86.98 =0.31
Gaussian Noise
61.23 +0.13
62.22 =0.12
62.09 =0.15
63.42 =0.07
63.84 =0.24
Shot noise
62.88 +0.15
63,98 +0.17
63.51 =0.03
64.93 #0.13
65.28
Impulse Noise
54.71
56.41 10.11
56.04 -0.16
58.37 10.37
58.64
Defocus blur
77.93 40.12
78.63 10.18
78.66 =0.07
79.11 10.17
78.94 20.12
Glass blur
62.37 4:0.88
63.32 +0.07
63.35 0.25
64.67 #0.18
65.12 00007
Motion blur
75.55 1:0.19
76.97 +.0.5
76.96 10.14
77.56 #0.12
77.81 00.14
CIFAR-10-C
Zoom blur
77.86 +4.06
78.90 1:.18
79.08 10.1
79.76 1003
79.32 =0,77
Snow
77.77 10.03
78.92 +:
78.95 -0.5
79.89 10.26
79.79 -0.06
Frost
78.51 4-0.99
79.67 4-0.09
79.95 0.06
80.52 1004
80,54 0.12
Fog
76.04 +0.17
77.54 +.0.00
77.72 0.09
78.44 #0221
78.53 0.22
Brightness
86,08 +0.13
86.75 +0004
86,98 =0.66
87.32 0014
87. 37.11 =0.11
Contrast
77.87 +0.02
79.48 +0.07
79.62 =0.0
80.77 10.35
81.20
Elastic transform
69.98
71.20
71.61 -0.09
72.52 10.19
72.66
Pixelate
66.78 40.29
68.27 40.17
68.77 00.5
70.50 20.20
71.11 2011
JPEG compression
65.62 +:0.8
66.78 +:0.08
66.72 00.01
67.05 #0.10
67.36 00.25
Mean
71.41
72.60
72.66
73.66
73.82
Table 16: Accuracy (%) on CIFAR-10, CIFAR-10. and CIFAR- 10-C datasets with different averaging
19

---

Darasct
CLIP
BS =
BS=2
BS =
BS = 16
=S
BS = 64
BS = 128
CIFAR-I0
88.74
89.87
89.39
1002
89.16
1007
88,93
10.16
89.14
10.04
89.51
10.12
90.16 10.13
91.05 1.006
CIFAR-10.1
83.25
84.55
84.32
10.15
83.88
84.12
20:37
84.35
10.21
84.87
10.16
85.52
86,98 1031
Gaussian Noise
35.27
38.55
43.85
1:026
45.41
10.10
47.95
10.15
51.79
1.0.27
56.35
10.11
60.87
1:0.3
63.84 1:024
Shot noise
39.67
42.57
46.87
1025
47.95
49.13
20.14
52.57
56.96
20.10
61.84
65.28 20:1
Impulse Noise
42.61
42.92
47.94
1029
48.20
10.18
48.69
10.11
50.53
10.18
53.32
10.19
55.81
10.11
58.64
10
Defocus blur
69.76
72.29
72.80
1:013
72.95
10.13
72.57
1:000
73.71
10.18
75.28
10.18
77.37
10.08
78.94
1.012
Glass blur
42.40
44.15
48.15
10.15
47.69
+000
48.96
=0044
52.59
+0.19
57.83
+0.24
62.16
+0.20
65.12 +0007
CIFAR 10-C
Motion blur
63.97
66.37
67.53
67.22
68.00
69.20
71.60
Zoom blur
69.83
71.50
72.60
10.14
72.30
1004
72.39
10001
73.19
10.06
75.01
10.09
77.03
10.27
79.32 10077
Snow
71.78
73.72
74.46
20.16
73.97
#0.19
74.12
#0:05
74,62
+0.22
76.06
+0.06
77.64
+0.06
79.79 #0006
Frost
72.86
75.67
76.50
20.3
75.98
+0.11
75.55
#0.16
76.32
+0.13
77.67
#0.03
78.82
+0.20
80,54 +0.12
Fog
67.04
68.88
70.25
73.10
Brightness
81.87
83.52
83.73
+0.10
83.38
+0:
83,31
+0,5
83,51
+0.11
84.49
#0.13
85.40
+0,77
87.11 #011
Contrast
64.37
67.02
69.67 #013
68.64
+0.14
69.08
#0,6
71.11
#0.17
74.58
+0.14
78.25
#0.22
81.20 +0222
Flastic transf.
60.83
62.04
64,25
1013
63.50
40
63.46
10
64.65
20.28
66.63
20.21
69.58
10.18
72.66
10.15
Pixelate
50.53
51.65
58.18 1036
10.14
56.30
=0020
10.21
1:008
67.43
10.11
71.11
10.13
JPEG compr.
55.48
58.12
60.17
59.44
10.04
61.20
10.09
63.15
65.32
67.36
Mean
59.22
61.26
63.60
63.47
63.94
65.66
68.34
71.21
73.82
Table 17:
Accuracy (%) on CIFAR-10, CIFAR-10.1 and CIFAR- 10-C datasets with ViT-B/16 as
visual encoder for different number of batches.
Dataset
T=1
T=2
T=4
T=6
T=8
CIFAR-10
89.42 10.84
90.74
10:30
90.98 10.14
91.34
10.16
91.05
-0.06
CIFAR-10.1
85.08
86.49
87.20
10.48
87.53
10.21
86.98 -0.31
Gaussian Noise
59.82 11443
62.05
10.62
62,79
10.18
63.49
.0.27
63.84 .0.24
Shot noise
62.32 11.32
63.35
10.43
64.73
10.31
65.02
10.10
65.28 10.21
Impulse Noise
54.07 +0.17
56.83
+0.33
57.53
+0.47
58.37
=0.08
58.64 =0.11
Defocus blur
77.09 +0.24
78.32 +0.32
78.92
10.16
79.17
=0.26
78, =0.12
Glass blur
60.64 40.29
63.77 40.43
64.42
+0.68
64.64
-0.39
65. =0.07
Motion blur
74.60 10.50
77.02 40.32
77.70
10.26
77.73
10.12
77.81 10.14
CIFAR- - 10-C
Zoom blur
77.40 10,29
78.93
10,46
10.54 79.28
79.33
=0.24
79.32 =0.7
Snow
76.96 +1.04
78,83 10.31
79.47
10.29
79.69
=0.33
79.79 =0.06
Frost
77.62 40.86
79.27
40:45
80.04 10.26
80.46
-0.17
80.54 =0.12
Fog
75.32 10.57
77.27
78.00 10.17
78.55
78.53 10.22
Brightness
85.13 10.58
10.22 86.74
87.07 1.0.00
87.13
10.21
87.11 2011
Contrast
77.18 10.68
79.74 10.31
80.32 10.07
80.69
81.20 10.22
Elastic transform
69.39 +0.39
71.40 +0.24
72.25 +0.14
72.28
-0.34
72.66 =0.15
Pixelate
66.26
68,86
69,47
71.00
-0.57
71.11 -0.13
JPEG compression
64.58 +0.58
66.28 +0.14
66.82 +0.24
67.16 10.18
67.36 -0.28
Mean
70.56
72.58
73.25
73.65
73.82
Table 18:
Accuracy (%) CIFAR-10, CIFAR-10.1 and CIFAR-10-C datasets with ViT-B/16 as visual encoder for different number of templates randomly picked over 5 runs.
20

---

Dataset
CLIP
TENT
TPT (BS-32)
CLIPArTT
WATT-P
WATT-S
CIFAR-10
88.74
91.69 #0.10
88.06 00.66
90.04 #0.13
91.41 -0.17
91.05 40,6
CIFAR-10.1
83.25
87.60 1045
81.80 0022
86.35 10:27
87.78 00.8
86.98 10.31
Gaussian Noise
35.27
41.27
#0.27
33,90 =0.08
59.90 00.36
61.89
=0.24
63.84 +.024
Shot noise
39.67
47.20
+0.23
38.20 =0.02
62.77 =0.07
63.52 00008
65.28 +0021
Impulse Noise
42.61
+0.31 48.58
37.66 =0.20
56.02
10.16
57.13 =0.02
58.64 +0.11
Defocus blur
69.76
77.12 #0.16
67.83 =0.28
76.74
=0.05
78.86 =0.09
78.94 +0.12
Glass blur
42.40
52.65 +0.30
38.81
=0.12
61.77
=0.16
62.88 =0.06
65.12 +0007
Motion blur
63.97
71.25 10.99
63.39
0.13
76.01
10.19
76.85
...66
77.81 10.14
LFFAR-1--
Zoom blur
69.83
76.20 10.19
68.95
0016
77.40
1.220
79.35
0.04
79.32 10007
Snow
71.78
78.29 10.00
70.16 =011
77.29
10.16
79.44
0.09
79.79 1:006
Frost
72.86
79.84 1009
72.39
=0.22
79.20
10.88
80.13 00.10
80.54 1.012
Fog
67.04
77.39 10.01
64.31 =0.28
75.74
20.14
77.68 =0.07
78.53 +0022
Brightness
81.87
87.78 10.03
81.30
86.59
1016
87.10
87.1 #0.11
Contrast
64.37
79.47 10.11
62.26 =0.31
77.82
10.4
80.04 =0.24
81.20 +.022
Elastic transform
60.83
70.00 10.25
56.43
70.20
71.76
72.66 10.15
Pixelate
50.53
63.74 10.18
42.80 =0.40
66.52
#0113
69.28 =0.09
71.11 +0.13
JPEG compression
55,48
62.64 #0.44
53.67 =0.25
63.51
#0.14
66.49 =0.14
67.36 +
Mean
59.22
67.56
56.80
71.17
72.83
73.82
Table 19: Accuracy (%) on CIFAR-10, CIFAR-10.1 and CIFAR- 10-C datasets with ViT-B/32 as visual encoder.
Dataset
CLIP
TENT
TPT (BS=32)
CLIPArTT
WATT-P
WATT-S
CIFAR-10
89.25
92.75 40.17
89.80 10.05
92.61 20.05
92.31 20.10
91.97 20.03
CIFAR-10.1
84.00
88.52 40.33
83.75 =0.21
88.72 +0.33
87.9 =0.11
88.10 =0.08
Gaussian Noise
37.75
31.04 10.38
35.35
=0.15
60.89
10.26
63.10
10.12
65.57
10.22
Shot noise
41.10
40.54 +0.41
41.03
10.19
65.19
+0,21
66.31
10.10
68.67
+0,37
Impulse Noise
51.71
58.03 10.16
54.86
=0.07
67.55 #0.09
69.62 20.12
70.39 10.11
Defocus blur
70.07
77.57 10.03
70.29
10.02
78.92 10.12
79.64 10.08
79.90 10.07
Glass blur
42.24
47.16 10.05
37.86
10.17
57.18 10.20
58.98
61.62 10.21
Motion blur
65.81
76.16 10.05
67.43
10.11
76.59 10.06
78.32 10.16
79.02 10.07
CIFAR-10-C
Zoom blur
72.50
79.64 10.12
72.91
10.02
79.62 10.11
80.67
81.10 10.08
Snow
73.23
81.68 +003
72.98
+0.32
81.13 +0029
81.99 10.10
82.54 10.18
Frost
76.52
83.22 10.05
10.16 75.87
81.24 40.08
83.41
83.46
Fog
68.35
80.78 10.15
69.13 +0.27
78.47 40.19
81.36 10.12
81.88 10.12
Brightness
83.36
89.85 +0.11
83,67
10.14
88,66
89.06 10.05
89.10
Contrast
61.90
79.24 10.19
62.16 =0.06
75.15 +0.07
81.57 +0.23
83.79 +0.12
Elastic transform
53.16
62.54 40.08
51.26 -0.23
69.49 +0.08
69.14 10.09
70.93
Pixelate
48.48
67.08
10.24
44.65
10.1
71.80 10.16
73.38
10.29
75.67 10.32
JPEG compression
56.05
65.42 10.0
56.73
10.07
66.42 10.25
69.02 10.10
69.65 10.23
Mean
60.15
68.00
59.75
73.22
75.04
76.22
Table 20: Accuracy (%) on CIFAR-10, CIFAR-10.1 and CIFAR-10-C datasets with ViT-B/16 as
visual encoder.
21

---

Dataset
CLIP
TENT
TPT (BS=32)
CLIPArTT
WATT-P
WATT-S
CIFAR-I00
64.76
71.73 #0.14
67.15 =0.23
71.34 +0.07
72.98 #0.07
72.85 #0.15
Gaussian Noise
15.88
12.28 +0.20
15,43 =0.03
19.01 #0.24
34.23 =0.03
35,95 #0.27
Shot noise
17.49
15.07 +021
16,88 =0.07
20.27 #0.11
36.68
37.96 =0.15
Impulse Noise
21.43
13.13 #0.16
22.12 =0.15
17.66 =0.10
43.17 =0.35
44,62 #0.2
Defocus blur
40.10
50.35 +0.03
41.08 =0.22
49.86
53.13 =0.12
53.80 =0.12
Glass blur
13.48
4.84 10.14
18.43 10.15
18.34 10.31
32.53 10.03
33.39 20.11
CIFAR MAR-100-C
Motion blur
39.82
49.85 40.37
40.85
0.26
50.00 +0.09
51.63 #0.06
52.72 0033
Zoom blur
45.45
54.76 10.04
46.77
0006
54.13 +0.08
56.81 #0.11
57.51 +0009
Snow
42.77
52.38 10.18
47.24
10.18
52.80 10.27
56.04
10.06
56.73
20.27
Frost
45.39
51.66 1004
48.61
10.14
49.56 10.08
56.00 10.11
56.48
10.34
Fog
38.98
50.74 10.14
39.92
10166
49.92 10.11
52.88 10.33
53.83
10.19
Brightness
52.55
64.26
55,83 =0.10
63.76
65.58 =0.07
66.67 10.19
Contrast
33.32
48.69 +0.08
33.13 =0.16
47.86
52.90 =0.06
55.06 #0.15
Elastic transform
24.39
33.56
27.36
=0.10
32.93
+0.23
39.82 =0.21
40.37
#0.26
Pixelate
21.89
36,20 +0.28
21.26
-0.10
39.49
40.21
45. =0.06
47.02 20.04
JPEG compression
27.21
30.80 10.05
30.97
10.10
35.56 +0.23
41.43 20.18
42.13 #0.21
Mean
32.01
37.90
33.73
40.08
47.86
48.95
Table 21: Accuracy (%) on CIFAR-100 100 and CIFAR- 100-C datasets with ViT-B/16 as visual encoder.
Dataset
CLIP
TENT
TPT (BS-32)
CLIPArTT
WATT-P
WATT-S
CIFAR-10
95.36
96.15 13 4006
95.18 =0.02
95.16
95.91 10.10
95.71 20.03
CIFAR-10.1
91.20
92.22 #0.25
91.32 0012
91.02 #0.02
92.97 10.13
92.10 +0.33
Gaussian Noise
64.64
68.87 +0.20
64.44
10.11
70.04 10.31
72.81
=0.09
72.73 10.03
Shot noise
67.82
71.95 +0.06
66,81
=0.19
71.44 10.16
74.45
10.16
74.60 +0.03
Impulse Noise
78.21
80.22 10.19
76.46
10.17
79.42 10.15
81.36
80.95 10.15
Defocus blur
80.73
83.10 10.03
79.01
10.23
81.75 10.19
100.10 83.20
83.15 10.18
Glass blur
50.29
57.12 10007
49.64 10.23
58.13 10.23
61.51
62.35 10.15
Motion blur
80.75
82.69 +0.11
78.85 +0.04
80.76 +0.12
82.60 10.13
13.1
82.61 +0.12
CIFAR-10-C
Zoom blur
82.75
84.91 +0.88
82.32
83.39
85.76 10.06
85.44
Snow
83.01
85.99 +0.11
82.69 10.10
84.48 +0.07
84.91 13.13
85.61 10.15
Frost
84.90
87.15 40.12
84.63 -0.88
85.21
87.17 10.13
86,88
Fog
78.44
81.30 +0.07
77.56
=0.17
79.27 #0.07
81,80 =0.11
10.11
81,79 +0.09
Brightness
91.67
93.07 40.44
90.94
10.04
91.87
92.78 +0.05
92.59
Contrast
84.20
87.93 +0.04
82,88
=0.09
86.19 +0.06
87.54 #0.12
87.38
+0.02
Elastic transform
65.45
69.96 10.12
64.81
10.14
67.45
10,24
71.19 10.07
71.25
10.09
Pixelate
75.10
77.89
10.0
72.92
10.12
77.11
10.10
77.88
10..3
77.67
10.16
JPEG compression
72.58
75.49 1007
71.18
10.19
74.46 10.11
75.88
10..16
75.84
10.18
Mean
76.04
79.18
75.01
78.06
80.05
80.06
Table 22: Accuracy (%) on CIFAR-10, CIFAR-10.1 and CIFAR-10-C datasets with ViT-I/144 as
visual encoder.
22

---

Dataset
CLIP
TENT
1BT,)))322)
CLIPArTT
WATT-P
WATT-S
CIFAR-I00
73.28
78.03 +0.08
76.85 -0.06
79.42 #0.08
79.33 #0.05
78,85 =0.19
Gaussian Noise
30.55
36.93 +0003
36.10 =0.11
41,46 #0.15
43,99 =0.13 10.13
44.13
=0.11
=0.11
Shot noise
34.58
40.96
38.23 =0.13
44.27 =0.09
46.28
=0.22
46.63
=0.17
Impulse Noise
44.89
49.09 10.14
49.69
=0.21
51.44
56.15 =0.04
=0.04
56.26=0.2
Defocus blur
48.88
55.23 #007
50,43
=0.19
56.55
#0.22
=7.46
57.66
Glass blur
23.46
27.02 4:233
24.35
+0.22
30.47
20.14
32.54 10.12
33,54 0.16
Motion blur
50.83
56.03 4:020
51.94 10004
56.98 20.18
58.22 10.10
57.81 0.05
CIFAR-100-6 10-100-C
Zoom blur
56.02
61.19 +0.10
56.96
10.16
62.56
20.04
20.04
62.94
62.74 0006
Snow
49.03
55.60 10.09
54.89
20.11
58.81
20.11
60.68 10.06
61.04
-0.27 =0.27
CIFAR
Frost
53.27
58.21 10.5
58.15
10.33
60.38
20.23
62.34
10.14
62.76 -0.22
Fog
48.51
53.37 10.25
49.26 10.13
54.38
10.4
54.71 10.31
$4.70 10.13
Brightness
60.53
67.34 #0.17
66.60
=0.10
69.63
#0.14
71.52
=0.07
71.60=0.0
Contrast
50.24
59.91 +0.13
53.64
=0.24
63.39
#0.13
62.77
=0.22
63.95 =0.04
Elastic transform
35.07
38.49 #0.12
35.72
=0.09
39.57
#0.39
41.28
=0.25
41,27 =0.15
Pixelate
43.86
48.37 #0.17
44.32
=0.10
50.45
20.16
51.15
10.15
51.22
=0.13
JPEG compression
39.11
44.42 10009
43.44 10.11
47.45
20.14
49.40
10.17
49.78 20.18
Mean
44.59
50.14
47.58
52.52
54.10
54.34
Table 23: Accuracy (%) on LIFAR-100 and CIFAR- 100-C datasets with ViT-L/14 as visual encoder.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract, we claim that we introduce a new method using Weight Averaging for Test-Time Adaptation. In the paper, we present this method and prove its performance compared to state-of-art.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The work has been tested on many datasets with several runs. We also point out that with a small number of classes, it seems less effective compared to when there are many classes. Additionally, we show that it not always the best when there is no domain shift.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All our assumptions are writing in the Introduction and the Methodology, everything is explained with some illustrations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the section Experimental Setup, we present all our different settings which are based the CLIPArTT method. We also provide our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Everything is detailed in the github link in the Implentation section in the supplementary material..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
mublic//guides/Codeeuuiiiis oow.aaaboisioooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //ips.cc/ cttpp/scccpubllcccuiiiiie guides
Pollccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Everything detailed in the section Experimental Setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
All experiments have been conducted at least three times to ensure statistical robustness.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Everything is detailed in the section Implementation of the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have considered all potential harms caused by the research process, societal impacts, and potential harmful consequences, as described the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper addresses a research field that not currently utilized for direct practical applications, thereby limiting its immediate social impact.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In paper, cite all sources that inspired our work, and we provide links in our git repository to the codes from which we drew inspiration.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document all aspects of our research and provide link to our implementa- tion, along with step-by-step explanations for reproducing the results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "GFSSETvrYQ", "reviewer": "Reviewer_wUmo", "summary": "This paper proposes a method called Weight Average Test-Time Adaptation (WATT) to improve test-time adaptation (TTA) for the CLIP model. The core idea is to use different text templates to construct multiple text prompts and adapt the model weights using these different prompts. During the evaluation stage, the prompts provide predictions based on the adapted weights and text embedding averaging across multiple prompts. The authors conduct comprehensive experiments that demonstrate the generalization and strong performance of the WATT method.", "questions": "* Is it possible to combine text prompt augmentation (this method) with TTA strategies that use image augmentation to achieve better results?\nSome templates may be incorrect. For example, as shown in Table 2, prompt T0: \"a photo of a {class k}\" and prompt T2: \"a bad photo of the {class k}\" refer to the same image, but why is this photo considered bad? If you add distortion or corruption, then it might be appropriate to call it bad. Also, for T6: \"art of the {class k},\" if you use diffusion to generate an art image of the original image, then the prompt would match.\n* Can you measure the similarity score of these text templates in the CLIP text encoder space?", "limitations": "Can the authors test whether this method can also improve the latest VLM, such as SigLIP?", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The idea is simple but works effectively.\n* Comprehensive experiments support the effectiveness of this method in the following aspects:\nRobustness, as reflected in Table 6.\nEfficient adaptation, as reflected in Table 4.\nGeneralization and state-of-the-art comparison, as shown in Table 7.", "weaknesses": "There are no clear weaknesses, but I do have some questions and potential limitations that do not affect my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q:* Is it possible to combine text prompt augmentation (this method) with TTA strategies that use image augmentation to achieve better results? Some templates may be incorrect. For example, as shown in Table 2, prompt T0: \"a photo of a {class k}\" and prompt T2: \"a bad photo of the {class k}\" refer to the same image, but why is this photo considered bad? If you add distortion or corruption, then it might be appropriate to call it bad. Also, for T6: \"art of the {class k},\" if you use diffusion to generate an art image of the original image, then the prompt would match.**\n\n*A:* To clarify how we chose the templates for weight averaging: The CLIP paper identifies 80 templates that enhance model robustness and performance. They ultimately conclude that 7 of these templates best summarize their model (see https://github.com/openai/CLIP/blob/main/notebooks/Interacting_with_CLIP.ipynb). In our work, we use these 7 generic templates and add the common one, “a photo of {},” based on their optimization. These templates are not specifically linked to the content of the images. However, linking these text templates to the images through augmentation or generation is a promising idea for future work. Thank you for the suggestion.\n\n***Q:* Can you measure the similarity score of these text templates in the CLIP text encoder space?**\n\n*A:* As suggested, we computed the similarity between the text templates for each class and averaged them into a single matrix, which we have included in the PDF. \nThe similarity between different templates for the same class can vary. This highlights that utilizing diverse templates, despite their individual variations, provides a richer set of information that significantly enhances the model's performance.\n\n***Q:* Can the authors test whether this method can also improve the latest VLM, such as SigLIP?**\n\n*A:* Thank you for your suggestion! We have incorporated SigLip into our code and conducted a comparative analysis with our Sequential method (WATT-S) on different datasets. The results are summarized in the table below:\n |  Dataset                | **SigLip** | **WATT-S** |\n | :-------------------- | :-------------------- | :-------------------- | \n | **CIFAR-10** | 66.35 | 75.02 &plusmn; 0.05|\n | **CIFAR-10.1** | 57.30 | 65.87 &plusmn; 0.21|\n | **CIFAR-10-C** | 37.52 | 45.29 &plusmn; 0.13|\n | **CIFAR-100** | 33.97 | 65.87 &plusmn; 0.21|\n | **CIFAR-100-C** | 14.43 | 20.05 &plusmn; 0.05|\n\nOur method, when applied to SigLip, shows significant improvements across all datasets, highlighting the effectiveness of WATT in enhancing performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fn99MtYOR3", "reviewer": "Reviewer_y4Me", "summary": "This paper proposes a test-time adaptation (TTA) method for CLIP by integrating various textual prompt templates into Weight Average (WA) methods (Ref [11], [22]). Experiments on multiple types of domain shifts show the effectiveness of the proposed method.", "questions": "Is the evaluation done online (like the setting in the TENT paper where inference and adaptation happen on each batch in data streams), or does the model adapt with a single batch and then perform inference on the whole test set?", "limitations": "The authors mention limitations in the checklist.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is effective yet simple.\n- The experiments span different domain shift scenarios and demonstrate better performance of the method.", "weaknesses": "- The second contribution at the end of Section 1 lacks sufficient evidence. Table 4 only shows that the proposed method can handle small batch sizes without comparison with others. Additionally, there are existing TTA works studying small batch size scenarios. (SAR [Towards Stable Test-Time Adaptation in Dynamic Wild World, 2023], MEMO [MEMO: Test Time Robustness via Adaptation and Augmentation, 2022], etc.). Recent diffusion model-based TTA methods (such as DDA [Back to the Source: Diffusion-Driven Test-Time Adaptation, 2023]) also show robustness for small batch sizes.\n- The technical contribution seems not that significant. The core idea is introducing multiple prompt templates into two previous WA methods, to my understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q:* Comparison with other methods and a batch size of 1.**\n\n*A:* Thank you for your insightful feedback. We appreciate your comments regarding the need for more comprehensive evidence to support our second contribution in Section 1. In response, we have tested the TTA methods mentioned in our paper with a batch size of 1. Additionally, we implemented the comparisons you suggested and adapted SAR [Towards Stable Test-Time Adaptation in Dynamic Wild World, 2023] and MEMO [MEMO: Test Time Robustness via Adaptation and Augmentation, 2022] to CLIP.\nWe decided not to implement DDA [Back to the Source: Diffusion-Driven Test-Time Adaptation, 2023]. While it is an excellent article, its benchmarks differ from ours. In DDA, the input is adapted rather than the encoder, and it aligns more with test-time training than fully test-time adaptation, as it requires the source dataset to train the diffusion model.\n\nThe results with a batch size of 1 are summarized in the table below:\n\n| Dataset       | **CLIP** | **TPT** | **CLIPARTT** | **SAR** | **MEMO** | **WATT-S** |\n| :------------ | :------- | :------ | :----------- | :------ | :------- | :--------- |\n| **CIFAR-10**  | 88.74    | 88.29   | 88.76        | 87.41   | 89.29    | 89.87      |\n| **CIFAR 10.1**| 83.25    | 82.85   | 83.15        | 82.32   | 83.80    | 84.55      |\n| **CIFAR-10-C**| 59.22    | 59.03   | 59.18        | 58.70   | 61.15    | 61.26      |\n\nAs can be seen, WATT-S achieves the highest accuracy in all cases. On CIFAR-10.1, it outperforms SAR by 2.23% and MEMO by 0.75%. As highlighted in our paper, this improvement is achieved without any image augmentation, a common practice in previous TTA approaches working with small batches.\n\n***Q:* The technical contribution seems not that significant. The core idea is introducing multiple prompt templates into two previous WA methods, to my understanding.**\n\n*A:* Our work introduces the use of multiple prompt templates in weight averaging (WA) methods, which, to our knowledge, is a novel approach in the TTA field. As noted by Reviewer HDdi, this approach deviates from conventional TTA techniques and offers a valuable extension to existing methods. We have also conducted a comparative analysis of WA with other common ensembling methods to highlight its effectiveness. \nMoreover, our paper is the first to demonstrate performance fluctuations due to using different text templates. Recognizing this, we leveraged these fluctuations as a benefit to enhance WA, whereas previous articles focused on using image augmentations or varying hyperparameters in the training time.\n\n***Q:* Is the evaluation done online (like the setting in the TENT paper where inference and adaptation happen on each batch in data streams), or does the model adapt with a single batch and then perform inference on the whole test set?**\n\n*A:* In the context of the TENT paper, the model adapts to a given batch, and updates its parameters from the most recent adaptation when a new batch is introduced. In this framework, adaptation and inference occur within the same batch, similar to TENT. However, upon introducing a new batch, the model reverts to the standard CLIP configuration before making the adaptation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMWMORiIRo", "reviewer": "Reviewer_HDdi", "summary": "The authors introduce a Test-Time Adaptation technique named Weight Average Test-Time Adaptation (WATT) for Vision-Language Models (VLMs) such as CLIP. WATT is the proposed method that improves test-time adaptation by borrowing ideas from different text prompt templates to develop pseudo-labels for model updates and also carrying out weight averaging so as to consolidate learned information from those generated labels. The approach is tested across different datasets, revealing its capacity to boost performance without any further metamorphosis of the model or additional trainable modules.", "questions": "1. Could you provide a more detailed comparison of the computational cost of WATT compared to other TTA methods?\n2. Have you considered evaluating WATT on real-world datasets with more complex domain shifts?\n3. How would WATT perform on other vision tasks such as segmentation or object detection?", "limitations": "The authors have addressed the limitations and potential negative societal impacts adequately. They highlight the focus on image classification and suggest future work extending the method to other tasks. However, they could discuss more on the computational resource requirements and scalability in practical deployments.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The novelty of the study is that it proposes a novel test-time approach. It involves averaging weights while using varying text prompts which deviates from the conventional TTA techniques— making it a high-quality extension to existing TTA methods.\n+ It ensures quality by using rigorous experimental setup: involving thorough evaluations on various datasets, with good performance demonstrated by improvements over the current leading techniques.\n+ It maintains clarity in its presentation: both of methodology and results. This has been made possible by visual aids and detailed elaborations that ensure better understanding, thus though we still need to work more on this document to achieve perfection.\n+ The importance of the proposed method lies in its ability to adapt using a single image without requiring any further changes to the model; this level of adaptability is highly significant for real-world applications across different fields.", "weaknesses": "- Limited Scope of Evaluation: State-of-the-art Methods [1, 2] published in recent CVPR/ICCV are missing for comparing in Experimental part and discussing in the Related Work section.\n[1] Diverse data augmentation with diffusions for effective test-time prompt tuning. ICCV, 2023.\n[2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR, 2024.\n- Computational Cost: The paper mentions the efficiency of WATT but does not provide a detailed comparison of computational costs with other TTA methods, which could be crucial for practical deployment.\n- Generalization to Other Tasks: The paper focuses on image classification tasks. Extending the approach to other vision tasks like segmentation or detection and evaluating its performance there could provide a more comprehensive understanding of its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q:* Limited Scope of Evaluation.**\n\n*A:* Thank you for your valuable feedback regarding the scope of our evaluation. We appreciate the recommendation to include comparisons with more recent state-of-the-art methods in both the Experimental section and the Related Work section. Specifically, we have taken note of the methods mentioned: DiffTPT [Diverse Data Augmentation with Diffusions for Effective Test-Time Prompt Tuning, ICCV 2023] and TDA [Efficient Test-Time Adaptation of Vision-Language Models, CVPR 2024]. Therefore, we compared them with the TTA method mentioned in our paper and SAR as noted by Reviewer y4Me. The results are summarized in the table below:\n\n| Dataset       | **CLIP** | **TENT** | **TPT** | **CLIPArTT** | **TDA** | **DiffTPT** | **SAR** | **WATT-P** | **WATT-S** |\n| :------------ | :------- | :------- | :------ | :----------- | :------ | :---------- | :------ | :--------- | :--------- |\n| **CIFAR-10**  | 88.74    | 91.69 &plusmn; 0.10 | 88.06 &plusmn; 0.06 | 90.04 &plusmn; 0.13 | 84.09 &plusmn; 0.04 | 83.07 &plusmn; 0.05 | 89.05 &plusmn; 0.02 | 91.41 &plusmn; 0.17 | 91.05 &plusmn; 0.06 |\n| **CIFAR-10.1**| 83.25    | 87.60 &plusmn; 0.45 | 81.80 &plusmn; 0.27 | 86.35 &plusmn; 0.27 | 78.98 &plusmn; 0.37 | 76.50 &plusmn; 0.29 | 83.65 &plusmn; 0.04 | 87.78 &plusmn; 0.05 | 86.98 &plusmn; 0.31 |\n| **CIFAR-10-C**| 59.22    | 67.56              | 56.80            | 71.17              | 48.00            | 56.77            | 60.45            | 72.83            | 73.82            |\n| **CIFAR-100** | 61.68    | 69.74 &plusmn; 0.16 | 63.78 &plusmn; 0.28 | 69.79 &plusmn; 0.04 | 60.32 &plusmn; 0.06 | 52.80 &plusmn; 0.08 | 64.44 &plusmn; 0.01 | 70.38 &plusmn; 0.14 | 70.74 &plusmn; 0.20 |\n| **CIFAR-100-C**| 29.43    | 35.19              | 30.46            | 41.51              | 22.08            | 22.89            | 31.92            | 44.68            | 45.57            |\n\nAccording to the TDA supplementary materials, we selected the weighting factor alpha as 5.0 and the sharpness ratio beta as 2.0, which are stated as optimal. However, these values did not appear to be the best choice for more challenging datasets like CIFAR-10-C or CIFAR-100-C, which contain various corruptions. Adjusting these parameters based on the dataset would not be consistent with the principles of a fully TTA method, which might explain their suboptimal performance in our results.\nRegarding DiffTPT, it generates 64 images per test image, making it challenging to use in a real-world TTA scenario. Similar to TDA, DiffTPT requires carefully chosen parameters to fit the dataset, whereas our method does not require dataset-specific tuning. This highlights the robustness and practicality of our approach in diverse real-world applications.\n\n***Q:* Computational Cost.**\n\n*A:* To address the reviewer's concern regarding the computational cost comparison of WATT with other TTA methods, we have conducted a thorough evaluation under consistent conditions using an NVIDIA A6000 GPU within the same Python environment. The table provided compares the adaptation time, memory usage, and the number of learnable parameters for various TTA methods, including our proposed WATT method.The table clearly demonstrates that WATT-S, a sequential implementation of WATT, maintains competitive adaptation time and memory usage compared to other methods like TENT and ClipArTT, which are efficient but lack the robustness of WATT's method. Additionally, the table highlights that WATT-P, with parallel model training, offers a faster adaptation time than WATT-P with a for-loop implementation, albeit at the cost of higher memory usage.\nIt's important to note that methods like DiffTPT and MEMO, which show significantly higher adaptation times, employ off-the-shelf diffusion models and AugMix augmentation, respectively, resulting in time-consuming processes that may be impractical for real-world scenarios. In contrast, the effectiveness of our WATT-S method makes it better suited for scenarios where a robust, rapid, and resource-efficient adaptation is crucial.\n\n| Method     | **Adaptation Time** | **GPU Memory**      | **Percentage of  Learnable Parameters** |\n| :--------- | :------------------ | :-------------- | :------------------------- |\n| **TENT**   | 0.3 sec.             | 1.5 GB          | 0.026%                     |\n| **ClipArTT**| 0.6 sec.             | 1.7 GB          | 0.026%                     |\n| **SAR**    | 0.4 sec.              | 1.4 GB          | 0.026%                     |\n| **MEMO**   | 165 sec.               | 2 GB            | 0.026%                     |\n| **DiffTPT**| 8.5 sec.          | 10 GB |         0.001%                    |\n| **WATT-P** | 23.2 sec.     | 1.5 GB  | 0.026%            |\n| **WATT-P (concurrent)** | 2.9 sec.     | 10 GB | 0.208%              |\n| **WATT-S** | 2.3 sec.              | 1.5 GB          | 0.026%                     |\n\nWe noticed that we used the word \"efficiency\" in lines 105 and 382. We apologize for this and should have used \"effectively\" instead.\n\n***Q:* Generalization to Other Tasks.**\n\n*A:* While our current work focuses on image classification, exploring the applicability of WATT to tasks such as segmentation and object detection is indeed an interesting direction for future research. It is worth noting that many recent TTA papers also primarily focus on classification tasks. Extending to other tasks would require tailored experimental settings and potentially different methodological adjustments (e.g., CLIP cannot be used directly for segmentation), which we believe are beyond the scope of this paper. We appreciate your suggestion and consider it a valuable avenue for further investigation.\n\n***Q:* Have you considered evaluating WATT on real-world datasets with more complex domain shifts?**\n\n*A:* Due to limited space, please refer to the global rebuttal for a general explanation of the datasets we used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the reviewers' insightful and constructive comments and are pleased to note that all three reviewers voted towards acceptance. We are also encouraged by the feedback highlighting the robustness and generalizability of our method (Reviewer wUmo), its superior performance across various domains (Reviewer y4Me), and the rigor of our experiments (Reviewer HDdi).

Below, we address all the questions raised by the reviewers, including additional baseline evaluations and clarifications as requested. For Reviewer wUmo’s convenience, we have included additional plots and results in the updated PDF.

--- 

###  **Continuation of Answer to Reviewer HDdi:**

***Q:* Have you considered evaluating WATT on real-world datasets with more complex domain shifts?**

*A:* Thank you for this remark suggesting that a better explanation of the datasets should be added to our supplementary materials. In our investigation, we use the VisDA-C dataset, which challenges models with two distinct domain shifts: the simulated shift and the video shift. The simulated shift includes 152,397 3D-rendered images across 12 diverse classes, while the video shift comprises 72,372 YouTube video frames spanning the same categories. This dataset addresses the diversity of imagery types applicable to a model, posing a significant challenge.

Moreover, we evaluate our proposed method on three other datasets: PACS, VLCS, and OfficeHome. These datasets help understand various domain shifts, including texture and style variations. The PACS dataset consists of 9,991 images across four domains (Art, Cartoons, Photos, Sketches) and seven classes. The VLCS dataset contains 10,729 images across four domains (Caltech101, LabelMe, SUN09, VOC2007) and five classes. Lastly, the OfficeHome dataset includes 15,588 images across four domains (Art, Clipart, Product, Real) and 65 classes. Evaluating across these distinct domain shifts showcases the generalizability of our method.
These datasets are more representative of real-world scenarios compared to CIFAR, with complex domain shifts.

Meta-review: The paper proposes "Weight Average Test Time Adaptation" (WATT) for VLMs like CLIP, aimed at enhancing their ability to handle domain shifts during inference without retraining. The method leverages multiple text prompt templates to adapt model weights at test time and aggregates predictions using weight averaging to improve performance across datasets with domain shifts, such as CIFAR and VisDA. The results show notable improvements in robustness and generalization without the need for model modifications or additional training.

The paper is technically solid and presents a practical approach to improving VLMs under domain shifts without retraining. Despite the limited novelty, the experimental results demonstrate benefits in terms of robustness and generalization, making it a valuable contribution to the test-time adaptation field. While there are some concerns about the breadth of evaluation and computational aspects, the overall impact on the VLMs and TTA subfield justifies its acceptance.
Decision: Accept (poster)