Title: WATT: Weight Average Test Time Adaptation of CLIP
Venue: NeurIPS 2024
Accepted: yes
Keywords: Weight Averaging

Test Time Adaptation

CLIP

Text Ensemble
Abstract: Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performances for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a new approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing the overall test performance by aggregating diverse textual cues.
Our findings underscore the effectiveness of WATT across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without the need for additional model transformations or trainable modules. Moreover, compared to other TTA methods, our approach can operate effectively with just a single image. The code is available at: https://github.com/Mehrdad-Noori/WATT.

WATT: Weight Average Test-Time Adaptation of CLIP

Abstract

Vision-Language Models (VLMs) such as CLIP have yielded unprecedented performances for zero-shot image classification, yet their generalization capability may still be seriously challenged when confronted to domain shifts. In response, we present Weight Average Test-Time Adaptation (WATT) of CLIP, a new approach facilitating full test-time adaptation (TTA) of this VLM. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Predictions are utilized as pseudo labels for model updates, followed by weight averaging to consolidate the learned information globally. Furthermore, we introduce a text ensemble strategy, enhancing the overall test performance by aggregating diverse textual cues. Our findings underscore the effectiveness of WATT across diverse datasets, including CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, VisDA-C, and several other challenging datasets, effectively covering a wide range of domain shifts. Notably, these enhancements are achieved without the need for additional model transformations or trainable modules. Moreover, compared to other TTA methods, our approach can operate effectively with just a single image. The code is available at: https://github.com/Mehrdad-Noori/WATT .

1 Introduction

The integration of vision and language modalities into a unified learning framework, known as Vision Language models (VLM), has shown remarkable effectiveness in a broad range of vision-related tasks [1, 2, 3]. Notably, these models excel in zero-shot generalization scenarios, where they demonstrate proficiency in tasks beyond their original training scope, without requiring additional fine-tuning supervision. Applications of models like CLIP [1] extend across diverse domains including video recognition [4], audio processing [5], and medical imaging [6]. These advancements underscore the pivotal role of such methods in shaping the trajectory of future research and applications in machine learning.

Despite its powerful capabilities, CLIP, like other traditional deep architectures such as Convolutional Neural Networks (CNNs), experiences performance degradation when confronted with domains it has not been trained on. Current research trends emphasize the importance of domain adaptation mechanisms in the deployment of CLIP [7, 8]. However, a significant challenge remains: swiftly and

Template

T

0

:

'

a

photo

of a

{class

k

}

'

T

1

:

'

itap

of

a {class

k

}

'

T

2

:

'

a

bad

photo of

the

{class

k

}

'

T

3

:

'

a

origami

{class

k

}

'

T

4

:

'

a

photo

of the

large

{class

k

}

'

T

5

:

'

a

{class

k

}

in

a

video

game

'

T

6

:

'

art

of

the {class

k

}

'

T

7

:

'

a

photo

of the

small

{class

k

}

'

(a)

T

0

T

1

T

2

T

3

T

4

T

5

T

6

T

7

T

0

T

1

T

2

T

3

T

4

T

5

T

6

T

7

1.00

0.81

0.93

0.77

0.94

0.89

0.91

0.95

0.81

1.00

0.73

0.62

0.75

0.71

0.69

0.75

0.93

0.73

1.00

0.74

0.90

0.88

0.89

0.91

0.77

0.62

0.74

1.00

0.74

0.75

0.77

0.77

0.94

0.75

0.90

0.74

1.00

0.85

0.88

0.93

0.89

0.71

0.88

0.75

0.85

1.00

0.88

0.87

0.91

0.69

0.89

0.77

0.88

0.88

1.00

0.89

0.95

0.75

0.91

0.77

0.93

0.87

0.89

1.00

Average Similarity Matrix Across All Classes

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

(b)

T

0

T

1

T

2

T

3

T

4

T

5

T

6

T

7

WATT

Original

89.80

90.37

90.50

88.42

89.93

89.95

90.13

88.54

91.05

60.19

61.01

61.17

58.24

58.84

58.35

59.62

61.13

63.84

Defocus Blur

77.23

77.07

78.00

75.98

76.39

77.45

77.08

75.59

78.94

Snow

76.57

77.36

77.93

75.08

77.45

77.09

77.05

75.57

79.79

JPEG Compression

64.65

65.36

65.24

64.16

64.18

64.36

64.78

65.32

67.36

(c)

effectively adapting the model to new domains while preserving its attractive zero-shot capabilities, thus obviating the need for retraining.

To tackle this challenge, we investigate the impact of different text prompt templates, listed in Figure 1a, on model adaptation. Despite these templates being related, as reported in Figure 1b, the cosine similarity between their embeddings varies greatly, suggesting that they encode complementary information about the classes. The variability of information in different templates can also be observed in Figure 1c, where the classification accuracy obtained with these templates on a subset of CIFAR-10 corruptions fluctuates by up to 3%. Given this insight, finding an effective way to leverage the knowledge from different text templates would be useful to yield a better adaptation. This motivates our work proposing Weight Average adaptation during Test-Time (WATT).

By strategically averaging the adapted weights derived from multiple text prompt templates, our method aims to harness the complementary strengths of individual templates, resulting in robust and enhanced performance across a wide range of domain shifts. To further illustrate this point, Figure 2 presents the test loss and adaptation error surfaces for three models that are separately adapted using three templates ( T 0 , T 1 , T 2 ) under the Gaussian noise corruption of the CIFAR-10-C dataset 2 . The central point in these landscapes, representing the final model obtained by averaging the weights of the separate models, demonstrates a convergence towards lower loss and error, highlighting the potential of weight averaging for test-time adaptation. Moreover, inspired by recent advancements in machine learning utilizing train-time weight averaging techniques [10, 11], the proposed WATT method can dynamically adjust to new data to tackle unforeseen distribution shifts without relying on class supervision.

We outline the main contributions of our work as follows:

We introduce a novel Test-Time Adaptation method for CLIP, which leverages weight averaging across various text templates at test-time.

T0

T1

T2

Loss

T0

T1

T2

Error (1-Acc)

1.4365

1.1460

0.9934

0.9133

0.8712

0.8491

0.8375

0.8247

0.4219

0.3689

0.3348

0.3129

0.2988

0.2897

0.2839

0.2734

Our WATT strategy yields highly competitive performances in comparisons to current TTA methods, and could bring improvement using only a single image at test time, a capability not present in previous approaches.

We rigorously evaluate our WATT methodology through comprehensive evaluations across different datasets characterized by diverse types and degrees of domain shifts, encompassing a total of 155 evaluation scenarios . Our experiments demonstrate the robustness and efficacy of WATT compared to existing adaptation methods.

2 Related work

Test-Time Adaptation (TTA) is crucial in domain adaptation, particularly with unlabeled target domain data and no access to source domain samples. The challenge lies in estimating the target domain's distribution and comparing domain characteristics indirectly. Recent advancements have highlighted the potential and limitations of adapting pre-trained models. A key focus has been on leveraging batch normalization layers for adaptation due to their ability to retain source domain information. Methods such as PTBN [12] and TENT [13] recalibrate batch statistics and optimize affine parameters via entropy minimization, though they often require image augmentations or large batches. MEMO [14] proposes a simple approach that does not require multiple test points; it uses single-test-point data augmentations and minimizes the marginal entropy of the model's average output to enforce consistency and improve robustness. LAME [15] introduces a closed-form optimization strategy that refines model predictions for target images by leveraging the Laplacian of feature maps to encourage clustering, thereby emphasizing feature similarities. In another study, SAR [16] addresses TTA instability by using batch-agnostic norms (e.g., group and layer norms) and a sharpness-aware entropy minimization approach. It filters noisy samples and guides the model to flat minima, improving robustness under mixed domain shifts and small batch sizes.

Recently, Test-Time Training (TTT) methodologies have emerged as prominent contenders in TTA [17, 18, 19, 20, 21]. This approach involves training a supplementary sub-branch alongside the primary network in an unsupervised manner, subsequently leveraging it to refine the model. Unlike previous methods, our approach operates on individual image batches, offering a significant advantage in TTA by avoiding the necessity of training additional branches from scratch.

In natural language processing, TPT [8] introduced entropy minimization for adapting models like CLIP, albeit with high computational costs due to learning an adapter at the text prompt with multiple transformations. DiffTPT [22] extends this by leveraging pre-trained diffusion models to generate diverse and informative augmented data, combining conventional augmentation methods used in TPT with diffusion-generated data to enhance adaptability. It also introduces a cosine similarity-based filtration technique for improved prediction fidelity. TDA [23] offers a training-free approach with a dynamic adapter, utilizing a lightweight key-value cache and pseudo label refinement, making it computationally efficient. CLIPArTT [24], fine-tunes normalization layers with minimal disruption to the model's knowledge, enhancing text supervision by introducing pseudo labels. Existing methods often lag behind supervised prompt adaptation techniques in performance. SwapPrompt [25] bridges this gap by leveraging self-supervised contrastive learning, employing a dual prompt paradigm. In

1 st  Adapted

CLIP

(𝜽 𝟏

)

𝐿𝑇𝑇𝐴(𝜃 1

)

𝑻 𝟎

A photo of

a {class

k

}

+

Test Image

2 nd  Adapted

CLIP

(𝜽 𝟐

)

𝐿𝑇𝑇𝐴(𝜃 2 )

𝑻 𝟏

itap of

a {class

k

}

+

Test Image

H th  Adapted

CLIP

(𝜽 𝑯

)

𝐿𝑇𝑇𝐴(𝜃 𝐻 )

𝑻 𝑯

a photo of the

small {class

k

}

+

Test Image

…

updating LN

weights of the

vision encoder

every iteration

weight averaging

every L iterations

(M times)

𝑻 𝟎

A photo of

a {class

k

}

𝑻 𝑯

a photo of the

small {class

k

}

Adapted CLIP

…

𝑇1

…

𝑇𝐶

𝑇2

Text Embedding

Averaging

Test Image

𝐼 1

𝐼

1 . 𝑇

1

…

𝐼

1 . 𝑇 C

𝐼

1 . 𝑇

2

Class is Dog

itap of

a {class

k

}

𝑻 𝟏

Text

Encoder

𝑇 1

1

…

𝑇 𝐶

1

𝑇2

1

𝑇 1

2

…

𝑇 𝐶

2

𝑇2

2

…

Text Embeddings

𝑇 1

𝐻

…

𝑇 𝐶

𝐻

𝑇2

𝐻

All Templates

Weight

Averaged

Vision

Encoder

Similarity Scores

Final Prediction

Adaptation Phase

Evaluation Phase

weight averaging

every L iterations

(M times)

updating LN

weights of the

vision encoder

every iteration

updating LN

weights of the

vision encoder

every iteration

comparison, our method combines prompt augmentation and fine-tuning of normalization layers, highlighting its effectiveness in test-time adaptation.

Weight Averaging (WA) is a powerful train-time technique for improving deep neural network generalization. Stochastic Weight Averaging (SWA) [26] averages weights of multiple models sampled from different training epochs, aiding smooth optimization trajectory and convergence to points with superior generalization. SWAD [10] refines SWA by densely sampling weights throughout training, enhancing generalization and robustness across tasks. This train-time refinement enhances WA's effectiveness in producing models with improved generalization. The Lookaround [11] optimizer iterates between an 'around step' and an 'average step', building on SWAD's advancements. In the 'around step', independently trained models using various data augmentations explore a broader loss landscape to find flatter minima. In the 'average step,' weights of these models are then averaged, guiding optimization towards lower loss regions. This method enhances robustness and generalization across tasks, improving upon SWA and SWAD by providing a more effective weight averaging process.

In contrast to existing approaches, WATT leverages varied text prompts to adapt vision-language models such as CLIP during testing. Our method also harnesses the benefits of weight averaging while addressing domain shifts without additional model transformations or trainable modules, thereby setting a new precedent in test-time adaptation.

3 Method

The proposed WATT method, summarized visually in Figure 3 comprises three main components, the first two in the Adaptation Phase and the third in the Evaluation Phase: 1) a light-weight transductive TTA strategy that adapts CLIP's visual encoder effectively by considering the similarity between all

batch samples in terms of their visual and text features; 2) a weight-averaging strategy using multiple text templates to generate diverse model hypotheses during adaptation; 3) an ensembling technique that boosts performance during evaluation by averaging the embedding of different text templates.

3.1 Transductive TTA loss

While our method can be employed with any TTA framework, in this work, we implement a strategy inspired by the transductive TTA approach of CLIPArTT [24] which effectively incorporates semantic relationships among batch samples.

Initially, our process involves executing inference using CLIP, a system comprising a visual encoder f v θ ( · ) that translates an image x into visual features z v ∈ R D , and a text encoder f t θ ( · ) which converts text prompts t into text features z t ∈ R D . During inference, we employ pre-defined text prompts assigned to each class within a dataset, such as t 0 k = ' a photo of a {class k } '. For a new image x i , the likelihood of belonging to class k is then computed using cosine similarity:

where τ is a softmax temperature parameter set to 0 . 01 is this work. This prediction is then stored to be used as pseudo labels for the model.

Denoting the normalized visual embeddings of the samples within the test batch as Z v ∈ R B × D and the instance-specific text embeddings as Z t ∈ R B × D , we compute an image-to-image similarity matrix S v = Z v ( Z v ) ⊤ ∈ [ -1 , 1] B × B modeling pairwise relationships in terms of image characteristics. Similarly, we construct a text-to-text similarity matrix S t = Z t ( Z t ) ⊤ ∈ [ -1 , 1] B × B , capturing the semantic relationships among text embeddings within the batch. Utilizing the computed pairwise similarity matrices, we generate pseudo labels Q = softmax ( ( S v + S t ) / 2 τ ) ∈ [0 , 1] B × B which are used with cross-entropy in our transductive TTA loss:

Drawing a link with the Stochastic Neighbor Embedding (SNE) method for dimensionality reduction [27], which minimizes the KL divergence between distributions modeling pairwise distances, our TTA loss ensures that the inter-modality (text-to-image) similarities of batch samples are aligned with their intra-modality ones (text-to-text and image-to-image).

3.2 Multi-Template Weight Averaging

We explore various text prompt templates suggested in the CLIP paper and detailed in Fig. 1a. As reported in Table 1c, these prompts achieve varying performance across different corruption types of CIFAR-10-C. We formulate prompts of the form t h k = template h (class k ) , where h ∈ { 1 , 2 , . . . , H } , encompassing a spectrum of textual cues tailored to elicit diverse responses from the model.

Two different approaches are investigated for our multi-template weight averaging (MTWA) strategy. The first one denoted as Parallel MTWA (WATT-P) , which follows recent optimization approaches like Lookaround [11], performs the adaptation separately for each text template, starting from the same parameters, and then averages the resulting adapted weights. The second one, called Sequential MTWA(WATT-S) , instead considers text templates sequentially without resetting the weights. These two approaches, which we illustrate and compare in Fig. 4, are detailed below.

𝑻 𝟎

M iterations

(here M=2)

Initial

Weights

𝑻 𝟏

𝑻 𝟎

𝑻 𝟏

𝑻 𝟎

𝑻 𝟎

Final

Weights

𝑻 𝟎

𝑻 𝟎

𝑻 𝟏

𝑻 𝟏

𝑻 𝟎

𝑻 𝟎

WATT - S

Final

Weights

Initial

Weights

𝑻 𝟏

𝑻 𝟏

M iterations

(here M=2)

𝑻 𝟏

𝑻 𝟏

Parallel MTWA. This approach optimizes the TTA loss in (2) separately for H different models, each utilizing a distinct template. Starting from the same visual encoder parameters θ , these models are updated in parallel for L iterations, resulting in updated parameters θ ′ h , with h ∈ { 1 , . . . , H } . The parameters are reset after each update, enabling each model to restart the adaptation from the same initial point. Subsequently, we aggregate the weights obtained from these H models by computing their average: θ avg = 1 H ∑ H h =1 θ ′ h . We repeat this step M times, and denote the overall process as '(after L iter) × M '.

Sequential MTWA. Our Sequential MTWA approach is inspired from the work of [26], where the averaging of weights across various stages of the training process is employed to mitigate variance and enhance generalization capabilities. Instead of resetting parameters for each model, we update parameters after each template's iteration. To ensure impartiality in the update sequence of templates, a random selection process is implemented, thereby disregarding any predetermined order.

3.3 Evaluation Phase

At test-time, predictions are computed using Equation 1 through two distinct methodologies. In the first approach, the text features z t 0 are derived from the initial text prompt t 0 k = ' a photo of a class k ', denoted as single _ temp . Conversely, the second method aggregates the text features from all templates by computing their mean, resulting in the prediction z t ens = 1 H ∑ H h =1 z t h , denoted as text _ avg (see Fig. 3).

4 Experimental Setup

Settings. In line with prior TTA methodologies, adjustments are made to all Layer Normalization layers within the visual feature extractor for test-time adaptation. The Adam optimizer is employed with a fixed learning rate of 10 -3 , wheras a smaller learning rate of 10 -4 is chosen for adaptation to the 3D renderings split, as it reflects a more pronounced shift. Throughout our experimentation process, a consistent batch size of 128 is maintained to ensure uniformity and facilitate meaningful comparisons across various scenarios.

Datasets. Following [24], we rigorously evaluate WATT's performance across diverse TTA datasets using established assessment techniques. These datasets simulate intricate domain shifts, providing nuanced insights into our approach's effectiveness. Additionally, we explore WATT's adaptability on the original dataset through zero-shot test-time adaptation. To ensure a thorough examination, we

1

2

4

6

8

Number of templates

70

75

80

85

90

Test-time accuracy

CIFAR10

CIFAR10.1

CIFAR10-C average

0

10 20

50

80

100

Number of iterations

30

40

50

60

70

Test-time accuracy

CIFAR100

Defocus Blur

Frost

Contrast

CIFAR100-C average

extend our analysis to include various domain generalization datasets, exposing our method to a broad spectrum of image categories for comprehensive evaluation. Our evaluation framework encompasses natural images , common corruptions , simulated shifts , video shifts , texture shifts and style shifts .

In our assessment of natural image analysis, we include CIFAR-10, CIFAR-10.1, and CIFAR-100, each comprising 10,000 images and offering varied data distributions. CIFAR-10.1 [28] introduces a natural shift from CIFAR-10, providing a comprehensive evaluation of our model's performance. We also incorporate the CIFAR-10-C and CIFAR-100-C datasets [28], augmented with 15 distinct corruptions across 5 severity levels, resulting in 75 common corruption scenarios. This comprehensive augmentation assesses the model's resilience effectively.

Our investigation also extends to the VisDA-C dataset [29], challenging models with simulated and video shifts across diverse imagery types. Additionally, we evaluate our method on three datasets mostly used in the field of domain generalization: PACS [30], VLCS [31], and OfficeHome [32] datasets, instrumental in understanding texture and style variations. These evaluations effectively demonstrate the generalizability of our method across distinct domain shifts.

Benchmarking. We conduct a comparative analysis of WATT against contemporary methods using ViT-B/32 as the backbone. Specifically, we incorporate an adapted version of TENT [13], customized for CLIP by the authors of [24], using 10 iterations. We also include SAR [16] and MEMO [14] in the same manner as TENT. Additionally, we compare with TPT [8], a novel adaptation technique for CLIP that heavily relies on image augmentations, as well as DiffTPT [22], an extension that incorporates diffusion models. Lastly, we include TDA [23], which employs a dynamic adapter, and CLIPArTT [24], a recent approach that utilizes pseudo labels generated through conformal learning.

5 Results

In this section, we present empirical findings from our WATT method through a series of ablation studies aimed at understanding the impacts of individual components. These studies inform subsequent experiments across diverse datasets. Leveraging insights from ablations, we conduct comprehensive experiments, benchmarking WATT against state-of-the-art techniques across various datasets.

5.1 Ablation Studies

In this section, unless otherwise specified, we focus on the Sequential MTWA variant of our method (see Section 3.2) and will use these findings as a reference for the Parallel MTWA method.

Comparison of the template used during testing. After updating the model, we proceed to compute the similarity between the image features and the text embeddings, enabling prediction. Typically, text embeddings originate from the text prompt ' a photo of a class k '. However, by employing multiple templates, we have the flexibility to alter this text embedding through the averaging of all text embeddings from each template. Table 1 conducts a comparative analysis revealing that this

averaged text embedding consistently yields superior results across all scenarios. Hence, we adopt this approach for next experiments.

Comparison of the number of templates. In Figure 5, we examine the performance variation relative to the number of utilized templates. In this investigation, we conduct 5 runs wherein the templates are randomly selected from a pool of 8 distinct templates (as outlined in Fig. 1a). Notably, when the distribution shift is minimal, as observed in CIFAR-10 and CIFAR-10.1, optimal performance is attained using 6 templates, with performance gradually diminishing thereafter. Conversely, in scenarios characterized by substantial corruptions, such as CIFAR-10-C, employing all 8 templates proves advantageous. Consequently, our focus moving forward will be on utilizing all 8 templates in our work.

Text Averaging vs Output Averaging vs Weight Averaging. Utilizing the averaging method within a VLM offers several possibilities, including averaging the weights, the outputs or the text embeddings before computing the logits. In Table 3, a comparison between these approaches is presented. It becomes evident that weight averaging consistently outperforms text embedding averaging across various datasets, showcasing a superiority of approximately 1% even with the less effective weight averaging method. This performance advantage is observed across CIFAR-10, CIFAR-10.1, and CIFAR-10-C, and persists even with larger numbers of classes, such as in CIFAR-100 and CIFAR100-C. When concentrating on output averaging, the results may be less evident with less effective weight averaging methods. However, they remain valid and even more accurate with superior weight averaging techniques. Therefore, our focus for future experiments will be on weight averaging as the preferred approach.

Best moment to do the Weight Averaging. Examining Table 3, it is evident that the parameters L and M discussed in Section 3 are crucial. Specifically, a large L (e.g., 10) combined with a small M (e.g., 1) is ineffective. Conversely, setting L = 1 and M = 10 yields optimal results for small distribution shift datasets, while L = 2 and M = 5 perform best on highly corrupted datasets. Given that TTA typically encounters substantial distribution shifts, we will use L = 2 and M = 5 in our subsequent experiments.

Performance over the number of iterations. In this section, we focus on the method incorporating a Parallel MTWA mechanism and examine the impact of the number of iterations on performance. As illustrated in Figure 6, the accuracy stabilizes after approximately 20 iterations. Although there is a slight improvement in performance beyond 50 iterations, the difference is marginal. Based on these observations, we have opted to use 50 iterations for our experiments.

Model performance across various batch sizes. In our investigation, we delve into the performance implications of TTA methods when operating under small batch sizes, a historical challenge in the field. Table 2 provides insights into this aspect, revealing substantial performance enhancements with increasing batch sizes. Notably, our WATT model showcases remarkable adaptability, demonstrating performance improvements even with a single image input contrary to alternative methods. Specifically, we observe enhancements of approximately 1% for CIFAR-10 and CIFAR-10.1, and an impressive 2% for CIFAR-10-C when compared to baseline. Moving forward, we maintain a batch size of 128 in our experiments, aligning with prevalent practices observed in contemporary state-of-the-art methodologies.

5.2 Comparison to SOTA methods

Performance evaluation with small batches. In existing TTA works that study small batch size scenarios, such as SAR [16] and MEMO [14], we implement these methods in the context of CLIP. We compare our method with other TTA approaches using a batch size of 1. As shown in Table 5, WATT-P achieves the highest accuracy across all cases, outperforming SAR by 2.23% and MEMO by 0.75% on CIFAR-10.1. Notably, this improvement is achieved without any image augmentation, which is a common practice in previous TTA approaches that deal with small batches.

Performance evaluation in the presence of natural or no domain shift. In Table 4, results show consistent performance enhancements with WATT, both with the Parallel and Sequential MTWA strategies, alongside the baseline. On CIFAR-10, performance improves by 2.67% with Parallel MTWAand 2.31% using Sequential MTWA. On CIFAR-10.1, improvements reach 4.53% and 3.73%, and on CIFAR-100, enhancements are 8.70% and 9.06%. While WATT consistently outperforms the baseline, TPT, and CLIPArTT, TENT yields superior results on CIFAR-10. WATT's effectiveness often correlates with the number of classes, showing better performance with more classes, indicating its strength in lower-confidence scenarios.

Performance evaluation in the presence of common corruptions. Table 4 shows that both WATT variants consistently outperform alternative methods across various corruptions and class numbers. Notably, WATT with Parallel MTWA improves performance by 16.48% on CIFAR-100 Gaussian Noise and by 17.01% on Glass Blur compared to the baseline, while WATT with Sequential MTWA shows improvements of 17.27% and 17.96% respectively. On common corruptions, the Sequential MTWA variant surpasses Parallel MTWA, with improvements of 0.99% on CIFAR-10 and 0.89% on CIFAR-100. According to the TDA supplementary materials, we selected the weighting factor alpha as 5.0 and the sharpness ratio beta as 2.0, which are stated as optimal. However, these values did not appear to be the best choice for more challenging datasets like CIFAR-10-C or CIFAR-100-C that contain various corruptions. Adjusting these parameters based on the dataset would not be consistent

with the principles of a fully TTA method, which might explain their suboptimal performance in our results. Regarding DiffTPT, it generates 64 images per test image, making it challenging to use in a real-world TTA scenario. Similar to TDA, DiffTPT requires carefully chosen parameters to fit the dataset, whereas our method does not require dataset-specific tuning. This highlights the robustness and practicality of our approach in diverse real-world applications.

Performance analysis under simulated and video shifts. Results on the 3D (simulated shift) and YT (video shift) splits of VisDA-C demonstrate a significant improvement in accuracy with our proposed WATT method compared to pure CLIP. The Sequential MTWA variant achieves the highest accuracy on both the 3D and YT splits, with scores of 85.36% and 84.69%, respectively, surpassing other adaptation methods including TENT, TPT, and CLIPArTT (see Table 6).

Performance analysis under texture and style shifts. Results on the OfficeHome, PACS, and VLCS datasets are presented in Table 6. On average, our proposed WATT method, with Parallel and Sequential MTWA variants, improves performance across the different domains of OfficeHome, PACS, and VLCS compared to other methods. This highlights its robustness in addressing texture and style shifts, which are especially challenging compared to other domain shift variants.

6 Conclusion

We introduce WATT, a Test-Time Adaptation method tailored for Vision-Language Models. Our approach harnesses Weight Averaging with different text prompts and incorporates text embeddings averaging to bolster prediction accuracy.

Through an extensive ablation study, we scrutinized the efficacy of employing varied text prompts and weight averaging. Comparative evaluations across Test-Time Adaptation and Domain Generalization datasets underscored the superiority of our method, particularly in scenarios involving distribution shifts and zero-shot performance enhancements compared to state-of-the-art approaches.

Looking forward, investigating the potential of text prompts and weight averaging in classification opens up promising avenues for future exploration. Our methodology, with its focus on template manipulation, suggests potential avenues for extension, such as incorporating alternative class descriptors, yielding valuable insights for future research. Moreover, expanding Test-Time Adaptation to encompass diverse scenarios, including segmentation or object detection with Vision-Language Models, holds significant potential for advancing our comprehension of model adaptability and performance across varied tasks.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision , pages 388-404. Springer, 2022.

Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976-980. IEEE, 2022.

Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21152-21164, 2023.

Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 16155-16165, October 2023.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 14274-14289. Curran Associates, Inc., 2022.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems , 34:22405-22418, 2021.

Jiangtao Zhang, Shunyu Liu, Jie Song, Tongtian Zhu, Zhengqi Xu, and Mingli Song. Lookaround optimizer: k steps around, 1 step average. Advances in Neural Information Processing Systems , 36, 2024.

Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006.10963 [cs, stat] , January 2021. arXiv: 2006.10963.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. 2021.

Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 38629-38642. Curran Associates, Inc., 2022.

Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8344-8353, 2022.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world, 2023.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML) , 2020.

David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ismail Ayed, and Christian Desrosiers. Tttflow: Unsupervised test-time training with normalizing flow. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2125-2126, Los Alamitos, CA, USA, jan 2023. IEEE Computer Society.

Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A Efros. Test-time training with masked autoencoders. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Gustavo A Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Ismail Ben Ayed, and Christian Desrosiers. Clust3: Information invariant test-time training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6136-6145, 2023.

David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Nc-ttt: A noise constrastive approach for test-time training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6078-6086, June 2024.

Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2704-2714, October 2023.

Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14162-14171, June 2024.

Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Light-weight adaptation of clip to new domains at test time. arXiv preprint arXiv:2405.00754 , 2024.

Xiaosong Ma, Jie ZHANG, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 , 2018.

Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems , 15, 2002.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. 2019.

Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2018.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision , pages 5542-5550, 2017.

Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision , pages 1657-1664, 2013.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5018-5027, 2017.

WATT: Weight Average Test-Time Adaption of CLIP Supplementary Material

A Implementation

Our proposed WATT method is implemented in Python using the PyTorch (version 2.0.1) framework. All experiments were conducted on an NVIDIA V100 32 GB GPU. However, due to the effectively and lightweight nature of our method, it can be executed on less powerful GPUs. Specifically, the adaptation with a batch size of 128 using the ViT/B32 backbone requires up to 4 GB of memory, making it feasible to use on a wide range of GPUs. For fairness and consistency, we re-implemented and ran all other methods, including CLIPArTT, TENT, and TPT, in the same environment. Each experiment was performed three times to ensure reliability (three trials per experiment). To facilitate reproducibility, we provide the original implementation and detailed step-by-step instructions in our repository, accessible via this github link .

B Template details

The CLIP paper identifies 80 templates that enhance model robustness and performance. They ultimately conclude that 7 of these templates best summarize their model (see this link ). In our work, we use these 7 generic templates and add the common one, ' a photo of {} ,' based on their optimization. These templates are not specifically linked to the content of the images.

C Dataset details

In our investigation, we use the VisDA-C dataset, which challenges models with two distinct domain shifts: the simulated shift and the video shift. The simulated shift includes 152,397 3D-rendered images across 12 diverse classes, while the video shift comprises 72,372 YouTube video frames spanning the same categories. This dataset addresses the diversity of imagery types applicable to a model, posing a significant challenge.

Moreover, we evaluate our proposed method on three other datasets: PACS, VLCS, and OfficeHome. These datasets help understand various domain shifts, including texture and style variations. The PACS dataset consists of 9,991 images across four domains (Art, Cartoons, Photos, Sketches) and seven classes. The VLCS dataset contains 10,729 images across four domains (Caltech101, LabelMe, SUN09, VOC2007) and five classes. Lastly, the OfficeHome dataset includes 15,588 images across four domains (Art, Clipart, Product, Real) and 65 classes. Evaluating across these distinct domain shifts showcases the generalizability of our method. These datasets are more representative of real-world scenarios compared to CIFAR, with complex domain shifts.

D Computational Cost

We conduct a thorough evaluation under consistent conditions using an NVIDIA A6000 GPU within the same Python environment. The Table 7 provided compares the adaptation time, memory usage, and the number of learnable parameters for various TTA methods, including our proposed WATT method.The table clearly demonstrates that WATT-S, a sequential implementation of WATT, maintains competitive adaptation time and memory usage compared to other methods like TENT and ClipArTT, which are efficient but lack the robustness of WATT's method. Additionally, the table highlights that WATT-P, with parallel model training, offers a faster adaptation time than WATT-P with a for-loop implementation, albeit at the cost of higher memory usage. It's important to note that methods like DiffTPT [22] and MEMO [14], which show significantly higher adaptation times, employ offthe-shelf diffusion models and AugMix augmentation, respectively, resulting in time-consuming processes that may be impractical for real-world scenarios. In contrast, the effectiveness of our WATT-S method makes it better suited for scenarios where a robust, rapid, and resource-efficient adaptation is crucial.

E Pseudo-code of our both methods

In Algorithms 1 and 2, we compare the two variants of WATT: one with Parallel MTWA (WATT-P) and the other with Sequential MTWA (WATT-S). The WATT-P model recalibrates its parameters for each template using the average parameters of m -1 , whereas the WATT-S model updates its parameters solely at the start of each new iteration m .

F Additional Ablation Studies

Cross Entropy vs Entropy Minimization. Two unsupervised loss functions were integrated into previous TTA methods: classical entropy minimization and the loss introduced by CLIPArTT [24], where predictions are utilized as pseudo labels for cross-entropy computation. In Table 8, a comparison between these two loss functions is presented across the original CIFAR-10 dataset and various corruptions from CIFAR-10-C. It is observed that, for these specific corruptions, entropy minimization generally outperforms with the different templates employed, except for Gaussian Noise . However, upon assessing the weighted average accuracy, computed after 10 iterations for each template, cross-entropy consistently emerges as the superior option. The marginal impact of the weighted average on entropy minimization suggests that, irrespective of the template used, the model updates in a consistent direction to enhance confidence, rendering cross-entropy the preferred choice for subsequent experiments.

Algorithm 2 WATT-S - model f , parameter θ

1:

for

m

∈ {

1

,

2

, . . . , M

}

do

2:

θ

avg

1

H

∑

H

h

=1

θ

h

3:

f

f

θ

avg

4:

for

h

∈ {

1

,

2

, . . . , H

}

do

5:

for

l

∈ {

1

,

2

, . . . , L

}

do

6:

θ

h

←L

TTA

(

f

θ

avg

(

template

h

))

7:

end for

8:

end for

9:

end for

G Experiments with another VLM

We investigate if our TTA method is working with other VLMs like SigLip, so we compare if after adaptation with our method the performance improves compared to the baseline in Table 9.

H Experiments on other Visual Encoders

We replicated the experiments from the main paper using alternative visual encoders, ViT-B/16 and ViT-L/14.

Performance evaluation in the presence of natural or no domain shift. As indicated in the main results, employing WATT, both with Parallel and Sequential MTWA, consistently enhances

performance alongside the baseline. This pattern persists across different visual encoders, as shown in Tables 10. Although WATT consistently outperforms the baseline and TPT, TENT and CLIPArTT may occasionally yield superior results depending on the visual encoder used.

Performance evaluation in the presence of common corruptions. Table 10 demonstrates a consistent trend where both WATT methods consistently outperform alternative methods across various corruptions and class numbers. Upon closer examination of Table 10, specifically with ViT-B/16 as the visual encoder, Sequential MTWA exhibits a significant performance advantage, surpassing the baseline by 16.07% and the leading method in the state-of-the-art by 3.00%. This trend becomes even more pronounced with an increased number of classes, where Sequential MTWA surpasses the baseline and CLIPArTT by 16.94% and 8.87%, respectively.

Performance analysis under simulated and video shifts. Our study reveals substantial accuracy improvements on the 3D (simulated shift) and YT (video shift) partitions of VisDA-C when employing different backbones. This enhancement is particularly notable with our proposed WATT method compared to pure CLIP. Notably, the WATT-S variant achieves the highest accuracy across both the 3D and YT partitions, outperforming various adaptation approaches including TENT, TPT, and CLIPArTT. Detailed comparisons can be found in Tables 11 and 12.

Performance analysis under texture and style shifts. Findings across the OfficeHome, PACS, and VLCS datasets are detailed in Tables 11 and 12. Across these varied domains, our WATT method demonstrates consistent performance enhancements, as evidenced by both its WATT-P and WATT-S variants. These improvements underscore the efficacy of our approach in mitigating the complexities of texture and style shifts, which pose particular challenges compared to other forms of domain shift.

I Detailed Experimental Findings

This section provides extensive tables with detailed information on the results, which were summarized in the main body of the paper.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we claim that we introduce a new method using Weight Averaging for Test-Time Adaptation. In the paper, we present this method and prove its performance compared to state-of-art.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The work has been tested on many datasets with several runs. We also point out that with a small number of classes, it seems less effective compared to when there are many classes. Additionally, we show that it is not always the best when there is no domain shift.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our assumptions are writing in the Introduction and the Methodology , everything is explained with some illustrations.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the section Experimental Setup , we present all our different settings which are based on the CLIPArTT method. We also provide our code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Everything is detailed in the github link in the Implentation section in the supplementary material..

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Everything is detailed in the section Experimental Setup .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments have been conducted at least three times to ensure statistical robustness.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Everything is detailed in the section Implementation of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have considered all potential harms caused by the research process, societal impacts, and potential harmful consequences, as described in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper addresses a research field that is not currently utilized for direct practical applications, thereby limiting its immediate social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In our paper, we cite all sources that inspired our work, and we provide links in our git repository to the codes from which we drew inspiration.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document all aspects of our research and provide a link to our implementation, along with step-by-step explanations for reproducing the results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a Test-Time Adaptation technique named Weight Average Test-Time Adaptation (WATT) for Vision-Language Models (VLMs) such as CLIP. WATT is the proposed method that improves test-time adaptation by borrowing ideas from different text prompt templates to develop pseudo-labels for model updates and also carrying out weight averaging so as to consolidate learned information from those generated labels. The approach is tested across different datasets, revealing its capacity to boost performance without any further metamorphosis of the model or additional trainable modules.\n\nSTRENGTHS:\n+ The novelty of the study is that it proposes a novel test-time approach. It involves averaging weights while using varying text prompts which deviates from the conventional TTA techniques— making it a high-quality extension to existing TTA methods.\n+ It ensures quality by using rigorous experimental setup: involving thorough evaluations on various datasets, with good performance demonstrated by improvements over the current leading techniques.\n+ It maintains clarity in its presentation: both of methodology and results. This has been made possible by visual aids and detailed elaborations that ensure better understanding, thus though we still need to work more on this document to achieve perfection.\n+ The importance of the proposed method lies in its ability to adapt using a single image without requiring any further changes to the model; this level of adaptability is highly significant for real-world applications across different fields.\n\nWEAKNESSES:\n- Limited Scope of Evaluation: State-of-the-art Methods [1, 2] published in recent CVPR/ICCV are missing for comparing in Experimental part and discussing in the Related Work section.\n[1] Diverse data augmentation with diffusions for effective test-time prompt tuning. ICCV, 2023.\n[2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR, 2024.\n- Computational Cost: The paper mentions the efficiency of WATT but does not provide a detailed comparison of computational costs with other TTA methods, which could be crucial for practical deployment.\n- Generalization to Other Tasks: The paper focuses on image classification tasks. Extending the approach to other vision tasks like segmentation or detection and evaluating its performance there could provide a more comprehensive understanding of its applicability.\n\nQUESTIONS:\n1. Could you provide a more detailed comparison of the computational cost of WATT compared to other TTA methods?\n2. Have you considered evaluating WATT on real-world datasets with more complex domain shifts?\n3. How would WATT perform on other vision tasks such as segmentation or object detection?\n\nLIMITATIONS:\nThe authors have addressed the limitations and potential negative societal impacts adequately. They highlight the focus on image classification and suggest future work extending the method to other tasks. However, they could discuss more on the computational resource requirements and scalability in practical deployments.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper proposes a test-time adaptation (TTA) method for CLIP by integrating various textual prompt templates into Weight Average (WA) methods (Ref [11], [22]). Experiments on multiple types of domain shifts show the effectiveness of the proposed method.

STRENGTHS:
- The proposed method is effective yet simple.
- The experiments span different domain shift scenarios and demonstrate better performance of the method.

WEAKNESSES:
- The second contribution at the end of Section 1 lacks sufficient evidence. Table 4 only shows that the proposed method can handle small batch sizes without comparison with others. Additionally, there are existing TTA works studying small batch size scenarios. (SAR [Towards Stable Test-Time Adaptation in Dynamic Wild World, 2023], MEMO [MEMO: Test Time Robustness via Adaptation and Augmentation, 2022], etc.). Recent diffusion model-based TTA methods (such as DDA [Back to the Source: Diffusion-Driven Test-Time Adaptation, 2023]) also show robustness for small batch sizes.
- The technical contribution seems not that significant. The core idea is introducing multiple prompt templates into two previous WA methods, to my understanding.

QUESTIONS:
Is the evaluation done online (like the setting in the TENT paper where inference and adaptation happen on each batch in data streams), or does the model adapt with a single batch and then perform inference on the whole test set?

LIMITATIONS:
The authors mention limitations in the checklist.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposes a method called Weight Average Test-Time Adaptation (WATT) to improve test-time adaptation (TTA) for the CLIP model. The core idea is to use different text templates to construct multiple text prompts and adapt the model weights using these different prompts. During the evaluation stage, the prompts provide predictions based on the adapted weights and text embedding averaging across multiple prompts. The authors conduct comprehensive experiments that demonstrate the generalization and strong performance of the WATT method.

STRENGTHS:
* The idea is simple but works effectively.
* Comprehensive experiments support the effectiveness of this method in the following aspects:
Robustness, as reflected in Table 6.
Efficient adaptation, as reflected in Table 4.
Generalization and state-of-the-art comparison, as shown in Table 7.

WEAKNESSES:
There are no clear weaknesses, but I do have some questions and potential limitations that do not affect my rating.

QUESTIONS:
* Is it possible to combine text prompt augmentation (this method) with TTA strategies that use image augmentation to achieve better results?
Some templates may be incorrect. For example, as shown in Table 2, prompt T0: "a photo of a {class k}" and prompt T2: "a bad photo of the {class k}" refer to the same image, but why is this photo considered bad? If you add distortion or corruption, then it might be appropriate to call it bad. Also, for T6: "art of the {class k}," if you use diffusion to generate an art image of the original image, then the prompt would match.
* Can you measure the similarity score of these text templates in the CLIP text encoder space?

LIMITATIONS:
Can the authors test whether this method can also improve the latest VLM, such as SigLIP?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12