Title: Frustratingly Easy Test-Time Adaptation of Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision-Language Models

Test-Time Adaptation

Robustness

Model Calibration
Abstract: Vision-Language Models seamlessly discriminate among arbitrary semantic categories, yet they still suffer from poor generalization when presented with challenging examples. For this reason, Episodic Test-Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature on TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that a surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO (TTA with “zero” temperature), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature to zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses or compares favorably w.r.t. the state-of-the-art while being almost 10× faster and 13× more memory friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work in this field. Code will be available.

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

Abstract

Vision-Language Models seamlessly discriminate among arbitrary semantic categories, yet they still suffer from poor generalization when presented with challenging examples. For this reason, Episodic Test-Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature on TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that a surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO ( TTA with 'zero' temperature ), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature to zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses or compares favorably w.r.t. the state-of-the-art while being almost 10 × faster and 13 × more memory friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work in this field. The code is available.

1 Introduction

Groundbreaking achievements in Vision-Language pretraining [31, 14, 33, 39, 52] have increased the interest in crafting Vision-Language Models (VLMs) that can understand visual content alongside natural language, enabling a new definition of zero-shot classification. Despite huge pretraining databases [34, 37], VLMs still face limitations, suffering from performance degradation in case of large train-test dissimilarity [24] and requiring the design of highly generalizing textual templates [56].

Test-Time Adaptation (TTA) can effectively improve the robustness of VLMs by adapting a given model to online inputs. Among the various TTA setups (such as 'fully' [45], 'continual' [47] or 'practical' TTA [49]), Episodic TTA [53] is particularly appealing, as it focuses on one-sample learning problems and requires no assumptions on the distribution of the test data. When presented with a test image x , the parameters θ of a model f are optimized through a TTA objective L before inferring the final prediction, and reset afterward.

The choice of L is, ultimately, what characterizes TTA methods the most, with the recent literature being dominated by the objective of Marginal Entropy Minimization (MEM) [53]. Given a collection A of N ∈ N data augmentation functions, a test image x is first augmented N times to obtain a set of different views X = {A i ( x ) } N i =1 = { x i } N i =1 . The marginal probability distribution p w.r.t. sample x is then defined as the empirical expectation of Softmax-normalized model outputs over X , i.e. :

Under this framework, the Shannon Entropy of p is a bona fide measure of how inconsistently and uncertainly the model predicts over X , making it a tantalizing candidate to minimize, i.e. :

where C is the number of semantic categories. Once L ent is computed, (some of) the parameters of f are typically updated for a few steps of Gradient Descent before inferring the final prediction over the source input x with updated parameters. Owing to its simplicity and effectiveness, MEM has become a de facto standard in modern TTA [53, 38, 33, 19, 42, 27].

In this work, we take the opposite direction and challenge this paradigm. By conducting an in-depth theoretical and empirical investigation, we find that: ① while effective in improving model robustness, MEMhas little effect on the prediction of p ; ② no matter the dataset, the label space, or the parameter initialization, VLMs become much better classifiers when p replaces the standard inference protocol. Building on these insights, we show that a surprisingly strong and optimization-free TTA baseline is subtly hidden within the MEM framework. We term this baseline ZERO, which is short for TTA with ' zero ' temperature. Instead of tuning any parameters, setting to zero the Softmax temperature before marginalizing over views makes p already stronger than the model after MEM. Notably, ZERO only requires a single forward pass through the vision encoder and no backward passes.

Wrapping up, the contributions of this paper are the following:

We theoretically show when the prediction obtained through ¯ p ( i.e. , arg max p ) is invariant to MEM, and empirically verify that MEM has largely no effect on arg max p ;

We theoretically and empirically demonstrate that the error rate of p is a lower bound to the base error of a VLM in the setup of TTA. Additionally, we identify augmentations-induced overconfidence as the primal factor undermining the reliability of p ;

Motivated by these theoretical insights, we introduce ZERO, a frustratingly simple TTA approach that recovers the reliability of p by tweaking a single parameter of the model: the temperature;

We thoroughly evaluate ZERO following the established experimental setup with a variety of model initializations. Our results show that ZERO surpasses or compares favorably to state-ofthe-art TTA methods while being much faster and more memory efficient ( e.g. , 10 × faster and 13 × more memory efficient than the established Test-Time Prompt Tuning [38]).

2 Understanding Marginal Entropy Minimization

In this Section, we take a step towards both theoretically and empirically understanding the paradigm of MEM. In particular, this section is devoted to answering the following research questions:

How does MEM affect the marginal probability distribution? And, in turn

How does the marginal probability distribution relate to the standard inference protocol?

First, we introduce MEM for VLMs by reviewing the established Test-Time Prompt Tuning (TPT) method [38] and its notation. Then, in Sections 2.2 and 2.3 we answer the research questions above.

2.1 Preliminaries

Zero-Shot Classification with VLMs employs a predefined template ( e.g. , ' a photo of a ') from which a set of context vectors t ctx is obtained by looking up a token embedding table. Expanding

the template with the class names ( e.g. , ' a photo of a laptop. ' for the class ' laptop ') makes up the entire set of input vectors [ t ctx , t 1 ] , . . . , [ t ctx , t C ] , with t i being the embeddings derived from the i -th class name. A text encoder E txt transforms these class descriptions into independent and normalized text embeddings z txt 1 , . . . , z txt C and an image encoder E img encodes an input image x into a normalized latent vector z img . Lastly, classification is carried out by picking the class c corresponding to the text embedding z txt c holding the maximum cosine similarity with z img .

MEMfor VLMs. Pioneered by MEMO [53] in the scope of unimodal neural networks, MEM was repurposed for TTA with VLMs by Test-Time Prompt Tuning [38]. In [38], a VLM such as CLIP [31] is adapted at test time by minimizing the same objective of Eq. (2). In contrast to optimizing all model parameters, TPT relies on the effectiveness of prompt tuning [56, 55, 15], optimizing only the context vectors derived from the token embeddings of the standard CLIP template ' a photo of a '. By explicitly enunciating the dependency on the context vectors t ctx and re-using the notation of Sec. 1, one can re-write the MEM objective of [38] as:

Here, τ is the temperature of the Softmax operator. In the rest of this section, we omit the dependency on τ for simplicity, writing p ( ·| x , t ctx ) . Similarly to [53], the objective of Eq. (3) is minimized for a single step of Gradient Descent to update the set of context vectors. The updated context vectors, denoted as t ∗ ctx , are then used to prompt the VLM and obtain the final prediction for x . For any class c this is simply z img x · z txt c ( t ∗ ctx ) , which is easily transformed into p ( y = c | x , t ∗ ctx ) via Softmax.

2.2 How does MEM affect the marginal probability distribution?

The recent literature on TTA shows that minimizing L ent significantly enhances the robustness of model outputs. However, the impact of this process on the marginal probability distribution p remains unclear. We start with a straightforward hypothesis: due to its nature, minimizing L ent tends to increase the probability of the most probable class of p ( ·| x , t ctx ) . More formally, denoting with ˆ c the prediction of p ( i.e. , ˆ c = arg max p ( ·| x , t ctx ) ), we hypothesize that p ( y = ˆ c | x , t ∗ ctx ) > p ( y = ˆ c | x , t ctx ) . If this hypothesis is realized, it comes as a natural consequence that minimizing L ent is unlikely to alter the prevailing class of p , thus resulting in a consistent prediction pre- and post-TTA where arg max p ( ·| x , t ctx ) = arg max p ( ·| x , t ∗ ctx ) .

Hence, the first contribution of this work is to show that the prediction of the marginal probability distribution p is invariant to Entropy Minimization under loose constraints on confidence and gradients. To lighten the notation of the proposition, let us first define the following function g :

i.e. , the probability assigned to class c given a latent image representation z img and class-wise text embeddings z txt 1 , . . . , z txt C . Additionally, let δg ( c, z img ) be the negative variation incurred to the function g when the context vectors t ctx are updated through Entropy Minimization:

where, for clarity, the dependency of the text embeddings z txt 1 , . . . , z txt C on the context vectors (either t ctx or t ∗ ctx ) is explicit. Using this notation, we can formalize the following proposition:

Proposition 2.1. Let z img 1 , . . . , z img N be the latent image representations resulting from the N views and ˆ c = arg max p ( ·| x , t ctx ) be the initial prediction of the marginal probability distribution. If the entropy of p is minimized and p ( y = ˆ c | x , t ctx ) > 1 N ∑ N i =1 δg (ˆ c, z img i ) then the prevalent class of p is invariant to MEM, i.e. , arg max p ( ·| x , t ctx ) = arg max p ( ·| x , t ∗ ctx ) .

In Appendix A, we provide a detailed proof of this proposition, highlighting that δ (ˆ c, z img ) is directly linked to the gradient w.r.t. the context vectors t ctx. This relationship emerges when writing any postupdate text embedding z txt c ( t ∗ ctx ) as a function of its pre-update counterpart z txt c ( t ctx ) . Specifically,

we can write z txt c ( t ∗ ctx ) = E txt ([ t ctx -λ ∇ t ctx ( L ent ) , t c ]) , which is equivalent to E txt ([ t ctx , t c ]) -λ ∇ t ctx E txt ([ t ∗ ctx , t c ]) t ∇ t ctx ( L ent ) after a first-order Taylor Expansion around t ∗ ctx . Consequently, the proposition holds by a condition relating confidence (through p ( y = c | x , t ctx ) ) and gradients (through δ (ˆ c, z img ) ). Alongside the proof, Appendix A presents evidence supporting this proposition for CLIP [31] on the ImageNet-1k validation set [4], as well as across various datasets for natural distribution shifts: ImageNet-A [13], ImageNet-R [12], ImageNet-v2 [32], and ImageNet-Sketch [46].

2.3 How does p relate to the standard inference protocol?

From prior work on Test-Time Augmentations (TTAug) with unimodal neural networks [40, 35], empirical evidence suggests that p ( ·| x ) is more robust than p ( ·| x ) . This observation leads to the hypothesis that the expected risk of predicting with p is lower than that of doing so with p . However, the literature lacks guarantees for this hypothesis, except for the peculiar case in which the risk function is the squared error, i.e. , ℓ ( a, b ) = ( a -b ) 2 [16]. 2

As the second contribution of this study, we show that the error rate of p ( ·| x ) does indeed lower-bound the error rate of p ( ·| x ) . We do so by revisiting the theory of model ensembling, and showing that analogous ideas can emerge for TTA.

̸

Preliminaries on model ensembling. From the theory of classifier ensembling [18], we know that if f 1 , . . . , f N are N ∈ N independent classifiers with error rate ϵ and x is an example whose label is y ∈ { 0 , 1 } , then the probability that any group of k classifiers picks the same wrong label f i ( x ) = ˆ y = y can be expressed with a Binomial distribution wrapping N Bernoulli processes:

̸

Revisiting model ensembling for TTA. Eq. (6) holds as long as all events modeled as Bernoulli processes are independent. Thus, we have an equivalent error estimate for the setup in which only a single classifier f is present and X y = { x i } N i =1 is a set of independent examples with the same underlying label y . Within this framework, any group of k examples in X y to which the classifier has assigned the same label ˆ y is also a set of independent Bernoulli processes, whose error probability is still quantified via Eq. (6). Note that this resembles the TTA setup in the presence of N views of the source sample x , as long as augmentations do not change their underlying labels. We refer the reader to Appendix H for a discussion about the independence assumption among different views.

p is better than p (if f is calibrated). The final step can be taken through the lens of model calibration [8], a property requiring that the confidence of a classifier matches its accuracy. For example, a calibrated classifier f whose confidence is 0 . 7 is expected to be correct 70% of the times. In the previous discussion, if we denote with k ( y ) the number of examples correctly labeled as y , then the accuracy of the classifier is exactly k ( y ) /N . It follows that there is a positive correlation between accuracy and confidence if f exhibits good calibration, i.e., ↑ k ( y ) /N = ⇒↑ p ( y ) . Thus, the probability of picking the wrong class with this marginal probability is approximated by Eq. (6). Given this relationship, we have that p ( y ) = max p ( · ) if k ( y ) matches or exceeds the majority within N . Thus, the probability of picking the wrong class with p is approximated by marginalizing out all values of k that satisfy this criterion, which entails that the error or p can be expressed with the cumulative distribution of (6):

̸

From the Condorcet Jury Theorem [36], we know that Eq. (7) is a monotonically decreasing function if the error ϵ is better than random guessing, which is likely to be the case for VLMs pretrained on a massive amount of web data such as CLIP. Hence, we conclude that the error of p is a realistic lower bound for the base model error ϵ over a set of independent data points sharing the same label .

Does this lower bound empirically realize? We evaluate if the error of p consistently lower bounds the error of p also in practical use cases, where model calibration is unknown and the label space is large. For this, we use CLIP-ViT-B-16 [5], the ImageNet validation set, and four datasets reflecting

/negationslash

A

1k

Sketch

v2

R

Dataset

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Expected Error over Classes [%]

E

y

∼Y

[

P

ˆ

y

=

y

(¯

p

)

]

E

y

∼Y

[

/epsilon1

(

y

)]

(a) Error of

p

vs

ϵ

(

y

)

.

(b) Reliability diagrams for IN-1k and its augmented version.

ImageNet-1k

Gap

Augmented

1.0

ECE(↓):1.93%

ECE(↓):13.24%

Accuracy(↑): 66.73%

Accuracy(↑): 67.58%

0.8

0.6

A

0.4

0.2

0.0

0.00

0.25

0.50

0.75

1.00(

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

̸

Natural Distribution Shifts [12, 13, 32, 46]. For all classes in each dataset, we first draw all images sharing the same label ( X y ). Then, we compute the expected error ϵ ( y ) of the model on this subset, together with the error of p (ideally, Eq. (6)). Lastly, we average these errors over the entire label space Y . We do not restrict to the cases where y is supported by the majority and we do not re-organize predictions in a one-versus-all scheme. Fig. 1(a) clearly shows that the error of p is a lower bound to the base error of the model also in practical use cases where the label space is large and guarantees on model calibration are possibly missing. Importantly, this phenomenon persists no matter the dataset .

3 Simple and surprisingly strong TTA (for free)

The main point of Section 2.2 is that MEM generally does not affect the predominant class of the marginal probability distribution p . On the other hand, from Section 2.3 one can conclude that through p the model becomes a much stronger classifier. Summarizing:

̸

̸

̸

Chaining observations together, it emerges that:

̸

̸

̸

i.e. , if all assumptions are met, the error of MEM ≥ error of p after MEM = error of p without MEM. All in all, this TTA framework is hiding a surprisingly strong and optimization-free baseline: p ! Next, we highlight the detrimental impact of data augmentations on this marginal probability distribution and introduce a simple trick to recover its reliability: zeroing-out the Softmax temperature.

3.1 Augmentations undermine the reliability of p

While augmentations are essential in TTA to obtain multiple views of the test instance, noisy views may constitute Out-of-Distribution (OOD) data, thus having the undesired effect of un-calibrating the model. To sidestep this issue, one can attempt to discriminate between in-distribution ( w.r.t. to the pretraining data) and OOD views. Given that low confidence is a common trait in OOD data, a viable way to discriminate is confidence-based filtering, such as in TPT [38]. Formally, a smaller set of confident views are obtained following X filt = { x i ∈ X | H ( p ( ·| x i , t ctx )) < ρ } , where ρ is a

̸

threshold retaining the views whose entropy is in the bottom-10% percentile (lowest entropy). Despite its effectiveness, this filter cannot help when the reliability of p is undermined by overconfidence .

Augmentations lead to poor calibration. We demonstrate the impact of augmentation-induced overconfidence using the same model and datasets of Section 2.3. For each dataset, we generate an augmented counterpart following the augmentation and filtering setup of TPT [38], i.e. : we augment an input N = 64 times using simple random resized crops and horizontal flips. Then, we only retain 10% of the N views according to confidence-based filtering, resulting in 6 views per sample. Consequently, each augmented dataset contains 6 × more data points than its plain counterpart. The Expected Calibration Error (ECE) [8] reported in Appendix C conveys that ① zero-shot CLIP is well-calibrated (ECE < 0 . 1 for all datasets), strongly supporting the theory of Section 2.3 and ② the augmented visual space greatly increases the calibration error .

Poor calibration is frequently linked to overconfidence. We investigate the reason for the increase in ECE by presenting reliability diagrams for the ImageNet validation set in Fig. 1(b). In a reliability diagram, every bar below the identity line y = x signals overconfidence ( i.e. , the confidence on the x-axis prevails over the accuracy on the y-axis), while the opposite signals under-confidence. Notably, in the scope of our experiments, overconfidence is the primal factor leading to an increase in the ECE. The error rate, in contrast, decreases slightly. In Appendix C, we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on the 2B subset of LAION [2, 34]. Importantly, this phenomenon further persists within this extended experimental suite.

3.2 ZERO: Test-Time Adaptation with 'zero' temperature

Since its reliability is severely undermined by augmentations-induced overconfidence, directly predicting through p is not an enticing baseline for TTA. Concurrently, we also know that the error rate does not decrease when predicting over the augmented visual space. Hence, we are interested in finding an efficient way to capitalize on these observations: relying on the predictions over the views, while ignoring potentially misleading confidence information. The key is to note that both desiderata are obtained by explicitly tweaking a single parameter of the model: the temperature . Specifically, setting the temperature to (the limit of) zero corresponds to converting probability distributions into one-hot encodings, hence exclusively relying on their arg max when marginalizing. Inspired by this idea we propose ZERO, Test-Time Adaptation with ' zero ' temperature.

Procedure. ZERO follows these simple steps: ① augment, ② predict, ③ retain the most confident predictions, ④ set the Softmax temperature to zero and ⑤ marginalize. The final prediction is the arg max of the marginal probability distribution computed after 'zeroing-out' the temperature, i.e. :

Efficient Implementation. In all its simplicity, ZERO is computationally lightweight. In closed set assumptions where the class descriptions (and thus their embeddings) are fixed, ZERO only requires a single batched forward pass through the vision encoder, just as much as needed to forward the N views. Additionally, since the temperature is explicitly tweaked, ZERO needs no backpropagation at all and can be implemented in a few lines of code. For reference, a PyTorch-like implementation [30] is reported in Algorithm 1.

where ✶ is an indicator function, whose output is 1 if x i ∈ X filt and 0 otherwise, and X filt is the set of confident views before tweaking the temperature, i.e. , x i ∈ X filt if H ( p ( ·| x i , t ctx , τ ) < ρ .

Equivalent perspective and final remark. We bring to attention a simple scheme which corresponds to ZERO: voting over (confident) augmentations. Drawing from the theory of

Algorithm 1 PyTorch-style code for ZERO

ensembling, note that the error rate of the voting paradigm is exactly described by Eq. (6). Essentially,

this means that ZERO capitalizes on the theoretical insights while circumventing practical issues stemming from augmentations. We also highlight that ZERO is subtly hidden within any TTA framework relying exclusively on MEM, since computing p is inevitable therein. For this reason, we refer to ZERO as a baseline for TTA. Our goal diverges from introducing a 'novel' state-of-the-art method for TTA. In contrast, we advocate the importance of evaluating simple baselines.

4 Experiments

In this section, we present a comprehensive experimental evaluation of ZERO. Similarly to [38, 33, 54], we always work in the setup of single test point adaptation. Our results show that ZERO, alongside its simplicity, is an effective and efficient approach for TTA.

4.1 Experimental Protocol

Baselines. We compare ZERO to three strategies for TTA with VLMs: ① TPT [38], ② PromptAlign [33], and ③ Reinforcement Learning from CLIP Feedback (RLCF) [54]. As introduced in Section 2, TPT works by minimizing the entropy of p . In contrast, PromptAlign relies on a pretrained MaPLe initialization [15] and pairs the MEM objective with a distribution alignment loss between layer-wise statistics encountered online and pretraining statistics computed offline. Finally, RLCF does not include MEM in its framework; Zhao et al. [54] shows that, if rewarded with feedback from a stronger teacher such as CLIP-ViT-L-14 , the smaller CLIP-ViT-B-16 can surpass the teacher itself.

Models. As different approaches consider different backbones in the original papers, we construct different comparison groups to ensure fair comparisons with all TTA baselines [38, 33, 54].

Group 1: When comparing to TPT, we always use CLIP-ViT-B-16 . Shu et al. [38] also reports CLIP-Ensemble , i.e. , CLIP enriched with an ensemble of hand-crafted prompts. While the design of TPT does not allow leveraging text ensembles (as also pointed out by concurrent work [42]), ZERO seamlessly integrates with CLIP-Ensemble . We denote this variant with ZERO+Ensemble.

Group 2: When comparing to PromptAlign, we follow Samadh et al. [33] and start from a MaPLe initialization for a fair comparison. MaPLe prompts are learned on ImageNet, following [33]. Within this group, we also report TPT on top of MaPLe , as in [33].

Group 3: When comparing to RLCF, we use both CLIP-ViT-B-16 and CLIP-ViT-L-14 as in [54]. Specifically, confidence-based filtering acts on top of the output of the first model, and the selected inputs are passed to the second model for the final output. Both forward passes are inevitable in RLCF, so this scheme corresponds to 'early-exiting' the pipeline, exactly as per MEM. RLCF can vary according to (i) the parameter group being optimized and (ii) the number of adaptation steps. We denote with Θ v the full image encoder tuning, with t ctx prompt tuning, and with t the number

of adaptation steps. For example, RLCF t ctx t =3 indicates RLCF with prompt tuning for 3 TTA steps. Note that, since all methods need to forward more than one image to the teacher model, the zero-shot baseline of this group is exactly zero-shot classification with CLIP-ViT-L-14 .

Pretrainings. This Section deals with models officially released by OpenAI [28]. Appendix B further reports experiments with LAION-pretrained CLIP models [2], as well as the soft prompt initialization with supervised Context Optimization ( CoOp ) from [56].

Benchmarks. We follow the established experimental setup of [38, 33], evaluating ZERO on Natural Distribution Shifts and Fine-grained Classification (also referred to as 'Cross-Datasets Generalization" in previous works). For the former, we consider the ImageNet validation set and the four datasets for Natural Distribution Shifts already presented in Section 2, commonly considered Out-of-Distribution (OOD) datasets for CLIP. For fine-grained classification, we evaluate all TTA methods on 10 datasets. Specifically, we experiment with Oxford-Flowers (FWLR) [25], Describable Textures (DTD) [3], Oxford-Pets (PETS) [29], Stanford Cars (CARS) [17], UCF101 (UCF) [41], Caltech101 (CAL)[6], Food101 (FOOD) [1], SUN397 (SUN)[48], FGVC-Aircraft (AIR) [23] and EuroSAT (ESAT) [11]. For all of these datasets, we refer to the test split in Zhou et al. [56] as per the common protocol.

Textual prompts. When +Ensemble is specified, we do not use dataset-specific templates. In contrast, we use the set of 7 generic templates highlighted in the official CLIP repository [28] across all datasets. When adapting MaPLe , we stick to the ImageNet-learned prompts released by [15] and evaluate them cross-datasets as in [33].

Implementation Details. The augmentation pool A only contains random resized crops and random horizontal flips. The only hyperparameter of ZERO is the percentile for confidence-based filtering, which is set to 0 . 3 after validation on ImageNet (following standard practice [51]) and kept fixed for all datasets . We inherit the setup of TPT with N = 64 , crafting 63 augmentations to collate with the source image. To ensure hardware differences do not play any role in comparisons, we execute all TTA methods under the same hardware setup by running the source code of each repository with no modifications. We always use 1 NVIDIA A100 GPU and FP16 Automatic Mixed Precision. Results are averaged over 3 different seeds. Unless otherwise specified, all tables report top-1 accuracy.

4.2 Results

Natural Distribution Shifts. Results for Natural Distribution Shifts are reported in Table 1.

Group 1 (TPT): ZERO surpasses TPT consistently on all datasets . Among OOD datasets, peak difference is reached with ImageNet-A, where ZERO outperforms TPT by +4 . 84% . Enriching ZERO with hand-crafted prompts improves results further, with an average margin of +3 . 66% w.r.t. TPT.

Group 2 (PromptAlign): Within the second comparison group, ZERO outperforms PromptAlign on all datasets , with +1 . 68% being the gap in average performance. ZERO consistently outperforms TPT also when the baseline initialization is MaPLe (by an average of +2 . 92% ). Please note that we omit evaluation on ImageNet for this group, since PromptAlign adopts token-level statistics from this dataset when adapting to test points, which would render the comparison unfair. For completeness, we report that zero-shot MaPLe achieves an accuracy of 70 . 72% on ImageNet, which is improved to 72 . 99% by adapting with ZERO ( +2 . 27% ).

Group 3 (RLCF): We follow [54] and report RLCF variants with t = 3 steps. In this group, ZERO outperforms RLCF in 5 out of 5 datasets, with a gap in the average performance of +1 . 25% . Importantly, RLCF is only close to ZERO with image encoder tuning; only prompt tuning is insufficient.

Fine-grained Classification. Results for fine-grained classification are shown in Table 2. To foster readability, the standard deviations of ZERO are separately reported in Table 11 (Appendix).

Group 1 (TPT): Default ZERO improves over the zero-shot baseline CLIP-ViT-B-16 , but is outperformed by TPT with an average margin of -0 . 57% . However, extending ZERO with hand-crafted prompts (something that TPT cannot do by design ) is sufficient to outperform TPT on 7 out of 10 datasets, and obtain an average improvement of +0 . 74% .

Group 2 (PromptAlign): On average, PromptAlign has an improvement of +0 . 5% over ZERO. However, note that this is mostly influenced by the performance on one dataset only (EuroSAT) and that, in contrast, ZERO surpasses PromptAlign in 7 out of 10 datasets . In line with the previous benchmark, ZERO better adapts MaPLe than TPT, again outperforming it in 7 out of 10 datasets.

Group 3 (RLCF): As Zhao et al. [54] do not report results on fine-grained classification, we use their code to evaluate four RLCF variants: Θ v and t ctx tuning, with t = 1 and t = 3 adaptation steps. We find that ZERO largely outperforms RLCF regardless of the configuration. Even with respect to the strongest RLCF Θ v t =3 variant, ZERO obtains an average improvement of +2 . 28% .

Computational Requirements. The complexity of ZERO does not scale linearly with the size of the label space, as it does for prompt-tuning strategies. To quantify the computational gain of ZERO w.r.t. other TTA methods, we report the runtime per image and peak GPU memory in Table 3 under the same hardware ( i.e. , 1 NVIDIA RTX 4090. We compare the computational requirements of ZERO to TPT and the RLCF pipeline in a worst-case scenario where the label space is large (ImageNet). We omit PromptAlign from our analysis since it has slightly worse computational performance than TPT.

ZERO is 9 . 5 × faster than TPT taking 12 . 61 × less memory, corresponding to an order of magnitude of computational savings in both time and space. Concerning the slowest RLCF variant (prompt tuning), ZERO is 15 × faster and takes 7 . 22 × less memory. In the faster RLCF Θ v , text classifiers are also cached; nevertheless, ZERO is 2 . 25 × faster and 3 . 5 × more memory friendly.

5 Related Work

Closest to our work is a recent and very active research thread focusing on Episodic TTA with VLMs [38, 33, 54, 42]. As discussed in the manuscript, these methods mostly rely on prompt learning, a parameter-efficient strategy that only trains over a small set of input context vectors [20]. Narrowing down to VLMs, notable examples of prompt learning approaches include CoOp [56], CoCoOp [55], and MaPLe [15]. Episodic TTA has also been explored with traditional unimodal networks, such as ResNets [10], where MEM is still a core component [53]. In this context, MEM has recently been enriched with sharpness- [27] or shape-aware filtering [19]. Due to its nature, Episodic TTA is completely agnostic to the temporal dimension and is powerful when no reliable assumptions on the test data can be taken. Some other works relax these constraints and integrate additional assumptions such as batches of test data being available instead of single test points [45]. When test data are

assumed to belong to the same domain, one can rely on various forms of knowledge retention as a powerful mechanism to gradually incorporate domain knowledge [21, 22] or avoid forgetting [26]. The synergy between TTA and retrieval is also emerging as a powerful paradigm when provided with access to huge external databases [9, 50]. We particularly believe this can be a promising direction.

Closely related to our work are also Test-Time Training (TTT) and TTAug. In TTT the same one sample learning problem of Episodic TTA is tackled with auxiliary visual self-supervised tasks, such as rotation prediction [43] or masked image modeling [7], which require specialized architecture heads and are not directly applicable to VLMs. TTAug has recently been theoretically studied [16]. It boils down to producing a large pool of augmentations to exploit at test time [35], or to learn from [44]. In all its simplicity, ZERO can be seen as a strong TTAug baseline for VLMs, which, differently from concurrent work [51], does not involve any form of optimization.

6 Limitations

ZERO can seamlessly adapt a wide range of VLMs on arbitrary datasets without requiring extensive computational resources and is backed by theoretical justifications. However, we delineate four major limitations to our method which we report here.

Preliminary observations. The first limitation concerns the preliminary observations which led to ZERO, such as augmentation-induced overconfidence or a comparable error rate between source and augmented datasets. These observations may not persist if VLMs or benchmarks change significantly in the future, potentially leading to poor adaptation. For example, we have observed a consistent failure case for TTA with EuroSAT [11], with ZERO incurring large performance drops w.r.t. simple zero-shot classification. In Appendix F we unravel this worst-case further.

Theoretical assumptions. The second limitation stems from theoretical assumptions, the core one being the invariance of the marginal probability distribution to marginal entropy minimization. While our proposition guarantees invariance if entropy is globally minimized and the negative variation to the probability of the most probable class is less than the initial probability itself, these theoretical assumptions may not hold all the time. In this work, we supported our assumptions with empirical verification but, as per the first limitation, these may not extend to the space of all models and datasets. We refer the interested readers to Appendix A for a more in-depth discussion about the invariance of the prediction of p to MEM.

Independence among views. A third worthy-of-note limitation relates to the independence assumption among the views from which the marginal probability distribution is obtained. As we discussed in Section 2.3, the views themselves do not have any direct dependency, but they are still partially related through the source image from which they stem. Related to this limitation, we hypothesize that extending ZERO in a Retrieval-Augmented TTA setup (or a cache-based one) could improve the results. The discussion on this topic is extended in Appendix H.

Linear complexity with respect to augmented views. Finally, despite being much lighter than the current state-of-the-art TTA strategies, ZERO's computational requirements in the visual branch scale linearly with the number of views, since all of them need to be independently forwarded. On this, we believe that exploring how to augment directly in the latent visual space to also circumvent the forward pass of the vision encoder is an intriguing direction.

7 Conclusions

We theoretically investigated Marginal Entropy Minimization, the core paradigm of the current research on Test-Time Adaptation with VLMs. Building on our theoretical insights, we introduced ZERO: a frustratingly simple yet strong baseline for TTA, which only relies on a single batched forward pass of the vision encoder. ZERO works by setting the temperature of the Softmax operator to 'zero' before marginalizing across confident views, which is equivalent, in terms of output, to the widely known paradigm of majority voting. Our experimental results on Natural Distribution Shifts and Fine-grained Classification unveil that ZERO favorably compares to state-of-the-art TTA methods while requiring relatively negligible computation. We hope our findings will inspire researchers to push the boundaries of TTA further.

Acknowledgements. The authors acknowledge the CINECA award under the ISCRA initiative for the availability of high-performance computing resources and support. Matteo Farina is supported by the PRIN project 'LEGO-AI' (Prot.2020TA3K9N) and the PAT project 'AI@TN". This work was supported by the projects EU Horizon ELIAS (No. 101120237), AI4TRUST (No.101070190), FAIR - Future AI Research (PE00000013), funded by NextGeneration EU, and carried out in the Vision and Learning joint laboratory of Fondazione Bruno Kessler and the University of Trento, Italy.

References

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In European Conference on Computer Vision (ECCV) , 2014.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2020.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W) . IEEE, 2004.

Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML) , 2017.

Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In International Conference on Learning Representations (ICLR) , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7), 2019.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML) , 2021.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Masanari Kimura. Understanding test-time augmentation. In International Conference on Neural Information Processing (ICONIP) . Springer, 2021.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for finegrained categorization. In IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W) , 2013.

Ludmila I Kuncheva. Combining pattern classifiers: methods and algorithms . 2014.

Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In International Conference on Learning Representations (ICLR) , 2024.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.

Zichen Liu, Hongbo Sun, Yuxin Peng, and Jiahuan Zhou. Dart: Dual-modal adaptive online prompting and knowledge retention for test-time adaptation. In AAAI Conference on Artificial Intelligence (AAAI) , 2024.

Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems (NeurIPS) , 2024.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does clip's generalization performance mainly stem from high train-test similarity? In International Conference on Learning Representations (ICLR) , 2023.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing . IEEE, 2008.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning (ICML) , 2022.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations (ICLR) , 2023.

OpenAI. Clip. URL https://github.com/openai/CLIP .

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2012.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML) , 2019.

Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Lloyd Shapley and Bernard Grofman. Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice , 1984.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jongwook Son and Seokho Kang. Efficient improvement of classification accuracy via selective test-time augmentation. Information Sciences , 2023.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. arXiv preprint arXiv:2403.12952 , 2024.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML) , 2020.

Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean-Philippe Thiran. Tesla: Testtime self-learning with automatic adversarial augmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR) , 2021.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2010.

Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Luca Zancato, Alessandro Achille, Tian Yu Liu, Matthew Trager, Pramuditha Perera, and Stefano Soatto. Train/test-time adaptation with retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of visionlanguage models: Do we really need prompt learning? arXiv preprint arXiv:2405.02266 , 2024.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with CLIP reward for zero-shot generalization in vision-language models. In International Conference on Learning Representations (ICLR) , 2024.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV) , 2022.

A Marginal Entropy Minimization does not influence arg max p

A.1 Proof of Proposition 2.1.

Proof. Let us denote the pre-TTA p init ( c ) = p ( y = c | x , t ctx ) and the post-TTA p end ( c ) = p ( y = c | x , t ∗ ctx ) , i.e. , the marginal probabilities before and after optimizing t ctx. Let c init and c end denote the predictions before and after TTA, i.e. , c init = arg max p init and c end = arg max p end .

To simplify the notation, let us use z ∗ txt to write any post-TTA text embedding z txt ( t ∗ ctx ) .

̸

Under the assumption that entropy is minimized (the optimal scenario for MEM), we have p end ( c end ) = 1 , and p end ( c ) = 0 ∀ c = c end .

Let us rewrite the final distribution p end using the function g introduced in Sec.2.2. Specifically, for any class c , we have:

Performing a first-order Taylor expansion on g , we have:

We can also write any post-TTA text embedding z txt c ( t ∗ ctx ) as a function of the text encoder E txt prompted with optimized context vectors:

Through another first-order Taylor expansion (this time on z txt c ( t ∗ ctx ) ), we have:

leading to an equivalent re-writing:

Substituting (15) into (12), we can express g as follows:

with d k denoting the k -th entry of the C dimensional vector d . From (12) and (16) the negative variation δg ( c, z img ) to g before and after MEM can be expressed as:

Finally, for any class, we can rewrite its final probability p end as a function of its initial probability p init and the variation of g before and after TTA for the same class:

From Eq.(18) we have that if p init ( c init ) > λ N ∑ i δg ( c init , z img i ) , then the final probability p end ( c init ) > 0 . In the optimal case for MEM the entropy of p end is minimized, which entails that only one class can have a probability strictly greater than 0. Hence, c init = c end .

A.2 Experimental verification

We support the previous proposition with empirical evidence, by manually counting how often the prediction of p is invariant to Test-Time Prompt Tuning by MEM. This experiment is easy to reproduce and consists of the following: augment N times, filter by confidence, compute p init , optimize by MEM, compute p end and check if arg max p init = arg max p end . We report the proportion of samples for which the proposition holds for all Natural Distribution Shifts datasets in Table 4, averaged over 3 runs with different seeds (the same used in Sec. 4 of the main body). Although the proposition only accounts for the cases where entropy is globally minimized, the table shows that the marginal probability distribution is largely invariant to MEM. In the best case (ImageNet-Sketch) MEM alters the prediction of p only 8 . 77% of the times. In the worst case (ImageNet-R), the prediction is unaltered for 96 . 78% samples.

A.3 Can invariance be anticipated?

In the proof of Proposition 2.1, we express the post-MEM embeddings as a function of the pre-MEM embeddings through a Taylor expansion. For this relationship to hold, the variation needs to be small. If the initial entropy is high, the gradients from MEM (and, thus, the variation between pre- and post-MEM embeddings) can be larger than what a Taylor expansion can accurately approximate. In such cases, Prop. 2.1 cannot be guaranteed. We execute a simple experiment using the validation set of ImageNet-1k, whose recipe is described below, to visualize this relationship.

We compute pre- and post-MEM marginal probability distributions. We sort the pre-MEM distributions in order of descending entropy (most to least uncertain) and quantize them into 10 bins. Bins shall be interpreted as follows: the leftmost bin contains the top 10% of samples with the highest entropy; the second bin contains samples outside the top-10% percentile but within the top-20%, and so on; the rightmost bin contains the bottom 10% of samples with the lowest entropy. For each bin we compute the invariance ratio, measuring how often the arg max of the pre-MEM p does not change after MEM. Finally, we display a histogram with this data in Figure 2.

Atrend appears: as the entropy decreases (left to right), invariance holds more and more often. Hence, intuitively, the most likely cases where invariance to MEM does not hold are those of high uncertainty in the initial marginal probability distribution. However, this may still be rare: even within the top 10% of most uncertain samples, invariance holds more than 82% of the time (leftmost bin).

1.0

0.9

Invariance ratio

0.8

0.7

0.6

Overall Invariance = 0.9573

0.5

2.831.861.41

1.1

0.840.62

0.41

0.25

0.12

0.04

Mean H(p init) per bin

B Additional Experiments: LAION-2B Pretraining, Context Optimization and Hyperparameter Inheritance

This Appendix deals with enriching the experiments of Section 4, which focused on models officially released by OpenAI [28]. Here we focus on the comparison with TPT [38] and extend the analysis to: ① CLIP-ViT-B-16 pretrained on the 2B English Subset of LAION-5B [34]; ② OpenAI's CLIP, transferred after supervised Context Optimization ( CoOp ) [56].

Implementation Details. For the experiments with LAION Pretraining, we use the open_clip repository, i.e. , the official code for [2]. The pretrained keyword for this model is laion2b_s34b_b88k . For CoOp we use the context vectors learned on ImageNet-1k officially released by [56]. The experimental setup is analogous to Section 4 in all details. We do not tune any hyperparameters for these different initializations, but inherit them from the experiments with OpenAI models.

B.1 LAION-2B Pretraining

Table 5 reports experiments on Natural Distribution Shifts, from which we observe no differences w.r.t. OpenAI models: ZERO largely outperforms TPT, and peak difference is reached with ImageNet-A [13]. Results on Fine-grained Classification are given in Table 7. We observe that ZERO improves the zero-shot baseline better with this pretraining, and overcomes TPT with an average margin of +0 . 4% . In contrast, ensembling textual prompts appears less effective. We speculate this is because the 7 templates were explicitly tuned and selected for OpenAI models. The worst-case scenario is confirmed with satellite imagery [11]; please refer to Appendix F for a deeper investigation.

B.2 Context Optimization (CoOp)

For this comparison, we follow [38] and report CoOp on Natural Distribution Shifts only, presenting results in Table 6. We further observe patterns consistent with OpenAI models, with ZERO providing large improvements over TPT. Also here, the best-case scenario persists with ImageNet-A.

B.3 Hyperparameter Inheritance

In all experiments so far, including Section 4 as well as Tables 5, 6 and 7, we employed a percentile for confidence-based filtering set to 0.3. This value was obtained after validation on ImageNet-1k with OpenAI's CLIP-ViT-B-16 and kept fixed for all models and datasets. Here, we show that ZERO obtains favorable performance even if the percentile for confidence-based filtering is not tuned in any way, but set to 0.1 by 'inheriting' the value used in TPT [38]. These results are given in Tables 8 and 9. Surprisingly, some datasets within the Natural Distribution Shifts benchmark benefit from this more restrictive filtering (ImageNet-A above all), while we observe that Finegrained classification tends to improve when more views are retained. The core findings, however, are entirely unchanged: the best

case remains ImageNet-A, the worst-case remains EuroSAT, and ZERO outperforms competitor in most datasets, no matter the experimental setup.

C Calibration and Overconfidence of CLIP on augmented Natural Distribution Shifts

In Section 3.1 of the manuscript, the validation set of ImageNet-1k is shown to convey that overconfidence emerges as a critical issue when predicting over augmented views. In this appendix, we expand the analysis to the 4 datasets for robustness to Natural Distribution Shifts (NDS) [13, 12, 32, 46]. For all datasets, we follow the augmentation setup of Sec.3.1, and generate augmented counterparts with 6 × more examples.

First, let us define the calibration of DNNs. Calibrating DNNs is crucial for developing reliable and robust AI systems, especially in safety-critical applications. A DNN is perfectly calibrated if the probability that its prediction is correct ( ˆ y = y ) given a confidence score random variable S is equal to its confidence score. The confidence score is commonly taken as the maximum of the output probability vector of the model, i.e. , s = max p ( · ) :

0

5

10

15

20

25

Expected Calibration Error [%]

IN-A

IN-1k

IN-Sketch

IN-v2

IN-R

8.3

1.9

4.9

3.0

3.6

22.2

13.2

20.2

15.4

9.6

source

augmented

To evaluate the expected calibration error (ECE), we typically split the dataset into M bins B m based on their confidence scores. We then calculate the accuracy of each bin, denoted as acc ( B m ) , and the average confidence, denoted as conf ( B m ) . The ECE is defined by the following formula:

Then, we show how the ECE of CLIP-ViT-B-16 varies between 'source' and augmented versions of all datasets (ImageNet-1k included) in Figure 3. From this experiment, we observe a large increase in the ECE across all datasets. In no cases, the ECE remains comparable to its default value when no augmentations are present. As we discussed in 3.1, the calibration error increases when the model is either more accurate than confident (signaling underconfidence ) or the opposite, signaling overconfidence . Reliability diagrams are a standard tool to understand which is the case, hence we show them for all 4 NDS Datasets in Fig.4. These results are entirely consistent with Sec.3.1: the calibration error increases exclusively due to overconfidence, no matter the dataset. In parallel, the error rate of CLIP-ViT-B-16 can either remain close to its default value ( e.g. , ImageNet-Sketch), slightly decrease ( e.g. , ImageNet-R and -v2) or largely decrease (ImageNet-A). We observe an identical pattern for CLIP models pretrained on LAION. For reference, see Figure 5.

D On the expected risk of p and p .

The expected risk of a classifier f is commonly defined as the expectation of the risk function ℓ over the joint distribution of data and labels.

In [16], the expected risk of a classifier f ( x ) = p ( ·| x ) , which predicts by marginalizing over several augmented views, is theoretically shown to lower-bound the empirical risk of a standard classifier f = p ( ·| x ) when the risk function ℓ is a squared error, i.e. , ℓ ( a, b ) = ( a -b ) 2 .

1.0

Gap

ImageNet-A

Augmented

1.0

ECE (↓):8.34%

ECE(↓):22.24%

0.8

0.6

0.4

0.2

0.0

ImageNet-v2

Augmented

1.0

ECE(↓):3.05%

ECE(↓):15.39%

0.8

Accuracy(↑):62.56%

0.6

0.4

0.2

0.0

ImageNet-R

Augmented

1.0

ECE(↓):3.55%

ECE (↓): 9.59%

0.8

0.6

0.4

0.2

0.0

ImageNet-Sketch

Augmented

1.0

ECE (↓):4.87%

ECE(↓):20.16%

0.8

Accuracy(↑): 46.09%

Accuracy(↑): 46.64%

0.6

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

Confidence

Confidence

Here, we show that such a bound can be extended to any risk function ℓ that checks the triangular inequality. Specifically, note that if ℓ satisfies the triangular inequality, then:

ImageNet-1k

Gap

Augmented

1.0

ECE(↓):8.55%

ECE(↓):23.24%

0.8

0.6

0.4

0.2

0.0

0.00

0.25

0.50

0.75

1.000.00

0.25

0.50

0.75

1.00

Confidence

Confidence

The above inequality is obtained following these simple steps:

Applying the expectation operator E over the joint distribution P XY to both sides of Eq.(22) leads to:

Hence, the empirical risk of p lower-bounds that of p for any risk function ℓ satisfying the triangular inequality.

E Tie breaking with ZERO

A caveat of ZERO are ties, i.e. , cases with multiple classes having identical probability within the marginal probability distribution. This is clear to see when viewing ZERO as its equivalent paradigm of voting among confident views, simply because more than one class may have an equal amount of 'votes'. Throughout all the experiments of this work, ties are broken greedily . If a tie results from the top views, the procedure for breaking it follows these two steps: ① sort the remaining views by ascending entropy (most to least confident) and ② scan the views until a prediction is encountered that breaks the tie. Other than this, many alternative are possible, such as relying on the most confident prediction. Specifically, we have explored the following alternatives:

greedy tie breaking, as discussed above;

relying on the most confident prediction;

computing several marginal probabilities for p , each by marginalizing over views with identical predictions, and picking the one with the lowest entropy for the final decision;

relying on the maximum logit (pre-Softmax);

using the averaged logits (pre-Softmax);

doing similar to point 2, using logits instead of probabilities;

random tie breaking;

and did not find consistent behaviour across all (fine-grained and NDS) datasets, suggesting this is indeed a minor component. We opted for greedy tie breaking due to its slightly better performance on the ImageNet validation set.

F A failure mode for TTA: satellite imagery

In our experiments, we find that an extremely OOD domain represents a consistent failure mode for TTA: satellite imagery. In all comparison groups, a zero-shot baseline largely outperforms any TTA strategy when evaluated on EuroSAT[11]:

in Group 1 the zero-shot baseline CLIP-Ensemble largely outperforms the best TTA strategy ZERO+Ensemble;

in Group 2 , zero-shot MaPLe outperforms PromptAlign;

in Group 3 the best RLCF Θ v t =1 pipeline lies far behind the zero-shot teacher CLIP-ViT-L-14 .

Here, we qualitatively and quantitatively report two main root causes for failures.

Qualitatively poor augmentations. In principle, TTA methods should rely on generic data augmentations, since not doing so would require going against the principles of the field by assuming some prior knowledge about the test data is available. As discussed in Sec.3, data augmentations are a doubled edged sword in TTA, and failing in crafting properly augmented views can potentially generate misleading or uninformative visual signals. We report some qualitative examples conveying this problem in Figure 7. In the Figure, we report three images from [11], together with the top-3 augmented views leading CLIP-ViT-B-16 to its most confident predictions. Each source image is reported with the groundtruth, and all views are reported with both the prediction and the confidence of CLIP. Visually, one can perceive that the simple data augmentation scheme of cropping and flipping, which has largely been proven successful in [38, 33] and in our work, does not provide informative views, since most are alike one another.

Quantitatively high error. Augmentations are used by all TTA methods discussed in this paper, hence the previous discussion holds for TPT as much as it does for PromptAlign, RLCF or ZERO.

Nevertheless, we highlight an additional caveat about satellite imagery which is particularly detrimental for ZERO, and relates to the base model error over augmentations. Recall that, in ZERO, the usage of p is backed by theoretical motivations, and the manual adaptation of the temperature is supported by two concurrent observations: augmentations-induced overconfidence and a comparable error rate between source and augmented images. Simply put, the latter condition is not verified for satellite imagery. To show this phenomenon, we follow the experimental setup of Sec.3.1 and examine the reliability

EuroSAT

Gap

Augmented

1.0

ECE (↓): 7.28%

ECE(↓):22.33%

Accuracy(↑):35.21%

0.8

0.6

0.4

0.2

0.0

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Confidence

Confidence

diagrams of EuroSAT[11] and of its augmented counterpart in Figure 6. As per Section 3.1, we display the ECE and the Top1-Accuracy on each version of the dataset. From this perspective, one can note that the base model error largely increases, in this domain, when augmented views are present. The accuracy on source images is 42 . 01% , dropping to 35 . 21% simply due to augmentations.

Both observations, combined, suggest that crafting augmentations for satellite imagery requires an ad-hoc treatment, which makes it a controversial benchmark for TTA.

G Natural Distribution Shifts vs Fine-grained Classification

Throughout the manuscript, one can observe that ZERO consistently provides larger improvements in Natural Distribution Shifts than it does in the Finegrained suite. We thus devote this section to digging deeper into this matter.

Perhaps unsurprisingly, we posit that ZERO improves over the zero-shot baseline if the zero-shot error rate of the model does not largely increase with augmented views. As Fig.1(b) displays, this is the case for all Natural Distribution Shifts datasets. To understand any different behaviors, we repeat the same experiment of Section 3.1 for the entire Fine-grained suite and report the results in Table 10.

Please note that, in the table, the percentile for confidence-based in filtering in ZERO is set to 0 . 1 , since the protocol for generating the augmented datasets follows the setup of TPT, which also uses a cutoff percentile of 0 . 1 , and that we omit EuroSAT since an analogous experiment was presented in the previous Appendix.

Overall, we observe a strong correlation between the error gap and the improvement provided by Zero, with Spearman's coefficient being -0 . 95 across datasets. This result shows that the correlation is negative, i.e. , the lower the error gap, the larger the improvement (or, in other words, the better the zero-shot performance on augmented views, the larger the improvement of ZERO). This pattern is also consistent with the experiments on EuroSAT reported in the previous Appendix. Understanding why augmentations induce larger or smaller errors may be a case-by-case matter that relates to the nature of the datasets. Here, we pinpoint two possible reasons:

The semantic space of the ImageNet variants of the Natural Distribution Shifts benchmark comprises many common categories, which may have appeared frequently during CLIP's pretraining. Hence, it seems reasonable that CLIP is robust w.r.t. augmented views of images belonging to these categories. In the Fine-grained classification suite, datasets such as SUN397 and Caltech101 also contain common object categories, which is consistent with the results shown above. In contrast, other datasets such as Flowers102 and Oxford-Pets span much less frequent concepts.

Other than the semantic classification space, images' visual appearance also plays an important role. For example, datasets such as FGVC-Aircraft and Stanford Cars still contain rare concepts, but ZERO largely improves over the baseline nonetheless. Our augmentation setup is simple, and only contains random resized crops and random horizontal flips, which can constitute a 'zoom-in' to a random portion of the image. For some benchmarks, this is useful as it may trigger CLIP's capabilities to recognize small details, such as logos, or even reading text, such as the car brand or the airline name. In contrast, more object-centric datasets such as Flower102, may lead to missing precious visual features ( e.g. , the stem).

In our work we did not search for the best data augmentations but rather stuck to an established setting, using the same augmentations setup for all datasets. Nevertheless, the performance of ZERO is linked to the impact that data augmentations have on how the model perceives images, and we believe this is an interesting research direction to pursue.

H Independence among views in the setup of Test-Time Adaptation

The theoretical framework of Section 2.3 models an ideal scenario, where independence holds among different inputs. To clarify, this means that the model's error on view x i should not be correlated with the error on any other view x j , which allows writing the compound error with a binomial distribution as in (6).

In practice, achieving perfect independence is challenging, if not impossible. Hence, a suitable approximation strategy to mitigate this issue is to promote diversity. In classical ensembling theory, a well-established approach is to train different models on different subsets of the available data. Similarly, the augmentation scheme of random cropping aligns with this approach by presenting the model with different portions of the image each time.

Moreover, ideally, the augmentation pipeline should not change the underlying label of the original input and guarantee that the model's error rate on augmented views remains comparable to the error rate on the original inputs belonging to the same category. In practice, this entails that augmentations should not disrupt the visual appearance of the image, and, consequently, some views may result in a slight or moderate correlation, because some 'parts' of the source image will overlap among them. An analogy with classical literature can be drawn also in this case. Specifically, when not enough data are available, overlaps among the training sets of different models are required to ensure convergence. Consequently, models producing slightly or moderately correlated predictions are more likely to emerge.

I Additional Implementation Details

Standard deviations. To complement the results on Fine-grained classification, we report the standard deviation of ZERO computed over 3 runs with different seeds in Table 11. These are not reported together with the average top1-accuracy in Tab. 2 to avoid an excessively dense table. On average, standard deviations are very small, suggesting that regardless of the inherent randomness of data augmentations, ZERO is relatively stable. Note that standard deviations in Group 2 ( i.e. , with MaPLe ) are slightly greater than those in the other groups. This fact does not stem from ZERO's or MaPLe 's greater instability, but from an experimental detail which we report here for completeness: while only one set of weights is officially released for each CLIP version [28], Khattak et al. [15] released 3 sets of pretrained weights for MaPLe , varying on the seed. To avoid picking one, we associated a set of weights to each of our runs, hence results from slightly different initializations are computed to match the experimental setup of Samadh et al. [33] (PromptAlign).

Reproducibility of TTA methods. For section 4, we reproduced all methods using the source code provided by the authors with the hardware at our disposal. This was done to ensure that hardware differences did not interfere with a correct evaluation. We found that all TTA strategies are highly reproducible, with negligible differences ( i.e. , ∆ < 0 . 1 ) which we omitted by reporting the numbers from the official papers. In case of larger differences, we reported reproduced results.

Forest

Highway or Road (0.76)

Highway or Road (0.73)

Highway or Road (0.75)

Herbaceous Vegetation Land

Sea or Lake (0.48)

Sea or Lake (0.44)

Sea or Lake (0.43)

Sea or Lake

Highway or Road (0.24)

Highway or Road (0.24)

Highway or Road (0.25)

(a) Source Image

(b) Most confident

(c) 2nd most conf.

(d) 3rd most conf.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This work studies TTA by MEM, and shows that a method obtained by manually adapting a single parameter is hidden therein. The method, called ZERO, is supported by theoretical justifications. Experiments show that ZERO compares favorably w.r.t. existing TTA methods while being faster and more memory friendly, which is what the abstract claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper contains a 'Limitations' section in the main body. It discusses theoretical assumptions and computational aspects. The paper discusses computational aspects also in the 'Experiments' section of the main paper, especially concerning the dataset size (which is the size of the label space, in the scope of this paper). The Appendix provides evidence for the theoretical proposition of this paper and reports failure modes for the presented approach, as well as in broader TTA.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: This paper contains a proposition, whose proof is in the Appendix. All other theoretical results are supported by empirical evidence.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper contains implementation details in both the 'Experiments' section of the main body and the Appendix. A PyTorch-like implementation is provided alongside the proposed strategy in Section 3 of the main body.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets we have used are all publicly available, including the data splits. The manuscript links to a public GitHub repository containing the implementation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are explicitly reported in the 'Experiments' section of the main paper, including hyperparameters and data splits.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The 'Experiments' section contains the standard deviations, computed over 3 runs, of the proposed method for one out of two benchmarks used. The standard deviations for the second benchmark are reported in the Appendix, to avoid formatting an excessively dense table, and are discussed. The empirical verification for the proposition of this paper is also executed for 3 runs, and standard deviations are reported in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 'Experiments', we explicitly tell which and how many GPU devices were used. Computational aspects concerning runtime and peak memory consumption are also reported in the same section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully inspected the code of ethics, and believe our work conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper theoretically analyzes a paradigm of Test-Time Adaptation with Vision-Language Models and presents a simple and effective method. We do not see a direct path from our method to a negative societal impact which is not inherently already related to VLMs.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe the content of this paper does not have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original creators of all models, datasets and algorithms used in this work are properly credited, with citations around the manuscript. We used their material for the only non-commercial purpose of developing this research paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There are no external assets related to this paper or its Appendix, except for the open-sourced implementation. The code is properly commented, on to guide readers into understanding it step-by-step, and a comprehensive 'readme' file for installation is given.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve crowdsourcing, nor the involvement of external human subjects except for the authors.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This research does not involve any participants except for the authors.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a TTA strategy called ZERO, in which the author carefully and thoroughly explains the entire work, from motivation to method and experiments, including numerous appendices. However, the observations mentioned are not novel enough, and the proposed method is not sufficiently flexible.\n\nSTRENGTHS:\n1. This paper features a clear analysis, allowing for a quick understanding of the problem the authors aim to solve. \n\n2. The highlighted presentation of the experimental section greatly aids reviewers in rapidly analyzing the experimental results.\n\nWEAKNESSES:\n1. The observations mentioned are not novel enough, as many studies have reported similar observations [1].\n\n2. The proposed method lacks flexibility and guarantees.\n\n3. The experimental results are insufficient, with not only a limited number of comparison methods but also incomplete results and a lack of comparison method results for different model architectures.\n\n4. Although the writing is clear, it uses many uncommon words in academic papers, causing difficulties in understanding.\n\nQUESTIONS:\nQuestions:\n\n1. The authors' motivation is not novel, as the phenomenon of over-under-confidence has been revealed in many studies[1], making it not a surprising discovery, could you please clarify your novelty?\n\n2. The authors' method of setting the temperature to 0 seems to be a very straightforward and naive strategy. In fact, in some studies, the temperature has been made learnable instead of being forcibly set to 0[2]. \n\n3. The authors' pseudocode is quite unusual, as it is the first time I have seen code used instead of pseudocode. Is there any insurmountable reason for this choice? \n\n4. The authors have too few comparison methods. In fact, there are already many TTA strategies [3]. Do these methods also have the same problem? \n\n5. Why are only a portion of the experimental results shown? I cannot find the missing results for different model architectures anywhere, including the appendices. \n\n6. Why is the temperature set to 0? This will cause the post-softmax distribution to be extremely sharp. \n\n7. Have the authors considered that label smoothing and sharpness-aware optimization strategies may solve this problem? \n\n8. Why do the authors use some unusual terms like \"prevalent class\"? it means? I cannot even find similar descriptions in papers on arXiv or Google Scholar.\n\n[1] Singla, Sumedha, et al. \"Augmentation by counterfactual explanation-fixing an overconfident classifier.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[2] Wang, Xiao, et al. \"Be confident! towards trustworthy graph neural networks via confidence calibration.\" Advances in Neural Information Processing Systems 34 (2021): 23768-23779.\n\n[3] Liang, Jian, Ran He, and Tieniu Tan. \"A comprehensive survey on test-time adaptation under distribution shifts.\" arXiv preprint arXiv:2303.15361 (2023).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the test-time adaption (TTA) of Vision-Language Models (VLMs), where the goal is to adapt the trained VLMs to unseen datasets/distribution. To this end, this work first revisits the commonly used Marginal Entropy Minimization (MEM) by showing its effect on marginal probability distribution $\\bar{p}$. Then, the relationship between $\\bar{p}$ and ${p}$ is shown. Based on this, a simple TTA method is proposed, which uses \"zero\" temperature when calculating the probability with softmax. The experimental results on several benchmarks demonstrate that the proposed method is effective, and can bring improvements over several baselines.\n\nSTRENGTHS:\n+ The proposed method (ZERO) is simple and straightforward. The experimental results are good and several baselines are included and discussed, which helps understand the effect of ZERO.\n\n+ Section 2.2 (how does MEM affect the marginal probability distribution) is interesting. It gives some insights that the prediction is invariant to entropy minimization\n\n+ Section 2.3 gives the reliability perspective into the marginal probability distribution ($\\bar{p}$). Showing the error of $\\bar{p}$ is a lower bound to the base error of the model is interesting. Giving empirical results is also helpful.\n\nWEAKNESSES:\n- The major concern is the invariance to entropy minimization (Section 2.2). I noticed that this work provides a discussion on this, but I would appreciate the authors give some cases/ examples where this assumption does not hold. \n\n- While I think the analysis of reliability (Section 2.3) is interesting, the claim may be too strong. Specifically, in Line 192 \"Poor calibration is **always** caused by overconfidence\". I do not think \"always\" is appropriate here. I understand the empirical results give such observation. However, this claim needs to consider various datasets/ models. For example, [a] shows the CLIP models trained on different datasets (LIAON and WiT) exhibit distinct behaviors in terms of calibration/reliability. That said, I would suggest the authors lower the claim and make it moderate. Also, I noticed the authors discuss this potential limitation, which I appreciate. Moreover, it would be better if the authors could show the results of Figure 1 (b) using CLIP models trained on LAION and WIT (OpenAI-provided CLIPs). This will make the empirical observation more convincing. \n\n    [a] A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)\n\n- I noticed TPT [37] and PromptAlign [32] evaluate models on the setting of Cross-Datasets Generalization. Why not report results on such a setting?\n\n\n**---- Post Rebuttal ---**\n\nI am not fully convinced by the response regarding the benchmark comparison, particularly since ZERO exhibits a significant performance drop on EuroSAT, which has not been clearly reported following the standard protocol. Other methods do not exhibit such a drop. An apples-to-apples comparison is fundamental to understanding the effectiveness of the proposed approach. While I acknowledge the merits of simplicity and analysis, I remain skeptical about the experimental evaluations.\n\nConsidering the current two evaluations and the lack of guarantee that “the predictions of the model, on average, are accurate,” I am not convinced of ZERO’s generalization to other tasks, backbones, or datasets.\n\nQUESTIONS:\n- Please discuss the cases where invariance to entropy minimization (Section 2.2) may not hold\n- Please claim in Line 192 \"Poor calibration is **always** caused by overconfidence\" moderate and provide more experiments to support such observation\n- Please correct my understanding of Zero temperature: the temperature (not negative) will not change the Softmax predicted class (the predicted class corresponds to the maximum predictions). I am not sure how Zero temperature impacts the predicted class. Please clarify how it improves the performance. \n- [Small suggestion] TPT [37] and PromptAlign [32] use the protocol that only a single test sample is given. This work follows such protocol. I would suggest this work highlight this. The current presentation is not very clear to me\n\nLIMITATIONS:\nThis work provides a discussion of potential limitations, including the augmentation-induced overconfidence might not hold on future VLMs, the invariance to entropy minimization may not hold on all models and datasets, and the independence among augmented views, the computational cost. I appreciate this work mentions these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work shows that Marginal Entropy Maximization (MEM), a leading class of methods for Test Time Adaptation (TTA) which involves maximizing the entropy of the predictive distribution marginalized over different views of the input, regularly results in the same argmax (and thus same final class prediction) as the argmax of this marginal distribution. This means that the entropy maximization step, which requires not insignificant computational overhead, often does not provide any benefit. \n\nThey then show that a very simple approach involving just setting the temperature of each predictive distribution to 0 prior to marginalization is a very strong baseline that outperforms MEM while being much more computationally efficient.\n\nSTRENGTHS:\n- The authors discover a critical flaw in MEM approaches \n- They propose a simple but very effective baseline that outperforms state of the art methods\n- They provide theoretical analysis of their approach \n\nI like this paper quite a bit. It exposes a critical flaw in a popular class of TTA methods, and provides a strong, theoretically-backed alternative method that is very simple to implement.\n\nWEAKNESSES:\n- In section 2.3 in the \"Revisiting model assembling for TTA.\" paragraph, do different views of the input really count as independent samples? it seems like they would be highly dependent on each other. \n- I would tone down some of the stronger claims just a bit. For instance, the paragraph header \"Poor calibration is *always* caused by overconfidence.\" might be overclaiming, since from my understanding this is an empirical observation on a sample of datasets, not an analytical fact.\n\nQUESTIONS:\n- In section 2.3 in the \"Revisiting model assembling for TTA.\" paragraph, do different views of the input really count as independent samples? it seems like they would be highly dependent on each other.\n\nLIMITATIONS:\nThe limitations section is sufficient. Though I think ideally it would be part of the main paper, rather than in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
This work carefully reviews the popular test-time adaptation (TTA) method, MEM, and finds that MEM has largely no effect on 
arg max(𝑝). Based on this understanding, this work further introduces a clean method called Zero, which shows decent performance for the TTA task.

STRENGTHS:
1. Great motivation for bringing insights into Test-Time Augmentations.
2. This is a simple method that can be easily integrated into the current model and algorithms. The performance on natural distribution shifts is strong, and the memory cost is impressively small.

WEAKNESSES:
Although the performance of the ZERO is good on natural distribution shifts, it is not as effective in fine-grained classification. It would be helpful to have more insights into why the performance in fine-grained classification is lacking.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
Please refer to the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-28