Frustratingly Easy Test-Time Adaptation of Vision-Language Models
Authors: Matteo Farina
Gianni Franchi
Giovanni Iacca
Massimiliano Mancini
Elisa Ricci
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Models
Test-Time Adaptation
Robustness
Model Calibration
Primary area: machine_vision

Abstract: Vision-Language Models seamlessly discriminate among arbitrary semantic categories, yet they still suffer from poor generalization when presented with challenging examples. For this reason, Episodic Test-Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature on TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that a surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO (TTA with “zero” temperature), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature to zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses or compares favorably w.r.t. the state-of-the-art while being almost 10× faster and 13× more memory friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work in this field. Code will be available.

Frustratingly Easy Test-Time Adaptation of Vision-Language Models
Matteo F Gianni Franchi Giovanni Iacea'
Massimiliano Mancini!
Elisa Ricei
University of Trento
U22S, ENSTA Paris, Institut Polytechnique de Paris Fondazione Bruno Kessler (FBK)
Abstract
Vision-Language Models seamlessly discriminate among arbitrary semantic cate- gories, yet they still suffer from poor generalization when presented with challeng- ing examples. For this reason, Episodic Test- Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO (77A with "zero" lemperature), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses compares favorably w.r.f. the state-of-the-an while being almost 10x faster and 13x more memory friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work this field. The code is available.
Introduction
Groundbreaking achievements in Vision-Language pretraining interest in crafting Vision-Language Models (VLMs) that can understand visual content alongside natural language, enabling new definition of zero-shot classification. Despite huge pretraining databases [34 37), VLMs still face limitations, suffering from performance degradation in case of
33 39 52
have increased the
large train-test dissimilarity (4 and requiring the design of highly generalizing textual templates (56).
Test- Test-Time Adaptation (TTA) can effectively improve the robustness of VLMs by adapting a given
model to online inputs. Among the various TTA setups (such as "fully" (45), "continual" [47] or "practical" TTA (49). Episodic TTA [33] is particularly appealing, as it focuses on one-sample learning problems and requires no assumptions on the distribution of the test data. When presented with test image the parameters 0 of a model are optimized through a TTA objective L before inferring the final prediction, and reset afterward.
Correspondence to: m. farinaaunitn.... Code
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The choice of C is. ultimately, what characterizes TTA methods the most, with the recent literature being dominated by the objective of Marginal Entropy Minimization (MEM) [53]. Given collection A of N E N data augmentation functions, a test image x is first augmented N times to obtain a set of different views X [A](
)- The marginal probability distribution p w.r.f. sample x is then defined as the empirical expectation of Softmax-normalizae model outputs over X, Le.:
p(-
xx).
N
Under this framework, the Shannon Entropy of p is a bona fide measure of how inconsistently and uncertainly the model predicts over X, making it a tantalizing candidate to minimize, iee.:
Lent = HI((( = Σ ply ply - c|x) = clx)),
(2)
c=
where C' is the number of semantic categories. Once Lent is computed, (some of) the parameters of if are typically updated for a few steps of Gradient Descent before inferring the final prediction over the source input with updated parameters. Owing
simplicity and effectiveness, MEM has
become a de facto standard in modern TTA [33
38
33
19
27).
In this work, take the opposite direction and challenge this paradigm. By conducting in- theoretical and empirical investigation, we find that: @ while effective in improving model robustness, MEM has little effect the prediction of p; @ matter the dataset, the label space, the parameter initialization, VL.Ms become much better classifiers when replaces the standard inference protocol. Building on these insights, show that surprisingly strong and optimization-free TTA baseline is subtly hidden within the MEM framework. We term this baseline ZERO, which is short for TTA with "zero" temperature. Instead of tuning any parameters, setting to zero the Softmax temperature before marginalizing over views makes p already stronger than the model after MEM. Notably, ZERO only requires single forward pass through the vision encoder and no backward passes.
Wrapping up. the contributions of this paper are the following:
1. We theoretically show when the prediction obtained through p (i.e., arg max is invariant to MEM, and empirically verify that MEM has largely no effect on arg max p7;
2. We theoretically and empirically demonstrate that the error rate of p is lower bound to the base eтror of VLM in the setup of TTA. Additionally, we identify augmentations- induced overconfidence as the primal factor undermining the reliability of p.:
3. Motivated by these theoretical insights, introduce ZERO, frustratingly simple TTA approach that recovers the reliability of p by tweaking single parameter the model: the temperature;
4. We thoroughly evaluate ZERO following the established experimental setup with a variety of model initializations. Our results show that ZERO surpasses or compares favorably to state-of- the-art TTA methods while being much faster and more memory efficient (e.g., 10x faster and 13x more memory efficient than the established Test-Time Prompt Tuning [38
2 Understanding Marginal Entropy Minimization
In this Section, we take a step towards both theoretically and empirically understanding the paradigm of MEM. In particular, this section is devoted to answering the following research questions:
1. How does MEM affect the marginal probability distribution? And, turn
2. How does the marginal probability distribution relate to the standard inference protocol?
First, we introduce MEM for VLMs by reviewing the established Test-Time Prompt Tuning (TPT) method 38 and its notation. Then, in Sections 2.2 and 2.3 we answer the research questions above.
2.1 Preliminaries
Zero-Shot Classification with VLMS employs predefined template (e.g., "a photo of a²") from which a set of context vectors tex is obtained by looking up token embedding table. Expanding

---

the template with the class names (e.g., "a photo of laptop. for the class "laptop") makes up the entire set of input vectors [tex. til...., [tex. tcl, with t, being the embeddings derived from the i-th class name. A text encoder Exs transforms these class descriptions into independent and normalized text embeddings , 2 and an image encoder Fimg encodes an input image x into a normalized latent vector 2m.__ Lastly, classification is carried out by picking the class e corresponding to the text embedding z holding the maximum cosine similarity with _!
MEM for VLMs. Pioneered by MEMO [53] in the scope of unimodal neural networks, MEM was re-
purposed for TTA with VLMs by Fest-Time Prompt Tuning [38]. In (38) a VLM such as CLIP [3]] adapted at test time by minimizing the same objective of Eq. (2). In contrast to optimizing all
model parameters, TPT relies on the effectiveness of prompt tuning [56] (5), optimizing only the
context vectors derived from the token embeddings of the standard CLIP template "a photo of
a". By explicitly enunciating the dependency on the context vectors t. and re-using the notation of
Sec. 1 one can re-write the MEM objective of [8 as:
C plu = c/x, tax, T) log (p(v = c/x, tekx T))
Cent = Hp(( [x, texx)) = Σ
1 N
N
exp 2
img
2, (tess)/
where p(y c/x, lexx; T)
exp Zz zk (tex)/T)
Here, F is the temperature of the Softmax operator. In the rest of this section, we omit the dependency on 7 for simplicity, writing p(-|x, tetx). Similarly to [53], the objective of Eq. (3) is minimized for single step of Gradient Descent to update the set of context vectors. The updated context vectors, denoted as txx are then used to prompt the VLM and obtain the final prediction for x. For any class this is simply Zx
img z (txx), which is easily transformed into p(y c|x, txx) via Softmax.
2.2 How does MEM affect the marginal probability distribution?
The recent literature on TTA shows that minimizing Lent significantly enhances the robustness of model outputs. However, the impact of this process on the marginal probability distribution p remains unclear. We start with a straightforward hypothesis: due to its nature, minimizing fent tends to increase the probability of the most probable class of p(- tex). More formally, denoting with è the prediction of P (i.e., č t. If this hypothesis is realized, it comes as natural consequence that minimizing Cent is unlikely to alter the prevailing class of p. thus resulting in a consistent prediction pre- and post-TTA where arg max p(- |x, t arg max P( |X, ).))
arg maxp( [x, tex)) we hypothesize that ply è|x, tzx) p(y
Hence, the first contribution of this work is to show that the prediction of the marginal probability distribution p is invariant to Entropy Minimization under loose constraints on confidence and gradients. To lighten the notation of the proposition, let us first define the following function g:
exp (z img
1xl /T)
g(c, z img 2
Σ exp (min.. 2 /T)
i.e., the probability assigned to class given a latent image representation z in and class-wise text embeddings z ..... zzz Additionally, let òg(c, mi)) be the negative variation incurred to the function g when the context vectors tex are updated through Entropy Minimization:
0g(c, z (ms)) g(c,z"ny (tax), . .., 26 (tex))
g(c, z (t(x) 2c (tax))
(5)
where, for clarity, the dependency of the text embeddings z ..... ziz on the context vectors (either tu or t(x) explicit. Using this notation, we can formalize the following proposition:
....,% .. be the latent image representations resulting from the N views Let zitHr ..... and é = arg max p(-|x, tax) be the initial prediction of the marginal probability distribution. If the entropy of P is minimized and ply c|x,, twxx) invariant to MEM, i.c., arg max (-|x tex: = arg max P |x, ).
Proposition 2.1.
-N
5g(ē, 2img)) then the prevalent class of T is
In Appendix A we provide detailed proof of this proposition, highlighting that 8(2, 2!)) is directly linked to the gradient w.r.l. the context vectors fax. This relationship emerges when writing any post- update text embedding 2.t (txx)) as function of its pre-update counterpart z.! (tax). Specifically,
3

---

we can write 2 Exa(ite e.. (Cene), tel), which is equivalent to Era(ftas tc])
CIx
A Eu t.)) Vt.. (Cnnt) after first-order Taylor Expansion around tax Consequently, the proposition holds by condition relating confidence (through p(u = c|x, tax }) and gradients (through 8(2, 2 (em)) Alongside the proof. Appendix A presents evidence supporting this proposition for CLIP
31 on the ImageNet- 1k validation set 4 as well as across various datasets for natural distribution
shifts: ImageNet-A [13,, ImageNet-R [12] ImageNet-v2
[2), and ImageNet-Sketoe [46].
2.3 How does p relate to the standard inference protocol?
From prior work on Test-Time Augmentations (TTAug) with unimodal neural networks 40 351, empirical evidence suggests that p--|x) is more robust than p(-|x). This observation leads to the hypothesis that the expected risk of predicting with p is lower than that of doing with p. However, the literature lacks guarantees for this hypothesis, except for the peculiar case in which the risk
function is the squared error, i.e., E(a, b) (a- b)2 [16]
As the second contribution of this study, we show that the error rate of p(-1x) x) does indeed lower-bound the error rate of p(-|x), We do so by revisiting the theory of model ensembling, and showing that analogous ideas can emerge for TTA.
Preliminaries on model ensembling. From the theory of classifier ensembling 18 we know that if .f.....JJ JN are N E independent classifiers with error rate e and x is an example whose label is E {0, 1}, then the probability that any group of & classifiers picks the same wrong label fi(x) = 0 F y can be expressed with a Binomial distribution wrapping N Bernoulli processes:
Potu(A (k) =
*(1 (()-c((N-))
(6)
k
Revisiting model ensembling for TTA. Eq. 66) holds as long as all events modeled as Bernoulli processes are independent. Thus, we have an equivalent error estimate for the setup in which only a single classifier f is present and x,, {x, )== is a set of independent examples with the same underlying label y. Within this framework, any group of & examples in Xy to which the classifier has assigned the same label ğ is also a set of independent Bernoulli processes, whose error probability is still quantified via Eq. (6. Note that this resembles the TTA setup in the presence of N views of the source sample x, as long as augmentations do not change their underlying labels. We refer the reader to Appendix H| discussion about the independence assumption among different views.
p is better than p (if f is calibrated). The final step can be taken through the lens of model calibration [8) property requiring that the confidence of a classifier matches its accuracy. For example, calibrated classifier whose confidence is 0.7 is expected to correct 70% of the times. In the previous discussion, if we denote with k(y) the number of examples correctly labeled as y. then the accuracy of the classifier is exactly kty)/N. It follows that there is positive correlation between accuracy and confidence if f exhibits good calibration, i.e., 1 k(y) - 1 p(y). Thus, the probability of picking the wrong class with this marginal probability is approximated by Eq. (6). Given this relationship, N. Thus, the probability of picking the wrong class with p is approximated by marginalizing out all values of & that satisfy this criterion, which entails that the error or can be expressed with the cumulative distribution of 6
have that p(y) maxp(; if k<y)) matches exceeds the majority within
N Σ
P (p) =
(x)
(1-)
k= k=IN/2+1
7 is a monotonically decreasing function functioon if the error is better than random guessing, which is likely to be the case for VLMs pretrained massive amount of web data such as CLIP. Hence, we conclude that the error of p is a realistic lower bound for the base model error over set of independent data points sharing the same label.
From the Condorcet Jury Theorem 36.. we know that Eq.
Does this lower bound empirically realize? We evaluate if the error of p consistently lower bounds the error of p also in practical use cases, where model calibration is unknown and the label space is large. For this, we use CLIP- -----iiT--111 [5), the ImageNet validation set, and four datasets reflecting
In Appendix D show that this bound generalizes to any function & satisfying the triangular inequality.

---

Powe
ImageNet-Ik
Angmented
Expected Error Clazzes [%6]
1.0
ECE LOBS
11.2% Accunncy
0.5
0.8
Accumey
Accuracy
0.4
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Dataset
Confidence
(a) Error p c(y))
(b) Reliability diagrams for IN-1k and its augmented version.
Figure I: Motivating findings. (a) Comparison between the expected error of CLIP-ViT-B-16 denoted as c(y), and the error of the marginal probability distribution obtained by marginalizing over examples with the same label, (p); (b) Reliability diagrams of CLIP-ViT-B-1 on the ImageNet validation set (left), and augmented version (right), showing that augmentations largely un-calibrate CLIP exclusively due to overconfidence while leading to slightly better overall accuracy.
For all classes in each dataset, we first draw all images sharing the same label (X,)) Then, we compute the expected error <(y) of the model on this subset, together with the error of p (ideally, Eq. ( Lastly, we average these emrrrs over the entire label space V. We do not restrict to the cases where is supported by the majority and we do not re-organize predictions in one-versus-all scheme. Fig. (Ta clearly shows that the eπror of is lower bound to the base error of the model also in practical use cases where the label space is large and guarantees on model calibration possibly missing. Importantly, this phenomenon persists matter the dataset.
Natural Distribution Shifts 12 13
32 46
Simple and surprisingly strong TTT (for free)
The main point of Section 2.2 that MEM generally does affect the predominant class of the marginal probability distribution p. On the other hand, from Section 2.3 one can conclude that through p the model becomes much stronger classifier. Summarizing:
From Section 2.2
arg max (p(-Ix, texx)) arg max (p(-Ix, t(x)
Peov (f(-1x t(x)) S Pozzy (p(-1x tex)
(8)
From Section 2.3
and, equivalently
Pa)y(p(-1x, t(xx)) S (oly tax))
Chaining observations together, it emerges that:
Potty (p(-)x tax)) 2 Pott (p(-1x, t(x)) Po tax))
Le., if all assumptions are met, the enror of MEM error of after MEM error of T without MEM. All in all, this TTA framework hiding a surprisingly strong and optimization-fua baseline: p! Next, we highlight the detrimental impact of data augmentations this marginal probability distribution and introduce simple trick to recover its reliability: zeroing-ond ooring-outtt out the Softmax temperature.
3.1 Augmentations undermine the reliability of p
While augmentations are essential TTA obtain multiple views of the test instance, noisy views may constitute cut-of-Diitributtion (OOD) data, thus having the undesired effect of un-calibrating the model. To sidestep this issue, one can attempt to discriminate between in-distribution (wott to the pretraining data) and OOD views. Given that low confidence is common trait in OOD data, viable way to discriminate confidence-basee filtering, such as TPT (38 Formally, a smaller set of confident views obtained following X X {x, X (p( X4, tex)) < p), where is
5

---

threshold retaining the views whose entropy is in the bottom 100% percentile (lowest entropy). Despite its effectiveness, this filter cannot help when the reliability of p is undermined by overconfidence.
Augmentations lead to poor calibration. We demonstrate the impact of augmentation-indut overconfidence using the same model and datasets of Section 2.3 For each dataset, we generate an augmented counterpart following the augmentation and filtering setup TPT (38), i.e.: we augment an input N 64 times using simple random resized crops and horizontal flips. Then, we only retain 10% of the N views according to confidence- beasddence-baed filtering, resulting in 6 views per sample. Consequently, each augmented dataset contains 6 more data points than its plain counterpart. The Expected Calibration Error (ECE) 8 reported in Appendix [C conveys that @ zero-shot CLIP is well-calibrated (ECE < 0.1 for all datasets), strongly supporting the theory of Section 2.3 and the augmented visual space greatly increases the calibration error.
Poor calibration is frequently linked to overconfidence. We investigate the reason for the increase
in ECE by presenting reliability diagrams for the ImageNet validation set in Fig. [b). In a reliability diagram, every bar below the identity line y = x signals overconfidence (i.e., the confidence on the x-axis prevails over the accuracy on the y-axis), while the opposite signals under-confidence. Notably, in the scope of our experiments, overconfidence the primal factor leading an increase the ECE, The error rate, in contrast, decreases slightly. In Appendix C we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on the 2B subset of LAION 12 34]. Importantly, this phenomenon further persists within this extended experimental suite.
3.2 ZERO: Test-Time Adaptation with "zerro" temperature
Since its reliability is severely undermined by augmentations-induees overconfidence, directly predicting through p is not an enticing baseline for TTA. Concurrently, we also know that the error rate does not decrease when predicting over the augmented visual space. Hence, are interested in finding an efficient way to capitalize on these observations: relying on the predictions over the views, while ignoring potentially misleading confidence information. The key is to note that both desiderata are obtained by explicitly tweaking a single parameter of the model: the lemperature. Specifically, setting the temperature to (the limit of) zero corresponds to converting probability distributions into one-hot encodings, hence exclusively relying on their arg max when marginalizing, Inspired by this idea we propose ZERO, Test-Time Adaptation with "zero" temperature.
Procedure. ZERO follows these simple steps: ① augment, @ predict, @ retain the most confident predictions, @ set the Softmax temperature zero and @ marginalize. The final prediction is the arg max of the marginal probability distribution computed after "zeroing-out" the temperature, i.e.:
N
ZERO(x, tax; C) = arg max cE(
Σ
(xx E ((s)) lim ply = c|x;; texx; (ex)) --0+
(10)
where 1 is an indicator function, whose output is 1 xf; E faa and 0 otherwise, and X is the set of confident views before tweaking the temperature, i.e., x, E X aee if (p((1xx,, tex T) < p.
Efficient Implementation. In all its simplic- ity. ZERO is computationally lightweight. In closed set assumptions where the class de- scriptions (and thus their embeddings) are fixed, ZERO only requires single batched forward pass through the vision encoder, just as much as needed to forward the N views. Additionally, since the temperature is explic- itly tweaked, ZERO needs no backpropagation at all and can be implemented in a few lines of code. For reference, Py Torch-like imple- mentation 30 is reported in Algorithm T
Algorithm Py Torch-style code for ZERO
z_txt pre-computed text embeddings (C,hdim) temp model 's original temp
augment takes (C,H,M) and returns (H,C,H,W) gamma filtering percentile (e.g., 0.1) def (image, 2_txt, N, garma, temp):
# step1: augment
views augnent (image, num_vieue-N) # step2: predict (unscaled logits) saddel.inage_eeccdeee (views) z_txt.a() # step3: retain most confident preds
filt confidence_filtert # step4: zero temperature zero_temp # step5: marginalize
temp, top=gamma)
wtt.:faffoo_riitdtyyppe.e .eps
Equivalent perspective and final remark. We bring to attention a simple scheme which corresponds to ZERO: voling over (confident) augmentations. Drawing from the theory of ensembling, note that the error rate of the voting paradigm is exactly described by Eq. 6 Essentially,
babar = (1_filt / zero bero_temp).... sooftmax (1) s0)
return p_bar.argmax:

---

Table Natural Distribution Shifts. TTA methods are grouped according to the baseline model and top-1 accuracy is reported. Bold text is the best method within each group.
Method
ImageNet
Sketch
Mcon
CLIP-ViT-B-16
Zero-Shot
66.73
Ensemble
68.34
49.89
61,88
77.65
48.24
61.20
TPT
68.98
54.77
63.45
77.06
47.94
62.44
ZERO
69.31:1:0.13
59.61=0.19
64.16:20.03
77.22:4:0.05
48.40:=0.07
63.74
ZERO.Raeebbl
71.17:10.06
62.75:10.14
65.23:10.08
80.75.1:0.02
30.59:10.08
NaPLe
Zero-Shot
76.98
60.28
TPT
58,08
64.87
78.12
48.16
62.31
PromptAlign
59.37
65.29
79.33
50.23
63.55
ZORO
63.32=00.26
66.81=0045
79.74::0.32
51.07:0..77
65.23
CLIP- PTPP-VIT-B.. B- 16 LLIP--VT-L--14
Zero-Shot
las
RLCF the
74.85
73.71
69.77
86.19
57.10
72.32
ZERO
75.52:20.03
75.15=00.26
70.37:-0.05
87.21::0.09
59.61=20044
73.57
this means that ZERO capitalizes on the theoretical insights while circumventing practical issues stemming from augmentations. We also highlight that ZERO subtly hidden within any TTA framework relying exclusively MEM, since computing is inevitable therein. For this reason, refer to ZERO as baseline for TTA. Our goal diverges from introducing "novel" state-of-the-art method for TTA. In contrast, we advocate the importance of evaluating simple baselines.
4 Experiments
In this section, we present a comprehensive experimental evaluation of ZERO. Similarly to 38 33 33,5,,, always work in the setup of single test point adaptation. Our results show that ZERO, alongside its simplicity, is an effective and efficient approach for TTA.
4.1 Experimental Protocol
Baselines. We compare ZERO to three strategies for TTA with VLMs: ① TPT 38 2 PromptAlign [33] and @ Reinforcement Learning from CLIP Feedback (RLCF) [54]. As introduced in Section TPT works by minimizing the entropy of p. In contrast, PromptAlign relies on pretrained MaPLe initialization [15] and pairs the MEM objective with a distribution alignment loss between layer-wise statistics encountered online and pretraining statistics computed offline. Finally, RLCF does not include MEM in its framework; Zhao et [54] shows that, if rewarded with feedback from a stronger the smaller CLIP-ViT-B V-1---1-
teacher such
surpass the teacher itself.
Models. As different approaches consider different backbones in the original papers. we construct different comparison groups to ensure fair comparisons with all TTA baselines [8
33
Shu al. 38 also reports ELIP-Ensemil i.e., CLIP enriched with an ensemble of hand-crafted prompts. While the design of TPT does not allow leveraging text ensembles (as also pointed out by concurrent work 42 seamlessly integrates with CLIP-Ensemble We denote this variant with ZERO+
Group 1:
When comparing to TPT, always
ZERO
Group 2:
When comparing PromptAlign, follow Samadh et al. 133 and start from MaPLe initialization for fair comparison. MaPLe prompts are learned on ImageNet, following [33 Within this group, we also report TPT on top of MaPLe, in 33
Group 3: When comparing RLCF, we use both CLIP-ViT-B-18 and CLIP-ViT-L-14 as in [54 Specifically, confidence-basee filtering acts on top of the output of the first model, and the selected inputs are passed to the second model for the final output. Both forward passes are inevitable in RLCF, so this scheme corresponds to "early-exiting" the pipeline, exactly as per MEM. RLCF can vary according to (i) the parameter group being optimized and (ii) the number of adaptation steps. We denote with θ the full image encoder tuning, with tex prompt tuning, and with t the number

---

of adaptation steps. For example, RLCF fon indicates RLCF with prompt tuning for 3 TTA steps. Note that, since all methods need forward more than one image to the teacher model, the zero-shot baseline of this group is exactly zero-shot classification with
1=
Pretrainings. This Section deals with models officially released by OpenAl 28) Appendix R further reports experiments with LAION-pretrainne CLIP models [2]. as well as the soft prompt initialization with supervised Context Optimization (CoOp) from [56].
Distribution Shifts and Fine- fine-grained Classification (also referred to evaluating ZERO on Natural Generalization" in previous works). For the former, we consider the ImageNet validation set and the four datasets for Natural Distribution Shifts already presented in Section 2 commonly considered
Benchmarks. We follow the established experimental setup of 38
(OOD) datasets for CLIP. For fine-grained classification, we evaluate all TTA methods on 10 datasets. Specifically, experiment with Oxford-Flowers (FWLR) [25), Describable Textures (DTD) 3
Oxford-Pets (PETS) 29), Stanford Cars (CARS) 17). UCF101 (UCF) 411, Caltech (CAL)
Food101 (FOOD) M. SUN397 (SUN) 48), FGVC-Aireraft (AIR) [23] and EuroSAT (ESAT)
For all of these datasets, we refer to the test split in Zhou et al. [56 as per the common protocol.
Textual prompts. When +Ensemble is specified, we do not use dataset-specific templates. In
contrast, we use the set of 7 generic templates highlighted in the official CLIP repository [28] across all datasets. When adapting MaPLe, we stick to the ImageNet-leamedd prompts released by [5]] and evaluate them cross-datasets as in (33.
Implementation Details. The augmentation pool A only contains random resized crops and random horizontal flips. The only hyperparameter of ZERO is the percentile for confidence-based filtering, which is set to 0.3 after validation on ImageNet (following standard practice [51]) and kept fixed for all datasets. We inherit the setup of TPT with N 64, crafting 63 augmentations to collate with the source image. To ensure hardware differences do not play any role in comparisons, we execute all TTA methods under the same hardware setup by running the source code of each repository with no modifications. We always 1 NVIDIA A100 GPU and FP16 Automatic Mixed Precision. Results are averaged over 3 different seeds. Unless otherwise specified, all tables report top-1 accuracy.
4.2 Results
Natural Distribution Shifts. Results for Natural Distribution Shifts are reported in Table
Group I (TPT): ZERO surpasses TPT consistently on all datasets. Among OOD datasets, peak difference is reached with ImageNet-A, where ZERO outperforms TPT by +4.84%. Enriching ZERO with hand-crafted prompts improves results further, with an average margin of +3.66% w.r.t. TPT.
Group 2 (PromptAlign): Within the second comparison group, ZERO outperforms PromptAlign on all datasets, with + 1.6% being the gap average performance. ZERO consistently outperforms TPT also when the baseline initialization is MaPLe (by an average of +2.92%). Please note that we omit evaluation on ImageNet for this group, since PromptAlign adopts token-level statistics from this dataset when adapting test points, which would render the comparison unfair. For completeness, we report that zero-shot MaPLe achieves an accuracy of 70.72% on ImageNet, which is improved to 72.99% by adapting with ZERO (+2.27%)
Group 3 (RLCF): We follow (54] and report RLCF variants with t steps. In this group, ZERO out- performs RLCF in 5 out of 5 datasets, with gap in the average performance of +1.25%. Importantly, RLCF only close to ZERO with image encoder tuning; only prompt tuning is insufficient.
Fine-grained Classification. Results for fine-grained classification are shown in Table To foster readability, the standard deviations of ZERO are separately reported in Table [I
(Appendix).
Group (TPT): Default ZERO improves over the zero-shot baseline CLIP-ViT-B-16 but is outper- formed by TPT with an average margin of -0.57%. However, extending ZERO with hand-crafted prompts (something that TPT cannot do by design) is sufficient to outperform TPT on 7 out of 10 datasets, and obtain an average improvement of +0.74%.
Group 2 (PrompiAlign): On average, PromptAlign has an improvement of +0.5% over ZERO. However, note that this is mostly influenced by the performance on one dataset only (EuroSAT) and that, contrast, ZERO surpasses PromptAlign in 7 out of 10 datasets. In line with the previous benchmark, ZERO better adapts MaPLe than TPT, again outperforming it in 7 out of 10 datasets.
8

---

Table Fine-grainee classification. TTA methods are grouped according the reference baseline,
top-1 accuracy is reported and bold text indicates the best performer of each group.
Method
FLWR
DTD
PETS
CARS
UCF
CAL
FOOD
SUN
AIR
ESAT
Mean
Median
CLIP-VIT-B-16
Zero-Shor
67.44
44.27
88.25
65.48
63.13
93.35
83.65
62.59
23.67
42.01
63.58
65.31
Ensemble
67.07
45.09
88.28
66.16
67.51
93.91
84.04
66.26
23.22
50.42
65.20
66.66
TPT
68.75
47.04
87.23
66.68
68.16
93.93
84.67
65.39
23.13
42.86
64.78
67.42
ZERO
67.68
46.12
87.75
68.04
67.77
93.66
86.53
65.03
25.21
34.33
64.21
67.72
ZEROsEraenei
67.17
45.86
87.83
68.97
69.18
94.41
86.77
67.63
25.21
42.17
63.52
68.30
NaPLe
Zero-Shor
72.23
46.49
90.49
65.57
68.69
93.53
86,20
67.01
24.74
48.06
66.30
67.85
TPT
72.37
45.87
90.72
66.50
69.19
93.59
86.64
67.54
24.70
47.80
66.49
68.36
PromptAlign
72.39
47,24
90.76
68.50
69:47
94.01
86.65
67.54
24.80
47.86
66.92
68.99
ZERO
71.62
47.89
90.60
68.58
69.87
94.48
87,20
68.20
26.25
39.47
66.42
69.23
CLIP-WV
CLIP-ViT-L-1E
Zero Zeno-Shot Shot
75.76
51.83
92.86
76.16
73.70
94.04
88.03
66.96
30.54
54.38
70.43
74.73
RLCF
1
71.58
50.34
89.01
69.76
69.84
94.09
85.90
67.33
23.71
46.87
66.84
69.80
RLCF
72.49
51.93
89.55
72.91
72.31
95,00
86.84
69.04
25.40
45.96
68.14
72.40
RLCF 5.
72.56
52.21
89.51
63.12
71.49
94.65
86.90
68.50
24.06
47.74
67.07
70.00
6.
RLCF
71.74
53.27
91.15
70.93
73.24
94.73
87.28
69.38
28.54
47.41
68.77
71.34
ZERO
76.41
53.63
94.08
78.39
74.68
95.21
90.66
69.61
33.62
44.21
71.05
75.55
Table 3: Computational requirements of different TTA methods.
CLIP-VAT-B-16
CLIP-VIT-B-16 CLIP-VAT-L-16
Metrie
RLCF Sex
RLCF Sy
Time [s]
0.57
0.06:10.01
1.20::.002 18.64
0.1810001 9.04
0.08=0.02
Mem (GB]
17.66
1.40
2.58
Group 3 (RLCF): As Zhao et al. [54] do not report results on fine-grained classification, we use their code evaluate four RLCF variants: O, and ferx tuning, with = 1 and 3 adaptation steps. We find that ZERO largely outperforms RLCF regardless of the configuration. Even with respect to the strongest RLCF C.
t=3 variant, ZERO obtains an average improvement of +2.28%.
Computational Requirements. The complexity of ZERO does not scale linearly with the size of the label space, as does for prompt-tuning strategies. To quantify the computational gain of ZERO w.r.f other TTA methods, report the runtime per image and peak GPU memory in Table under the same hardware (i.e., NVIDIA RTX 4090. We compare the computational requirements ZERO to TPT and the RLCF pipeline in worst-case scenario where the label space is large (ImageNet). We omit PromptAlign from our analysis since has slightly worse computational performance than TPT.
ZERO is 9.5x faster than TPT taking 12.61 less memory, corresponding to order of magnitude of computational savings both time and space. Concerning the slowest RLCF variant (prompt tuning), ZERO is 15x faster and takes 7.22x less memory. In the faster RLCF 9,, text classifiers also cached: nevertheless, ZERO is 2.25x faster and 3.5x more memory friendly.
Related Work
Closest to our work is a recent and very active research thread focusing on Episodic TTA with VLMs [38 33 54] 42). As discussed in the manuscript, these methods mostly rely on prompt learning,
parameter-effiiien strategy that only trains over small set of input context vectors [20]. Narrowing down to VLMs, notable examples of prompt learning approaches include CoOp [56]. CoCoOp [55]
and MaPLe 15
Episodic TTA has also been explored with traditional unimodal networks, such
as ResNets 10 where MEM is still a core component [53] In this context, MEM has recently been enriched with sharpness- 27 or shape-aware filtering 19 Due to nature, Episodic TTA completely agnostic to the temporal dimension and is powerful when no reliable assumptions on the test data can taken. Some other works relax these constraints and integrate additional assumptions such as batches of test data being available instead of single test points [45.. When test data

---

assumed to belong to the same domain, one can rely on various forms of knowledge retention as powerful mechanism to gradually incorporate domain knowledge [2] [2 or avoid forgetting [26]..
The synergy between TTA and retrieval is also emerging as a powerful paradigm when provided with access to huge external databases
[9] 50]
We particularly believe this can be promising direction.
Closely related to our work are also Test-Time Training (TTT) and TTAug, In TTT the same one sample learning problem of Episodic TTA is tackled with auxiliary visual self-supervised tasks, such as rotation prediction 3 or masked image modeling Z7, which require specialized architecture heads and are not directly applicable VLMs. TTAug has recently been theoretically studied [I6]. boils down to producing a large pool of augmentations to exploit at test time [35), or to learn from [4]. In all its simplicity, ZERO can be seen as strong TTAug baseline for VLMs, which, differently from concurrent work [31], does not involve any form of optimization.
6 Limitations
ZERO can seamlessly adapt a wide range of VLMs on arbitrary datasets without requiring extensive computational resources and is backed by theoretical justifications. However, delincate four major limitations to our method which we report here.
Preliminary observations. The first limitation concerns the preliminary observations which led to ZERO, such as augmentation-inducer overconfidence or comparable error rate between source and augmented datasets. These observations may not persist if VLMs or benchmarks change significantly in the future, potentially leading to poor adaptation. For example, we have observed a consistent failure case for TTA with EuroSAT [1I] with ZERO incurring large performance drops w.r.f. simple zero-shot classification. In Appendix we unravel this worst-case further.
Theoretical assumptions. The second limitation stems from theoretical assumptions, the core one being the invariance of the marginal probability distribution to marginal entropy minimization. While our proposition guarantees invariance if entropy is globally minimized and the negative variation to the probability of the most probable class less than the initial probability itself, these theoretical assumptions may not hold all the time. In this work, we supported our assumptions with empirical verification but, as per the first limitation, these may not extend the space of all models and datasets. We refer the interested readers to Appendix A for a more in-depth discussion about the invariance of the prediction of p to MEM.
Independence among views. A third worthy-of-note limitation relates to the independence assump- tion among the views from which the marginal probability distribution obtained. As we discussed in Section 2.3] the views themselves do not have any direct dependency. but they are still partially related through the source image from which they stem. Related this limitation, we hypothesize TTA setup (or a cache-basee based one) could improve the
that extending ZERO in a Setrieval-Augmenne results. The discussion on this topic extended in Appendix H
Linear complexity with respect to augmented views. Finally, despite being much lighter than the current offte-of-the-art TTA strategies, ZERO'S computational requirements in the visual branch scale linearly with the number of views, since all of them need to be independently forwarded. On this, believe that exploring how to augment directly in the latent visual space to also circumvent the forward pass of the vision encoder is an intriguing direction.
7 Conclusions
We theoretically investigated Marginal Entropy Minimization, the core paradigm of the current research on Test-Time Adaptation with VLMS.. Building on our theoretical insights, we introduced ZERO: a frustratingly simple yet strong baseline for TTA, which only relies on single batched forward pass of the vision encoder. ZERO works by setting the temperature of the Softmax operator to "zero" before marginalizing across confident views, which is equivalent, in terms of output, to the widely known paradigm of majority voting, Our experimental results on Natural Distribution Shifts and Fine-grained Classification unveil that ZERO favorably compares to -tate-of-the-art TTA methods while requiring relatively negligible computation. We hope our findings will inspire researchers to push the boundaries of TTA further.
10

---

Acknowledgements. The authors acknowledge the CINECA award under the ISCRA initiative for the availability of high-performance computing resources and support. Matteo Farina is supported by the PRIN project "I "LEGO-AI (Prot. PPr0A.0000 and the PAT project "AI@TN" This work was supported by the projects EU Horizon ELIAS (No. 101 120237), A14TRUST (No. 101070190), FAIR Future AI Research (PE00000013), funded by NextGeneration EU, and carried out in the Vision and Learning joint laboratory of Fondazione Bruno Kessler and the University of Trento, Italy.
References
[1] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. food-101-minng discriminative components with random forests. European Conference on Computer Vision (ECCV), 2014.
[2] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmano, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEF/CVF Conference on Computer Vision and Pultern Recognition (CVPR), 2023.
[3] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
[6] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEF/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPR-W)) W). IEEE, 2004.
[7] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning (ICML), 2017.
[9] Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In International Conference on Learning Representations (ICLR), 2024.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[11] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7), 2019.
[12] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distributio generalization. In JEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[13] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Kun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and languale representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021.
[15] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In IEFF/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
11

---

[16] Masanari Kimura. Understanding test-time augmentation. In International Conference on Neural Information Processing (ICONIP). Springer, 2021.
[17] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W), 2013.
[18] Ludmila I Kuncheva. Combining pattern classifiess: methods and algorithms. 2014.
[19] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. International Conference on Learning Representations (ICLR), 2024.
[20] Xiang Lisa Li and Percy Liang. trefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1t International Joint Conference on Natural Language Processing (Volume I: Long Papers); 2021.
[21] Zichen Liu, Hongbo Sun, Yuxin Peng, and Jiahuan Zhou. Dart: Dual-modal adaptive online prompting and knowledge retention for test-time adaptation. In AAAI Conference Artificial Intelligence (AAAI), 2024.
[22] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchho Xu. Swapprompt: Test-time prompt adaptation for vision -aiiugle models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[23] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- 2013.
grained visual classification of aircraft. arXiv preprint
[24] Prasanna Mayilvahanan, Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does clip's generalization performance mainly stem from high train-test similarity? In International Conference on Learning Representations (ICLR), 2023.
[25] Muria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing.
JEEE, 2008.
[26] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Confer- ence on Machine Learning (ICML), 2022.
[27] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference Learning Representations (ICLR), 2023.
[28] OpenAI. Clip. URLLaattee//ggttbbb.coo//ppppppp
[29] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2012.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high nigh-peeformance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[32] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019.
[33] Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for sero-shot generalization. In Advances Neural Information Processing Systems (NeurIPS), 2023.
12

---

[34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[35] Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag, Better aggregation in test-time augmentation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[36] Lloyd Shapley and Bernard Grofman. Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice, 1984.
[37] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2018.
[38] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[39] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[40] Jongwook Son and Seokho Kang. Efficient improvement of classification accuracy via selective test-time augmentation. Information Sciences, 2023.
[41] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212,042, 2012.
[42] Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. arXiv preprint prXiv:2403.19955 2024.
[43] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
[44] Devavrat Tomar, Guillaume Vray, Behzad Bozorgtabar, and Jean- Saan-Philippe Thiran. Tesla: Test- time self-learning with automatic adversarial augmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[45] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021.
[46] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[47] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
[49] Longhui Yuan, Binhui Xic, and Shuang Li. Robust test-time adaptation in dynamic scenarios.
In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[50] Luca Zancato, Alessandro Achille, Tian Yu Liu, Matthew Trager, Pramuditha Perera, and Stefano Soatto. Train/test-time adaptation with retrieval. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[51] Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision- language models: Do we really need prompt learning? arXiv preprint arXiv:2405.02266. 2024.
[52] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IFFF/CVF International Conference on Computer Vision (ICCV), 2023.
13

---

[53] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[54] Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with CLIP reward for zero-shot generalization in vision-language models. In International Conference on Learning Representations (ICLR), 2024.
[55] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (ICCV), 2022.
14

---

A Marginal Entropy Minimization does not influence arg max
A.1 Proof of Proposition 2.1.
Proof. Let us denote the pre-TTA pn ply = c|x, tex) and the post-TTA p ply = c|x, tx ie., the marginal probabilities before and after optimizing tux Let e and can denote the predictions before and after TTA, i..,, c arg max pinit and enod = arg max prnd.
To simplify the notation, let use nMx write any post-TTA text embedding zzx (tax))
Under the assumption that entropy minimized (the optimal scenario for MEM), we have (c = 1, and yaT (c) = 4A cond
Let us rewrite the final distribution pen using the function g introduced Sec 2.2 Specifically, for any class e, we have:
img
2 (tww)/T)
pend (c)
1 N
exp
2
1 N
21 E
g(., zi iing
20 (
(11)
2E exp
2 (tTw)/T)
Performing first-order Taylor expansion on g, we have:
g(c, img zi
2(x1 g(c,
img
lxt
(12)
V a (g) ([z)
ZC
sixl
[ Pal
We can also write any post-TTA text embedding (t ) as function of the text encoder Eua prompted with optimized context vectors:
2 (t = Exer Exee(taa tc)) = Exr((tttr - XV tex H, tc))
(13)
Through another first-order Taylor expansion (this time on z (tExx) D). we have:
- (1 = Eral (1texx tel) + (V E (x (z
z (tex))
(14)
E (ftxx, tel) A(Ve,, EAS) '. (H),
leading to equivalent re-writing:
1xt (txx) = Ze
nxt (tam) A(Ve.. ea Eva) E (H)
(15)
Substituting 15p into 12 we can express g as follows:
g(c,z, jine
21 stxt
2 = g(c, img txt
2
2 A(V (am....... egg)'d
(16)
where RC d (Ve.. E [texx t (Vl))vVe. ([tax txl)) Wk (1....,.), C}.
with de denoting the k-th entry of the C' dimensional vector d. From 12 and (16) the negative
variation óg(c, z mm) to before and after MEM can be expressed as:
óg(c, me X(V pa pg gl
(17)
Finally, for any class, we can rewrite its final probability and as function of its initial probability jiii and the variation of before and after TTA for the same class:
pnd (c) = 1 (ii(c))
Σ
8g(c, 2 img)
(18)
From Eq. 18) we have that if pint (niit( (cinil) 2 E, dg(cinn, 2, img ) then the final probability prn (cinit) In the optimal case for MEM the entropy of yend is minimized, which entails that only one class can have a probability strictly greater than O. Hence, dinit and
15

---

Table Empirical evidence supporting Proposition 2.1.
Preposition
IN-IK
IN-A
IN-v2
IN-R
IN Sketch
arg max prd arg max
[%6]
95.73+0.05
95.55:0.12
94.8620.17
96.7810.08
91.23+0.009
A.2 Experimental verification
We support the previous proposition with empirical evidence, by manually counting how often the prediction of p is invariant to Test-Time Prompt Tuning by MEM. This experiment is easy to reproduce and consists of the following: augment N times, filter by confidence. compute p" optimize by MEM, compute pnd and check if arg max p" = arg maxp We report the proportion of samples for which the proposition holds for all Natural Distribution Shifts datasets in Table 4 averaged over 3 rums with different seeds (the same used in Sec. Ff of the main body). Although the proposition only accounts for the cases where entropy is globally minimized, the table shows that the marginal probability distribution is largely invariant to MEM. In the best case AimagNet-SSkttch MEM alters prediction p only 8. 8.77% the times. the worst case (ImageNet-R), the prediction unaltered for 96. 78% samples.
A.3 Can invariance be anticipated?
In the proof of Proposition 2. express the post-MEM embeddings as function of the pre-MEM embeddings through Taylor expansion. For this relationship to hold, the variation needs to be small. If the initial entropy is high, the gradients from MEM (and, thus, the variation between pre- and post-MEM embeddings) can be larger than what a Taylor expansion can accurately approximate. In such cases, Prop. 2.1 cannot guaranteed. We execute simple experiment using the validation set of ImageNet-1K, 1k, whose recipe is described below, to visualize this relationship.
We compute pre- and POst-MEM marginal probability distributions. We the pre-MEM distribu- tions in order of descending entropy (most to least uncertain) and quantize them into 10 bins. Bins shall be interpreted as follows: the leftmost bin contains the top 10% of samples with the highest entropy; the second bin contains samples outside the top-10% percentile but within the top-20%, and so on; the rightmost bin contains the bottom 10% of samples with the lowest entropy. For each bin we compute the invariance ratio, measuring how often the arg max of the pre-MEM p does not change after MEM. Finally, we display histogram with this data in Figure 2
trend appears: as the entropy decreases (left to right), invariance holds more and more often. Hence, intuitively, the most likely cases where invariance MEM does not hold those of high uncertainty in the initial marginal probability distribution. However, this may still be rare: even within the top 10% of most uncertain samples, invariance holds more than 82% of the time (leftmost bin).
1.0
Invarianos ratio
0.8
0.7
Invarionce
0.5
2.83
1.41
1.1
0.62
0.41
0.12
0.04
Mean
Hi Inity
per
bin
Figure 2: Entropy of the PTe-TTA marginal probability distribution the invariance ratio.
16

---

Table Results on Natural Distribution Shifts when adapting CLIP-ViT-B ViT---16 pretrained on the 2B English subset of LAION-5B. Top-1 accuracy is reported, and bold text indicates the best performer.
Method
ImageNet
Sketch
Mcan
CLIP--VIT-B-16 (LAION2B)
Zero-Shot
69.27
37.08
61.27
78.83
54.85
60.26
Ensemble
70.43
38.32
62.28
80.41
55.54
61.40
TPT
70.61
41.94
62,96
80.40
55.48
62.28
ZERO
71.39
48.71
63.53
80.59
55.82
64.01
ZERO,Beeebble
72.14
49.02
64.32
82.42
56.53
64.89
Table 6: Results on Natural Distribution Shifts when adapting OpenAT's CLIP-ViT-B-16, with CoOp-learned prompts. Top-1 accuracy is reported, and bold text indicates the best performer.
Melhod
ImageNet
CoOp
TPT
73.64
57.77
66.72
78.03
49.56
65.14
ZERO
74.12
61.57
67.15
78.43
49.77
66.21
Table Fine-grained Classification with CLIP-ViT-B- pretrained on the 2B English subset of LAION-5B. Top-1 accuracy is reported, and bold text indicates the best performer.
Method
FLWR
DTD
PETS
CARS
CAL
FOOD
SUN
AIR
Mean
Modian
CLIP-VIT-B--A (LAIOR2B)
Kero-Shut
69.71
54.43
89.37
89.94
64.02
95.82
81.38
70.60
26.04
47.05
68.84
70.16
Ensemble
68.70
54.55
87.76
89.98
67.64
96.51
81.64
70.62
25.68
49.64
69.27
69.66
TPT
69.47
54.53
89.00
90.72
66.68
96.16
81.76
71.34
26.73
48.81
69.52
70.41
ZERO
70.82
55.20
89.77
91.95
67.23
96.13
83.65
71.21
28.25
45.01
69.92
71.02
ZERO4
68.01
55.95
87.67
91.87
69.11
96.54
83.83
71.09
28. 10
47.10
69.93
70.10
B Additional Experiments: LAION-2B Pretraining, Context Optimization and Hyperparameter Inheritance
This Appendix deals with enriching the experiments of Section 4 which focused on models officially released by OpenAl (28) Here we focus on the comparison with TPT 38 and extend the analysis
to: ① CCIP-ViT- B-16 pretrained on the 2B English Subset of LAION 5B 34 OpenAl's CLIP, transferred after supervised Context Optimization (CoOp) [56).
Implementation Details. For the experiments with LAION Pretraining, we use the open_clip repos- itory, i.e., the official code for 2 The pretrained keyword for this model is laion2b. s34b_ _88k. For CoOp we use the context vectors learned on ImageNet-Ik 1k officially released by [566. The experi- mental setup is analogous to Section in all details. We do not tune any hyperparameters for these different initializations, but inherit them from the experiments with OpenAl models.
B.1 LAION-2B Pretraining
Table reports experiments on Natural Distribution Shifts, from which we observe no differences w.E.f
OpenAl models: ZERO largely outperforms TPT, and peak difference is reached with ImageNet-A
[3]. Results on Fine- line-grained Classification are given in Table 7 We observe that ZERO improves zero-shot baseline better with this pretraining, and overcomes TPT with an average margin of +0.4%. In contrast, ensembling textual prompts appears less effective. We speculate this is because the 7 templates were explicitly tuned and selected for OpenAl models. The worst-case scenario confirmed with satellite imagery [[] please refer to Appendix F for deeper investigation.
17

---

Table Natural Distribution Shifts (percentile 0.1). TTA methods are grouped according to the baseline model and top-1 accuracy is reported. Bold text is the best method within each group.
Method
ImageNet
Sketch
Average
CLIP-ViT-B-I-
Zero-Shot
66.73
47.87
46.09
59.11
Ensemble
68.34
49.89
61.88
77.65
48.24
61.20
TPT
68.98
$4.77
63.45
77.06
47.94
62.44
ZERO
69.06:2:0.04
61.35:1:026
64.13-+0.17
77.28=00008
48.29:10.04
64.02
ZERO.,Raemble
70.93:10.02
64.06:1:.099
65.16:10.21
80.75:10.08
50.32:10.09
NaPLe
Zero-Shot
50,90
64.07
76.98
49.15
60.28
TPT
58.08
64.87
78.12
48.16
62.31
PromptAlign
59.37
65.29
79.33
50.23
63.55
ZORO
64.65::0.24
66.63-00.32
79.75=0.41
50.73=20.62
65.44
CLIP- SLIP-VVT-B--1- CLIP-VAT-1--
68.82
fas RLCF the
65.45
74.85
73.71
69.77
86. 19
57.
72.32
ZERO
74.48:1:.12
77.07:10.35
69.53:20.1
86.87=00.05
$8.59=00088
73.31
Table 9: Finegrained classification (percentile 0.1). Formatting follows other tables.
Method
FLWR
DTD
PETS
CARS
UCF
CAL
FOOD
SUN
AIR
ESAT
Mean
Median
CLIP-ViT-B-1E
Zero Shot
67.44
44.27
88.25
65.48
65.13
93.35
83.65
62.59
23.67
42.01
63.58
65.31
Ensemble
67.07
45.09
88.28
66.16
67.51
93.91
84.04
66.26
23.22
50.42
65.20
66.66
TPT
68.75
47.04
87.23
66.68
68.16
93.93
84.67
65.39
23.13
42.86
64.78
67.42
ZERO
67.07
45.80
86.74
67.54
67.64
93.51
84,36
64.49
24.40
39.60
64.11
67.31
ZURO.ir
66.82
45.86
87.20
68.48
68,57
94.14
84.58
66.90
24.42
43.77
65.07
67.69
NaPLe
Zero
72.23
46.49
90.49
65.57
68.69
93.53
86.20
67.01
24.74
48.06
66.30
67.85
TPT
72.37
45.87
90.72
66.50
69.19
93.59
86.64
67.54
24.70
47.80
66.49
68.37
PromptAlign
72.39
47.24
90.76
68.50
69.47
94.01
86.65
67.54
24.80
47.86
66.92
68.98
ZERO
71.20
47.70
90.17
67.91
69.49
94.12
86.78
67.55
25.57
41.05
66.15
68.70
CLIP- ViT V1T---16
CLIP V:T-L-14
ZhroShot
75.76
51.83
92.86
76.
73.70
88.03
66.96
30.54
54.38
74.73
RLCF futs
67.33
23.71
fus
72.91
72.31
69.04
RLCF
72.56
52.21
89.51
63.12
71.49
94.65
86.90
68.50
24.06
47.74
67.07
70,00
RLCF by
71.74
53.27
91.15
70.93
73.24
94.73
87.28
69.38
28.54
47.41
68.77
71.34
ZERO
75.34
54.22
92.90
77.33
87.57
68.05
32.11
42.74
69.90
74.80
B.2 Context Optimization (CoOp)
For this comparison, we follow (38 and report CoOp Natural Distribution Shifts only, presenting results Table We further observe patterns consistent with OpenAl models, with ZERO providing large improvements over TPT. Also here, the best-case scenario persists with ImageNet-A.
B.3 Hyperparameter Inheritance
In all experiments so far, including Section 4 as well as Tables 5 6 and we employed percentile for sefiidnncebassed filtering set to 0.3. This value was obtained after validation on ImageNet- 1k with OpenAl's CLIP-VAS -II----V---16- and kept fixed for all models and datasets. Here, we show that ZERO obtains favorable performance even if the percentile for confidence basedd filtering is not tuned in any way, but set to 0.1 by "inheriting" the value used TPT 38 These results are given in Tables and Surprisingly, some datasets within the Natural Distribution Shifts benchmark benefit from this more restrictive filtering (ImageNet-A above all), while we observe that Finegrained classification tends to improve when more views are retained. The core findings, however, entirely unchanged: the best
18

---

case remains ImageNet-A, the worst-case remains EuroSAT, and ZERO outperforms competitor in most datasets, no matter the experimental setup.
C Calibration and Overconfidence of CLIP on augmented Natural Distribution Shifts
In Section 3.1 of the manuscript, the validation set of ImageNet-Ik ik is shown convey that overconfi- dence emerges as critical issue when predicting over augmented views. In this appendix, we ex- pand the analysis to the 4 datasets for robustness to Natural Distribution Shifts (NDS) [33 [2] 32 46]. For all datasets, we follow the augmentation setup of S.1 and generate augmented counterparts with 6x more examples.
IN-A
22.2
1N-52ecch
20.2
IN
IN-K
sngesezted
First, let define the calibration of DNNs. Cal- ibrating DNNs is crucial for developing reliable and robust Al systems, especially in safety-critical applications. A DNN is perfectly calibrated if the probability that its prediction is correct (ÿ y) given confidence score random variable $ is equal its confidence score. The confidence
30 25
20
20
Expected Calibration Error [%6
Figure Expected Calibration Error (ECE) 8 of CLIP-ViT-B-D across 5 datasets for robust- ness to natural distribution shifts. Blue is the ECE of zero-shot CLIP, and orange the ECE of zero-shot CLIP on an augmented version of the dataset after confidence-based thresholding.
score is commonly taken as the maximum of the output probability vector of the model, i.e.,
maxp(.)
P(ù = y|s =
(19)
To evaluate the expected calibration error (ECE), we typically split the dataset into M bins Bm based on their confidence scores, We then calculate the accuracy of each bin, denoted as acc( Bm). and the average confidence, denoted as (m.) The ECE is defined by the following formula:
1 M
ECE
Σ
|ccc - [m))
(20)
Then, we show how the ECE of CLIP-ViT-B-16 varies between "source" and augmented versions of all datasets (ImageNet-Ik included) in Figure From this experiment, we observe large increase in the ECE across all datasets. In no cases, the ECE remains comparable to its default value when no augmentations are present. As we discussed in 3.1, the calibration error increases when the model is either more accurate than confident (signaling umderconfidence) or the opposite, signaling overconfidence. Reliability diagrams are standard tool understand which is the case, hence we show them for all 4 NDS Datasets in Fig 4 These results are entirely consistent with Sec 3.1 the calibration error increases exclusively due to overconfidence, no matter the dataset. In parallel, the eтror rate of CLIP- ELIP-ViT-BI --i---i-BB--- 16 can either remain close to its default value (e.g.. ImageNet-Skeete -Seetchh. slightly decrease (e.g., ImageNet-R and -v2) or largely decrease (lmageNet-A). We observe an identical pattern for CLIP models pretrained on LAION. For reference, see Figure
D On the expected risk of p and p.
The expected risk of a classifier f is commonly defined as the expectation of the risk function over the joint distribution of data and labels.
R(F) B(z.g)>Pxy [(y f(x))
(21)
In 16., the expected risk of classifier f(x) p(-1x), which predicts by marginalizing over several augmented views, is theoretically shown to lower-bound the empirical risk of standard classifier p(-[x) when the risk function É is squared error, i.e., f(a, (a - 2
19

---

Accuracy
Gap
ImageNet-A
Augmented
1.0
BCB
ECB 22.24%
0.8
Ancuray (1] 47.87%
Accuracy (1 58.5%
Accuracy
0.6
0.2
0.0
Image Net-v2
Augmented
1.0
ECE (L 3.0%%
ECE (4): 15.09%
0.8
Accurncy (1)) 60L.0%
Accuracy 62..6%
Accuracy
0.6
0.4
0.2
0.0
ImageNet-R.
Augmented
1.0
ECE (L): 3.35%
ECE (4): 0.50%
0.8
Accuracy (1): 73.96%
Accuracy (1)) 75.19%
Accuracy
0.6
0.4
0.2
0.0
ImageNet-Skcee Sketch
Augmented
1.0
BCB (1
ECB (J): 20.10% Accurscy (t') 16.61%
0.8
Accurscy (1)) 46.00%
Accuracy
0.6
0.1
0.2
0.0
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
Confidence
Confidence
Figure Reliability diagrams (20 bins) for CLIP-ViT-B--- the 4 datasets for Natural Distribution Shifts. In each row, left is the ECE on the source dataset, right on the augmented and filtered version. Row 1: ImageNet-A [3] Row 2: ImageNet-v2 [32] Row ImageNet-R [12): Row 4: ImageNet-
Sketch 46].
Here, show that such bound can be extended to any risk function & that checks the triangular inequality. Specifically, note that & satisfies the triangular inequality, then:
(x))(x)) N
(g.p(xx)))
(22)
20

---

ImageNet-lk 1k
Accuracy
Gap
Augmented
1.0
ECE [J]: 8.53%
ECE (4): 28.24%
Accuracy (+ 66.73%
Accurncy (-): 68.80%
0.8
Accuracy
0.6
0.4
0.2
0.0
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Figure 5: Reliability diagram (10 bins) for CLIP-ViT-BB B-16 pretrained on LAION-25 when trans-
ferred zero-shot on ImageNet-1k, 1k. (left) Source Dataset, (right) Augmented version of the dataset.
The above inequality is obtained following these simple steps:
N
1 N Ee ((xxxx)
1 N
- -()]( = 1
Σ
p(x))
Σ
-((x-((x))) (23)
Applying the expectation operator over the joint distribution Pxy to both sides of Eq. 22 leads to:
R(d)
R(p) R(p).
(24)
N 1=1
Hence, the empirical risk of p lower-bounds that of p for any risk function é satisfying the triangular inequality.
E
Tie breaking with ZERO
A caveat of ZERO ties, Le., cases with multiple classes having identical probability within the marginal probability distribution. This is clear to see when viewing ZERO as its equivalent paradigm of voting among confident views, simply because more than class may have equal amount of "votes".. Throughout all the experiments of this work, ties are broken greedily. a tie results from the top views, the procedure for breaking follows these steps: @ sort the remaining views by ascending entropy (most to least confident) and @ scan the views until a prediction is encountered that breaks the tie. Other than this, many alternative are possible, such as relying on the most confident prediction. Specifically, we have explored the following alternatives:
1. greedy tie breaking, discussed above;
2. relying on the most confident prediction;
3. computing several marginal probabilities for p. each by marginalizing over views with identical predictions, and picking the one with the lowest entropy for the final decision;
4. relying on the maximum logit (pre-Softmax);
5. using the averaged logits (pre-Softmax);
6. doing similar to point 2, using logits instead of probabilities;
7. random tie breaking;
and did not find consistent behaviour across all (fine-grained and NDS) datasets, suggesting this is indeed minor component. We opted for greedy tie breaking due its slightly better performance on the ImageNet validation set.
21

---

A failure mode for TTA: satellite imagery
In our experiments, we find that extremely OOD domain represents consistent failure mode for TTA: satellite imagery. all comparison groups, a zero-shot baseline largely outperforms any TTA strategy when evaluated on EuroSAT[II]
in Group the zero-shot baseline CLIP CLIP-Ensemble largely outperforms the best TTA strategy Zmeoelme
in Group 2, zero-shot MaPLe outperforms PromptAlign;
in Group 3 the best RLCF pipeline lies far behind the zero-shot teacher CLIP-ViT-L-14,
Here, we qualitatively and quantitatively report two main root causes for failures.
Qualitatively poor augmentations. In principle, TTA methods should rely generic data aug- mentations, since not doing so would require going against the principles of the field by assuming some prior knowledge about the test data is available. As discussed in Sec data augmentations are a doubled edged sword in TTA, and failing in crafting properly augmented views can potentially generate misleading uninformative visual signals. We report
qualitative examples conveying this problem in Figure 7 In the Figure, we report three images from [[]. together with the top-3 augmented views leading CLIP-ViT-B-16 reported with the groundtruth, and all views are reported with both the prediction and the confidence of CLIP. Visually, one can perceive that the simple data augmentation scheme of cropping and flipping, which has largely been proven successful in 38 33 and in our work, does not provide informative views, since most are alike one another.
its most confident predictions. Each source image is
Quantitatively high error. Augmentations are used by all TTA methods discussed in this paper, hence the previous discussion holds for TPT as much it does for PromptAlign, RLCF ZERO. Nevertheless, we highlight an ad-
ditional caveat about satellite im-
agery which is particularly detri- mental for ZERO, and relates the
EurSATT
Gap
Augmented
1.0
80B 14 T.0K5
[C) 22 a0s
base model error over augmenta- tions. Recall that, in ZERO, the usage of P is backed by theoret- ical motivations, and the manual adaptation of the temperature is sup- ported by two concurrent observa- tions: iugmentations-inducedd over- confidence and comparable error rate between source and augmented images. Simply put, the latter con- dition is not verified for satellite im- agery. To show this phenomenon, we follow the experimental setup of Sec[..] and examine the reliability diagrams of EuroSAT(II] and of its augmented counterpart in Figure As per Section 3.1 display the ECE and the Tocc-Acc on each version of the dataset. From this perspective, one can note that the base model error largely increases, in this domain, when augmented views are present. The accuracy on source images is 42.01% %, dropping to 35.21% simply due to augmentations.
Accuracy (1): 42.033
Accunacy (1): 56.255
Accuracy
0.6
0.4
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Figure 6: Reliability diagrams of CLIP-ViTS SAII--Vi-B--11 16 for EuroSat and its augmented version, generated following 3.1
Both observations, combined, suggest that crafting augmentations for satellite imagery requires an ad-hoc treatment, which makes a controversial benchmark for TTA.
G Natural Distribution Shifts vs Fine-grained Classification
Throughout the manuscript, one can observe that ZERO consistently provides larger improvements in Natural Distribution Shifts than does in the Finegrained suite. We thus devote this section to digging deeper into this matter.
22

---

Table 10: Comparison among ① CLIP's zero-shot accuracy, CLIP's accuracy on the augmented counterpart of the dataset, and @ ZERO. The augmented datasets are crafted following the protocol of Section 3.1 "Gap" is defined as CLIP's zero shot accuracy minus its accuracy on the augmented dataset. "Improvement" is defined as the accuracy of ZERO minus that of zero-shot CLIP. Spearman's s coefficient between "Gap" and "Improvement" equals -0.95: as the "Gap" decreases (i.e., the lower the error augmented views) ZERO provides more substantial improvements.
Method
FLWR
DTD
PETS
CARS
UCF
CAL
FOOD
SUN
AIR
3: Zero
67.44
44.27
88.25
65.48
65.13
93.35
83.65
62.59
23.67
3 Augmented
66.19
44,90
86.17
65.88
63,59
92.62
83.25
62.97
23.52
3 7FRO (pere - 0.1)
67.07
45.80
86.74
67.54
67.64
93.51
84.36
64.49
24.40
Gap 1 8
+1.25
-0663
+22008
-0.40
-0.46
+-0.73
+0.40
-0.38
+0.15
Improvemeent
-0.77
+1.53
1.51
+2.06
+2.51
+0.16
+0.71
+1.90
+0.73
Perhaps unsurprisingly, we posit that ZERO improves over the zero-shot baseline if the zero-shot error rate of the model does not largely increase with augmented views. As Fig[[b) displays, this is the case for all Natural Distribution Shifts datasets. To understand any different behaviors, we repeat the same experiment of Section for for the entire Fine-grained suite and report the results in Table T0 Please note that, in the table, the percentile for confidence-based in filtering in ZERO is set to O.T, since the protocol for generating the augmented datasets follows the setup of TPT, which also uses a cutoff percentile of 0.1, and that omit EuroSAT since analogous experiment presented in the previous Appendix.
Overall, we observe strong correlation between the error gap and the improvement provided by Zero, with Spearman's 's coefficient being -0.95 across datasets. This result shows that the correlation is negative, i.e., the lower the error gap, the larger the improvement (or, in other words, the better the zero-shot performance on augmented views, the larger the improvement of ZERO). This pattern is also consistent with the experiments on EuroSAT reported in the previous Appendix. Understanding why augmentations induce larger or smaller errors may be a case-by-case matter that relates to the nature of the datasets. Here, pinpoint two possible reasons:
The semantic space of the ImageNet variants of the Natural Distribution Shifts benchmark comprises many common categories, which may have appeared frequently during CLIP's pretraining. Hence, it seems reasonable that CLIP is robust w.r.f. augmented views of images belonging to these categories. In the Fine-grained classification suite, datasets such as SUN397 and Caltech101 101 also contain common object categories, which is consistent with the results shown above. In contrast, other datasets such as Flowers 102 and Oxford-Pets span much less frequent concepts.
Other than the semantic classification space, images' visual appearance also plays an important role. For example, datasets such FGVC-Aircraft and Stanford Cars still contain rare concepts, but ZERO largely improves over the baseline nonetheless. Our augmentation setup is simple, and only contains random resized crops and random horizontal flips. which can constitute "zoom-in" to random portion of the image. For some benchmarks, this is useful as it may trigger CLIP's capabilities to recognize small details, such logos, or even reading text, such as the car brand or the airline name. In contrast, more object-centric datasets such as Flower102, may lead to missing precious visual features (e.g., the stem).
In our work we did not search for the best data augmentations but rather stuck to an established setting, using the same augmentations setup for all datasets. Nevertheless, the performance of ZERO is linked to the impact that data augmentations have on how the model perceives images, and believe this is an interesting research direction to pursue.
H Independence among views in the setup of Test- Time Adaptation
The theoretical framework of Section 2.3 models an ideal scenario, where independence holds among different inputs. To clarify, this means that the model's error on view should not be correlated with the error on any other view x,, which allows writing the compound error with binomial distribution as in 6
23

---

Table 11: Standard deviations of ZERO for Fine- tine-grained classification. Each cell refers to Tab. 2
Methed
FLWR
DTD
PETS
CARS
UCF
CAL
FOOD
SUN
AIR
ESAT
CLIP-ViI-B-16
ZERO
+0.12
+0.07
+0.06
#0.15
=0.24
#0.14
#001
#0.10
#0.12 +:047
#0.11
7FRO; Esscuble /
1:0.07
10.26
10.16
10.04
+0.07
10.19
2004
18.18
10.08
MaPLe
ZERO
+0.51
1041
10.52
10.66
10.10
+0.40
1:033
1.4.7
CLIP-VIT-B-16
CLIP-VIT-L-RE
ZERO
::0.11
+0009
+0.15
10.19
10.14
20.04
#0.08
+0.03
+0032
1:019
In practice, achieving perfect independence is challenging, if not impossible. Hence, a suitable approximation strategy mitigate this issue promote diversity. In classical ensembling theory, well-established approach is to train different models on different subsets of the available data. Similarly, the augmentation scheme random cropping aligns with this approach by presenting the model with different portions of the image each time.
Moreover, ideally, the augmentation pipeline should not change the underlying label of the original input and guarantee that the model's error rate on augmented views remains comparable to the error rate the original inputs belonging the same category. In practice, this entails that augmentations should not disrupt the visual appearance of the image, and, consequently, some views may result in slight moderate correlation, because some "parts" of the source image will overlap among them. An analogy with classical literature can be drawn also in this case. Specifically, when not enough data are available, overlaps among the training sets of different models are required to ensure convergence. Consequently, models producing slightly or moderately correlated predictions are more likely to emerge.
I Additional Implementation Details
Standard deviations. To complement the results on Fine-grained classification, we report the standard deviation of ZERO computed over 3 runs with different seeds in Table [[] These are not reported together with the average topl-accuracy in Tab, to avoid an excessively dense table. On average, standard deviations are very small, suggesting that regardless of the inherent randomness of data augmentations, ZERO is relatively stable. Note that standard deviations in Group 2 (i.e., with MaPLe) are slightly greater than those the other groups This fact does not stem from ZERO's 's or MaPLe's 's greater instability, but from experimental detail which report here for completeness: while only one set of weights is officially released for each CLIP version (28). Khattak et al. [15] released 3 sets of pretrained weights for MaPLe, varying the seed. To avoid picking one, associated a set of weights to each of our runs, hence results from slightly different initializations are computed match the experimental setup of Samadh al. 33
(PromptAlign).
Reproducibility of TTA methods. For section 4 we reproduced methods using the source code provided by the authors with the hardware at our disposal. This was done to ensure that hardware differences did not interfere with a correct evaluation. We found that all TTA strategies are highly reproducible, with negligible differences (Le.., A 0.1) which we omitted by reporting the numbers from the official papers. case larger differences, we reported reproduced results.
24

---

Forest
Highway or Road (0.76)
Highway or Road (0.73)
Highway or Road (0.75)
Herbaceoous Vegetation Land
Sea or Lake (0.48)
Sea or Lake (0.44)
Sea or La (0.43)
Sea or Lake
Highway or Road (0.24)
Highway or Road (0.24)
Highway or Road (0.25)
(a) Source Image
(b) Most confident
(c) 2nd most conf.
(d) 3rd most conf.
Figure 7: Examples of satellite imagery taken from EuroSAT[1]] along with augmentations leading high confident predictions. Column (a) reports source images with their label. Columns (b-d) report views sorted by entropy (lowest to highest), paired with the prediction and the confidence of CLIP-ViT-B-18 (31]
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This work studies TTA by MEM, and shows that method obtained by manually adapting a single parameter is hidden therein. The method, called ZERO, is supported by theoretical justifications. Experiments show that ZERO compares favorably w.E.. existing TTA methods while being faster and more memory friendly, which is what the abstract claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper contains "Limitations" section in the main body. It discusses theoretical assumptions and computational aspects. The paper discusses computational aspects also in the "Experiments" section of the main paper, especially concerning the dataset size (which is the size of the label space, in the scope of this paper). The Appendix provides evidence for the theoretical proposition of this paper and reports failure modes for the presented approach, as well as in broader TTA. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
26

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper contains proposition, whose proof the Appendix. All other theoretical results are supported by empirical evidence.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper contains implementation details in both the "Experiments" section of the main body and the Appendix. Py -ircch-likee implementation is provided alongside the proposed strategy in Section 3 of the main body.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets we have used are all publicly available, including the data splits. The manuscript links to a public GitHub repository containing the implementation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/guiden wwtp///euuuiisissooooooooeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are explicitly reported in the "Experiments" section of the main paper, including hyperparameters and data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
28

---

Answer: [Yes]
Justification: The "Experiments" section contains the standard deviations, computed over runs, of the proposed method for one out of two benchmarks used. The standard deviations for the second benchmark are reported in the Appendix, to avoid formatting an excessively dense table, and are discussed. The empirical verification for the proposition of this paper is also exccuted for runs, and standard deviations are reported in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section "Experiments", we explicitly tell which and how many GPU devices were used. Computational aspects concerning runtime and peak memory consumption are also reported in the same section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp//pppoos..ccc////////ta
Answer: [Yes]
Justification: We have carefully inspected the code of ethics, and believe our work conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper theoretically analyzes paradigm of Test-TTime Adaptation with Vision-Language Models and presents a simple and effective method. We do not see direct path from our method negative societal impact which is not inherently already related to VLMs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe the content of this paper does not have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
30

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original creators all models, datasets and algorithms used this work are properly credited, with citations around the manuscript. We used their material for the only non-commercial purpose of developing this research paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There external assets related to this paper its Appendix, except for the open-sourced implementation. The code is properly commented, on to guide readers into understanding it step-by-step, and comprehensive "readme" file for installation is given.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does involve crowdsourcing, nor the involvement of external human subjects except for the authors.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
31

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This research does not involve any participants except for the authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "HDNiPmiF08", "reviewer": "Reviewer_Wz3n", "summary": "This work carefully reviews the popular test-time adaptation (TTA) method, MEM, and finds that MEM has largely no effect on \narg max(𝑝). Based on this understanding, this work further introduces a clean method called Zero, which shows decent performance for the TTA task.", "questions": "Please refer to the weakness.", "limitations": "Please refer to the weakness.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Great motivation for bringing insights into Test-Time Augmentations.\n2. This is a simple method that can be easily integrated into the current model and algorithms. The performance on natural distribution shifts is strong, and the memory cost is impressively small.", "weaknesses": "Although the performance of the ZERO is good on natural distribution shifts, it is not as effective in fine-grained classification. It would be helpful to have more insights into why the performance in fine-grained classification is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 - Performance on Fine-grained datasets vs Natural Distribution Shifts.**\\\nWe thank the reviewer for raising this interesting point, allowing us to further investigate our method. \n\nA possible explanation may be linked to Sections 2.3 and 3.1 of the manuscript. Specifically, Zero improves over the zero-shot baseline if the error rate does not largely increase with augmented views. As Fig.1(b) of the manuscript displays, this is the case for all Natural Distribution Shifts datasets. For Fine-grained classification, we discuss here three distinct datasets for which Zero exhibits different behaviors: \n1. $\\text{Flowers102}$. Zero does not improve over the baseline here. \n2. $\\text{Caltech101}$. Zero marginally improves here. \n3. $\\text{SUN397}$. Zero largely improves here.\n\nTo understand these different behaviors, we repeat the same experiment of Section 3.1 for the entire Fine-grained suite and report here the results for the aforementioned datasets, formatted as follows: $[ \\text{zero-shot accuracy}, \\text{augmented version accuracy}, \\text{error gap}]$.  \n\nBy $\\text{error gap}$, we refer to:\n$\\text{error gap} = \\text{zero-shot accuracy} - \\text{augmented version accuracy}$. \n\nWe additionally include the accuracy of Zero.\n\n**Results [%]**\n1. $\\text{Flowers102} = [67.44, 66.19, 1.25]$ | $\\text{Zero} = 67.07$ ($-0.37$ w.r.t. zero-shot baseline)\n2. $\\text{Caltech101} = [93.35, 92.62, 0.73]$ | $\\text{Zero} = 93.51$ ($+0.16$ w.r.t. zero-shot baseline)\n3. $\\text{SUN397} = [62.59, 62.97, -0.38]$ |  $\\text{Zero} = 64.49$ ($+1.90$ w.r.t. zero-shot baseline)\n\nFor completeness, **all results are reported in Table 1 of the PDF attachment of the general response**. \n\nOverall, there is a strong correlation between the error gap and the improvement provided by Zero, with Spearman’s coefficient being $-0.95$ across all datasets. This result shows that the correlation is negative, i.e., the lower the error gap, the larger the improvement. This pattern is also consistent with the experiments on EuroSAT reported in the Appendix of the manuscript (App.E, Tab.5).\n\nHence, the reviewer’s question is directly linked to this follow-up question: *Why do augmentations increase or decrease the error gap with different datasets?* While this may be a case-by-case matter, we pinpoint two possible reasons:\n\n1. The **semantic space** of the ImageNet variants of the Natural Distribution Shifts benchmark comprises many common categories, which may have appeared frequently during CLIP’s pretraining. Hence, it seems reasonable that CLIP is robust w.r.t. augmented views of images belonging to these categories. In the Fine-grained classification suite, datasets such as SUN397 and Caltech101 also contain common object categories, which is consistent with the results shown above. Other datasets, such as Flowers102 and Oxford-Pets, span much less frequent concepts, and CLIP is less robust w.r.t. their augmented views.\n\n2. Other than the semantic classification space, also the **visual appearance of images** plays an important role. For example, datasets such as FGVC-Aircraft and Stanford Cars still contain rare concepts, but Zero largely improves over the baseline nonetheless (main paper, Tab.2, first comparison group). Our augmentation setup is simple, and only contains random resized crops and random horizontal flips, which can constitute a “zoom-in” to a random portion of the image. For some benchmarks, this is useful as it may trigger CLIP’s capabilities to recognize small details, such as logos, or even reading text, such as the car brand or the airline name. In contrast, for Flower102, these may lead to some loss of precious visual features, such as the stem. \n\nTo conclude: In our work we did not search for the best data augmentations but rather stuck to an established setting, using the same augmentations setup for all datasets. Nevertheless, the performance of Zero is linked to the impact that data augmentations have on how the model perceives images, and we believe this is an interesting research direction to pursue. \n\nWe also think that this may be a useful discussion, and plan to include it in the Appendix of the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mA9yYu13u4", "reviewer": "Reviewer_7eTE", "summary": "This work shows that Marginal Entropy Maximization (MEM), a leading class of methods for Test Time Adaptation (TTA) which involves maximizing the entropy of the predictive distribution marginalized over different views of the input, regularly results in the same argmax (and thus same final class prediction) as the argmax of this marginal distribution. This means that the entropy maximization step, which requires not insignificant computational overhead, often does not provide any benefit. \n\nThey then show that a very simple approach involving just setting the temperature of each predictive distribution to 0 prior to marginalization is a very strong baseline that outperforms MEM while being much more computationally efficient.", "questions": "- In section 2.3 in the \"Revisiting model assembling for TTA.\" paragraph, do different views of the input really count as independent samples? it seems like they would be highly dependent on each other.", "limitations": "The limitations section is sufficient. Though I think ideally it would be part of the main paper, rather than in the Appendix.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors discover a critical flaw in MEM approaches \n- They propose a simple but very effective baseline that outperforms state of the art methods\n- They provide theoretical analysis of their approach \n\nI like this paper quite a bit. It exposes a critical flaw in a popular class of TTA methods, and provides a strong, theoretically-backed alternative method that is very simple to implement.", "weaknesses": "- In section 2.3 in the \"Revisiting model assembling for TTA.\" paragraph, do different views of the input really count as independent samples? it seems like they would be highly dependent on each other. \n- I would tone down some of the stronger claims just a bit. For instance, the paragraph header \"Poor calibration is *always* caused by overconfidence.\" might be overclaiming, since from my understanding this is an empirical observation on a sample of datasets, not an analytical fact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 - Q1: Independence among views.** We agree and we thank the reviewer for this remark, which allows us to enrich our manuscript further.\n\nThe theoretical framework of Section 2.3 models an ideal scenario, where independence holds among different inputs. To clarify, this means that the model's error on view $\\mathbf{x}_i$ should not be correlated with the error on another view $\\mathbf{x}_j$, which allows writing the compound error with a binomial distribution.\n\nIn practice, achieving perfect independence is challenging, if not impossible. Hence, a suitable approximation strategy to mitigate this issue is to promote diversity. In classical ensembling theory, a well-established approach is to train different models on *different subsets* of the available data. Similarly, our augmentation scheme of random cropping aligns with this approach by presenting the model with *different portions* of the image each time.\n\nMoreover, ideally, the augmentation pipeline should not change the underlying label of the original input and guarantee that the model’s error rate on augmented views remains comparable to the error rate on the original inputs belonging to the same category ($\\epsilon(y)$ in the caption of Figure 1(a) of the manuscript). In practice, this entails that augmentations should not disrupt the visual appearance of the image, and, consequently, some views may result in a slight or moderate correlation, because some “parts” of the source image will overlap among them. An analogy with classical literature can be drawn also in this case. Specifically, when not enough data are available, overlaps among the training sets of different models are required to ensure convergence. As a consequence, models producing slightly or moderately correlated predictions are more likely to emerge.\n\nWhile we have tried to highlight this potential limitation in the dedicated section, we believe a tailored discussion may be helpful for the readers. Hence, we propose to replace lines 139-140 with a pointer to an in-depth appendix on this subject, where we will include this discussion.\n\n**W2: Overconfidence and \"always\".** We agree with this remark. As we acknowledge in the Limitations section, this is an empirical observation stemming from the combinations of models and datasets that we tested, and could not extend to the space of all existing VLMs and datasets (or those that will arise in the future). \n\nAs suggested by Reviewer `a5fT`, **Figures 1(a) and 1(b) of the PDF attachment show additional experiments with LAION-pretrained CLIP models, which confirm our initial observations**. We hope that these will strengthen the analytical section, and plan to include these results in the Appendix.\n\nFinally, we propose detailed changes to the manuscript to tone down some passages from lines 192 to 197 (shared with reviewer `a5fT`):\n1. line 192, paragraph header: “Poor calibration is frequently linked to overconfidence.”\n2. lines 195-196: “Notably, in the scope of our experiments, overconfidence is the primal factor leading to an increase of the ECE.”\n3. line 197: “In Appendix B, we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on LAION. Importantly, this phenomenon further persists within this extended experimental suite.”\n\nThank you for this suggestion.\n\n**About Limitations.** We are glad that our efforts in highlighting the limitations of our work were appreciated! We agree that, ideally, a Limitations section shall be part of the main body of the paper. This year NeurIPS is granting an extra page for the final revision of the manuscript upon acceptance. We will use it to follow this suggestion if this is our case. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhBceLtaH3", "reviewer": "Reviewer_a5fT", "summary": "This work studies the test-time adaption (TTA) of Vision-Language Models (VLMs), where the goal is to adapt the trained VLMs to unseen datasets/distribution. To this end, this work first revisits the commonly used Marginal Entropy Minimization (MEM) by showing its effect on marginal probability distribution $\\bar{p}$. Then, the relationship between $\\bar{p}$ and ${p}$ is shown. Based on this, a simple TTA method is proposed, which uses \"zero\" temperature when calculating the probability with softmax. The experimental results on several benchmarks demonstrate that the proposed method is effective, and can bring improvements over several baselines.", "questions": "- Please discuss the cases where invariance to entropy minimization (Section 2.2) may not hold\n- Please claim in Line 192 \"Poor calibration is **always** caused by overconfidence\" moderate and provide more experiments to support such observation\n- Please correct my understanding of Zero temperature: the temperature (not negative) will not change the Softmax predicted class (the predicted class corresponds to the maximum predictions). I am not sure how Zero temperature impacts the predicted class. Please clarify how it improves the performance. \n- [Small suggestion] TPT [37] and PromptAlign [32] use the protocol that only a single test sample is given. This work follows such protocol. I would suggest this work highlight this. The current presentation is not very clear to me", "limitations": "This work provides a discussion of potential limitations, including the augmentation-induced overconfidence might not hold on future VLMs, the invariance to entropy minimization may not hold on all models and datasets, and the independence among augmented views, the computational cost. I appreciate this work mentions these limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method (ZERO) is simple and straightforward. The experimental results are good and several baselines are included and discussed, which helps understand the effect of ZERO.\n\n+ Section 2.2 (how does MEM affect the marginal probability distribution) is interesting. It gives some insights that the prediction is invariant to entropy minimization\n\n+ Section 2.3 gives the reliability perspective into the marginal probability distribution ($\\bar{p}$). Showing the error of $\\bar{p}$ is a lower bound to the base error of the model is interesting. Giving empirical results is also helpful.", "weaknesses": "- The major concern is the invariance to entropy minimization (Section 2.2). I noticed that this work provides a discussion on this, but I would appreciate the authors give some cases/ examples where this assumption does not hold. \n\n- While I think the analysis of reliability (Section 2.3) is interesting, the claim may be too strong. Specifically, in Line 192 \"Poor calibration is **always** caused by overconfidence\". I do not think \"always\" is appropriate here. I understand the empirical results give such observation. However, this claim needs to consider various datasets/ models. For example, [a] shows the CLIP models trained on different datasets (LIAON and WiT) exhibit distinct behaviors in terms of calibration/reliability. That said, I would suggest the authors lower the claim and make it moderate. Also, I noticed the authors discuss this potential limitation, which I appreciate. Moreover, it would be better if the authors could show the results of Figure 1 (b) using CLIP models trained on LAION and WIT (OpenAI-provided CLIPs). This will make the empirical observation more convincing. \n\n    [a] A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)\n\n- I noticed TPT [37] and PromptAlign [32] evaluate models on the setting of Cross-Datasets Generalization. Why not report results on such a setting?\n\n\n**---- Post Rebuttal ---**\n\nI am not fully convinced by the response regarding the benchmark comparison, particularly since ZERO exhibits a significant performance drop on EuroSAT, which has not been clearly reported following the standard protocol. Other methods do not exhibit such a drop. An apples-to-apples comparison is fundamental to understanding the effectiveness of the proposed approach. While I acknowledge the merits of simplicity and analysis, I remain skeptical about the experimental evaluations.\n\nConsidering the current two evaluations and the lack of guarantee that “the predictions of the model, on average, are accurate,” I am not convinced of ZERO’s generalization to other tasks, backbones, or datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 - Q1: Invariance to Entropy Minimization.** Invariance to MEM is strongly related to the uncertainty of the marginal probability distribution pre-TTA. The lower the initial entropy, the lower the impact of MEM on the $\\arg \\max$.   \n\n**Theory**: this is related to the proof of Proposition 2.1. To guarantee invariance, we express the post-MEM embeddings as a function of the pre-MEM embeddings through a Taylor expansion, which implies that the variation is small. If the initial entropy is high, the gradients from MEM (and, thus, the variation between pre- and post-MEM embeddings) can be larger than what a Taylor expansion can accurately approximate, so Prop. 2.1 cannot be guaranteed.  \n\n**Empirical evidence**: To visualize the aforementioned relationship, we compute pre- and post-MEM marginal probability distributions. We sort the pre-MEM distributions in order of *descending entropy* and quantize them into 10 bins. Bins shall be interpreted as follows:\n1. the leftmost bin contains the top 10% of samples with the highest entropy;\n2. the second bin contains samples outside the top-10% percentile but within the top-20%, and so on;\n3. the rightmost contains the bottom 10% of samples with the lowest entropy.\n\nFor each bin we compute the invariance ratio, measuring how often the $\\arg \\max$ of the pre-MEM $\\overline{p}$ does *not* change after MEM (the higher the better). **Finally, we display a histogram with this data in Figure 2 of the PDF attachment**. \n\nA trend appears: *as the entropy decreases (left to right), invariance holds more often*. \nHence, to answer: the most likely cases where invariance to MEM does not hold are those of high uncertainty in the marginal probability distribution. However, this may still be rare: even within the top 10% of most uncertain samples, invariance holds more than 82% of the time (leftmost bin). \n\nThe reported experiment was conducted on the validation set of ImageNet-1k with OpenAI’s CLIP-ViT-B-16, but identical trends were observed for all Natural Distribution Shifts datasets. We plan to report these results in the revised appendix, together with this discussion and a few examples. We are thankful to the reviewer for this comment. \n\n**W2 - Q2: Overconfidence and \"always\".** We agree. Following the suggestion, we report the same experiment with LAION-pretrained CLIP models, using the code of [j]. **In the PDF attachment, the reviewer can find results for LAION-2B and LAION-400M**.\n\nNotably, these CLIP variants comply with the already observed patterns, i.e.: the ECE increases with augmented views, with overconfidence being the leading cause. We propose to include these results in a dedicated Appendix. \n\nWe further propose to tone down ll. 192-197 (shared with `7eTE`):\n1. line 192, paragraph header: “Poor calibration is frequently linked to overconfidence.”\n2. lines 195-196: “Notably, in the scope of our experiments, overconfidence is the primal factor leading to an increase of the ECE.”\n3. line 197: “In Appendix B, we also experiment across all datasets for Natural Distribution Shifts and different CLIP models pretrained on LAION. Importantly, this phenomenon further persists within this extended experimental suite.”\n\nWe thank the reviewer for this comment.\n\n**W3: Cross-datasets generalization.** These results are already in the manuscript, in the 2nd comparison group of Tab. 2 (MaPLe). We pinpoint that there are slightly different meanings to the term “cross-datasets generalization”, according to the recent literature on TTA, and that confusion may arise from these. We recap them here:\n1. This experiment was first designed in the TPT paper to compare supervised prompt learning (e.g. CoOp [55]) vs instance-specific prompt learning. In our case there is no learning, so this would not apply.\n2. The PromptAlign paper extends this setting to see how prompt learning methods, trained on a source dataset like ImageNet, can benefit from instance-specific TTA when evaluating “cross-datasets”. \n\nThis second experiment is already in Table 2: MaPLe prompts are learned on ImageNet and the model is adapted by Zero “cross-datasets”. We referred to these as the \"Fine-grained classification\" experiments.\n\nHowever, we notice that we did not explicitly mention that our MaPLe initialization comes from ImageNet. We sincerely apologize. As a remedy, we propose the following:\n1. Before l. 252: “MaPLe prompts are learned on ImageNet, following [32]”;\n2. In l. 267, we will insert: “When adapting MaPLe, we stick to the ImageNet-learned prompts and evaluate it cross-datasets as in [32].”\n\n**Q3: zero temperature.** Consider this example in a 3-way problem. Let $x_1$, $x_2$, and $x_3$ be three views and $p_1$, $p_2$, and $p_3$ be their probabilities with the default temperature. Let $y=2$ be the correct label.\\\n$p_1 = [0.9, 0.1, 0.0]$ - incorrect\\\n$p_2 = [0.3, 0.6, 0.1]$ - correct\\\n$p_3 = [0.25, 0.6, 0.15]$ - correct\n\nThis simple example is designed to meet the observations of the manuscript:\n1. the error rate of the model is low, but \n2. the model may suffer from overconfidence ($p_1$). \n\nThe resulting marginal probability distribution $\\bar{p} = [0.483, 0.433, 0.083]$ would be wrong because of the influence of a wrong overconfident example. \n\nSetting the temperature to zero before marginalizing, it would be (approximating):\\\n$p_1 = [1, 0, 0]$\\\n$p_2 = [0, 1, 0]$\\\n$p_3 = [0, 1, 0]$\n\nwith a resulting $\\bar{p} = [0.33, 0.67, 0]$, which would be correct and avoid the influence of overconfidence. \n\nThe takeaway is that due to the effect of data augmentations, confidence information can be misleading, but we can still rely on the fact that the predictions of the model, on average, are accurate. Setting the temperature to zero discards confidence information, but retains the $\\arg\\max$ (i.e., the prediction).\n\n**Q4: single-test point.** To clarify, we will include the following before l. 252: “Similarly to [37, 32, 53], we always work with a single test point at a time.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VJizSdg6mf", "reviewer": "Reviewer_Wx8e", "summary": "This paper presents a TTA strategy called ZERO, in which the author carefully and thoroughly explains the entire work, from motivation to method and experiments, including numerous appendices. However, the observations mentioned are not novel enough, and the proposed method is not sufficiently flexible.", "questions": "Questions:\n\n1. The authors' motivation is not novel, as the phenomenon of over-under-confidence has been revealed in many studies[1], making it not a surprising discovery, could you please clarify your novelty?\n\n2. The authors' method of setting the temperature to 0 seems to be a very straightforward and naive strategy. In fact, in some studies, the temperature has been made learnable instead of being forcibly set to 0[2]. \n\n3. The authors' pseudocode is quite unusual, as it is the first time I have seen code used instead of pseudocode. Is there any insurmountable reason for this choice? \n\n4. The authors have too few comparison methods. In fact, there are already many TTA strategies [3]. Do these methods also have the same problem? \n\n5. Why are only a portion of the experimental results shown? I cannot find the missing results for different model architectures anywhere, including the appendices. \n\n6. Why is the temperature set to 0? This will cause the post-softmax distribution to be extremely sharp. \n\n7. Have the authors considered that label smoothing and sharpness-aware optimization strategies may solve this problem? \n\n8. Why do the authors use some unusual terms like \"prevalent class\"? it means? I cannot even find similar descriptions in papers on arXiv or Google Scholar.\n\n[1] Singla, Sumedha, et al. \"Augmentation by counterfactual explanation-fixing an overconfident classifier.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[2] Wang, Xiao, et al. \"Be confident! towards trustworthy graph neural networks via confidence calibration.\" Advances in Neural Information Processing Systems 34 (2021): 23768-23779.\n\n[3] Liang, Jian, Ran He, and Tieniu Tan. \"A comprehensive survey on test-time adaptation under distribution shifts.\" arXiv preprint arXiv:2303.15361 (2023).", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper features a clear analysis, allowing for a quick understanding of the problem the authors aim to solve. \n\n2. The highlighted presentation of the experimental section greatly aids reviewers in rapidly analyzing the experimental results.", "weaknesses": "1. The observations mentioned are not novel enough, as many studies have reported similar observations [1].\n\n2. The proposed method lacks flexibility and guarantees.\n\n3. The experimental results are insufficient, with not only a limited number of comparison methods but also incomplete results and a lack of comparison method results for different model architectures.\n\n4. Although the writing is clear, it uses many uncommon words in academic papers, causing difficulties in understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1-Q1: Novelty and relationship to over-under-confidence.** To clarify our novelties: (1) we provide theoretical tools to understand the pitfalls of Marginal Entropy Minimization; (2) the proposed baseline only requires manual tweaking of a single parameter, a single forward pass of the vision encoder, and no optimization.  \n  \nWe politely disagree with the statement “the motivation is not novel, as the phenomenon of over-under-confidence has been already revealed”. Here are the reasons:  \n1. We know that over-under-confidence are widely known phenomena in the fields of uncertainty estimation and model calibration. Hence, we do not claim their discovery in any passage of the manuscript.  \n2. Our core motivation is not over-under-confidence, but: (1) demonstrating that the $\\arg\\max$ of the marginal probability distribution is largely invariant to MEM (Sec 2.2), and (2) establishing that the marginal probability distribution can be regarded as a lower bound for the error of the standard inference protocol (Sec 2.3). *These findings are not readily available in the existing literature; they are key contributions and novelties of our work.* As also acknowledged by `Wz3n`, we believe these are “great motivations for bringing insights into Test-Time Augmentations”. \n\n**Q2: Setting the temperature to 0 is \"straightforward and naive\".**\nWe agree that Zero is indeed simple, but we see this as a strength rather than a weakness, as pointed out by `Wz3n`, `7eTE`, and `a5fT`.  We firmly believe that simplicity should be rewarded, since it allows more practitioners to use our work.\n\n**Q6: Why is the temperature set to 0?**\nFig. 1(b) suggests that incorrectly classified views tend to suffer from overconfidence. When marginalizing, a highly confident but wrong prediction may greatly influence the resulting marginal distribution, leading to an overall incorrect prediction. Adapting the temperature addresses this by removing the dependency on confidence. Please, see the response to `a5fT` (Q3) for further clarifications.\n\n**cont. - learnable temperature.**\nWe are aware of works treating the temperature as a learnable parameter. Other than the provided reference, this is also done in [7], a pioneer of modern research on calibration, as well as in CLIP. Our goal, however, largely differs from learning parameters; in contrast, we aim to show that a surprisingly strong TTA baseline emerges with no optimization at all.\n\n**W2: Method lacks flexibility and guarantees.**\nOn flexibility: our method is fairly simple and can be easily integrated into a variety of VLMs (as also `Wz3n` points out). Its only prerequisite is a softmax-based classification pipeline. \nOn guarantees: as also acknowledged by `7eTE`, our work features a theoretical and empirical analysis (Sec. 2.2 to 3.1) that tries to justify its design. This is missing even in some influential work in this field, e.g., [52, 37, 32]. \nWe are happy to expand our answer in case the reviewer points to specific flexibility/guarantees issues we did not cover.\n \n**Q3: Unusual pseudocode.**\nWe provided the pseudocode in a code-like fashion following some influential examples from the literature, including CLIP (see [a-h]). While traditional pseudocode may hide how functions are implemented, code-like snippets provide an intuitive way to grasp how simple and easy to adopt the proposed methodology is.\n\n**Q4: Too few comparisons, many TTA methods exist [3]. Do they have the same problem?**\nWe strove to compare with the state-of-the-art among MEM-based methods (TPT, PromptAlign) as well as the latest state-of-the-art on TTA for VLMs “Reinforcement Learning from CLIP Feedback” [53], published at ICLR 2024. This year ICLR ended 4 days before the abstract submission deadline for NeurIPS. We believe this is a good effort to keep up with the pace of AI research. \nOur work also features an analysis of Marginal Entropy Minimization, hence, to answer the question, our theoretical insights extend to strategies relying on this objective, including those listed in the provided survey paper. Other than TPT and PromptAlign, some examples are [18,26,41,52]. \n\nWe are happy to enrich our manuscript in case the reviewer points to specific comparisons that may be relevant to this work.\n\n**Q5: Missing results.**\nOur focus is on VLMs, hence we followed the established experimental protocol of the field of TTA with VLMs by evaluating different CLIP variants (ViT-B-16 and MaPLe), following [37,32]. Additionally, we experimented with a newer combination of CLIP-ViT-B-16 and CLIP-ViT-L-14, for which [37,32] did not present results. We did not omit any results and believe that our evaluation is fair.\n\n**Q7: Label smoothing.**\nIn our context, applying label smoothing as in [i] would not bring benefits since it would not affect the $\\arg \\max$ of the marginal probability distribution. Thus, it would incur the same issues highlighted in the example of the response to `a5fT` (Q3). \n \n$\\bar{p}^{LS} = 1/N \\sum p_i^{LS} = 1/N \\sum \\left[(1-\\alpha) * p_i  + \\alpha/C\\right] = (1-\\alpha) * 1/N \\sum p_i + \\alpha/C$  \n$\\bar{p}^{LS} = (1-\\alpha) * \\bar{p} + \\alpha/C \\implies \\arg \\max \\bar{p}^{LS} = \\arg \\max \\bar{p}$\n\n**cont. SAM.**\nA core contribution of our work is to show that Marginal Entropy, a very popular objective for TTA, has some pitfalls that can be circumvented with a simple and optimization-free approach. While we agree that sharpness-aware strategies can synergize with TTA, proposing an optimization-based alternative is out of the scope of our work. \n\n**Q8: Unusual terms like \"prevalent class\".**\nWe thank the reviewer for pinpointing this potential misunderstanding! To clarify, we will replace “prevalent class” with “most probable class”. This refers to the class with the highest probability within a distribution. We notice the reviewer mentioned \"many uncommon words.\" Could the reviewer provide examples? We are willing to clarify and update the manuscript accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General Comment**
We sincerely thank all reviewers for the time and effort devoted to reviewing our manuscript. 

Above all, we profoundly appreciate that the simplicity of the proposed baseline has been praised almost unanimously among reviewers (`Wz3n`, `a5fT`, `7eTE`).  

On theoretical results, we are glad that our “clear” theoretical analysis (`Wx8e`) has been pointed out as a strength that “discovers a critical flaw in a popular class of TTA methods” (`7eTE`) while providing two “interesting insights”, supported by “helpful empirical results” (`a5fT`), which constitute “great motivations for bringing insights into Test-Time Augmentations” (`Wz3n`).

On experimental results, we are happy that our strategy was described as a “simple but very effective baseline that outperforms state-of-the-art methods” (`7eTE`), all with a “memory cost impressively small” (`Wz3n`). We are also glad to read that the presentation was appreciated: “the highlighted presentation of the experimental section greatly aids reviewers in rapidly analyzing the experimental results” (`Wx8e`), “several baselines are included and discussed, which helps understand the effect of ZERO” (`a5fT`).

Finally, we appreciate that our efforts in highlighting the limitations of our work were acknowledged (`a5fT`, `7eTE`).

### **Rebuttal Content**
Concerning doubts and weaknesses, we provide detailed responses to each reviewer. Summarizing:  
1. We follow the suggestion of reviewer `a5fT` and conduct additional experiments with LAION-pretrained CLIP models about overconfidence, complementing Section 3.1 of our submission. These results align with the initial observations of the manuscript; 
 
2. Although not requested, we support our answers with additional experimental verification if applicable. This applies to (1) the analysis of invariance to entropy minimization (`a5fT`) and (2) motivating why Fine-grained datasets incur less improvement than Natural Distribution Shifts datasets (`Wz3n`);  

3. We propose explicit ad-hoc re-phrasings to the manuscript whenever possible. We hope this is helpful to clarify any misunderstandings;

When applicable, figures portraying the outcome of additional experiments are provided in the PDF attachment. We hope that our responses are comprehensive, clear, and satisfactory to the reviewers. We look forward to engaging in a fruitful discussion otherwise.

### **Response Formatting**  
All responses are organized into questions and weaknesses. For example, **Q1** refers to the first question, while **W1** to the first weakness. 

Responses may contain references. When these are numbered (e.g., [32]) they refer to the references of the manuscript. When these are lettered (e.g., [a]) they refer to the list below.  

**References for the rebuttal**\
[a] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." ICML 2021.\
[b] Caron, Mathilde, et al. "Unsupervised learning of visual features by contrasting cluster assignments." NeurIPS 2020.\
[c] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." ICCV 2021.\
[d] Zhang, Hongyi, et al. "mixup: Beyond Empirical Risk Minimization." ICLR 2018.\
[e] Cubuk, Ekin D., et al. "Randaugment: Practical automated data augmentation with a reduced search space." CVPR-W 2020.\
[f] Yu, Jiahui, et al. "CoCa: Contrastive Captioners are Image-Text Foundation Models." TMLR.\
[g] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." ICLR 2024.\
[h] Tolstikhin, Ilya O., et al. "Mlp-mixer: An all-mlp architecture for vision." NeurIPS 2021.\
[i] Szegedy, Christian, et al. "Rethinking the inception architecture for computer vision." CVPR 2016.\
[j] Cherti, Mehdi, et al. "Reproducible scaling laws for contrastive language-image learning." CVPR 2023.

Meta-review: I would like to thank the reviewers for spending a lot of time reading the paper carefully and providing valuable suggestions. I also thank the authors for providing the detailed responses to the reviewers’ questions and addressed many concerns from the reviewers. After the rebuttal and discussion, there are still disagreement among the reviewers. The AC read through the manuscript, all reviews, the discussion, and the rebuttal. One reviewer is concerned about the performance report on EuroSAT and another reviewer is concerned about the novelty of the paper. __Thanks so much again for the time that you spent on reading and discussion of the paper!__ The AC agrees that the method is simple yet provide outstanding performance with careful analysis. The AC believe we should promote simple-and-effective methods while avoid making methods unnecessarily complex. Also, the AC agrees that the authors should report the results and analysis of performance on EuroSAT in the main paper. The authors are highly encouraged to improve the paper quality according to the reviewers' feedback in the camera-ready version. The AC decided to accept this submission.
Decision: Accept (poster)