CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
Authors: Marc Lafon
Gustavo Adolfo Vargas Hakim
Clément Rambour
Christian Desrosiers
Nicolas THOME
Venue: NeurIPS 2025 poster ()
Keywords: Test-Time Adaptation
Vision-Language Models
Unsupervised Image Classification
Online Adaptation
Primary area: general_machine_learning
TL;DR: We propose CLIPTTA, a contrastive test-time adaptation method for CLIP that improves both accuracy and OOD detection in closed- and open-set settings.

Abstract: Vision-language models (VLMs) like CLIP exhibit strong zero-shot capabilities but often fail to generalize under distribution shifts. Test-time adaptation (TTA) allows models to update at inference time without labeled data, typically via entropy minimization. However, this objective is fundamentally misaligned with the contrastive image-text training of VLMs, limiting adaptation performance and introducing failure modes such as pseudo-label drift and class collapse. We propose CLIPTTA, a new gradient-based TTA method for vision-language models that leverages a soft contrastive loss aligned with CLIP’s pre-training objective. We provide a theoretical analysis of CLIPTTA’s gradients, showing how its batch-aware design mitigates the risk of collapse. We further extend CLIPTTA to the open-set setting, where both in-distribution (ID) and out-of-distribution (OOD) samples are encountered, using an Outlier Contrastive Exposure (OCE) loss to improve OOD detection. Evaluated on 75 datasets spanning diverse distribution shifts, CLIPTTA consistently outperforms entropy-based objectives and is highly competitive with state-of-the-art TTA methods, outperforming them on a large number of datasets and exhibiting more stable performance across diverse shifts.

CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
Marc Lafon 1.1
Gustavo A. Vargas Hakim .3
Clément Rambour
Christian Desrosier
Nicolas Thome
Conservatoire National des Arts et Métiers, CEDRIC, F-75141 Paris, France 2Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
3TTS Montreal, Canada
Institut universitaire de France (IUF)
Abstract
Vision-language models (VLMs) like CLIP exhibit strong zero-shot capabilities but often fail to generalize under distribution shifts. Test-time adaptation (TTA) allows models to update at inference time without labeled data, typically via entropy minimization. However, this objective is fundamentally misaligned with the contrastive image-text training of VLMs, limiting adaptation performance and introducing failure modes such pseudo-label drift and class collapse. We propose CLIPTTA, new gradient-based TTA method for vision-language models that leverages soft contrastive loss aligned with CLIP's pre-training objective. We provide a theoretical analysis of CLIPTTA 's gradients, showing how its batch- aware design mitigates the risk of collapse. We further extend CLIPTTA the open-set setting, where both in-distribution (ID) and out-of-distrion disstiibuttion (OOD) samples are encountered, using an Outlier Contrastive Exposure (OCE) loss improve OOD detection. Evaluated on 75 datasets spanning diverse distribution shifts, CLIPTTA consistently outperforms entropy-based objectives and is highly competitive with state-of-the--rt TTA methods, outperforming them on large number of datasets and exhibiting more stable performance across diverse shifts. Source code is available CLIPTTA Repository.
Introduction
Vision-language models (VLMs), such CLIP [1] and ALIGN [2], are multimodal foundation models with strong zero-shot performance in downstream classification tasks. Yet, their ability to generalize to specialized domains, e.g., medical imaging or corrupted inputs, remains limited without adaptation, making this an active area of research.
Test-Time Adaptation (TTA) addresses the adaptation of pre-trained models to new downstream tasks during inference, without access ground-truth labels, typically by updating model parameters via gradient-based optimization [3, 4, 5, 6, 7]. This label-free adaptation is particularly valuable for deploying VLMs in real-world applications where annotation is scarce and costly, such as medical image processing [8]. numan--obo interaction [9]. and federated learning [10].
Entropy minimization is the most common TTA objective [11, 12, 13, 14], it mirrors the cross- entropy training of standard classifiers. However, it fundamentally misaligned with the contrastive
Equal contribution
Corresponding author: laro.lafon @lccnam.net
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Aligned objectives
(a) Standard classifier pre-training
(b) Standard TTA
Model
Cross-entropy
Model
Entropy
dog
Unlabeled image
Objective Mismatch
(c) CLIP pre-training
(d) Our propused TTA approach for CLIP
Psendo-captioning
CLIP
Image-text contrastive
CLIP
Image-text soft-comtrastive loss
"My puppy happy!"
Batch of unlabeled images
Aligned objectives
Figure Motivation for CLIPTTA. Standard test-time adaptation (TTA) methods often rely on entropy minimization (b), which aligns with the cross-entropy loss used in classifier training (a) but is misaligned with CLIP's contrastive pre-training (c), hindering adaptation due to incompatible gradient dynamics. CLIPTTA instead uses soft contrastive loss aligned with CLIP's objective, reinforeing alignment between images and their predicted pseudo-caption analysis shows that this contrastive, batch-awer sich--wwree formulation improves robustness pseudo-label drift and class collapse-tve two failure modes common to entropy-based TTA methods.
within the batch (d). Our gradient
image-text pre-training objective of VLMs like CLIP, illustrated in Fig. 1, potentially hindering adaptation due to differing gradient dynamics. Recent works have attempted to improve TTA of CLIP by leveraging visual-textual similarities in a transductive manner [6, 7]. yet the objective misalignment remains unresolved. Furthermore, when labeled data is available, recent work on fine-tuning [15] demonstrates that using the exact same loss function as during CLIP pre-training leads to better performance on downstream tasks.
This mismatch in objectives leads fundamental issues during adaptation: entropy-minimizzatin methods are prone to pseudo-label drifi, where the model reinforces its own mistakes. This can lead to class collapse, where predictions concentrate
set of classes regardless of the input [16, 13], severely hindering adaptation. Numerous efforts have been made to reduce the adverse impact pseudo-label misclassification [12, 13, 14, 3]. However, these methods make predictions for each sample independently, without accounting for other predictions in the batch, which limits their robustness. This becomes especially critical when the source model's accuracy low or when input batches contain out-of diistribitiinnnonnn (OOD) samples that belong to unknown classes [17, 12, 18].
Together, these observations raise central question:
How design an adaptation loss that is more suited for gradient-based test-time adaptation of contrastive vision-language models such as CLIP?
In this work, we introduce CLIPTTA, a new test-time adaptation method tailored to vision-language models. employs soft contrastive image-text loss that mirrors CLIP's pre-training objective, providing natural continuity in adaptation. As illustrated in Fig. 1, this design reflects our central assumption: adaptation losses should align with the model's multimodal contrastive training paradigm. Importantly, the contrastive nature of the CLIPTTA loss links predictions within batch, incorporating mechanisms to mitigate the risk of class collapse caused by noisy pseudo-labels. It also demonstrates increased robustness in open-set scenarios, where both in-distribution (ID) and out-of out-of-distribution (OOD) samples are present. We further augment with discriminative loss to separate from OOD samples, improving performance under open-set conditions.

---

Our contributions can be summarized as follows:
We introduce CLIPTTA, a new TTA method for CLIP based on a soft contrastive image-text loss aligned with its pre-training objective, offering a principled alternative entropy minimization. We provide a theoretical analysis of CLIPTTA's gradients, showing how its batch-aware design improves robustness to pseudo-label drift and class collapse-two key failure modes of standard gradient-based TTA methods.
We extend CLIPTTA to open-set adaptation with an Outlier Contrastive Exposure (OCE) loss, improving ID/OOD separation and robustness under distribution shift.
We conduct extensive benchmarking across 75 diverse datasets, spanning four types of distribution shifts: corruptions, domain shifts, coarse grained, and fine-grained classification. Empirical results show that our soft contrastive loss consistently outperforms entropy-based objectives for gradient- based TTA of vision-language models, establishing it as more effective alternative. In addition, CLIPTTA is highly competitive with state-of-the-at TTA methods, outperforming them on a large number of datasets and exhibiting more stable performance across diverse shifts. It also achieves notable gains in accuracy and OOD detection under open-set conditions.
2 Related work
Test-time adaptation (TTA) seeks to adapt model to new datasets on the fly in the absence of labels. This process is performed on independent data streams that showcase only a small portion of the full data distribution. Aiming to adapt deep classifiers to new domains, TENT [1I] proposed the widely exploited technique of entropy minimization. The entropy loss is chosen for its link with cross-entropy, with the intent of extending the model's training in unsupervised way. Building on this principle, several approaches have been proposed: filtering out unimportant samples based on an entropy criterion in ETA [12], and further filtering those with small gradients in SAR [13], minimizing the marginal distribution's 's entropy across image transformations in MEMO [19], meta-learning the TENT loss via conjugate pseudo-labels [20], storing the most confident samples in memory for a cleaner adaptation in RoTTA [14], or combining entropy minimization with a clustering loss constraint in TTC [21]. While these methods rely additional mechanisms such as filtering or confidence-bart based selection, CLIPTTA achieves robustness to pseudo-label drift and collapse by a simple modification of the adaptation objective. Contrastive learning approaches have also been explored, such as AdaContrast [22], where a student-teacher model is trained using pseudo-labels obtained from weak and strong image augmentations as in MoCo [23]. In contrast, our contrastive adaptation refers to visual-text interactions in the context of VLMs. To the best of our knowledge, this is the first attempt to explore this particular contrastive TTA formulation for VLMs.
TTA for VLMs. Several methods have been proposed to adapt VLMs new streams of unseen data. CLIPArTT [6] introduces a new loss function specifically tailored to VLMs, combining image- to-image and text-to-text similarities to generate pseudo-labels and utilizing small subset of probable classes to form new image-wise text prompts. WATT [7] extends this idea with prompt ensembling and weight averaging. While CLIPArTT's loss better leverages CLIP's s multimodal structure than entropy minimization, it remains heuristically driven and loosely aligned with CLIP's contrastive training objective. Complementary to these, other methods explore alternative adaptation paradigms. TPT [3] performs adaptation through prompt tuning [24]: rather than updating the model's internal weights, optimizes a small set of text prompts using entropy minimization. Although it uses pradient-based adaptation, this approach is fundamentally distinct from traditional TTA methods that typically update normalization parameters, and it comes with a high computational cost due to its reliance on multiple augmentations per image. TDA [4] adopts an even more distinct approach: it operates in gradient-free manner by building positive and negative caches of past predictions, which are then used as pseudo-labels to simulate few-shot episodes as in [25]. While TDA achieves strong results on Imagenet variants, we found it to perform poorly under other types of distribution shifts, such corruptions. In contrast, approach, CLIPTTA, requires only simple modification of the loss function and delivers robust performance across all TTA benchmarks.
Open-set TTA is a more challenging branch of TTA, where batches are polluted with out-of- distribution (OOD) samples that belong unknown classes. Open-set TTA methods aim at detecting
3

---

plane
car
2
LOCE Eq.
pseudo maptins
duy
phoco of <class>
Text Encoder
N
sumples
2
OOD /DD Detection
Pseudo- captioning
CELIFTTA Eq.(9)
2
Vision Encoder
[ 2(
Z
Image--cn Gest similoitty matrix
Bu
B
data streom
Figure Illustration of CLIPTTA CLIPTTA in Sec. 3.1 consists of soft contrastive loss specifically designed for TTA of VLMs like CLIP. We show in Sec. 3.2 that CLIPTTA is robust to class collapse and pseudo-label errors. Finally, we add an OCE loss to be robust to OOD samples in batches in Sec. 3.3 and improve the ID/OÓD detection and accuracy in open-set scenarios.
OOD samples from in-distribution (ID) samples, and improve the model's accuracy on ID images. OSTTA [17] uses an entropy heuristic based on a student-teacher model to disregard OOD samples and apply entropy minimization on the ID ones. SoTTA [26] uses the maximum predicted probability to filter and store the most confident samples in memory, and applies TENT on them. On the contrary.
STAMP [27] filters samples and their augmentations via entropy, also preserve them memory for entropy minimization. UniEnt [28] addresses the problem more explicitly by modeling the samples' outlier score as a mixture of two Gaussian distributions, later using entropy minimization on the II) samples and entropy maximization on the OOD samples. As in the closed-set set scenario, these methods do not transfer optimally VLMMs, since entropy does not connect well with CLIP's pre-training loss. Our adaptation loss aligns with CLIP's pre-training, and we propose discriminative OOD loss that directly aligns with ID/OOD separations metrics.
CLIPTTA
We introduce CLIPTTA, contrastive test-time adaptation method tailored VLMs such as CLIP, as illustrated in Fig. 2. By aligning the adaptation objective with CLIP's image-text contrastive pre-training described Sec. 3.1, CLIPTTA improves robustness to pseudo-label errors and class collapse through its batch-aware formulation, as demonstrated by gradient analysis in Sec. 3.2. Combined with the Outlier Contrastive Exposure loss introduced Sec. 3.3, improves both OOD detection and accuracy for robust adaptation in open-set scenarios.
3.1 Contrastive adaptation loss at test-time
Let us denote CLIP's visual encoder as fö. (-) and its textual encoder as (+), with model parameters
0 (O.., 0z). Given an image and textual prompt t, the normalized visual and text features are Ze = für (x) and = fú, (t). To classify an image in downstream task, we construct class- specifc captions of the form t = "A photo of class > for each class c, and compute the probability of classifying image as class
exp(z). 2f/T)
( (x:)
exp(xt 2 /TT
where r is a fixed temperature parameter.
Since ground truth captions are unavailable at test-time, we generate pseudo-captions for batch of N samples { by associating each image æ; to the caption of its predicted class = ta, where arg max, ((t)) We denote 2, the representation of f. Given two pseudo- labeled image-text pairs (æq, f.) and (I), E2), we define p(tylæz) and p(æ, (&)) as the probabilities that x, matches &, and that f, matches æy, respectively:
exp(2 T2/ TT) p((y(ox)) Σ exp(=) 2
exp(2! (2:/TT
and p(x;)(:)
(2)
ΣN exp(z!) 2, 2//7T)

---

Although Eq. (1) and Eq. (2) appear similar, they differ in their softmax normalization: Eq. (1) normalizes over C' classes, while Eq. (2) normalizes over the N predicted classes in the batch.
natural strategy for adapting CLIP at test time is to reuse its contrastive loss on pseudo-labeled image-text pairs (æ,, f.). However, this assumes pseudo-labels are correct and ignores uncertainty in the predictions. Instead, we retain alignment with CLIP's training objective while relaxing reliance on hard pseudo-labels. To this end, we introduce a soft contrastive loss that leverages the full distribution over pseudo-captions:
N
N
N
Li-coom
pl logp(ts
plar log ((gp(1)))
j=1
imagge->ext
text lexi-iinage
This loss retains CLIP's contrastive structure while explicitly modeling uncertainty in pseudo-labels. As shown in Fig. 2, the first term computes the entropy over the image-to-ted probability distribution (row-wise), and the second term the entropy over the text-to-image probability distribution (column- wise) within the batch. Analogous entropy minimization, which replaces hard cross-entropy with a soft and uncertainty-aware loss, our soft contrastive loss is a principled extension of the VLMs' contrastive scheme. Furthermore, it demonstrates enhanced robustness pseudo-label errors, as studied in Sec. 3.2. To ensure fair comparisons, we use only the image-to-text term of Eq. (3) in the main experiments. as most gradient-bass TTA methods update only the visual encoder. The effect of simultancously updating the text encoder is evaluated in Appendix C.
Final training objective. Following prior TTA research [29, 17, 27, 14], we also incorporate stan- dard techniques such as entropy regularization and a class-wise confident memory (CCM) to enhance adaptation. The regularization loss, based on negative marginal entropy, diversifies the predictions by uniformizing the prediction distribution across classes. Defining q. E (2;) as the batch-wise average probability for class e (i.e., over probabilities in Eq. (1)), the regularization loss is feg (0) = E Ge log Ge- The final CLIPTTA loss integrates the soft-contrastive loss Eq. (3), the regularization term, and the CCM memory. Memory batches MM, equal in size test batches, are used to compute the adaptation loss:
LCUPTTA (0) - Cucom(()) (0) + L (0)] + Areg (0),
[
where LM (0) is the soft-contrastive loss computed on the memory batch, and Areg controls the regularizer's strength. By averaging the loss over current and memory batches, the method effectively leverages confident past predictions improve adaptation while reducing sensitivity to noisy data.
3.2 Gradient Analysis
We analyze the gradient of the soft contrastive loss Ly to understand how it enables robust test-time adaptation, particularly in the presence of pseudo-label errors and class imbalance. The key insight is that, unlike entropy-based losses, La-cont batch-aware, allowing the model to dynamically correct prediction errors and reducing the risk of class collapse.
Proposition 3.1 (Gradient of Soft-Contrastive Loss), Let N₂ be the number of samples in the batch pseudo-labeled as class k and qik = q(tx) (x;) as in Eg. (1), The gradient of Lv-comt W.E.t. z is:
V2 Le
Σ
2,'
Whe =11,
(5)
1=1
k=1
NR qik
where Big = ((fz)o0z) + logp(t [2:))
and Wk.i
Σ
No qic
Proof. See Appendix A.I.
5

---

This expression shows that the gradient for sample a; aggregates contributions from all pseudo- captions the batch, each weighted by P.g. Each contribution consists of two effects. The first term, - acts as an attractive force pulling z toward pseudo-caption Ł In contrast, the second term, >x : introduces a repulsive force that pushes the embedding away from dominant class directions since classes that more frequently predicted exert stronger repulsion.
Importantly, the coefficients Sr, are key to allowing the gradient of a sample to point toward a class different from its pscudo-label, enabling error correction by leveraging predictions from other samples in the batch, as illustrated on a toy dataset in Appendix B. They amplify the contribution of confident and semantically similar pairs in the gradient update, allowing the model to rely more reliable examples. For instance, if æ; is misclassified as class k'' but is close to another sample z; whose pseudo-caption reflects the correct class k, a large 3rg steers the update toward z Such correction is not achievable by sample-wise objectives like TENT, which systematically reinforce the predicted class regardless its correctness.
Proposition 3.2 (Gradient Vanishing under Class Imbalance). As one class dominates the batch (N₂ + N), the gradient of Ls-cont vanishes:
|| L..courl
0.
N
Proof. See Appendix A.I.
To further illustrate, consider a binary classification setting with classes and b, where is the most predicted class the batch (i.e., No > No). In that case, the gradient in Eq. (5) becomes:
Na Natta + Noqo
[3 - Bebbbaa
(z, 21)
The magnitude of this gradient depends on batch composition. As class imbalance grows, the becomes smaller, reducing the overall gradient magnitude.
coefficient N. Ng
Nomu Netu
This self-regulation property acts as a built-in dampening mechanism that slows adaptation before collapse occurs, helping prevent convergence dominant classes, preserving stable updates, and giving the model a chance to recover from poor pseudo-labeling. In contrast, entropy-based objectives such as TENT continue to reinforce dominant class predictions even as imbalance increases, accelerating collapse rather than preventing it (see derivation in Appendix A.
3.3 Outlier Contrastive Exposure loss
In this section, we extend CLIPTTA to the open-set setting, where the model is exposed to batches composed of images from both known classes (ID samples) and unknown classes (OOD samples) during adaptation. Our primary objective is to design an effective ID/OOD filtering mechanism focus adaptation
ID samples only. For that purpose, we the MCM [30] score, defined as = max, qtt xz.), which is the most popular OOD scoring function in the context of OOD detection for VLMs. For input image æ,, we further define the outlierness filtering weight:
W2 sigmoid (s - -),,
(8)
where a is an adaptive and learnable threshold. Using these weights, an image æ; will be considered reliable if w; > 0.5 and will be regarded as OOD otherwise.
While effective OOD filtering helps to improve TTA performance an open-set setting, we argue that we can leverage filtered-out OOD samples to improve the ID/OOD detection performance during adaptation. this end, we introduce the Outlier Contrastive Exposure (OCE) loss that aims at improving the OOD score separation between ID and OOD samples;

---

N Zim1
wist
Wi w;
LOCE
N Ziml We
food
In the open-set scenario, our optimization objective then becomes ming,oa LCLPPTTA + CocE where we update the parameters of the model ( and the ID / OOD threshold parameter in an end-to-end fashion. Our OCE loss differs from the UniEnt loss [28] since it purely discriminative, enforcing more direct separation between ID and OOD features, and since it learns the separation threshold a.
4 Experiments
CLIPTTA evaluated on four families of adaptation benchmarks: corruptions (CIFAR- 10/100-C, Imagenet-C) with 15 perturbations, domain shifts (VisDA-C, PACS, OfficeHome, Imagenet- Domains), semantic datasets, including coarse- (CIFAR-10/100) and fine-grained classification (Imagenet, and 10 datasets from the CLIP zero-shot suite). In total, this represents thorough evaluation over 75 datasets. A detailed description provided in Appendix C.2. open-set TTA, SVHN and Places-365 serve OOD counterparts for CIFAR- R and Imagenet, respectively.
Datasets.
Metrics. We report classification accuracy as the primary performance metric. In the open-set setting, we additionally report the area under the ROC curve (AUC) and the false positive rate at 95% ID true positive rate (FPR95) as OOD detection metrics.
Implementation details. We use ViT-B/16 as CLIP's backbone in all experiments. Adaptation performed with batches of 128 images using the Adam optimizer and learning rate of 10-1 over 10 iterations. Experiments are conducted in non-episodic manner, i.e., without restoring the model's parameters after each batch. Following the standard TTA protocol, we adapt the affine parameters of the visual encoder's normalization layers. In the open-set setting, add 128 OOD images per batch, as done prior work [28, 27J. The regularization and OCE losses' weights are set to Aveg = and Xooe = 1, respectively. We validate that CLIPTTA is stable variations of its hayper-parameter in Appendix C. Experiments were performed on two NVIDIA V100 32GB GPUs.
4.1 Main results
What is the best loss function for TTA CLIP? We compare CLIPTTA with the two prevailing
families of test-time objectives: (i) TENT-style losses, including TENT [11], ETA [12], SAR [13] and RoTTA [14], and (ii) CLIPArTT-derive losses, such as CLIPArTT [6] and WATT [7]. Table
Corruptions
Domain shifts
C-10CC
C-100-C
Inagenet C
Average
VisDA-C
PACS
Officelomme
Imagenet
Average
CLIP [1]
60.2
35.2
25.5
40.3
87.1
96.1
82.5
59.4
81.3
TENT [11]
56.4
31.4
17.6
35.1
89.3
96.6
83.4
60.2
82.3
ETA [12]
61.3
38.9
26.8
42.3
88.3
96.7
84.1
59.9
82.3
SAR [13]
67.8
43.2
33.6
48.2
87.8
96.2
83.8
60.6
82.1
RoTTA [14]
58.0
33.6
24.6
38.7
83.7
95.8
82.5
61.6
80.9
CLIPArTT
68.1
48.0
33.3
49.8
84.1
96.3
82.0
60.7
80.8
WATT [7]
66.0
38.5
43.5
87.7
96.2
83.4
61.8
82.1
CLIPTTA (ours)
80.7
52.6
41.1
58.1
89.6
97.5
84.2
63.4
83.7
Table I:
Comparison with gradient-based TTA methods. CLIPTTA outperforms entropy mini- mization methods [11, 12, 13, 14] and CLIP-specific TTA methods based on CLIPArTT's loss [6, 7] on all corruptions and domain shift datasets.
7

---

CLIP
CLIP
Accur 1%0
80
79
78
T6NT
ETA
RoTTA
CUPMIT
WXT7
CLIPITA
TEWT
ETA
Ske
Rotta
CURAITY
WAYT
(ours)
TDA
(a) Coarse- grained results
(b) Fine -rainnd results
Figure 3: TTA results on semantic datasets. Top-1 I aceuracy on coarse graiied datasets (CIFAR-1K 10 and CIFAR- 100) (a) and on 11 fine-grained datasets (including Imagenet) (b), Comparison with
gradient-based TTA methods [11, 12, 13, 14, 6, 7] and alternative state-of-the-art TTA methods [3, 4].
reports top-1 accuracy under synthetic corruptions and real domain shifts. We highlight two key findings. First, CLIPTTA substantially improves over zero-shot CLIP, with large gains when initial accuracy is low: +20.5
CIFAR-10-C, + +17.4 pts on CIFAR-100-C, and +15.6 pts on Imagenet-C. In contrast, other methods perform poorly under the same conditions, which can be attributed the increased likelihood of class collapse and pseudo-label drift when initial accuracy is low. Further analysis and finer-grained experimental evidence are presented in Appendix A. and Appendix B, respectively. Second, CLIPTTA is the only method that consistently achieves top performance across all benchmarks. While competing methods demonstrate strengths in specific scenarios, they fall short overall. For example, ETA performs best among the TENT-style methods on domain-shil datasets but still lags by 1.4 pts on average. Similarly. CLIPAITT is most competitive on corruption benchmarks but remains 8.3 pts behind. This trend persists across both coarse- and fine- aneegrained datasets (Fig. 3, with extended results in Appendix C.3). Altogether, these results establish our soft contrastive loss the most reliable and broadly effective objective for gradient-based TTA of CLIP.
How does CLIPTTA perform against other CLIP-based TTA methods? We further benchmark CLIPTTA against two recent state-of-the-ee the-art TTA methods tailored to CLIP: TPT [3], which adapts through text prompt tuning instead of updating normalization parameters, and TDA [4], gradient- free approach that adjusts CLIP's logits using cached predictions. As shown in Table 2, CLIPTTA improves top-1 accuracy by an average +19.4 pts over TPT and +15.6 pts over TDA. achieves state-of-the-art results on nearly all domain-shift benchmarks, with the sole exception of the ImageNet- suite, where TDA benefits from per-dataset hyperparameter tuning. However, we note that TDA lags far behind corruption datasets, standard benchmark in TTA, with average gap of over 15 pts compared to CLIPTTA Figure further confirms CLIPTTA's advantage across both coarse- and fine- vee-grained recognition tasks. While TPT and TDA designed for single-image batches, analysis in Appendix C.3 shows that CLIPTTA remains competitive even in this challenging setting, thanks to its use of the CCM memory and maintains stable performance across wide range of batch sizes.
Corruptions
Domain shifts
C-10-C
C-100-C
Imagenet-C
Average
VisDA-C
PACS
OfficeHome
Imagenct-D
Average
CLIP [1]
60.2
35.2
25.5
40.3
87.1
96.1
82.5
59.4
81.3
TPT [3]
58.0
33.6
38.7
85.0
94.0
81.7
62.4
80.8
TDA [4]
63.4
37.4
26.8
42.5
86.6
96.1
83.0
65.0
82.8
CLIPTTA (ours)
80.7
52.6
41.1
58.1
89.6
97.5
84.2
63.4
83.7
Table 2:
Comparison with other CLIP-based TTA methods. CLIPTTA outperforms TPT and TDA on most corruptions and domain shifts datasets and is second best on Imagenet-D.

---

How does CLIPTTA perform in the presence of semantic OOD samples? Table 3 presents results on the open-set scenario on Imagenet, where OOD detection needs to be performed alongside classification. First, we note that all closed-set methods, except ours, perform noticeably worse than zero-shot CLIP OOD detection, highlighting CLIPTTA's strong robustness to OOD sample contam- ination during adaptation. Notably, TENT and CLIPArTT suffer severe performance degradation in both classification and OOD detection, likely due to outlier interference in their pseudo-labeling process. Second, when equipped with our OCE loss, CLIPTTA consistently outperforms specialized open-set TTA methods, which use heuristic OOD detection mechanisms, achieving +2.3 points AUC over UniEnt and +8.4 points AUC over SoTTA. Our soft contrastive objective reliably preserves and improves both accuracy and OOD detection. unlike these intropy-based methods, which tend to degrade CLIP's initial performance. Additional results on other datasets are reported Appendix C.3.
ACC1
AUCT
FPR950
CLIP [1]
66.7
90.1
43.8
TENT [11]
12.4
49.9
89.4
ETA [12]
67.1
89.6
46.1
SAR [13]
58,8
62.0
75.7
CLIPArTT [6]
31.2
61.1
87.5
WATT
67.1
87.4
53.4
TDA [4]
66.8
82.1
59.8
CLIPTTA (OIr))
67.6
93.5
25.7
OSTTA [17] i
66.9
84.9
59.2
SoTTA[26]
66.7
89.3
47.1
STAMP [27]
29.7
63.0
80.2
UniEnt [281+
65.2
95.4
17.1
CLIPTTA
(ours)
67.6
97.7
9.7
Table Open-set TTA results with Imagenet as ID dataset and Places as OOD dataset. f denotes open-set TTA methods.
4.2 Model analysis
Ablation study. Table presents an ablation of CLIPTTA 's components on four closed- benchmarks. The first key observation that the soft contrastive loss (o.) alone accounts for the vast majority of the over- all performance gains, demonstrating the im- portance of aligning the adaptation objective with CLIP's pre-training. In low-accuracy set- tings, com significantly outperforms TENT, achieving gains of +19.4 points on CIFAR- 100-C and +22.7 points on ImageNet-C, where TENT even degrades CLIP's performance. This confirms the vulnerability of entropy-basee methods to pseudo-label drift and class collapse and supports the enhanced robustness of Ls-cont, as theoretically analyzed in Sec. 3.2. On average, a. improves loss (freg) further improves overall results by +0.4 pts, while incorporating the confident memory (M) brings additional +0.8 pts gains.
C-100
C-100-C
IN
IN-C
FG
Avg.
CLIP
68.1
35.2
66.7
25.5
64.8
52.1
TENT
72.9
31.4
66.5
17.6
65.1
50.7
C.comm
74.2
50.8
68.8
40.3
68.3
60.5
Ly-com Lus
74.9
52.4
69.1
38.6
69.8
60.9
Cy-com Lus M
75.3
52.6
69.6
41.1
69.9
61.7
Table 4: Ablation analysis. Accuracy in the closed- set setting on CIFAR-100, CIFAR- Imagenet, Imagenet-C, and 11 fine mene-graindd datasets (FG).
TENT +9.8 pts. Adding the regularization
On CLIPTTA'S robustness. Figure 4 provides empirical insights supporting the gradient analysis in Sec. 3.2, by illustrating CLIPTTA's stability and robustness over batches CIFAR-10-C. CLIPTTA the only method that steadily improves accuracy throughout adaptation while all competing objectives plateau degrade (Fig.4a). This stability is closely linked to CLIPTTA's ability to maintain prediction diversity. As shown in Fig,4b, prediction entropy remains nearly constant for CLIPTTA, whereas TENT exhibits a sharp drop in entropy, indicating collapse toward small subset of classes. This behavior results in harmful label drift. Fig. 4c tracks the deterioration ratio, defined as the fraction of initially correct predictions that become incorrect during adaptation. TENT reaches over 30% deterioration, compared to less than 7% with CLIPTTA These findings confirm the significant stabilizing effect of our batch-awe aware contrastive loss during adaptation.

---

80
242
0.30
CLPTTA
TENT
Top-1 Accursyy
75
Dicerlerriio
0.25
0.20
1.4
60
TEMT
CUPKIT
TENT
0.05
ITA
WATT
SAR
CLITTA dourst
20
Batches
60
80
0 20
60
10
20
30
40
Batches
50
60
70
80
(a) Accuracy vs batches
(b) Class collapse
(c) Deterioration ratio
Figure 4: CLIPTTA accuracy and robustness on CIFAR-IO-C. (a) In the non-episodic setting,
CLIPTTA steadily improves top-1 accuracy across batches while competing methods degrade. (b)
CLIPTTA maintains high prediction entropy, preserving diversity predicted classes, whereas TENT shows marked entropy collapse. (c) The deterioration ratio, defined as the fraction of initially correct predictions that become incorrect, increases significantly for TENT but remains low for CLIPTTA
Conclusion
This work introduces CLIPTTA, showing that using simple soft contrastive loss be highly beneficial to adapt VLMs in pseudo-label TTA. By careful analysis of our loss and its gradient, show that method brings robustness to the class collapse and pseudo-label drift issues. We also introduce a contrastive outlier exposure loss to tackle the open-set TTA setting. Extensive experiments conducted wide range of benchmarks demonstrate that method significantly outperforms previous baselines on both closed-set and open-set adaptation. Ablation experiments and model analyses strengthen the foundations of our contribution. While CLIPTTA shows strong results, it also presents some limitations. First, it focuses on global image-tex, text interactions, which may limit adaptation fine-grained localized shifts. Second, our evaluation restricted to classification tasks, and extending the approach to broader multimodal settings remains an open direction. Finally, like most test-time adaptation methods, CLIPTTA assumes additional computation and time are available at inference, which may not always be practical in real-world deployments. Future work includes investigating more thoroughly the interplay between visual and textual adaptation during test-time optimization, aiming to better exploit cross-modal consistency. Another promising direction to design gradient-based adaptation strategies that operate efficiently in real-time environments, for instance within embodied or interactive agents.
Acknowledgment
This work supported by the French National Research Agency (Agence Nationale de la Recherche, ANR) under grants from project DIAMELEX (ANR-20-CE45-0026) and project RODEO (ANR-24-CE23-5888)) It granted access to the HPC resources of IDRIS under the allocation AD011013370R2 made by GENCI.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[2] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 1, 18-24 July 2021, Virtual Event, pages 4904-4916. PMLR, 2021.
[3] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision- lisounlaagguage models.
10

---

In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 14274-14289. Curran Associates, Inc., 2022.
[4] Adilbek Karmanov, Dayan Guan, Shijiam Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages #112-11177, 2024.
[5] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: zero-shot enhancement of clip with parameter-free attention. In Proceedings of the Thirtt-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI
AAAI Press, 2023.
[6] Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Light-weight adaptation of clip to new domains at test time, 2024.
[7] David Osowicchi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. Watt: Weight average test-time adaption of clip. arXiv:2406. 13875, 2024.
[8] Jeya Maria Jose Valanarasu, Pengfei Guo, Vibashan VS, and Vishal M. Patel. On-the-fly test-time adaptation for medical image segmentation. In Ipek Oguz, Jack Noble, Xiaoxiao Li, Martin Styner, Christian Baumgartner, Mirabela Rusu, Tobias Heinmann, Despina Kontos, Bennett Landman, and Benoit Dawant, editors, Medical Imaging with Deep Learning, volume 227 of Proceedings of Machine Learning Research, pages 586-598. PMLR, 10-12 Jul 2024.
[9] Qiongjie Cui, Huaijiang Sun, Jianfeng Lu, Bin Li, and Weiqing Li. Meta-auxiliary learning for adaptive human pose prediction. Proceedings of the AAAI Conference Artificial Intelligence, 37(5):6166-6174, Jun. 2023.
[10] Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. Adaptive test-time personalization for federated learning. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 77882-77911 Curran Associates, Inc., 2023.
[11] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[12] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16888-16905. PMLR, 17-23 Jul 2022.
[13] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
[14] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15922-15932, 2023.
[15] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[16] Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adaptation: survey. International Journal of Computer Vision pages 1-34, 2024.
11

---

[17] Jungsoo Lee, Debasmit Das, Jaegul Choo, and Sungha Choi. Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16380-16389, October 2023.
[18] Skyler Scto, Barry-John Theobald, Federico Danicli, Navdeep Jaitly, and Dan Busbridge. Realm: Robust entropy adaptive loss minimization for improved single-sample test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision ( (WACV), pages 2062-2071, January 2024.
[19] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 18629-38642 Curran Associates, Inc., 2022.
[20] Sachin Goyal, Mingjie Sun, Aditi Raghunanthan, and Zico Kolter. Test-time adaptation via conjugate pseudo-labekt Advances in Neural Information Processing Systems, 2022.
[21] Guoliang Lin. Hanjiang Lai, Yan Pan, and Jian Yin. Improving entropy-based test-time adaptation from clustering view. arXiv:2310.20327, 2023.
[22] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 295-305, 2022.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[24] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcieen prompt tuning. (eXiv:2104.0861,, 2021.
[25] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-fre free adaption of clip for few-shot classification. In European conference on computer vision, pages 493-510. Springer, 2022.
[26] Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
[27] Yongcan Yu, Lijun Sheng. Ran He, and Jian Liang. Stamp: Outlier-aware test-time adaptation with stable memory replay. arXiv:2407.15773. 2024.
[28) Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 23975-23984, June 2024.
[29] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, volume 13693 of Lecture Notes in Computer Science, pages 440-458. Springer, 2022.
[30] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out- of-distribution detection with vision-language representations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 35087-35102 3022 Curran Associates, Inc., 2022.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada, 2009.
12

---

[32] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248- 218-255,,20 2009.
[33] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[34] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXivv1306.5151, 2013.
[35] Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training ex- amples: An incremental bayesian approach tested 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 178-178, 2004.
[36] Jonathan Krause, Michael Stark. Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[37] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613 2014.
[38] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12()):2117-2226, 2019.
[39] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. 2008
[40] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[41] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[42] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database; Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3499 2010.
[43] K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv: urXiv:1212.0400) 2012.
[44] Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang. Judy Hoffmaan, and Kate Saenko. Visda: synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
[45] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[46] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conferrncce on computer vision, pages 5542-5550, 2017.
[47] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027 2017.
[48] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 2389- PMLR, 2019.
13

---

[49] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: A critical analysis of out-of disrriiittitttonn generalization. In Proceedings of the IEEF/CVF international conference on computer vision, pages 8340-8349, 2021.
[50] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[51] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[52] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[53] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-1464, 2018.
[54] Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. If your data distribution shifts, use self- learning. Transactions on Machine Learning Research, 2022.
[55] Sarthak Kumar Maharana, Baoming Zhang, Leonid Karlinsky, Rogerio Feris, and Yunhui Guo. Batelip: Bimodal online test-time adaptation for clip. In International Conference on Computer Vision (ICCV), 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims of the paper can be found in the paper's abstract, as well as in
Sec. 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Some elements of limitation are discussed in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Theoretical propositions in Sec. 3.2 are accompanied with their respective assumptions and proofs in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We included a detailed description of each setting, including datasets, baselines hyperparameters, and our own method's perpparaametess
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We provided an anonymizes repository that includes all the elements for reproducibility.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guides/
Sewiiddee/CooeSSmmissseeee ctlbbmmsssinooocciy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/pplteiie wwtpccccpppccccccccccccccccccccc guides CooeeSbmm ssi esionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all the necessary evaluation details in Sec. Appendix C, Appendix C.2, and Appendix C.I.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include statistical significance analysis in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information can be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http://///eee..eeeeeee cctppbccc//eeeecccccccccccceeeee
Answer: [Yes]
Justification:
The research conducted in this paper complies with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The topic of this paper does not imply any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This work does not include any element that requires safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All authors properly credited with the ownership of the different assets in this paper, and the terms of use are clearly stated in the submission.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide new library for Test-Time Adaptation of VLMs in Python/PyTorch, released in our source code.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Declaration is not required.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttpsr///.euu..eeeeeee Seeeurrpsscc/Coofeeeeeeee wcc.ooneeeeeccse/22222.. /LLM) for what should or should be described.
21

---

CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation Appendix
The appendix is organized as follows. In Appendix A, present detailed theoretical analysis of the gradients of the CCUPTTA loss. In Appendix B. we provide additional insights into how LCLrTA helps mitigate collapse and pseudo-labeling errors. Finally, in Appendix C, we elaborate on the experimental protocol and include additional experimental results.
A
Theoretical Analysis
A.1 Gradient Analysis
In this section, we provide detailed analysis of the gradients of the TENT loss LTNTT, CLIP's contrastive loss Leom. (i.e. using hard pseudo-captions), our soft contrastive loss Lo the regular- ization loss Lrgg and the CLIPTTA loss LeurTAA. Furthermore, we show how, benefiting from the information of other predictions in the batch, both contrastive losses allow to avoid collapse. Finally, when combined with the regularization loss, CLIPTTA allows mitigating the effect of pseudo label errors.
Let's consider batch of examples T1, NN and let's write Ng. the number of predictions assigned to the htth class. To simplify the computations, we place ourselves in the case where only the parameters of the visual encoder are updated.
Gradient of CTENT. We recall that the TENT loss writes as follows:
CINTT
Σ 4sk log 9sk
with qak the probability of image æ; being classified as class k (see Eq. (I)), The gradient of LTETt
w.r.t. 2;
V TENT
Σ
V z, Qik log qek = Σ (1 + log lsh Z
k=1
k=1
(1 + log qsk) qsk Σ
qte (z
x()
(10)
4sk
log
gio
qsu 2,
gic
k=1
C
From Eq. (10) can see that the gradient will always push z, in the direction of the predicted class i because in that case have log the 0, Vc / k. And there is mechanism allowing reduce the magnitude of the gradient towards the predicted class when we approaching situation of collapse.
Gradient of Ceont. Using the notation introduced the main paper, let represent the pseudo- caption associated with the example æ; in the batch, and let p(t, (z;) denote the probability of æ; matching f, within the batch. Specifically, we have:
E
p(t, (x.)
The unsymmetrized version of CLIP's contrastive loss writes:
N
N Σ
N
C + log Σ =TT
Leont.
(ogp(f.(m:)
T,a log Σ
Σ
-
1=1
t=1
k=1

---

where 2 is the embedding of the pseudo caption associated with image æy and 2 is the embedding of class k. Let's compute the gradient of Lonnt. w.r.t. 2j:
V 2; k== NeetTat Σe=1 Noet.2
NRe
V 21 fcomt -2
e== Noe2125 2
NREt
Ee oF
= +
E 0z 21
Ees1 Noe=122
2
k=
(11)
Nk qik
-1
Zt
No qic
k= mmml
21
Wkii Z6
with Wkii Denn No qic"
Ne Osk
From Eq. (11), observe that CLIP's contrastive loss consistently drives the visual embedding 2; toward the embedding of its predicted class & Wk, 1. However, the gradient's magnitude is influenced by the proportion of predictions assigned to the same class within the batch. Specifically, as the system approaches a collapse scenario (i.e., Wk. + 1), the gradient of feom. diminishes and eventually vanishes:
|| s4 Ccom.
(12)
Gradient of Lvccon
The unsymmetrized version of our Cycom loss writes:
N
Lecont
Σ
Σ p()s(az) logpll
Let's compute the gradient of Lvcomt W.r.t. z;:
N
(p((,(x:)) logp((, ax;)
j=1
Σ
p(t (z, v. (fgg(f,lxx) + logp(t/lz.)
Using the fact that Vp = Pv logp, we have:
N
V Ly.com =
Σ
p(f, V. logp(t, x.)) + log (x) log (og)(t)læ;)
==1
Σ [1 + log log lx. V.. logp(t,(ou)
=
Now we can use the fact that Vz,- log (og = -2 + Ee Wk,t 2/ based on the computation Loom in Eq. (11). Therefore, we have:

---

V Cs-om
Bepl- +
Σ Wkis 2,
(13)
with BG, - p(( (z;) + log p(( p(())==))
The gradient of L--ceet does not solely push the visual embedding toward the predicted class. Instead, it incorporates other predictions within the batch to guide the gradient direction, thereby mitigating the risk of pseudo-labcling errors. However, similar to CLIP's contrastive loss, the gradient diminishes as we approach a collapse scenario. In the case of collapse, where all examples in the batch are predicted belong to the same class e, the following conditions hold: (a, = 1 and wkk = 0, vk * c, and
2/² v. Consequently, the term [- + ΣE= k=1 Wk, z{] cancels out, leading to a null gradient.
Binary classification case. We derive Eq. (7) in the main paper, starting from Eq. (13), and assuming that the classification task comprises two classes K = {a,b], with N = N₂ + N₂ as the total batch size. To build on the intuition of the working mechanisms of our soft contrastive loss, we adopt the case where class a is dominant in the batch (i.e., N₂ >> Nı). First, we expand on the second sum term inside Eq. (13), as follows:
Na
Naga z
Naliazi + N
Wkit Wa + 1
+
Nagba N698
Na + Noqo
Noqaa + Noqbb
k=1
(14)
We notice that we can partition the main sum term in Eq. (13) into two sums that account for the N samples predicted as class a, and the N₂ samples predicted as class b:
N Σ 1=1
V
21 Lx-cont
Bigl- 2 + Q = NaBial-24 22 + Q1 + N1. Q]
Naqia
Nuqa
-NBiaal222
2t
*X1
Natia Noqo
Nagio + Noqu
Nagia
Nigo
+ No,Bol
Naqia + Nogio
Nagia + Noqib
=NaBial
Nogab
21 + Natia + Nodi z
Nuqa
(15)
N Nagia Noga
-Natia
Nayia x11
+ NN,Bal Nagia + Nogib
2 Nagia = Nogib
N.N₂ (2) 24) Bio Natha Nogs
N.
Biagit Na4sa + Nooib
(2) -1-)
Na (2) -2()
= [Bi B3,biial Naqua + Noqa
As pointed out, the increasing dominance of class (N₂ - 0) reduces the gradient to O, vanishing the negative effect of class collapse.
Gradient of Creg. The regularization loss Creg writes
Crg =
Σ To log go.
(16)
where Te correspond to the average predicted probability for class inside the batch.
3

---

Let's compute the gradient of Creg w.r.t. 24:
reg
Pi log Pk Σ
(1 + log
VxPP
k=1
Therefore, only need to compute V P.P
N
E *
C
P. V
qik
N
V2
4ik
N
V
2:
e #
N
lik Σ
Pis
z
1=1
j=1
2=1
Then we have:
s. Freg Σ
log qh 9it Σ
EgllE z()
k=1
log
Th )ak
Σ
lik Σ
+ log (,) 2
(17)
N
k=1
j7k
jzk
Kk
Ki log
qtk
4
k=1
From Eq. (17), we observe that the gradient is influenced by the ratios log driving it towards classes that are underrepresented conjunction with our soft contrastive loss creates powerful combined effect, enabling the effective relabeling misclassified examples, as discussed in Appendix B.
4," batch predictions. The use of the regularization loss in
Gradient of CCLPTIA We recall from the main paper that the final CLIPTTA loss combines both Lesont and Greg, thus benefiting both from an enhanced adaptation loss as well a mechanism to combat pseudo-labeling errors (we omit the effect of the memory for simplicity):
LeUPTIA L. + Areghreg:
(18)
Therefore the gradient of LCLPTIA writes:
N
C Whi 21 + Aven N
C
V z. COLPTIT
Biol-22 +
gh Rg log
1aas
z
(19)
k=1
k=1
Depending on the composition the hatch, we can see that LCUPTTA will strongly benefit from the contribution of the soft-contrastive loss provide accurate adaptation, will be able to correct misclassified examples due to the positive interaction of the combined corrective terms in V as L--onnt and V., Creg.
A.2 Analysis of OCE Loss
As discussed in the main paper, our outlier contrastive exposure (OCE) in Eq. (9) of the main paper is special case of the intra-class variance minimization:
DN N wi(s; 2
ΣN (1 wi)(s; Mood
0² Pid
N >m Wi
Pood
N Σ (1 -w;)
(20)

---

with Pia = E and Pood
N E (1 - w₂). This is further condensed into the loss function
Eq. (21):
a Piatfu Pood
(21)
Here, samples from the same distribution might tend to collapse into single point. An alternative formulation inter-class variance maximization, as shown in Eq. 22:
iner PiaPood (ood)
(22)
The impact of the proportions Pia and Pood is twofold. First, when neither ID OOD samples detected, the respective proportion nullifies, and the inter-class variance reaches its minimum. On the contrary, an equilibrium can be reached with both Pid Pood 0.5, which displays the implicit assumption of equally distributed scores between ID and OOD. We argue that this constraint limits the flexibility of the OOD detection at test time; as the nature incoming samples is unknown, allowing for non-uniform distribution in the detection can help filter out less useful samples. Secondly, the product of these probabilities would reduce the scale of the loss, especially compared to the other components of our CLIPTTA framework, which limits its impact on the adaptation of the model. Hence, a fully contrastive metric can attain the same detection objective by diminishing the latter negative effects:
(Jua Mand))
(23)
B Discussion on CLIPTTA's robustness
We further study the properties of CLIPTTA, to expand the insights on the working mechanisms that assist its success. Initially, the accuracy across batches (see Fig. in the main paper) serves as straightforward depiction of (a) the general preeminence of CLIPTTA over other methods, particularly entropy-baa -asse techniques, and (b) the collapse effect in methods such as TENT. To elaborate on the underlying advantages of our method, we examine the adaptation process more closely, first in controlled toy example, then using CIFAR-10-C across all of its corruptions.
Mitigating pseudo-label errors. In Fig. 5, present controlled toy ample demonstrating how CLIPTTA effectively mitigates misclassifica- tions. This example features a batch of six samples a three-class classi- fication problem. It focuses on the gradient orientations of the TENT, CLIPTTA, and regularized CLIPTTA losses for a single misclassified and ambiguous sample. The sample in question exhibits high probabilities for both the predicted and correct la- bels, indicating low confidence. The gradient of the CLIPTTA loss is di- rected toward the correct label, work- ing to minimize the difference be- tween the top two probabilitics-a be- havior further amplified by the reg- ularized CLIPTTA loss. In contrast, TENT prioritizes increasing the high- est probability, thereby reinforcing the incorrect prediction.
Cecaro
43-
0.2
614
Figure 5: Gradient Behavior: TENT vs. CLIPTTA Illus-
tration of gradient directions for TENT, CLIPTTA, and reg- ularized CLIPTTA losses misclassified sample (circled in red). While TENT (red arrows) reinforces the incorrect prediction to reduce entropy, CLIPTTA and its regularized version (green arrows) aim to minimize top-2 probability differences, guiding the correction.
provide quantitative insights on the CIFAR- 10-C dataset Fig. 6. Fig. 6-a, we observe the collapse of TENT, while CLIPTTA maintains robust performance. To further analyze this, quantify
5

---

TENT
PTT
18-
100
60
30
70077e6
40
Gasthe
8
25
Cashhe
05
(a) Accuracy
(b) Improvement ratio
(c) Deterioration ratio
Figure 6: Improvement & Deterioration ratios CIFAR-IO-C. (a) While TENT's accuracy collapses, CLIPTTA shows consistent improvement. (b) The improvement ratio quantifies the proportion of misclassified examples correctly relabeled after adaptation.
(c) The deterioration ratio captures the proportion correctly classified examples that become misclassified aost-adaptation.
two key metrics: the "improvement ratio" shown in Fig. 6-b, which represents the proportion of misclassified examples that are correctly classified after adaptation, and the "detcrioration ratio" shown in Fig. 6-c, which denotes the proportion of correctly classified examples that become misclassified after adaptation. CLIPTTA outperforms TENT by achieving a higher improvement ratio and lower deterioration ratio.
C Experimental details
We provide further information about the experimental setup that was conducted in the main paper. This includes the specifics of the experimental protocol, the baselines and benchmarks that were considered, as well as an extension of the empirical results.
C.1 Detailed experimental protocol
In our experiments. follow the widely explored non-episodic TTA setting [11, 17, 28], in which the model is adapted continually to batches of data, without recovering its original weights. This poses challenge, as adaptation risks of severely degrading the model, which can aggravate adaptation goes longer. As some of the considered baselines were originally conceived for an episodic setting (e.g. CLIPArTT [6]), some conditioning was applied in order to amplify their performance this scenario.
C.2 Details on baselines and datasets
Benchmarks.
We provide more detailed information about the datasets that compose the different benchmarks used through the main paper. For all the experiments, images of different sizes were reshaped for compatibility with CLIP (i.e., 224x224).
Natural images. We employed CIFAR-10 and CIFAR- 100 [31], both containing 10,000 images of size 28 x8,, and spanning 10 and 100 classes, respectively. We use Imagenet [32] as larger-scale dataset, with 1000 classes and 50,000 images in total.
Corruptions. Transformed variants of the previous benchmarks are built by applying 15 different corruptions such as gaussian fog, or pixelate. This results in CIFAR- and CIFAR-100 [33] and Imagenet-C. Each corruption utilized its highest severity level (e.g.. level 5), yielding the most complex version of each dataset. The number of images in each corrupted set and their size correspond to the previous benchmark, which results 45 different datasets evaluate in total.
Fine-grained classification datasets are popular choice in zero-shot classification with CLIP, as they span a wide semantic variety in their classes. We utilize Imagenet as well 10 other datasets covering: Aircraft [34], Caltech101 [35], Cars [36], DTD [37], EuroSat [38], Flowers102 [39], Food101 [40], Pets [41], SUN397 [42], and UCF101 [43]. The specific details each dataset are condensed in Table 5.

---

Dataset
Classes
Size
Category
Aircraft
100
3,333
Transportation
Caltech101
100
2,465
Objects
Cars
196
8,041
Transportation
DTD
47
1,692
Textures
EuroSat
10
8,100
Satellite
Flowers 102
102
2,463
Flora
Food101
101
30,300
Food
Pets
37
3,669
Fauna
SUN397
397
19,850
Scenes
UCFI01
101
3,783
Actions
Table 5: Detailed information of the fine-grained classification benchmark.
Dataset
Domain
Size
Photo 1,670
Dataset
Domain
Classes
Size
PACS
Cartoon
2,344
Imagenet
Natural
1,000
50,000
Sketch
3,929
Imagenet- V2
Natural
1,000
10,000
Art painting
2,048
Imagenet-S
Sketch
1,000
50,000
Art 965
Imagenet-R
Art
200
30,000
Clipart
2,535
Imagenet-A
Adversarial
7,500
7,500
OfficeHome
Product
2,470
Real world
1,495
(a) PACS and OfficeHome
(b) Imagenet-Domains
Table 6: Detailed dataset statisties. (a) PACS and OfficeHome. (b) Imagenet-Domains.
Domain generalization. This is a set of datasets popularly use the context of Domain Adaptation. We use Visda-C [44], which includes 12 common classes and contains two main sets: of 152,397 3D renderings and set of 55,388 of images cropped from MS COCO [45]. We also incorporate PACS [46], with seven classes, and OfficeHome [47] with 65 classes, which include images in four different styles, as summarized in Table 6a). Finally, we include the challenging Imagenet- Domains benchmark, involving four variants of Imagenet: Imagenet-V2 [48], Imagenet-R [49], Imagenet-S [50], Imagenet-A [5]], each of which is detailed in Table 6b).
dittr--diittrbuttonn datasets. In our open-set TTA setup, each ID dataset the natural and corrupted image benchmarks is paired with corresponding OOD dataset. The classification task is performed only on ID samples, while OOD samples are solely used for detection (i.e., recognizing and rejecting unknowns). Thus, OOD class labels are not meaningful in this context. Following prior work [28, 27]. we use SVHN [52] (26,032 street view digit images) as the OOD set for CIFAR-10 and CIFAR-100. and Places365 [53] (1. (1.8M scene images) for ImageNet, In the corrupted setting (i.e., 10/10-10/10-CC C and ImageNet-C), we use SVHN-C and Places365 as OOD sources, matched by corruption type (e.g., JPEG compression) and set to maximum severity.
Baselines. We group baselines into three categories based on their adaptation strategy. The first group includes entropy-based methods for standard classifiers such as TENT [11], ETA [12], SAR [13], RoTTA [14], OSTTA [17], SoTTA [26], STAMP [27]. and UniEnt [28]. These methods typically operate by minimizing the conditional entropy of the model's predictions and require adaptations to work with CLIP's Dision-language outputs. The second group comprises CLIP-specific methods such as CLIPArTT [6] and WATT [7], which modify the loss or prompt structure to better leverage CLIP's multimodal nature. The third group includes alternative CLiP-based adaptation approaches: TPT [3], which performs prompt tuning entropy minimization, and TDA [4], which operates without gradients using memory-based episodic scheme. All baselines are implemented following their respective publications. For CLie-bbseed methods, minimal changes were needed to integrate into our framework. For non-CLIP methods, use CLIP's image-to-text similarities (as

---

defined in Eq.1, Sec.3) as classification logits. Entropy- based baselines directly apply their loss to these logits. Hyperparameter details are provided below when applicable.
ETA: similarity threshold of c and an entropy threshold a = 0.4 are used. These are kept for all cases.
SAR: an entropy threshold 0.4 and an exponential moving average (EMA) weight m = 0.2 are used for all cases. The SAM optimizer is employed.
timeliness weight A and uncertainty weight X 1, memory capacity equivalent to the batch size. These are kept for all cases.
RoTTA:
TDA: we use the same values used for Imagenet in the original paper. We employ Apos 2.0, Spus 2.0, Oneg = 0.117, Bneg 1.0, entropy thresholds H, (0.2, 0.5), entropy masks M, = (0.03, 1.0), and positive and negative shot capacities of 2 and 3, respectively.
CLIPArTT: take K = most probable classes in all datasets, except for K 5 in VisDA-C, which uses a learning rate of 1 x 10-5.
WATT: we use two adaptation iterations per text prompt, and two repa-ieptt are used. A learning rate x 10-5 used for VisDA-C.
we use the confidence threshold т 1/|C), with C the number of classes. The memory capacity is equal to the batch size. The SAM optimizer is employed.
SoTTA:
UniEnt: we use Areg = and Avod
1.
C.3 Extended experimental results
Adapting the text encoder. CLIPTTA is evaluated across a diverse set of datasets by adapting not only the visual encoder but also the text encoder, as shown in Ta- ble 7. Updating the text encoder proves beneficial many cases, particularly for se- mantically complex datasets where CLIP's pre-trained embeddings may lack sufficient specialization. This evident in datasets focused on fine-grained classification, such as SUN397 and OxfordPets, where incorpo- rating text encoder updates yields notable improvements. However, updating the encoder can sometimes have detrimental ef- fects, especially on datasets containing gen- eral or well-represented concepts, such as EuroSat. Despite being visually challeng- ing, the broad and commonly encountered class labels such datasets may already be adequately represented in CLIP's original text embeddings.
CLIPTTA (Vision only)
CLIPTTA (Vision Text)
Dataset
CIFAR-10
95.0
93.5
(-1.5)
CIFAR-100
74.9
75.0
(+0.1)
ImageNet
69.1
69,6
(+0.5)
ImageNet- V2
62.7
63.1 (+0.4)
ImageNet-A
54.0
54.2 (+0.2)
ImageNet-R
80.1
79.9 (.0.2)
ImageNet-S
50.8
51.2 (+0.4)
Aircraft
26.5
26.9 (+0.4)
Caltech101
94.2
94.4 (+0.2)
Cars
66.7
67.1 (+0.4)
DTD
46.5
48.1 (+1.6)
EuroSat
80.3
72.9 (-7.4)
Flowers
71.3
71.7 (+0.4)
Food101
86.7.
86.8 (+0.1)
OxfordPets
91.6
92.4
(+0.8)
SUN397
65.2
67.5
(+2.5)
UCF101
69.3
70.3
(+1.0)
Median
69.2
70.3 (+L1)
these cases, further adaptation of the text encoder may disrupt this alignment, leading to performance degradation. This behavior underscores the importance of selectively adapting the text encoder based on the semantic complexity of the dataset.
Table Impact of updating the text encoder.
Moreover, the results highlight the trade-off between generalization and specialization when jointly adapting both encoders. While semantically complex datasets benefit from increased specialization, datasets with simpler or well-represented class concepts risk losing the robust general- ization capabilities inherent to CLIP's pre-trained representations. This suggests that targeted or dataset-specific strategy for adapting the text encoder may be more effective in leveraging potential.

---

Open-set TTA on corrupted datasets. Table 9 reports results in the challenging open-set set- ting under corruption shifts. This scenario challenging because models must adapt to noisy in-distribution samples while maintaining robustness to unseen OOD classes. As previously observed, TENT highly unstable in these settings, suffering from severe model collapse that is exacerbated by corrupted inputs. Its accuracy drops to 2.1% 1% on ImageNet-C and 10.6% on CIFAR- 100-C, with poor OOD detection (FPR95 above 95%). confirming its sensitivity to pseudo-label noise.
In contrast, CLIPTTA with the OCE loss maintains high performance across all benchmarks, achieving the best overall results on both accuracy and OOD detection. On average, on the corrupted datasets, it improves over UniEnt by +5.8 points in accuracy and reduces FPR95 by nearly 20 points. These gains demonstrate the benefit of aligning the adaptation objective with CLIP's pre-training loss while integrating an explicit OOD detection signal. The results confirm that CLIPTTA is well-suited for open-set test-time adaptation, even under strong distribution shifts such as corruptions.
CIFAR-I0
CIFAR-100
ImageNet
Average
ACC1
AUC1
FPR95]
ACC1
AUC1
FPR95]
ACC1
AUCT
FPR95]
ACC1
AUC1
FPR95.
CLIP
89.3
98.5
5.2
68.1
86.8
83.5
66.7
90.1
43.8
74.7
91.8
44.2
TENT [1I]
93.0
42.3
89.3
69.1
36.2
94.8
12.4
49.9
89.4
58.2
42.8
91.2
OSTTA [17]
90.9
60.5
72.8
70.9
43.3
93.8
66.9
84.9
59.2
76.2
62.9
75.3
SoTTA [26]
89.5
98.5
4.9
68.9
88.5
76.3
66.7
89.3
47.1
75.0
92.1
42.8
STAMP [27]
89.9
98.6
5.5
67.5
87.7
80.0
29.7
63.0
80.2
62.4
83.1
55.2
UniEnt [28]
94.2
99.9
0.0
72.7
97.8
8.7
65.2
95.4
17.1
77.3
97.7
8.6
CLIFTTA + OCB (ours)
94.6
99.8
0.4
74.9
98.4
7.6
67.6
97.7
9.7
79.0
98.6
5,9
Table 8: Open-set TTA results. Top-1 accuracy with ViT-B/16 backbone on the open-set setting.
CIFAR-I0-C
CIFAR- 100-C
Imagenet-C
Average
ACC1
AUC
FPR95
ACC1
AUC
FPR95,
ACC1
AUC
FPR95
ACC1
AUCT
FPR95]
CLIP
60.2
88.0
58.0
35.2
67.0
93,8
24.6
68.6
89.2
40.0
74.5
80.3
TENT [11]
26.9
50.7
91.2
10.6
43.1
95.0
2.1
48.0
95.0
13.2
47.3
93.7
OSTTA [17]
62.7
53.4
85.2
34.5
36.3
92.6
31.2
75.1
79.8
42.8
54.9
85.9
STAMP [27]
60.4
88.0
57.1
34.5
66.8
93.8
53.8
92.9
34.6
69.5
81.3
UniEnt [28]
78.7
98.6
5.7
48.9
91.0
31.0
23.6
44.8
90.8
50.4
78.1
42.5
CLIPTTA + OCE (ours)
79.1
98.7
6.1
50.4
96.7
19.2
39.0
89.0
43.2
56.2
94.8
22.8
Table 9: Open-set TTA results on Corrupted Datasets. Top-1 accuracy with ViT-B/16 backbone on the open-set setting.
Overall performance across shift types. To provide global overview of our evaluation, we summarize in Tab. 10 the average performance of all methods across the four distribution shift categories considered in this work: (i) synthetic corruptions (CIFAR-10/1004 domain shifts (ImageNet variants such as Sketch and Paintings), (iii) coarse coarse-grained semantic shifts (CIFAR-10, CIFAR-100), and (iv) fine-grained semantic shifts (11 CLIP zero-shot datasets, including IeageNet). This table offers a consolidated view of the results presented the following sections. As shown, CLIPTTA consistently achieves the best performance across all shift types, with particularly large improvements in challenging settings where the base CLIP model performs poorly. For example, under synthetic corruptions, CLIP reaches 40.3% accuracy on average, while CLIPTTA attains 58. 1%, a gain of +1 17.8 points and 9.1 points above the second-best method. When the base model already performs well-such such in domain shift benchmarks the improvements are smaller (83.7% vs. 82.8% for TDA). These results suggest that CLIPTTA 's benefits are more pronounced when the initial model accuracy low. Detailed results for each dataset are provided in the following sections.
ImageNet-C), (ii)
Domain shifts benchmarks. We provide the extended results of the different domain shifts (Ta- ble 1), including Imagenet-Domains (Table 11), VisDA-C (Table 12), OfficeHome (Table 13), and PACS (Table 14). CLIPTTA achieves the best performance across all of these datasets on average, demonstrating great flexibility across domains. Our method also obtains highly competitive results independently in each sub-datasct.

---

Method
Corruptions
Domain Shifts
Coarse Coarse-Grained
Fine-Grained
CLIP
40.3
81.3
78.7
64.7
TENT [11]
35.1
82.3
83.9
65.3
ETA [12]
42.3
82.3
84.3
65.4
SAR [13]
48.2
82.
82.7
65.3
RoTTA [14]
38.7
80.9
79.0
62.1
CLIPArTT [6]
49.8
80.8
80.8
64.4
WATT [7]
43.5
82.1
81.7
65.3
TPT [3]
38.7
80.8
78.6
65.6
TDA [4]
42.5
82.8
80.6
67.7
CLIPTTA
58.1
83.7
85.2
69.8
Table 10: Overall performance across shift types. Average accuracy (%) across four categories of distribution shifts. Detailed per-dataset results provided in the following sections.
ImageNet
ImageNet-A
ImageNet-Vi ImageNet-R
ImageNet-S
Average
CLIP
66.7
47.8
60.8
74.0
47.8
59,4
TPT (NeurIPS '22)
69.0
54.8
63.5
77.1
47.9
62.4
TDA (CVPR
69.5
60.1
64.7
80.2
50.5
65.0
TENT (ICLR '21)
66.5
51.3
60.2
79.4
43.7
60.2
ETA (CML '22)
67.4
49.2
60.9
46.8
59.9
SAR (CCLR (23)
66.7
51.5
79.6
44.6
60.6
RoTTA (CVPR '23)
68.4
51.2
62.5
78.1
47.8
61.6
CLIPArTT (WACV '25)
67.6
50.7
61.2
76.2
47.9
60.7
WATT (NewrPP '24)
69.0
51.1
62.5
78.1
48.2
61.8
CLIPTTA (ours)
69.6
54.0
62.7
80.2
50.8
63.4
Table 11:
Detailed results on the Imagenet-Donen -Domains benchmark.
Synthetic 3D
MS COCO
Average
CLIP
87.2
86.7
87.0
TPT [3]
85.5
84.5
85.0
TDA [4]
86.6
86.5
TENT [11]
93.2
85.3
89.3
ETA [12]
91.1
85.4
88.3
SAR [13]
88.1
87.5
87.8
RoTTA [14]
80.6
86.7
83.7
CLIPArTT [6]
82.2
86.0
84.1
WATT
88.4
87.0
87.7
CLIPTTA (ours)
86.9
Table 12.
Detafled resufts on the two domains of the Visda-C dataset. Real
Art
Clipart
Product
Average
CLIP
83.2
68.0
89.1
89.8
82.5
TPT [3]
82.5
88.5
81.7
TDA [4]
83.2
68.8
89.8
90.4
83.0
TENT [11]
84.1
68.8
90.0
90.5
83.4
ETA [12]
84.3
70.8
90.4
90.7
84.1
SAR [13]
84.4
70.9
89.6
90.3
83.8
RoTTA [14]
82.9
68.0
89.1
89.8
82.5
CLIPArTT [6]
82.6
68.4
87.6
89.6
82.0
WATT [7]
83.8
69.0
90.0
90.5
83.4
CLIPTTA (ours)
84.2
70.7
91.0
91.0
84.2
Table 13: Detailed results the four domains of the OfficeHome (OH) dataset.
10

---

Photo
Art
Cartoon
Sketch
Average
CLIP
99.9
97.4
99.1
88.1
96.1
IPT [3]
99.5
95.3
93.9
87.2
94.0
TDA [4]
99.9
97.5
98.9
88.1
96.1
TENT [11]
99.8
98.0
99.2
89.1
96.6
ETA [12]
99.8
97.9
99.3
89.8
96.7
SAR [13]
99.9
97.5
99.1
88.2
96.2
RoTTA [14]
99.9
98.8
88.1
CLIPArTT [6]
99.5
96.9
98.3
90.4
96.2
WATT [7]
99.9
97.6
99.2
88.4
96.2
CLIPTTA (ours)
99.9
98.0
99.3
92.0
97.5
Table 14: Detailed results on the four domains of the PACS dataset.
CIFAR 10
CIFAR-100
Average
CLIP
89.3
68.1
78.7
TPT [3]
89.8
67.4
78.6
TDA [4]
91.4
69.8
80.6
TENT [11]
94.9
72.9
83.9
ETA [12]
94.8
73.7
84.3
SAR [13]
92.1
73,2
82.7
RoTTA [14]
89.4
68.5
79.0
CLIPAITT [6]
88.4
73.2
80.8
WATT
92.5
70.8
81.7
CLIPTTA (ours)
95.1
75.3
85.2
Table 15: Closed-set TTA on coarse-grained datasets. Top-1 accuracy with ViT-B/16 backbone on coarse-grained datasets (CIFAR-10 and CIFAR-100).
ImageNet
Aircrutt
Calechio1
Cars
DTD
EaroSAY
Flowers102
Food10)
OxferdPets
SUN399
UCF101
Averge
CLIP
66.7
24.8
92.2
65.5
44.1
48.3
70.7
84.8
88.4
62.3
64.7
64.7
TPT [3]
69.0
24.8
94.2
66.9
47.8
42.4
69.0
84.7
87.8
65.5
68.0
65.6
TDA [4]
69.5
23.9
94.2
67.3
47.4
58.0
71.4
86.1
88.6
67.6
70.7
67.7
TENT [11]
66.5
15.5
93.8
58.4
71.3
86.5
89.5
68.0
65.3
ETA [12]
67.4
24.8
93.0
65.2
44.4
47.5
71.4
85.9
89.2
63.6
66.6
65.4
SAR [13]
66.7
21.9
93.9
50.2
70.9
86.5
67.7
65.3
RoTTA [14]
68.4
22.3
94.0
58.1
45.2
24.2
70.5
81.6
87.0
64.9
66.8
62.1
CLIPAITT [6]
67.5
24.0
92.7
64.0
43.4
46.7
67.0
84.2
87.1
64.2
67.0
64.4
WATT [7]
69.0
23.6
94.1
65.8
44.7
40.0
71.4
86.2
88.7
66.3
68.2
65.3
CLIFTTA (ours)
69.6
26.5
94.2
66.7
46.5
80.3
71.3
86.7
91.6
65.2
69.3
69.8
Table 16:
Closed-set TTA on fine-grained datasets. Top-1 accuracy comparison CLIPTTA against other TTA methods on a suite of 11 fine-grained datasets.
Semantic datasets. We report closed-set adaptation results on both coarse- and fine- ine-grained clas- coarse-grained
sification tasks in Tables 15 and 16. On coarse grained benchmarks (CIFAR-10 10 and CIFAR-100), CLIPTTA achieves the highest accuracy on both datasets, with a strong average of 85.2%, outperform- ing all TENT-based and CLIP-based methods, including CLIPArTT and WATT. Notably, it improves over TENT by +0.2 points on CIFAR-10 10 and +2.4 points CIFAR- 100. and remains significantly ahead of zero-shot CL.IP (+6.5 points on average). fine-grained datasets, CLIPTTA consistently ranks among the top methods, achieving the best average accuracy across the 1 datasets (69.8%).
11

---

Backbone
Dataset
Base
TENT
CLIPTTA
ImageNet
58.2
58.0
59.8
CLIP-ResNeth
CIFAR-10 10
68.7
72.2
84.0
CIFAR-100
40.6
44.7
55.8
ImageNet
62.0
61.4
68.2
CLIP-ViT-B/32
CIFAR-10
88.7
93,2
93.4
CIFAR-100
64.0
69.6
72.5
ImageNet
73.5
74.2
74.6
CLIP-ViT-L/14
CIFAR-10
95.4
97.3
97.5
CIFAR-100
75.9
82.1
82.4
ImageNet
75.7
75.5
75.9
SigLIP-VIT-B/L
CIFAR-10
92.5
96.0
96.0
CIFAR-100
70.9
78.2
78.7
Table 18: Evaluation across architectures. CLIPTTA consistently improves over both the base model and TENT across various CLIP and SigLIP backbones. Results top-1 accuracies (%).
Despite its simplicity, it performs favorably compared to more complex CLIP-specific methods such TPT, TDA, and CLIPArTT, which rely on prompt tuning heuristic loss modifications. CLIPTTA performs particularly well on datasets like EuroSAT (+22.3 over CLIPArTT) and OxfordPets (+4.5 over TDA) while maintaining competitive results the others. These findings highlight the ro- bustness of our adaptation objective across both coarse- and fine-grained tasks without requiring task-specific tuning architectural modifications.
Statistical significance. We con- duct additional runs CLIPTTA to as- sess its sensitivity to random initializa- tion, reporting the mean accuracy and 95% confidence interval in Tab. 17. The results indicate that CLIPTTA is highly stable, with very low variance across independent runs. The tight confidence intervals (e.g., +0.01 on ImageNet) confirm the reliability and reproducibility the observed performance gains, further supporting the robustness of the method across different datasets.
CIFAR-10
CIFAR-100
Imagenet
CLIPTTA
94.9 + 0.03
75.3 + 0.07
69.1 + 0.01
Table 17: Accuracy of CLIPTTA averaged over three random initializations (mean + 95% CT).
Evaluation across architectures. To assess the generality of CLIPTTA beyond the default ViT-B/16 backbone, we extended our experiments three additional CLIP variants (ResNet50, ViT-B/32, and ViT-L/14) and another contrastive vision-language model, SigLIP- SigL.I--iTTB/16. ViT-B/16. As shown in Tab. 18, CLIPTTA consistently improves over both the base model and TENT across all architectures and datasets. For example, on CLIP-ResNet5i it yields gains of +12.0 and +11.1 points on CIFAR-10 and CIFAR-100, respectively, highlighting robustness across training paradigms. Improvements on SigLIP are smaller, with saturation on easier datasets such as CIFAR-10 (96.0%). We expect larger gains on more challenging corrupted benchmarks. These results confirm that our soft contrastive objective generalizes beyond specific backbone, though all models here were evaluated with shared hyperparameters. Further cesiin--specialyy adapting the loss to SigLIP's pre-training objective-come yield additional improvements.
Additional baseline comparisons. We further evaluate CLIPTTA against two additional baselines for test-time adaptation of vision-language models: RPL [54] and BATCLIP [55]. Both methods extend entropy minimization (as used in TENT) with additional mechanisms to improve robustness. RPL leverages lseudo-labels and generalized cross-entropy loss to mitigate overconfidence, while BATCLIP adapts both visual and textual encoders using two auxiliary objectives projection matching loss aligning visual prototypes with text features, and an inter-class separability loss promoting distinctiveness. As shown in Tab. 19, CLIPTTA surpasses both RPI. and BATCLIP across CIFAR-IO-C. CIFAR-100-C. and ImageNet-C. demonstrating the effectiveness of our soft contrastive objective compared to more complex adaptation strategics. All results are reported using the CLIP
12

---

ViT-B/16 backbone under identical evaluation settings.
Method
CIFAR-I0-C
CIFAR-100-C
ImageNer-C
Average
CLIP
60.2
35.2
25.5
40.3
RPL [54]
61.5
38.5
25.1
41.7
BATCLIP [55]
73.9
42.1
30.7
48.9
CLIPTTA
80.7
52.6
41.1
58.1
Table 19: Comparison with RPL and BATCLIP. Top-1 accuracy (%) on corrupted benchmarks using the CLIP VIT-B/16 backbone.
Runtime and memory usage. We compare the runtime and GPU memory usage of our soft contrastive loss with other gradient-based test-time adaptation methods under identical conditions, using single A6000 GPU and a batch size 128. As shown in Tab. 20, CLIPTTA introduces negligible memory overhead compared TENT and achieves competitive runtime, outperforming several recent approaches such as ETA, SAR, CLIPArTT, and WATT. Although TPT [3] is designed for single-image adaptation, its reliance test-time augmentations leads similar memory demands and higher latency. Despite its contrastive formulation, CLIPTTA remains efficient even with small batch sizes: for instance, on CIFAR-10 ( it attains 93.4% accuracy with batch size of substantially surpassing TENT (40.3%) and TPT (89.8% under the same conditions. This demonstrates that CLIPTTA well suited to low-resource or streaming scenarios where batch size constrained.
Method
Time (s)
Memory (Gb)
CLIP
11
1.2
TENT [11]
312
8.1
ETA [12]
364
8.3
SAR [13]
705
8.1
CLIPArTT [6]
335
8.1
WATT [7]
1260
8.3
TPT [3]
894
9.8
L--cont
315
8.1
Table 20: Runtime and memory usage. Comparison of adaptation time and GPU memory for gradient-based TTA methods. All results obtained on single A6000 GPU with batch size 128.
C.4 Hy perpparrreee analysis.
In this section, evaluate the sensitivity of CLIPTTA to its key hyperparameters: the regularization weight Avgg the OOD loss weight Poces and the adaptation batch size. Results show that CLIPTTA remains robust across wide range of values, requiring minimal tuning for strong performance.
Effect of Areg- Figure 7 shows the impact of the regularization weight Areg on CIFAR-100 accuracy. We observe that CLIPTTA is remarkably stable for values ranging from 0.5 to 2.0, with accuracy consistently above 75% in this range. Performance peaks around Areg 1, which we use as the default. Beyond that, accuracy gradually declines, indicating that overly strong regularization may suppress beneficial updates. Overall, this confirms that CLIPTTA does not require precise tuning of Areg perform well and that wide range of values yields near-optimal performance.
Effect of Aoce- Table 21 reports the impact of Auce on ImageNet in the open-set setting. While accuracy stays stable for small values, OOD detection improves substantially: AUROC increases from 93.5% (no OCE) to 97.7% at Xuce 1, and FPR95 drops by 16 points. Performance remains robust the range [0.25-2], confirming the stability of the OCE loss.
13

---

Ages
0
0.25
0.5
1
2
5
10
20
100
Acc
67.6
67.6
67.6
67.6
67.5
67.3
66.4
64.5
56.6
AUC
93.5
97.5
97.6
97.7
97.8
98.0
98.4
98.8
99.2
FPR
25.7
10.1
9.8
9.7
8.8
7.8
6.3
4.7
2.3
Table 21: Impact of Poee on Imagenet. Effect of Aose on accuracy and open-set metrics AUROC (AUC) and false positive rate (FPR).
Effect of batch size. Table 22 compares the performance of TENT, the standalone soft contrastive objective (Ls-cout), and CLIPTTA CIFAR-10 10 for batch sizes ranging from to 512. While CLIPTTA benefits slightly from larger batches, its performance remains stable across all sizes. Remarkably, even in the extreme case of batch size 1, CLIPTTA achieves 93.4% accuracy-fai far above TENT (40.3 %) and Lo- (89.3 %). This robustness is enabled by the Class-wise Confident Memory (CCM), which stores past confident samples and allows contrastive adaptation even when only single test example is available.
Batch size
16
32
64
128
256
512
TENT [11]
40.3
89.8
92.6
94.6
94,4
94.7
94.9
94.7
94.6
L CLIPTTA
89.3
90.1
93.6
94.7
94,9
94.9
95.0
94.9
94.9
93.4
94.7
94.7
94.8
94.8
95.0
95.1
95.1
95.2
Table 22: Effect of batch size CIFAR-10. Accuracy (%) for different batch sizes. While CLIPTTA benefits from larger batches, remains competitive even in the case of image adaptation thanks to the Class-wise Confident Memory (CCM).
75
Accuracy (%) T
74
72
70
Accuracy
10
A
Areg
Figure 7: Impact of Xveg on CIFAR-100. Effect of Areg on the closed-set accuracy of CLIPTTA when evaluated on CIFAR-100.
14


Reviews:
[{"review_id": "ouBnvK72sj", "reviewer": "Reviewer_7qJh", "summary": "This paper presents CLIPTTA, a novel test-time adaptation (TTA) framework tailored for vision-language models such as CLIP. CLIPTTA addresses the limitations of entropy-based adaptation objectives by introducing a soft contrastive image-text loss, which better aligns with CLIP’s original contrastive pretraining. This loss helps reduce pseudo-label drift and mitigates class collapse during adaptation. Additionally, the paper proposes an Outlier Contrastive Exposure (OCE) loss, aimed at improving in-distribution (ID) and out-of-distribution (OOD) separation for more robust adaptation under distribution shifts. CLIPTTA is evaluated across an extensive benchmark of 75 datasets and shows consistent improvements over standard entropy-based approaches (e.g., TENT) and is competitive with recent state-of-the-art methods such as CLIPArTT.", "questions": "1) Could the authors provide a runtime and memory usage comparison with other TTA methods? Given the reliance on large batch sizes for contrastive loss, how does CLIPTTA scale to lower-resource settings?\n\n2) Can the authors provide more detailed per-dataset or per-shift-type analysis? Are there specific types of distribution shifts (e.g., adversarial, synthetic corruptions) where CLIPTTA struggles or particularly excels?\n\n3) In Figure 4, how does CLIPTTA compare to baselines in terms of class collapse and deterioration ratio? Could this be discussed more explicitly?\n\n4) How does the Class-wise Confident Memory (CCM) handle OOD samples during updates? Are there safeguards to prevent incorporating noisy or low-confidence predictions?\n\n5) Does optimizing the OCE loss introduce a trade-off in classification performance on in-distribution samples? If so, how significant is this trade-off?", "limitations": "Missing limitations and not properly discussed in the conclusion.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1) The introduction of the soft contrastive loss is a principled design choice, closely aligned with CLIP’s pretraining, and serves as a more appropriate alternative to entropy minimization for VLMs.\n\n2) The proposed OCE loss improves robustness by explicitly encouraging separation between ID and OOD samples, which is critical for real-world generalization.\n\n3) The paper is clearly written and well-structured, with strong empirical results across a diverse and challenging set of benchmarks.\n\nWeaknesses:\n\n1) Computational Efficiency and Scalability:\nThe method’s reliance on large batch sizes for effective contrastive loss raises concerns about efficiency and scalability. Although the paper mentions using two NVIDIA V100 GPUs, it does not report runtime, memory usage, or how performance scales with smaller hardware setups. A quantitative analysis of resource requirements would strengthen the work.\n\n2) Unexplored Failure Cases of Contrastive Loss:\nThe paper does not examine scenarios where soft contrastive loss might break down—e.g., when pseudo-labels are highly correlated within a batch, or when the batch contains a high proportion of OOD samples. Such conditions could undermine the contrastive objective, and identifying these limits would be helpful.\n\n3) Limited Analysis Across Dataset Types:\nAlthough evaluated on 75 datasets, the paper offers limited insight into how CLIPTTA performs under different types of distribution shift (e.g., adversarial, synthetic, or natural corruptions). Highlighting specific strengths or weaknesses across shift categories would improve understanding of the method’s robustness.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns during this rebuttal period. While there are some clarity issues and missing discussion regarding the limitation of the work, I think that these issues can be fixed in the camera ready.", "rebuttal": "We thank the reviewer for their insightful questions, which help us clarify our work.\n\n1. __Q1 & W1- Runtime and memory usage:__ \nTo fulfill the reviewer’s request, we compare the runtime and GPU memory usage of our soft contrastive loss $L_{\\text{s-cont}}$ with other gradient-based test-time adaptation methods. All methods were evaluated under the same conditions using a single A6000 GPU and a batch size of 128. As shown in the table below, CLIPTTA introduces minimal memory overhead compared to TENT and achieves competitive runtime, outperforming several recent approaches such as ETA, SAR, CLIPArTT, and WATT.  Methods like TPT [7], although designed for single-image adaptation, rely on test-time augmentations that introduce similar memory overheads and longer execution times.\n\n    | Method | Time (s)  | Memory (Gb)  |\n    |------------|------|--------------------|\n    | CLIP                 | 10.74 | 1.195 |\n    |TENT [1]           | 312 | 8.085 |\n    | ETA [2]            | 364.21| 8.252 |\n    | SAR [3]           |  704.88 | 8.085 |\n    | CLIPArTT [4]  | 334.96 | 8.087 |\n    | WATT [5]        | 1259.91 | 8.256 |\n    | TPT [7]        | 894.21  | 9.784  |\n    | $L_{\\text{s-cont}}$ (ours)  | 315.01 | 8.085 |\n\n    Although $L_{\\text{s-cont}}$ is a contrastive objective, **it does not require large batch sizes to perform well**. As discussed in our responses to reviewers JXSL and 26qe, CLIPTTA achieves strong performance even in the extreme case of batch size 1, thanks to the Confident Consistency Memory (CCM), which stores past confident samples. For example, on CIFAR-10, CLIPTTA reaches 93.4% accuracy with a batch size of 1, significantly outperforming TENT (40.3%) and TPT (89.8%) under the same conditions. This demonstrates that CLIPTTA remains practical for low-resource or streaming settings, where batch sizes may be limited. \n\n2. __W2 - Unexplored Failure Cases of Contrastive Loss__:\nWe agree with the reviewer that our soft contrastive loss is batch-aware and, by design, assumes the presence of sufficiently good pseudo-labels within the batch to enable effective adaptation. This reliance on pseudo-label quality is not unique to our method but shared by all gradient-based test-time adaptation (TTA) techniques, including entropy-based approaches like TENT, which can also suffer from collapse or error reinforcement under high label noise.\n\n    To empirically assess robustness in such settings, we highlight the case of ImageNet-C, where the zero-shot CLIP accuracy is only 25.1%, implying that ~75% of pseudo-labels are incorrect. Despite this, CLIPTTA reaches 41.1% accuracy, significantly outperforming all baselines, including TENT, which collapses to 17.6% (lower than CLIP's accuracy). This suggests that our batch-aware loss is considerably more robust to pseudo-label drift than existing methods, as it can still leverage confident and semantically consistent examples for adaptation, even under substantial noise.\n\n    That said, in extreme scenarios where few or no pseudo-labels are correct, adaptation may fail—as is the case for other self-supervised TTA methods. We thank the reviewer for raising this important point and will include a discussion of this limitation in the final manuscript.\n\n3. __Q2 & W3 - Per-shift performance analysis:__ \nAs mentioned in the main paper (lines 63–64 and 216), we structure our evaluation around four distinct types of distribution shifts:\n    - **Synthetic corruptions**, including CIFAR-10/100-C and ImageNet-C;\n    - **Domain shifts**, comprising datasets like Sketch, Paintings, and other ImageNet variants;\n    - **Coarse-grained semantic shifts**, such as CIFAR-10 and CIFAR-100;\n    - **Fine-grained semantic shifts**, evaluated on 11 CLIP zero-shot datasets, including ImageNet.\n\n    To complement the reviewer’s request, we provide in the table below a consolidated view of CLIPTTA’s performance across these shift categories, compared to other recent TTA baselines:\n    | Method       | Corruptions | Domain Shifts | Coarse-Grained | Fine-Grained |\n    |--------------|:-----------:|:-------------:|:--------------:|:------------:|\n    | CLIP         |       40.3      |  81.3    |     78.7    |  64.7   |\n    | TENT [1]     |      35.1     |  82.3    |    83.9   |    65.3  |\n    | ETA [2]      |   42.3          |    82.3     |  84.3    |   65.4   |\n    | SAR [3]      |     48.2       |   82.1    |   82.7    |  65.3   |\n    | RoTTA [6]    |   38.7       |     80.9    |    79.0     |  62.1   |\n    | CLIPArTT [4] |  49.8      |    80.8     |    80.8    |  64.4  |\n    | WATT [5]     |  43.5       |    82.1    |     81.7     |   65.3    |\n    | TPT [7]      |    38.7      |    80.8   |    78.6     |    65.6   |\n    | TDA [8]      |   42.5      |   82.8   |     80.6    |    67.7     |\n    | CLIPTTA (ours)      |   **58.1**    |    **83.7**    |    **85.2**   |   **69.8**   |\n\n    From these results, we observe that CLIPTTA consistently outperforms all other methods across the four shift types, with particularly strong gains in challenging settings. The largest improvements occur when the base model (CLIP) performs poorly. For example, on corruption datasets, CLIP achieves 40.3% accuracy on average, while CLIPTTA reaches 58.1%—a +17.8 point gain, and 9.1 points above the second-best method.\n\n    In contrast, when CLIP already performs well—such as on domain shift datasets with 81.3% base accuracy—the gains are smaller (CLIPTTA reaches 83.7%, vs 82.8% for TDA). These results suggest that CLIPTTA’s improvement is more influenced by the base model’s initial performance than by the specific type of distribution shift. Finally, we did not identify any dataset or shift type where CLIPTTA consistently underperforms compared to other methods.\n\n4. __Q3: Extended analysis on class collapse:__ \nAs requested by the reviewer, we compare the entropy of the predictions produced by CLIPTTA and several baselines over the course of adaptation on CIFAR-10-C. The table below reports the mean predictive entropy at different batch indices during test-time adaptation:\n    |  | Batch 0  | Batch 20   | Batch 40  | Batch 60 | \n    |------------|:------:|:------:|:------:|:------:|\n    | TENT [1]                                      | 2.19 | 1.75 | 1.44 | 1.19 | \n    | ETA [2]                                       | 2.17 | 2.18 | 2.18 | 2.17 | \n    | SAR [3]                                      | 2.17 | 2.13 | 2.13 | 2.15 | \n    | CLIPArTT [4]                              | 2.19 | 2.24 | 2.2 | 2.13 | \n    | WATT [5]                                     | 2.21 | 2.18 | 2.16 | 2.10 |\n    | CLIPTTA  (ours)     | __2.20__ | __2.27__ | __2.28__ | __2.29__ | \n\n    As shown, CLIPTTA maintains high predictive entropy throughout adaptation, approaching the theoretical maximum of 2.3 for CIFAR-10. This indicates effective preservation of class diversity and avoidance of class collapse, in contrast to entropy-minimization methods like TENT.\n    A similar trend is observed for the deterioration ratio: CLIPTTA maintains or improves accuracy over time, whereas methods like TENT can degrade after a few batches. However, this metric is less straightforward to interpret for approaches like ETA and SAR, which adapt to dynamically filtered subsets of the batch. We also clarify that Fig. 4b and 4c were primarily designed to compare the adaptation dynamics of our soft contrastive loss with TENT. We will consider extending the analysis to all baselines in the final version.\n5. __Q4 - Safeguards against OOD contamination in the CCM Memory:__ \nTo clarify, the Class-wise Confident Memory (CCM) retains the top-k most confident samples ever observed for each class. This design ensures that only consistently high-confidence predictions are eligible for inclusion, effectively preventing the incorporation of noisy or uncertain examples. Moreover, we filter out samples detected as OOD before updating the CCM. Even if an OOD sample were to bypass this filter, it would still need to rank among the most confident examples ever seen to be included—something we did not observe in practice. Finally, the OCE loss is explicitly designed to reduce the confidence of OOD inputs, further mitigating the risk of contaminating the CCM. These safeguards together ensure the reliability of the memory.\n\n6. __Q5: Impact of OCE loss on classification:__ We evaluate the trade-off between OOD detection and in-distribution (ID) classification accuracy when optimizing the OCE loss in Section C.4 of the supplementary material (Table 17, reproduced below). The OCE loss is weighted by a hyperparameter $\\lambda_\\text{oce}$.\n\n    | $\\lambda_\\text{oce}$ | 0 | 0.25 | 0.5 | 1 | 2 | 5 | 10 | 20 | 100 |\n    |-----|------|------|------|------|------|-----|-----|-----|-----|\n    | Acc | 67.6 | 67.6 | 67.6 | 67.6 | 67.5 | 67.3 | 66.4 | 64.5 | 56.6|\n    | AUC | 93.5 | 97.5 | 97.6 | 97.7 | 97.8 | 98.0 | 98.4 | 98.8 | 99.2 |\n    | FPR | 25.7 | 10.1 | 9.8    | 9.7   | 8.8   | 7.8   | 6.3   | 4.7   | 2.3| \n\n    As shown, the accuracy on ID samples remains remarkably stable across a wide range of $\\lambda_\\text{oce}$ values—from 0.25 to 5—while OOD detection improves steadily. However, when $\\lambda_\\text{oce} \\geq 10$, ID accuracy drops despite continued gains in OOD detection, indicating a trade-off that only emerges at high values.\n\n[1] Tent: Fully test-time adaptation by entropy minimization. ICLR 2021\n\n[2] Efficient test-time model adaptation without forgetting. ICML 2022\n\n[3] Towards stable test-time adaptation in dynamic wild world. ICLR 2023\n\n[4] CLIPArTT: adaptation of CLIP to new domains at test-time. WACV 2025\n\n[5] WATT: Weight average test-time adaption of clip. NeurIPS 2024\n\n[6] Unified entropy optimization for open-set test-time adaptation. CVPR 2024\n\n[7] Test-time prompt tuning for zero-shot generalization in vision-language models.  NeurIPS 2022\n\n[8] Efficient Test-Time Adaptation of Vision-Language Models. CVPR 2024"}, {"review_id": "Gu9NsiKVcK", "reviewer": "Reviewer_4cbr", "summary": "This paper introduces CLIPTTA, a test-time adaptation method for vision-language models like CLIP. It uses CLIP-style contrastive loss and a soft OOD filtering mechanism based on maximum softmax scores. A learnable threshold and optional memory bank further improve adaptation. Experiments show considerable performance in both closed- and open-set settings.", "questions": "Please refer to weaknesses", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n\nThis paper presents a well-structured study on test-time adaptation for CLIP, evaluated across a wide range of datasets. The method demonstrates consistently strong performance, highlighting its robustness and general applicability.\n\n__Weaknesses:__\n\n1. The definition of ID/OOD is somewhat unclear for vision-language models like CLIP, given its undisclosed training data. A sample flagged as OOD by confidence-based filtering may simply reflect underrepresentation, not a true open-set case. A more formal definition would strengthen the framing.\n\n2. The adaptation is performed over 10 steps per sample, which raises questions about test-time efficiency in real-time settings.\n\n3. The method is evaluated solely on CLIP, without examining other vision-language backbones (e.g., SigLIP) or standard CLIP backbones (e.g., ResNet, ViT-L), which limits the generality.\n\n4. The ablation is restricted to typical TTA benchmarks and does not include cross-dataset evaluations commonly used in CLIP adaptation work. Comparing with non-CLIP-specific methods like Tent may also be suboptimal.\n\n5. The paper allows the CLIP logit scale parameter to be updated during test-time adaptation but does not analyse or report its behaviour. Given that CLIP’s pre-trained logit scale is calibrated for contrastive learning, adapting it at test time could introduce instability or overfitting.", "was_revised": true, "final_justification": "Thank you for your response and the additional experiments. I have raised my score accordingly. However, since your work focuses on CLIP adaptation, restricting the evaluation to the standard TTA setting is somewhat limited. I strongly recommend including the new experiments in W3 that address cross-dataset adaptation rather than solely corruption-based shifts.\n\nRegarding your response to W2, there are indeed other works proposing training-free adaptation, such as TDA and TCA, which perform adaptation using augmented or single-batch data. This makes your claim of uniqueness less convincing. Please include a discussion of these related works in the final version.\n\nOverall, the paper presents strong experimental results. I recommend some minor refinement in your final version.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, which helped us identify areas for clarification and improvement. Below, we respond to each point in detail, including additional experiments and clarifications where appropriate.\n\n1) **W1 - Definition of ID/OOD in the CLIP setting**: While the notion of in-distribution (ID) versus out-of-distribution (OOD) may appear ambiguous for vision-language models like CLIP, we follow the task-specific definition of ID/OOD separation as introduced in the MCM paper [1], as mentioned in Section 3.3 of the submission. To clarify, in this context, ID refers to the downstream classification task (e.g., ImageNet-1k or CIFAR-100), which defines the relevant label space, while OOD refers to semantically unrelated samples outside that space. Following [1], this setup has been widely adopted in the CLIP-based OOD detection community [2–4], and we will be glad to clarify and give more details in the related work section.\n\n    Since our objective is to adapt CLIP to a specific task at test time, it is essential to avoid updating the model based on inputs that lie outside the downstream label space. For example, when adapting to ImageNet-1k, including samples from unrelated datasets such as Places or SUN would deteriorate the adaptation process. Our confidence-based filtering addresses this by discarding low-confidence predictions unlikely to belong to the ID distribution.\n    We agree that low-confidence inputs may include underrepresented or ambiguous ID examples. However, such samples also carry a higher risk of adaptation errors. Excluding them is often beneficial in practice, especially in the absence of labels. To improve filtering beyond static thresholds, we introduce the OCE loss, which learns to better distinguish ID from OOD samples during test-time, enhancing adaptation in open-set scenarios.\n\n2) **W2 - Test-time efficiency**:\nWe follow the standard practice in gradient-based TTA, where adaptation is performed using 10 gradient steps [5-7]. This setup provides a good balance between adaptation effectiveness and computation cost, and allows for a controlled and fair comparison with existing baselines. Importantly, our method remains efficient in practice. As detailed in our response to Reviewer 7qJh (Q1), our soft contrastive loss performs adaptation in less time and with minimal memory overhead compared to other competitive TTA methods.\n\n3) **W3 - Limited backbone evaluation**:\nAs requested by the reviewer, we extended our experiments during the rebuttal period to assess the generality of CLIPTTA beyond the default ViT-B/16 backbone. Specifically, we evaluated the method on three additional CLIP architectures (ResNet50, ViT-B/32, and ViT-L/14), as well as another contrastive vision-language backbone (SigLIP-ViT-B/16). \n\n    | Backbone         | Dataset    | Base  | TENT  | CLIPTTA (ours) |\n    |------------------|------------|:-----:|:-----:|:-------------------:|\n    | CLIP-ResNet50     | ImageNet   | 58.2  | 58.0  | **59.8**            |\n    |                  | CIFAR-10   | 68.7  | 72.2  | **84.0**            |\n    |                  | CIFAR-100  | 40.6  | 44.7  | **55.8**            |\n    | CLIP-ViT-B/32     | ImageNet   | 62.0  | 61.4  | **68.2**            |\n    |                  | CIFAR-10   | 88.7  | 93.2  | **93.4**            |\n    |                  | CIFAR-100  | 64.0  | 69.6  | **72.5**            |\n    | CLIP-ViT-L/14     | ImageNet   | 73.5  | 74.2  | **74.6**            |\n    |                  | CIFAR-10   | 95.4  | 97.3  | **97.5**            |\n    |                  | CIFAR-100  | 75.9  | 82.1  | **82.4**            |\n    | SigLIP-ViT-B/16   | ImageNet   | 75.7  | 75.5  | **75.9**            |\n    |                  | CIFAR-10   | 92.5  | **96.0**   | **96.0**            |\n    |                  | CIFAR-100  | 70.9  | 78.2  | **78.7**            |\n\n    Overall, CLIPTTA consistently outperforms both the base model and TENT across all architectures and datasets. For instance, on CLIP-ResNet50, CLIPTTA improves performance over TENT by +12 points on CIFAR-10 and +11.1 points on CIFAR-100, demonstrating its effectiveness across diverse architectures and training paradigms. On the other hand, performance gains appear more modest on SigLIP, with saturation observed on some datasets such as CIFAR-10, where both TENT and CLIPTTA reach 96.0%. We anticipate, however, that improvements could be more pronounced on more challenging benchmarks, such as corrupted datasets with lower zero-shot accuracy. This remains an avenue we were unable to explore within the rebuttal timeline.\n\n    These results confirm that our soft contrastive objective’s effectiveness is not tied to a specific backbone or contrastive formulation. We emphasize, however, that these are preliminary results obtained using a shared hyperparameter setting across all models. We anticipate that further tuning—such as adjusting the learning rate for each backbone—could lead to improved performance. Moreover, adapting the adaptation loss to better align with SigLIP’s pre-training objective could further improve performance and is a promising direction for future work.\n\n4) **W4 - Limited ablation and baselines**:\nWe have already evaluated our method on the 11-dataset cross-dataset benchmark commonly used in CLIP adaptation works in Figure 3(b) of the main paper, with detailed results provided in supplementary material C.3 Table 15. As requested, we provide additional ablations on this benchmark using the soft contrastive loss Lscont, its regularized variant, and memory-enhanced adaptation. As shown in the table below, $L_{s-cont}$ accounts for the vast majority of the overall performance\ngains over entropy minimization (TENT), and incorporating the regularization loss and the CCM memory further boosts performance. \n\n    | Method | Average Accuracy |\n    | :---- | :---: |\n    | CLIP | 64.8 |\n    | TENT | 65.1 |\n    |$L_{s-cont}$| 68.3 |\n    | $L_{s-cont}$ \\+ $L_{reg}$| 69.8 |\n    |$L_{s-cont}$ \\+  $L_{reg}$ \\+ $M$ | 69.9 |\n\n    Regarding the relevance of baselines such as TENT, we emphasize that the primary goal of this paper is to address the question: **What is the best loss function for gradient-based test-time adaptation of CLIP?** This motivates our evaluation protocol, which focuses on comparisons with both generic gradient-based methods like TENT and CLIP-specific gradient-based methods, including CLIPArTT [6] and WATT [7].\n\n    In addition, we include comparisons with other state-of-the-art CLIP-specific methods such as TPT [8] and TDA [9], as reported in Table 2 of the main paper. These results show that CLIPTTA outperforms all prior methods on most datasets. Notably, while TDA excels on the cross-dataset benchmark, it performs worse than the frozen CLIP model on others, such as VisDA-C, highlighting its lack of robustness across shifts. In contrast, CLIPTTA maintains strong performance across all evaluated datasets. Please see our response to Reviewer 7qJh (Q2 & W3) for a detailed breakdown of performance across different types of distribution shifts, and see our response to Reviewer JXSL (Q2) for additional comparisons with two other baselines (RPL[10] and BATCLIP [11]).\n\n5) **W5 - Logit scale parameter adaptation**:\nWe clarify that CLIP’s softmax temperature (logit scale) is not updated during test-time adaptation, in our method or any of the baselines evaluated in this paper. We will make this explicit in the final version.\n\n[1] Delving into Out-of-Distribution Detection with Vision-Language Representations. NeurIPS 2022\n\n[2] LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning. NeurIPS 2023\n\n[3] GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection. IJCV 2025\n\n[4] Negative Label Guided OOD Detection with Pretrained Vision-Language Models. ICLR 2024\n\n[5] Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR 2021\n\n[6] CLIPArTT: Adaptation of CLIP to New Domains at Test Time. WACV 2025\n\n[7] WATT: Weight Average Test Time Adaptation of CLIP. NeurIPS 2024\n\n[8] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. NeurIPS 2022\n\n[9] Efficient Test-Time Adaptation of Vision-Language Models. CVPR 2024\n\n[10] If your data distribution shifts, use self-learning. TMLR 2022\n\n[11] BATCLIP: Bimodal Online Test-Time Adaptation for CLIP, arxiv 2024"}, {"review_id": "x0SWdWS5Ec", "reviewer": "Reviewer_JXSL", "summary": "In this paper, the authors introduce CLIPTTA, a novel gradient-based test-time adaptation (TTA) method for vision-language models that employs a soft contrastive loss aligned with CLIP’s pre-training objective. They provide a theoretical analysis of CLIPTTA’s gradients, demonstrating how its batch-aware design enhances robustness against pseudo-label drift and class collapse. Extensive experiments validate the effectiveness of the proposed approach.", "questions": "1. The authors only consider CLIP in this paper, can the proposed method be applied to other similar vision-language models?\n\n2. How does the proposed method compare with RPL [1] and BATCLIP [2], which are both specifically designed for the same task?\n\n3. While the proposed method may help prevent convergence to dominant classes, it remains unclear how it facilitates recovery from poor pseudo-labeling. Further explanation is needed.\n\n\n[1] If your data distribution shifts, use self-learning\n\n[2] BATCLIP: Bimodal Online Test-Time Adaptation for CLIP", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea to leverage soft contrastive image-text loss for adapting CLIP is well-motivated.\n\n2. The method is clearly explained and the authors include sufficient analysis to show the benefits of the proposed method.\n\n\nWeaknesses:\n\n1. Several concepts are introduced without sufficient explanation—for example, gradient dynamics, pseudo-label drift, and natural continuity.\n\n2. It is unclear how the proposed loss function helps minimize the domain gap during test-time adaptation.\n\n3. The proposed method relies on a batch of test samples and cannot be applied to a single test sample.", "was_revised": true, "final_justification": "My concerns are addressed and I keep my current rating.", "rebuttal": "We thank the reviewer for their valuable feedback and insightful questions. Below, we provide detailed responses to each concern raised.\n\n1) **Q1 - Extension to other similar vision-language models**:\nThe reviewer rightly asks whether our method can generalize to other VLMs beyond CLIP.  We extended our experiments during the rebuttal period to assess the generality of CLIPTTA beyond the default ViT-B/16 backbone. Specifically, we evaluated the method on three additional CLIP architectures (ResNet50, ViT-B/32, and ViT-L/14), as well as another contrastive vision-language backbone (SigLIP-ViT-B/16). \n\n    | Backbone         | Dataset    | Base  | TENT  | CLIPTTA (ours) |\n    |------------------|------------|:-----:|:-----:|:-------------------:|\n    | CLIP-ResNet50     | ImageNet   | 58.2  | 58.0  | **59.8**            |\n    |                  | CIFAR-10   | 68.7  | 72.2  | **84.0**            |\n    |                  | CIFAR-100  | 40.6  | 44.7  | **55.8**            |\n    | CLIP-ViT-B/32     | ImageNet   | 62.0  | 61.4  | **68.2**            |\n    |                  | CIFAR-10   | 88.7  | 93.2  | **93.4**            |\n    |                  | CIFAR-100  | 64.0  | 69.6  | **72.5**            |\n    | CLIP-ViT-L/14     | ImageNet   | 73.5  | 74.2  | **74.6**            |\n    |                  | CIFAR-10   | 95.4  | 97.3  | **97.5**            |\n    |                  | CIFAR-100  | 75.9  | 82.1  | **82.4**            |\n    | SigLIP-ViT-B/16   | ImageNet   | 75.7  | 75.5  | **75.9**            |\n    |                  | CIFAR-10   | 92.5  | **96.0**   | **96.0**            |\n    |                  | CIFAR-100  | 70.9  | 78.2  | **78.7**            |\n\n    Overall, CLIPTTA consistently outperforms both the base model and TENT across all architectures and datasets. For instance, on CLIP-ResNet50, CLIPTTA improves performance over TENT by +12 points on CIFAR-10 and +11.1 points on CIFAR-100, demonstrating its effectiveness across diverse architectures and training paradigms. On the other hand, performance gains appear more modest on SigLIP, with saturation observed on some datasets such as CIFAR-10, where both TENT and CLIPTTA reach 96.0%. We anticipate, however, that improvements could be more pronounced on more challenging benchmarks, such as corrupted datasets with lower zero-shot accuracy. This remains an avenue we were unable to explore within the rebuttal timeline.\n\n    These results confirm that our soft contrastive objective’s effectiveness is not tied to a specific backbone or contrastive formulation. We emphasize, however, that these are preliminary results obtained using a shared hyperparameter setting across all models. We anticipate that further tuning—such as adjusting the learning rate for each backbone—could lead to improved performance. Moreover, adapting the adaptation loss to better align with SigLIP’s pre-training objective could further improve performance and is a promising direction for future work.\n\n\n2) **Q2 - Comparison with RPL[1] and BATCLIP[2]**:\nWe thank the reviewer for pointing us to the related works RPL [1] and BATCLIP [2]. Both address the same task—test-time adaptation of VLMs—and extend entropy minimization (as used in TENT) with mechanisms to mitigate its limitations. RPL leverages pseudo-labels and uses a generalized cross-entropy loss to enhance robustness, while BATCLIP adapts both visual and textual encoders using two auxiliary objectives: a projection matching loss to align visual prototypes with text features, and an inter-class separability loss to enforce distinctiveness. In this sense, both remain conceptually close to TENT but aim to address its failure modes.\n\n    In terms of performance, CLIPTTA significantly outperforms both RPL and BATCLIP on CIFAR-10-C, CIFAR-100-C, and ImageNet-C (see table below). These results (from the BATCLIP paper, using ViT-B/16) follow the same setting as ours and enable direct comparison. We will include these comparisons in the final version to highlight the advantages of our simpler soft contrastive loss.\n\n    | Dataset      | CLIP    | RPL   | BATCLIP | CLIPTTA (ours) |\n    |---------------|:-----:|:-----:|:-------:|:----------------:|\n    | CIFAR-10-C    | 60.2| 61.5 | 73.9   | **80.7**         |\n    | CIFAR-100-C   | 35.2| 38.5 | 42.1   | **52.6**         |\n    | ImageNet-C   | 25.5 | 25.1 | 30.7   | **41.1**         |\n\n3) **Q3 & W2 - On CLIPTTA recovery from poor pseudo-labelling**:\nThe ability to recover from poor pseudo-labeling is a subtle yet critical aspect of CLIPTTA's robustness. Unlike methods with explicit correction mechanisms, CLIPTTA relies on the structure of its soft contrastive loss—specifically the $\\beta_{i,j}$ coefficients in Eq. (5)—to guide updates.\n\n    This can be understood by contrasting it with TENT: In TENT, the gradient is computed independently for each sample and always reinforces its own current prediction, whether correct or not. Once a sample is misclassified, updates tend to push it further in the wrong direction.\n\n    In contrast, CLIPTTA uses a batch-aware loss where each sample’s gradient is influenced by its similarity to others in the batch via the $\\beta_{i,j}$ terms:\n\n    $\\beta_{i,j} = p(\\hat{t}_j \\mid \\mathbf{x}_i) \\left(1 + \\log p(\\hat{t}_j \\mid \\mathbf{x}_i)\\right)$,\n\n    where $\\hat{t}\\_{j}$ is the pseudo-caption of sample $x_j$, and $p(\\hat{t}\\_{j} | x_{i})$ is the probability that $x_i$ is assigned to $\\hat{t}_{j}$.  These coefficients are large when the probability $p(\\hat{t}\\_j \\mid \\mathbf{x}\\_i) $ that $x_i$ and $x_j$ share the same pseudo-label is high, which typically happens when the two samples are semantically similar. Consider three test samples:\n    - $x_1$ is misclassified as class $c’$.\n    - $x_2$ is confidently and correctly classified as class $c$ (the true class of $x_1$).\n    - $x_3$ is unrelated.\n\n    In TENT, $x_1$ is pulled toward $c’$, reinforcing the mistake. In CLIPTTA, the loss considers the pair ($x_1$, $x_2$). Since $x_1$ assigns a high score to the pseudo-caption of $x_2$, the coefficient $\\beta_{1,2}$ is large. This pulls the gradient update for $x_1$ toward the correct class $c$, enabling implicit correction. The unrelated $x_3$ contributes little due to a small $\\beta_{1,3}$.\n\n    This behavior is discussed in section 3.2 of the main paper and Appendix A and B. The key idea is that CLIPTTA allows each sample to benefit from other confident and semantically aligned predictions in the batch. This reduces the risk of reinforcing early pseudo-labeling errors.\n    We hope this additional explanation and example help clarify the role of the $\\beta_{i,j}$ coefficients in allowing the model to recover from wrong pseudo-labels.\n\n4) **W1 - Clarification of key concepts**:\nThe notions of gradient dynamics (L.31), pseudo-label drift (L.37), and natural continuity (L.48) are meant to provide intuition about the behavior of TTA in the context of CLIP and motivate our loss design. To clarify:\n    - **Gradient dynamics** refer to the direction and magnitude of updates during optimization. When the adaptation loss deviates significantly from the pre-training objective, it may lead to misaligned gradients (in direction and/or magnitude) and suboptimal updates. \n    - **Pseudo-label drift** describes the phenomenon where early adaptation errors (i.e., incorrect pseudo-labels) reinforce themselves over time, gradually pushing the model toward biased or incorrect predictions.\n    - **Natural continuity in adaptation** refers to designing the adaptation process to behave as similarly as possible to the pre-training phase. By aligning the loss structure, we promote stable updates.\n\n5) **W3 - CLIPTTA on a single test sample**:\nAs rightfully noted by the reviewer, the soft contrastive objective $L_{\\text{s-cont}}$ requires at least two samples per batch to perform meaningful updates. When the batch size is 1, both the loss and its gradient vanish, resulting in no adaptation. However, as detailed in Section C.4 of the supplementary material, CLIPTTA remains applicable in this setting by leveraging the Class-wise Confident Memory (CCM), which stores past confident samples. This enables the computation of $L\\_{\\text{s-cont}}$ even when only a single test example is present in the current batch.\n\n    We evaluate the performance of TENT, the standalone soft contrastive loss ($L\\_{\\text{s-cont}}$), and CLIPTTA across various batch sizes on CIFAR-10. Results are shown in the table below. This comparison highlights that:\n\n    - TENT performs poorly at batch size 1 (40.3%), as the only mechanism it can rely on to prevent reinforcing incorrect predictions in low-batch regimes is through gradient averaging over the batch samples.\n    - The standalone $L_{\\text{s-cont}}$ is equivalent to CLIP at batch size 1 (since the loss vanishes), but quickly improves as batch size increases.\n    - CLIPTTA achieves strong performance even with a single test sample (93.4%), thanks to the CCM. At batch size 2, it already surpasses both TENT and $L_{\\text{s-cont}}$, and shows only marginal gains beyond batch size 64, indicating robustness to the batch size.\n\n    | Batch size |  1    | 2    | 8    |  16 |  32   | 64   | 128  | 256  | 512  |\n    |:------------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n    | TENT   |  40.3 |  89.8 | 92.6 | 94.6 | 94.4 | 94.7 |94.9| 94.7 | 94.6 |\n    | $L_{s-cont}$    |  89.3 |  90.1 | 93.6 |  94.7 |  94.9 |  94.9 |  95.0 | 94.9|  94.9| \n    | CLIPTTA   | 93.4 | 94.7 | 94.7  | 94.8 | 94.8 | 95.0 | 95.1 | 95.1 | 95.2 |\n\n    In addition, we compare to TPT and TDA, two recent CLIP-specific TTA methods designed for batch size 1. On CIFAR-10, they achieve 89.8% and 91.4% respectively—substantially below CLIPTTA’s 93.4%, confirming the effectiveness of our memory-based strategy in this regime.\n\n[1] If your data distribution shifts, use self-learning. TMLR 2022\n\n[2] BATCLIP: Bimodal Online Test-Time Adaptation for CLIP, arxiv 2024"}, {"review_id": "HLIPAiNQUa", "reviewer": "Reviewer_26qe", "summary": "This paper tackles test-time adaptation (TTA) for vision-language models (e.g. CLIP). The authors hypothesize that the classic entropy minimization objective is misaligned with the contrastive training paradigm of CLIP. To address this, they propose a batch-aware contrastive loss for TTA in vision-language settings. Besides, they introduce an outlier contrastive exposure loss to enhance outlier detection in open-set scenarios. Extensive evaluation on 75 datasets verifies robustness and effectiveness of the proposed method for TTA.", "questions": "1. How does the number of samples in a batch affect the performance of the proposed contrastive loss for TTA (compare with entropy minimization loss such as TENT)?\n2. Though classic models trained with cross-entropy cannot compute the text-to-image entropy (the second term in eq. 3), the image-to-text entropy (the first term in eq. 3) could still be used. What’s the performance when applying the proposed contrastive loss to such classic models? How does it compare to traditional entropy minimization in this context?\n3. In Fig. 4 (b), futher ablations such as (1) TENT + $\\mathcal{L}\\_{\\text{reg}}$, and (2) CLIPTTA w/o $\\mathcal{L}_{\\text{reg}}$ would be helpful to verify the individual contribution of the contrastive loss in preventing class collapse.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The idea for extending classic entropy minimization to better align with CLIP’s contrastive training is well motivated. The proposed batch-aware contrastive loss is conceptually clear and easy to follow.\n- The empirical evaluation is thorough and extensive, spanning a large number of datasets. The improvement shows solid improvements, especially under data corruption, where baseline solutions often fail to adapt.\n\n**Weaknesses**\n1. The performance of the proposed batch-aware contrastive loss seems highly depends on batch size (i.e., the number of samples in a batch). However, the effect of batch size on adaptation performance is not studied and discussed.\n2. While the authors hypothesize that the misalignment between pre-training and adapting objective is crucial for TTA, this claim remains untested. For example, the performance of applying a similar contrastive loss to classic cross-entropy pre-trained models (e.g., ResNet pre-trained on ImageNet) is unclear. Based on the authors’ hypothesis, will classic entropy minimization techniques outperform the proposed contrastive loss for classic models?\n3. Fig. 4b suggests that CLIPTTA avoids class collapse by preserving prediction diversity. However, the regularization term ($\\mathcal{L}_{\\text{reg}}$), which explicitly maximizes entropy in a batch, likely plays a major role in this effect. It is unclear how much of the improvement in preventing class collapse is due to the proposed contrastive loss itself vs the regularization.", "was_revised": true, "final_justification": "My concerns regarding batch size & the analysis of class collapse have been addressed. As for the mismatch between training & adaptation losses, I encourage the authors to refine the motivation to avoid potential misinterpretation. Given most of the issues have been addressed, I will maintain my score, and look forward to improvements in the final version.", "rebuttal": "We appreciate the reviewer's comments and suggestions, which will help enhance the quality of our work. \n\n1. __Q1 - Effect of batch size in adaptation:__ \nAs rightfully noted by the reviewer, the soft contrastive objective $L_{\\text{s-cont}}$ requires at least two samples per batch to perform meaningful updates. When the batch size is 1, both the loss and its gradient vanish, resulting in no adaptation. However, as detailed in Section C.4 of the supplementary material, CLIPTTA remains applicable in this setting by leveraging the Class-wise Confident Memory (CCM), which stores past confident samples. This enables the computation of $L\\_{\\text{s-cont}}$ even when only a single test example is present in the current batch.\n\n    We evaluate the performance of TENT, the standalone soft contrastive loss ($L\\_{\\text{s-cont}}$), and CLIPTTA across various batch sizes on CIFAR-10. Results are shown in the table below. This comparison highlights that:\n\n    - TENT performs poorly at batch size 1 (40.3%), as the only mechanism it can rely on to prevent reinforcing incorrect predictions in low-batch regimes is through gradient averaging over the batch samples.\n    - The standalone $L_{\\text{s-cont}}$ is equivalent to CLIP at batch size 1 (since the loss vanishes), but quickly improves as batch size increases.\n    - CLIPTTA achieves strong performance even with a single test sample (93.4%), thanks to the CCM. At batch size 2, it already surpasses both TENT and $L_{\\text{s-cont}}$, and shows only marginal gains beyond batch size 64, indicating robustness to the batch size.\n\n    | Batch size |  1    | 2    | 8    |  16 |  32   | 64   | 128  | 256  | 512  |\n    |:------------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n    | TENT   |  40.3 |  89.8 | 92.6 | 94.6 | 94.4 | 94.7 |94.9| 94.7 | 94.6 |\n    | $L\\_{s-cont}$    |  89.3 |  90.1 | 93.6 |  94.7 |  94.9 |  94.9 |  95.0 | 94.9|  94.9| \n    | CLIPTTA   | 93.4 | 94.7 | 94.7  | 94.8 | 94.8 | 95.0 | 95.1 | 95.1 | 95.2 |\n\n    In addition, we compare to TPT and TDA, two recent CLIP-specific TTA methods designed for batch size 1. On CIFAR-10, they achieve 89.8% and 91.4% respectively—substantially below CLIPTTA’s 93.4%, confirming the effectiveness of our memory-based strategy in this regime.\n\n\n2. __Q2 - Our soft contrastive loss on cross-entropy models:__\nWe appreciate the reviewer’s perspective and would like to clarify that our intent was not to suggest that the adaptation loss should necessarily mirror the pre-training objective in all cases. Rather, our motivation for the soft image-text contrastive loss $L_{\\text{s-cont}}$ stems from the specific structure of vision-language models (VLMs) like CLIP. This design choice is further supported by insights from the FLYP [1] study, which shows that in few-shot adaptation settings, using a contrastive objective aligned with CLIP’s pre-training yields stronger performance than standard cross-entropy. In our case, the proposed batch-aware adaptation offers two key advantages for CLIP: aligning with the pre-training objective and mitigating pseudo-label errors.\n\n    We agree with the reviewer that, while text-to-image entropy (the second term in Eq. 3) cannot be computed for classic models trained with cross-entropy, the image-to-text entropy (first term in Eq. 3) remains applicable and could be used for adaptation. To fulfill the reviewer’s request, we conducted a preliminary experiment using a ResNet-50 model pre-trained on ImageNet with standard cross-entropy. We compare test-time adaptation using TENT and our proposed $L\\_{\\text{s-cont}}$, on the ImageNet-C benchmark with Gaussian noise corruption:\n    | Dataset       | ImageNet-C (Gaussian noise) |\n    |:------:|:---------------------------:|\n    | Base    |            20.9             |\n    | TENT    |            21.0             |\n    | $L_{s-cont}$   |            25.0     |\n\n    We observe that $L_{\\text{s-cont}}$ yields a +4.1pt gain over the base model, while TENT provides only a marginal +0.1pt improvement. This suggests that the robustness properties of our batch-aware loss—particularly its ability to mitigate pseudo-label drift—may also benefit test-time adaptation of non-VLM models.\n    That said, our main experiments on the same dataset using a CLIP-ViT-B/16 backbone show a much larger improvement: from 13.0% (CLIP base) to 29.0% with CLIPTTA (+16pt). This suggests that part of the gain comes from better robustness, but a substantial component stems from the alignment between the adaptation loss and CLIP’s pre-training objective.\n\n3. __Q3 - Additional ablations on collapse analysis:__ \nTo fulfill the reviewer’s request, we conducted an additional ablation study to isolate the contribution of the regularization loss $L_{reg}$ in preventing class collapse during adaptation. The table below reports the evolution of the entropy of the predictions on CIFAR-10-C across batches for: (1) TENT with and without $L_{reg}$, and (2) CLIPTTA with and without $L_{reg}$.\n\n    | Method                                | Batch 0 | Batch 20 | Batch 40 | Batch 60 |\n    |---------------------------------------|:-------:|:--------:|:--------:|:--------:|\n    | TENT                                  | 2.19    | 1.75     | 1.44     | 1.19     |\n    | TENT + $L_{\\text{reg}}$               | 2.18    | 2.24     | 2.26     | 2.25     |\n    | CLIPTTA w/o $L_{\\text{reg}}$ ($L_{s\\text{-}cont}$) | 2.18    | 2.26     | 2.26     | 2.26     |\n    | CLIPTTA ($L_{s\\text{-}cont}$ + $L_{\\text{reg}}$)   | **2.20**    | **2.27**     | **2.28**     | **2.29**     |\n\n    We agree that the regularization loss $L_{reg}$ plays a beneficial role in preventing class collapse by promoting prediction diversity. However, the results also show that $L_{s\\text{-cont}}$ alone (i.e., CLIPTTA w/o $L_{reg}$) achieves nearly identical entropy behavior to TENT + $L_{reg}$, highlighting the inherent robustness of the soft contrastive loss to class collapse. Finally, combining $L_{s\\text{-cont}}$ with $L_{reg}$ further improves entropy in predictions, reaching values close to the theoretical maximum (2.3) for CIFAR-10, and confirming their complementarity.\n\n[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR 2023"}]

Author rebuttal:

Meta-review: This paper proposes a test-time adaptation framework for vision-language models like CLIP, termed CLIPTTA. They show the effectiveness of the method for both closed and open-set settings. The reviewers appreciated the clear writing, problem addressed (ID and OOD) and extensive evaluation. Based on the overall contributions and the positive reviews, the recommendation is to accept the paper. The authors are strongly encouraged to address the remaining concerns regarding explaining the limitations of the work, etc. in the final version.
Decision: Accept (poster)