Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions
Authors: Wenxuan Bao
Ruxi Deng
Jingrui He
Venue: NeurIPS 2025 poster ()
Keywords: vision-language models
test-time adaptation
Primary area: deep_learning

Abstract: Pretrained vision-language models such as CLIP achieve strong zero-shot generalization but remain vulnerable to distribution shifts caused by input corruptions. In this work, we investigate how corruptions affect CLIP’s image embeddings and uncover a consistent phenomenon we term as embedding variance collapse, where both intra-class and inter-class variances shrink as corruption severity increases. We find that this collapse is closely tied to performance degradation, with inter-class variance strongly correlated with classification accuracy. To explain this phenomenon, we analyze how corruptions alter the structure of the embedding space. Our theoretical results suggest that the visual encoder tends to encode corruption-related signals, which dilute class-discriminative features and compress the representation geometry. We further show that maximizing inter-class variance, even when estimated from pseudo-labels, can provably enhance embedding quality. Based on this insight, we propose Mint, a simple test-time adaptation method that maximizes pseudo-label-based inter-class variance on the fly using a mean accumulator and a gradient accumulator. Mint operates effectively with small batch sizes and consistently improves performance across multiple corruption benchmarks and CLIP architectures. Our code is available at https://github.com/baowenxuan/Mint.

Mint: A Simple Test-Time Adaptation of Vision- Language Models against Common Corruptions
Wenxuan Bao'-, Ruxi Deng" Jingrui He
University of Illinois Urbana-Champaign bobao4,ruxidd jingniing inois. edu
Abstract
Pretrained vision-language models such as CLIP achieve strong zero-shot gener- alization but remain vulnerable distribution shifts caused by input corruptions. In this work, we investigate how corruptions affect CLIP's image embeddings and uncover a consistent phenomenon we term as embedding variance collapse, where both intra-class and inter-class variances shrink as corruption severity in- creases. We find that this collapse is closely tied to performance degradation, with inter-class variance strongly correlated with classification accuracy. To ex- plain this phenomenon, we analyze how corruptions alter the structure of the embedding space. Our theoretical results suggest that the visual encoder tends to encode corruption- and compress the representation geometry. We further show that maximizing inter-class variance, even when estimated from pseudo-labels, can provably en- hance embedding quality. Based on this insight, we propose Mint, a simple test-time adaptation method that maximizes beeudo-label-bad inter-class vari- ance the fly using cumulative prototypes and gradient estimates. Mint operates effectively with small batch sizes and consistently improves performance across multiple corruption benchmarks and CLIP architectures. Our code is available https: //gg::///tttbb www.coom/////ceee hoom/baoxenxuan/Minn
signals, which dilute class-diseriminative features
1 Introduction
have demonstrated strong zero- However, their performance can degrade significantly under distribution shifts, such as common image corruptions [16]. Test- time adaptation (TTA) has emerged as a promising strategy for improving model robustness under distribution shifts. by adapting the model during test-time without accessing source data or target labels 24 40 38). This property makes TTA particularly suitable for the adaptation of pretrained VLMs, where the source training data is often large-scale, proprietary, unavailable at deployment.
Pretrained vision-language models (VLMs) such as CLIP 32
shot generalization across wide range of vision tasks 47 33 29
Most existing TTA methods for VLMs focus modifying the text prompt embedding to improve image-text alignment [35 12 I B1 25 36), or leveraging similarities between different image embeddings adapt the model prediction 45 19
While these approaches achieve strong performance on standard benchmark datasets, they often overlook a key issue: the quality of image significantly degrade under corruption. Some recent methods
embeddings themselves attempt to address this by adjusting the image encoder's 's normalization layers to align image-to-imags or text-to-text similarities. However, such techniques typically require large batches perform effective adaptation, making them unsuitable for many online TTA scenarios where only a few test samples are available at time. Furthermore, these methods offer limited insight into why common
[15] 28
Equul contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

corruptions cause accuracy degradation, and most lack theoretical analysis, making it difficult to understand when and why they succeed or fail.
In this work, we take a step back and ask: how exactly does corruption affect CLIP's image embeddings? To answer this, evaluate the intra-class and inter-class variances of the embeddings using ground-truth labels, referred to as GT-intra and GT-inter. respectively. This analysis reveals consistent and intriguing pattern: as corruption severity increases, both GT-intra and GT-inter variances consistently decrease. We term this phenomenon variance collapse, which implies that under corruptions, embeddings of images tend to become more similar, regardless of whether they belong to the same class or different classes. The phenomenon is illustrated in Figure [ Moreover, we observe strong correlation between inter-class variance and classification accuracy, suggesting that variance collapse is a key factor contributing to performance degradation.
To better understand and counteract variance collapse, we conduct a theoretical analysis of image embeddings" variances under distribution shifts. Our analysis shows that the simultane- ous reduction of GT-intra and GT-inter variances can be attributed to the visual encoder project- ing corruption-relatet related patterns into the embed- ding space, which dilutes class- diiss--iiiiiiaiiiee information. Our theoretical analysis further shows that even in the absence of ground-truth labels during adaptation, maximizing the inter- class variance computed from -seudo-labels (PL-inter) by updating the LayerNorm parame- ters can provably improve the quality of image embeddings and lead to more accurate classifi- cation.
Clea
Con
mages
Figure I: Variance collapse. Under corruptions, embeddings of images tend to become similar, re- gardless of whether they belong to the same class or different classes.
Motivated by this result, we design Mint, a simple test-time adaptation method that Maximizes the PL-inter variance on the fly. Mint is designed to operate reliably even when the batch size extremely small, which is common in online adaptation settings. To enable stable adaptation under such constraints, Mint incorporates two key components: a mean accumulator and gradient accumulator. The mean accumulator maintains cumulative averages of image embeddings for each pseudo-class and for the entire set of samples observed so far. This allows the estimation of PL-inter variance within each batch without requiring access to the full test set. In parallel, the gradient accumulator keeps track of the average update direction across batches, which reduces noise in parameter updates and improves adaptation stability. These two components together allow Mint to enhance diassdiicrrm signals and suppress corruption-related patterns in the image embedding space. We evaluate our method across a wide range of corruption benchmark datasets and CLIP architectures to demonstrate its robustness and generality. In all settings, Mint consistently outperforms existing TTA methods for VLMs, while also offering significant efficiency advantages. Our contributions are summarized follows:
We identify a phenomenon we refer to as variance collapse in CLIP image embeddings, where both intra-class and inter-class variances decrease corruption severity increases.
We provide theoretical analysis that attributes this collapse to the visual encoder embedding corruption-related patterns, and show that maximizing Pl--inter can improve embedding quality.
We propose Mint, simple TTA method that maximizes PL-inter on the fly using mean accumulator and a gradient accumulator, enabling effective adaptation even with extremely small batch sizes.
We demonstrate that Mint consistently improves the performance of CLIP models across multiple corruption benchmarks and architectures, outperforming existing TTA methods in both accuracy and efficiency.
2

---

2 Related works
Test-time adaptation (TTA) adapts a source model to an unlabeled target domain during testing, without access to source data, making it suitable for pre-trained VLMs like CLIP. Early TTA methods
for CLIP focus on modifying the text encoder or embedding, via prompt tuning 35
prompt store embeddings of high-confidence samples and use image similarity to guide predictions. Other training-free free methods
12 34
weighting M. or ensembling 44 BT] Memory-based approaches [9
8 11 apply augmentations and confidence selection to enhance robustness without updating any model parameters. Although effective, most methods overlook key issue: the quality of image embeddings degrades under corruptions. Recent approaches [15] 28] attempt to mitigate this by adjusting normalization layers, aligning the image-image and text-text similarities. However, they are very sensitive to batch size and introduce high computational overhead, limiting their use in online
TTA settings.
Inter-class separability has been widely explored in supervised learning, where a common goal is to increase the distance between classes while reducing the variance within each class. A classic example is the Fisher score [TO]. which measures this separation and has been used for feature selection [14] and to improve domain adaptation by applying the Fisher criterion on the labeled source domains [46.. However, computing such metrics typically requires access to ground-truth labels, which are unavailable in test-time adaptation. More recently, Matcha [5] extended similar ideas to graph-based TTA by leveraging soft pseudo-labels. While effective, this method assumes simultaneous access to all nodes in the test graph, making it unsuitable for online adaptation scenarios where only small batches are available.
We provide broader discussion of related works in Appendix A.1
3 Analysis
Preliminary CLIP 32 is a VLM consisting of an image encoder and a text encoder, which aligns images with their corresponding textual descriptions. Pretraining on a large-scale image-text dataset enables CLIP to perform zero-shot prediction. Specifically, for a classification task with C' classes, the text encoder embeds class descriptions (e-g.. "a photo of {class}") into normalized text embeddings T = [4,, |c]] E &xdd where d is the embedding dimension. Given a test image, the image encoder produces a normalize image embedding z; E TRá, and prediction is made via the maximum similarity score arg max, z, ty. However, CLIP's performance degrades noticeably under common image corruptions
28 15
as its image encoder was not explicitly trained for robustness.
3.1 Variance collapse
In this subsection, investigate how common corruptions affect the image embeddings extracted by CLIP's visual encoder, and how these changes influence classification accuracy. Motivated by Fisher score [10 46 [4] and contrastive learning [32 37 objectives, we posit that high-quality image embeddings should exhibit low intra-class variance (i.e., samples from the same class are close) and high inter-class variance (i.e., samples from different classes are well separated). To formalize this intuition, given a target dataset with C classes and N images, we define the following variances:
GT--total variance: VGT 1 DE
> 1 N 2i-1
Vie
GT-inter variance: yGT = 1 DC |
GT-intra variance: yGT
EC
N Die =
intra = C
Σ Vic
where - + z; is the average embedding for all images, 2.
EN EN Iie is the average Σn Wio
embedding of class e, and y: = [ya..... W/c]] E 1 is the one-hot ground-truth label of image i, where yir- = 1 if image corresponds to class c, and yic. = 0 otherwise. Note that these variances are computed using the ground-truth labels. To distinguish them from the pseudo-label-basee counterparts introduced later, we denote these metrics with GT- prefix (c.g.. GT-intra, GT-inter).
3

---

Coastian Volee
Cellccr 6
Comuptton
Mooe
Ekr
hord
1.10
50
Ausuneyy
Soettr Cerbret
Stow
0.20
3.15
30
23
0 98
0:05
GTiinte Variance
All types of variances decrease as the severity of corruption increases (severity=0 indi- cates original CIFAR-100 datasets without cor- ruptions).
Figure 2:
Figure 3: GT-inter variance is highly correlated with accuracy, with correlation 0.98. compari- son, the correlations between accuracy and GT- intra, GT-ttotal are 0.86 and 0.94, respectively.
We compute the GT-total, GT-inter, and GT-intra variances on the corruption benchmark 16 which includes 15 types of common corruptions, each evaluated 5 severity levels. Figure presents the results on four representative corruptions. Additional results across all corruption types and datasets included in Appendix C.1 We observe that for all types of corruptions, both GT--intra and GT-inter variances consistently decrease as the severity increases. This indicates that the pairwise similarity between image embeddings increases under corruptions, regardless of whether the images belong to the same class or not. We refer to this phenomenon as variance collapse, where the image embeddings become increasingly indistinguishable under stronger corruptions. Furthermore, we find that variance collapse is closely linked to the drop in accuracy. We computed all three variance metrics along with the corresponding classification accuracy under 76 corruption settings (15 corruption types x 5 severity levels, plus the clean setting) on CIFAR-100-4 100-C. As shown in Figure 3] the inter-class variance exhibits an extremely strong correlation with accuracy, indicating that this collapse could be a key factor driving the performance decline.
3.2 Theoretical explanation
In this subsection, we provide theoretical explanation for the emergence of variance collapse. For clarity, we consider balanced binary classification problem with E {0, Motivated by [39], we assume that each image can be mapped disentangled latent representation v; (uch, wit wahin mioiis] E IRd, composed of four components:
1. Task- ugt:: Semantic features that are directly predictive of the class label, ofh = 1t if y/s = and ogth p if y/ 0.
2. mask-irrelevant features "hTh Features unrelated to classification, such as background. is preserved during pretraining due to CLIP's general representation learning objective. We assume w.th Rademacher i.e., uniformly distributed in {-1, 1
3. Structured distribution shift uhhift Features representing systematic distribution changes in the target domain, such weather conditions digital transforms. We assume wi Ö, where indicates the severity of corruptions or distribution shifts.
4. Unstructured noise vi Random noise introduced by the corruption process. We assume oooise 8 Rademacher i.e., uniformly distributed in {-1, danee
Notice that by controlling the ratio of s, ||ll|| ||0 we can freely adjust the ratio for each component. Following the structure of CL.PP's visual encoder, we assume that the latent representation first passes through LayerNorm layer [ with linear transformation, followed by normalization to unit length. For analytical simplicity, we omit the demeaning step in LayerNorm and ignore the bias term in its parameters under which the image embedding can be formulated as
z; = normalize
① w
(1)
[x
'This simplitication is also known RMSNorm 41

---

where @ represents element-wise multplicatiion of vectors, w E R is the learnable weight of the LayerNorm layer, normalize()) denotes l normalization. For simplicity, we assume 1 at initialization, w is updated during the adaptation.
Theorem 3.1 (Variance collapse).
When the sample size - +00,
dur + 52 dnoose
Ver
VGT intra
(2)
11 + dirr + x2
2 + 82 dhoosee
11 der + 82 110112 + 82 dnaise
where s denotes the corruption severity. As increases, inter strictly decreases. In addition, intra also decreases when ||6||2 2 v //noise/ dive ||al||2-
Theorem 3.1 characterizes how the GT-inter and GT-intra variances change with increasing corruption severity. Combined with the empirical trends observed in Figure 2 this suggests that common corruptions often induce significant structured distribution shifts, reflected as large in the latent space. As result, the image encoder tends embed corruption-relatee patterns into the representation itself. This dilutes class-disriminntivee features and introduces bias into the resulting image embeddings.
3.3 Maximization of inter variance
Theorem 3.1 also supports that GT-inter variance has strong relevance to classification accuracy, as it reflects the proportion of task-relevant features within the overall feature representation. This insight motivates the idea that maximizing GT-inter variance should lead improved classification accuracy under distribution shifts. However, several challenges arise in the context of TTA. First, the ground-truth labels are unavailable, so we must rely on pseudo-labels, i.e., the model's own prediction, which are noisy due to distribution shifts. Second, model updates in TTA are typically restricted to small subset of parameters, such LayerNorm weights, for better efficiency. In this part, we show that even under these constraints, using only pseudo-labels and updating only LayerNorm parameters, maximizing inter-class variance remains an effective and theoretically justified strategy for improving robustness to distribution shifts.
Theorem 3.2 (Maximization of PL-inter variance). When the sample size - Dc.
C(Ej))
4a Ny 1
||
& w
[|2 + || O w
+ l noise @ uoise
DPL. 2,
2
wob
11
|wirr] 52. 110 w
122 +s²
where C(Ei,) (E
Byy Cov U.). oir Cov(e'r (.). and n
(1-Eai)2+
Cov(u K.). Furthermore, when oz
| 1 2 and oi adwaar
we have
ddor
(407) 1| 113) + 2 Byy
9² 32110112 + (40 Anoise - ||o 112)
V with PLL
C(Eji)
112
(1
dir + 32 (108 + dnoice))
(4)
PPL inter
V wiit
PPL inter
52 52 < O.
(5)
|| 1 + d + s2 [11812 + dmoise)
This implies that when we perform single gradient ascent step to maximize the PL-inter variance by updating the parameters of LayerNorm, the parameters associated with structured distribution shifts (i.e., Wahin) are necessarily suppressed. Furthermore, as long as the current prediction is reasonably accurate, meaning it depends more on task-relevant features than on task-irrelevant components or unstructured noise, maximizing PL-inter variance will increase the weights associated with task-relevant features (i.e., Wos). As result, this process reweighs the components in the final image embedding, enhancing the influence of task-relevan features while suppressing the effects of distribution shifts.
4 Proposed method: Mint
In this section, we introduce our proposed algorithm Mint, which maximizes the PL-inter variance on the fly. While the previous section provides theoretical justification that maximizing Pl--inter variance can improve test-time robustness, directly computing PL-inter variance requires access to the entire
5

---

8
88
Parp
Adaptation
Inference
Figure 4: Overview of Mint. Left: Adaptation phase. Given
test image batch, we compute the PL-inter variance with the help of a mean accumulator and maximize it via gradient ascent. A gradient accumulator aggregates update directions across batches to robustly update the LayerNorm parameters. Right: Inference phase. The mean accumulator is used to adjust the text embeddings, and final predictions are made based on the similarity between image and text embeddings.
test dataset. However, in the online TTA setting, the model typically adapts using only small batch or even single sample at a time. This leads to noisy and potentially biased gradient directions that deviate from the true optimization target. To address this, we reparameterize the PL-inter variance and employ both a mean accumulator and a gradient accumulator to aggregate information across batches, enabling more accurate approximation of the gradient in streaming setting. Figure 4 gives an overview of our method.
4.1 Mean accumulator
The most straightforward way to estimate PL-inter variance is to assume that the current batch fully represents the test data distribution and compute PL-inter using only samples in that batch. Under this approach, objectives across batches are computed independently. However, this naive strategy introduces significant noise, and even bias, into the PL-inter estimate. For instance, ImageNet-C contains 1,000 classes, but due deployment and memory constraints, the test-time batch size is typically limited to just few dozen samples. As result, most classes present in batch are represented by only single sample, causing their estimated class means degenerate into the samples themselves. In such cases, the distance between a sample and its class mean becomes zero, preventing us from estimating PL-intra variance. As a consequence, the computed objective ends up approximating PL-total variance rather than true PL-inter variance, which degrades adaptation performance under small batch sizes.
To address this issue, we first reparameterize PL-inter variance as the difference between PL-total and PI-intra variance. With detailed proof in Appendix B.I. this decomposition be written as:
1 C
Σ
21
N
Die
C
Die | žel12
(6)
Σ Dic
E Die
DPL
DTT.
where 2; is the embedding for i-th image, 1 DE is the global average embedding, IN EN, fic2: is the average embedding of all images predicted as class e by CLIP. This and E. = ΣN Dic
reformulation reveals that maximizing PL-inter is equivalent jointly maximizing PL-total variance and minimizing PL-intra variance, encouraging each embedding z; to move away from the global mean & and toward its corresponding class mean že, with the gradient direction approximately given by E --- - when the sample size sufficiently large. This insight suggests that more accurate estimates lead better gradient directions. Therefore, instead of estimating these means
of Σ and &. using only the current batch, we use a mean accumulator to maintain cumulative averages & and { Every time when observe new image with embedding z; and CLIP's prediction as pseudo-label,
K +1
K Kg. +
Zis Sg.
Kg, + Zi.
K
++11
where K is the total number of seen samples, and K3, is the number of seen samples with pseudo- label Gr. After replacing the class and global means in Equation (6) with the cumulative averages, the

---

final objective for the b-th batch By becomes
C₁ EiER, Vic zz E112
Ch
Σ EiEB, Die ||
Ecll2
Vitter (Bb]
(8)
Cb
EiEB, Vic
Cb
EiEB, Use
where C. the number of unique classes present in batch Bg.
4.2 Gradient accumulator
While the mean accumulator mitigates systematic bias in the objective by stabilizing the estimates of class and global means, does not eliminate the noise the individual gradient contributions from Z,, which are still computed over the current batch. To further reduce gradient estimation error, we introduce simple gradient accumulator that mimics adaptation with a larger effective batch size. Specifically, for the b-th batch, if the gradient computed the current batch is gb. maintain cumulative average gradients g over the seen batches:
b b-
g 9 gb.
and update the LayerNorm parameters in the direction of g. We perform only single step of update on each batch.
4.3 Adjust text embedding
In addition to estimating PL-inter variance, the cumulative class means can also be leveraged to adjust the text embeddings, thereby improving alignment between the image and text modalities. Motivated by prior works in TTA [18] and Bayesian estimation, we adopt simple, training-free approach to refine the text embeddings using accumulative embedding means. Specifically, maintain separate mean accumulator to store the image embeddings produced by the adapted image encoder. The refinement of text embeddings is given by
E. normalize
Kprioe + to
Kprior
Kprior + K R00) c=1....,C C,
K
(10)
where Kprior is hyperparameter controlling the strength of prior. This design enables dynamic adjustment of the text embedding. In the early stage of adaptation, the image embedding means may be less reliable, so we assign more weight to the original text embedding tc. As adaptation progresses and the quality of the estimated class-wis wise means &. improves, we gradually place more weight on E..
The final prediction is given by arg max,, 2, ty. After making prediction each batch, reset both the image encoder and the optimizer state to their initial values. However, the mean accumulator and gradient accumulator are preserved and carried over the next batch, allowing information aggregated from previous samples to guide the adaptation on subsequent inputs.
Experiments
In this section, we use experiments answer the following research questions
RQ1: Can Mint effectively improve the performance of CLIP models under common corruptions, especially in low batch size scenarios?
RQ2: Does Mint effectively mitigate the variance collapse?
RQ3: How efficient is Mint in terms of computational time?
Setup and baselines We test Mint with different combination of model architectures and corruption datasets [66]: ViT-B/32 [9] on CIFAR-10-C [21], ViT-B/16 on CIFAR-100-C and ViT-L/14 on ImageNet-C 77. all with corruption severity of 5. We consider standard TTA setting, where model is adapted to each type of corruption independently. We compare Mint with a wide range of existing TTA methods designed for VLMs. VTE g and Zero [ aggregate image embeddings
from multiple augmentations. TPT 35 and TPS 36 minimize the marginal entropy to encourage

---

Table 1: Mean accuracy (%) on Corruption benchmarks. Error bars are deferred to Appendix C.3
VIT-B/32 on CIFAR-1O-C
Method
Venue
Noise
Blur
Weather
Digital
Avg.
Gaass.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
ICML/21
35.5
40.0
43.2
70.0
41.4
64.5
70.2
70.8
72.3
66.7
81,4
64.5
59.6
48.2
56.7
59.0
Easemble
38.8
42.7
42.8
72.6
43.9
66.8
71.7
73.9
75.8
68.9
83.7
67.2
61.9
51.8
58.6
61.4
TPT 35
22urIPS'22
42.9
46.2
47.1
71.5
46.4
68.1
72.7
73.7
75.9
68.9
83.7
73,9
62.5
50.3
58.2
62.8
CVPR 24
41.2
44.1
43.3
73.9
45.1
68.1
73.6
74.0
76.7
69.6
84,0
66.6
62.3
54.7
58.4
62.4
DMN.7S
CVPR 24
37.6
41.5
42.5
69.4
43.8
65.9
70.5
70.2
71.2
64.0
80.7
58.6
59.4
54.9
58.1
59.2
VTE
8
ECCV-W"24
47.6
50.5
49.8
70.4
49.8
70.2
73.4
74.4
77.3
71.4
83.6
81.2
65.5
55.1
58.8
65.3
NcurIPS'24
74.5
80.6
WATT-S T-S XX
NeurIPS 24
53.2
54,9
50.7
75.0
35.4
71.1
74.8
75.4
77.0
72.7
84.2
73.1
63.4
61.1
62.3
61.1
TPS 36
W5CV
45.5
49.4
19.2
73.8
50.7
71.4
76.0
77.0
79.2
73.3
85.3
79.5
67.2
56.6
61.8
66.4
CLIPArTT
WACV
49,9
73.0
74.1
84.3
71.4
60.5
Mint
59.0
62.4
54.2
75.8
61.8
77.1
78.9
79.0
78.9
75.2
86.3
76.9
70.1
66.6
63.4
71.0
VIT-R/16 CIFAR-100-C
Method
Venue
Noise
Blur
Weather
Diginl
Aug.
Ganes.
Shot
Impul.
Defoc.
Glass
Mation
Zaom
Snow
Frost
Fog
Brii.
Contr.
Elustic
Pixel
JPBG
CLIP
ICME:21
19.7
214
25.3
42.5
20.2
43.1
48.0
48.4
49.7
41.7
57.0
34.5
29.2
23.9
32.4
35.8
Ensemble
22.9
24.3
29.6
43.6
20.1
43.7
48.7
48.9
50.3
41.8
58.1
35.2
29.2
26.3
33.6
37.1
TPT
NeurIP5'22
49.0
42.7
57.5
38.0
32.5
TDA
19
CVPR 24
23.8
26.0
32.5
45.7
21.5
44.4
50.5
49.5
51.5
42.8
59.2
36.8
29.7
28.1
34.3
38.4
DMN-ZS
45
CVPR'24
23.9
25.6
31.7
45.5
21.6
45.0
51.1
49.6
52.0
43.0
60.3
36.0
30.5
27.5
34.7
38.5
3
ECCV-W
20.2
39.9
18.5
47.6
43.
55.7
49.9
30.4
30.3
30.6
36.6
Zero
11
NeurIPS 24
19.9
21.5
29.6
40.4
18.5
39.6
44.8
47.8
48.3
43.3
55.8
50.0
30.6
30.4
30.7
36.8
WATES
2X
NeurIPS'24
27.5
36.4
47.5
26.8
46.8
51.6
52.3
46.6
61.0
43.5
34.3
35,9
37.3
41.9
TPS
25
22.6
24.4
31.0
44.0
20.1
43.6
49.0
50.5
51.3
44.3
59.1
45.1
30.6
28.8
53.8
38.6
CLIPArTT
15
WACV'25
24,9
27.1
32.5
47.4
23.4
47.2
$2.0
$1.6
52.5
46.5
61.2
41.2
33.7
32.6
37.0
40.7
Mint
29.4
30.8
38.6
50.7
27.1
49.9
55.5
$3.0
51.8
50.6
68.6
48.1
36.8
34,4
38.7
44.1
WT-L/14 on ImageNet-C
Method
Venue
Noise
Blur
Weather
Digital
Avg.
Gauss.
Shot
Impul.
Defoc:
Glass
Motion
Zoom
Snow
Frust
Tog
Bri,
Contr.
Elustic
Pixel
JPEG
CLIP 32
ICML'21
27.4
29.4
28.7
34.5
25.3
41.0
36.7
49.8
44.1
49.7
65,4
35.1
30.3
53.5
42.2
39.6
29.1
52.4
$2.7
67.8
34.5
32.4
56.2
TPT 35
NcurIPS/22
27.2
29.1
29.3
35.7
26.6
41.1
38.1
51A
46.3
51.6
67.7
39.4
32.1
55.9
45.5
41.1
TDA 19
CVPR 24
29.1
30.5
31.0
37.7
28.0
44.5
39.5
$3.4
47.8
53.6
68.3
36.8
33.3
56.7
44.4
42.3
DMN-ZS
45
CVPR'24
29.0
30.4
30.4
37.5
27.3
44.3
39.3
$2.5
46.7
$2.7
67.8
34,9
32.4
56.2
44.3
41.7
VTE 8
ECCV-W
23.2
26.5
24.9
34.5
25.8
39.7
38.2
49.0
45.7
49.8
67.0
44.4
32.1
55.8
46.5
40.2
7ero
NeurIPS
24.1
26.9
25.8
35.8
26.9
39.4
49.5
46.2
50.7
66.8
44.9
32.6
56.4
47.4
40.9
WATT-S 28
24
31.7
33.5
34.6
38.7
31.3
45.2
41.2
$2.7
47.8
54.5
67.5
42.9
34.8
56.1
45.9
43.9
TPS 30
WACV
28.9
31.0
30.7
37.8
28.0
43.4
40.8
53.3
47.9
53.5
69.2
438
33.3
57.1
47.0
43.1
CLIPArTI
W5CV'25
29.2
Mint
33.0
34,3
37.3
39.6
37.2
46.6
45.1
55.2
46.6
57.5
67.7
48.9
43.9
58.2
54.6
47.0
Table 2: Accuracy (mean + s.d. %) of Mint with various batch size.
Mint
Architacture
CLIP
BS
=
BS
BS
BS 20
BS 50
BS 100
BS
VIT.B/32
CIFAR-10.4
0.1
71.0
7..0
709
70.6
VIT-/16
CHIR--10ICC
35.8
43.1 0.1
43.1 43.1+ 0.1
43.3 0.1
43.6 0.1
44.1 +.1
44.+01
44. +
44.6 0.1
VIT-L/14
ImageNet-C
39.6
0,1
46.7 0.1
46.8 0.1
47.0 0.2
47.1
4688 47.0
consistency across augmented views. TDA 19 and DMN-ZS 45 leverage sample-wise similarity adjust predictions. WATT-S 28 and CLIPArTT 15 improve modality alignment by aligning mamage-to-image and text-to-text similarities. Unless otherwise specified, we use default batch size of 20 during adaptation. Mint uses Adam [20] optimizer with learning rate 0.007 for ViT-B models and 0.015 for ViT-L/14, and K Kprior = 10,000. Hyperparameter settings for baselines are provided in the Appendix
Main results (RQ1) The experimental results are summarized in Table We observe that training- free methods generally perform worse, as they do not update the model during adaptation. Among them, TPS achieves relatively strong performance by adjusting the text embeddings. CLIPArTT and WATT-S, which allow updates to the image encoder, perform best among the baselines. However, these methods do not share information across batches, which limits their overall effectiveness. Across all settings, Mint consistently improves accuracy and achieves the best performance. Compared to the strongest baselines, Mint yields absolute gains of 3.9%, 2.2%, and 3.1 %, respectively.
Robustness to batch size (RQ1) To evaluate the robustness of Mint under different test-time conditions, run it with batch sizes ranging from I to 200, using the same set of hyperparameters
8

---

33490.0 POD
dMbois bar
0Y98
Figure 5: Mint alleviates variance collapse.
No accumnllasrs
Mint CLIP
Keen sctumdeltrooi
75
Gradinn accumuleter only
Ensenbble
46
Accuracy
70
Accuncyy
44
42
60
40
20
100
Learning Rate
Kans (x20*)
Figure 6: Ablation study.
Figure 7: Hyperparameter sensitivity.
across all settings. As shown in Table 2 Mint consistently maintains strong performance across this range. Even in the extreme case of batch size I, achieves significant accuracy gains, demonstrating its effectiveness in highly constrained online adaptation scenarios.
Variance collapse (RQ2) We investigate the underlying mechanism of Mint by analyzing its effect on the image embeddings. Specifically, we evaluate the PL-inter variance, GT--inter variance, and classification accuracy CIFAR-100-C before and after adaptation, under four representative types of corruption (same as Figure 2 As shown in Figure [3] Mint successfully increases PL-inter variance by design, and this also leads clear improvement GT-inter variance. The increased GT-inter variance is accompanied by rise in accuracy, indicating that Mint effectively mitigates variance collapse. Additional results across all corruption types and datasets provided in Appendix C.4
Efficiency (RQ3) We compare the testing time of Mint with baseline algorithms CIFAR- 10----0------ by measuring the time required to process one corruption type (10,000 images). As shown in Table Mint runs substantially faster than other training-baser based TTA methods. This efficiency primarily stems from its simple design and the fact that it performs only a single model update per batch, unlike methods that require multiple iterative updates during adaptation. Notably, Mint is only slower than CLIP and other training-free free and augmentation free baselines.
Table 3: Comparison of testing time.
Method
Testing Time
Accuracy (%)
(%)
CLIP
21s
35,8
23m21x
VTF
9m45s
36.6
+0.8
Zero
9mn50s
TDA
33s
38.4
+2.6
DMN-2S
30s
38.5
+2.7
9m158s
CLIPAFIT
7md0ss
40.7
+4,9
NiDt
Im07s
44.1
+8.3
Ablation study To understand the individual contributions of the two accumulators in Mint, we perform ablation study CIFAR-10-C comparing the full method with the following variants: (1) Mean accumulator only, which removes the gradient accumulator; (2) Gradient accumulator only. which removes the mean accumulator; and (3) No accumulators, which disables both components. We observe in Figure 66 that both accumulators contribute to the performance of Mint, especially under small batch sizes. The mean accumulator essential for estimating PL-inter variance in extremely small batches, including the batch size of 1. Without it, gradients cannot be computed when the batch contains only single class instance, rendering adaptation ineffective. Meanwhile, the gradient accumulator improves adaptation quality by reducing the noise in gradient estimates across batches. Overall, Mint exhibits the strongest robustness and performance when both accumulators are used, validating the necessity of their complementary roles in the online test-time adaptation setting. Additionally, we explore adapting different layers in the visual encoder and find that updating all LayerNorm layers yields the best performance (sce Appendix C.5

---

Hyperparameter sensitivity We study the sensitivity of Mint to its two hyperparameters: the learning rate and the prior strength K prior- across three datasets. Results on ImageNet-C are shown in Figure 7 with results on CIFAR-10-C and CIFAR- 10FAR-100-C included in Appendix C.6 We observe that Mint remains stable across broad range of hyperparameter values, without requiring precise tuning. In particular, we find that learning rate of 0.009 and prior size of K prior = 10,000 consistently perform well across different datasets and architectures, demonstrating the robustness and generality of the method.
Additional experiments We further evaluate Mint on clean datasets (uncorrupted CIFAR-10 10, CIFAR-100, and ImageNet), ImageNet variants (ImageNet-A, -V2, -R, and -Sketch), and corruption benchmarks under the mixture-of-domet in Appendix C.7 C.8 and C.9 Mint demonstrates consistently strong performance across these
setting [27]. The corresponding results are provided
scenarios, confirming its broad applicability.
6 Conclusion
In this work, we identify variance collapse in image embeddings as a key factor behind CLIP's per- formance degradation under corruptions. Through theoretical analysis, we attribute this phenomenon to the image encoder encoding corruption-eelteed patterns, which dilutes flass-dscrrminatte signals. We further show that maximizing inter-class variance, even when computed using pseudo labels, can provably enhance performance. Based on this insight, we propose Mint, simple yet effective test-time adaptation method. Mint leverages cumulative mean and gradient accumulators to oper- ate robustly in low-batch-sizze online settings. Extensive experiments on corruption benchmarks demonstrate its strong performance and efficiency.
Acknowledgments and Disclosure of Funding
This work is supported by National Science Foundation under Award No. IIS-2416070, IIS-2117902. The views and conclusions are those of the authors and should not be interpreted as representing the official policies the funding agencies or the government.
References
[1] James Urquhart Allingham, Jie Ren, Michael W. Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 547-568. PMLR, 2023.
[2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/ abs/1607.06455 2016.
[3] Wenxuan Bao, Ruxi Deng, Ruizhong Qiu, Tianxin Wei, Hanghang Tong, and Jingrui He. Latte: Collaborative test-time adaptation of vision-language models in federated learning. In IEEF/CVF International Conference Computer Vision, ICCV 2025, Honolulu, Hawaii, USA, October 19-23, 2025. IEEE, 2025.
[4] Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. Adaptive test-time personalization for federated learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[5] Wenxuan Bao, Zhichen Zeng, Zhining Liu, Hanghang Tong, and Jingrui He. Matcha: Mitigating graph structure shifts with test-time adaptation. In The Thirteenth International Conference on Learning Representations, 2025.
[6] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yer Kuo, Sizhuo Ma, Jian Wang. Robustsam: Segment anything robustly on degraded images. In JEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 4081-4091. IEFE, 2024.
10

---

[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society, 2009.
[8] Mario Döbler, Robert A. Marsden, Tobias Raichle, and Bin Yang. A lost opportunity for vision-language models: A comparative study of online test-time adaptation for vision-language models. CoRR, abs/2405. 1007/ 2024.
[9]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021.
[10] Richard O. Duda, Peter E. Hart, and David G. Stork. Pattern classification, 2nd Edition. Wiley, 2001.
[11] Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustrat- ingly easy test-time adaptation of vision-language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024.
[12] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmenta- tion with diffusions for effective test-time prompt tuning. In /EEF/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2704-2714, 14. IEEE, 2023.
[13] Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation to test-time corruption. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 1 17-24, 2023, pages 11786-11 1166. IEEE, 2023.
[14] Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized fisher score for feature selection. In UAI 2011, Proceedings of the twenty-Sevnntt Conference on Uncertainty in Artificial Intelligence, Barcelona, Spain, July 14-17, 201 pages 266-273, AUAI Press, 2011.
[15] Gustavo Adolfo Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. Clipartt: Adaptation of CLIP new domains at test time. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025, Tucson, AZ, USA, February 26- - March 6, 2025, pages 7092- 7101. IEER, 2025.
[16] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In 7th International Conference on Learning Representations, ICIR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[17] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448 456. JMLR 2015.
[18] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-1,, 2021, virtual, pages 2427-2440, 2021.
[19] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El-Saddik, and Eric P. Xing. Efficient test-time adaptation of vision- lasion-angguag models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16- 22, 2024, pages 14162- 14171. IEEE, 2024.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, San Diego. CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

[22] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang. and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In The Twelfth International Conference on Learning Representations, ICI. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[23] Yoonho Lee, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[24] Jian Liang, Ran Hc, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, pages 1 1-34, 2024.
[25] Xiaosong Ma, Jie Zhang, Song Guo, and Wennhho Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[26] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Confer- ence on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16888-16908 PMLR, 2022.
[27] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang. Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[28] David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. WATT: weight average test time adaptation of CLIP. In Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
[29] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-On -r,, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 2065-2074. IEEE, 2021.
[30] Priyank Pathak, Shyam Marjit, Shruti Vyas, and Yogesh S Rawat. LRO.FM: LOW- RESOLUTION ZERO-SHOT CLASSIFICATION BENCHMARK FOR FOUNDATION MOD- ELS. In The Thirteenth International Conference on Learning Representations, 2025.
[31] Sarah M. Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating eustomized prompts for zero-shot image classification. In IEEF/CV International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 15645- 15655. IEEE, 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021.
[33] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language- guided dense prediction with context-aware prompting. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18061-18070. IEEE, 2022.
Language-guided
[34] Jameel Abdul Samadh, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Shahbaz Khan, and Salman H. Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
[35] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in lanoualanguagge models.
12

---

In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28: December 9, 2022, 2022.
[36] Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2025, Tucson, AZ, USA, February 26 March 6, 2025, pages 825-835. IEEE, 2025.
[37] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807 hal///007.3748, 2018.
[38] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[39] Olivia Wiles, Sven Gowal, Florian Stimberg. Sylvestre-Alvee Rebuffi, Ira Ktena, Krishnamurthy Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[40] Zehao Xiao and Cees G. M. Snoek. Beyond model adaptation at test time: A survey. CoRR, abs/2411.03688 2024.
[41] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12360- 12371, 2019.
[42] Ce Zhang. Simon Stepputtis, Katia P. Sycara, and Yaqi Xie. Dual prototype evolving for test- time generalization of vision-language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canuda, December 10 15, 2024, 2024.
[43] Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
[44] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, volume 13695 of Lecture Notes in Computer Science, pages 493-510. Springer, 2022.
[45] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision- laniuage models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 287 18718-28728 28728. IEEE, 2024.
[46] Yinghua Zhang, Yu Zhang, Ying Wei, Kun Bai, Yangqiu Song, and Qiang Yang. Fisher deep domain adaptation. In Proceedings of the 2020 SIAM International Conference Data Mining, SDM 2020, Cincinnati, Ohio, USA, May 7-9, 2020, pages 469-477. SIAM, 2020.
[47] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region- based tanguage-image pretraining. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pages 16772-16782. IEEE, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we claim that this paper focuses on test-time adaptation of vision-language models. Contributions are clearly listed at the end of the introduction and in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix A.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We clearly state the main assumptions in Section 3 Formal statements and proof's are given in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the paper, we clearly specified the methods we used obtain the experi- mental results and all the hyperparameters used in the process, which can fully support the reproducibility of the experiment. More details are provided Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We use open-source datasets, and provide the code in the supplemental
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcccceooooooop http:/guuuddeeeoeSSuubbbiiiii wwwninsssonpppo ssinoolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide key information in Section 5 and other details in Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report results with mean and standard deviation from five independent runs with different random seeds. Notice that for space limits, we provide the error bar of Table in Appendix C.3 instead.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix C.2.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We strictly adhere the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix A.3
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release new data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in the paper are properly credited, and their licenses and terms of use have been explicitly mentioned and respected if provided in the original paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neurr /ttpa/rrps.cc/ooofeerrres/////// cc//oonferences/2225//LL. for what should or should be described.
20

---

Appendix
Contents
Discussion
22
A.I Additional related works
22
A.2 Limitations
22
A.3 Broader impacts
22
Theoretical analysis
23
B.
Variance decomposition
23
B.2
Theoretical setup
24
B.3
Change of variances under corruption
25
B.4
Adaptation
26
Experiments
28
Effect of corruptions
28
Experiment details
31
C.3
Full results for RQI
32
C,4
Full results for RQ2
32
C.5
Ablation on layers to adapl
34
C.6
Hyperparameter sensitivity
34
Experiments on clean datasets
35
C88
Experiments on ImageNet variants
35
C.9
Mixture corruption datasets
35
21

---

A Discussion
A.1 Additional related works
In this subsection, we discuss additional related work on general test-time adaptation. Many of these methods have inspired recent advances TTA algorithms for VLMs.
Generic test-time adaptation (TTA) Most TTA methods aim to improve model accuracy by optimizing a carefully designed unsupervised loss on unlabeled test data. prominent line of work minimizes the entropy of model predictions, based on the intuition that entropy quantifies prediction uncertainty. Pioncered by Tent [38]. these methods typically update the running statistics and affine parameters of batch normalization 17 layers. However, entropy minimization is often unstable, and many subsequent works 226
27 22 focus on improving its robustness. One important variant is
43 which captures model's uncertainty across different augmentations of the same input. This idea has inspired several follow-up TTA approaches [35 36 for VLMs.
marginal entropy
Another line of potential approaches focuses on restoring uncorrupted images from corrupted ones, using generative techniques such as diffusion models [K] super-resolution (30 6 These methods do not require adapting the model at test time. However, as noted in [27]. they often perform well on certain types of corruption but poorly on others, indicating limited generalization across corruption types.
A.2 Limitations
While our analysis reveals consistent variance collapse pattern across multiple datasets and cor- ruption types, it primarily focuses on natural distribution shifts and classification tasks. Extending analysis and algorithm broader types of distribution shifts (e.g., adversarial perturbations) and more diverse tasks (c.g., object detection, semantic segmentation) represents an important direction for future work.
A.3 Broader impacts
Our work focuses on understanding and mitigating the degradation of rision-language models under distribution shift, particularly in the context of image corruption. On the positive side, improving model robustness can enhance the reliability of real-world applications such as accessibility tools, autonomous systems, and content moderation, especially under suboptimal conditions. By providing theoretical insights and simple, efficient test-time adaptation methods, our work contributes toward safer and more dependable AI deployments.
We do not anticipate significant negative societal impacts. Our method is unsupervised, operates solely test time, and does not require access sensitive data any form of user interaction. Nonetheless, as with all where robustness could amplify existing biases be deployed without appropriate oversight. We encourage future work to consider fairness and accountability as these methods are applied more broadly.
techniques, there is potential for misuse in contexts
22

---

B Theoretical analysis
B.1 Variance decomposition
In this section we give formal proof of variance decomposition. Lemma B.1.
VeT
VeT
VGT intert
Vi
V.L
(II)
Proof. We define "prior" for each class c 1, 1,... C:
1
Dc
Vic
(12)
N
1=1
The class means are
N
Dio
Die Zz-
2,
(13)
i=1
V/ic
De
The global mean is
2
(14)
N
Notice that for all class c 1,... C, we have
N
N
1
Vic
1
Vic
1
yic ze Dc
(z:
2c) =
Zi
N
Ve
N
ye
N
i=1
1=1
1
yic ne
(definition of Ee))
N
Tlc
2.
Ze
(definition of j.)
Therefore,
GT Total
1
Vic
|| 212
N.C
De
1=1
1
yice 2| 2e + 2c xx12
N.C
Te
1
Vic
(||2: - 2cl12 + 11 - =112 + =1 (z: - 2c) (Ec - 8)
N.C
Te
N
N
c
1
Vie ||:: - zell +
Σ
Vie 1| 21/
N.C
Te
N C
i=1
Vic || - zell
+ Σ
E -
(definition of j.)
N
Te
yoT + GT
By replacing each Vie with Dic. Σ, with Σ., and & with 2, and repeating the above steps, it is straightforward to prove VPL = D + PL
23

---

B.2
Theoretical setup
This section introduces the setup and assumptions of our theoretical analysis. For simplicity, we focus on a binary classification setting where C' = 2. While the standard notation of label for image i yi Bio Viil "ul""= IR?, we write Y: Yi1 for brevity, with mild abuse of notation. We also assume there is no label imbalance, i.e., Pr(y = 0) Pr(m = =)
Image latent representation Motivated by 39 we assume that each image can be mapped to disentangled latent representation v; [ut u!";; bihiitt pooiss]] R. composed of four components:
1. Class-relevantteanee ttth E Rd;; Semantic feature that are directly predictive of the class label, with = μs for = and t9th = -14 for = 0.
2. Class-irrelevant feature v!th Rd=; Features that are unrelated to the classification task, such as background information. is preserved during pretraining due to CLIP's general rep- resentation learning objective. assume ve"" ~ Rademacher i.e., uniformly distributed
in {
3. Structured distribution shift ozhht E Rdaa:: Features representing systematic distribution changes in the target domain, such weather conditions or digital transforms. We assume ozhill = s. 6, where s indicates the severity of corruption or distribution shift.
4. Unstructured noise upaise Random noise introduced by the corruption process. We assume
y moise ~ 3 Rademacher i.e., uniformly distributed in { {-1, A
Notice that by controlling the ratio of s, || |1112, we can freely adjust the ratio for four compo- nents.
LayerNorm and image embedding Following the structure CLIP's visual encoder, we assume that the latent representation first passes through LayerNorm layer (2) with linear transformation, and then normalized to unit length. For analytical simplicity, we omit the demeaning step LayerNorm and ignore the bias term in its parameters. This simplification is also known as RMSNorm [41]. Under this simplification, the image embedding can expressed as
U;
2; normalize
(15)
Var v
where @ represents element-wise multiplication of vectors, w = [we wi''; whliin, Roisse] E R is the LayerNorm weights, and normalize(-) denotes & normalization. For simplicity, we w at initialization. wr updated during TTA. Since / /Var(e;] is just scalar, the equation above can be further reduced to
21 ④ w
2; = normalize (v, 3 w)
(16)
10, @ w|l2
Text embedding and prediction Let to, denotes the text embedding for class 0 and respec- tively. The model prediction is given by
0, when 2 to 2 2, t
y1
1, when 2, T_to to 2, t
(17)
24

---

B.3 Change of variances under corruption
This section studies the behavior of various types of variance with increasing corruption severity Theorem 3.1 (Variance collapse). When the sample size + +00,
|
dirr + s² dnoise
VGT inter
P
VGT P
(2)
12 dar + 5² 1 + s² dnoise"
14 dir 5² 1|01 8² dnoise
where s denotes the corruption severity. As increases, Vinee strictly decreases. In addition, Varr also decreases when ||ól12 2 v dmoise dier ||ae||2-
Proof. We first compute the normalizing factor for each image:
a @ w l = ||
@ 1
+ || O " 1 + ||
@ w 12
w Woise
|l C 11 + 1|w /w"1 1 + +s² 11ō @ w 1 s² 11 2o 1
(at initialization w 1) Notice that this normalizing factor the same for each image, and function of w and severity s. Let
|| + dur + x² (115)1 2 + daoose)
Z(w, v |l O w || 11 + s2 || wishit| 1 + s² ||
denote the normalizing factor. Under infinite sample size, the total mean z and class means Eo. can be expressed as:
L. R2 Z(w, s)
[0; 0; 8
w shin, 0)
L B[zi|y/ 0] Z(w, 8)
[ O wh 0; d @ w shit, 0]
21 P E[zilyy: = 1] = 2(w,8) 8)
[- ol 0, w shift, 0)
The GT-total variance:
> 2 El|z; B2
(11 2 c
1/2 + | "r + + || noise (12)
7(w, 8)2
|| woek 1/3 + I
in
+ s2. 11 moise |
11 @ woek
+ /1
$2. ② we
+ s²
hoise
111111 + dim + s2 dnuise
(at initialization u 1)
|1 dur + s2 (110112 + daoise)
The GT-inter variance:
VGT
P,
Σ
E[zilyi =c - 1z 112
2 c=1
|110012
Z(w, 5)2
|100 ② nht
webb|1
+ +
82. 1 s unshi
2/2 + 52
|1Aa113
(at initialization up 1)
|11/2 + dnr
32. (11011 d
And the GT-intra variance:
GT
GT
GT
1 ir
+ s2
1/2 moise
2
1| 3 wolks + | 12
+ 82 16
u shif
2 Banooss !|
durr + 8² dnoice
(at initialization w 1)
|101// + dur + s2 ( + dnsise)
25

---

B.4 Adaptation
In this section, we derive how maximizing the pseudo-label inter-class (PL-inter) variance influences the learned representation, when only the LayerNorm parameters are updated during test-time adaptation.
Lemma B.2.
When the sample size N + +00,
1
1
PL inter
|[Cov(z,, 9:)
2
(Eg:)²
(1 Eg;)²
Proof.
E
U x,1
Ez, + Cov(2 (4) Ez, +
Cov( 06)
E |z Ui =
Eji
Eg;
Cov 1/s) Edjs
E z. D0
=
Ezz
PL
1
E3 1i: c] E2,112
+ (11 2 (Eg,)²
1
(1 - )²,) |Cov(zz,, 0:)1
1
2
Remark B.3.
Cov(zi, 0s) Cov(zi, 2, - -z, to)
where Σx, is the covariance matrix of z; and to. t1 are the text embedding of class 0 and 1. This implies that maximizing Pl.-inter will enhance those features that (1) have high variance, and (2) are more relevant to the classification task described by the text embedding.
Theorem 3.2 (Maximization of PL-inter variance). When the sample size N oc,
yPL inter IS C(E.)))
2 || O "1 112 + 1| O w + |Onoiis @ wonks 11
2
11 C work + || | 1 + 11 @ 10 + s2 -||
where C(Ei;) (Eg,)
[I-EB, Jiu Cov 1i), our Cov (s)) and Onoise
Cov(u Coosee moive (.). Furthermore, when oipy 2 | and
0 2 4dwar we have
4dir
= C(Eis) (4o (y dier Wii P.P.. inter
|| 123) + 40 By
2110112 (Ao Anoiee || noise I12)
112 0,
(1 dim 52 (1081 + )
(4)
PL inter
V Wing Y inter
((Err)
8. < 0.
(5)
dir + s2 (1)))
)moise)
Proof. Similar to the procedure of deriving GT-inter, we start by computing the total mean á and pseudo-class means 20, 21.
F Ez,
10;0; 5 w thin, 0]
Z(w, s)
P Elz; = =1]
Eig; U. ④ w)
Eij
Z(w, s)
20
P E[
bi 0] Eij
E[(1 01) v. ④ w)
Z(w,s)
where Z(w, s) factor we defined in the proof of Theorem
v I| c w
_ + w
2 + s². 1 @ wiish _ + s2 [ 1 is the normalizing
3.1 For four components of the feature:
Eig with
c c
(2y) - 1) p eh 2Cov v.) p O ohls
Blg wiT @ in
= Cov (Brr, o O iir
Big vi
hiit
w
shill
= LEg; - 8 c shin
E a. w moise]
poise
= Cov (or, a muise
w moise
26

---

And similarly,
B[(1 0:) cls =
[[([ 9:) (2y: 1 -2Cov( Vi) 1 o w
[B[(1 úi) v.
" Cov (it 0²1) 3 w
[B[ 9t) ustilt
w
E( U1) s 6 whiit
[[ úr) moise
w moise
Cov (Ūt, moise)
w noise
Therefore, we have
PIPL
4
1
||E
cl Ez
2
1/2
1
1
1
(Bij)² (1 Eij.)²
i(u, 8)²
(ACov(Di:))) u)²
1//..11100 @ ch
1/ + ||Cov (Uis v"
12 + ||Cov (ūts x 2oise
w noise
1
1
40 || @ wh
112 + ||om @ w
ir 12 o 2 maisee 1|
(Bi.)² (1-EA;) || we + / [/ + s2
||ō wehin
1/ + +s² moiee
As a simple correctness check, when the pseudo-label ú ys, Vi, substituting Eý; Egg; 2
1
Jyy = Var(y: f. dur = 0, and noose 0 recovers the result of GT-inter variance Vor in Theorem 3.
Finally, we compute the gradients w.r.. four components of w at initialization. Note that although the pscudo-labels depend the model parameters, this dependence involves arg max operation and is thus non-differentiable Therefore, during optimization, we treat the pseudo-labels as fixed constants and do not backpropagate through them. Let C(E),,) (Eg.)² + (1 (1-B0,)2*
)LP
C(Bid)
(4o by 11
|| C " (22 + (4o s² 1| + whlhi (12) + (40 by
noise
O Donise
to Roise
(12)
inter
2
Z (ww, s)*
24² o ch
(4o dir - || + 40 by 1011 + (40
by Anoise (1 3 + dim + 52 (108) + dnoise)) ))²
|| o noise 2
C(Egi)
C(Bji
4 by
|| ols 1/2 + | o w 1|3 + || moise 2u noise
upptin )I
Inter
2
Z(w, 8)1
82 28 2
w
shift
PL
-C((B)))
|| + du + 22 (1 1 + daoise) 52 8²
inter
27

---

C Experiments
C.1 Effeet of corruptions
C.1.1 CIFAR-10-C
Impul Ne.ke
1103
2110
0.00
130
$2%
Dutcous Bkr
Stao Bu
0.71
Vouer
Zoan Bar
1.2
0.0
3.00
Siee
lever
0.30
Sea/y
Serrtty
State
Seatity
Datts
Figure 8: Effect of different levels of corruptions on VIT-B/32 on CIFAR-1O-C.
comppttio
severity
Original
level
Accuracy (86
Zoom
Snow
60-
Poost
Fog
50
40
0.95
0.01
0.02
0.03
0.04
0.05
0.06
0.07
GT-inter Variance
Figure 9: Correlation of GT-inter variance and classification accuracy of ViT-B/32 on CIFAR-IO-C.
Table 4: Pearson correlation coefficients between accuracy and variances ViT-B/32 CIFAR-10.
C.
y GTT total
intra
Inter
Accuracy
0.9104
0.8286
0.9483
28

---

C.1.2 CIFAR-100-C
Inpudee Nalbe
Dutsous Dky
Clan Bur
Veuee Bar
Zoan Bur
0.35
Seatyy
$196
Jeaarty
Ebetic
1.22
0.23
0.00
3.20
Figure 10: Effect of different levels of corruptions on ViT-B/16 on CIFAR-100-C.
compuption
severity
Driginll
level
60-
Accuracy (80)
50-
Motion
Zoom
Snow
Prost
30
20
p 0.98
0.02
0.03
0.04
0.05
0.06
0.07
GT-inter Variance
Figure 11: Correlation of GT-inter variance and classification accuracy of ViT-B/16 on CIFAR-100-C.
Table 5: Pearson correlation coefficients between accuracy and variances VIT-B/16 on CIFAR- 100-C.
tocal
Intra
inter
Accuracy 0.9364
0.8560
0.9752
29

---

C.1.3 ImageNet-C
bod
00
Centist
1botc Teeefum
Redien
1600 Dorraresd
06.
80-
Soartyy
Figure 12: Effect of different levels of corruptions on ViT-L/14 on ImageNet-C.
comppttio
severity
Originll
level
Gauss.
level
Shot
level
Imgul.
level
Deloc:
level
Accuracy (8a
Snow
Frost
Brit.
Canis.
Elsstic
Pool
30
JItG
p
0.175
0.250
0.275
Gi-inter Variance
Figure 13: Correlation of GT-inter variance and classification accuracy of ViT-L/14 on ImageNet-C.
Table 6: Pearson correlation coefficients between accuracy and variances on ViT-L/14 on ImageNet-
C.
>TT
LTT intra
DCT
Accuracy 0.8937
0.7753
0,9332
30

---

C.2 Experiment details
C.2.1 Baselines
For all methods, expect from TPT 33 and CLIPArTT 15 template in 44
which modifies the prompts, we the
"itpp of {class}"
bad photo of the {class)"
"a origami {class}"
"a photo of the large {class}"
"a {class} in a video game"
"art the {class}"
"a photo of the small {class}"
The text embedding for each class y is computed by
t, normalize
tu.
where ty,x
The following is our detailed handling method for other baselines and the usage of hyperparameters. The above hyperparameters are derived from those used experiments reported in previous papers.
For all augmentation- baselines (TPT B5 TPS [6), Zero II]. VTE (8).
AugMix to augment each test image 63 times to obtain batch of 64 images, which includes the original image. We select 10% of samples in the batch with lowest entropy aggregate.
In TPT [35), the number of prompt tokens is 4, the prompt is initialized with "a photo of a", and class-specific contexts are disabled. We use the Adam optimizer and adopt learning rate of 0.005, consistent with the setting used for ImageNet in the original papers.
In TDA [19], positive cache is enabled with shot capacity of 3, an adaptation strength (a) of 2.0, and a sharpness ratio (3) of 5.0. The negative cache enabled with a shot capacity of 2, an adaptation strength (ar) of 0. 17,17, and sharpness ratio (3) of 1.0, an entropy threshold between 0.2 and 0.5, and mask threshold between 0.03 and 1.0.
In DMN-7S 45, the positive cache is enabled with shot capacity of 50, adaptation strength (o) of 0.3, and sharpness ratio (B) of 5.5.
In TPS 36). we also use the Adam W optimizer and adopt a learning rate of 0.005, consistent with the setting used for ImageNet in the original papers.
In WATT-S 28). the learning rate is 0.001, the weight averaging is performed in sequential manner, with iterations per template and 5 total rounds of averaging.
In CLIPArTT [5], the learning rate is 0.001, the adaptation process for 10 steps, and the top predicted classes are used to construct the pscudo-label prompt.
C.2.2 Compute resources
All of our experiments conducted single NVIDIA Tesla V100 with 32GB memory, except for experiments on large batch size are conducted on single NVIDIA Tesla A100 with 80GB memory.
C.2.3 Licenses
The corruption benchmark is licensed under the Apache-2.0 License, as indicated at Ihttps:// // github. hatb..////eeeddryyss//ouusssees CLIP is licensed under the MIT License, as stated at
httpss//gittbb.coom/pemaa////
31

---

C.3 Full results for RQ1
Table 7: Accuracy (mean (s.d.) %) on corruption benchmarks with different batch sizes.
VIT-
Banch Size
Weuther
Axg:
Causs.
Sbor
Impul.
Defoc:
Gliss
Musion
Zoom
Saow
Frost
Pog
Brit.
Contr:
Elaste
Pixel
JPBG
57.9
76.2
79.2
m9
78.9 mm 75 15.2
(6.km.1)
75.6mm
30.3
63.4
63.3
20.5
18.00
76.2m.
6000m
16.8a)
70:11 1/3
78.9 mm 75 25:3
66-4mm1)
75.6
70.333
63.4
63.2m
20.5
5
6016
62.9
a3.3
76.222
6m
77.0ms
792
8-.1
78.9 mm 75.1
86.4-.1)
75.8.0
70.4933
646mm
63.2
71.0
10
59.8 mu 02.5m2
54.0a2
76.0m
61.3mm
mm
79.123
89.0
7888 mm
mm
86.482
75.8.m
70.383
65.7
63.28
71,0
50
54.2.
75.882
618mm
77.
78942
79.0
78.9
75:2
M..3..
75.9
701 mm
06/63
03442
71,080
62.0
75.6mm
62.4m
77.am
78.6
86.4a.0)
67.3
634g2
71.0
38.0
61.3
73.0aa
77.mm
7846
78.5
78.6
74.9
86.2..
77.0gm
68.0gu
67.0wa
Butch
Nosee
Blu
Weaher
Avg.
Carss:
Shot
Defoc
Gioes
Morion
Zoom
Stow
Frost
Pog
Bmi.
Centr
Elarse
Pixol
JPFC
25.9 mm
31.2
50.
27:0
(9.6
55.3tt:
53.0.33:
51.6
30.
(5.5
05.6.83:
36.75 36.7
34.4
38.7
43.1
25.9 mm
26:2m
$1.2.85
50.
49.7
55.345
53.0
51.6mm
50.
55.5
55.5-835
45.783
36 16.7
34.5
38.7m
43.1 mm
24.8
17.5
90.mm
27.
49.
$1.0.
m1.2mm
90:3
65.5
47.0mmm
36 mm
34.4m
38.7m
41. 41.3mm
10
mm
28.9raa:
38.0aa2
50.6..00
27.1 mm 49.7 mm
55441
33.0
51.7 mm 50.
(3.5
47.5
36.8
34.4m
33.7
43.6
20
19.4
30
38.6-22
50.7633
27.1 mm 499m2
55:5:0
33.0
5188m
30.602
(3.6m.1)
mm
34.4mm
35.7
44.1 10
30
35.0n
39.4-2
30.9..0
26
30.0>
55.5:2
33.1
mm9
30.5
65.5..
69.2mm
m3
34.6m
35.3
44.5 44.500
100
31 mm
35.
40000
51.0g2
27. mm 50.1 ms
55.5
83.
1100c
09.5m.)
36
34.6m
37.44
pm
51.2
$3.6
51.3c
63.
49.6.3)
34.
36.5
44.6
ImageNee
Busch
Nose
Blu
Weaher
Avg.
Canss:
Defoc
Morien
Zoom
Stow
Frost
B5nt.
Elarsc
Pixel
JPS3
31.6
13.2
16.3
41.0aa2
37.6
41,3124
54.0.
43.mm
57.4
67.7.a.)
45.763
41.5 mm
15.7
54.5
45.8mm
2
31.9
35.0
$1.2.
80.1.
376
46.as
42.122
54.533
438
577
67.74m
47.785
41.5
55.1mt
46.2
12:4
354mn::
11.1
99.mmm
376
467as
434i5
55.6.
46.mm
57.
89:8m
420
57.
55.3e
46.7mm
10
mm
35.6:::
16..am
99.mmm
376mm
463mg
44.1
$5.8n
m52mm
57.5
60-7am
494.m
428
58
98.1mm
55.2a
46.8
20
a3
38.3
31.3a2
39.66.
37.26
468m2
45:1a5
35.2%-
m66mm
57.
45.9
439
58.2
54.427
47.0
30
33.2 mm
35.1
31.645
38.80
369
47.mm
35.0
483mm
375
67.4n
47.0mm
45.1m
57.am
54.84
47.1
100
338
34995:
31.88
38.60
312nn
474ms
457724
35.0gm
489
67.2a3
44.20
460m2
54.04
41.0
Pm
38.7am
47.am
43.3a%
$4.7.m
37.5
67.3
42.0a
4633m)
57.0mm
$3.5u.
46.8
C.4 Full results for RQ2
132
1.30
-90
2.54
1:24
01/000
051-04
1.00
UMMS Tandow
0:8
194 Songraa
088
071144
Pride
Puidde
Figure 14: Variance collapse mitigation on CIFAR-I0-C.
32

---

230
Stukke Node
Vroue NeAe
140
GMB
2008 BN
1.20
0.403
01/100
8.00
1.35
8.00
0.22
6.00
042
8:42
Figure 15: Variance collapse mitigation on CIFAR-100-C.
Figure 16: Variance collapse mitigation ImageNet-C.
33

---

C.5 Ablation on layers to adapt
Mint adapts all LayerNorm layers in the vision encoder. Altemnttve choices include updating only the last block, the last MLP layer, or the first patching layer. Our comparison shows that updating all LayerNorm layers yields the best performance. For further discussions on which layers are most effective to adapt, we refer readers to related studies 23
Table 8: Comparison of different parts of the image encoder to update at test-time on CIFAR-10-C with ViT-B/32.
Layers to adapt
Accuracy (%)
Last block
63.4
Last MLP
62,9
Patching layer
63.6
All LayerNorm (Mint)
71.0
C.6 Hyperparameter sensitivity
In this subsection, we provide the results of hyperparameter sensitivity experiments on CIFAR-10-C and CIFAR-100-C.
CLIP
Ensemble
75
70
Accuracy (%)
65
60
55
0.003
0.006
0.009
0.012
100
Learning Rate
Kpror (x10²)
Figure 17: Hyperparameter sensitivity on CIFAR-IO-C.
CLIP
Ensemble
44
Accuracy (%)
42
40
38
36
34
0.003
0.006
0.009
0.012
0.015
1
2
10
20
50
100
+
Learning Rate
Kyoor
(x10²)
Figure 18: Hyperparameter sensitivity CIFAR-100-C.
34

---

In this subsection, compare between Mint and currently outstanding baselines, CLIPArTT [15]. WATT-S 28 TDA 119, and DMN-ZS (55,, on mixture of 15 types of corruption datasets on CIFAR-10-C 10-C with V/T-B/32, CIFAR- 100-C with VIT-B/16, and ImageNet-C with ViT-L/14, While C.7 Experiments on clean datasets
In this subsection, we offer the comparison between Mint and currently outstanding baselines, CLIPArTT IS). WATT-S (28). TDA [19, DMN-ZS [45, Tent [38], and ETA [26], on clean
(non-coryptiin) datasets with ViT-B/32 on CIFAR-10, ViT-B/16 on CIFAR-100, and ViT-L/14 on ImageNet.
Table 9: Accuracy (%) on clean datasets.
Method
ViT-B/32 on CIFAR-10
ViT-B/16 on CIFAR-100
ViT-I/14 on ImageNet
CLIP
88.3
68.4
73.0
CLIPAITT
89.1
70.2
72.1
WATT-S
89.8
72.3
74.5
TDA
89.6
70.1
73.4
DMN-ZS
90.2
69.4
73.1
Tent
91.1
72.2
73.4
ETA
91.4
73.0
73.6
Mint
91.6
74.1
75.6
C.8 Experiments ImageNet variants
In this subsection, we provide the comparison between Mint and currently outstanding baselines,
CLIPArTT (15). WATT-S (28), TDA [19), DMN-ZS [45), Tent (38), and ETA (26 in ImageNet
variants (-A, -V2, -R, -Sketch) datasets with ViT-B/16.
Table 10: Accuracy (%) on ImageNet variants with ViT-B/16.
Method
ImageNet-A
ImageNet-Ve V2
ImageNet-R
ImageNet-Sketel)
CLIP
49.2
60.4
72.7
44.9
CLIPArTT
49.6
72.8
45.0
WATT-S
51.7
61.2
75.7
47.0
TDA
51.0
61.2
73,9
46.4
DMN-ZS
49.7
60.5
73.0
45.4
Tent
51.9
61.0
77.0
45.4
52.0
61.0
77.4
46.8
Mint
54.7
62.6
78.1
48.4
C.9 Mixture corruption datasets
the results in the main text are obtained by testing on each corruption type separately. here we first mix the data from all 15 corruptions together and then perform evaluation on this mixed-domain setting, following the setup in 27
Table 11: Accuracy on Mixture of 15 Types of Corruptions.
Method
CIFAR-10-C
CIFAR-100-C
ImageNet-C
59.0
35.8
39.6
TDA
62.1
38.3
42.3
DMN-ZS
60.2
36.0
39.9
WATT-S
63.6
39.0
43.9
CLIPAITT
56.9
38.7
40.5
Mint
65.9
39.8
45.2
35


Reviews:
[{"review_id": "WP8fRfyOi9", "reviewer": "Reviewer_Cqv3", "summary": "This paper investigates the degradation of CLIP's performance under input corruptions and introduces a novel phenomenon, variance collapse.\nThe authors analyze variance collapse and provide a theoretical explanation: corrupted images weaken class-discriminative features, leading to a performance drop.\nTo address this, the authors propose Mint, a test-time adaptation method that maximizes inter-class variance estimated from pseudo-labels.\nMint consists of a mean accumulator to estimate PL-inter variance and a gradient accumulator to stabilize updates.\nExtensive results show that Mint improves the classification performance of CLIP across three commonly used corruption benchmarks.", "questions": "Please refer to the above weaknesses and provide corresponding explanations.", "limitations": "Yes, the authors provide the limitations and potential negative societal impact in the supplementary material.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n1. The authors provide a solid theoretical framework connecting the variance collapse to performance degradation, enhancing the interpretability of the proposed method.\n2. Experimental results show that Mint outperforms existing TTA methods on several corrupted benchmarks.\n3. The submission is well written and easy to follow.\n\n### Weaknesses:\n\n1. The decomposition in Eq.5 may be redundant. If certain classes have no samples, they contribute nothing to the computation of the inter-class variance. The average embedding of these classes can effectively be treated as the global center. In this way, the calculation of the left term of Eq.5 can be directly performed while retaining the update of Eq.6. If there is a meaningful numerical difference between the two formulations, the authors should provide quantitative evidence to justify the added complexity.\n2. The experiments are limited to corrupted benchmarks and do not report classification accuracy on clean images. Moreover, the paper omits evaluation on other standard distribution shift benchmarks commonly used for VLMs (e.g., ImageNet-A, -V, -R, -K).\n3. Is Mint sensitive to optimization space? The authors need to provide the results of optimizing the prompt or the text encoder and explain whether the image encoder must be selected for optimization.\n4. In Table 3, the authors only select some training-based methods for time comparison, but explain that they select the top-performing algorithm. In fact, training-free methods such as TDA are better than TPT in experimental results, but they were not selected. The authors should provide the time of high-performance training-free methods for reference.\n\n### Minor Weakness:\n1. The statement ‘where $K_{prior}$ is the strength of prior’ in Line 249 would make one think that $K_{prior}$ is an adaptive variable rather than a hyperparameter.\n2. There is a conflict between the statement in Lines 49-53 about ‘simultaneous reduction of GT-intra and GT-inter variances ... dilutes class-discriminative information.’ and the method that continues to minimize PL-intra variances.", "was_revised": true, "final_justification": "The paper presents a clearly written TTA method that consistently outperforms strong baselines. The rebuttal satisfactorily addresses all concerns, adding results on clean and natural distribution shift benchmarks, clarifying the decomposition of Eq. 5, explaining optimization choices, and providing runtime comparisons. Given its strong empirical results and authors' detailed responses, the reviewer decided to raise the score to borderline accept.", "rebuttal": "Thank you for taking the time to review our work and for your thoughtful feedback. We appreciate your recognition of our theoretical framework and experimental results. We also acknowledge your concerns and value your suggestions. Below, we respond to each point in detail and clarify potential misunderstandings.\n\n# W1. Decomposition in Eq (5)\nThank you for the question. You are right that even if certain classes have no samples, we can still evaluate the left-hand side of Eq (5) with the mean accumulator defined in Eq (6). However, **the key issue with this formulation is that the loss becomes disconnected from the current batch, making it non-differentiable with respect to the current features** $\\{z\\_i\\}$. This prevents backpropagation and gradient-based optimization. Our reformulation (right-hand side of Eq. 5) addresses this by preserving differentiability via the use of batch-level embeddings $\\{z_i\\}$, while still leveraging $\\tilde{z}_c$ and $\\tilde{z}$ as running estimates. This enables optimization using stochastic gradients and facilitates information sharing across batches. \n\n# W2. Clean images and ImageNet benchmarks\nThank you for the helpful suggestion. We have added results on clean datasets corresponding to the three corruption datasets (CIFAR-10, CIFAR-100, and ImageNet), as well as four widely-used natural distribution shift datasets: ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-Sketch. *The table below reports accuracy under the same no-augmentation setting as used in our main experiments.* We compare against the strongest baselines from the paper, including CLIPArTT and WATT. \n\nTable A: Performance on Clean Datasets\n|Method|ViT-B/32 on CIFAR-10|ViT-B/16 on CIFAR-100|ViT-L/14 on ImageNet|\n|-|-|-|-|\n|CLIP|88.3|68.4|73.0|\n|CLIPArTT|89.1|70.2|72.1|\n|WATT-S|89.8|72.3|74.5|\n|Mint|**91.6**|**74.1**|**75.6**|\n\nTable B: Performance on Natural Distribution Shifts Benchmark (ViT-B/16)\n|Method|ImageNet-A|ImageNet-V2|ImageNet-R|ImageNet-Sketch|\n|-|-|-|-|-|\n|CLIP|49.2|60.4|72.7|44.9|\n|CLIPArTT|49.6|60.5|72.8|45.0|\n|WATT-S|51.7|61.2|75.7|47.0|\n|Mint|**54.7**|**62.6**|**78.1**|**48.4**|\n\n# W3. Sensivity to optimization space\nThank you for the question. In our method, optimizing the image encoder is essential, as our loss is explicitly designed to address the variance collapse in image embeddings. The PL-inter variance objective is defined on image features and is not differentiable with respect to the text encoder or prompt, since we treat the model’s predictions as hard pseudo-labels. As a result, **updating only the text prompt or text encoder yields zero gradients and cannot adapt the model.**\n\n# W4. Time comparison\nThank you for the suggestion. By “top-performing algorithms,” we refer to CLIPArTT and WATT-S, which achieve 4.9% and 6.1% accuracy gain, respectively. Following your advice, we have added all previously omitted training-free baselines (Ensemble, Zero, VTE, DMN, TDA) to the runtime comparison in Table 3, ordered by accuracy. This provides a more comprehensive reference for efficiency comparison. \n\nTable 3: (Updated) Comparison of computation time for\none corruption on CIFAR-100-C.\n|Method|Testing Time|Accuracy (%)|Gain (%)|\n|-|-|-|-|\n|CLIP|21s|35.8|-|\n|TPT|23m21s|36.0|+0.2|\n|VTE|9m45s|36.6|+0.8|\n|Zero|9m50s|36.8|+1.0|\n|Ensemble|21s|37.1|+1.3|\n|TDA|33s|38.4|+2.6|\n|DMN-ZS|30s|38.5|+2.7|\n|TPS|9m58s|38.6|+2.8|\n|CLIPArTT|7m40s|40.7|+4.9|\n|WATT-S|50m20s|41.9|+6.1|\n|Mint|1m07s|44.1|+8.3|\n\n# Minor 1. Hyperparameter $K_{prior}$\n\nThank you for pointing this out. We agree that the current phrasing may be misleading. We will revise the sentence to avoid confusion.\n\n# Minor 2. Statement conflict\n\nThank you for the question. We did observe a simultaneous reduction in both GT-inter and GT-intra variances. Among them, GT-inter variance, which reflects the separation between class centers, is naturally desirable to be large. On the other hand, a decrease in GT-intra variance, which reflects within-class variation, is not necessarily harmful.\n\nIn practice, we found that GT-inter variance correlates more strongly with accuracy than GT-intra and GT-total (see Tables 4,5,6 in Appendix C.1). Therefore, in our method, we choose to maximize PL-inter variance (computed as PL-total minus PL-intra), without explicitly penalizing PL-intra further in the loss."}, {"review_id": "NvdxgXnk8E", "reviewer": "Reviewer_EhQi", "summary": "This paper proposes a novel test-time adaptation (TTA) method for vision-language models (VLMs) like CLIP targetting image corruption.\nThe authors found a phenomenon in which the variance of image embeddings decreases, and the embeddings become less distinguishable under image corruption, which is called variance collapse.\nThe mechanism of variance collapse and the fact that variance collapse is closely related to model performance is also theoretically explained.\nBased on this finding, a novel TTA method called Mint, which maximizes the inter-class variance estimated with pseudo labels, is proposed.\nExperimental results show that Mint outperforms prior TTA methods for VLMs under image corruption.", "questions": "- Q1: Since $\\mathcal{V}^\\text{PL}\\_\\text{inter} = \\mathcal{V}^\\text{PL}\\_\\text{total} - \\mathcal{V}^\\text{PL}\\_\\text{intra}$, $\\mathcal{V}^\\text{PL}\\_\\text{intra}$ should have negative correlation with accuracy. However, L136 and Fig. 3 state that $\\mathcal{V}^\\text{PL}\\_\\text{intra}$ increases with severe corruption and has a positive correlation with performance. Is the sign flipped?\n- Q2: In Tabs. 1 and 2, how the model and dataset combinations, i.e., (ViT-B/32, CIFAR10-C), (ViT-B/16, CIFAR100-C), and (ViT-L/14, ImageNet-C), were selected?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper is well-structured, and the motivation is clear.\n- The finding of variance collapse is interesting.\n- The design choice of Mint based on variance collapse is reasonable.\n\n## Weaknesses\n- W1: It is questionable whether the assumption for the theoretical analysis that the image features can be disentangled into the four components. Under image corruption, such as noise, the information is lost and cannot be recovered. Although this assumption is motivated by Wiles et al. [33], they mainly consider attributes rather than corruption.\n- W2: It would be interesting to validate that variance collapse is specific to image corruption experimentally. For example, comparing vanilla ImageNet and ImageNet-A/R/S would support the claim.\n- W3: Maximizing PL-total variance while minimizing PL-intra may be related to neural collapse [a,b]. Discussing neural collapse would improve the coverage of related works.\n\n[a] Kothapalli, Neural Collapse: A Review on Modelling Principles and Generalization, TMLR, 2023.     \n[b] Papyan et al., Prevalence of neural collapse during the terminal phase of deep learning training, PNAS, 2020.", "was_revised": true, "final_justification": "Considering the other reviews and the authors' rebuttal, my concerns have been addressed. I keep my score.\nAlthough it is questionable whether the theoretical assumption holds, the finding on the variances is intriguing.", "rebuttal": "Thank you very much for your positive and encouraging feedback. We're pleased that you found the paper well-structured, the motivation clear, and the proposed phenomenon and method compelling. We truly appreciate your support and are happy to address any remaining points or clarifications below.\n\n# W1. Assumption of disentangled components\n\nThank you for raising this point. We would like to clarify that the disentanglement assumption is primarily a conceptual tool to make the theoretical analysis more interpretable and analytically tractable. \n\nIn fact, our analysis can be extended to a more general setting where each feature vector undergoes a rotation, such that each dimension becomes a mixture of the original components. While this generalization does not change the conclusion of Theorem 3.1, the gradient derivation in Theorem 3.2 would involve additional rotation terms, making the result harder to interpret while offering limited new insight. \n\nThat said, we acknowledge that in extreme scenarios, such as when corruption completely entangles class-relevant signals (e.g., classifying different noise patterns themselves), the disentanglement assumption may indeed fail. In these cases, adaptation methods relying on discriminative features become fundamentally ineffective, as meaningful class-specific signals no longer exist or cannot be distinguished from corruption.\n\n# W2. ImageNets benchmark\n\nThank you for the insightful suggestion. Following your recommendation, we computed the GT-total, GT-intra, and GT-inter variances on ImageNet, ImageNet-A, ImageNet-R, and ImageNet-Sketch using the ViT-B/16 backbone. While this comparison may not be as tightly controlled as our corruption-based experiments, due to differences in image content and number of classes (e.g., ImageNet and ImageNet-Sketch have 1,000 classes, while ImageNet-A and ImageNet-R contain only 200), it still provides valuable insights.\n\nTable A: Variances on Natural Distribution Shifts Benchmark (ViT-B/16)\n|Variance type|ImageNet|ImageNet-A|ImageNet-R|ImageNet-Sketch|\n|-|-|-|-|-|\n|GT-total|0.479|0.496|0.445|0.387|\n|GT-intra|0.257|0.346|0.330|0.224|\n|GT-inter|0.222|0.150|0.115|0.163|\n\nOn ImageNet-Sketch, we observed a clear variance collapse pattern consistent with our corruption benchmarks: both GT-inter and GT-intra variances decrease. However, on ImageNet-A and ImageNet-R, we observed that GT-inter variance still decreases, but GT-intra variance increases instead. \n\nThat said, the consistent drop in GT-inter variance across all three datasets supports the potential applicability of Mint to a broader range of distribution shifts, as it directly targets inter-class variance. Therefore, we evaluated Mint on these datasets, and the results are as follows:\n\nTable B: Performance on Natural Distribution Shifts Benchmark (ViT-B/16)\n|Method|ImageNet-A|ImageNet-R|ImageNet-Sketch|\n|-|-|-|-|\n|CLIP|49.2|72.7|44.9|\n|CLIPArTT|49.6|72.8|45.0|\n|WATT-S|51.7|75.7|47.0|\n|Mint|**54.7**|**78.1**|**48.4**|\n\nNotably, Mint also outperforms our strongest baselines (CLIPArTT, WATT-S) from the main paper on all three datasets, further supporting its effectiveness beyond corruption benchmarks.\n\n# W3. Neural collapse\n\nThank you for the insightful suggestion. We agree that the idea of maximizing PL-total variance while minimizing PL-intra shares a conceptual connection with neural collapse phenomena, which also involve the alignment of features and separation of class means. We will include a discussion of neural collapse in the revised version to better contextualize our work within this broader literature.\n\n# Q1. Variances\n\nThank you for the question. In Line 136 and Figure 3, we are referring to the GT- variances (e.g., $\\mathcal{V}\\^{GT}\\_{inter}$, $\\mathcal{V}\\^{GT}\\_{intra}$), not the PL-variances. As shown in Tables 4–6 in Appendix C.1, all $\\mathcal{V}\\^{GT}\\_{inter}$, $\\mathcal{V}\\^{GT}\\_{intra}$ and $\\mathcal{V}\\^{GT}\\_{total}$ positively correlate with accuracy, i.e., they decrease as corruption severity increases, and so does performance.\n\nThis is due to the phenomenon we described in Line 173: “As a result, the image encoder tends to embed corruption-related patterns into the representation itself.” Consequently, embeddings from different classes become more similar, reducing both inter- and intra-class variance, and degrading performance.\n\n# Q2. Model and dataset combination\n\nThank you for the question. The chosen model-dataset combinations reflect practical usage scenarios, where more efficient, lightweight models (e.g., ViT-B/32) are often used for simpler tasks like CIFAR-10-C, while stronger models (e.g., ViT-L/14) are typically employed for more complex tasks such as ImageNet-C. This setup ensures that the evaluation remains realistic and meaningful across a range of difficulty levels."}, {"review_id": "xhL8rwcleS", "reviewer": "Reviewer_3aNj", "summary": "Work discusses variance collapse under corruptions i.e., embeddings of images tend to become more similar, regardless of whether they belong to the same class or different classes. A strong correlation between inter-class variance and classification accuracy is further shown. The proposed method uses Test time augmentation to improve model accuracy on the noise by    \n1)\tMaximizing intra-variance among features using pseudo labels.    \n2)\tUsing mean accumulator (average image embeddings for each pseudo-class) and gradient accumulator to improve TTA under limited batch.    \n3)\tOnly updating the LayerNorm parameters to improve image embedding.    \nAuthors postulate that models should show low intra-class variance (same class embedding should be closer) and high inter-class variance (different classes embedding separated) for higher accuracy.", "questions": "Please refer to the above weakness. Empirically, as the severity of noises goes up, features start to mix into one another is not novel and has been done before. The main contribution of theoretically showing how severity affects features is novel, but also needs to clearly explain the steps, which I believe severely weakened the paper. Also, the case for why using test time augmentation is not clear, given previous works which have seen these noise before, may easily outperform such a approach.", "limitations": "No limitation as such.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 1, "originality": 4, "strengths_and_weaknesses": "## Strengths: \n1)\tPaper gets credit for mathematically justifying some known solution for low resolution (and similar other kinds of noises).   \n    a.\tEffect of noise severity of feature variance.   \n    b.\t“maximizing PL-inter is equivalent to jointly maximizing PL-total variance and minimizing PL-intra variance”.    \n2)\tCode is provided with supplementary, hopefully they will make the code public, as it's not going to be an easy implementation.    \n3)\tSmall batch size in TTA has a series of problems like “noisy and biased gradients” and few samples across classes, where the distance between a sample and its mean becomes zero. Thus, the work opts for a mean accumulator and a gradient accumulator to aggregate information across batches, which is the first of its kind.      \n4)\tCost-efficient as the model updates only the layer norm during test time.    \n5)\tAblation successfully establishes the effectiveness of each component.    \n\n\n## Weakness \n1.\t**The idea of Variance Collapse is not novel.** Features collapse under low resolution is a fairly well-researched topic across severities. The work simply extends the observation to other forms of noise.  Missing reference works, and the tone of writing indicates novelty. Here are some works indicating a similar property:    \n    a)\t**Feature collapse on foundation models:** LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models, ICLR’25   \n    b)\t**Overlapping of positive and negative instances of class under low resolution**: Dive into the Resolution Augmentations and Metrics in Low Resolution Face Recognition: A Plain yet Effective New Baseline, AAAI’23.   \n   c)\t**Overlap of feature centers:** Recognizability Embedding Enhancement for Very Low-Resolution Face Recognition and Quality Estimation, CVPR’23.    \n\n2.\tOne of the main contributions of the paper is the theoretical justification for variance collapse with noise. But the theoretical proof is not at all an easy read, with tons of confusion in the derivation of Theorem 3.1. & Theorem 3.2   \n    a.\t**Confusion regarding layer norm** (equation number missing) Line 165. Why are authors using L2 norm as Layer norm? In traditional layer norm there is no L2 normalization,                        \n   Z =  (  (v – mean_dim) / (variance_dim) **0.5  )  * w + beta   \nAuthors have instead used two L2 normalization steps, which is not how PyTorch has implemented layer norm either (ref: https://docs.pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html). The original layer norm doesn't allow L2 normalization either.    \n   b.\tA core assumption seems to be feature vector distangled as: $v_i = v_i^{cls} +  v_i^{irr} + v_i^{shift} + v_i^{noise}$  & weight as $w = w^{cls} + w^{irr} +  w^{shift} + w^{noise}$, where  $v_i^j \\bigodot w^k = 0 \\text{ if } k \\ne j $. \nHow weights are broken into these components is not clear, and why $v_i^j \\bigodot w^k = 0 \\text{ if } k \\ne j $, for example $v_i^{cls} \\bigodot w^{irr} = 0 $. If its not addition, and instead concatenation, it will raise more questions.     \n    c. Appendix: Line 576 (equation number missing) appears to be using Cauchy Cauchy-Schwarz inequality. I’m not sure why  $|| u \\bigodot w^{cls} || =  || u || $ (not sure why $w^{cls}$ is assumed to be taken from a similar distribution as that of u. Similarly, for other multiplication products as well.    \n    d.\tLine 580 (equation number missing). Under the infinite sum, average (mean) becomes an l2 normalized vector? Clarity for the equation is required, with missing steps and lack of assumptions, it's not easy to understand how Theorem 3.1, and Theorem 3.2 are derived. I can only take the author's word for it if these equations are derived correctly.     \n   e.\tB.4 Adaptation: What is conv? How is equation line 589 (equation no missing) derived?    \n   f.\t[Line 190] *“maximize the PL-inter variance by updating the parameters of LayerNorm, the parameters associated with structured distribution shifts (i.e., wshift) are necessarily suppressed”* how? Which equation 3 or 4? \n  g.\t[Line 194] Maximizing PL-inter variance will increase the weights associated with task-relevant features (i.e., wcls)? How?    \n  h.\tText writes inferences without actually pointing to what equation it's using to derive those inferences. This is not possible for me to understand.     \n   i.\t[line 176] *“GT-inter variance has strong relevance to classification accuracy, as it reflects the proportion of task-relevant features within the overall feature representation”*. How is accuracy reflected in Theorem 3.1, which primarily connects to variance to noise severity. \n\n\n3.\tAblation results requested for :   \n    a.\tWhat happens if you maximize variance inter (instead of total - intra)   \n    b.\tOther variations, updating last block? Last MLP layer?   \n\n4.\t**Fundamental flaw with logic: Why test time augmentation?**   \nMissing works include Super resolution (generate HQ sample on fly), Lr-TK0 [1] (pretraining model on synthetic diffusion generated images with noise), Robsut SAM [2] (prompts help improve accuracy)    \nIf the Idea is to have never seen the noise before, why do authors treat the model training on each noise independently [line 265]. Separating noise means the authors already know what noise looks like. Not training on it is purely a design choice. Ideally, should evaluate the **same model across all noise**, i.e., training using an amalgam of all noises. if the assumption is to not been seen the noise before.    \n[1] LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models, ICLR’25 (feature collapse on foundation models)   \n[2] RobustSAM: Segment Anything Robustly on Degraded Images, CVPR’24   \n\n5. How does this lab-simulated noise translate to the real world? Can the authors show how the model would be affected when faced with real-world images, which often have a large variety of noise? \n\nMinor Complaint: \nFigure 3 does not clearly convey what it’s trying to convey. I don’t see how severity is playing any role here.\nMissing citation: Feature Augmentation based Test-Time Adaptation, WACV'25, it has a running mean (mean accumulator) for test time augmentation.", "was_revised": true, "final_justification": "After careful consideration and seeing additional experiments, I have revised my evaluation to a weak reject. Below, I outline both the strengths and concerns that informed my decision:\n\nGain for the research community :     \n- The paper offers a theoretical perspective on feature/variance collapse under noise. The derivation, while complex, attempts to formalize an important phenomenon.\n- The idea of using test-time adaptation (TTA) via layer norm updates to improve robustness is interesting.\n\nConcerns regarding the work:    \n- The core theoretical derivation is difficult to follow (with missing steps) and currently resides in the supplementary material. A clearer and more integrated presentation in the main paper would significantly enhance accessibility.\n- The key goal of the work is to improve the robustness of the model on **one kind of noise** (results in paper). A simple Google search reveals a missing comparison with prior work (e.g., Super-resolution, LRTK0, RobustSAM), which solves a similar problem. This weakens the empirical justification for the proposed approach. A deeper search may reveal more methods, but the bare minimum Super resolution needs to be incorporated.\n- In my *subjective opinion*, the authors have put artificial constraints on the problem statement to spare their work against comparison with existing approaches, namely, 1) No training, 2) No information about the noise the model is improving robustness against. \n- However, a similar argument can be made against the work, as existing approaches improve the robustness in zero-shot conditions without any information about the domain of test set images.  Additionally,  \"No information about the noise during inference\" ensures the model will be exposed to *only one type of noise*, which is weirdly convenient (for robustness results in paper).  \n- Thus, not training, or training a simple classifier to inform the model about the type of noise, are simply design choices in my *subjective opinion*, which would have put them in direct comparison with existing methods. \n- Most types of noise (e.g., Gaussian blur, impulse noise, motion blur, etc.) can be easily simulated and addressed with existing approaches. But since comparison with existing work is missing, we don't know how good this solution really is. Comparison with only TTA approaches is too narrow in my opinion.\n- *What would have been an acceptance from my side?* **No assumptions about test set**. 1 TTA-Mint model evaluated on all noises (not 15 noises 15 TTA Mints). Table 2 and Table 3 results would be ablation to show improvement on individual noises, but **the main result would be 1 model exposed to all \"amalgam\" of noises**.", "rebuttal": "Thank you for your thoughtful and encouraging feedback. We appreciate your recognition of our theoretical insights and efforts to address the challenges of small-batch TTA. Below, we respond to your concerns point by point.\n\n# W1. Idea of variance collapse\nThank you for the helpful feedback and references. While we agree feature collapse under low-resolution inputs has been previously studied, our work explores a broader set of 15 corruption types spanning noise, blur, weather, and digital transformations (e.g., shot noise, motion blur, snow), many fundamentally different from low-resolution. Importantly, our key contribution is not only observing variance collapse but quantifying it, providing theoretical insights, and designing a corresponding adaptation loss. We appreciate your suggestions on relevant literature and will include these references to better contextualize our contributions.\n\n# W2. Confusion in theoretical analysis\nThank you very much for carefully reading through our theoretical analysis. Below, we address your concerns and clarify the derivations: \n\n## a. Layer norm\nWe used a simplified form of LayerNorm for analytical tractability. **The first** $\\text{normalize}(\\cdot)$ in our derivation comes from a simplified but general form of LayerNorm. Let the input of LayerNorm to be $v_i = [v_{i1}, \\cdots, v_{id}]^\\top \\in \\mathbb{R}^d$, and the output be $u_i\\in\\mathbb{R}^d$. The standard LayerNorm can be written as\n$$u_i = \\frac{v_i - \\bar{v}\\_i}{\\sqrt{\\frac{1}{d}\\sum_{j=1}^d(v_{ij} - \\bar{v}_i)^2}} \\odot w + b$$\n\nwhere $\\bar{v}\\_i=\\frac{1}{d}\\sum_{j=1}^d v_{ij}$. As mentioned in Line 563, *for analytical simplicity, we omit the demeaning step and the bias term*, which leads to the following form:\n$$u_i = \\frac{v_i}{\\sqrt{\\frac{1}{d}\\sum_{j=1}^d v_{ij}^2}} \\odot w $$\nThis corresponds to the widely used RMSNorm [1], a common simplification of LayerNorm that retains the normalization effect. Factoring out the constant $\\sqrt{1/d}$ and writing the result in terms of L2 normalization:\n$$u_i = \\sqrt{d} \\cdot \\text{normalize}(v_i) \\odot w$$\nFinally, **the second** $\\text{normalize}(\\cdot)$ in our derivation is not part of LayerNorm or RMSNorm, but instead comes from the CLIP architecture (see Figure 3 in [2]), where the output image embeddings are L2 normalized before computing cosine similarity. \n\nWe will revise the text to make this simplification process clearer and avoid ambiguity. \n\n[1] Root Mean Square Layer Normalization. NeurIPS 2019\n\n[2] Learning Transferable Visual Models From Natural Language Supervision. ICLR 2021\n\n## b. Feature disentaglement\nTo clarify, we use concatenation to construct the feature and weight vectors, and $\\odot$ denotes the Hadamard (element-wise) product, not the dot product. $v^{cls}$ and $w^{irr}$ applied to different subspaces and their Hadamard product is not well-defined.\n\nThis decomposition follows a common analytical framework used in prior works [3,4], and is mainly intended to help identify which part of the feature vector each component of $w$ operates on, making the theoretical results more interpretable. In a more general setting where $v_i$ undergoes a rotation, each dimension becomes a mixture of the original components, and the gradient on $w$ will be a rotated version of our current result in Theorem 3.2. While the essence of the conclusion remains, such formulations would significantly complicate the interpretation. \n\n[3] A fine-grained analysis on distribution shift. ICLR 2022. \n\n[4] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors. ICLR 2024. \n\n## c. Inequality\nWe would like to clarify that the derivation in Appendix Line 576 does not use the Cauchy-Schwarz inequality. All expressions are exact equalities. The identity holds because we assume that $w^{cls}$ is initialized as an all-one vector. As a result, the Hadamard product $u\\odot w^{cls}=u$ holds for any $u$, and no distributional assumption is involved.\n\n## d. Convergence of means\nWe would like to clarify that the expression on Line 580 does not treat the mean vector as L2-normalized. The normalization factor $Z(w, s)$ arises from the final normalization step in CLIP's image encoder, and applies to each individual feature vector $z_i$; it is not related to the sample average.\n\nSpecifically, the convergence $\\bar z \\xrightarrow{p} \\mathbb{E}z_i$ follows from the Weak Law of Large Numbers, indicating that the empirical average converges to the expected value. Each feature vector is defined as:\n$$z_i=\\frac{1}{Z(w, s)} \\cdot [v^{cls} \\odot w^{cls}; v^{irr} \\odot w^{irr}; \\delta \\odot w^{shift}; v^{noise} \\odot w^{noise}]. $$\nTo compute the expectation, we note that $\\mathbb{E} v^{cls} = 0$ due to class balance, $\\mathbb{E} v^{irr} = 0,\\mathbb{E} v^{noise} = 0$ due to the Rademacher distribution. Thus, we obtain:\n$$\\bar{z}\\xrightarrow{p}\\mathbb{E}z_i=\\frac{1}{Z(w, s)} \\cdot [0; 0; s \\cdot \\delta \\odot w^{shift}; 0]$$\n\n## e. B.4 Adaptation\n “Cov” in Line 589 refers to the covariance operator. The first two lines follow directly from the definition of covariance $Cov(X,Y) = \\mathbb{E}[X,Y] = \\mathbb{E}[X] \\mathbb{E}[Y]$. The third line follows from the Weak Law of Large Numbers.\n\n## f&g. Theorem 3.2\nThis refers to Equation 4, where the gradient of PL-inter variance with respect to $w^{shift}$ is negative. Since we maximize $\\mathcal{V}\\^{PL}\\_{inter}$ via gradient ascent, the update takes the form:\n$$w^{shift} \\leftarrow w^{shift} + \\eta \\cdot \\nabla_{w^{shift}} \\mathcal{V}^{PL}_{inter}$$\nwhere $\\eta$ is the learning rate. Because the gradient is negative, this update reduces the magnitude of $w^{shift}$, effectively suppressing the structured shift component.\n\nSimilarly, we show that the gradient with respect to $w^{cls}$ is positive, meaning the same gradient ascent step will increase $w^{cls}$, thus amplifying task-relevant features.\n\n## h. Equation reference\nThank you for the feedback. We appreciate the suggestion and will revise the text to more clearly reference the equations associated with each inference to improve readability and traceability. If there are specific instances that are considered unclear, we would greatly appreciate it if you could point them out, so we can address them more precisely in the revision.\n\n## i. Accuracy\nTheorem 3.1 shows that GT-inter variance increases with corruption severity, indicating a lower proportion of task-relevant features in the overall representation. This implies a reduced signal-to-noise ratio, which typically leads to lower accuracy. We will clarify this connection in the text.\n\n# W3. Ablation study\n## a. Maximize inter variance\nIf we directly maximize the PL-inter variance as defined by the LHS of Equation 5, we need to remove the mean accumulator (otherwise, the objective has zero gradients with respect to the current batch). We maximize the objective of\n$$\\frac{1}{C_b} \\sum_{c=1}^{C_b} \\\\| m_c - m \\\\|\\_2\\^2$$\nwhere we use batch statistics\n$$m_c = \\frac{\\sum_{i \\in \\mathbb{B}\\_b} \\hat{y}\\_{ic} z_i}{\\sum_{i \\in \\mathbb{B}\\_b} \\hat{y}\\_{ic}}, m = \\frac{1}{\\mathbb{B}\\_b} \\sum_{i \\in \\mathbb{B}\\_b} z_i$$\nas approximations of $\\tilde{z_c}$ and $\\tilde{z}$. The results are shown below for different batch sizes. \n|Method|BS=1|BS=5|BS=20|BS=100|\n|-|-|-|-|-|\n|Maximize inter|61.4|67.8|68.9|69.9|\n|Mint(total-intra)|70.5|71.0|71.0|70.9|\n\nNotice that this objective is numerically equivalent to the \"gradient accumulator only\" variant in Figure 6, and thus yields the same results in practice.\n\n## b. Other variations\nWe tried several variations, including updating the last block, the last MLP layer, and the patching layer of the image encoder, but found that updating all LayerNorm layers yields the best performance. This choice is also consistent with prior works such as CLIPArTT and WATT, which also adapt LayerNorm.\n|Method|Accuracy|\n|-|-|\n|Last block|63.4|\n|Last MLP|62.9|\n|Patching layer|63.6|\n|All LayerNorm (Mint)|71.0|\n\n# W4. Why TTA\nThank you for the comment. We would like to clarify the motivation and setup of our experiment setup.\n- **We follow the most standard TTA setting widely used in the literature** [5,6,7,8]. Specifically, we do not train a model on each corruption. Instead, we use the same pretrained VLM, and perform TTA separately on each corruption type, using only unlabeled test samples. This per-corruption evaluation is a common and practical protocol designed to assess the average performance of TTA algorithms under different corruptions, in a controlled and interpretable manner. \n- **This setup does not assume the model “knows” the corruption types.** During training, the model is never exposed to these corruptions. The corruptions are only encountered at test time, during adaptation, which is consistent with the goal of evaluating corruption-agnostic TTA methods.\n- Regarding the statement “not training on it is purely a design choice”, this is **not the case**. Our focus is on the adaptation of pretrained VLMs, where we only have access to public checkpoints and no access to training data. This is a practical constraint, not an arbitrary design decision, and reflects real-world deployment conditions for foundation models.\n\n[5] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models. CVPR 2024\n\n[6] Efficient test-time adaptation of vision-language models. CVPR 2024\n\n[7] WATT: weight average test time adaptation of CLIP. NeurIPS 2024\n\n[8] Clipartt: Adaptation of CLIP to new domains at test time. WACV 2025\n\n# W5. Translation to the real world\nWe conduct experiment on Natural Distribution Shifts Benchmark (please refer to Table B in our response to Reviewer Cqv3). Mint remains effective when faced with real-world images.\n\n# Figure 3\nIn Figure 3, severity is encoded by the color intensity of each point, with darker shades indicating higher corruption severity levels.\n\n# Missing citation\nWe will include the WACV’25 paper in our revision and clarify its connection to our method."}, {"review_id": "LStHcKvf6w", "reviewer": "Reviewer_HhB3", "summary": "The paper introduces Mint, a test-time adaptation technique aimed at improving model robustness against image corruptions. Mint leverages pseudo-labels to maximize inter-class variance, addressing the authors’ observation that both intra- and inter-class variances of CLIP image embeddings tend to collapse under corruptions, which is closely linked to performance degradation. Experimental results demonstrate that Mint significantly improves robustness across various corruption benchmarks, even when limited test data is available.", "questions": "Table 2 shows that increasing the batch size does not always lead to improved accuracy, which appears counterintuitive given the common assumption that more data should yield better estimates of unknown data statistics. What could be the underlying reasons for this behavior?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper is clearly written and easy to follow.\n\n* The proposed method is based on insightful observations about variance collapse of intra- and inter-class embeddings, also supported by a theoretical rationale.\n\n* The usage of mean and gradient accumulators is effective in low batch size cases, where only a few test samples are available.\n\n* Mint demonstrates time efficiency, compared to many TTA methods.\n\n* The paper includes comprehensive ablation studies and analysis that highlight the roles of different accumulators and the influence of hyperparameters.\n\nWeaknesses:\n\n* The paper lacks comparative results showing how SOTA methods perform under varying batch sizes. While Mint is evaluated across different batch sizes, it is unclear how it compares to SOTA methods in these settings, i.e. it is uncertain whether Mint maintains its advantage in more constrained batch size scenarios.", "was_revised": true, "final_justification": "I did not find any significant technical flaws. The exchange of comments and responses between reviewer 3aNj and the authors helped clarify several doubts. In particular, the responses of the authors made the setup of corruption robustness and TTA more understandable, and their responses regarding “confusion in theoretical analysis” indicate that they can revise the manuscript to improve its quality. Given the overall strength of the work, I intend to give a score of accept.", "rebuttal": "Thank you for your detailed and encouraging review. We appreciate your recognition of the clarity of the writing, the theoretical support for our variance-based perspective, and the effectiveness of our design under low-batch test-time scenarios. Below, we respond to your comments and provide clarifications where needed.\n\n# W1. Comparative results under varying batch sizes\n\nThank you for the suggestion! To address this concern, we conducted additional experiments comparing Mint with the most competitive baselines (CLIPArTT and WATT-S) under varying batch sizes $B = 1, 2, 5, 10, 20, 50$. The results show that Mint consistently outperforms these baselines, **particularly in more constrained settings (e.g., batch size = 1, 2, 5, 10)**, where it demonstrates greater stability and a more pronounced performance advantage. These findings further highlight Mint's effectiveness and robustness across different batch sizes. \n\nTable A: Accuracy, ViT-B/32 on CIFAR-10-C (CLIP: 59.0)\n|Method|BS=1|BS=2|BS=5|BS=10|BS=20|BS=50|\n|-|-|-|-|-|-|-|\n|CLIPArTT|59.2|59.3|61.7|63.5|64.4|64.5|\n|WATT-S|63.6|63.9|64.1|64.8|67.1|70.6|\n|Mint|70.5|70.5|71.0|71.0|71.0|71.0|\n\n\nTable B: Accuracy, ViT-B/16 on CIFAR-100-C (CLIP: 35.8)\n|Method|BS=1|BS=2|BS=5|BS=10|BS=20|BS=50|\n|-|-|-|-|-|-|-|\n|CLIPArTT|39.2|39.3|39.4|40.0|40.7|41.4|\n|WATT-S|35.9|36.4|36.9|38.8|41.9|42.0|\n|Mint|43.1|43.1|43.3|44.1|44.5|44.5|\n\nTable C: Accuracy, ViT-L/14 on ImageNet-C (CLIP: 39.6)\n|Method|BS=1|BS=2|BS=5|BS=10|BS=20|BS=50|\n|-|-|-|-|-|-|-|\n|CLIPArTT|39.2|39.7|40.3|40.5|40.7|41.2|\n|WATT-S|42.1|42.4|42.7|43.2|43.9|45.0|\n|Mint|45.8|46.2|46.7|46.8|47.0|47.1|\n\n# Q1. Batch size vs. accuracy\n\nThis is because Mint leverages both a mean accumulator and a gradient accumulator, which allow it to aggregate information from all previously seen test samples, not just the current batch. As a result, **a smaller batch size does not necessarily imply \"less data\" for adaptation, and thus does not always lead to degraded performance.** And when the batch size becomes sufficiently large, Mint's performance tends to stabilize, and the small variations observed are primarily due to randomness rather than systematic effects. \n\nAs shown in Figure 6, when either the mean accumulator or the gradient accumulator is removed, the resulting ablated variants fail to fully utilize information from prior batches. In these cases, performance drops more significantly as batch size decreases, highlighting the importance of both components in maintaining robustness across batch sizes."}]

Author rebuttal:

Meta-review: This paper investigates why pretrained VLMs like CLIP are vulnerable to distribution shifts caused by input perturbations or corruptions. The authors identify what they call "variance collapse", where intra-class and inter-class variances collapse as corruption increases. Their theoretical results argue that the visual encoder tends to encode corruption-related signals, which suppresses discriminative features. They also show that maximizing inter-class variance can counteract this. 

This work has several strengths. Notably, the writing is clear and easy to follow, the work introduces and formalizes an interesting and interpretable phenomenon (variance collapse), and the proposed method (MINT) is a simple, efficient yet effective TTA algorithm. 

Several concerns were identified during the initial review stage. Primarily, questions were raised regarding some of the equations, decomposition and theoretical claims [Cqv3, EhQi]; experiments were only on corrupted benchmarks and no clean image accuracy was reported [Cqv3], questions regarding why some sate of the art methods were excluded from comparison [Cqv3],  whether variance collapse applies only to corrupted images [EhQi]; concerns regarding the novelty of variance collapse [3aNj]; lack of clarity for some of the theoretical/technical components [3aNj]; missing related works (Super-resolution, LRTK0, RobustSAM) [3aNj]; the TTA problem setting requiring *no* fine-tuning or training of any sort potentially being too restrictive and not a sufficient reason to avoid comparison with other methods [3aNj].

The authors provided substantial clarifications and new experiments during rebuttal: added results on clean datasets and natural distribution shift benchmarks; addressed concerns about Eq. (5) vs Eq. (7), explaining why their formulation avoids gradient scale decay; Expanded baselines to include online training-free TTA methods (TDA, Tent, ETA), showing Mint still performed best; cited additional related works and acknowledged overlaps with prior observations of feature collapse.

After the rebuttal, most weaknesses were adequately addressed, though some concerns remained. Primarily, reviewer 3aNj raised concerns RobustSAM, LR-TK0, and Super-resolution likewise "assume knowledge of the corruption type" and also do not "retrain the original pretrained model", and that they should be included in the empirical evaluation as "the choice of whether to use prompts, SR, or layer norms seems more like a design preference than a fundamentally different objective." Further, "not training, or training a simple classifier to inform the model about the type of noise, are simply design choices in my subjective opinion, which would have put them in direct comparison with existing methods." Thus, restricting the comparison to include only other TTA methods can be seen as an artificial limitation, as the methods cited by the reviewer are still aiming to achieve the same objective. 

While the authors have included results on a mixture of 15 types of corruptions during the rebuttal, all other experiments and empirical analysis were done on test sets that included only a single type of perturbation, which may be overly artificial. 

Other concerns surround the clarity of the mathematical formulations. The authors would be required to update their manuscript to include the clarifications made during the rebuttal.  

While I agree that methods identified by reviewer 3aNj should be included in the related work, I feel like this work offers interesting theoretical insights and proposes a simple yet sound method for easily adapting pretrained models to be more robust to corruptions. The new results added during the rebuttal, especially results on domain shifts and mixture of noise types, strengthens the work to the point where I recommend acceptance.
Decision: Accept (poster)