Statistics Caching Test-Time Adaptation for Vision-Language Models
Authors: Zenghao Guan
Zhou Yucan
Wu Liu
Xiaoyan Gu
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
CLIP
Test time
Transfer Learning
Primary area: applications

Abstract: Test-time adaptation (TTA) for Vision-Language Models (VLMs) aims to enhance performance on unseen test data. However, existing methods struggle to achieve robust and continuous knowledge accumulation during test time. To address this, we propose Statistics Caching test-time Adaptation (SCA), a novel cache-based approach. Unlike traditional feature-caching methods prone to forgetting, SCA continuously accumulates task-specific knowledge from all encountered test samples. By formulating the reuse of past features as a least squares problem, SCA avoids storing raw features and instead maintains compact, incrementally updated feature statistics. This design enables efficient online adaptation without the limitations of fixed-size caches, ensuring that the accumulated knowledge grows persistently over time. Furthermore, we introduce adaptive strategies that leverage the VLM's prediction uncertainty to reduce the impact of noisy pseudo-labels and dynamically balance multiple prediction sources, leading to more robust and reliable performance. Extensive experiments demonstrate that SCA achieves compelling performance while maintaining competitive computational efficiency.

Statistics Caching Test-Time Adaptation for Vision-Language Models
Zenghao Guan 1.233.2.3, Yucan Zhou', Wu Liu Xiaoyan Gu Gu 12,3+ Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, Staae Key Laboratory of Cyberspace Security Defense, Tianjin University, University of Science and Technology of China zenghaoguan. manghaoguan.cs@@malll com, guxiaoyan@iie ac. cn
Abstract
Test-time adaptation (TTA) for Vision-Language Models (VLMs) aims to enhance performance unseen test data. However, existing methods struggle to achieve robust and continuous knowledge accumulation during test time. To address this, we propose Statistics Caching test-time Adaptation (SCA), novel cache- based approach. Unlike traditional fcature-caching methods prone to forgetting, SCA continuously accumulates task-specific knowledge from all encountered test samples. By formulating the reuse of past features as least squares problem, SCA avoids storing raw features and instead maintains compact, incrementally updated feature statistics. This design enables efficient online adaptation without the limitations of fixed-size caches, ensuring that the accumulated knowledge grows persistently over time. Furthermore, we introduce adaptive strategies that leverage the VI.M's prediction uncertainty reduce the impact of noisy pseudo- labels and dynamically balance multiple prediction sources, leading to more robust and reliable performance. Extensive experiments demonstrate that SCA achieves compelling performance while maintaining competitive computational efficiency. The code is available at this link.
Introduction
Recently, large-scale vision-language models (VLMs), such as CLIP [1], have shown strong gen- eralization across wide range of tasks, thanks to pre-training billions of image-text pairs from the web. These models learn powerful crossl representations, enabling impressive zero-shot performance without task-specific supervision [1, 2, 3, 4]. However, despite their success, adapting VLMs to real-world tasks remains challenging. Most existing approaches rely on a small number of labeled samples from the target domain for fine-tuning [5, 6, 7]. which limits their use in scenarios where annotation is expensive or unavailable [8, 9, 10, 11]. This motivates the need for test-time adaptation (TTA) for VLMs, where the model adapts new domains the fly using only unlabeled test data, improving robustness without extra labeling cost [12, 13, 14, 15, 16].
Existing TTA methods for rision-language models (VLMs) generally fall into two categories: (1) test-time prompt tuning methods and (2) cache-based methods. Test-time prompt tuning [12, 13, 17, 18. 19] adapts the prompt to each individual sample in the test data stream using entropy minimization objective on randomly augmented samples. To avoid prompt degradation caused by errors in previous unsupervised optimization steps, these methods typically process each test sample independently and reset the model after evaluating it. These approaches, while simple, fail to leverage
"Corresponding author is Xiaoyan Gu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

initial prompt
Pi
updated prompt
Pseudo-
#olass Feature
Reset
new
Cache
INEW
Cadhee
Vo
1/1
D
1/
Aclass
Helles
False, Low entropy
#oless
High entropy
(a) Test- prompt tuning
(b) Existing cache-based TTA
Visual Feature
Cache
Accumulative Dynamic
threstoaat sslected
G Gl + x
Efficient
P
C1u-1 C1a-1 + XPP,
(c) Statistics caching TTA
statistics accumulation
dynamic pseuco laDe!
Figure 1: Comparison of our SCA (c) with test-time prompt tuning methods (a) and existing cache-based methods (b). We each test sample's feature and soft pseudo-label update the cached feature statistics, enabling continuous knowledge accumulation without forgetting.
information from previously seen samples. As result, the model may repeatedly forget useful patterns, which reduces its ability to adapt and generalize across the test set. Moreover, test-time tuning methods require backpropagation through the text encoder at every test-time training step to learn the text prompt, leading to substantial computational overhead.
Another type of work is cache-based methods [14, 20, 15, 21J. These methods use class-wise feature cache to retain information during inference. Since ground -ruth labels are unavailable, test samples assigned to cache slots based zero-shot pseudo-labels. To improve cache quality under limited capacity, low-entropy samples are retained while high-entropy ones are discarded.
bache-based methods have garnered significant attention due to their potential for knowledge accumu- lation and computational efficiency. However, critical question arises: this acclaimed knowledge accumulation process truly robust and effective over time? Since low entropy does not guarantee correctness, confidently incorrect samples may enter and persist in the cache. Previous studies have primarily highlighted this detrimental impact on knowledge quality, often reflected in decreased classification accuracy. However, they often overlook a more insidious consequence: whether and how such errors fundamentally disrupt and hinder the process of accumulating new knowledge. Here, we analyze the issue from two perspectives:
Firstly, a misclassified low-entropy sample can overwrite pre- viously correct entry the cache, leading the forgetting of useful knowledge that had already been acquired. To demon- strate this effect, we conducted controlled experiment on the Flowers 102 [22] using a SOTA cache-based method, DPE [14].
-327
(Forgett ng)
Specifically, we construct a controlled setup with two cache update orders: (1) using only set of standard samples, and (2) using the same set followed by a few deliberately selected low- entropy misclassified samples that guaranteed to enter the cache. The second sequence simulates the injection of "forget- ting triggers," allowing to measure the performance drop by directly comparing the performance before and after forgetting. As shown in Fig. DPE suffers from forgetting, which causes a decline in performance as useful information is overwritten.
Ours
6,45
DPE
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Performanc Drop (Blocking
Figure/ 2: Performance drop on Flowers using ViT-B/16. Negative values mean the performance proves instead of drops.
Secondly, a misclassified low-entropy sample, once admitted, might persistently occupy cache slot due to high confidence score. This block potentially correct but higher-entropy samples from entering the cache, thereby

---

hindering the intended knowledge accumulation and refinement. To demonstrate this, we simulated a 'bad case' scenario, where several low-entropy but incorrect samples were placed the beginning of the online test sample sequence. As depicted in Fig.2, under these conditions, the cache was consistently dominated by incorrect features, preventing effective knowledge accumulation and resulting in significant performance drop for DPE compared to a standard evaluation order.
The preceding analysis reveals that while existing TTA methods attempt to adapt, they face significant hurdles in achieving robust and continuous knowledge accumulation. Test-time prompt tuning resets for each sample, inherently discarding prior information and preventing any meaningful accumulation of knowledge from the test stream. Existing cache- based methods, while aiming for accumulation, are susceptible to forgetting due to limited cache capacity and can suffer from blocked accumulation when noisy feature- pairs lead to the entrenchment of incorrect information. This highlights a critical need for an approach that can consistently learn from all past samples and robust to pseudo-label noise.
To this end, we propose Statistics Caching test-time Adaptation (SCA), a simple yet effective cache- based test-time approach for VLM. As illustrated in Fig. 1, SCA is designed to effectively accumulate task-specific knowledge from all previously encountered test samples without the forgetting issues inherent in feature-caching methods. We achieve this by modeling the effective use of previous test sample features as a least squares problem. Crucially, solving this problem requires only the corresponding feature statistics, not the full set of historical raw features. Therefore, unlike prior cache-based methods that store limited subset of raw features, SCA maintains and incrementally updates these compact feature statistics in the cache. This core design choice enables the continuous integration of information from every incoming sample, addressing the problem of forgetting due to fixed-size feature caches and ensuring that the accumulated knowledge grows persistently over time.
Furthermore, to ensure the quality of this continuously accumulated knowledge, especially in the presence of noisy pseudo-labels, we introduce a dynamic soft pseudo-label assignment strategy. This strategy intelligently utilizes the uncertainty information from the VLM's initial predictions to generate more robust pseudo-labels for updating the feature statistics. Finally, we propose an instance-level adaptive fusion strategy that combines the cache (statistics-derived) logit and the textual (zero-shot) logit using prediction entropy. This strategy estimates the confidence of the statistics-derived classifier and adaptively adjusts its weight relative to the VLM's textual logits. As a result, it enables more robust and flexible predictions with relatively little hyperparameter tuning effort.
We evaluate SCA on 15 diverse datasets across out-of-domain and cross-domain benchmarks, where it achieves notable performance gains over state-of-the- -frte-rr- prompt tuning and cache-based methods. Moreover, SCA avoids time-consuming backpropagation and achieves over 10x computational efficiency compared to TPT [12], making it much more practical for resource-limites settings. Our contributions are summarized as follows:
We propose Statistics Caching test-time Adaptation (SCA), novel cache-based method that stores feature statistics instead of raw features, enabling effective accumulation of task-specific knowledge from all previously seen test samples without forgetting.
We introduce a dynamic soft pseudo-labeling strategy that leverages uncertainty to reduce error accumulation. Additionally, we propose instance-level adaptive fusion mechanism based on prediction entropy, enabling robust predictions without extensive hyperparameter tuning.
Extensive experimental results show that SCA achieves strong performance across 15 diverse datasets, matching surpassing tate-of-the-art methods while maintaining high computational efficiency.
2 Preliminaries
Zero-shot classification in CLIP. CLIP [1] enables zero-shot image classification via pre-trained visual (-) and textual (-) encoders mapping inputs to shared D-dimensional space RD, To classify an image into one of K classes {o1, Cx }. text prompts Tę are generated for each class
(e.g.. "a photo of (CLS]"). The image and prompts embedded:
fi ( and fn. (Tk) for 1,..,, K.
Here, fi and fr. are the image and text features, respectively. CLIP computes the cosine similarity cosine(fr, fr.) between the image feature and each text feature. These similarities determine class
3

---

probabilities using a softmax function with temperature K:
exp (cosine((fr..JI1)/)
p(c, 1)
K Σj= exp (cosine fr,.ffr) fr)/rs)
The image is assigned the class cj corresponding to the maximum probability p(cx/T).
Test-time prompt tuning. These methods [12, 13, 23, 17, 18, 19] optimize learnable prompts for each test sample. As ground-truth labels are unavailable during testing, the optimization is guided by the prediction entropy of selected sample augmentations:
K
min L(v; min
Σ k=1
pl ck A v) log = A(T), v),
(3)
where denotes the learnable prompt, A(7) represents the set of selected augmentations with lower prediction entropy for test sample I, and p(ù ck A(T), denotes the averaged predicted
probability over these augmentations.
Cache-based test-time adaptation. These methods [14, 20, 15, 21] maintain fixed-size cache of M entries per class to store features from test samples. This cache acts as an external memory, allowing the model to reference past examples during inference. By continuously updating the cache based on minimal-entropy filtering strategy, the model can gradually accumulate useful information and improve prediction consistency. Final predictions are obtained by combining zero-shot logits from the textual classifier with similarity scores from the cache:
Zk = 0egZtext + atj Zeache
M
= onf In + o2 Σ
expl 1ppl-3(1- - (T
where Zeahce denotes the logit for class k, and f: is the 7th feature of category & in the cache. Their performance relies heavily on fusion hyperparameters on and 02-
3 Statistics Caching Test-time Adaptation
To avoid forgetting of previously learned knowledge, we aim to make full use of past test samples observed up to time step guide model adaptation. Denote X1xt l:t E R= xd as the features extracted from all test samples up to time t, and Y E [ as their corresponding one-hot encoded ground-truth labels. Here, Nia represents the cumulative number of test samples up to time We make full use of historical samples by formulating it as a least squares problem:
min X 1:t W, Y W W:
(5)
Here, we use y to control the strength of the regularization. The closed-form solution of Eq. is:
W. X 11 + - 1:t.
(6)
W, denotes the classifier learned by leveraging the feature representations of all previously observed test samples up to time f. It captures accumulated knowledge across test instances, enabling more informed and robust predictions.
However, in practical test-time scenarios, storing historical sample features X 1s is typically infeasible due to increasing storage overhead potential privacy concerns. Moreover, ground-truth labels Y 1:t are not accessible during testing. Existing methods typically rely on CLIP's zero-shot predictions to assign ascudo-labels for cache construction. However, these pscudo-labels are often noisy, leading to the inclusion of unreliable feature-labe pairs that degrade the quality the updated classifier. To tackle these challenges, propose the following parts.
Statistics Accumulation. In fact, computing W, does not require storing the growing feature matrix X l:r as test samples accumulate. Instead, it is sufficient to maintain two compact feature statistics
that are independent of the sample size: G = I X1x 1:t E Rddd and C X 1.r Y,, 1:s E R Gram matrix G1. represents the second-order feature statistics, capturing the pairwise relationships between the features, and Ct:: represents the first-order statistics, which is essentially the weighted

---

Cache
Image Encoder
Visual Feature
Statistics A
Logit
Adaptive Fusion
Textual Logit
Text Encoder
A photo of [class name]
Textual
Classifier
Final Logit
Figure 3: Framework of our SCA. Each test sample updates the feature statistics in the cache. Based on these statistics, we build a closed-form classifier that captures knowledge from all test samples. Finally, apply instance-level adaptive fusion strategy to combine the textual and cache logits.
sum of the features and the labels. Due the linear characteristics of these feature statistics, update them additively to support online updates during test-time scenarios:
G G + x, X,, Cla = C1+-1 X, Y.
Here, X, E R xd and Y, E I denote the features of the current test sample and its corre- sponding label, respectively. Finally, these two statistics are then used compute W, , as follows:
W, (G + + C
(8)
This approach efficiently accumulates knowledge and builds a strong classifier that captures informa- tion from all previously samples.
Dynamic soft pseudo-label assignment. The above discussion assumes access to true labels when accumulating feature statistics. In practice, we typically rely on pseudo-labels generated by CLIPPss zero-shot predictions to update C1a- common but problematic practice [14, 15] to convert these predictions into hard pscudo-labels by selecting the single most confident class ground truth. This approach ignores the nuance of the full probability distribution, collapsing all uncertainty into a binary decision, and can amplify errors when the top prediction is only marginally more probable or even incorrect [24]. A continuous stream such errors will inevitably skew the learned cache classifier and degrade its accuracy.
To mitigate this issue, we dynamically generate the soft pseudo-label Y, for each test sample X based on its predicted class probabilities Prett from the CLIP textual (zero-shot) classifier. Specifically, we first identify candidate set of classes based on each sample. Without loss of generality, we (2)
( are sorted descending order. In this case, Piext denotes the probability of the class initially deemed most likely by the zero-shot classifier.
assume the class probabilities Piext (Piext> Piex:
(1)
We select the smallest number of classes, denoted by Ω such that their cumulative probability meets or exceeds threshold r:
Ω min
Σ
(k) Piext 2 7(1 ((a))
U
Then the soft pseudo-label Y, is generated as:
(K) Piexi
I[(k E 22)
(k)
exp
(10)
Eien exp Pixt
(1)
By allowing |S2] vary with and the sample's confidence, our method produces softer pseudo-labels for uncertain samples and sharper labels when the model is confident. This dynamic assignment strategy better captures the sample-specific uncertainty inherent the zero-shot predictions and effectively curbs the overconfidence often associated with hard labeling schemes. Finally, the pseudo- label Y, is used to update the first-order statistics Č C shown in Eq.7. The cache classifier W, is then computed Eq.8 to obtain the cache logits Zeache X, W..
5

---

Table 1: Experimental results on the cross-domain benchmark with two backbones of CLIP. The best and second-bes best results are shown in bold and underlined, respectively.
Method
Aircraft
Caltech
Cars
DTD
EurSAT
Flower Food101
Pets
SUN397
UCF101
Average
ILIP-ResNet-S0
15.66
85.88
55.70
40.37
23.69
61.75
73.97
83.57
58.80
58.84
55.82
Ensemble
16.11
87.26
55.89
40.37
25.79
62.77
74.82
82,97
60.85
59.48
56.63
CoOp (5) [BCV
15.12
86.53
55.32
37.29
26.20
61.55
75.59
87.00
58.15
59.05
56.18
CoCoOp [26] (CVPR223
14.61
87,38
56.22
38.53
28.73
65.57
76.20
88.39
59.61
57.10
57.23
TPT [12) (2
17.58
87.02
58.46
40.84
28.33
62.69
74.88
84.49
61.46
60.82
57.66
Di [13] (CCVV233
17.60
86,89
60.71
40.72
41.04
63.53
79.21
83,40
62,72
62.67
59.85
TDA [14] [C]PR'24]
17.61
89.70
57.78
43.74
42.11
68.74
77.75
86,18
62.53
64.18
61.03
DPB [15] [NeudPS'244
19.80
90.83
59.26
50.18
41.67
67.60
77.83
85,97
64.23
61.98
61.93
BCA [27] [CVPR'25)
19.89
89.70
58.13
48.58
42.12
66.30
77.19
85,58
63.38
63,51
61.44
SCA (Ours)
20.82
91.03
59.73
53.31
43.33
71.09
76.51
86.15
64.50
66.40
63.29
BLT--V
23.67
93.35
65.48
44.27
42.01
67.44
83.65
88.25
62.59
65.13
63.58
Ensemble
23.22
93.55
66.11
45.04
50.42
66.99
82.86
86.92
65.63
65.16
64.59
CoOp [5] [UCV
18.47
93.70
64.51
41.92
46.39
68.71
85,30
89.14
64.15
66.55
63.88
CoCoOp (26) [C)PR/22]
22.29
93.79
64.90
45.45
39.23
70.85
83.97
90.46
66.89
68.44
64.63
TPT [12] (NeurPP5222
24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
Di [13] [CCV1231
25.60
92.49
67.01
47,00
43.13
70.10
87.23
88.22
65.74
62.67
65.47
TDA [14) [C)PR/24]
23.91
94.24
67.28
47,40
58.00
71.42
86.14
88.63
67.62
70.66
67.55
DPE [15) [NcudPS'20 24]
28.95
94.81
67.31
54.20
55.79
75.07
86.17
21.14
70.07
70.44
69:40
Dynaprompt [23] (ICLR22)
24.33
94.32
67.65
47.96
42.28
69.95
85,42
88.28
66.32
68.72
65.52
BCA [27] (CVPR'25)
28.59
94.69
66.86
53.49
56.63
73.12
85.97
90.43
68.41
67.59
68.59
SCA (Ours)
28.50
94.85
68.49
57.09
57.16
76.09
86.09
91.44
70.27
73.43
70.34
Instance-level Adaptive Fusion. Previous methods [25, 15, 20] fuse zero-shot (textual) logit and cache logit using fixed dataset-level coefficient, which typically requires extensive hyperparameter tuning on the validation set or even the test set. In contrast, we propose an instance-level fusion strategy that leverages entropy to estimate classifier confidence and adaptively weights the logits, enabling more robust and adaptive predictions.
For each test sample, we compute the prediction entropy for each classifier and denote them as Heaehe and H Lower entropy indicates higher confidence, so we derive confidence weights follows:
[a 222 = softmax( -B [Heache: Htexl),
(11)
where 3 controlling the sharpness of the distribution. This enables the model to evaluate their relative confidence, relying more on the zero-shot model when the cache model is uncertain and shifting toward the cache model as its confidence increases. Entropy is used it is the most common confidence metric. We then compute the aggregated logits as:
Zfinll 01 Zeache a2 Ztext
(12)
By automatically and adaptively adjusting the fusion coefficient for each test sample, our method captures instance-level uncertainty and improves robustness across diverse data distributions. We also provide an option to amplify the cache logits using the same sharpness parameter 3 in a divisive manner. Using default settings (B 0.5) gives strong performance across multiple datasets. In other words, a single set of hyperparameters can be applied to fuse predictions across different datasets, reducing the effort required to tune the fusion parameters @1, 02
4 Experiments
4.1 Experimental Settings
Datasets.
Following prior work [12, 13, 15, 27], we evaluate our method using two established
benchmarks: the cross-domain benchmark and the dutto--distiibution (OOD) benchmark. (1) The cross-domain benchmark assesses the model's 's ability to generalize across various domains, each with its own set of classes. This benchmark includes 10 diverse image classification datasets from distinct domains: FGVCAircraft [28], Caltech101 [29], StanfordCars [30], DTD [3]]. EuroSAT [32],

---

Table 2: Experimental results the OOD benchmark with two backbones of CLIP. The best and second-bes results are shown in bold and underlined, respectively.
Method
ImageNet
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-S
Average
Average
CLIP RelP-ResNet-50 [1]
58.16
21.83
51.41
56.15
33.37
44.18
40.69
Ensemble
59.81
23.24
52.91
60.72
35.48
46.43
43.09
CoOp [5] [IICV222]
63.33
23.06
55.40
56,60
34.67
46.61
42.43
CoCoOp [26] (CVPR'22]
62.81
23.32
55.72
57.74
34.48
46.81
42.82
TPT [PT[12] [NeurPS
60.74
26.67
54,70
59.11
35.09
47.26
43.89
DiTTPT [13] [ICCV123]
60.80
31,06
55.80
58.80
37.10
48.71
45.69
TDA [14] [CVPR'24]
61.35
30.29
55.54
62.58
38.12
49.58
46.63
DMN ZS [20] (CVPR'24)
63.87
28.57
56.12
61.44
39.84
49.97
46.49
DPE [15] 24c
63.41
30.15
56.72
63.72
40.03
50.81
47.66
BCA [27) [CVPR'25]
61.81
30.35
56.58
62.89
38.08
49.94
46.98
SCA (Ours)
63.11
30.64
56.51
62.85
39.33
50.49
47.33
CLIP-VYT-B/10 [I]
66.73
47.87
46.09
59.11
57.20
Ensemble
68.34
49.89
77.65
48.24
61.20
59.42
CoOp [5] [UCV/22]
71.51
49.71
64.20
75.21
47.99
61.72
59.28
CoCoOp (26) [CVPR
71.02
50.63
64.07
76.18
48.75
62.13
59.91
TPT [12] [NcudPS
68.98
$4.77
63.45
77.06
47.94
62.44
60.81
DifTTT [13] [ICCV233]
70.30
55.68
65.10
75.00
46,80
62.28
60.52
TDA [14] [CVPR]22]
69.51
60.11
64.67
80.24
50,54
65.01
63.89
DMN-ZS [20] [CVPR/24]
72.25
58.28
65.17
78.55
$3,20
65.49
63.80
DPE [15] (44
71.91
$9.63
65.44
80.40
52.26
65.93
64.43
Dynaprompt (23) [[CLR
69.61
56.17
64.67
78.17
48.22
63.37
61.81
BCA (27) [CVPR/25]
70.22
61.14
64.90
80.72
50.87
65.37
64.16
SCA (Ours)
71.75
60.33
52.50
66.16
Flowers 102 [22], Food101 [33], OxfordPets [34], SUN397 [35]. and UCF101 [36]. (2) The OOD benchmark measures the model's 's robustness against natural shifts data distribution. evaluates performance ImageNet [37] and four challenging variants: ImageNet-A [9], ImageNet- V2 [10], ImageNet-R [38], and ImageNet-Sketch [39].
Implementation Details. Consistent with prior work [15, 25], our experiments adopt ResNet-50 [40] and VIT-B/16 [41] ]] as the visual encoders for CLIP, with a batch size of to satisfy online processing requirements. For textual prompts, employ the prompt ensembling strategy as used in previous works [15, 42, 20, 25]. For data augmentation, we adopt the approach from DPE [12], generating 63 randomly resized crops per test image for the OOD benchmark. No data augmentation is applied in the cross-domain benchmark. By default, we set the ridge coefficient y to le4, threshold 7 to 0.1, sharpness coefficient / to 0.5. All experiments are conducted using top-1 accuracy to measure classification performance.
single NVIDIA A100 RTX GPU,
Baselines. We compare our method with zero-shot CL.IP using either simple prompt prompt ensemble, few-shot methods such as CoOp [5], with 16-shot annotated samples per class, alongside the following established CLIP test-time adaptation approaches: (I) prompt-tuning methods:
TPT [12], DifTTTT [13], Dynapromp((23): (2) cache-based methods: TDA [14]. DMN-ZS [20], DPE [15]. We also include recent baseline BCA [27] that adapts likelihood using Bayes rules. To ensure a fair comparison, we exclude RLCF [43] and COSMIC [21], which rely on auxiliary strong models (e.g., ViT-L-14) during test-time adaptation. We also exclude some transductive TTA baselines [44, 45] that rely on utilizing inter-sample relations within a large batch. The results of these baselines directly taken from their respective original papers.
4.2 Results and Discussions
Cross- Dooss-Domain Benchmark. Table presents results on the cross-domain benchmark. While CoOp [5] and CoCoOp [26] improve CLIP's generalization, they rely on training-based adaptation with labeled data, which restricts their practical applicability. In contrast, our SCA outperforms CoOp by an average of 7.11% 11% and 5.71 CoCoOp by an average of 6.06% and 6.46% on two backbones in training- free manner, demonstrating superior generalization without relying on labeled examples. Compared to test-time prompt tuning methods, SCA achieves 4.82% gain in average accuracy over the strongest baseline, Dynaprompt [23]. It also consistently outperforms cache-based approaches, surpassing TDA [14] by 2.25% and 2.81 %, and DPE [15] by 1.36% and 0.94% on ResNet-50 and ViT-B/16 backbones, respectively. Notably, unlike existing cache-based methods [15, 14] that rely
7

---

on dataset-specife fusion hyperparameters (a1. a2) for optimal performance. our method uses the pair, while different benchmarks or
same configuration for all datasets within a backbones use different configurations, highlighting its strong generalization.
OOD Benchmark. In Table 2, we further evaluate the generalization capability of our proposed method on the OOD benchmark by comparing it with state-of-the approaches. Our proposed method consistently outperforms the CoOp [5] and CoCoOp [26] with substantial improvements across all datasets. Specifically, it improves average accuracy by about 5% on both backbones. These results demonstrate the strong generalization capability of our approach and its effectiveness in enhancing performance without any additional supervised training. On the ResNet-50 backbone, our method performs slightly below the strongest baseline DPE [15]; however, achieves this level of performance without any training overhead, unlike DPE, which requires additional training. In contrast, on the ViT-B/16 backbone, SCA consistently outperforms all existing methods, with average accuracy improvements ranging from 0.23% to 3.72% across various OOD datasets.
Computation Efficiency. We measure time per image for several baselines on SUN397 (19,850 samples) using single 80 GB NVIDIA RTX A100 GPU as an example of computational efficiency. Table 3 presents the corresponding results. Since our proposed SCA avoids consuming backpropa- gation, it offers a notable advantage in computational efficiency compared to prompt tuning methods. Specifically, our approach is over 10x faster than TPT [12]. Compared to the cache-based methods TDA [14], our approach is slightly less efficient but delivers notable performance gains, especially on the cross-domain benchmark.
Table 3: Computation efficiency compari- son on SUN397 using VIT-B/16.
Method
fref
Time Per Image (s)
CLIP [1]
V
0.007
TPT [12]
X
0.218
TDA [14]
V
0.009
DPE [15]
X
0.043
SCA (Ours)
0.012
Storage Overhead. Previous cache-based methods that store feature-label pairs with fixed size M have a storage cost of M K d. Our cached feature statistics accumulate knowledge from all samples and require (K + d) x d storage. While this may be less efficient when the number of classes K is small, our method offers lower storage overhead as K grows, for example on large-scale datasets like ImageNet.
4.3 Ablation Studies
Effectiveness on statisties accumulation and dynamic soft label assignment. To investigate the roles of our statistics accumu- lation and dynamic soft label assignment, we report the performance on 10 fine-grained recognition datasets involved in cross-domain datasets using VIT-B/16 in Table 4. [W/o statistics accumulation] means that we follow [14, 15] by using fixed-size cache of M feature-label pairs per class to build our classifier (Eq. 6). [W/o dynamic soft label assignment] means that we use hard pseudo-labels for statistics accumulation. We have the following key observations: (1) Replacing statistics accumulation with direct feature caching leads to performance drop compared to our method. While increasing the cache size gradually improves performance, the gains eventually plateau. Interestingly, when the cache size reaches 8, outperforms the state-of-the-arr baseline DPE [15] (69.40). This improvement stems from our use of second-order feature information (the Gram matrix G) to construct the cache-based classifier, whereas prior cashe-based methods rely solely on first-order information. Studies other areas of machine learning similarly conclude that second-order feature statistics significantly enhance performance [46, 47, 48, 49]. (2) Using hard labels to update statistics leads to a performance drop. This highlights the importance of the extra information in soft labels and demonstrates the effectiveness of our dynamic label assignment.
Table 4: Effectiveness on statisties accumulation and dynamic soft label assignment. We report the averaged results on 10 datasets in cross-domain benchmark.
Method
Avg
w/o statisties accumulation (M 4)
69.31
w/o statistics accumulation (M = -8)
69.47
w/o statistics accumulation (M -16)
69.75
w/o statistics accumulation (M = 32)
69.72
w/o dynamic soft label assignment
69.32
with both
70.34
Ablations on fusion coefficients. In our SCA, the fusion coefficient is automatically adjusted for each test sample. We perform an ablation study by manually setting a fixed fusion coefficient, varying a from 0.1 to 0.9. As shown in Fig. 4, the optimal fusion value differs across datasets. While the
8

---

adaptive coefficients may not be optimal for every case, they consistently outperform most fixed coefficients across different datasets. Compared some rache-based methods [14, 15] that require costly parameter searches to tune fusion hyperparameters for each dataset, our approach uses single set of hyperparameters for all datasets to fuse the prediction and still deliver strong performance. This achieves a good trade-off between performance and tuning effort.
Norma Mizealizzedao Accuracy
0.8
0.6
0.4
0.2
Aircratt
Caltech
Cars
DTD
EUroSAT
Flower
Food101
Pets
SUN397
UCF101
Figure 4: Normalized accuracy with varying fusion parameters on ViT-B/16.
Accuracy 190)
70.35
Accuracy 199
70
70.2
70.30
70.0
70.25
Sample
Figure 5: Left: hyperparameter analysis of sharpness coefficient 3. Middle: hyperparameter analysis of threshould Right: sensitivity analysis of test-time sample order.
Hyperparameter Analysis. Here, we present the results of hyperparameter analysis for the threshould and the sharpness coefficient 3 on 10 fine grained recognition datasets. As shown in Fig. 5 (Left and Middle), the performance remains relatively stable when the parameters are set within a reasonable range. This suggests that our method is not sensitive to these hyperparameters, making it more robust and easier to apply in practice.
Sensitivity to test time sample order. Since our SCA is designed for online test-time adaptation, its performance may affected by the order in which test samples are encountered. To assess this sensitivity, we conduct experiments on 10 datasets from the cross-domain benchmark, each with varying test-time sample orders. As shown in Fig. 5 (Right), our method demonstrates stable perfor- mance across different orders and consistently outperforms the state-of-the-art bascline DPE [15]. highlighting its robustness to input sequence variations in online settings.
Related Work
Test-time prompt tuning. These methods tune the text prompts using the test samples. TPT [12] optimizes the prompt by minimizing prediction entropy to promote consistency across augmented views of the same input. DiffTPT [13] follows the TPT's framework and uses a pre-trained diffusion model to generate more diverse augmentations. C-TPT [18] optimizes prompts by minimizing calibration error, while RLCF [43] introduces an additional CLIP model reward signal to guide test-time prompt tuning. Despite their effectiveness, they typically treat each test sample in isolation, resetting the model for every instance without leveraging information from previously seen samples. To effectively accumulate previously learned knowledge, it is essential to incorporate cache module. For instance, Dynaprompt [23] uses a prompt buffer and introduces a dynamic selection strategy to update the appropriate prompt stored in the buffer. Although this approach facilitates the accumulation of historical knowledge, it, like other test-time tuning methods, requires backpropagation through the text encoder at each test-time step, resulting in significant computational overhead.
Cache-based test-time adaptation. These methods store high-confidence visual features during test time and utilize them to provide extra information for current prediction. TDA [14] and DMN [20] establish visual caches from test data and pseudo-labels to retrieve class prototypes during test time. Similarly, DPE [15] maintains a dynamic cache that facilitates the training of residual dual prototypes. COSMIC [21] enhances the semantic diversity of pseudo-labels by leveraging fine-
grained visual features from DINOv2. These methods demonstrate high computational efficiency and

---

strong performance due to the rich information provided by features stored in the cache. However, they suffer from forgetting and blocking issues during knowledge accumulation. Additionally, fusing the predictions derived from the cache features with the VI.M's zero-shot predictions often requires extensive and careful tuning. Our SCA overcomes these challenges through statistics accumulation combined with dynamic label assignment and fusion strategy.
6 Conclusion
We propose Statistics Caching test-time Adaptation (SCA), cache-based method that stores feature statistics instead of raw features, allowing effective accumulation of task-specific knowledge without forgetting. We also introduce a dynamic soft pseudo-labeling strategy that uses uncertainty to reduce error accumulation, and an instance-level adaptive fusion mechanism based on prediction entropy for robust predictions without heavy hyperparameter tuning. Experiments on 15 diverse datasets show that SCA matches or exceeds state-of -fate-of-the-art performance with high computational efficiency.
While our method enables test-time accumulation of task-specific knowledge without training and achieves strong performance, it still shares some limitations with existing methods. Firstly, storing feature statistics introduces some storage overhead. Secondly, our method only mitigates error accumulation during test time to a certain extent, and the performance still falls short of the ideal setting where ground-truth labels available to build the classifier. Morcover, similar to existing CLIP TTA methods, our approach mainly focuses on the single-domain scenario, where all test samples come from single dataset. Extending CLIP-based TTA to the multi-domaie setting where test samples from multiple domains are mixed is an important yet underexplored direction. In such cases, naive accumulation may even lead to negative effects due to domain discrepancies. Since feature representations from different domains often exhibit distinctive patterns, a promising direction to identify each sample's s domain and accumulate feature statistics separately for each domain. In future work, we plan to further investigate more effective ways to leverage feature statistics and develop improved strategies for robust knowledge accumulation across domains.
Acknowledgements
This work is supported by the Chinese Academy of Sciences under grant No. XDB0690302, the National Key Research and Development Program of China (NO. 2024YFE0203200), and the National Nature Science Foundation of China (NO. U24A20329).
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark al., "Learning transferable visual models from natural language supervi- sion," in International Conference on Machine Learning. PMLR, 2021, pp. 8748-8763, 1, 3, 7.8
[2] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, "Coca: Contrastive captioners are image-text foundation models." " Transactions on Machine Learning Research, 2022. [Online]. Available:
1
[3] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, "Lit: sero-shot transfer with locked-image text tuning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 123 18 133.
[4] Y. Li, F. Liang, L. Zhao, Y. Cui. W. Ouyang, J. Shao, F. Yu, and J. Yan. "Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm." " in International Conference on Learning Representations, 2022. [Online]. Available: http://opennevvee.eefffommm htt.://wwweeeoooommd//iiilli.e.
[5] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Learning to prompt for vasion-language models," International Journal of Computer Vision, vol. 130, no. 9, pp. 2337-2348, 2022. 1, 6, 7,
[6] R. Zhang. W. Zhang, R. Fang, P. Gao, K. Li, Dai, Y. Qiao, and H. Li, "Tip-adapter: Training- free adaption of clip for few-shot classification," European Conference on Computer Vision. Springer, 2022, pp. 493-510.
10

---

[7] T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang, "Task residual for tuning vision- vision-langue language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 10 (0899-10909)
[8] D. Hendrycks and T. Dietterich, "Benchmarking neural network robustness to common corruptions and perturbations," in International Conference on Learning Representations, 2019. [Online]. Available:
[9] D. Hendrycks, Zhao, Basart, Steinhardt, and D. Song, "Natural adversarial examples." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, 2021, pp. 15 262-15 271. 7,7, 13, 16
[10] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, "Do imagenet classifiers generalize to imagenet?" in International Conference on Machine Learning. PMLR, 2019, pp. 5389-5400.
15, 16
[11] C. Zhang. S. Stepputtis, J. Campbell, K. Sycara, and Y. Xie, "Hiker-sgg: Hierarchical knowledge enhanced robust scene graph generation." " Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2024, pp. 28 233-28245
[12] M. Shu, W. Nie, D.-A. Huang. Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao, "Test-time prompt tuning for zero-shot generalization in vision-language models," Advances in Neural Information Processing Systems, vol. 35, pp. 14 242-4-14 289, 2022. 1,3, 4, 6, 8, 9, 16, 17
[13] C.-M. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo, "Diverse data augmentation with diffusions for effective test-time prompt tuning," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2704-2714. 1, 4, 6, 7, 16
[14] A. Karmanov, D. Guan, S. Lu, A. E. Saddik, and E. Xing, "Efficient test-time adaptation of vision-language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 2, 4, 5, 6, 7, 8, 9, 14, 16
[15] C. Zhang, S. Stepputtis, K. Sycara, and Y. Xie, "Dual prototype evolving for test-time gen- cralization vision-language models," Advances in Neural Information Processing Systems, vol. 37, pp. 32 111-32 136, 2024. 2, 4, 5, 7, 8, 9, 14, 16, 17
[16] Z. Han, Yang, G. Wang, Li, Q. Xu, M. Shou, and C. Zhang, "Dota: Distributional test-time adaptation of vision-language models," arXiv preprint arXiv:2409. /9379, 2024. 1
[17] J. Abdul Samadh, M. H. Gani, N. Husscin, M. U. Khattak, M. M. Naseer, F. Shahbaz Khan, and S. H. Khan, "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization," Advances in Neural Information Processing Systems, vol. 36, pp. 80 80396-80 80 413, 2023. 1,4
[18] H. S. Yoon, E. Yoon, J. T. J. Tee, M. Hasegawa-Johnson. Y. Li, and C. D. Yoo, "C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion," arXiv preprint arXiv:2403. 14/19, 2024. 1, 4,9
[19] D.-C. Zhang, 7. Zhou, and Y.-F. Li, "Robust test-time adaptation for zero-shot prompt tuning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 15, 2024, pp. 16714-1677 722. 1,4
[20] Y. Zhang, W. 7hu, Tang, 7. Ma, K. Zhou, and Zhang, "Dual memory networks: A versatile adaptation approach for vision-language models." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28 718-28 728. 2, 4, 6, 7, 9, 14, 16, 17
[21] F. Huang, J. Jiang, Q. Jiang, H. Li, F. N. Khan, and Z. Wang, "Cosmic: Clique-oriented semantic multi-space integration for robust clip test-time adaptation," arXiv preprint arXiv:2503.23:: 2025. 2, 7.9
[22] M.-F. Nilsback and A. Zisserman, "Automated flower classification over large number of classes," Indian Conference on Computer Vision, Graphics and Image Processing. IEEE, 2008, pp. 722-729, 2, 7, 15, 16
11

---

[23] Z. Xiao, S. Yan, J. Hong. J. Cai, X. Jiang, Y. Hu, J. Shen, C. Wang, and C. G. Snoek, "Dy- naprompt: Dynamic test-time prompt tuning." in The Thirteenth International Conference on Learning Representations. 4, 6, 7,9, 16
[24] J. Zhang, Q. Wei, F. Liu, and L. Feng, "Candidate pseudolabel learning; Enhancing vision- language models by prompt tuning with unlabeled data," in Forty-first International Conference on Machine Learning. 5
[25] Z. Deng, Z. Chen, S. Niu, T. Li, B. Zhuang, and M. Tan, "Efficient test-time adaptation for super- resolution with second-order degradation and reconstruction," Advances in Neural Information Processing Systems, vol. 36, pp. 74 171667777777 701, 2023. 6,
[26] K. Zhou, Yang. C. C. Loy, and Z. Liu, "Conditional prompt learning for vision-language mod- cls." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022, pp. 16 16816-166 825. 6, 7, 8
[27] L. Zhou, M. Ye, N. Li, X. Zhu, L. Deng, H. Liu, and Z. Lei, "Bayesian test-time adaptation for vision-language models," arXiv preprini arXiv:2503.092455 2025. 6, 7, 16
[28] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, "Fine-grained visual classification of aircraft," arXiv preprint arXiv:1306.55 5151, 2013. 6, 15, 16
[29] L. Fei-Fei, R. Fergus, and P. Perona, "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories," Computer Vision and Image Understanding, vol. 106, no. I, pp. 59-70, 2007. 6, 15, 16
[30] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, "3d object representations for fine-grained categorization," in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2013, pp. 554-561. 6, 15, 16
[31] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures in the wild," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2014, pp. 3600-3613 6, 15, 16
[32] P. Helber, B. Bischke, A. Dengel, and D. Borth, "Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217-2226, 2019. 6, 15, 16
[33] L. Bossard, M. Guillaumin, and L. Van Gool, "Food-101 101-mining discriminative components with random forests," in European Conference on Computer Vision. Springer, 2014, pp. 446-461. 7, 15, 16
[34] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, "Cats and dogs," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2012, pp. 3498-3505, 7, 15, 16
[35] Xiao, J. Hays, K. A. Fhinger, A. Oliva, and Torralba, "Sun database: Large-scale scene recognition from abbey to z0o," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2010, pp. 3485-3492, 7, 15, 16
[36] K. Soomro, videos in the wild," " arXiv preprint 12Xiv:1212.04400 2012. 7, 15, 16
Zamir, and M. Shah, "Ucf101: A dataset of 101 human actions classes from
[37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2009, pp. 248-255. 7, 15, 16
[38] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo et al., "The many faces of robustness: A critical analysis of -ut-of-distribution generalization," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1340-8349, 7, 15, 16
[39] H. Wang. S. Ge, Z. Lipton, and E. P. Xing, "Learning robust global representations by penalizing local predictive power," in Advances Neural Information Processing Systems, vol. 32, 2019, pp. 10506-10518. 7, 15, 16
12

---

[40] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. 7
[41] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2021. [Online]. Available: https://opmmiiiiiwooo/ooouuiittd
7
[42] S. Pratt, I. Covert, R. Liu, and A. Farhadi, "What does a platypus look like? generating customized prompts for zero-shot image classification," in Proceedings of the IEEE/CVF
International Conference on Computer Vision, 2023, pp. 15 691-15 701.
[43] S. Zhao, X. Wang, L. Zhu, and Y. Yang, "Test-time adaptation with CLIP reward for zero-shot generalization in rision-language models," 2024. 7,
[44] C. Zhang, K. Xu, Z. Liu, Y. Peng, and Zhou, "Scap: Transductive test-time adaptation via supportive clique-based attribute prompting," arXiv preprint arXiv:2503.12866, 2025. 7
[45] M. Zanella, C. Fuchs, C. De Vlceschouwer, and B. Ayed, "Realistic test-time adaptation of vision-language models," arXiv preprint arXiv:2501.33729 2025. 7
[46] H. Li, Y. Zhou, X. Gu, B. Li, and W. Wang, "Diversified semantic distribution matching for dataset distillation." " in Proceedings of the 32nd ACM International Conference on Mullimedia, 2024, pp. 7542-7550.
[47] Z. Guan, Y. Zhou, and X. Gu, "Capture global feature statisties for one-shot federated learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 16, 2025, pp. 16 942-16 950. 8
[48] M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. Van den Hengel, "Ran- pac: Random projections and pre-trained models for continual learning," Advances in Neural Information Processing Systems, vol. 36, pp. 12 12022-12 12 0233, 2023. 8
[49] Z. Guan, G. Zhu, Y. Zhou, W. Liu, W. Wang, J. Luo, and X. Gu, "Stsa: Feder- ated class-incremental learning via spatial-temporal statistics aggregation," arXiv preprint 2rXiv:2506.01327, 2025. 8
[50] D. Osowiechi, M. Noori, G. Vargas Hakim, M. Yazdanpanah, A. Bahri, M. Cheraghalikhani. S. Dastani, F. Beizace, I. Ayed, and C. Desrosiers, "Watt: Weight average test time adaptation of clip." Advances in neural information processing systems, vol. 37, pp. 48 015 48 044, 2024. 16
[51] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, "Tent: Fully test-time adaptation by entropy minimization," in International Conference on Learning Representations. 16
[52] S. Cui, J. Xu, Y. Li, X. Tang, J. Li, J. Zhou, F. Xu, F. Sun, and H. Xiong, "Bayestta: Continual- temporal test-time adaptation for iision-languee models via gaussian discriminant analysis," " arXiv preprint arXiv:2507.08600 2025. 17
[53] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, "Maple: Multi-modal prompt learning," " in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, 2023, pp. 19 11 113-19 122. 17
13

---

Statisties Caching Test-Time Adaptation for Vision-Languaae Models
Appendix
A Details on forgetting and blocking issues
Since low-entropy zero-shot predictions not always correct (as shown in the following figure), they can lead to forgetting and blocking issues that hinder the accumulation of new knowledge in existing cache-based methods [14, 15, 20J. We conduct experiments illustrate this (Figure 2 in the Introduction). Below, we present the experimental settings and analysis detail.
Entropy: 2.7949
Predict:
Probability
0.2
Truth:
0.1
0.0
18
Class Index
Entropy: 2.2227
Predict: 15
Probability
0.3
Truth:
0.2
0.0
40
Class Index
Figure AI: An example of wrong high confidence sample.
Forgetting: misclassified low-entropy sample can overwrite previously correct entry in the cache, leading to the forgetting of useful knowledge that had already been acquired. To better illustrate forgetting issues, we conduct an additional set of control experiments. Firstly, we identify small set of "forgetting trigger samples." For each dataset, we randomly select 10 classes and choose the single lowest-entropy misclassified sample for each class. To ensure these 10 trigger samples are admitted by the cache, we filter out any correctly classified samples from the entire test set that have even lower entropy. From the remaining pool of test samples, we randomly draw 500 samples to serve as our fixed "final test samples." All other remaining samples are designated as "initial samples," which allow the cache to accumulate sufficient class-specific knowledge. We then consider the following two sequences for updating the model's cache:
Sequence A: [initial samples]
Sequence B: [initial samples, forgetting trigger samples]
The forgetting trigger samples, with their low entropy, designed to infiltrate the cache built from the initial samples, causing it to forget previously learned information. other words, the cache updated with Sequence A represents the state before forgetting, while the cache updated with Sequence represents the state after forgetting. To quantify the impact of forgetting, we measure the performance difference between Sequence A and Sequence B on the final test samples, where the performance drop is defined as the accuracy of Sequence minus the accuracy Sequence B. The results summarized in the following figure.
We have several observations: (1) The SOTA baseline DPE highly vulnerable to forgetting. (2) SCA is inherently robust to forgetting, as its accumulation design prevents error samples from fully
14

---

DPE
Perforancc Drop (1)
1.03
0.24
0.35
0.33
0.47
0.344
0.00
0.04
0.12
0.49
0.12
5.08
0.00
10.0
0.24
0.27
Ainrraft
Cattech
Cars
DTD
EurosAr
Flower
Food101
Pets
SUN
UCF101
Sum
Figure A2: Forgetting issues.
overwriting the stored knowledge. (3) Remarkably, SCA even leverages these "forgetting triggers" to improve performance (a negative drop), turning vulnerability into strength.
Blocking: misclassified sample with low entropy can, once inserted into the cache, occupy a cache slot persistently due to its high confidence score. This prevents correct but higher-entropy samples from being stored, thereby impeding subsequent knowledge accumulation and refinement. To illustrate this effect, simulate blocking cases by randomly selecting 20% of all classes and, for each selected class, identifying the single misclassified sample with the lowest entropy. These samples are placed the start of the test sequence, and the performance drop is measured against the standard evaluation order. As shown in the figure below, the performance degradation caused by blocking is considerably smaller for SCA than for DPE. This advantage arises because SCA's statistics- based accumulation mechanism is not constrained by fixed cache slots, thus avoiding the persistence of early high-confidence errors and enabling effective learning from later samples.
DPE
Performance Drop (4)
2.0
1.58
1.5
1,26
A61
6.23
0.03
0.0
Aircralt
Caltech
Cars
DTD
EurSAT
Flower
Food101
Pets
SUN397
UCF101
Avettge
Figure A3: Blocking issues.
B Experiment
B.1
Details on Dataset
We evaluate our methods on 15 datasets, including 10 [29], DTD [31], EuroSAT [32], FGV-
CAircraft [28], Flowers 102 [22], Food101 [33], OxfordPets [34], StanfordCars [30]. SUN397 [35].
UCF101 [36], ImageNet [37], ImageNet-V2 [10], ImageNet-Sketch [39], ImageNet-A [9], and
ImageNet-R [38]. Table B.1 provides detailed statistics for each dataset, including the number of
15

---

classes, the sizes of the training, validation, and testing sets, as well as the corresponding original task for each dataset.
Table B1: Summary of the 15 image classification datasets used in experiments. The last four ImageNet variant datasets are designed for evaluation only and contain no training or validation splits.
Benchmark
Dataset
Classes
Splits
Task
frain
vel
Caltech101 [29]
100
4,128
1,649
2,465
Object recognition
DTD [31]
47
2,820
1.128
1,692
Texture recognition
EuroSAT [32]
10
13,500
5,400
8,100
Satellite imagery
FGVCAircraft [28]
100
3,334
3,333
3,333
Fine-gra seae-graind aircraft rocognition
Flowcrs [22]
102
4,093
1,633
2,463
Fine-grained flower rocognition
Cross-Done Prss--Domiin
Food101 [33]
50,500
20,200
30,300
Fine- dine-ggained food recognition
OxfordPets [34]
37
2,944
736
3,669
Fine-grained pel revognition
StanfordCars [30]
196
6,509
1,635
8,041
Fine- tine-graied car recognition
SUN397 [35]
397
15,880
3,970
19,850
Scene recognition
UCFI01 [36]
7,639
1,898
3,783
Action recognition
ImageNet [37]
1,000
1.28M
50,000
Object recognition
ImageNet- V2 [10]
1,000
10,000
Robustness (collocation shift)
Dut-of-Distriuution
ImageNet-Skelchh [39]
1,000
50,889
Robustness (sketch domaii)
ImageNet-A [9]
200
7,500
Robustness (adversarial attack)
ImageNet-R [38]
200
30,000
Robustness (multi- (multi-domain)
B.2 Baseline Details
TPT [12], a prompt tuning method designed to minimize self-entropy across predictions from multiple augmented views.
DifTTT [13], an improved variant of TPT that leverages cittusiio-basel augmentations to refine prompt optimization.
TDA [14], a cache-basee based method that builds positive and negative caches to improve the prediction of the current test sample.
DMN-ZS [20], cache-based method that constructs memory banks from test data and pseudo- labels to enhance the prediction of the current test sample.
DPE [15], a cache-based method that trains multi-modal prototypes with historical test data to accumulate task-specific knowledge for test-time adaptation.
BCA [27]. a method that not only updates class embeddings to adapt the likelihood but also updates each class's prior using the posterior of incoming samples.
Dynaprompt [23], prompt-tuning method that maintains a prompt buffer and introduces a dynamic selection strategy to adaptively leverage relevant information from previous test samples for test- time adaptation.
B.3 Comparison with other baselines
DMN-ZS [20] is cache-based method that builds memory banks using test data and pseudo-labels to boost the prediction accuracy of the current test sample. In the original paper, DMN-ZS [20] uses a large cache size (M 50) and conducts extensive hyperparameter search with ground- truth labels from the test set to determine the fusion hyperparameter for each downstream task. Removing hyperparameter search leads to a significant drop in performance. We provide a version that uses the same small cache size (M 3) as other mache-based methods and applies a fixed fusion hyperparameter across all datasets. As shown in Table B2, our SCA substantially outperforms DMN-ZS under fixed fusion hyperparameter and small cache size (M 3), and exceeds the performance of DMN-ZS ZS configured with a large cache and per-dataset hyperparameter tuning as reported in the original paper.
Additionally, we provide results of other two gradient-bant based baselines WATT [50] and TENT [51] on the cross-domain benchmark using CLIP ViT-B/16.
16

---

Table B2: Experimental results the cross-domain benchmark with CLIP ViT-B/16. DMN-ZS (M=50) refers to the original results, where the fusion hyperparameter is exhaustively tuned for each dataset. DMN-ZS (M=3, fixed) uses the same small cache size as other cache-based methods and applies a fixed fusion hyperparameter across all datasets. The results of TENT and WATT are directly derived from [52].
Method
Aircruft
Caltech
Cars
DTD
EuroSAT
Flower
Food101
Pets
SUN397
UCF101
Average
DMN-ZS (M=50)
30.03
95.38
67.96
55.85
59,43
74.49
85.08
92.04
70.18
72.51
70.30
DMN-ZS (M=50, fixed)
29.76
94.60
67.53
55.08
56.79
75.27
81.60
89.89
69.95
72.22
69.27
DMN-2S (M=3)
28.56
94.77
66.73
53.84
56,75
75.49
84,50
91.03
68.99
72.59
69.33
DMN-ZS (M=3, fixed)
25.59
91.52
55.57
46.63
46.14
74.83
71.57
72.17
60.77
65.16
61.00
TENT
23.52
93.87
65,96
45,51
49.36
67.40
82,89
87.35
65.58
65,08
64.65
WATT
24.27
93.27
66.37
46.39
55.95
68.21
83.26
88.09
65.89
66.01
65.77
SCA (Ours)
28.50
94.85
68.49
57.09
57.16
76.09
86.09
91.44
70.27
73.43
70.34
B.4 Impact of prompt initialization
We follow the existing cache- based methods that use prompt ensembling strategy [20, 15] to initialize the prompt. This practice is widely adopted in the literature to ensure a robust and fixed zero-shot classifier, which serves as the foundation for these feature-space adaptation methods.
For prompt-tuning based methods, the use of a single prompt for methods like TPT and DiffTPT is also the standard protocol, their focus is demonstrating adaptation of the prompt itself.
Additionally, we present results using high-quality prompts obtained through training as the enhanced prompt initialization for prompt-tuning based TTA [12]. Specifically, we use the prompts trained by MaPLe [53] as the initialization, a widely adopted approach in prompt-tuning based TTA. As shown in the table below, the performance of prompt-tuning based methods is significantly lower than that of basee methods.
Table B3: Experimental results on the cross-domain benchmark with CLIP ViT-B/16.
Method
Aircraft
Caltech
Cars
DTD
EuroSAT
Flower Food101
Pets
SUN397 UCF101 Average
TPT
24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
MPP+M
24.70
93.59
66.50
45.87
47,80
72.37
86.64
90.72
67.54
69.19
66.49
SCA (Ours)
28.50
94.85
68.49
57.09
57.16
76.09
86.09
91.44
70.27
73.43
70.34
B.5 Impact of the number of classes and test samples on performance
Impact of the number of classes on performance: The number of classes inevitably impacts performance, this dictates the inherent difficulty of the task. However, our results show that there is no consistent inverse relationship between the performance gain of SCA and the number of classes. In other words, the effectiveness of our SCA does not systematically degrade as the number of classes increases.
Impact of the number of test samples on performance: The performance of any cache-based TTA method, including strong baselines like DPE and our SCA, is indeed influenced by the amount of test data available for adaptation. Here, we evaluate both SCA and DPE on subsets of the test data (10% 50%, and 100%) for two datasets: DTD (1692 testing samples) and SUN397 (19,850 testing samples). As shown in the figure below, while both methods experience performance degradation with fewer test samples, SCA shows smaller drop. This is because SCA leverages every encountered testing sample to incrementally update its feature statistics. Unlike existing cache-based methods that may filter or discard samples, SCA ensures that no information is wasted. By fully leveraging all encountered test samples, SCA builds a more comprehensive and accurate statistical model, resulting in notable performance ad advantage under data- scenarios.
17

---

DTD
SUN397
57
DPE
DPE
69.5
Accuracy 181
68.5
53
67.5
10%
50%
100%
10%
505
100%
Trining Data Ratio
Training Data Ratio
Figure B4: Performance curve.
C
Broader Impacts
This paper focuses test-time adaptation (TTA) for vision-language models (VLMs). In real-world applications, data often come from domains that differ significantly from the training distribution, leading to performance degradation. TTA enables VLMs adjust these unseen distributions on the fly, without requiring access to labeled data. Our research improves both the effectiveness and efficiency of TTA on VL.Ms, paving the way for wider adoption in practical applications.
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
It fine include aspirational goals motivation long as it is clear that these goals attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations in appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-speciffcation, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not. penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theory assumptions.
19

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report the setup throughout the paper in the Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might specific model and empirical evaluation, it may be necessary
suffice, if the contribution to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with open-source dataset instructions for how construct the dataset).
model (e.g., a large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
Code will be released after being accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
20

---

Please see the NeurIPS code and data submission guidelines Subeee ssi ssionPolicy) cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips cc/public///ccccccccccccccccc.le htte//guiiee///oeeeeee Staeeeee/ooeeeuumiseeeeeeeeeeeee wwt.ssuiissssssoooooooooooooiooo ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/detaíls
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4.1, we provide detailed descriptions of data splits and the proposed method's byperparameters.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not report the error bars our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
21

---

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have provided the experiment setting and hardware and software details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
Code of ethics
Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte::///eurippp cc/public//chicccuuuude wew.eniedidiiineeeeeeeee
Question:
Answer: [Yes]
Justification:
This work does not incorporate any ethic concerns of NeurIPS. The datasets and models are commonly used in the community, and the method does not incorporate potential concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In appendix.
Guidelines:
The answer NA means that there is no societal impact the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g,, gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets and baselines, used libraries properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseds has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- ee-aakaaedd, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The datasets and baselines, used libraries are well documented and cited.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations. and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does not involve LLMs as any important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy sectps:/nneurppee http:///.......eeeeeeee///////// for what should or should not be described.


Reviews:
[{"review_id": "qJgYhdkyCB", "reviewer": "Reviewer_GK6p", "summary": "This paper introduces Statistics Caching Test-Time Adaptation (SCA), a training-free test-time adaptation method for CLIP that replaces fixed-size caches with running class-wise statistics. Instead of storing features, it updates class means using soft pseudo-labels weighted by confidence. Predictions are made by combining this stats-based classifier with CLIP’s original output. The method avoids cache issues like forgetting or blocking and shows strong accuracy gains across benchmarks with the adaptive instance-level fusion strategy.", "questions": "1. Lines 176–177 state that allowing |Ω| to vary with τ and the sample’s confidence results in softer pseudo-labels for uncertain samples and sharper ones when confident. Could you provide more evidence for this claim, either empirically or through visualization?\n\n2. The results on ImageNet with the ResNet backbone appear weaker than DPE. Could you clarify why your method underperforms in this setting and whether it's due to architecture sensitivity or some other factor?\n\n3. In your efficiency comparison (Table 3), do the reported numbers include the cost of data augmentation? If not, could you provide a breakdown to help interpret the practical runtime tradeoffs?\n\n4. How did you determine the default values for your hyperparameters? Were they tuned on a subset of data, or selected based on prior work?\n\n5. Why are there no ablation studies on ImageNet? Given its scale and diversity, this would provide a more rigorous validation than DTD, which is relatively small.\n\n6. How would you explain the performance drop between Table 1 and Table 2? Is it due to the number of classes, the nature of the domain shift, or semantic similarity in CLIP’s textual space? Some analysis here would help interpret the results.\n\n7. Lines 242–244 present an important insight. Could you consider highlighting this point earlier or making it more prominent in the Table 3 caption, for instance? It feels like a core comparison factor for those who are not familiar with DPE and might be overlooked.\n\nIf my concerns are addressed, I will raise my score.", "limitations": "The method introduces some storage overhead due to maintaining feature statistics. It partially mitigates error accumulation, but in settings with low domain shift, the performance gains seem to become smaller.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The motivation is clear, and has a well-structured presentation with Figure 1.\n2. Replaces fixed-size caches with statistical summaries, directly addressing forgetting and blocking issues in cache-based adaptation.\n3. SCA outperforms prior methods like TDA and DPE across multiple benchmarks, showing robustness with the proposed hyperparameter strategy picking.\n4. Computationally efficient: the method avoids optimization and maintains low overhead per sample, with inference time comparable to other lightweight approaches.\n\n**Weaknesses:**\n1. The results on ResNet backbone with OOD benchmark is weaker than previous SOTA DPE.\n2. No ablations on bigger scope dataset like ImageNet, small-scale datasets like DTD is used for ablation studies.\n3. Efficiency comparisons lack clarity, unclear whether data augmentation cost is included.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed response. As I said, I’m updating my score to \"Accept\" since all concerns were resolved. If the paper gets accepted, please make sure to include the clarifications and changes I suggested. I also encourage open-sourcing the project to make it accessible to others.", "rebuttal": "We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our paper well-motivated. According to your valuable comments, we provide detailed feedback.\n### **W1 & Q2 & Q6: Performance on OOD benchmark**\n\nThank you for your insightful question regarding the performance comparison with the SOTA baseline DPE [1] on the OOD benchmark using a ResNet-50 backbone. Here we provide the explanation:\n\nOur proposed SCA is an entirely **training-free** approach. It operates by accumulating feature statistics (first and second-order moments) from all past samples and constructs a classifier via a **closed-form solution** (as detailed in Eq. 6, 7, and 8). The performance of this design is therefore directly and heavily dependent on the intrinsic quality and linear separability of the features provided by the backbone network. \n\nThe OOD benchmark is a particularly demanding scenario; its large number of classes and fine-grained **stylistic shifts** place extreme requirements on the discriminative power of the features. That's why performance drop occurs between Cross-Domain benchmark and OOD benchmark **(Q6: Performance drop between Table 1 and Table 2)**. \n\nWhile ResNet is a powerful CNN architecture, its features may be less semantically rich or linearly separable compared to those from ViT backbones. In this context, DPE's additional training phase can effectively compensate for minor imperfections in the ResNet-50 feature space, allowing it to find a slightly better decision boundary and thus achieve a marginally higher accuracy. Our purely statistical SCA is more constrained by the inherent quality of these features.\n\n[1] Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models, NeurIPS'24\n\n### **W2 & Q5: Ablation on ImageNet**\n\nThank you for this suggestion. In our paper, our ablation studies were primarily conducted on the cross-domain benchmark for faster validation. Here, we provide the results of ablation study on ImageNet:\n\n| Method Configuration               | ImageNet Accuracy |\n| :--------------------------------- | :---------------: |\n| w/o statistics accumulation (M=4)  |       70.79       |\n| w/o statistics accumulation (M=8)  |       70.81       |\n| w/o statistics accumulation (M=16) |       71.02       |\n| w/o statistics accumulation (M=32) |       71.08       |\n| w/o dynamic soft label assignment  |       70.65       |\n| **SCA (with both)**                |     **71.75**     |\n\nAs shown in table, we have several observations: (1) our full method (`71.75%`) consistently outperforms all variants of the feature caching baseline (`w/o statistics accumulation`). This demonstrates that our approach of accumulating knowledge from all samples into compact statistics is fundamentally more effective than maintaining a limited-size cache of discrete features, even on large-scale datasets. (2) Dynamic soft label assignment plays a great role in performance gain. This underscores the critical importance of this component, especially on a challenging, large-scale dataset with 1000 classes where the risk of generating incorrect hard pseudo-labels is extremely high. This confirms that our uncertainty-aware labeling is essential for robust adaptation.\n\n### **W3 & Q3: Efficiency comparison (whether data augmentation cost is included)**\n\nThank you for this important question regarding the efficiency comparison. \n\nThe numbers reported in Table 3 exclude data augmentation, as none of the methods use it on the cross-domain benchmark. This is a standard evaluation protocol followed by prior work in this area, including the baselines we compare against like DPE and TDA. Therefore, the reported runtimes reflect the full processing time for each method under this no-augmentation setting.\n\n### **Q1: Varying $|\\Omega|$ and its impact on pseudo-label sharpness**\n\nThanks for your insightful question. The core logic behind this claim is tied to how the candidate set $\\Omega$ is formed based on the initial CLIP prediction probabilities ($p_\\mathrm{text}$) and the threshold $\\tau(1-p_\\mathrm{text}^{(1)})$. Here $p_\\mathrm{text}^{(1)}$ is the value of the top-ranked class in $p_\\mathrm{text}$.\n\n* **For a confident sample:** The initial prediction $p_\\mathrm{text}$ may exhibit a very high probability for one class ($p_\\mathrm{text}^{(1)}$). According to Eq. (9), the candidate set $\\Omega$ will likely contain only a small number of classes (small $|\\Omega|$) that exceed the threshold $\\tau(1-p_\\mathrm{text}^{(1)})$. Upon renormalization in Eq. (10), the resulting pseudo-label $\\hat{Y}_t$ becomes a near one-hot vector, making it **sharp**. \n* **For an uncertain sample:** The initial prediction $p_\\mathrm{text}$ shows probabilities spread across multiple classes, none of which is individually large. To meet the cumulative probability threshold $\\tau(1 - p_\\mathrm{text}^{(1)})$, Eq. (9) includes multiple classes in the candidate set $\\Omega$ (resulting in a large $|\\Omega|$). As a result, the final pseudo-label $\\hat{Y}_t$ distributes its probability mass across these candidates, making it **soft.**\n\nCompared to using hard labels, which completely ignore uncertainty information, our dynamic threshold provides a way to adapt the label's sharpness based on the model's own confidence.\n\n### **Q4: Hyperparameters selection**\n\nWe select hyperparameters based on the validation set of DTD, a relatively challenging dataset (with CLIP ViT zero-shot accuracy of 44.27). Additionally, DTD has a small number of test samples, which allows for efficient validation. That's why DTD appears frequently across multiple tables.\n\n### **Q7:  Suggestion on highlighting \"whether need training\" in Table 3**\n\n\nThanks for your insightful suggestion, which helps make our paper clearer. In the final version, we will add a \"Requires Training\" column to Table 3 to highlight that our method, unlike DPE, is training-free.\n\nWe deeply appreciate your efforts and would be happy to further clarify or discuss any remaining concerns. Please feel free to let us know if additional modifications or explanations are required."}, {"review_id": "mXyUZRzd1C", "reviewer": "Reviewer_mpEP", "summary": "This paper investigates test-time adaptation (TTA) for vision-language models (VLMs) such as CLIP. In this field, two approaches have been proposed: (i) test-time prompt tuning, which optimizes trainable prompt parameters, and (ii) feature caching, which preserves past features and uses them for inference on new test samples. Test-time prompt tuning is computationally expensive because it optimizes prompts using backpropagation with unsupervised objectives such as entropy minimization. On the other hand, feature caching is computationally efficient because it does not perform optimization and instead uses past effective features classified with pseudo labels for inference. However, the paper argues that existing feature caching suffers from \"forgetting\" and \"blocking\" due to misclassified sample features because the cache size is limited. To address this, the paper proposes statistics caching test-time adaptation (SCA) to extract useful knowledge from all past test sample features. SCA formulates the classification problem using past features, such as ridge regression, and sequentially updates the Gram matrix required to obtain the closed-form solution. Additionally, to handle noisy pseudo-labels, it introduces dynamic soft pseudo-label assignment, which uses only the class with the highest prediction probability. SCA constructs the final prediction as the entropy-weighted sum of the zero-shot logits from CLIP and the logits computed from the cache. Experiments demonstrate that SCA achieves superior performance and high computational efficiency compared to recent TTA methods for VLMs, including test-time prompt tuning and feature caching.", "questions": "Please see the weakness section and address the concerns.", "limitations": "The main paper does not include a limitation section. The limitation should be discussed in the main paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n+ **S1.** The proposed method in this paper significantly reduces storage costs compared to existing methods by representing information from all past test samples as a Gram matrix. Leveraging all of the test samples for TTA might be a new perspective.\n+ **S2.** The dynamic soft pseudo-label assignment proposed in this paper has been successful in significantly improving performance.\n### Weaknesses\n- **W1.** The forgetting and blocking issues arising from existing feature caching, which are the primary motivations for this paper, may not be practically significant challenges. First, as shown in Figure 2, these issues have only been reported in a limited number of datasets, indicating limited generalizability. Second, as shown in Table 4, the performance improvements of the proposed method are primarily attributed to dynamic soft label assignment. Without dynamic soft label assignment, the average DPE and SCA performance would be comparable (69.40 in Table 1 vs. 69.32 in Table 4). Since this technique is not directly and necessarily related to feature caching, it can be combined with DPE. Addressing these concerns would further strengthen the clarity of the paper's contributions.\n- **W2.** The negative aspects that may be introduced by the proposed method have not been sufficiently discussed. The proposed method sequentially accumulates knowledge from all test samples into a Gram matrix. As the paper claims, this certainly has positive effects, but it may also have negative effects. For example, in situations where multiple domain shifts coexist, if samples belonging to a specific domain shift become dominant, the Gram matrix will contain statistics biased toward that domain shift. In such a situation, if samples with minor domain shifts appear, can the proposed method robustly handle them? The paper raises a primary research question, “Is this acclaimed knowledge accumulation process truly robust and effective over time?” on L42, but the evidence that the proposed method addresses this question is insufficient.\n- **W3.** The method for determining hyperparameters is unclear. The paper introduces three hyperparameters, $\\gamma, \\beta, \\tau$, into the proposed method and states that these are fixed to specific values in the experiments (L211-212). How were these fixed values discovered? As a result, the sensitivity of $\\beta$ and $\\gamma$ is confirmed in Figure 5, but nothing is mentioned about $\\gamma$. If the tuning was based on the average value of the test set, this would damage the reliability of the paper. In any case, the protocol for determining the hyperparameters should be clearly stated.", "was_revised": true, "final_justification": "The rebuttal and follow-up discussions have largely addressed my concerns with concrete evidence. Therefore, I recommend that the paper be accepted. Please see our discussions for more details.", "rebuttal": "We sincerely thank the reviewer for their constructive and insightful feedback. The comments help us to significantly improve the clarity and rigor of our paper. We address the concerns below.\n\n### **W1 (part 1): On the significance of the forgetting and blocking issues**\n\nThank you for raising this important point about the significance of forgetting and blocking issues. We clarify that these are indeed practically significant because they stem from a **fundamental algorithmic vulnerability**. \n\nThis vulnerability is inherent in any cache-based method that combines a fixed-size cache with a confidence-based replacement strategy. For any given dataset, a population of both low-entropy, incorrect samples and high-entropy, correct samples typicall always exist. Every time a low-entropy, incorrect sample arrives, there is a chance it will displace a correct feature from the cache, resulting in **forgetting**. And when a highly confident incorrect sample arrives early, **blocking** issues occurs.\n\nIn a real-world online setting, we cannot control or guarantee the arrival order of these samples. This makes the system's performance highly susceptible to \"unlucky\" sequences that trigger failure modes. Much like a security flaw in software, this design represents a **persistent threat.** While it may not be exploited in every single instance, its potential to cause performance degradation when triggered. \n\n### **W1 (part 2): On the synergy between statistics accumulation and dynamic soft labeling**\n\nThank you for this insightful question. We respectfully argue that our dynamic soft labeling technique is **not a simple \"plug-and-play\" module** that can be easily combined with DPE for similar gains. DPE, like other cache-based methods, maintains a cache of **discrete, raw feature vectors** in distinct, class-specific slots. This **discrete** storage design is ill-suited for a soft pseudo-label like `{'cat': 0.7, 'dog': 0.3}`, as it raises following challenges:\n*   **How should the feature be stored?** Should it be placed in the 'cat' cache with a weight of 0.7 and the 'dog' cache with a weight of 0.3? This would require storing an additional confidence weight with each feature, complicating the cache structure.\n*   **How would this affect cache occupancy?** If one sample now occupies slots in multiple class caches, does it count towards the size limit $M$ for each of them? This could lead to a single, highly uncertain sample disproportionately blocking space for many other more confident samples.\n*   **How would the replacement mechanism work?** When considering replacing a feature in the 'cat' cache, should its confidence be its standalone prediction confidence, or should it be discounted by its soft label weight (0.7)? If a new, confident 'cat' sample arrives, should it be allowed to replace the 'cat' portion of our soft-labeled sample? Doing so would break the integrity of the original soft label, leaving a partial entry in the 'dog' cache that no longer reflects the full uncertainty of the original prediction.\n\nIn contrast, in our SCA framework, dynamic soft label assignment is highly compatible with statistics accumulation. More specifically, they form a synergistic relationship:\n\n* **Statistics accumulation *unlocks the full potential* of soft labeling:** The update rule for our first-order statistics, $\\boldsymbol{C} _{1:t-1}+\\boldsymbol{X} _t^\\top\\boldsymbol{Y} _t$ naturally handles a soft probability vector $\\boldsymbol{Y} _t$. The contribution of each feature to the class correlations is weighted by its soft probability, which is mathematically elegant and effective.\n* **Soft labeling *enables* robust statistics accumulation:** Accumulating knowledge from every single sample is powerful but risky. If we were to use hard labels, errors would quickly accumulate and corrupt our statistics. Our dynamic soft labeling strategy acts as an uncertainty-aware regularizer to make our statistics accumulation strategy robust.\n\nIn conclusion, our dynamic soft labeling is not a simple plug-and-play module. It is tightly integrated with our statistics accumulation framework, with each part supporting the other. This synergy is key to our strong performance.\n\n### **W2: Negative aspects of accumulation in multiple domain settings**\n\nThank you for your insightful comment. Consistent with existing CLIP TTA methods, our approach primarily focuses on the single-domain setting, where test data come from a single dataset. Extending CLIP TTA methods to the multi-domain setting, where test samples from different domains are mixed, is a valuable and important direction. However, to the best of our knowledge, there are currently no TTA methods for VLMs that explicitly claim to address this scenario. \n\nIndeed, in the multi-domain setting, accumulation in our scheme may introduce negative effects. As a potential solution, we note that feature representations from different domains often exhibit some degree of difference. This property can be used to identify a sample's domain and accumulate feature statistics separately for each domain. We consider this a promising direction for future work and will add a discussion of this limitation and potential extensions in the final version.\n\n### **W3: Hyperparameters selection**\n\nWe select hyperparameters based on the validation set of DTD, a relatively challenging dataset (with CLIP ViT zero-shot accuracy of 44.27). Additionally, DTD has a small number of test samples, which allows for efficient validation. That's why DTD appears frequently across multiple tables.\n\nThe main text omits the hyperparameter analysis of the ridge coefficient $\\gamma$. Here, we present results on a cross-domain benchmark using ViT-B/16 by varying $\\gamma$ from 0 to 1e6. As shown in the table below, setting $\\gamma$ to zero leads to numerical instability due to a singular Gram matrix. Within the range from 1e2 to 1e6, performance remains stable, indicating that any value within this interval is appropriate.\n\n|         |  0   |   1   |  10   |  100  |  1e3  |  1e4  |  1e5  |  1e6  |\n| ------- | :--: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Avg Acc | None | 70.22 | 70.24 | 70.31 | 70.31 | 70.34 | 70.35 | 70.30 |\n\n### **L1: limitation section**\n\nThanks for your suggestion. We will move it from suplementary material to the main text in the final version.\n\nWe greatly appreciate your insightful review and are open to discussing any remaining issues. Please feel free to contact us if additional changes or clarifications are needed."}, {"review_id": "azYxBj3e4Z", "reviewer": "Reviewer_EE62", "summary": "This work focuses on test-time adaptation for vision-language models. The authors claim that existing methods struggle to achieve robust and continuous knowledge accumulation during test time and propose Statistics Caching test-time Adaptation (SCA), a novel cache-based approach. Specifically, SCA formulates the reuse of past features as a least squares problem, avoiding the storage of raw features and instead maintaining compact representations. In addition, SCA introduces adaptive strategies to reduce the impact of noisy pseudo-labels and dynamically balance multiple prediction sources. Extensive experiments on two benchmarks demonstrate that SCA achieves strong performance.", "questions": "Please see the weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. The paper proposes Statistics Caching test-time Adaptation (SCA), a novel cache-based approach that enables robust and continuous knowledge accumulation during test time. Storing compact information instead of raw features is a promising alternative to enable efficient adaptation.\n\n2. Comprehensive experiments, including comparisons with state-of-the-art methods and ablation studies, are provided.\n\n**Weakness**\n\n1. The motivation requires more detailed explanation. Although the authors provide Figure 2(a) to illustrate that exsiting approach leads to performance degradation, it would be helpful to analyze what types of features are forgotten during adaptation and how knowledge forgetting affects performance. In addition, how do low-entropy and high-entropy samples affect the learning process and contribute to information forgetting? The authors could also provide performance curves of the proposed method as a comparison to better demonstrate its effectiveness.\n\n2. The authors formulate feature storage as a least squares problem. It would be better if they could provide some theoretical justification or insights to explain why this approach yields better compact features for test samples. Also, the authors could compare it with other feature compaction strategies, such as using class prototypes and updating them with certain rules.\n\n3. Prediction fusion is a common strategy in VLM adaptation for improving prediction quality. It is somewhat conflicting to use entropy to weight predictions from different sources, especially since the authors themselves note that entropy is not a reliable measure of prediction accuracy (e.g., lines 40–69). Have the authors tried alternative weighting strategies, and have they analyzed the robustness of the entropy-based weighting in assigning correct importance to different predictions?\n\n4. I noticed that the authors use a prompt ensembling strategy to initialize prompts. However, prior work such as TPT and DiffTPT uses only a single prompt (\"a photo of a\") for initialization. This may introduce unfairness in comparison, as the ensemble strategy can yield a stronger baseline.\n\n5. Some feature visualizations should be provided, as they would help illustrate how stored features contribute to adaptation.", "was_revised": true, "final_justification": "My concern has been addressed. Overall, I am inclined to give a borderline accept.", "rebuttal": "We would like to thank the reviewer for taking the time to review our work. We appreciate your recognition of the originality of our paper. According to your valuable comments, we provide detailed feedback.\n\n### **W1: Motivation**\n\nThank you for this constructive feedback. *Firstly, we briefly introduce the mechanism of existing cache-based methods.*\n\nExisting cache-based methods, like TDA, maintain a fixed-size feature cache for each class during test time. The process works as follows:\n\n**1. Cache Update Mechanism:**\nFor each incoming test sample, the model first generates a pseudo-label (predicted class) and a corresponding confidence score (**typically measured by prediction entropy**). It then attempts to update the cache associated **with the predicted class**:\n\n*   The model locates the specific cache for the predicted class.\n*   If the cache for the predicted class is not full: The sample's feature is added to it.\n*   If the cache for the predicted class is full: The new sample's confidence is compared to the confidence of the features already stored in it. If the new sample is **more confident** than the **least confident feature** in the cache for predicted class, its feature **replaces** that one. Otherwise, the new sample is discarded.\n\n**2. Prediction:**\nThe final prediction for a new sample is a combination of the model's original zero-shot prediction and a new prediction derived from comparing the sample's feature similarity to the features stored across all class-specific caches.\n\n*Then we provoide a analysis to address your specific points:*\n\n**What types of features are forgotten during adaptation.** To be precise, the forgetting issue we address in this paper refers to a correct but relatively low-confidence (high-entropy) feature being overwritten by an incorrect but high-confidence (low-entropy) one. For example, on the DTD dataset, a canonical \"striped\" texture feature might be overwritten by a misclassified \"lined\" texture that the model confidently mistakes for \"striped.\"\n\n**How it Affects Performance:** the model's predictions rely on comparing new samples to these cached features. When a correct cached feature is replaced by an incorrect one, the model's ability to correctly match samples is impaired, leading to misclassifications and a performance drop, as shown in Figure 2(a). Our SCA method, by accumulating statistics from *all* samples rather than maintaining a fixed-size \"winner-takes-all\" cache, is immune to this specific type of forgetting.\n\n**The Role of Low and High-entropy samples:** \n\n* Correct low-entropy (high-confidence) samples serve as strong positive examples. They reinforce the quality of class representations in the cache, leading to more accurate future predictions.\n* Incorrect low-entropy samples may replace correct ones in the cache, causing **forgetting**. Because of their high confidence, these incorrect samples can hold cache slots for a long time, **blocking** later higher-entropy but correct samples.\n* High-entropy (low-confidence) samples typically fail to enter the cache, which leads to the loss of potentially useful, though uncertain, information. Our SCA enables these potential samples to contribute to the accumulated statistics in a controlled way through dynamic soft label assignment.\n\nIn the original version, we included more details on the forgetting and blocking issues in the **Supplementary Material (Appendix A)** due to page limits. In the final version, we will move this content into the main paper and add more visualizations, such as **the performance curves of our method in Figure 2(a)**, to make the motivation clearer.\n\n### **W2: Rationality of our accumulation scheme**\n\nThank you for this excellent question, which allows us to elaborate on the theoretical motivation behind our approach.\n\n**1. Theoretical Justification for the Least Squares Formulation:**\n\nOur core idea is to move beyond simple feature matching and instead learn a linear classifier ($\\boldsymbol {W} _{t}$) directly in the feature space, which adapts over time. The least squares formulation \n\n$$ \\min_{\\boldsymbol{W} _t}  \\|\\| \\boldsymbol{X} _{1:t} \\boldsymbol{W} _{t} - \\boldsymbol{Y} _{1:t}  \\|\\| _\\mathrm{F}^2 + \\gamma \\|\\|\\boldsymbol{W} _{t} \\|\\| _\\mathrm{F}^2$$\n\nis the classic way to achieve this for classification. The closed-form soultion is \n\n$$\\boldsymbol{W} _t=(\\boldsymbol{X} _{1:t}^\\top\\boldsymbol{X} _{1:t}+\\gamma\\boldsymbol{I})^{-1}\\boldsymbol{X} _{1:t}^\\top\\boldsymbol{Y} _{1:t}.$$\n\nAs this solution shows, to compute the optimal classifier $\\boldsymbol {W} _t$, we **do not need to store the entire, ever-growing feature matrix $\\boldsymbol{X} _{1:t}\\in \\mathbb{R}^{n _{1:t} \\times d}$** ($n _{1:t}$ denotes the number of features extracted from all test samples up to time $t$, $d$ denotes the feature dimension). We only need store two **fixed size** statistics :\n\n*   $\\boldsymbol{G} _{1:t}=\\boldsymbol{X} _{1:t}^\\top \\boldsymbol{X} _{1:t}=\\boldsymbol{G} _{1:t-1}+\\boldsymbol{X} _t^\\top\\boldsymbol{X}_t$ (the Gram matrix): This captures the **second-order statistics** of the features. It encodes the covariance structure between all feature dimensions.\n*   $\\boldsymbol{C} _{1:t}=\\boldsymbol{X} _{1:t}^\\top \\boldsymbol{Y} _{1:t}=\\boldsymbol{C} _{1:t-1}+\\boldsymbol{X} _t^\\top\\boldsymbol{Y} _t$: This captures the **first-order statistics** relating features to their class labels.\n\nTherefore, formulating the problem as least squares naturally leads to a highly compact and efficient online learning scheme. We are not arbitrarily \"compacting features.\" Instead, we are storing the essential sufficient statistics necessary to train a robust linear classifier using all the data encountered so far. This provides a strong theoretical grounding for our statistics caching approach.\n\n**2. Comparison with Class Prototype Methods:**\n\nClass prototype methods only store and update the mean feature vector for each class. This is a **first-order statistics** approach. It implicitly assumes that classes form simple, spherical clusters in the feature space and only captures the *center* of each class. It's conceptually similar to a nearest-neighbor classifier. \n\nOur SCA additionally uses **second-order statistics**. This means our model understands not just the *center* of each class's data cloud, but also its shape, orientation, and variance. This allows the learned classifier to create much more sophisticated decision boundaries.  We also provide the results of class prototypes methods on cross-domain benchmark using ViT-B/16:\n\n||Avg.|\n|-|:-:|\n|Class Prototype|65.23|\n|SCA|70.34|\n\nAs shown in the table above, our method significantly outperforms Class Prototype methods.\n### **W3: Robustness of weighting strategies for prediction fusion**\n\nThanks for your suggetsion. We provide the average accuracy of our SCA on cross-domain benchmark using different weighting strategies, including softmax probability [1] and mahalanobis distance [2] :\n\n||Entropy (Default) |Softmax|Mahalanobis distance|\n|-|-|-|-|\n|SCA|70.34|70.27|70.22|\n\nAs shown in the table, our SCA method is compatible with various weighting strategies and show robust performance. We selected entropy due to its simplicity and widespread adoption in the literature.\n\n[1] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks, ICLR'17\\\n[2] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS'18.\n\n### **W4: Prompt Initialization**\n\nWe follow the existing cache-based methods that use prompt ensembling strategy to initialize the prompt. This practice is widely adopted in the literature to ensure a robust and fixed zero-shot classifier, which serves as the foundation for these feature-space adaptation methods. \n\nFor prompt-tuning based methods, the use of a single prompt for methods like TPT and DiffTPT is also the standard protocol, as their focus is on **demonstrating adaptation of the prompt itself.**\n\nAdditionally, we present results using high-quality prompts obtained through training as the enhanced prompt initialization for prompt-tuning based TTA [1]. Specifically, we use the prompts trained by MaPLe [2] as the initialization, a widely adopted approach in prompt-tuning based TTA.  As shown in the table below, the performance of prompt-tuning based methods is significantly lower than that of cache-based methods.\n\n| | Aircraft | Caltech | Cars  |  DTD  | EuroSAT | Flower | Food101 | Pets  | SUN397 | UCF101 | Average |\n| - | :------: | :-----: | :---: | :---: | :-----: | :----: | :-----: | :---: | :----: | :----: | :-------: |\n| TPT |  24.78|  94.16 | 66.87 | 47.75 |  42.44  | 68.98  |  84.67  | 87.79 | 65.50  | 68.04  | 65.10   |\n| TPT+MaPLe |  24.70   |  93.59  | 66.50 | 45.87 |  47.80  | 72.37  |  86.64  | 90.72 | 67.54  | 69.19  | 66.49   |\n| DPE |  28.95 |  94.81  | 67.31 | 54.20 |  55.79  | 75.07  |  86.17  | 91.14 | 70.07  | 70.44  | 69.40   |\n| Ours  |  28.50   |  94.85  | 68.49 | 57.09 |  57.16  | 76.09  |  86.09  | 91.44 | 70.27  | 73.43  | 70.34 |\n\n[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS'22\\\n[2] Maple: Multi-modal prompt learning, CVPR'23\n\n### **W5: Feature visualizations**\n\nThank you for this suggestion. Existing cache-based methods store the features of specific samples, allowing for the use of visualization techniques, such as t-SNE, to directly assess cache quality. In contrast, our SCA method does not store the features themselves but rather feature statistics that summarize information from all past samples, making such visualization methods less applicable. In the final version, we will include performance curve visualizations to compare the impact of the features or feature statistics stored by different methods on adaptation.\n\nThank you for your thoughtful review. If there are any concerns or if further clarification is needed, we are more than happy to provide additional details accordingly."}, {"review_id": "9FqQN9tIFR", "reviewer": "Reviewer_JWVo", "summary": "This paper introduces SCA, a Test-Time Adaptation strategy for VLMs (e.g. CLIP), that uses a cache of statistics. The method deviates from previous cache-based methods in that the features are not directly stored, and pseudolabels are not fully trusted based on a low entropy criterion. The pipeline is fast, as it does not require gradient propagation through the model (as in prompt tuning alternatives), and only fuses the zero-shot logits with the cache logits, using adaptive fusion parameters.", "questions": "1. What is the impact of adapting on a single image and not with a batch? Specially when computing the Gram matrix G, having a few more samples could introduce more information from the batch in a transductive way.\n2. What would the performance be on corrupted datasets (see weaknesses). Some research ignore those datasets that were traditional in the field of TTA, and that significantly decrease the performance of CLIP. \n3. The ablation study on Table 4 is not totally significant, can other datasets be included?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes an alternative to popular cache based approaches for TTA, that does not require storing features directly. The main advantage is that the criterion does not rely on low entropy, which might be deceiving; incorrect features could be stored accidentally. \n2. The statistics-based cache is easy to compute, and the pseudo-label selection approach is interesting to avoid taking zero-shot predictions directly. \n3. Having adaptive fusion hyperparameters is very useful, as most of previous works finetune their hyperparameters on each dataset.\n4. Results show a competitive performance in several of the widely used benchmarks.\n\nWeaknesses:\n\n1. This work disregards another branch of TTA research that is also based on gradient, but that has proven to be competitive, such as WATT, or even TENT.\n2. Other datasets that are popular in the field of TTA are not considered; corruptions (CIFAR-10/100-C, Imagenet-C), VisDA-C, etc. \n3. The symbol \\theta is used equally for the softmax temperature as for the cumulative probability threshold in Eq. 9.", "was_revised": true, "final_justification": "The paper proposes an interesting idea that would help push the field of gradient-free TTA for VLMs forward, mainly by desisting from keeping a memory of raw features, and instead focusing on their statistics. The results are relevant and the rebuttal answers support their initial claims. I will increase my score to 5: accept.", "rebuttal": "We thank the reviewer for taking the time to review our work. We appreciate your positive feedback on the quality and clarity of our paper. Below, we provide detailed responses to your valuable comments.\n\n### **Q1: Impact of single and batch adaptation for accumulation**\n\nThank you for your insightful question. We agree that adapting on a batch introduces more information that accelerate the accumulation of feature statistics. Specifically, when adapting on a single image, the accumulation of feature statistics is indeed slower, as each test step incorporates only one sample. In contrast, batch adaptation allows the model to leverage multiple samples simultaneously, enabling faster accumulation. For fair comparison, we follow the setting in previous works that compare in single adaptation (batch size = 1).\n\n### **W2 & Q2: Performance on corrupted datasets**\n\nWe thank the reviewer for this excellent suggestion to broaden our empirical evaluation. Our initial experiments focused on the challenging OOD and cross-domain benchmarks standard in recent VLM-TTA literature. However, we agree that evaluating robustness against common corruptions provides a more complete and compelling picture of our method's capabilities.\n\nFollowing this valuable advice, we have conducted additional experiments on **CIFAR-10-C, CIFAR-100-C, ImageNet-C, and VisDA-C**, using the same CLIP ViT-B/16 backbone. We compare our method, SCA, against the strong state-of-the-art baseline, DPE. \n\n| Method         | CIFAR10-C | CIFAR100-C | ImageNet-C |  VisDA-C  |\n| -------------- | :-------: | :--------: | :--------: | :-------: |\n| CLIP           |   60.14   |   35.16    |   24.47    |   86.78   |\n| DPE            |   62.17   |   36.48    |   24.81    |   86.43   |\n| **SCA (Ours)** | **62.84** | **36.72**  | **25.64**  | **86.95** |\n\nAs shown in table,  SCA is effective against both domain shifts and common corruptions, broadening its applicability. \n### **Q3: Ablation study on other datasets**\n\nThank you for this suggestion. In our paper, our ablation studies were primarily conducted on the cross-domain benchmark for faster validation. Here, we provide the results of ablation study on large scale ImageNet:\n\n| Method Configuration               | ImageNet Accuracy |\n| :--------------------------------- | :---------------: |\n| w/o statistics accumulation (M=4)  |       70.79       |\n| w/o statistics accumulation (M=8)  |       70.81       |\n| w/o statistics accumulation (M=16) |       71.02       |\n| w/o statistics accumulation (M=32) |       71.08       |\n| w/o dynamic soft label assignment  |       70.65       |\n| **SCA (with both)**                |     **71.75**     |\n\nAs shown in table, we have several observations: (1) our full method (`71.75%`) consistently outperforms all variants of the feature caching baseline (`w/o statistics accumulation`). This demonstrates that our approach of accumulating knowledge from all samples into compact statistics is fundamentally more effective than maintaining a limited-size cache of discrete features, even on large-scale datasets. (2) Dynamic soft label assignment plays a great role in performance gain. This underscores the critical importance of this component, especially on a challenging, large-scale dataset with 1000 classes where the risk of generating incorrect hard pseudo-labels is extremely high. This confirms that our uncertainty-aware labeling is essential for robust adaptation.\n\n### **W1: Comparison with WATT and TENT**\n\nWe prvoid results of WATT and TENT on cross-domain benchmark using CLIP ViT-B/16. We will prvoide results of WATT and TENT using different backbone and datasets in our final version.\n\n\n|                | Aircraft | Caltech | Cars  |  DTD  | EuroSAT | Flower | Food101 | Pets  | SUN397 | UCF101 | Average |\n| -------------- | :------: | :-----: | :---: | :---: | :-----: | :----: | :-----: | :---: | :----: | :----: | :-----: |\n| CLIP-ViT-B/16  |  23.67   |  93.35  | 65.48 | 44.27 |  42.01  | 67.44  |  83.65  | 88.25 | 62.59  | 65.13  |  63.58  |\n| TENT           |  23.52   |  93.87  | 65.96 | 45.51 |  49.36  | 67.40  |  82.89  | 87.35 | 65.58  | 65.08  |  64.65  |\n| WATT           |  24.27   |  93.27  | 66.37 | 46.39 |  55.95  | 68.21  |  83.26  | 88.09 | 65.89  | 66.01  |  65.77  |\n| DPE            |  28.95   |  94.81  | 67.31 | 54.20 |  55.79  | 75.07  |  86.17  | 91.14 | 70.07  | 70.44  |  69.40  |\n| **SCA (Ours)** |  28.50   |  94.85  | 68.49 | 57.09 |  57.16  | 76.09  |  86.09  | 91.44 | 70.27  | 73.43  |  70.34  |\n\n\n### **W3: Ambiguous notation**\n\nThank you for pointing this out. We will fix it in the final version for clarity.\n\nThank you for your thoughtful review. If there are any concerns or if further clarification is needed, we are more than happy to provide additional details or revise accordingly."}]

Author rebuttal:

Meta-review: This paper proposes Statistics Caching Test-Time Adaptation (SCA), a training-free method for CLIP. SCA replaces fixed caches with running class-wise statistics, thereby avoiding issues such as forgetting and blocking in prior methods, while also achieving good performance.

After multiple rounds of discussion, the authors provided further clarifications and experimental results to support their claims. The reviewers agreed that the concerns were resolved and reached a consensus to accept the paper. The AC also endorsed this decision and recommended acceptance.

Kindly ensure that the new rebuttal results and discussions are incorporated into the final version.
Decision: Accept (poster)