Training-Free Test-Time Adaptation via Shape and Style Guidance for Vision-Language Models
Authors: Shenglong Zhou
Manjiang Yin
Leiyu Sun
Shicai Yang
Di Xie
Jiang Zhu
Venue: NeurIPS 2025 poster ()
Keywords: vision-language model
test-time adaptation
transfer learning
Primary area: applications

Abstract: Test-time adaptation with pre-trained vision-language models shows impressive zero-shot classification abilities, and training-free methods further improve the performance without any optimization burden. However, existing training-free test-time adaptation methods typically rely on entropy criteria to select the visual features and update the visual caches, while ignoring the generalizable factors, such as shape-sensitive and style-insensitive factors. In this paper, we propose a novel shape and style guidance method (SSG) for training-free test-time adaptation in vision-language models, aiming to highlight the shape-sensitive (SHS) and style-insensitive (STI) factors in addition to entropy criteria.
Specifically, SSG perturbs the raw test image with shape and style corruption operations, and measures the prediction difference between the raw and corrupted one as perturbed prediction difference (PPD). Based on the PPD measurement, SSG reweights the high-confidence visual features and corresponding predictions, aiming to highlight the effect of SHS and STI factors during the test-time procedure. Furthermore, SSG takes both PPD and entropy into consideration to update the visual cache, aiming to maintain the stored sample with high entropy and generalizable factors. Extensive experimental results on out-of-distribution and cross-domain benchmark datasets demonstrate that our proposed SSG consistently outperforms previous state-of-the-art methods while also exhibiting promising computational efficiency.

Training-Free Test-Time Adaptation via Shape and Style Guidance for Vision-Language Models
Shenglong Zhou', Manjiang Yin?, Leiyu Sun', Shicai Yang', Di Xie Jiang Zhu
Hikvision Research Institute
2 University of Science and Technology of China
Abstract
Test-time adaptation with pre-trained vision-language models shows impressive zero-shot classification abilities, and training-free methods further improve the performance without any optimization burden. However, existing training- training-free test-time adaptation methods typically rely on entropy criteria to select the visual features and update the visual caches, while ignoring the generalizable factors, such as Shape-sensitive and style-insensitive factors. In this paper, we propose a novel shape and style guidance method (SSG) for training-free test-time adaptation in lsiiun-aaguage models, aiming to highlight the shape-sensitive (SHS) and style- insensitive (STI) factors. Specifically, SSG perturbs the raw test image with shape and style corruption operations, and measures the prediction difference between the and corrupted ones perturbed prediction difference (PPD). Based on the PPD measurement, SSG reweights the high-confidence visual features and corresponding predictions, aiming to highlight the effect of SHS and STI factors during the test-time procedure. Furthermore, SSG takes both PPD and entropy into consideration update the visual cache, aiming maintain the stored sample with high entropy and generalizable factors. Extensive experimental results on out-of-distribution and cross-domain benchmark datasets demonstrate that proposed SSG consistently outperforms previous state-of-thc-an methods while also exhibiting promising computational efficiency.
Introduction
Vision-language models (VLMs) have demonstrated impressive effectiveness in downstream vision tasks, such as classification and generation. As the representative model, CLIP I is trained on extensive noisy image-text pairs achieve multi-modality alignment, and leverages the learned embedding space to enable zero-shot image classification. Through the learned embedding space, images can be directly recognized by matching image features with text embeddings from different classes. Nevertheless, CLIP still faces challenges when encountering domain and distribution shifts during test-time inference. As common real-world scenario, out-of-distribution (OOD) issues compromise CLIP's s ability to maintain consistent cross-moda feature alignment, resulting in degraded performance.
To mitigate the out-of-distributtin issues, test-time adaptation 2-6 is introduced to improve the model's s generalization ability. Test-time adaptation only demands the unlabeled data stream to adapt the model during inference, which is beneficial for real-world utilization. Recently, along with the rapid development of large-scale vision-language models, studies about the test-time adaptation of vision-lane Langugle models have been popular. As a pioncering work, Test-time Prompt Tuning [ (TPT) is proposed learn adaptive prompt from the single test data, which is built random
Conresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

augmentations and self-entropy optimization. Following TPT, DiffTPT [8] brings in diffusion- based augmentations to provide more various views of the single data, thus facilitating better prompt tuning during test time. Although such prompt tuning methods show promising performance in terms of test-time adaptation, they require gradient descent to optimize learnable prompts which is time-consuming with large overhead, especially for large-scale vision-language models, and thus infeasible in some computationally constrained scenarios.
Different from prompt tuning, training-free methods utilize memory banks or caches to store the high- quality test samples and make use of these stored caches to improve the model's prediction adaptively, Taking TDA [9] as an example, it constructs a dynamic key-value cache with the pseudo labels of a stream of test samples as values and the corresponding extracted features as keys. During the test-time procedure, TDA exploits the valuable knowledge encoded in CLIP based on augmentations and entropy selection, while maintaining the cache update through historical entropy comparison. Then TDA produces output predictions based on the similarity between the test sample and stored data in dynamic caches. Compared with prompt tuning methods, training-free free methods do not involve backpropagation optimization thus improving the inference efficiency obviously. However, existing training- free test-time adaptation methods typically focus on the entropy criteria, including using entropy to select the high-confidence samples from augmented views and updating the visual caches through entropy comparison, thus ignoring the exploration of generalizable factors which is important in the field of transfer learning.
In the field of transfer learning, generalizable factors, such as shape-sensitive factors and style- insensitive factors, are beneficial for models against domain shifts. Geirhos [10] conducts large-scale experiments and verifies the advantages of shape-sensitive factors to improve the model's robustness and transfer ability. MixStyle [1] proposes to force the model to learn style-insensitive factors by randomly mixing instance style statistics, which promotes the model's 's classification accuracy in domain generalization. De [12) claims to combine the commonly positively-corelate with label factors with entropy to assist the backpropagation optimization of single-modality models such as ResNer N/t5.. However, vision-language models face the dual-modality misalignment, and generalizable factors have not been explored in the field of pest-time adaptation in large-scale vision lsiiunlanguaae models such as CLIP, especially for the training- free methods.
In this paper, we propose shape and style guidance (SSG) method, which is the first exploration of generalizable factors for the training- frree test-time adaptation in vision-language models. First of all, we demonstrate the benefits of generalizable factors from the perspective of disentangled theoretical analysis, and then design SSG to exploit shape-sensitive (SHS) and style-insensitive (STI) factors to represent generalizable factors. SSG proposes perturbation prediction difference (PPD) to determine SHS and STI factors quantitatively. Specifically, SSG perturbs the shape/style of test-time images efficiently, and measures the prediction difference between the raw image and the perturbed one the PPD. Then, based on the PPD. SSG reweights the high-confidence visual features and corresponding predictions, aiming to highlight the effect of SHS and STI factors during the test-time procedure. Mcanwhile, PPD can be utilized to cast as the cache criteria with entropy to update the dynamic visual cache, and the sample with lower entropy and higher PPD is stored in the visual cache, which means not only high confidence but also high SHS and STI factors are considered. We conduct comprehensive experiments on out-of -fit---diittiuution and cross-domain benchmarks with two representative CLIP backbones ResNet-50 and ViT-B/16. Experimental results show SSG obviously outperforms state-of-the-an methods, while maintaining promising test-time inference efficiency.
2 Related Work
Vision-Language Models. Vision-I Language Models have shown impressive generalization abilities through contrastive pre-training with numerous text-image pairs [1 15) CLIP [ works by
Vision-Language
harmoniously aligning 400 million image-text pairs to predict the most relevant text description for
a given image. Recently, adapting CLIP for downstream applications has garnered considerable
attention and has been widely explored in various approaches [1 Among these, CoOp [16] introduces the concept of learnable prompts (22)25), while CoCoOp [7 further refines these prompts by conditioning them on image embeddings to achieve better generalization. Maple (26), on the other hand, applies to prompt to both the vision and language branches, thereby enhancing the alignment between multimodal embeddings. These advancements have improved performance, yet they still require training on the target data with ground-truth labels. Differently, SSG aims to explore test-time adaptation, yet they still require training the target data with ground-truth labels.

---

Test-Time Adaptation. Tent [2] minimizes generalization error on shifted data by employing test-time entropy minimization. Specifically for vision-language models, Test-time Prompt Tuning (TPT) 7 dynamically optimizes prompts during evaluation to improve zero-shot generalization. TPT works by generating multiple augmented versions of a test sample and minimizing the entropy of the model's predictions across these versions to ensure consistent results. Recent advancements like DifITPT (8] leverage diffusion models to create semantically consistent augmented images for entropy minimization, while PromptAlign 2 aligns token statistics between test samples and the source distribution. TPS (28] adjusts per-class prototypes within the embedding space, and adapts by learning shift vector for each prototype. DPE [29] proposes the dual- modalityy prototype learning for the test-time adaptation in lasiun- models. However, these methods necessitate gradient
descent operations augmented images, posing significant computational and time challenges.
Recently, training-free test-time adaptation popular which uses the cache model to facilitate the prediction of test samples in parammeetriic manner. Methods such as T3A (6] utilize prototypes as downstream classifiers, dynamically adjusting their weights to improve performance. For vision- language models, TDA [ introduces both positive and negative caches to obtain high-quality test samples from the target domain. BoostAdapter [30] introduces the instance-aware boosting samples through the augmentation operations based on TDA. However, current training-free test-time adaptation methods vision-languag models typically ignore the generalizable factors such as shape- seapessensitive and style-insensitive factors, and we bring them into this field for the first time.
3 Method
3.1
Preliminaries
Zero-Shot CLIP. CLIP consists of two pre-trained parallel encoders, visual encoder E, and textual encoder E,. For a C-class classification task, CLIP extracts the visual feature frest from a single test image Ttest and extracts textual features W from C' class-specific prompts. Then CLIP performs zero-shot predictions by computing the similarities between ftest and W as
P (frest) ) = ftest WT-
Training-Free Test-Time Adaptation. To avoid the heavy optimization burden while achieving promising performance, training-free test-time adaptation methods typically maintain dynamic visual cache, which stores few-shot high-quality samples' visual features as keys Q and corresponding pscudo-labels as values L. Given the test-time visual feature fiest, the adapted predictions can be calculated by the stored cache as
Peache (Jrest = A (Jrest QT) i,
where A(2) cxeepppp((1111 =)) is adaptation function with a weighting factor a and sharpness ratio 3 in [18] The final prediction is the combination of raw prediction P (frest and the adapted prediction Peache (ftest).
The Effect of Entropy. As discussed in Sec I existing training-free test-time adaptation methods focus entropy criteria obtain the visual features and construct the visual cache, and we describe the effect of entropy in detail.
First, entropy involves the generation of visual features frett As presented in TPT 77, considering the test-time image Tisst is single, current methods typically augment the single image to various views as [1r] (..,sst...,,:f ). then obtain the corresponding visual features [flest: ..., -..; f and prediction Then, entropy is utilized to select the high-confidence samples, which means features with lower entropy of prediction are selected. We term the selected visual features as = (f fross } and prediction as IP. = { (P(ftt)) ..., P(f pk= )} The test-time visual feature fist and corresponding prediction P(fust is the average of entropy-selected F., and P.. Training- free test-time adaptation methods such as TDA [9] basically follow the above procedure to maintain promising performance. Though it is reasonable not to augmented views in test-time adaptation, the performance can easily suffer by the complicated samples without various augmentation and entropy- based selection.
(P(fies) ..., P(ffk..))
Second, entropy involves the update of the dynamic visual cache. Considering the visual cache's capacity is limited, entropy is selected as the cache criteria to determine which sample should be
3

---

We
Hondi Grated (CLASS)
PPP
Perturbatton
Permutation
Augmermed Image
Frest
PPD
Style Perturbation
Teal-Time
Imagen
P9Dunes
Xpuet
FV
Perturbation
Collrr
Hum
Trerafomaaion
Acjustment
The generation of Permuted Prediction Difference (PPD). PPD consists of PPD and PPDah. which is used to determine shape-sensitive and style-insensitive factors quantitatively, SSG proposes to perturb the shape of test-time images and measure the prediction difference as PPD where the perturbation is the patch-shuffle of image content. Meanwhile, SSG perturbs the style of test-time images and measures the prediction difference PPDat, where the perturbation the colour affine transformation with hue adjustment.
Figure 1:
stored. Specifically, the current test-time sample compares its entropy H(ftest W with stored samples' entropy H(gen WT) in the visual cache, where H denotes the entropy function and Gent is the 'uncertain' key in visual cache [Р] The lower entropy sample's visual feature and pseudo-label will be stored finally. Through the entropy comparison, the high-confidence sample is maintained in the visual cache.
Although entropy plays an important role in existing training-free free test-time adaptation methods in VLMs, generalizable factors are totally ignored in the whole procedure, which can be proved beneficial as depicted below.
Theoretical Analysis about Generalizable Factors. As claimed in Sec generalizable factors are beneficial for transfer learning. Following the previous works [2 31, we give the theoretical analysis of the generalizable factors' important roles under the training-fres free test-time adaptation in vision-language models.
From the previous work 12 we can obtain that there is disentangled latent vector (æx
(v1 (z), ... Vd. (z)) corresponding to the input image Ziest and feature fuest. For convenience, we assume v, [0, 1] and focus on binary classification, where label y E {-1,1]. 1}. Then corr.ain corr (y*rain, v!rain) is the correlation between the train label trainn and the i-th factor virain corresponding to 2.tin obtain different partitions from v based on above correlations:
define
define corr corr (ytes, v!o") the same way. After that, can
V ppp {vrl corr train > corr > 0] Vpn {v: corr. > 0, corr test < <0}.
regard ppp as the generalizable factors which are commonly positively-correlane and pm as the spurious factors which are only positively- cettivelyycorrelateed with the label during the training- time. As a result, we have a theoretical conclusion that a sample x E geest is a harmful sample for training-fic free test-time adaptation in the vision-languag model if it satisfies the following condition:
with the label,
Vyp(x)) (Ear~agg [Vrp (x')] Ex [Yop (x')))
+ V Vpn(x) (Earmary [vp (x')] -Egsxxin [vp (z')] <0,
where Xtestt {x|
(x E Diest y y) Though we get the basically same conclusion as shown in [12), we give the insightful proof process, which is specially designed for training- free scenarios and is different from the case of entropy optimization. The detailed proof is provided in the Appendix.
From the Eq. we can obtain that the sample with high confidence can be harmful, thus generalizable factors Vpp should be highlighted and spurious factors Vpn should be avoided, which approves the claims about the benefits of generalizable factors for training-free test-time adaptation in VL.Ms. Detailed analysis can be found in the Appendix.

---

frse
Entropy
Peeudo Label
F.
PPD
PPD
fcest
Dynemio Vicual Cache
feest
Cache
| Upcatte
Cacha Keys (Visual Features)
Cocch Oriieraa (Entropyy PPD
Cache (Class Labels)
Adappted Prodetion
Final Prodiction
Gass 1
Entropy
Pseudo
Label
P(iet)
Ciess
P(fwa)
Dynamic Visual Cache
Figure 2: The effect Permuted Prediction Difference (PPD). First, based on PPD, SSG reweights the entropy-selected augmented visual features F. to get fust and IP to get P(fust) which highlights the shape-sensin sensitive factors and inseeinssnsttive factors. Second, PPD can be applied with entropy to cast as the cache criteria during the update of the visual cache, which means that not only high confidence considered, but also high chape-sensitive and style-insensitive factors are considered. Finally, the reweighted visual feature ftest queries the updated dynamic visual cache to obtain the adapted predictions, and generate the final predictions.
3.2 Shape and Style Guidance (SSG)
To highlight the generalizable factors, we leverage the prominent shape-sensitive [10 and style- insensitive factors [[]] to represent them. In detail, we design shape and style guidance (SSG) to bring in the shape-sensitive and style-insensitive factors. First, SSG designs measurement methods to determine shape-sensitive and style-insensitive factors quantitatively, Then, SSG utilizes the designed measurement affect the training-free test-time adaptation procedure in addition entropy criteria. is worth mentioning that we consider the test-time adaptation procedure with augmented images for the general case.
Shape-Sensitive Factors Measurement. To determine the stape-sensitive factors, SSG proposes to perturb the shape of test-time images first, and measure the prediction difference between the raw image and the perturbed one. The larger the difference is, the more the test-time image is affected by shape-sensitive factors. Specifically, given the augmented images {x (zlest: 2x SSG utilizes the shape perturbation to obtain the corrupted images {fi L.K &estt By the way, considering the simplicity and effectiveness, SSG chooses the patch-shuffle operation to permute the image, which corrupts the shape of the object but preserves local information [12 32]. We also utilize the entropy to select the high-confidence visual features iP_ and prediction as iP... After that, based on the pseudo-lahel from raw prediction, we measure the difference between the raw prediction P., and the perturbed one I.. and term it as Shape Perturbed Prediction Difference (PPD,t)
PPDsh (Fes Pc) = )
(5)
where y represents the prediction specific class according to the pseudo-label from the raw prediction.
Insensiivee Factors Measurement. Besides the shape-sensitive factors, SSG also proposes to determine the style-insensitive factors. Following the same way, SSG perturbs the style of the
the test-time image, and measures the prediction difference. The smaller the difference more the test-time image is affected by style-insensitive factors. Specifically, SSG utilizes the style perturbation to obtain the corrupted images {a ..., .KK ). Though other advanced style translation operations such as instance statistic permutation or fourier- based style translation can be applied, SSG chooses the efficient colour transformation along with hue adjustment to perturb the style image, considering the efficiency of test-time adaptation. After the entropy selection and difference measurement the same way, SSG obtains the Style Perturbed Prediction Difference (PPD...)
PPD (PerPo)
(Pa-P.)) (Pc-Pe))
(6)
After obtaining the PPD and PPD,t measurements, SSG combines them as Perturbed Prediction Difference (PPD) to influence the training-fin free test-time adaptation procedure. Because SSG aims to
5

---

highlight the shape-sensitive and styleinnenssiii factors, PPD is calculated as
PPD = PPD.h (Pes P... ~PPDst (P F..)
The whole and detailed process of obtaining PPD is shown in Fig. along with the specific operations sh and st
The Effect of PPD. As shown in Fig. 2 PPD can be deeply influenced the training-free freen test-time adaptation procedure in VLMs. First, SSG reweights the entropy-selecte augmented visual features Fe by PPD to get ftest, rather than the simple average operation as stated before, shown as
k+
frest E
PPD (Pe), * f
(8)
is noticeable that PPD in Eq. is processed by the softmax normalization to maintain the feature scale. Meanwhile, the same operation can be applied to entropy-selected augmented prediction P.- Through the above operations, SSG highlights the shape- sensitvvne and inyensinsensieive factors in high-confidence samples, which can further improve the model's generalization ability during the test time.
Second, PPD can be applied with entropy to cast as the cache criteria during the update of the visual cache, such H(fret W PPD. It means that not only high-confidence is considered, but also high shape-sensitive and style-insensitive factors are considered. Meanwhile, the stored visual feature also obtains the generalizable factors as shown in Eq. g which further boosts the test-time performance. Finally, the reweighted visual feature fist queries the updated dynamic visual cache to obtain the adapted predictions, and generate the final predictions.
4 Experiments
4.1 Experimental Settings
Datasets. We follow previous works 7 g to evaluate our method on two benchmarking scenarios, in-
cluding aut-of-distribution benchmark and cross-domain benchmark.
benchmark
aims to evaluate the model's robustness to natural distribution shifts on 4 ImageNet (3 vari-
ants, including ImageNet-A (44, ImageNet-V? V2 [35), ImageNet-R (36], and ImageNet-Skktchh [37]
The cross-domain benchmark aims to evaluate the transferring performance on 10 diverse recog- nition datasets, including FGVCAircraft
[38] Caltech101 39). StandfordCars [40], DTD 41
EuroSAT 42). Flowers 102
[43], Foo
44], OxfordPets [45] SUN397 [46). and UCF101 47
We follow the split in CoOp 16]. and more details are shown Appendix.
Baselines. We first compare SSG with the zero-shot CLIP prediction along with the ensemble version of 80 hand-crafted prompts from [ Then, we compare our method SSG with test-tima adaptation approaches for CLIP. For the prompt tuning methods, we compare SSG with TPT [7]. DiffTPT (8] and PromptAlign 48). We also compare with recent prototype learning methods including TPS
28
and DPE 29 For the training-free methods, we compare SSG with TDA 19. DMN-ZS 49 and
BoostAdapter 30 with promising performance. Besides the test-time adaptation methods, we also
show the performance of the representative work CoOp [16), Co-CoOp [17 and Maple (26
Implementation Details. We adopt two types of backbone including ResNet-50 (50 and ViT- B/16 [51] as the visual encoder of CLIP. Following TPT [7] and TDA [9], we set the batch size as I and generate 63 augmented views for each test image, while setting the ks the top-10%. All experiments are conducted on single 24GB NVIDIA RTX 4090 GPU. We build our SSG on the naive cache-based method, which only contains the positive cache and can be considered as the TDA [9] without negative cache. For the cache implementation details, we follow the TDA 9 default setting for the positive cache. Specifically, the utilised cache in SSG is dynamic key-value cache, whose memory size is 3 for all datasets. Considering our SSG is plug and play for existing training-fre methods, so we also provide the enhanced version by combining Boost with SSG the new version +

---

Table 1: Performance comparisons out-of-distribution benchmark. We present top-1 accuracy(%) and employ both ResNet-50 and ViT-B/16 visual backbones CLIP. The best results are highlighted in bold, and the second best results are highlighted in underline. The standard deviation is in brackets
Method
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-S
OOD Average
CLIP-ResNet-50)
21.83
51.41
56.15
33.37
40.69
Ensemble
23.24
52.91
60.72
35.48
43.09
CoOp [66
23.06
55,40
56.60
34,67
42.43
TPT
26.67
54.70
59.11
35.09
43.89
DifFTPT
31.06
55.80
58,80
37.10
45.69
TDA 9
30.29
55.54
62.58
38.12
46.63
DMN ZS
49
28.57
56.12
61,44
39.84
46.49
TPS 28
30.48
54.96
62.87
37.14
46.36
DPE
30.15
56.72
63.72
40.03
47.66
BoostAdapter
30
35.12
56.14
62.66
38.87
48.20
SSG (Ours)
31.54 (0.25)
56.78 (0.18)
63.77 (0.20)
39.11 (0.12)
47.78 (0.19)
SSG (Ours)
35.92 (0.22)
57.29 (0.17)
63.91 (0.22)
39.51 (0.11)
49.16 (0.18)
CLIP-ViT-B/IG
47.87
60.86
73,98
46.09
57.20
Ensemble
49.89
61,88
77.65
48.24
59.42
CoOp [6
49.71
64,20
75.21
47.99
59.28
Co-CoOp 17
50.63
64.07
76.18
48.75
59.91
TPT
54.77
63.45
77.06
47.94
DiffTPT
g
55.68
65.10
75.00
46.80
60,52
PromptAlign
48
59.37
65.29
79.33
50.23
63.55
TDA 9
60.11
64.67
80.24
50.54
63.89
DMN-ZS
49
58.28
65.17
78.55
53.20
63.80
TPS 23
60.08
64.73
80.27
49.95
63.76
DPB
59.63
65.44
80.40
52.26
64,43
BoostAdapter
30
64.53
65.51
80.95
51.28
65.57
SSG (Ours)
62.02 (0.14)
65.32 (0.12)
81.33 (0.18)
$2.13 (0.13)
65.20 (0.13)
SSG (Ours)
65.16 (0.18)
65.84 (0.14)
81.54 (0.15)
52.42 (0.13)
66.24 (0.15)
4.2 Experimental Results
Out-of-distribution Benchmark. As shown in Table compare SSG with other state-of- the-art methods on 4 nut-of-distribution ImageNet variants. Compared with test-time prompt tuning methods TPT, SSG improves the average accuracy by about 3.9% the ResNet-50 backbone and 4.4% on the ViT-B/16 backbone. Compared with prototype learning methods, both our SSG and SSG+ show superior performance, especially for the ViT-B/16 backbone. It is worth mentioning that, our SSG does not demand the heavy optimization burden compared with prompt-tuning and prototype learning methods, while achieving better performance. Compared with the training-free test-time adaptation method TDA and DMN-ZS, our SSG provides an obvious improvement, Compared with the current SOTA method BoostAdapter, our SSG almost outperforms it all datasets except for ImageNet-Ar A. Mcanwhile, our SSG further enhances the performance compared with BoostAdapter, which verifies the effectiveness of our SSG.
Efficiency Comparison. As the training-free free test-time adaptation methods, the inference ef- ficiency is an important advantage against other methods such test-time prompt learning and prototype learning methods. Therefore, we com- pare our SSG with other test-time adaptation methods for Dision-languse different methods on the ImageNet-A dataset and the experiments are conducted on single 24GB NVIDIA RTX 4090 GPU. In our SSG, the extra introduced computations are the shape and style perturbation, but these perturbation opera- tions are extremely simple and do not bring in heavy computation overhead. Besides, the operations comparison are also lightweight. The detailed comparison results are shown in Table Compared with test-time prompt tuning and prototype learning methods, it is obvious that our SSG shows significant improvement in terms of testing time due to the design
Table 2: Comparison results in terms of efficiency (testing time) and effectiveness (accuracy)
Method
Testing Time
OOD Accuracy
Gain
CLIP
-3 min
57.20
h 36 min
60.81
+3.61
models. We evalu-
DifTIT
3
60.52
+3.32
TDA
~15 min
63.89
+6.69
DPE
79
~28 min
64.43
+7.23
SSG (Ours)
-15 min
65.20
+8.00
based on PPD such reweighting

---

Table 3: Performance comparisons on cross-datesets generalization. We also present top-1
aceuracy(%) for all methods on two backbones of CLIP. The best results are highlighted in bold.
Method
Aireraft
Caltcch
Cars
DTD
EuroSAT
Flower
Food101
Pets
SUN397
UCF101
Average
CLIP-RN-50 [1]
15.66
85.88
55.70
40.37
23.69
61.75
73.97
83,57
58.80
58.84
55.82
Ensemble
16.11
87.26
55.89
40.37
25.79
62.77
74.82
82.97
60.85
59.48
56.63
CoOp [6
15.12
86.53
55.32
37.29
26.20
61.55
75.59
87.00
58.15
59.05
56.18
TPT 7
17.58
87.02
58.46
40.84
28.33
62.69
74.88
84.49
61.46
60.82
57.66
DifTTPT
17.60
86.89
60.71
40.72
41.04
63.53
79.21
83.40
62.72
62.67
59.85
TDA 9
17.61
89,70
57.78
43.74
42.11
68.74
77.75
86.18
62.53
64.18
61.03
DPE 20
19.80
90.83
59.26
50.18
41.67
67.60
77.83
85.97
64.23
61988
61.93
BoostAdapter
18.93
88.4
59.67
43.85
44.40
68.25
78.78
85.75
62.83
64.42
61.54
SSG (Ours)
21.63
90.67
60.75
52.19
43.26
69.02
77.73
86.64
64.23
62.15
62.83
SSG (Ours)
22.14
90.68
60.99
52.31
44.58
69.07
78.91
86.32
64.41
64.77
63.42
CLIP-VIT-B/EE
23.67
93.35
65,48
44.27
42.01
67.44
83.65
88.25
62.59
65.13
63,58
Ensemble
23.22
93.55
66.11
45.04
50.42
66.99
82.86
86.92
65.63
65.16
64.59
CoOp LT
18.47
93:70
64.51
41.92
46.39
68.71
85.30
89.1
64.15
66.55
63.88
CoCoOp 10
22.29
93.79
64,90
45.45
39.23
70.85
83.97
90.46
66.89
68.44
64.63
TPT 0
24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
DiATTT
25.60
92.49
67.01
47.00
43.13
70.
87.23
88.22
65.74
62.67
65.47
PromptAlign
24.80
94.01
68.50
47.24
47.86
72.39
86.65
90.76
67.54
69.47
66.92
TDA 99)
23.91
94.24
67.28
47.40
58.00
71.42
86.14
88.63
67.62
70.66
67.53
DPE 20
28.95
94.81
67.31
54.20
55.79
75.07
86.17
91.14
70.07
70.44
69.40
BoostAdapter
27.45
94.77
69.30
45.69
61.22
71.66
87.17
89.51
68.09
71.93
68.68
SSG (Ours)
30.21
94.93
68.65
56.15
$9.91
74.54
86
91.58
70.10
69.73
70.24
SSG (Ours)
31.97
95.24
69.97
55.47
62.54
74.64
86.99
91.89
70.64
72.69
71.20
of the training- fraieing-free method. Specifically, the computational efficiency of SSG is more than 8x times higher than that of TPT and almost x times higher than that of DPE, while improving the performance about 4.4% and 0.8%. Though SSG still gives a slower testing time compared with zero-shot CLIP, it obtains obvious OOD accuracy improvement of about 8.0%.
It is worth mentioning that, SSG applies the shape/style perturbation to the raw input images before the model forward procedure, and we concatenate them into a batch to feed into the model, thus the forward passes can be shared and maintain the forward time nearly unchanged. Meanwhile, due to the design of triple number inputs per sample to share the forward process for maintaining inference time, our SSG consumes more memory usage (3862 MiB) compared with TDA (3098 MiB). Our SSG increases the memory usage within an acceptaale scope, as both 3862MiB and 3098MiB are generally considered to be acceptable memory usage ranges (< 4GB) in most practical scenarios. Thus, our SSG achieves a modest trade-off given the gains in accuracy and comparable inference time. Compared with DPE (9986 MiB), our SSG shows obviously lower memory usage (3862 MiB) while achieving better OOD accuracy, which demonstrates the effectiveness of SSG.
Benchmark. As shown in Table we compare our SSG with other sate-of-the-ant methods on 10 fine-grained cross-domain datasets. Compared with the test-time prompt tuning methods TPT and Dif our SSG outperforms them obviously in the average accuracy. Compared with the prototype learning method DPE, our SSG gives a promising performance without any parameter optimization. Compared with the BoostAdapter method, our SSG also gives superior performance on 7 out of 10 datasets, showing the effectiveness of the whole design.
4.3 Ablation Studies
First, conduct experiments to explore the influence of shape-sensitive and style-insensitive factors. Second, we conduct experiments to determine the influence hyperparameters on the shape perturbation operation. Third, we conduct experiments to explore the effect of the PPD's operations including PPD reweighting and comparison.
Shape-Sensitive and Styls-Insensitive Factors. SSG aims to bring in generalizable factors to assist test-time adaptation, thus designing PPDah and PPD,, to measure shape-sensitive and style- insensitive factors respectively. We conduct experiments to clarify the different effects of PPD and PPD along with the full version as PPD shown Eq. There are three experiments, the first one only involves the PPD measurement which means only the insensivsessiivee factors are brought
8

---

65
63
06.0
53.0
65.5
R0
65
65.5
PP Suree
52.5
Accuracy
61.5
Accuracy
64
Arcuracy
65.0
64.0
64.5
51.5
63.5
62
59
64.0
51.0
63.0
61
63.5
50.5
000 Average
Imagehett-A
62.0
000 Average
ImageNet-V22
63.0
ImageNet-5
50.0
VIT-B/16
Patch numbers (VIT-8/16)
VIT-B/16
Figure 3: Ablation experiments about important components of SSG. Left: we conduct experiments to explore the influence of shape-sensitive and style-insensitive factors through PPD PPD,t and PPD measurements, Middle: we conduct experiments to determine the influence of hyperparameters (patch number) on the shape perturbation operation. Right: we conduct experiments to explore the effect of the PPD's operations including PPD reweighting and comparison.
into the test-time adaptation procedure. The second one only considers the shape-sensitive factors and PPDsh measurement, and the last one is the full model consisting of both PPDah and PPDst. which is termed PPD.
We conduct experiments on the ViT-B/16 backbone, and the results on OOD datasets and the single ImageNet-V2 V2 shown in Fig. 3 left. The results show that PPDah achieves more achievement than PPD which shows that shape-sensitive factors represent better generalizable factors than style-insensitive factors to some degree. Though style and shape considered the disentangled factors in some cases, shape-sensitive factors are more straightforward for the generalizable concept. But it is worth mentioning that, the full model which utilizes the PPD shows the best performance, which means the style-insensitive factors are good complement to shape-sensitive factors.
Hyperparameters in Shape Perturbation Operation. To obtain the PPDah measurement, SSG introduces the shape perturbation operation to generate the shape-corrupted images. Specifically, PPD chooses the simple patch shuffle operation as the shape perturbation, and achieves the promising performance. As the important hyperparameter in patch shuffle, the patch number in the image desired to be explored. We choose different patch numbers to conduct experiments ViT-B/16, and show the results in Fig. 3 middle. The choice patch numbers range from 2 to 16, which covers the extreme cases in the regular patch shuffle operation.
According to the accuracy of ImageNet-R and ImageNet-A, we can find the patch number as (4, 6) performs the better performance, both smaller and larger patch numbers decrease the accuracy obviously, especially for the larger patch number such as 16. We analyze the results from the property of shape corruption. A small patch number such 2 may not corrupt the semantic instances due to the small instance size, so the shape-sensitive factors are not captured well. For the large patch number of 16, the local information is totally corrupted, thus the prediction is casily affected by many factors rather than the shape-sensitive factors. Therefore, a proper patch number demanded when applying the patch shuffle, and we choose patch number 4 in default for all main experiments, and the performance shows a stable improvement.
Hyperparameters about the combination between PPD and PPD.:. For the combination of PPD and PPD,t in Eq. 7 considering the training-fir free methods' efficiency and no extra learnable parameters, we combine them in a simple way by the hyperparameter as and achieve the best performance compared with alone PPDs//PPD, as shown Fig. We conduct experiments about different ratios to combine PPD and PPD such as rescaling PPD with 0.5/1.552.. factors to obtain PPD. The OOD accuracy of SSG varies from 65.20 to 65. 166600/655.1.. This result shows the performance range is relatively stable, which shows the robustness of SSG performance. Considering the situation in actual scenarios, the scale of PPDst can be simply set to 1.0.
The Effect of PPD Operation. Once obtained the PPD measurement, SSG applies it to reweight the ceggh-oofideece samples to generate the test-time visual feature, while assisting the visual cache selection along with the entropy criteria. To clarify the different effects of these two operations, we design three ablation experiments. The first one only involves the PPD reweighting, and second one only involves the PPD comparison. The third one is the full model with both PPD reweighting and comparison. We conduct experiments on ViT-B/16. and show the accuracy results on all OOD datasets and the single ImageNet-S dataset in Fig. right. According to the results,

---

Table 4: Comparison results based on the OpenCLIP (ViT-L/14) backbone.
Method
ImageNet-A
ImageNet-V
ImageNet-R
ImageNet-S
OOD Average
OpenCLIP
53.88
67.69
87.42
63.18
68.13
TDA 9
61.27
68.42
88.41
64.67
70,69
DPE 29
61.09
70.83
89.18
66.33
71.86
SSG (Ours)
62.85
69.97
89.67
66.56
72.26
we can find reweighting operation achieves better performance and a combination of them gets the best performance. It is worth mentioning that PPD comparison for the update of the visual cache is helpful for the PPD reweighting. illustrates that even for the single test test-time image without any augmentation, the PPD comparison still contributes to the construct of the visual cache.
SSG other Vision-Language models. Our SSG can easily be applied to other vision- language models, and we follow DPE to involve the OpenCLIP (ViT-L/14) [52) as an example for a fair and public comparison. We show the comparison results in Table 4 We observe that SSG still outperforms training-free TDA by 1.57% on average across datasets, showing that our method generalizes well the larger-scale vision-language model.
Conclusion
In this work, we propose novel shape and style guidance (SSG) method for the training-free test-time adaptation fields in VLMs. SSG first brings the generalizable factors such as shape-sensitive and style- insensitive factors into the test-time adaptation procedure. To determine the generalizable factors, SSG designs the shape perturbed prediction difference (PPDsh) and style perturbed prediction difference (PPDat) measurement, by utilizing the shape perturbation and style perturbation to corrupt the raw image. Based the perturbed prediction difference (PPD), SSG reweights the high-confidence visual features for a better generalizable representation, and introduces PPD as the cache criteria along with the entropy.
Despite the promising performance of our SSG, also has some limitations. Though SSG proposes to explore the generalizable factors, the current utilized shape and style perturbation operation is straightforward, and more advanced and efficient operations, such as fourier-based or channel-based methods, can be discussed in the future. At the same time, there always based medical image analysis) that depend on style information, and how to explore the generalizable factors these tasks is one promising research direction.
tasks (e.g.. texture-
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision." in International conference machine learning. PMLR, 2021, pp. 8748-8763.
[2] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, "Tent: Fully test-time adaptation by entropy minimization," in International Conference Learning Representations, 2021.
[3] Y. Sun, Wang. Z. Liu, Miller, Eiros, M. Hardt, "Test-time training with ceffssupeviisoo for generalization under distribution shifts," in International conference on machine learning. PMLR, 2020, pp. 9229-9248.
S. Zhou, Xiong, and F. Wu, "Test-time adaptation via style and structure guidance for histological image registration," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp- 7677-7685.
T. Yang. S. Zhou, Y. Wang, Y. Lu, and N. Zheng, "Test-time batch normalization." arXiv preprint arXiv:2205.100 2022.
[6] Y. Iwasawa and Y. Matsuo, "Test-time classifier adjustment module for model-agnostic domain generaliza- tion," Advances in Neural Information Processing Systems, vol. 34, pp. 2427 2440, 2021.
[7] M. Shu, W. Nie, D.-A. Huang. Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao, "Test-tine time prompt tuning for zero-shot generalization in pision-langual models." Advances in Neural Information Processing Systems, vol. 35, pp. 14 289, 2022.
10

---

[8] C.-M. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo, "Diverse data augmentation with diffusions for effective test-time prompt luning," " in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 2704-2714.
[9] A. Karmanov, D. Guan, Lu, A. Saddik, and F. Xing, "Efficient test-time adaptation vision-language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 122-1221-2 171.
[10] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, "Imagenet-trained biased towards texture; increasing shape bias improves accuracy and robustness," in International conference on learning representations, 2018
[11] K. Zhou, Yang. Y. Qiao, and Xiang, "Domain generalization with mixstyle," International Conference on Learning Representations. 2021.
[12] J. Lee, D. Jung, S. Lec, J. Park, J. Shin, U. Hwang, and S. Yoon, "Entropy is not enough for test-time adaptation: From the perspective of disentangled factors." in The Twelfth International Conference on Learning Reprexentations, 2024.
[13] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, "Sealing up visual and vision-language representation learning with noisy text supervision," " International conference on machine learning. PMLR, 2021, pp. 4904-4916.
[14] J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation," in International conference muchine learning. PMLR, 2022, pp. 888-12900.
[15] J. Li, D. Li, S. Savarese, and Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in International conference on machine learning. PMLR, 2023, pp. 19 7777711777 742.
[16] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Learning to prompt for vision-language models," International Journal of Computer Vision, vol. 130, no. 9. pp. 2337-2348, 2022.
[17] "Conditional prompt learning for lision-language models," in Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, 2022.
[18] R. Zhang, W. Zhang, Fang, Gao, K. Li, Dai, Qiao, and H. Li, "Tip-adapter: Training-free adaption of clip for few-shot classification," " in European conference on computer vision. Springer, 2022, pp. 493-510.
[19] X. Li, D. Lian, Z. Lu, J. Bai, Z. Chen, and X. Wang, "Graphadapter: Tuning vision-language language models with dual knowledge graph," Advances in Neural Information Processing Systems, vol. 36, 2024.
[20] X. Zhu, R. Zhang, B. He, A. Zhou, D. Wang. B. Zhao, and P. Gao, "Not all features matter: Enhancing few-shot clip with adaptive prior refinement," Proceedings of the IEEE/CVF Intermational Conference on Computer Vision, 2023, pp. 2605-2615
[21] Z. Lu, Bai, X. Li, Z. Xiao, and X. Wang, "Beyond sole strength: Customized ensembles for generalized vision-language models," in International Conference on Machine Learning. PMLR, 2024, pp. 32 924 32938.
[22] B. Lester, R. Al-Rfou, and N. Constant, "The power of scale for parameter-efffiient prompt tuning." in Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, 2021, pp. 3045-3059.
[23] Y. Zha, J. Wang, T. Dai, B. Chen, Z. Wang, and S.-T. Xia, "Instance-aware dynamic prompt tuning for pre-trained point cloud models," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 14 161-14 170.
[24] S. Yang. J. Bai. K. Gao, Y. Yang, Li, and S.-T. Xia, "Not all prompts secure: A switchable backdoor attack against pre-trained vision transfomers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24431-24 441.
[25] X. Liu, K. Ji. Y. Fu, W. Tam, Z. Du, Z. Yang. and J. Tang. "P-tuning: Prompt tuning can be comparable to fine- tne-tunnng across scales and tasks." in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 61-68.

---

[26] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, "Maple: Multi-modal prompt learning." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2023, pp.
19 113-19122.
[27] J. Abdul Samadh, M. H. Gani, N. Hussein, M. U. Khattak, M. M. Naseer, F. Shahbaz Khan, and S. H. Khan, "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization." Advances in Neural Information Processing Systems, vol. 36, 2024.
[28] E. Sui, Wang, and Yeung- Levy, "Just shift it: Test- time prototype shifting for zero-shot generalization with vision- lanioon models," in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEFF, 2025, pp. 825-835.
[29] C. Zhang, S. Stepputtis, K. Sycara, and Y. Xie, "Dual prototype evolving for test-time generalization of vision-language models," Advances in Neural Information Processing Systems, vol. 37, pp. 32 111-32 136, 2024.
[30] T. Zhang, J. Wang. H. Guo, T. Dai, B. Chen, and S.-T. Xia, "Boostadapter: Improving vision-language test-time adaptation via regional bootstrapping"" Advances in Neural Information Processing Systems, vol. 37. pp. 67 795-67 825, 2024.
[31] O. Wiles, S. Gowal, F. Stimberg. S.-A. Rebuffi, I. Ktena, K. D. Dvijotham, and A. T. Cemgil, "A fine- grained analysis on distribution shift," in International Conference on Learning Representations, 2022.
[32] F. M. Carlucci, A. D' D'Innocente, S. Bucci, B. Caputo, and T. Tommasi, "Domain generalization by solving jigsaw puzzles," " in Proceedings the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2229 -2388.
[33] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248-255.
[34] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, "Natural adversarial examples," in Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 15222-15271.
[35] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, "Do imagenet classifiers generalize to imagenet?" in International conference on machine learning. PMLR, 2019, pp. 5389-5400.
[36] D. Hendrycks, S. Basart, N. Mu. S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo al., 'Thh many faces of robustness: A critical analysis of out-of-distribattion generalization, in Proceedings of the IEFF/CVF international conference on computer vision, 2021, pp. 8340-8349.
[37] H. Wang. S. Ge, Z. Lipton, and E. P. Xing, "Learning robust global representations by penalizing local predictive power," " Advances in Neural Information Processing Systems, vol. 32, 2019.
[38] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, "Finc-grained visual classification of aircraft," arXiv preprint / 305.515,, 2013.
[39] I.. Fei-Fei, R. Fergus, and Perona, "Iearrnig generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories," " in 2004 conference on computer vision and pattern recognition workshop. IEFF, 2004, pp. 178-178.
[40] Krause, M. Stark, J. Deng, and L. Fei-Fei, "3d object representations for fine-grained categorization," in Proceedings of the IEEE international conference on computer vision workshops, 2013, pp. 554-561.
[41] M. Cimpoi. S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures the wild," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 3606-3613.
[42] P. Helber, B. Bischke, A. Dengel, and D. Borth, "Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7. pp. 2217-2226, 2019.
[43] M.-E. Nilsback and A. Zisserman, "Automated flower classification over a large number of classes." in 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008, pp. 222729.
[44] L. Bossard, M. Guillaumin, and L. Van Gool, "Food- 101-mining discriminative components with random forests." in Computer vision-ECCO ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. Springer, 2014, pp. 446-461.
[45] O. M. Parkhi, A. Vedaldi. A. Zisserman, and C. Jawahar, "Cats and dogs," in 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012, pp.
12

---

[46] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "Sun database: Large-scale scene recognition from abbey z00," in 2010 IEEE computer society conference computer vision and pattern recognition. IEEE, 2010, pp. 3485-3492.
[47] K. Soomro, "Ucf101: dataset of 101 human actions classes from videos in the wild," arXiv preprint arXiv:1212.0402. 2012.
[48] J. Abdul Samadh, M. H. Gani, N. Hussein, M. U. Khattak, M. M. Naseer, F. Shahbaz Khan, and S. H. Khan, "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization." " Advances in Neural Information Processing Systems, vol. 36, pp. 80 996-8011,, 2023.
[49] Y. Zhang, W. Zhu, Tang, Ma, K. Zhou, Zhang, "Dual memory networks: versatile adaptation approach for vision-language models," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, 28 718-28 728.
[50] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedingy of the IEEE conference computer vision and pattern recognition, 2016, pp. 770-778.
[51] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, and X. Zhai, "An image is worth 16x 16 words: transformers for image recognition atscale," in International Conference on Learning Representations, vol. I, no. 2, 2021. p. 3.
[52] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco. C. Gordon, C. Schuhmann. L. Schmidt, and J. Jitsev, "Reproducible scaling laws for contrastive language-image learning," in Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 281--289..
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We reflect the contributions and scopes clearly in the abstract and introduction as mentioned in Section [
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 5
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Section and Appendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 3
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We use the public dataset as mentioned in Section 4 and we will release the codes after the final decision.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct the statistic significance test as mentioned in Appendix.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Please refer Section 4
9. Code of ethics
14

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics http://pppp.//a.poo..ccoo///a/aa
Answer: [Yes]
Justification: We have complied with the NeurlPS Code of Ethics in all respects.
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broader impacts as mentioned in Appendix.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have received proper acknowledgement and respected the licenses and terms of use.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this paper does not involve LLMs as any important, original, or non-standard components.
15


Reviews:
[{"review_id": "NPQNF2nfoo", "reviewer": "Reviewer_BH2W", "summary": "This paper proposes a training-free test-time adaptation method for vision-language models, called Shape and Style Guidance (SSG). SSG aims to improve robustness to domain shifts by incorporating generalizable visual factors, specifically shape-sensitive and style-insensitive cues. It perturbs test-time images using shape and style transformations and measures the prediction difference compared to the original, forming a Perturbed Prediction Difference (PPD). This signal is used to reweight high-confidence visual features and guide cache updates, emphasizing features that are robust and informative. Experiments on several out-of-distribution and cross-domain benchmarks show that SSG achieves strong improvements over existing methods.", "questions": "1. Could you please clarify whether SSG is implemented on top of TDA, Tip-Adapter, or another cache-based method? It would also be helpful to specify the cache structure (e.g., queue types and sizes, similarity function), and explain whether the entropy-minus-PPD criterion applies to both the positive and negative caches. If so, how are negative PPD values handled?\n\n2. Could you please include or comment on an augmentation-only ablation—e.g., applying shape and style perturbations in TDA without PPD-based reweighting or cache updates? \n\n3. How is the parameter k* determined? Is it a fixed value, dataset-dependent, or adaptively selected per image? Please consider clarifying this in the main text.\n\n4. When computing PPD, are the same top-k* low-entropy indices used across the original, shape-perturbed, and style-perturbed augmentations? If so, please make this explicit, as it’s important for reproducibility.\n\n5. SSG appears to use roughly 3× more augmentations than TDA, yet reports similar inference time. Could you please clarify whether forward passes are reused or shared across branches?\n\n6. Have you tested SSG on other vision-language models, such as SigLIP or SigLIP-2? It would be helpful to know if the approach generalizes beyond CLIP.\n\n7. In Table 1, the “ensemble” row is somewhat ambiguous. Could you please explain whether this refers to CLIP predictions averaged over all augmentations, or something else?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Well Written.** The paper is well written and clearly structured, making the proposed method easy to follow and understand.\n\n2. **Interesting Idea.** Introducing shape-sensitive and style-insensitive cues for test-time adaptation is a novel and compelling idea that adds a new perspective to training-free adaptation.\n\n3. **Practicality.** The method builds on training-free adaptation techniques and avoids gradient-based updates, making it more practical and efficient for real-world deployment compared to optimization-based approaches.\n\n4. **Thorough empirical validation.** The experimental evaluation is thorough, with comparisons against a wide range of state-of-the-art baselines across both out-of-distribution and cross-domain benchmarks, where the proposed method consistently outperforms others.\n\n\n\n**Weakness**\n\n1. **Novelty.** While the paper is thoughtfully designed and thoroughly evaluated, the core idea of perturbation‐based reweighting feels like incremental improvement of existing cache-based TTA techniques rather than a transformative advance.\n\n2. **Efficiency concerns.** SSG is reported to have similar inference time to TDA, despite using roughly three times more augmentations per image (original, shape-perturbed, and style-perturbed). Unless forward passes are reused, this seems inconsistent. Clarifying these details is important to accurately assess the true computational cost.\n\n3. **Cache design ambiguity.** The paper does not clearly specify whether SSG is built on TDA, Tip-Adapter, or another cache-based method, leaving key implementation details—such as queue structure, and memory size—undefined. Additionally, if based on TDA, it is not explained how the entropy-minus-PPD criterion affects the negative cache or whether it is applied symmetrically. These ambiguities limit interpretability and reproducibility.\n\n4. **Missing augmentation-only ablation.** A missing ablation is a version of TDA that uses the same 3× augmentation strategy as SSG (original + shape + style), but without applying the PPD-based reweighting or entropy-PPD cache update. This would isolate the benefit of the diverse views themselves versus the actual contribution of SSG's mechanisms. \n\n5. **Missing details and analysis of** k*. One parameter,  k* , determines how many low-entropy augmentations are used for prediction reweighting and cache updates. However, the paper does not specify how k* is selected, nor does it study its impact on performance. Given its central role in PPD computation and feature aggregation, a sensitivity analysis or ablation would help assess the method's robustness.\n\n6. **Inconsistent ablations.** While the ablation studies are informative, they are inconsistent. Different components are analyzed on different datasets (e.g., ImageNet-V2, -R, -A, -S) and with different backbones (ResNet vs. ViT), making it difficult to compare relative contributions. A more rigorous evaluation would unify ablations on a consistent benchmark or report average OOD accuracy for all variants.\n\n7. **Missing statistical variance in main results.** Although the appendix includes significance testing, no variance metrics (e.g., standard deviations) are reported in the main tables. Given the sensitivity of test-time adaptation to augmentation noise and pseudo-labeling, reporting mean ± std across runs would improve transparency and trust in the reported gains.\n\n\n8. **Minor Issues.**\n- The paper likely uses the same top-k* indices across original and perturbed augmentations when computing PPD, but this should be stated explicitly to avoid confusion.\n- The subsection “Hyperparameters about Style-Insensitive Factors” focuses on the weighting between PPD_sh and PPD_st, but omits the actual style perturbation settings. It may be helpful to rename or expand the section to include these details.\n- Figure and terminology consistency. In the caption of Figure 1, both branches are labeled PPD_sh, though the right should be PPD_st. Acronyms for style-insensitive factors (e.g., “SIS” vs. “STI”) should also be standardized for clarity.\n\n---\nI welcome the authors' responses to the above concerns and am open to revising my assessment accordingly based on their clarifications.", "was_revised": true, "final_justification": "I thank the authors for their thorough and well-organized rebuttal. The additional clarifications, ablation studies, and efficiency/memory analyses have satisfactorily addressed most of my concerns.\n\nWhile I feel the methodological novelty is moderate, the integration of shape-sensitive and style-insensitive factors offers an intresting enhancement.\n\nGiven that my other concerns have been addressed, I am raising my score to 4.", "rebuttal": "We appreciate your thorough review of our paper, along with your constructive comments. We provide our feedback on weaknesses (W) and questions (Q) as follows.\n\n**W1**: *\"While the paper is thoughtfully designed and thoroughly evaluated, the core idea feels like an incremental improvement of existing cache-based TTA techniques.\"*\n\n**A1**: Thank you for the approval of our thoughtful design and thorough evaluation. While our SSG is easy to implement with existing cache-based VLM TTA methods, we do not think it is an incremental improvement.\n\nFirst, SSG reveals the effect of generalizable factors for the cache-based (training-free) VLM TTA, which has not been studied before. Similar to TDA (2024CVPR) or BoostAdapter (2024NeurIPS), our SSG gives a new perspective for the cache-based TTA framework, and the corresponding experimental results prove the effectiveness.\n\nSecond, we give the theoretical analysis for the generalizable factors under training-free TTA scenarios. Different from the entropy minimization scenarios, it is not trivial to derive the proof based on the properties of cache-based methods, as shown in Line15 - Line21 in the Appendix.\n\nThird, SSG gives a potential direction to explore a further cache-based approach by designing more advanced generalizable factors.\n\n**W2&Q5**: *\"Efficiency concern. Could you please clarify whether forward passes are reused or shared across branches?\"*\n\n**A2**:  Thanks for your thoughtful feedback. The forward passes are indeed shared across branches in our SSG, and we give the detailed explanations below.\n\nFirst, SSG utilised shape and style augmentation is extremely simple to implement. For the shape augmentation, we use the patch-shuffle operation, which is efficiently implemented by `erarrange func` in `einops packages`. For the style augmentation, we use the colour transformation with hue adjustment, which is easily implemented by the `kornia package`.\n\nSecond, SSG applies the shape/style augmentation to the raw input images before the model forward procedure, and we concatenate them into a batch to feed into the model, thus the forward passes can be shared and maintain the forward time nearly unchanged.\n\nThird, after model forward and obtaining the logits, the following PPD calculation, reweighting and cache update are simple mathematical calculations, which do not introduce much more inference burden. \n\nFinally, we give the detailed inference time comparison below. SSG needs a little more data processing time due to the shape/style augmentation, and the total increased inference time is relatively small. \n| Methods    | Data Process | Model Forward | Rest Part | Total  |\n| ---------- | ------------ | ------------- | --------- | ------ |\n| TDA        | 15 ms        | 82 ms         | 34 ms     | 132 ms |\n| SSG (Ours) | 21 ms        | 83 ms         | 35 ms     | 139 ms |\n\nFurthermore, we supplement the efficiency comparison in ImageNet-V, which contains 10000 images, in the following table.\n| Methods    | Testing Time | OOD Accuracy | \n| ---------- | ------------ | ------------ | \n| TDA        | 22 min       | 63.89        | \n| DPE        | 41 min  | 64.43        | \n| SSG (Ours) | 23 min 10 s       | 65.20        | \n\nFrom the above table, even for the 1w image scale, our SSG only increase 1 minute compared with TDA while improving the OOD accuracy by about 1.3%. Our SSG only demands 56% inference time compared with DEP while achieving better performance.\n\n**W3&Q1**: *\"Cache design ambiguity.\"*\n\n**A3**: Thank you for pointing it out. We build our SSG on the naive cache-based method, which only contains the positive cache and can be considered as the TDA without a negative cache.  \n\nFor the cache implementation details, we follow the TDA default setting for the positive cache. Specifically, the utilised cache in SSG is a dynamic key-value cache, whose memory size is 3 for all datasets. For the similarity function in adapted prediction, we utilise Eq.2 in the manuscript following TDA and Tip-Adapter.\n\nLast but not least, we combine BoostAdapter with SSG as SSG+. In SSG+, we only apply our SSG operations for the positive cache in BoostAdapter, while keeping the negative cache and related operations identical to BoostAdapter.\n\n**W4&Q2**: *\"Missing augmentation-only ablation.\"*\n\n**A4**: Thanks for your suggestions. We supplement the augmentation-only ablation results below. Model-(a) means our SSG applying shape and style perturbation without PPD-based reweighting or cache updates, and model-(b) means our full SSG.\n| OOD Accuracy        | (a)    | (b)    |\n| ------------------- | ------ | ------ |\n| SSG (CLIP-RN50)     | 46.38 | 47.78 |\n| SSG (CLIP-ViT-B/16) | 63.58 | 65.20 |\n\nFrom the above table, we can find that our full model shows an obviously better performance than model (a). \n\n**W5&Q3**: *\"Details and analysis of k\\*.\"*\n\n**A5**: Thanks for your questions. We follow the operation in TPT and TDA, choosing k* as the top 10%,  and we keep the k* value for all experiments. We supplement the ablation results about the k* value, as shown below.\n| OOD Accuracy        | 5%     | 10%    | 15%    | 20%    | \n| ------------------- | ------ | ------ | ------ | ------ | \n| SSG (CLIP-RN50)     | 47.45 | 47.78 | 47.59 | 47.55 | \n| SSG (CLIP-ViT-B/16) | 64.89 | 65.20 | 65.04 | 65.02 | \n\nAs shown in the above table, we can find that setting k* as the top-10% achieves the best performance, which is consistent with the conclusion from TPT.\n\n**W6**: *\"Inconsistent ablations.\"*\n\n**A6**: Thanks for your valuable suggestions. We unify the ablation experiment based on the ViT-B/16 backbone and report the average OOD accuracy.\n\nFirst, we report the ablation results about shape-sensitive and style-insensitive factors. As shown in Fig.3 in the manuscript, we have given the comparison results, and we summarise below, while we supplement the baseline performance on which we built our SSG as stated in **A3**.\n| OOD Accuracy        | Baseline | PPD_st | PPD_sh | PPD    |\n| ------------------- | -------- | ------ | ------ | ------ |\n| SSG (CLIP-ViT-B/16) | 63.46   | 64.08 | 64.54 | 65.20 |\n\nSecond, we report the ablation results about hyperparameters (patch number) on the shape perturbation (patch shuffle) below. From the table, the same conclusion is shown as patch number as {4, 6} gives the better performance, and we set patch number to 4 for all experiments.\n| OOD Accuracy        | 2      | 4      | 6      | 8      | 16     |\n| ------------------- | ------ | ------ | ------ | ------ | ------ |\n| SSG (CLIP-ViT-B/16) | 64.38 | 65.20 | 64.91 | 63.47 | 62.52 |\n\nThird, the ablation results about the combination hyperparameters between PPD_st and PPD_sh are shown in Line317 in the manuscript, and we omit them for saving space.\n\nFourth, we report the ablation results about the effect of the PPD operation below. According to the results, we can find that both reweighting and comparison operations give an improvement,  and a combination of them gets the best performance.\n| OOD Accuracy        | PPD reweight | PPD comparison | Full Model |\n| ------------------- | ------------ | -------------- | ---------- |\n| SSG (CLIP-ViT-B/16) | 64.12       | 64.65         | 65.20     |\n\n**W7**: *\"Missing statistical variance in main results. Reporting mean ± std across runs would improve transparency and trust in the reported gains.\"*\n\n**A7**: Thanks for your feedback. We supplement the statistical variance and show it below. Due to the space limitation, more variance will be provided in the final manuscript.\n| Method               | ImageNet-A   | ImageNet-V   | ImageNet-R   | ImageNet-S   | OOD Average  |\n| -------------------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| SSG (CLIP-RN50)      | 31.54 (0.25) | 56.78 (0.18) | 63.77 (0.20) | 39.11 (0.12) | 47.78 (0.19) |\n| SSG (CLIP-ViT-B/16)  | 62.02 (0.14) | 65.32 (0.12) | 81.33 (0.18) | 52.13 (0.13) | 65.20 (0.13) |\n\n**W8-(1)&Q4**:  *\"When computing PPD, are the same top-k* low-entropy indices used across the original, shape-perturbed, and style-perturbed augmentations?\"*\n\n**A8**: We indeed utilise the same top-k* low-entropy indices used across the original, shape-perturbed and style-perturbed augmentations.\n\n**W8-(2)**: *\"Rename the subsection.\"*\n\n**A9**: We will rename this subsection as \"Hyperparameters about the combination between PPD_st and PPD_sh\" to clarify.\n\n**W8-(3)**: *\"Typos.\"*\n\n**A10**: We will accordingly modify the typos and standardise the acronyms for style-insensitive factors as STI.\n\n**Q6**:  *\"Have you tested SSG on other vision-language models?\"*\n\n**A11**: Thanks for your constructive comments. Our SSG can easily be applied to other vision-language models, and we follow DPE to involve the OpenCLIP (ViT-L/14) as an example for a fair and public comparison. We show the comparison results below.\n|                     | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Average |\n| ------------------- | ---------- | ---------- | ---------- | ---------- | ----------- |\n| OpenCLIP (ViT-L/14) | 53.88      | 67.69      | 87.42      | 63.18      | 68.13       |\n| TDA                 | 61.27      | 68.42      | 88.41      | 64.67      | 70.69       |\n| DPE                 | 61.09      | **70.83**  | 89.18      | 66.33      | 71.86       |\n| SSG (Ours)          | **62.85**  | 69.97      | **89.67**  | **66.56**  | **72.26**   |\n\nWe can observe that our SSG still outperforms training-free  TDA by 1.57%/ on average across 4 datasets, showing that our method generalizes well to larger-scale VLMs. And more vision-language models such as SigLIP or DFN will be explored in the future.\n\n**Q7**: *\"The ensemble row is somewhat ambiguous.\"*\n\n**A12**: We follow TPT and DPE to involve the \"Ensemble\" as the comparison method.  \"Ensemble\" means the baseline zero-shot CLIP prediction using the ensemble of 80 hand-crafted prompts from [1].\n\n[1] Learning transferable visual models from natural language supervision. ICML 2021."}, {"review_id": "kgKYC3zgPe", "reviewer": "Reviewer_8eZw", "summary": "This paper proposes SSG, a training-free test-time adaptation method for CLIP models that adds shape and style awareness. It introduces Perturbed Prediction Difference (PPD) to measure prediction sensitivity to shape versus style changes. Using this, SSG reweights features to focus on shape-relevant signals and filters cache updates to retain only informative, uncertain samples. The method boosts zero-shot robustness under domain shift and outperforms prior work without extra tuning or high cost.", "questions": "1. How would you describe the SSG method, is it a cache with an adapter? I feel like the structure of SSG needs more introduction at the beginning of Section 3.2. If you are using a cache, what is the cache capacity? Can you please explain the cache collection process?\n\n2. SSG updates the cache using both entropy and PPD criteria. How are these combined? For instance, do you require a sample to have both low entropy and high PPD to be cached, or is it a weighted combination? Can you also clarify the PPD criteria, does a higher PPD indicate better? This part is unclear. More details on the cache maintenance strategy would clarify the adaptation process.\n\n3. The results on ImageNet are missing. Could you please provide results on the ImageNet dataset with a main comparison to TDA, DPE, and BoostAdapter for reference?\n\n4. Could you please provide the results of the SSG+ method in Table 3 (cross-dataset generalization setting)?\n\n5. Equation 4 lacks intuition. How can we see from it that high confidence can be harmful? Can you provide more evidence for this claim?\n\n6. In the ablation studies, you introduce PPD reweighting and comparison. Could you identify all the operations under a single name, SSG or PPD, because it gets confusing what refers to what?\n\n7. Can you show in Figure 3 the effect of reweighting and a comparison using the ViT backbone to make the ablation study more consistent?\n\n8. There is no reference to Figure 1 in the manuscript. Can you please add the reference sentence in a future revision?\n\nIf the above concerns are addressed, I would be happy to raise my score. If space does not allow, feel free to skip Questions 6, 7, and 8.", "limitations": "The current shape/style augmentation operations are basic and more sophisticated techniques could be used. The assumption that emphasizing shape always helps is not discussed as a potential weakness.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The motivation is well-defined: most existing methods rely on entropy-based criteria to select features for the cache.\n- SSG is a novel approach that injects domain generalization insights into test-time adaptation of vision-language models.\n- The method is training-free and computationally efficient.\n- The results are strong. SSG achieves consistent accuracy gains over previous methods on two benchmarks. The enhanced version of the method, SSG+, further improves the performance. \n\n**Weaknesses:**\n- The method assumes shape features are always more reliable than style, but that’s not always true, some tasks depend heavily on style information. The paper doesn’t acknowledge this limitation.\n- The comparison on the ImageNet dataset is missing, which weakens the evaluation. Additionally, the backbone used in the ablation study is inconsistent.\n- The paper throws in too many abbreviations (SSG, SHS, SIS, PPD, STI), which makes it harder to follow and breaks the flow.\n- Key details in the method description are underexplained. The Questions section below outlines several points that need clarification to improve the paper’s clarity.", "was_revised": true, "final_justification": "As I mentioned earlier, I would increase my score if my concerns were addressed, therefore, I am raising my score to \"Accept\". The authors have now provided additional evidence supporting the applicability of their method. If the paper is accepted, I encourage the authors to open-source their work.", "rebuttal": "We sincerely thank you for your great efforts and insightful questions. We provide our feedback on weaknesses (W) and questions (Q) as follows.\n\n**W1**: *\"The method assumes shape features are always more reliable than style, but that’s not always true.\"*\n\n**A1**: Thanks for your valuable feedback. Motivated by Shape-Bias[1] and MixStyle [2], SSG utilizes the shape features (shape-sensitive factors and style-insensitive factors) to represent the generalizable factors, aiming to improve model's robustness. Meanwhile, we agree that there are always some tasks (e.g., texture-based medical image analysis) that depend on style information, and we will supplement this limitation in the \"Conclusion\" section.\n\n[1] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, 2018ICLR; [2] Domain generalization with mixstyle, 2021ICLR.\n\n**W2-(1)&Q3**: *\"The comparison result on the ImageNet dataset is missing.\"*\n\n**A2**: Thanks for your suggestion. We supplement the SSG performance on the ImageNet below.\n| CLIP-RN50    | ImageNet | CLIP-ViT-B/16 | ImageNet |\n| ------------ | -------- | ------------- | -------- |\n| TPT          | 60.74    | TPT           | 68.98    |\n| TDA          | 61.35    | TDA           | 69.51    |\n| DPE          | 63.41    | DPE           | 71.91    |\n| BoostAdapter | 61.54    | BoostAdapter  | 69.92    |\n| SSG          | 62.54    | SSG           | 71.21    |\n| SSG+         | 62.87    | SSG+          | 71.66    |\n\nAs shown in the above table, SSG/SSG+ gives superior performance compared with training-free methods TDA/BoostAdapter.\n\n**W3&Q6**: *\"Too many abbreviations (SSG, SHS, SIS, PPD, STI) in the paper.\"*\n\n**A3**: Thanks for your valuable feedback. For simplicity, we standardize the acronyms for \"style-insensitive factors\" as STI and \"shape-sensitive factors\" as SHS. Meanwhile, we maintain the SSG for the \"style and shape guidance\", and modify the perturbed prediction difference from PPD to SSG-score, while naming the operation \"PPD reweight\" as SSG-RW and \"PPD comparison\" as SSG-CP. We will modify all abbreviations in the final manuscript to make it clear.\n\n**Q1**: *\"How would you describe the SSG method, is it a cache with an adapter?  What is the cache capacity? Can you please explain the cache collection process?\"*\n\n**A4**: Thanks for your feedback. As you pointed out, our SSG can be described as a cache-based method with an adapter. As described in Section 3.1, SSG contains the dynamic visual cache, and SSG's final prediction is the combination of zero-shot prediction $P(f_{\\mathrm{test}})$ from Equation 1 and the adapted prediction $P_{cache}(f_\\mathrm{test})$ from Equation 2. \n\nWe supplement more details about cache design to clarify. Following the positive cache in TDA, we utilize the dynamic key-value cache for SSG, and keep the cache capacity as 3 for all datasets. The key-value pair corresponds to the testing features $f_{\\mathrm{test}}$ and the pseudo label $\\hat{l}$ from the prediction $P(f_{\\mathrm{test}})$.\n\nWe supplement more details about cache collection for clarify. For the cache collection process, the key-value cache is initially empty and then accumulates a sufficient number of key-value pairs during the test-time adaptation. SSG progressively incorporates test predictions with lower update criterion while limiting cache capacity. Specifically, if the number of the collected key-value pair is less than the cache capacity (3 for SSG), current testing key-value will add to the cache directly. If the number of collected key-value pair is more than the capacity, SSG will choose the existing key-value pair with highest update criterion, and compare it with the current key-value pair, and add the key-value pair with lower update criterion to the cache.\n\nThe update criterion is set to entropy of prediction $\\mathrm{H}(P(f_{\\mathrm{test}}))$ for previous cache-based method such as TDA or BoostAdapter, and our SSG modifies the update criterion as the combination of entropy and PPD (SSG-score), such as $ \\mathrm{H}(P(f_{\\mathrm{test}})) - \\mathrm{PPD}$  as described in last paragraph (Line213 - Line218) in Section 3.2. Meanwhile, SSG only contain one cache without any other negative cache like TDA. \n\nFollowing your suggestion, we will add more introduction at the beginning of Section 3.2.\n\n**Q2**: *\" How are entropy and PPD criteria combined? Does a higher PPD indicate better?\"*\n\n**A5**: Thanks for your valuable question. For the first question, we combine the entropy and PPD criteria by the minus operation, such as $ \\mathrm{H}(P(f_{\\mathrm{test}})) - \\lambda*\\mathrm{PPD}$. Through this operation, a sample with low entropy and high PPD will be cached just as you suppose. We set $\\lambda$ as 1 for all experiments, and the ablation experiments about the $\\lambda$ are shown in Section B.2 in the appendix, which are shown below.\n| OOD Average | 0.5   | 1.0   | 2.0   | 5.0   |\n| ----------------------- | ----- | ----- | ----- | ----- |\n| SSG(CLIP-ViT-B/16)      | 64.91 | 65.20 | 64.73 | 64.59 |\n\nFor the second question, a higher PPD means the sample contains more generalizable factors (shape-sensitive and style-insensitive factors), which is complementary to the entropy criteria for better performance. The ablation experiments about the effect of PPD reweighting (SSG-RW) and PPD comparison (SSG-CM) in Figure 3 in the manuscript show the improvement from the high PPD, and we show the results here.\n| OOD Accuracy        | PPD reweight | PPD comparison | Full Model |\n| ------------------- | ------------ | -------------- | ---------- |\n| SSG (CLIP-ViT-B/16) | 64.12        | 64.65          | 65.20      |\n\n**Q4**: *\"Results of the SSG+ method in Table 3.\"*\n\n**A6**: Thanks for your question. We supplement the results of the SSG+ method in Table 3 below, and SSG+ shows the better performance in almost all datasets compared with SSG in Table 3 in the manuscript.\n| Method              | Aircraft | Caltech | Cars  | DTD   | EuroSAT | Flower | Food101 | Pets  | SUN397 | UCF101 | Ave.  |\n| ------------------- | -------- | ------- | ----- | ----- | ------- | ------ | ------- | ----- | ------ | ------ | ----- |\n| SSG+(CLIP-RN50)     | 22.14    | 90.68   | 60.99 | 52.31 | 44.58   | 69.07  | 78.91   | 86.32 | 64.41  | 64.77  | 63.42 |\n| SSG+(CLIP-ViT-B/16) | 31.97    | 95.24   | 69.97 | 55.47 | 62.54   | 74.64  | 86.99   | 91.89 | 70.64  | 72.69  | 71.20 |\n\n**Q5**: *\"Equation 4 lacks intuition.\"*\n\n**A7**: Section A in the appendix shows more evidence for Equation 4, and we briefly summarise here. \n\nEquation 4 in the manuscript (Equation 10 in the appendix) is derived from the definition of the harmful samples. We define the harmful sample as one that reduces the differences in the mean logits between classes during the test-time procedure. Intuitively, we want the margin or differences between classes larger as much as possible, to obtain robust classification results. \n\nBased on this definition, we first use the properties of training-free (cache-based) VLM TTA to obtain the change in logits due to model's update in Equation 6,  shown as \n $\\Delta\\left(w\\cdot\\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)\\right)=\\Delta w(\\mathbf{x})\\cdot\\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right) \\approx \\mathbf{v}(\\mathbf{x}) \\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)$, \nwhere stored visual feature can be regarded as the residual update of the raw classifier. \nThen we obtain the change in the gap between the mean logits of samples from two classes as Equation 7, shown as $\\mathbf{v}(\\mathbf{x})\\cdot(E_{X_{+1}^{{test}}}[\\mathbf{v}(\\mathbf{x}^{{test}})]-E_{X_{-1}^{{test}}}[\\mathbf{v}(\\mathbf{x}^{{test}})]).$\nFinally, based on the disentangled factors, we approximate Equation 7 to Equation 10 (Equation 4 in the manuscript) as $\\mathbf{v_pp}(\\mathbf{x})\\cdot(E_{X_{+1}^{ {test }}}[\\mathbf{v_pp}(\\mathbf{x}^{ {test }})]-E_{X_{-1}^{{test}}}[\\mathbf{v_pp}(\\mathbf{x}^{{test}})])+\\mathbf{v_pn}(\\mathbf{x})\\cdot(E_{X_{+1}^{ {test}}}[\\mathbf{v_pn}(\\mathbf{x}^{{test}})]-E_{X_{-1}^{{test}}}[\\mathbf{v_pn}(\\mathbf{x}^{{test}})]).$\n\nTo make the differences large, we need to make Equation 10 positive. According to Equation 9, the term multiplied by v_pp is positive, while the term multiplied by v_pn is negative, so we need to highlight the generalizable factors v_pp. Meanwhile, according to Equation 10, the sample with high confidence but with significant v_pp factors is also harmful for the differences between the two classes.\n\n**Q7**: *\"Can you show in Figure 3 the effect of reweighting and a comparison using the ViT backbone?\"*\n\n**A8**: We supplement the average OOD accuracy results about the effect of reweighting (SSG-RW) and comparison (SSG-CM) based on the ViT-B/16 backbone below. According to the results, we can find that both reweighting and comparison operations give an improvement,  and a combination of them gets the best performance.\n| OOD Accuracy        | PPD reweight | PPD comparison | Full Model |\n| ------------------- | ------------ | -------------- | ---------- |\n| SSG (CLIP-ViT-B/16) | 64.12        | 64.65          | 65.20      |\n\nMoreover, we give the OOD average accuracy for the ablation results in the middle of Figure 3 to make the ablation consistent. This ablation experiment is about the patch number on the shape perturbation (patch shuffle). According to results, we set the patch number to 4 for all experiments.\n| OOD Accuracy        | 2      | 4      | 6      | 8      | 16     |\n| ------------------- | ------ | ------ | ------ | ------ | ------ |\n| SSG (CLIP-ViT-B/16) | 64.38% | 65.20% | 64.91% | 63.47% | 62.52% |\n\n**Q8&Typos**: *\"There is no reference to Figure 1 in the manuscript. And Formatting Concerns.\"*\n\n**A9**: Thanks for your valuable suggestions. We refer the Figure 1 in Line203, and we will move it to the first paragraph of Section 3.2 in the final version. And we will correct all typos in the final version."}, {"review_id": "0FhmDzgdvZ", "reviewer": "Reviewer_B6DJ", "summary": "The paper presents a training-free test-time adaptation method using dynamic key-value cache with pseudo-labels. The key contribution is employing perturbed prediction difference (PPD) as weighting criteria for entropy-based cache updates. PPD computation relies on two generalizable factors: shape-sensitive (SHS) and style-insensitive (SIS) perturbations. The method demonstrates improvements over existing approaches across two benchmarks using different backbone architectures.", "questions": "1. How does PPD and the theorical analysis differ from DeYO's work, and shouldn't this prior work be more explicitly credited rather than claimed as a contribution?\n\n2. Can you provide comparisons against simple baseline augmentations (e.g., random combinations of Gaussian noise and rotations) with PPD computation to demonstrate that your specific perturbation choices are essential? Additionally, please clarify whether the entropy computation uses the same shape and style perturbations as PPD, or different augmentations such as those employed in TPT?\n\n3. Can you provide the missing ablation studies for ResNet-50, particularly showing the individual effects of PPD_sh and PPD_st, as well as the impact of patch shuffling on CNN-based architectures?\n\nI am open to raising my evaluation score if the authors provide convincing responses to the questions outlined above and address the clarification points regarding target distribution access and the foundational method underlying SSG.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## **Strengths** \n**Writing Quality:**  The paper is exceptionally well-written with clear, accessible sections and comprehensive explanations.\n\n**Literature Positioning:** Authors effectively position their work as the first training-free test-time adaptation method to extend beyond entropy-only updates for visual-cache memory, incorporating style and shape factors.\n\n**Method Design:**  The approach is elegant and intuitive. Using PPD as an entropy weighting factor represents an interesting innovation that yields good performance improvements.\n\n**Computational Efficiency:** The method maintains low testing time with negligible computational overhead from additional shape and style augmentations.\n\n## **Weaknesses** \n\n**Theoretical Attribution:** The authors claim as a contribution the demonstration of generalizable factors through disentangled theoretical analysis (L68-70). However, this analysis directly replicates the theoretical framework from DeYO [1] without proper attribution. While the paper mentions following previous works (L151), it fails to clearly acknowledge that the theoretical analysis is identical to DeYO's contribution, which undermines the claimed novelty.\n\n**Insufficient Baselines:** The paper lacks comparison against straightforward augmentation baselines that combine geometric and photometric transformations. Although Table 1 in supplementary materials shows some shape perturbations, this evaluation is insufficient to demonstrate that the specific choice of style and shape perturbations is crucial rather than simply computing PPD on any reasonable augmentation strategy.\n\n**Limited Technical Novelty:** The core technical contribution, PPD, has been previously studied in DeYO under the name Pseudo-Label Probability Difference (PLPD). The authors do not clearly acknowledge that or distinguish their approach sufficiently from PLPD in DeYO. This raises questions about the technical novelty of the proposed method.\n\n**Incomplete Ablations:** The left and middle ablations of Figure 3 are only shown for Vit-B/16 but not for ResNet-50. Given that CNNs process information differently than transformers, it would be valuable to assess how PPD_sh and PPD_st perform individually on CNN architectures, as well as understand the effect of patch shuffling.\n\n### **Notes** \n\n**Typo:** in L13, it should be SIS instead of STI\n\n**Target Distribution Access:** In L91, the authors claim SSG lacks information about the target distribution. However, since SSG applies augmentations to test images and computes both entropy and PPD on them, it does have access to target distribution information. The distinction should be clarified between having access to target distribution versus training on it, as training-free methods can still operate on and extract information from test data.\n\n**Method Foundation:** The experiments section mentions SSG+ as a combination of BoostAdapter and SSG, but it's unclear what existing method SSG builds upon. Given the similarity to TPT (entropy-based approach with SSG adding PPD weighting), clarification is needed about whether SSG extends TPT or represents an independent development of similar concepts.\n\n[1] : Lee, Jonghyun, et al. \"Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors.\" ICLR 2024", "was_revised": true, "final_justification": "I appreciate the authors' responses to all of my questions. Their explanations have adequately resolved my initial concerns, and I will therefore raise my score to accept", "rebuttal": "We sincerely thank you for your constructive comments, and we  provide our feedback on weaknesses (W) and questions (Q) as follows.\n\n**W1&W2&Q1**: *\"How does PPD and the theoretical analysis differ from DeYO's work, and shouldn't this prior work be more explicitly credited rather than claimed as a contribution?\"*\n\n**A1**: Thanks for your feedback. We will depict the difference between our SSG and DeYO in the following points.\n\nFirst, we give the **theoretical analysis about** **generalizable factors for the training-free VLM TTA methods for the first time**. We appreciate the insight and theoretical contribution from DeYO, and we formally mention DeYO in Line151 and Line154, but DeYO only considers the entropy minimization back-propagation scenarios rather than considering the training-free (cache-based) cases. We extend the theoretical analysis to the training-free methods, and we think it is not trivial to accomplish this extension. Specially, DeYO calculates the gradient $\\partial \\operatorname{Ent}_\\theta(\\mathbf{x})/{\\partial w}$ when adapting through entropy minimization loss to determine the update of $w$, then determines the change in the gap between the mean logits of samples belonging to different classes. However, our SSG utilises the properties of the cache-based method, which can directly determine the update of $w$ by the stored visual feature $\\mathbf{v}(x)$ without the derivations from the gradient of entropy, such as $\\Delta\\left(w \\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)\\right)=\\Delta w(\\mathbf{x})\\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right) \\approx \\mathbf{v}(\\mathbf{x}) \\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)$, where stored visual feature can be regarded as the residual update of the raw classifier. Then we can obtain the change in the gap between the mean logits of samples from two classes as Equation 7 in the Appendix. Though the conclusion is the same, the core derivation of the two methods is different, as shown in Line15-Line21 in the Appendix and stated in Line 165 in the manuscript. We reveal the theoretical support for the cache-based (training-free) methods in a reasonable way, which can not be intuitively supported by DeYO theory without any modification. \n\nSecond, **our SSG is designed for the training-free VLM TTA settings, which are different from the DeYO settings.** DeYO aims to filter samples based on PLPD to benefit the entropy back-propagation. However, our SSG does not demand the entropy back-propagation, and aims to use PPD to highlight the augmented views in a reweight way, and supplement generalizable factors for the cache update. In terms of setting and specific operations, our SSG is different from DeYO. Moreover, DeYO only conducts the experiments on the single-modality scenarios, while our SSG conducts and verifies our design on large-scale vision-language models (CLIP) with promising performance and efficiency.\n\nThird, though SSG (PPD) and DeYO (PLPD) introduce the generalizable factors, **our SSG not only considers the shape-sensitive factors but also involves the style-insensitive factors** motivated by the MixStyle, which makes the generalizable factors more comprehensive and sound. And the ablation results about style-insensitive factors in Figure 3 left in the manuscript show the promising improvement, which is a good supplement to style-insensitive factors, as denoted in **A5**. As for the specific operations of PPD, we think it is common to measure prediction differences to reveal specific factors.\n\nBased on the above main points, we think our SSG is different from DeYO, and we will give more discussions about DeYO to make it more explicitly credited in Sections 3.1 and 3.2 in the final manuscript.\n\n**W2-1**: *\"The paper lacks comparison against straightforward augmentation baselines that combine geometric and photometric transformations.\"*\n\n**A2**: Thanks for your valuable comments. We supplement the comparison experiments between: (a) straightforward augmentation baselines, which means our SSG model without PPD-related operations but considers geometric and photometric transformations in the augmentations, and (b) our full SSG model.\n\n| OOD Accuracy    | (a)    | (b)    | OOD Accuracy        | (a)    | (b)    |\n| --------------- | ------ | ------ | ------------------- | ------ | ------ |\n| SSG (CLIP-RN50) | 46.38% | 47.78% | SSG (CLIP-ViT-B/16) | 63.58% | 65.20% |\n\nFrom the above table, we can find that our full model shows an obviously better performance than model (a). The reason is that geometric and photometric augmentation usually does not provide a more accurate prediction than the regular augmentation as shown in TPT/TDA, as they cause perturbations to the original image.\n\n**W2-(2)&Q2-(1):** *\"Can you provide comparisons against simple baseline augmentations (e.g., random combinations of Gaussian noise and rotations) with PPD computation to demonstrate that your specific perturbation choices are essential?\"*\n\n**A3**: Thanks for your suggestions. We supplement the comparison experiment between (a) PPD with Gaussian noise and rotations, and (b) PPD with shape patch-shuffle and style colour transformation with hue adjustment.\n| OOD Accuracy | (a)    | (b)    | OOD Accuracy | (a)    | (b)    |\n| ------------- | ------ | ------ | --------- | ------ | ------ |\n| SSG (CLIP-RN50)           | 46.02% | 47.78% | SSG (CLIP-ViT-B/16)       | 63.08% | 65.20% |\n\nFrom the above table, we can find that model (b) gives a much better performance than model (a). In fact, to obtain the PPD_st and PPD_sh, we need to corrupt/perturb the image's shape/style information, and patch-shuffle and colour transformation are suitable to reach this goal, because the augmented images obviously lose or change the shape/style information. But for the rotation and Gaussian noise augmentation, the shape information is perturbed slightly and the more than style information is destroyed, which even gives a negative impact on the performance.\n\n**Q2-(2)**: *\"Additionally, please clarify whether the entropy computation uses the same shape and style perturbations as PPD, or different augmentations such as those employed in TPT?\"*\n\n**A4**: Thanks for your question. We follow the augmentation in TPT/TDA to make the 63 augmented image views, and compute the entropy for these 64 (along with the raw image) images. For obtaining PPD_st and PPD_sh, we apply shape and style perturbations to the selected images (with high confidence) from 64 views, as described in Line182-Line183 and Line194-Line195.\n\n**W4&Q3**: *\"Can you provide the missing ablation studies for ResNet-50, particularly showing the individual effects of PPD_sh and PPD_st, as well as the impact of patch shuffling on CNN-based architectures?\"*\n\n**A5**: Thanks for your feedback. We supplement the ablation results with average OOD accuracy based on the CLIP-RN50 backbone.\n\nFirst, we supplement the ablation results about shape-sensitive and style-insensitive factors. Furthermore, we supplement the baseline method performance as described in **A8**. From the table, both shape-sensitive and style-insensitive factors give the better performance, while the combination gieves the best performance.\n\n| OOD Accuracy    | Baseline | PPD_st | PPD_sh | PPD    |\n| --------------- | -------- | ------ | ------ | ------ |\n| SSG (CLIP-RN50) | 46.33%   | 46.85% | 47.11% | 47.78% |\n\nSecond, we report the ablation results about hyperparameters (patch number) on the shape perturbation (patch shuffle) below. From the table, the same conclusion is show as patch number as {4, 6} gives the better performance, and we set patch number as 4 for all experiments.\n\n| OOD Accuracy    | 2      | 4      | 6      | 8      | 16     |\n| --------------- | ------ | ------ | ------ | ------ | ------ |\n| SSG (CLIP-RN50) | 47.03% | 47.78% | 47.51% | 46.93% | 45.23% |\n\nThird, to show the effect of the patch-shuffle, we supplement the ablation experiments about the different shape perturbation types as shown below, corresponding to the results in Table 1 in the appendix. It is obvious that patch-shuffle gives the best performance. The reason is that, pixel-shuffle is difficult to clarify the background and object, center-occlusion misses the image where the object is not at center, and rotation does not impact the object shape information efficiently.\n\n| OOD Accuracy    | Pixel-shuffle | Center-occlusion | Patch-shuffle | Rotation |\n| --------------- | ------------- | ---------------- | ------------- | -------- |\n| SSG (CLIP-RN50) | 44.63%        | 46.79%           | 47.78%        | 47.03%   |\n\n**W5**: *\"Typos.\"*\n\n**A6**: Thanks for your comments. We will correct all typos in the final version.\n\n**W6**: *\"Target Distribution Access.\"* \n\n**A7**: Thanks for your clarification. We will modify the Line91 in Section 2 to \"...yet they still require training on the target data with ground-truth labels. Differently, SSG aims to ..., where the model does not demand training on the target data with ground-truth labels.\".\n\n**W7**: *\"Method Foundation.\"*\n\n**A8**: Thanks for your feedback. We build our SSG on the naive cache-based method, and this baseline method only contains one positive cache and can be considered as the cache-based method TDA without a negative cache, which is different from the entropy-based TPT.\n\nFor the cache implementation details of baseline and our SSG, we follow the TDA default setting for the positive cache. Specifically, the utilised cache is a dynamic key-value cache, whose memory size is 3 for all datasets following TDA. For the cache collection process, the key-value cache is initially empty and then accumulates a sufficient number of key-value pairs during the test-time adaptation.For the similarity function in adapted prediction, we utilise Eq.2 in the manuscript following TDA and Tip-Adapter.\n\nWe will add the method foundation in the Subsection \"Implementation details\"."}, {"review_id": "oNGeOUFIOV", "reviewer": "Reviewer_rBMJ", "summary": "The paper proposes a method named Shape and Style Guidance (SSG) for training-free test-time adaptation in vision-language models. \nIt introduces the concepts of perturbed prediction difference (PPD) to measure the importance of shape-sensitive (SHS) and style-insensitive (SIS) factors during test-time adaptation.\nSSG incorporates PPD into the test-time procedure by reweighting high-confidence visual features and corresponding predictions based on PPD.\nThe method updates the dynamic visual cache not only by entropy but also by considering PPD, allowing the storage of samples with both high confidence and generalizable factors.\nThe paper conducts a number of experiments and claims state-of-the-art performance under specific experimental settings.", "questions": "1. Novelty and Distinction from DeYO\n\n * Concern : The paper's novelty appears limited, as many core ideas and mechanisms closely resemble those in the DeYO method. \n\n(1)PPD vs. PLPD: The proposed PPD (Perturbed Prediction Difference) and DeYO's PLPD (Pseudo-Label Probability Difference) seem to be conceptually identical. Both measure prediction differences to reveal sample dependencies on specific factors.\n\n(2) Generalizable Factors vs. Commonly CPR (Commonly Positively-coRrelated) factor: The paper claims to introduce \"generalizable factors\" (e.g., shape and style), but these concepts are already discussed in DeYO, which emphasizes the importance of shape as a CPR factor. The addition of style seems to be a minor extension.\n\n(3) Cache Update Strategy: The cache update strategy based on \"lower entropy and higher PPD\" appears to be very similar to DeYO's sample selection strategy based on \"lower entropy and higher PLPD\" (as shown in DeYO's Fig. 4). \n\n(4) Theoretical Analysis: The section on \"Theoretical Analysis about Generalizable Factors\" seems highly similar to DeYO's section on \"Entropy is Not Enough.\" Please clarify the unique theoretical contributions of this paper compared to DeYO. What new insights or theoretical advancements are provided that were not covered in DeYO?\n\n* Request: Please provide a detailed comparison and clarification of how the proposed method fundamentally differs from DeYO. Highlight the specific innovations and contributions that go beyond what DeYO has already achieved.\n\n2. Experimental Contributions\n\n* Concern: The Shape Guidance proposed in this paper has already been introduced in DeYO. The addition of Style Guidance appears to be a minor alteration. If the authors can show that the inclusion of style significantly enhances the performance or robustness of the model in ways that shape alone cannot, this would to some extent address concerns about limited novelty.\n\n* Request: Please provide experimental evidence or analysis that demonstrates the unique benefits of incorporating style in addition to shape. How does the inclusion of style guidance improve the model's performance or robustness in scenarios where shape guidance alone is insufficient?", "limitations": "The paper mentions the computational overhead of previous methods like TPT but does not demonstrate a significant improvement in computational efficiency compared to other training-free methods.The authors should provide a detailed comparison of computational efficiency with other training-free methods, highlighting any specific optimizations or improvements that their method offers.\n\nThis work utilizes both shape perturbation and style perturbation to corrupt the raw image. However, it is reasonable not to heavily use augmented views in test-time adaptation, as such operations (e.g., cropping, color perturbations) may lead to the loss or displacement of key semantic information. Forcing consistency across these augmented views can suppress the model's ability to discern fine-grained semantics, thereby affecting the final prediction.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nQuality: The paper is generally easy to follow and the methodology is clearly described. The experiments cover various out-of-distribution and cross-domain benchmarks, and the results are presented in a structured manner.\n\nWeaknesses:\n\nClarity: The paper fails to clearly articulate its differences from DeYO. The distinctions between the proposed method and DeYO are not well-explained, which affects the clarity of the paper. \n(DeYO: Entropy is not enough for test-time adaptation: From the perspective of disentangled factors.arXiv preprint arXiv:2403.07366, 2024.)\n\nOriginality: The paper's novelty is questionable due to significant similarities with the DeYO method. For detailed concerns regarding originality, please refer to the question section below. \n\nSignificance: The improvements over previous methods are incremental. The overall significance of the contributions is somewhat limited by the lack of clear differentiation from existing work.", "was_revised": true, "final_justification": "I appreciate the authors for their thorough and well-organized rebuttal, which has addressed part of my concerns. While I feel the novelty is moderate, the additional clarifications and ablation studies offers an enhancement. Therefore, I raise my score to 4.", "rebuttal": "Thanks for your comments, and we provide our feedback on questions (Q) and limitations (L) as follows.\n\n**Q1**: *\"Novelty and Distinction from DeYO.\"*\n\n**A1**: Thanks for your feedback. We will provide a detailed comparison and clarification between our SSG and DeYO in the following points.\n\nFirst, corresponding to (4), we give **the theoretical analysis about generalizable factors for the training-free VLM TTA methods for the first time**. We appreciate the insight and theoretical contribution from DeYO, and we formally mention DeYO in Line151 and Line154, but DeYO only considers the entropy minimization back-propagation scenarios rather than considering the training-free (cache-based) cases. We extend the theoretical analysis to the training-free methods, and we think it is not trivial to accomplish this extension. Specially, DeYO calculates the gradient $\\partial \\operatorname{Ent}_\\theta(\\mathbf{x})/{\\partial w}$ when adapting through entropy minimization loss to determine the update of $w$, then determines the change in the gap between the mean logits of samples belonging to different classes. However, our SSG utilises the properties of the cache-based method, which can directly determine the update of $w$ by the stored visual feature $\\mathbf{v}(x)$ without the derivations from the gradient of entropy, such as $\\Delta\\left(w \\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)\\right)=\\Delta w(\\mathbf{x})\\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right) \\approx \\mathbf{v}(\\mathbf{x}) \\cdot \\mathbf{v}\\left(\\mathbf{x}^{\\text {test}}\\right)$, where stored visual feature can be regarded as the residual update of the raw classifier. Then we can obtain the change in the gap between the mean logits of samples belonging to the two classes as Equation 7 in the Appendix. Though the conclusion is the same, the core derivation of the two methods is different, as shown in Line15-Line21 in the Appendix, and we reveal the theoretical support for the cache-based (training-free) methods in a reasonable way, which can not be intuitively supported by DeYO theory without any modification. \n\nSecond, corresponding to (3), **our SSG is designed for the training-free VLM TTA settings, which are different from the DeYO settings.** DeYO aims to filter samples based on PLPD to benefit the entropy back-propagation, however, our SSG does not demand the entropy back-propagation. Our SSG aims to use PPD to highlight the augmented views in a novel reweight way, and supplement generalizable factors for the cache update. Based on the theoretical analysis, it is natural to highlight \"lower entropy and higher PPD\", but in terms of setting and specific operations, our SSG is different from DeYO. Moreover, DeYO only conducts the experiments on the single-modality scenarios, while our SSG conducts and verifies our design on large-scale vision-language models (CLIP) with promising performance and efficiency.\n\nThird, corresponding to (2), though SSG (PPD) and DeYO (PLPD) introduce the generalizable factors, **our SSG not only considers the shape-sensitive factors but also involves the style-insensitive factors**. Style-related factors are important for the transfer learning methods, which have been proven in MixStyle[1], FourierDG[2], DSU[3] and more works. Therefore, the introduction of style-insensitive factors is important for the generalizable factors, which makes it more comprehensive and sound. Meanwhile, we have conducted the ablation results about style-insensitive factors in Figure 3 left in the manuscript, and style-insensitive factors show a promising improvement, which is a good supplement to their importance. Furthermore, we supplement more ablation results in the following **A2**, and get the same conclusion.\n\nCorresponding to (1), we acknowledge that PPD measures prediction differences to reveal sample dependencies on specific factors, which is similar to DeYO's formula. Meanwhile, we also think it is a common and efficient operation which has been deployed in many different transfer learning fields like [4] [5], and it is reasonable to use this operation to determine the generalizable factors.\n\nBased on the above main points, we think our SSG is different from DeYO, and we will give more discussions about DeYO to make it more explicitly credited in Sections 3.1 and 3.2 in the final manuscript.\n\n[1] Domain generalization with mixstyle, 2021ICLR; [2] A fourier-based framework for domain generalization,2021CVPR; [3] Uncertainty modeling for out-of-distribution generalization, 2022ICLR; [4] Style Normalization and Restitution for Generalizable Person Re-identification, 2020CVPR; [5] Generalized lightness adaptation with channel selective normalization, 2023ICCV.\n\n**Q2**: *\"Experimental Contributions. If the authors can show that the inclusion of style significantly enhances the performance or robustness of the model in ways that shape alone cannot, this would to some extent address concerns about limited novelty.\"*\n\n**A2**: Thanks for your suggestion. We have conducted detailed ablation results about the effect of PPD_st (style-insensitive factors) in Figure 3 left in the manuscript, and we supplement the average OOD accuracy based on CLIP-RN50 for further support. In the below table, the baseline method does not contain style-insensitive and shape-sensitive factors, PPD_st refers to style-insensitive factors, PPD_sh refers to shape-sensitive factors, and PPD refer the full model.\n\n| OOD Accuracy        | Baseline | PPD_st | PPD_sh | PPD    |\n| ------------------- | -------- | ------ | ------ | ------ |\n| SSG (CLIP-RN50)     | 46.33%   | 46.85% | 47.11% | 47.78% |\n| SSG (CLIP-ViT-B/16) | 63.46%   | 64.08% | 64.54% | 65.20% |\n\nFrom the above table, it is obvious that incorporating only the style-insensitive factors (PPD_st), the performance is improved from 46.33%/63.46% to 46.85%/64.08%, and further incorporation with shape-sensitive factors, the performance further enhance. The improved performance obviously demonstrates the effectiveness of the inclusion of style, and makes the generalizable factors more comprehensive and sound.\n\n**L1**: *\"The authors should provide a detailed comparison of computational efficiency with other training-free methods, highlighting any specific optimisations or improvements that their method offers.\"*\n\n**A3**: Thanks for your suggestions. \n\nFirst, SSG utilised shape and style augmentation is extremely simple to implement. For the shape augmentation, we use the patch-shuffle operation, which can be efficiently implemented by `erarrange func` in `einops packages`. For the style augmentation, we use the colour transformation with hue adjustment, which can be easily implemented by the `kornia package`.\n\nSecond, we apply the shape/style augmentation to the raw input images before the model forward procedure, and we concatenate them into a batch to feed into the model, which can be considered as the processing time is nearly similar.\n\nThird, after model feedforward and obtaining the logits, the following PPD calculation, reweighting and cache update is a simple mathematical calculation, which does not introduce much more inference burden. \n\nIn conclusion, our SSG shows the promising inference efficiency, and we give the detailed inference time comparison below. \n\n| Method     | Data Process | Model Forward | Rest Part | Total  |\n| ---------- | ------------ | ------------- | --------- | ------ |\n| TDA        | 15 ms        | 82 ms         | 34 ms     | 132 ms |\n| SSG (Ours) | 21 ms        | 83 ms         | 35 ms     | 139 ms |\n\nIt is obvious that SSG needs more data processing time due to the shape/style augmentation, and the other two parts need similar inference time, but the increased inference time is relatively small. Furthermore, we supplement the efficiency comparison in ImageNet-V, which contains 10000 images, in the following table.\n\n| Method     | Testing Time | OOD Accuracy | Gain  |\n| ---------- | ------------ | ------------ | ----- |\n| TDA        | 22 min       | 63.89        | +6.69 |\n| DPE        | 41 min  | 64.43        | +7.23 |\n| SSG (Ours) | 23 min 10 s`      | 65.20        | +8.00 |\n\nFrom the above table, even for the 1w image scale, our SSG only increase 1 minute compared with TDA while improving the OOD accuracy by about 1.3%. At the same time, our SSG only demands 56% inference time compared with DEP while achieving better performance.\n\n**L2**: *\"It is reasonable not to heavily use augmented views in test-time adaptation, as such operations (e.g., cropping, colour perturbations) may lead to the loss or displacement of key semantic information. Forcing consistency across these augmented views can suppress the model's ability to discern fine-grained semantics, thereby affecting the final prediction.\"*\n\n**A4**: Thanks for your comments. We do not force consistency across the raw images and their shape or style augmented images. Actually, we use shape and style augmentation to corrupt the raw images' shape and style information, aiming to measure the shape-sensitive and style-insensitive factors to further represent the generalizable factors."}]

Author rebuttal:

Meta-review: This paper proposes SSG, a training-free test-time adaptation method for CLIP models that adds shape and style awareness. It introduces Perturbed Prediction Difference (PPD) to measure prediction sensitivity to shape versus style changes. Using this, SSG reweights features to focus on shape-relevant signals and filters cache updates to retain only informative, uncertain samples.  The method demonstrates improvements over existing approaches across two benchmarks using different backbone architectures.

The paper initially received an average rating of 3 (borderline reject). T During rebuttal, the authors answered the reviewers questions to the satisfaction of the reviewers and two reviewers increased scores to "accept" and the other two to "borderline accept". There is some concern that the novelty of the method is incremental over the previous DeYO approach.

The ACs feel that the paper clearly meets the standards of a NeurIPS paper. The authors are encouraged to include some of the explanations and new results presented in the rebut in their revised paper.
Decision: Accept (poster)