Title: Training-Free Test-Time Adaptation via Shape and Style Guidance for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision-language model

test-time adaptation

transfer learning
Abstract: Test-time adaptation with pre-trained vision-language models shows impressive zero-shot classification abilities, and training-free methods further improve the performance without any optimization burden. However, existing training-free test-time adaptation methods typically rely on entropy criteria to select the visual features and update the visual caches, while ignoring the generalizable factors, such as shape-sensitive and style-insensitive factors. In this paper, we propose a novel shape and style guidance method (SSG) for training-free test-time adaptation in vision-language models, aiming to highlight the shape-sensitive (SHS) and style-insensitive (STI) factors in addition to entropy criteria.
Specifically, SSG perturbs the raw test image with shape and style corruption operations, and measures the prediction difference between the raw and corrupted one as perturbed prediction difference (PPD). Based on the PPD measurement, SSG reweights the high-confidence visual features and corresponding predictions, aiming to highlight the effect of SHS and STI factors during the test-time procedure. Furthermore, SSG takes both PPD and entropy into consideration to update the visual cache, aiming to maintain the stored sample with high entropy and generalizable factors. Extensive experimental results on out-of-distribution and cross-domain benchmark datasets demonstrate that our proposed SSG consistently outperforms previous state-of-the-art methods while also exhibiting promising computational efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method named Shape and Style Guidance (SSG) for training-free test-time adaptation in vision-language models. \nIt introduces the concepts of perturbed prediction difference (PPD) to measure the importance of shape-sensitive (SHS) and style-insensitive (SIS) factors during test-time adaptation.\nSSG incorporates PPD into the test-time procedure by reweighting high-confidence visual features and corresponding predictions based on PPD.\nThe method updates the dynamic visual cache not only by entropy but also by considering PPD, allowing the storage of samples with both high confidence and generalizable factors.\nThe paper conducts a number of experiments and claims state-of-the-art performance under specific experimental settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality: The paper is generally easy to follow and the methodology is clearly described. The experiments cover various out-of-distribution and cross-domain benchmarks, and the results are presented in a structured manner.\n\nWeaknesses:\n\nClarity: The paper fails to clearly articulate its differences from DeYO. The distinctions between the proposed method and DeYO are not well-explained, which affects the clarity of the paper. \n(DeYO: Entropy is not enough for test-time adaptation: From the perspective of disentangled factors.arXiv preprint arXiv:2403.07366, 2024.)\n\nOriginality: The paper's novelty is questionable due to significant similarities with the DeYO method. For detailed concerns regarding originality, please refer to the question section below. \n\nSignificance: The improvements over previous methods are incremental. The overall significance of the contributions is somewhat limited by the lack of clear differentiation from existing work.\n\nQUESTIONS:\n1. Novelty and Distinction from DeYO\n\n * Concern : The paper's novelty appears limited, as many core ideas and mechanisms closely resemble those in the DeYO method. \n\n(1)PPD vs. PLPD: The proposed PPD (Perturbed Prediction Difference) and DeYO's PLPD (Pseudo-Label Probability Difference) seem to be conceptually identical. Both measure prediction differences to reveal sample dependencies on specific factors.\n\n(2) Generalizable Factors vs. Commonly CPR (Commonly Positively-coRrelated) factor: The paper claims to introduce \"generalizable factors\" (e.g., shape and style), but these concepts are already discussed in DeYO, which emphasizes the importance of shape as a CPR factor. The addition of style seems to be a minor extension.\n\n(3) Cache Update Strategy: The cache update strategy based on \"lower entropy and higher PPD\" appears to be very similar to DeYO's sample selection strategy based on \"lower entropy and higher PLPD\" (as shown in DeYO's Fig. 4). \n\n(4) Theoretical Analysis: The section on \"Theoretical Analysis about Generalizable Factors\" seems highly similar to DeYO's section on \"Entropy is Not Enough.\" Please clarify the unique theoretical contributions of this paper compared to DeYO. What new insights or theoretical advancements are provided that were not covered in DeYO?\n\n* Request: Please provide a detailed comparison and clarification of how the proposed method fundamentally differs from DeYO. Highlight the specific innovations and contributions that go beyond what DeYO has already achieved.\n\n2. Experimental Contributions\n\n* Concern: The Shape Guidance proposed in this paper has already been introduced in DeYO. The addition of Style Guidance appears to be a minor alteration. If the authors can show that the inclusion of style significantly enhances the performance or robustness of the model in ways that shape alone cannot, this would to some extent address concerns about limited novelty.\n\n* Request: Please provide experimental evidence or analysis that demonstrates the unique benefits of incorporating style in addition to shape. How does the inclusion of style guidance improve the model's performance or robustness in scenarios where shape guidance alone is insufficient?\n\nLIMITATIONS:\nThe paper mentions the computational overhead of previous methods like TPT but does not demonstrate a significant improvement in computational efficiency compared to other training-free methods.The authors should provide a detailed comparison of computational efficiency with other training-free methods, highlighting any specific optimizations or improvements that their method offers.\n\nThis work utilizes both shape perturbation and style perturbation to corrupt the raw image. However, it is reasonable not to heavily use augmented views in test-time adaptation, as such operations (e.g., cropping, color perturbations) may lead to the loss or displacement of key semantic information. Forcing consistency across these augmented views can suppress the model's ability to discern fine-grained semantics, thereby affecting the final prediction.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a training-free test-time adaptation method using dynamic key-value cache with pseudo-labels. The key contribution is employing perturbed prediction difference (PPD) as weighting criteria for entropy-based cache updates. PPD computation relies on two generalizable factors: shape-sensitive (SHS) and style-insensitive (SIS) perturbations. The method demonstrates improvements over existing approaches across two benchmarks using different backbone architectures.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths** \n**Writing Quality:**  The paper is exceptionally well-written with clear, accessible sections and comprehensive explanations.\n\n**Literature Positioning:** Authors effectively position their work as the first training-free test-time adaptation method to extend beyond entropy-only updates for visual-cache memory, incorporating style and shape factors.\n\n**Method Design:**  The approach is elegant and intuitive. Using PPD as an entropy weighting factor represents an interesting innovation that yields good performance improvements.\n\n**Computational Efficiency:** The method maintains low testing time with negligible computational overhead from additional shape and style augmentations.\n\n## **Weaknesses** \n\n**Theoretical Attribution:** The authors claim as a contribution the demonstration of generalizable factors through disentangled theoretical analysis (L68-70). However, this analysis directly replicates the theoretical framework from DeYO [1] without proper attribution. While the paper mentions following previous works (L151), it fails to clearly acknowledge that the theoretical analysis is identical to DeYO's contribution, which undermines the claimed novelty.\n\n**Insufficient Baselines:** The paper lacks comparison against straightforward augmentation baselines that combine geometric and photometric transformations. Although Table 1 in supplementary materials shows some shape perturbations, this evaluation is insufficient to demonstrate that the specific choice of style and shape perturbations is crucial rather than simply computing PPD on any reasonable augmentation strategy.\n\n**Limited Technical Novelty:** The core technical contribution, PPD, has been previously studied in DeYO under the name Pseudo-Label Probability Difference (PLPD). The authors do not clearly acknowledge that or distinguish their approach sufficiently from PLPD in DeYO. This raises questions about the technical novelty of the proposed method.\n\n**Incomplete Ablations:** The left and middle ablations of Figure 3 are only shown for Vit-B/16 but not for ResNet-50. Given that CNNs process information differently than transformers, it would be valuable to assess how PPD_sh and PPD_st perform individually on CNN architectures, as well as understand the effect of patch shuffling.\n\n### **Notes** \n\n**Typo:** in L13, it should be SIS instead of STI\n\n**Target Distribution Access:** In L91, the authors claim SSG lacks information about the target distribution. However, since SSG applies augmentations to test images and computes both entropy and PPD on them, it does have access to target distribution information. The distinction should be clarified between having access to target distribution versus training on it, as training-free methods can still operate on and extract information from test data.\n\n**Method Foundation:** The experiments section mentions SSG+ as a combination of BoostAdapter and SSG, but it's unclear what existing method SSG builds upon. Given the similarity to TPT (entropy-based approach with SSG adding PPD weighting), clarification is needed about whether SSG extends TPT or represents an independent development of similar concepts.\n\n[1] : Lee, Jonghyun, et al. \"Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors.\" ICLR 2024\n\nQUESTIONS:\n1. How does PPD and the theorical analysis differ from DeYO's work, and shouldn't this prior work be more explicitly credited rather than claimed as a contribution?\n\n2. Can you provide comparisons against simple baseline augmentations (e.g., random combinations of Gaussian noise and rotations) with PPD computation to demonstrate that your specific perturbation choices are essential? Additionally, please clarify whether the entropy computation uses the same shape and style perturbations as PPD, or different augmentations such as those employed in TPT?\n\n3. Can you provide the missing ablation studies for ResNet-50, particularly showing the individual effects of PPD_sh and PPD_st, as well as the impact of patch shuffling on CNN-based architectures?\n\nI am open to raising my evaluation score if the authors provide convincing responses to the questions outlined above and address the clarification points regarding target distribution access and the foundational method underlying SSG.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SSG, a training-free test-time adaptation method for CLIP models that adds shape and style awareness. It introduces Perturbed Prediction Difference (PPD) to measure prediction sensitivity to shape versus style changes. Using this, SSG reweights features to focus on shape-relevant signals and filters cache updates to retain only informative, uncertain samples. The method boosts zero-shot robustness under domain shift and outperforms prior work without extra tuning or high cost.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The motivation is well-defined: most existing methods rely on entropy-based criteria to select features for the cache.\n- SSG is a novel approach that injects domain generalization insights into test-time adaptation of vision-language models.\n- The method is training-free and computationally efficient.\n- The results are strong. SSG achieves consistent accuracy gains over previous methods on two benchmarks. The enhanced version of the method, SSG+, further improves the performance. \n\n**Weaknesses:**\n- The method assumes shape features are always more reliable than style, but that’s not always true, some tasks depend heavily on style information. The paper doesn’t acknowledge this limitation.\n- The comparison on the ImageNet dataset is missing, which weakens the evaluation. Additionally, the backbone used in the ablation study is inconsistent.\n- The paper throws in too many abbreviations (SSG, SHS, SIS, PPD, STI), which makes it harder to follow and breaks the flow.\n- Key details in the method description are underexplained. The Questions section below outlines several points that need clarification to improve the paper’s clarity.\n\nQUESTIONS:\n1. How would you describe the SSG method, is it a cache with an adapter? I feel like the structure of SSG needs more introduction at the beginning of Section 3.2. If you are using a cache, what is the cache capacity? Can you please explain the cache collection process?\n\n2. SSG updates the cache using both entropy and PPD criteria. How are these combined? For instance, do you require a sample to have both low entropy and high PPD to be cached, or is it a weighted combination? Can you also clarify the PPD criteria, does a higher PPD indicate better? This part is unclear. More details on the cache maintenance strategy would clarify the adaptation process.\n\n3. The results on ImageNet are missing. Could you please provide results on the ImageNet dataset with a main comparison to TDA, DPE, and BoostAdapter for reference?\n\n4. Could you please provide the results of the SSG+ method in Table 3 (cross-dataset generalization setting)?\n\n5. Equation 4 lacks intuition. How can we see from it that high confidence can be harmful? Can you provide more evidence for this claim?\n\n6. In the ablation studies, you introduce PPD reweighting and comparison. Could you identify all the operations under a single name, SSG or PPD, because it gets confusing what refers to what?\n\n7. Can you show in Figure 3 the effect of reweighting and a comparison using the ViT backbone to make the ablation study more consistent?\n\n8. There is no reference to Figure 1 in the manuscript. Can you please add the reference sentence in a future revision?\n\nIf the above concerns are addressed, I would be happy to raise my score. If space does not allow, feel free to skip Questions 6, 7, and 8.\n\nLIMITATIONS:\nThe current shape/style augmentation operations are basic and more sophisticated techniques could be used. The assumption that emphasizing shape always helps is not discussed as a potential weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free test-time adaptation method for vision-language models, called Shape and Style Guidance (SSG). SSG aims to improve robustness to domain shifts by incorporating generalizable visual factors, specifically shape-sensitive and style-insensitive cues. It perturbs test-time images using shape and style transformations and measures the prediction difference compared to the original, forming a Perturbed Prediction Difference (PPD). This signal is used to reweight high-confidence visual features and guide cache updates, emphasizing features that are robust and informative. Experiments on several out-of-distribution and cross-domain benchmarks show that SSG achieves strong improvements over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Well Written.** The paper is well written and clearly structured, making the proposed method easy to follow and understand.\n\n2. **Interesting Idea.** Introducing shape-sensitive and style-insensitive cues for test-time adaptation is a novel and compelling idea that adds a new perspective to training-free adaptation.\n\n3. **Practicality.** The method builds on training-free adaptation techniques and avoids gradient-based updates, making it more practical and efficient for real-world deployment compared to optimization-based approaches.\n\n4. **Thorough empirical validation.** The experimental evaluation is thorough, with comparisons against a wide range of state-of-the-art baselines across both out-of-distribution and cross-domain benchmarks, where the proposed method consistently outperforms others.\n\n\n\n**Weakness**\n\n1. **Novelty.** While the paper is thoughtfully designed and thoroughly evaluated, the core idea of perturbation‐based reweighting feels like incremental improvement of existing cache-based TTA techniques rather than a transformative advance.\n\n2. **Efficiency concerns.** SSG is reported to have similar inference time to TDA, despite using roughly three times more augmentations per image (original, shape-perturbed, and style-perturbed). Unless forward passes are reused, this seems inconsistent. Clarifying these details is important to accurately assess the true computational cost.\n\n3. **Cache design ambiguity.** The paper does not clearly specify whether SSG is built on TDA, Tip-Adapter, or another cache-based method, leaving key implementation details—such as queue structure, and memory size—undefined. Additionally, if based on TDA, it is not explained how the entropy-minus-PPD criterion affects the negative cache or whether it is applied symmetrically. These ambiguities limit interpretability and reproducibility.\n\n4. **Missing augmentation-only ablation.** A missing ablation is a version of TDA that uses the same 3× augmentation strategy as SSG (original + shape + style), but without applying the PPD-based reweighting or entropy-PPD cache update. This would isolate the benefit of the diverse views themselves versus the actual contribution of SSG's mechanisms. \n\n5. **Missing details and analysis of** k*. One parameter,  k* , determines how many low-entropy augmentations are used for prediction reweighting and cache updates. However, the paper does not specify how k* is selected, nor does it study its impact on performance. Given its central role in PPD computation and feature aggregation, a sensitivity analysis or ablation would help assess the method's robustness.\n\n6. **Inconsistent ablations.** While the ablation studies are informative, they are inconsistent. Different components are analyzed on different datasets (e.g., ImageNet-V2, -R, -A, -S) and with different backbones (ResNet vs. ViT), making it difficult to compare relative contributions. A more rigorous evaluation would unify ablations on a consistent benchmark or report average OOD accuracy for all variants.\n\n7. **Missing statistical variance in main results.** Although the appendix includes significance testing, no variance metrics (e.g., standard deviations) are reported in the main tables. Given the sensitivity of test-time adaptation to augmentation noise and pseudo-labeling, reporting mean ± std across runs would improve transparency and trust in the reported gains.\n\n\n8. **Minor Issues.**\n- The paper likely uses the same top-k* indices across original and perturbed augmentations when computing PPD, but this should be stated explicitly to avoid confusion.\n- The subsection “Hyperparameters about Style-Insensitive Factors” focuses on the weighting between PPD_sh and PPD_st, but omits the actual style perturbation settings. It may be helpful to rename or expand the section to include these details.\n- Figure and terminology consistency. In the caption of Figure 1, both branches are labeled PPD_sh, though the right should be PPD_st. Acronyms for style-insensitive factors (e.g., “SIS” vs. “STI”) should also be standardized for clarity.\n\n---\nI welcome the authors' responses to the above concerns and am open to revising my assessment accordingly based on their clarifications.\n\nQUESTIONS:\n1. Could you please clarify whether SSG is implemented on top of TDA, Tip-Adapter, or another cache-based method? It would also be helpful to specify the cache structure (e.g., queue types and sizes, similarity function), and explain whether the entropy-minus-PPD criterion applies to both the positive and negative caches. If so, how are negative PPD values handled?\n\n2. Could you please include or comment on an augmentation-only ablation—e.g., applying shape and style perturbations in TDA without PPD-based reweighting or cache updates? \n\n3. How is the parameter k* determined? Is it a fixed value, dataset-dependent, or adaptively selected per image? Please consider clarifying this in the main text.\n\n4. When computing PPD, are the same top-k* low-entropy indices used across the original, shape-perturbed, and style-perturbed augmentations? If so, please make this explicit, as it’s important for reproducibility.\n\n5. SSG appears to use roughly 3× more augmentations than TDA, yet reports similar inference time. Could you please clarify whether forward passes are reused or shared across branches?\n\n6. Have you tested SSG on other vision-language models, such as SigLIP or SigLIP-2? It would be helpful to know if the approach generalizes beyond CLIP.\n\n7. In Table 1, the “ensemble” row is somewhat ambiguous. Could you please explain whether this refers to CLIP predictions averaged over all augmentations, or something else?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}