Title: Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language Models

Foundation Model

CLIP

Test-Time Adaptation

Generalization

Out-of-Distribution Robustness
Abstract: Vision–Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions, and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8× faster with a 12× smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces STS, an efficient Test-Time Adaptation technique for VLMs such as CLIP. Inspiring by the processing pipeline of previous works such as TPT, STS treats adaptation as an alignment problem between the visual and text spaces, particularly focusing on modifying the text features. A small set of parameters are learned in order to shift the right values matrix from the Singular Value Decomposition (SVD) of text features. A marginal entropy objective (based on augmentations on a single image) is used to adapt the linear variables, alongside L2 regularization. The method obtains competitive results on popular baselines for TTA.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\n1. The paper introduces an unexplored and interesting idea: adaptation can be carried out by modifying the text features in the most significant semantic directions based on SVD decomposition. This ensures to \"steer\" the text representation towarda a better alignment with the input images without incurring in expensive backpropagation, and enforcing a black-box adaptation.\n\n2. The experiments show competitive results on several of the popular datasets used in the field, and integrates MaPLe as well as CLIP, including previous adaptation strategies from the state-of-the-art. \n\n3. The writing is clear and the explanations are easy to follow. \n\nWeaknesses:\n\n1. Although the main contribution (SVD on the text space) is interesting, the cohesion with the rest of the ideas is not entirely clear.  The paper ZERO [5] proposes some intuition, but it's not discussed here. \n\n2. Popular datasets such as Imagenet-Domains and the CLIP's zero-shot suite are used, but some of the previous (and customary) datasets from TTA are disregarded; corruptions such as CIFAR-10/100-C or Imagenet-C, simulations as VisDA-C, etc. It is a trend in recent works to ignore the previous protocols (see TENT for instance).\n\n3. The comparison against the state-of-the-art is based solely on methods based on TPT. The paper intents to reduce the adaptation complexity, but that does not invalidate other gradient-based methods such as WATT, TENT, or parameter-free methods like TDA. The only difference is that they do not learn text prompts, but instead finetune the visual encoder.\n\nQUESTIONS:\n1. For instance, what's the advantage of performing marginal entropy minimization instead of conditional entropy minimization on filtered samples?\n\n2. Also, are the augmentations needed? What is the impediment of using a single image? Is the strategy of augmenting an image related solely to the fact that the marginal entropy is the objective function?\n\n3. If the domain shift is on the visual side, why not applying STS on the visual features so that they align better with the class concepts embedded in the text space?\n\n4. What is the performance on corruptions? Datasets such as CIFAR-10/100-C or Imagenet-C can be included. Oftentimes important gains can be obtained.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Spectrum-Aware Test-Time Steering (STS), a lightweight test-time adaptation (TTA) method for vision-language models (VLMs), specifically addressing the challenge of zero-shot generalization under domain shifts. STS works by steering latent text representations within a low-dimensional spectral subspace obtained from the singular value decomposition (SVD) of initial textual embeddings. This steering optimizes a minimal set of parameters (spectral coefficients) at inference, significantly enhancing model robustness and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of using an SVD-based spectral subspace to guide adaptation is novel and well-motivated, leveraging intrinsic low-dimensionality in text embeddings.\n2. The approach is parameter-efficient, significantly reducing computational overhead compared to existing TTA strategies.\n\nWeaknesses:\n1. The method's adaptation mechanism is linear, which may limit its effectiveness in handling more complex, nonlinear domain shifts. Such a limitation could restrict generalization performance under extreme or highly varied conditions.\n2. Although the paper presents an ablation study on the computational aspects, additional detailed ablations exploring the sensitivity to the choice of subspace dimensionality or impact of specific hyperparameters (like the number of augmented views) could further clarify robustness.\n3. The experimental results appear suboptimal on certain datasets. What could be the reasons for this?\n\nQUESTIONS:\n1. Although the method is simple and easy to understand, I believe it requires some theoretical justification or relevant citations to support the claims made in the paper.\n2. Is data augmentation for images necessary? Can the method perform without image augmentation?\n3. Can TTA-based methods or the method proposed in this paper be applied to a wider range of domains?\n4. Line 243: Inconsistent capitalization in the title.\n5. Line 176: Citation error.\n6. Lines 247, 253, and 256: Tables are missing citations.\n7. Line 312 in the provided document appears incomplete.\n\nLIMITATIONS:\nSee questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Spectrum-Aware Test-Time Steering (STS), a lightweight method for adapting vision-language models like CLIP to distribution shifts without using labels or updating model weights. STS identifies a low-dimensional semantic subspace via SVD of text prototypes and learns a small set of coefficients to shift all class embeddings in this space. The shift is optimized at test time by minimizing the entropy of predictions across augmented views, promoting confident and consistent outputs. STS achieves better or comparable performance to existing test-time adaptation methods (e.g., TPT, TPS), while offering reasonable gains in speed and memory efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nThe following are the strengths of the paper. \n\n1. The idea of constraining prototype shifts to a spectral subspace derived via SVD is reasonable. It introduces a well-motivated alternative to existing methods like TPT and TPS by grounding adaptation in semantically meaningful directions rather than unconstrained latent shifts or prompt tuning. \n\n2. STS operates entirely in latent space without requiring any backpropagation through the encoder or modifications to the model. This makes the approach efficient, reducing memory footprint compared to prior methods. \n\n3. The paper demonstrates consistent gains over zero-shot CLIP and prior TTA methods like TPT and TPS across a wide range of distribution-shifted and cross-domain datasets. The trend remains the same with a stronger MaPLe initialization. \n\n4. The paper is well written. The method is introduced step by step and the intuition behind using SVD to define the adaptation space is well-motivated. The use of entropy minimization as a label-free optimization objective is standard but well integrated. \n\nThe following are the weaknesses of the paper. \n\n1. All experiments are conducted only on CLIP ViT-B/16 which is now considered relatively outdated. With the increasing use of larger and stronger backbones like CLIP-L/14, SigLIP-SO400M [1], SigLIP2-SO400M [2] and Perception Encoder [3], it is unclear how STS scales or performs in those settings. \n\n2. While the core method is evaluated well, the paper lacks deeper ablations. For instance, how performance varies optimization steps, sensitivity to initialization, or impact of using alternative subspaces (e.g., PCA instead of SVD). These would improve our understanding of robustness and generality. \n\n---\n[1] SigLIP (SO‑400M): Zhai, Xiaohua, et al. \"Sigmoid Loss for Language Image Pre‑Training.\" arXiv preprint arXiv:2303.15343, 2023.\n\n[2] SigLIP 2 (SO‑400M variant): Tschannen, Michael, et al. \"SigLIP 2: Multilingual Vision‑Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.\" arXiv preprint arXiv:2502.14786, 2025.\n\n[3] Perception Encoder: Bolya, Daniel, et al. \"Perception Encoder: The best visual embeddings are not at the output of the network.\" arXiv preprint arXiv:2504.13181, 2025.\n\nQUESTIONS:\n1. How does STS perform on stronger VLMs such as CLIP-L/14, SigLIP2-SO400M or Perception Encoder L/14 & G/14? Even a small-scale evaluation (e.g., on a subset of datasets) would be highly informative. A detailed justification for why such models were not considered (e.g., computational limitations) would also be appreciated but ideally with pointers to expected behavior or work-in-progress. \n\n2. Can you provide a deeper ablation on the choice of spectral subspace, such as comparing SVD to PCA? A strong response showing that the method is either robust to subspace choice or that the current design is optimal, would strengthen confidence in the generality of STS and may improve my rating.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the well-known efficiency challenge of test-time prompt tuning (TPT) by proposing Spectrum-Aware Test-Time Steering (STS). The key idea behind STS is to selectively tune parameters at the output layer of the model. Specifically, it extracts a spectral subspace from the textual embeddings and learns to adapt a small number of per-sample shift parameters in a spectrum-aware manner. Empirical results demonstrate that the proposed method accelerates TPT by up to 8x and reduces GPU memory usage by up to 12x, all without compromising performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. This paper addresses an important issue in test-time prompt tuning, enhancing the practicality of this test-time technique.\n2. The empirical results are promising, significantly reducing computational costs while preserving performance.\n3. The writing is clear, and the paper is well-structured and easy to follow.\n\n### Weaknesses\n\n1. **Limited Scope of Evaluation**: While STS is presented as a plug-and-play technique applicable to various TPT methods, the experiments are restricted to TPT alone. Evaluating STS on additional methods, such as DiffTPT [1] and PromptSRC [2], would provide stronger evidence of its generalizability and effectiveness.\n2. **Missing Baselines**: The paper omits several important baselines that would help contextualize the contributions of STS. For example: 1) Directly ensembling results with negative entropy, which incurs no additional computational cost but can boost performance. 2) Updating only the last transformer layer, a potentially effective and efficient alternative. 3) Adding an identity matrix to the text features and optimizing its parameters. Including these intuitive baselines and demonstrating the gains of STS over them would significantly strengthen the paper.\n3. **More Performance-Efficiency Comparisons**: As the paper focuses on addressing efficiency, it would be valuable to compare STS with test-time training-free techniques for zero-shot image classification tasks, such as [3, 4]. These approaches demonstrate strong zero-shot performance while maintaining efficiency. Incorporating such comparisons and discussing the implications would elevate the paper's contributions and provide a more comprehensive analysis.\n\n [1] Self-regulating Prompts: Foundational Model Adaptation without Forgetting. NeurIPS 2023.\n [2] Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning. ICCV 2023.\n [3] A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models. ICML 2023\n [4] AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation. NeurIPS 2024\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of test-time adaptation of CLIP-like vision-language models, where the goal is to adapt the pretrained CLIP model on the fly for each test sample. A pioneering work that addresses the same task is [29], where the test-time adaptation is performed using prompt learning. This paper adopts the same framework, composed of image augmentation to multiple views and confident views selection. Yet, instead of tuning the prompt, which requires full backpropagation over the text encoder, it performs a singular value decomposition of the text embeddings matrix and identifies a spectral subspace with a basis formed from the largest right singular vectors. Then a vector is learned in this basis, which leads to a steering of initial class prototypes (i.e. text embeddings). The proposed method is faster and requires less memory than test-time prompt tuning, and its effectiveness is shown for out-of-distribution generalization and cross-dataset generalization.   \n\n[29] Shu et al., Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS 2022\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is written with a clear language and is easy to follow, contributions are well-conveyed and the method is well-illustrated (e.g. Figure 1). \n\n* Performing test-time adaptation of CLIP in the latent space instead of prompt tuning is a very interesting direction, especially thanks to the faster optimization and lower memory usage it induces.  \n\n**Weaknesses**\n\n*1- Missing important details* \n\nThere are several missing details about the method and experiments. For example, the actual value of $k_t$, the number of optimizable parameters for each dataset, is missing. The value of $\\lambda_R$, the regularization loss weight, is not specified as well.\n\n*2- The method*\n\nOn the method's level, the necessity of SVD decomposition is not justified enough. In other words, why adaptation isn't performed by just learning parameters $R\\in\\mathbb{R}^{C \\times D}$, initialized with zeros, on top of $Z_{T_{\\text{init}}}$ similarly to [A]? This means that there might be no justified need for optimization in a subspace, and a residual can be directly learned on top of the text prototypes. Additionally to $R\\in\\mathbb{R}^{C \\times D}$, which is class-dependent, another choice can be explored by learning $r\\in\\mathbb{R}^{D}$, a shared residual across all classes.\n\n*3- The experiments*\n\n* On the experimental level, when not using ensembling (i.e. 7 templates to encode class names), STS performance is very close to TPT except on ImageNet-A (cf. Table 1). The reasons why such a difference occurs only and specifically for this dataset are not discussed. Such discussion/interpretation is crucial to better understand insights about the effectiveness of the method, as well as estimating where using the proposed setting could be the best. \nOtherwise, when the ensembling is used, the performance gain is more significant. This is a good signal about the complementarity between using ensembling and the proposed adaptation approach, which can't be leveraged in prompt tuning methods like TPT. However, using the ensembling imposes exploring different design choices for the method. One of them could be the residual mentioned above [A]. Another choices could be simply tuning $Z_{T_{\\text{init}}}$, or using adapters [8], or tuning the projection matrix of the vision encoder [B]. All the mentioned methods allow for ensembling and do not perform prompt tuning, so no full backpropagation is performed. Such comparisons help corroborate the effectiveness of STS as design. \n\n* In Tables 1 and 2, STS is tested on top of MaPLe. This setting is a bit confusing as there are missing details about the so-called \" MaPLe initialization\". How is MaPLe trained in both tables? Is it trained in a few-shot labeled setting? On which dataset is it trained? Calling the MaPLe-initialized model \"zero-shot\" in the same tables is also confusing. Additionally, STS seems to degrade MaPLe performance in Table 2. Why does it happen in Table 2, while the same observation doesn't hold in Table 1? Interpreting this might help practical choices: deciding when to use STS on top of a method like MaPLe and when not.\n\n[A] Yu et al., Task Residual for Tuning Vision-Language Models. CVPR 2023                                          \n[8] Gao et al., Clip-adapter: Better vision-language models with feature adapters. IJCV 2024                              \n[B] Fahes et al., CLIP’s Visual Embedding Projector is a Few-shot Cornucopia. arxiv 2024\n\n\nMinor:\n\n* Line 176: Figure ?? -> Figure 2\n* Lines 247, 253: Table 1 not clickable\n* Line 256, 293: Table 3 not clickable \n* Line 306: whe -> when\n* Table 2 caption: cross-datesets -> cross-datasets\n\nQUESTIONS:\n* The paper claims that inference is 8$\\times$ faster than test-time prompt tuning (Lines 16,17). What does inference exactly refer to in this context? Lines 51-52 in the supplementary material indicate that inference includes the optimization steps, since it claims that inference efficiency decreases with more steps. This might be confusing since inference usually refers only to the computation of a prediction from an input, using only a forward pass.\n\n* Why is the cross-dataset generalization setting called so? I might have missed something here, but this setting is referred to in few-shot adaptation [18,39] when training on ImageNet and testing the generalization ability of the adapted model on other datasets. However, in this paper, in Table 2, test-time adaptation is performed separately for each test image in each dataset. Thus, the cross-dataset generalization name might be confusing as there is no training on one dataset and testing on others.  \n\n[18] Khattak et al., Maple: Multi-modal prompt learning. CVPR 2023                                 \n[39] Zhou et al., Conditional prompt learning for vision-language models. CVPR 2022\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}