Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
Authors: Konstantinos M. Dafnis
Dimitris N. Metaxas
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Foundation Model
CLIP
Test-Time Adaptation
Generalization
Out-of-Distribution Robustness
Primary area: applications
TL;DR: We propose Test-Time Spectrum-Aware Latent Steering (STS), a lightweight adaptation that learns a few per-sample steering parameters in latent space to align visual and text embeddings and boost zero-shot robustness.

Abstract: Vision–Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions, and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8× faster with a 12× smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
Konstantinos M. Dafnis* Department of Computer Science
Dimitris N. Metaxas
Department of Computer Science Rutgers University
Rutgers University
Abstract
Vision LaniunLL Models (VLMs) excel zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strate- gies have recently emerged as powerful techniques for adapting VLMs to single unlabeled image. However, existing adaptation strategies, such test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test- Test-Time Steering (STS), a lightweigtt adaptation framework that extracts a spectral sub- space from the textual embeddings to define principal semantic directions and learns to steer latent representations in a spectrum- aware manner by adapting a small num- ber of par-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses compares favorably against state-of-the-ar test-time adaptation methods, while introducing only handful of additional parameters and achieving inference speeds up 8x faster with 12x smaller memory footprint than conventional test-time prompt tuning. The code is available at https: ///g//gttbb///// com/ wwwfnis/////ccc.c..ao..ao seanfiissSSS...
Introduction
Vision-Language Models (VLMs), such CLIP [30], have marked a paradigm shift in artificial intelligence, demonstrating remarkable zero-shot generalization capabilities across a multitude of downstream visual tasks. By learning rich joint representations from vast quantities of image-text data, these models can often perform tasks like image classification without task-specific training, relying instead on natural language prompts define categories [32, 43J. This ability significantly reduces the need for extensive labeled datasets and the maintenance of numerous specialized models, paving the way for more versatile and scalable AI systems.
Despite their impressive zero-shot performance, the efficacy of VLMs can substantially compro- mised when encountering dut-of-distribution (OOD) data, where test samples exhibit characteristics different from those seen during pre-training [32]. Such distribution shifts are common in real- wellworldd applications, leading (TTA) has emerged as a crucial strategy, enabling models to dynamically adapt to unlabeled test samples on the fly, thereby enhancing robustness while preserving the benefits of zero-shot learning [36, 35, 23]. Episodic TTA, which adapts the model for each individual test sample, is particularly relevant for VLMs operating diverse and unpredictable environments [32, 7].
degradation in model reliability. To mitigate this, Test-Time Adaptation
Current TTA approaches for VLMs often focus on optimizing learnable components. Test-Time Prompt Tuning (TPT) [32] and its variants [39, 9] adapt textual prompts by minimizing objectives
"Conrepondencee to: ker03@escrleetma1..rue
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

like marginal entropy over augmented views of test sample. Although effective, these methods typically require backpropagation through the large text encoders of VLMs, leading to considerable computational overhead and increased memory usage during inference [34]. Other strategies involve parameter-efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA) [17, 18] applied to parts of the VL.M. However, these approaches often necessitate access to and modification of the model's internal architecture, deviating from a truly black-box paradigm and potentially limiting their applicability to proprietary models or those with fixed structures. The challenge remains to develop TTA methods that are both highly efficient and minimally invasive while effectively addressing domain shifts.
To address these limitations, we introduce Spectrum-Awar Test-Time Steering (STS), a novel TTA framework for VLMs that operates by efficiently adapting text representations within low- dimensional subspace defined by their Singular Value Decomposition (SVD). Instead of learning prompt vectors or modifying encoder weights, STS pre-computes a semantic basis from the SVD of the initial class text embeddings. At test time, for each incoming sample, our method learns a small set of coefficients that define a shift vector within this principal SVD subspace. This shift is then applied to the initial text prototypes, effectively steering them in the joint embedding space to better align with the current visual input. This approach directly manipulates representations in the latent space in a highly targeted and parameter-efficiet manner.
The core strength of STS lies in its strategy of adapting representations within a structured, low- dimensional subspace, a design motivated by the observation that embeddings derived from pre- trained deep neural networks are typically characterized by low intrinsic dimension [2]. This implies that their essential information resides within a lower-dimensional manifold, which can be effectively identified through spectral decomposition methods such as SVD. By operating on the principal singular vectors derived from the initial text embeddings, our method explicitly leverages the inherent semantic geometry of the VLM's text feature space. These singular vectors capture the most salient axes of variation among class concepts, providing a robust and semantically grounded basis for adaptation. Performing test-time adaptation within this constrained subspace inherently regularizes the learning process, fostering enhanced stability against noisy augmentations or idiosyncrasies of individual test samples. This targeted manipulation not only preserves the rich knowledge encoded in the frozen VLM but also ensures that adaptations are focused along directions of maximal semantic relevance, rather than allowing unconstrained shifts in the high-dimensional embedding space.
This work has four primary contributions:
We propose STS, a novel TTA method that, to the best of our knowledge, is the first to leverage the SVD-defined latent subspace of text embeddings for efficient and effective adaptation of VLMs to unlabeled test data.
Our method exhibits significant computational advantages: it avoids backpropagation through the VLM encoders and adapts only a minimal number of parameters (the SVD sub- space coefficients), leading to substantially lower latency and memory footprint compared to conventional prompt tuning techniques.
STS operates as a black-box adaptation mechanism, treating the VLM encoders as fixed fea- ture extractors without requiring knowledge of or modifications their internal architectures. This makes our approach broadly applicable and non-invasive.
Extensive experiments on several benchmark datasets for natural distribution shifts and cross-dataset generalization demonstrate that STS achieves state-of-the-an performance, enhancing the zero-shot capabilities of VLMs efficiently and effectively.
2 Related work
Vision-Language Models. Pre-trained on extensive image-text datasets via self relf-supervised learning. lasion-language models (VLMs) like CLIP [30] and ALIGN [19] exhibit impressive generalization abilities. For instance, CLIP's exceptional zero-shot performance largely stems from the scale and variety of its training data. However, effectively adapting these models to specific downstream tasks, especially in data-scares scenarios, continues to pose significant challenges. Efforts to improve the transferability of vision-language models have led to the use of prompt tuning techniques, including CoOp [43], CoCoOp [42], and MaPLe [21], I], and adapter-based methods, such as Tip-Adapter [40]
2

---

and HLLP-Adapter [10]. However, a prevalent assumption in these techniques is the accessibility of labeled data from the target domain, condition that is frequently incompatible with the requirements for rapid deployment in real-world applications. Therefore, we focus on test-time adaptation, defined as the challenge of a model adapting to a target domain using only the unlabeled test instances, with no access to any training data or ground-truth labels from that specific domain.
Test-Time Adaptation. Test-Time Adaptation (TTA) aims to improve model robustness and generalization by adapting pre-trained model unlabeled test data encountered during inference [35, 23, 36, 11, 26, 4, 32]. In the context of VLMs, several TTA strategies have emerged.
A dominant paradigm involves tuning the learnable prompt vectors at test time. Time Prompt Tuning (TPT) [32] pioneered this by optimizing textual prompts for each test sample to minimize the entropy of predictions over augmented views. Subsequent works have built upon TPT. such as DifTPP [9], which employs diffusion models for more diverse augmentations, and C-TPT [39], which focuses improving model calibration during TTA. Although these methods adapt VLMs without labeled data, they generally incur significant computational costs and memory overhead due the need for backpropagation through the large VLM encoders update the prompt parameters.
To address the efficiency concerns of prompt tuning, training-free approaches have been proposed. Most of these methods operate in an on-line streaming scenario, using memory banks to retain information from previously seen test inputs [41, 20]. However, such methods suffer from critical drawbacks that limit their applicability to real-life scenarios. First, maintaining memory banks significantly increases memory consumption, which becomes prohibitive for resource-constraing devices large-scale deployments. Second, their efficacy hinges the assumption of well- distributed stream of data, an unrealistic expectation practice, as imbalanced or nonstationary test distributions may prevent the memory bank from accumulating sufficient representative samples within a reasonable time-frame. This dependency renders them unreliable for time-sensitive applications scenarios with sparse or bursty data streams. Furthermore, memory- -based methods risk performance degradation when test samples arrive in biased sequences, as stored information may reflect transient patterns rather than meaningful statistical trends. Our STS method, while involving lightweight optimization step, shares the goal of minimizing encoder backpropagation and architectural changes.
Other approaches explore parameter-efficient fine-tuning (PEFT) techniques at test time. For example, TTL [18] adapts LoRA [17] parameters within the VLM's attention layers. Although more efficient than full fine-tuning, such methods still typically require modification of the underlying model architecture, differing from black-box approaches where the VLM encoders are treated fixed.
Directly adapting or modulating representations the latent embedding space offers an alternative to prompt tuning architectural modifications. Test-Time Prototype Shifting (TPS) [34] proposes learning shift vectors for pre-computed class prototypes directly in the embedding space, thereby avoiding backpropagation through encoders and achieving significant efficiency gains. This is conceptually related to our work. However, TPS learns unconstrained shift vectors for each class prototype. Our proposed STS method advances the idea of latent space adaptation by introducing spectrum-awand mechanism. Instead of learning arbitrary shifts, STS learns compact coefficients that define shifts along principal semantic axes derived from the SVD of text embeddings. This constrains the adaptation to a dimessonsiinna,, semantically meaningful subspace, aiming for both efficiency and effectiveness.
3 Method
In this section, we first provide preliminary details on Vision Vision-Language Models (VLMs) and the test-time adaptation setting. Subsequently, we introduce our proposed Spectrum-Aware Test-Time Steering (STS) framework. We detail its components for identifying principal spectral subspace from text embeddings, the mechanism for latent steering via a shared, learnable coefficient vector operating within this subspace, and the test-time optimization objective.
3

---

Singular Value Decomposition
Tani
X
Hurble
Text Prompt
Teatual Prototypis T
Predictions of all. augmented views
Backpropegation
Text Encoder
(CLASS).
Augmented Test Imagls
Confident Views Selection
Average
Image Encoder
666
armry
Figure 1: Overview of our proposed STS framework. Given text and image inputs, encoders E-)- and visual embeddings. A probability distribution
and E (-) extract text embeddings/proootypn FCLIP(U ye test) computed based on these embeddings. Then we perform refinement step of test-time adaptation, where we tune the learnable www-imensional coefficients to generate a small steering the text prototypes close the gap between the source and target distributions. Marginal entropy of the CLIP similarities of the shifted embeddings and the class prototypes is minimized.
3.1 Preliminaries
Vision-Language Models (VLMs). Pre-trained VLMs, such as CLIP [30], comprise a visual encoder (-) and textual encoder E.).), which map images and text descriptions into shared D-dimensional embedding space. For a C-class zero-shot image classification task, a set of initial text prototypes, Zrm {e C RCDD derived by encoding class names, typically using prompt templates (e.g.. "a photo of (class name)"). Given an input image x, its visual embedding 2, (x) R.D The predicted probability for class es:
exp(sim((er &e, (2Tm)e)/3 De)/T)
ply c|., ZT..
E exp(sim &os (2mm)))) /T/
where sim(s.) denotes cosine similarity, and is CLIP's learned temperature.
Test-Time Adaptation (TTA). Episodic TTA seeks to adapt a model for each unlabeled test
sample Pest by optimizing small set of parameters friA using objective Lrta derived from Tist (typically via augmentations). The adapted model f(; (TTA) is then used for prediction, and fYIA is reset for subsequent samples.
3.2 Spectrum-Aware Test-Time Steering (STS)
STS adapts VLMs at test time by learning to steer the initial text prototypes. This steering is enacted by single, shared vector of learnable coefficients that operates within a low-dimensional semantic subspace. This subspace critically derived from spectral properties (Singular Value Decomposition) of the initial text embeddings, ensuring adaptations are both efficient and aligned with principal semantic variations.
3.2.1 Spectral Subspace Identification from Text Prototypes
The core of STS lies in identifying robust, low-dimensional subspace from the initial text prototypes ZTT E CCDD to guide the adaptation. High-dimensional embedding spaces, while expressive, can be susceptible noise and may contain redundant information for specific adaptation tasks. Pre-trained

---

features from deep neural networks often exhibit a low intrinsic dimensionality [2], implying that their essential information can be effectively captured within a owwer-dimensionll manifold. By projecting the adaptation process onto such manifold, defined by the principal spectral components, STS aims to achieve more stable, generalizable, and semantically meaningful adaptations.
To this end, we perform a Singular Value Decomposition (SVD) on the initial text prototypes Zm.. Using the reduced SVD:
Zm. UTSTV
where Up E BCck' contains the left singular vectors, ST E IK' xk' is a diagonal matrix of K' min(C, ( D) singular values 81 2 82 2 ... 2 sR 2 0, and V E xD has rows corresponding to the right singular vectors. The columns of Vi E Dx (i.e., the right singular vectors of m represent an orthonormal basis for the subspace capturing the principal directions of variance in the text prototype data. These directions correspond to the most significant semantic axes that differentiate the class concepts as represented by ZTr.
We select the top k right singular vectors from VT (associated with the ky largest singular values) to form our textual adaptation basis Br [U1, 22....... Ux,] E Dxx,, The choice of k, < D is pivotal for efficiency and robustness. A small k, focuses adaptation on the most dominant semantic variations, potentially filtering out noise associated with higher-order components and leveraging the aforementioned low intrinsic rank of deep features [2]. Empirically, as we see in Figure 2, a small & often captures the vast majority (e.g., >90%) of the total variance (sum of squared singular values, or "energy") ") of ZT... Thus, for automatic and principled selection of kt, we employ the optimal hard thresholding strategy proposed by Gavish and Donoho [12]. This method determines an optimal singular value threshold w based on the aspect ratio of the matrix ZeT,, (i.e., C/D) and the median of its singular values. The rank Ky then the count of singular values 8; such that 8; wt. This pre-computed basis Bf defines the k-diimensional spectral subspace for our test-time steering.
3.2.2 Latent Steering via Subspace Coefficients
For each incoming test image Tiest STS learns single, shared vector of k, learnable coefficients E IR. These coefficients determine the magnitude and direction of the text prototype steering along each of the k, basis vectors in By. The steering vector (shift) Azy E R,,, which is applied to all class prototypes, is reconstructed from its ky-dimensional representation y:
AZr BTY-
The adapted text prototype (FT )c for each class c is then obtained by:
(2Tamma = normalize ((zTa)e + AzT).
(4)
The only parameters learned at test time are the k, coefficients in y.
3.2.3 Test-Time Optimization Objective
The shared steering coefficients ? are optimized for each Fiss using an unsupervised objective based on prediction consistency over N augmented views of Twat denoted (z(6) } Visual embeddings Zv (00) (x(0)) are extracted using the frozen E,..
Confidence Filtering. Following prior TTA works [32, 7], views are filtered based on prediction confidence using the initial (unadapted) text prototypes ZT Logits ( and probabilities p.()) are computed per view. Views with prediction entropy H H(P())) falling within the top-p percentile of init
confidence (lowest entropy) are retained, forming Zvi = {20 (1') of size NEk-
Marginal Entropy Minimization. For z E Zvan logits with current adapted text prototypes
Trague (from Eq. 4 using current 7) are: 110) (ddgpeaae) = sim( (Zrrsnne)))) The marginal probability distribution Patagied is:
Na P Padapted (c) Nan
Netmaxx.LLl' (c)).
(5)
5

---

Spectral Compression Text Prototypes
Singular
500
Sinqular Energy
Number of Singular Values
400
300
200
100
322
100
27
23
ImageNet
V2
Sketch
Plower1 02
DTO
OxfordPets
UCF101
Cattech101
Aircraft
BureSAT
StanforCCar
101
SUN997
Figure 2: Most spectral energy in CLIP text prototypes is captured by small subset of singular values, highlighting strong low-rank structure across datasets.
The primary objective is to minimize the Shannon entropy of Padaptedi
een H (Paaappud)
P Pattpped (c) log Padupted (c).
(6)
An 1.2 regularization term is added for Azr: freg ARl Azrll2- The total loss is:
LSTS = Lent + freg.
The coefficients initialized to and optimized to minimize LsTS-
3.2.4 Inference with Adapted Prototypes
After optimization yielding y", the final adapted text prototypes Zram are computed. The final class prediction ý for Feest is:
ό argmax
softmax, (sim(z!"), (2rmn)))//))
(
(8)
Ni
4 Experiments and Results
We conduct experiments on diverse range of benchmark datasets to assess the performance and robustness of our method, specifically testing its out-of-domain generalization across different domains.
4.1 Experimental Setup
Datasets. We conduct comprehensive evaluation of our method across diverse of benchmark datasets, with particular focus on out-of-domain generalization. To assess the model's ability to handle distribution shifts, we utilize several ImageNet variants, including ImageNet-A [16],
ImageNet-Ve V2 [31], ImageNet-R [14], and ImageNet-Skenn SSeeehhhh (also referred to as ImageNet-Kö [37]. These datasets serve established
(OOD) benchmarks for ImageNett enabling rigorous assessment the model's robustness under varying conditions and data distributions.
For Fine-graines Classification (also referred to as "Cross-Datasets Generalization in previous works),
in line with [32], we include Flowers [27], DTD [5], Pets [29], UCF [33], and Caltech
[8]. These datasets facilitate the evaluation of the model's capacity to distinguish fine-grained

---

Table 1: Comparison of top-1 accuracy (%) across ImageNet and its OOD variants. The best results in each section are highlighted in bold. Underline indicates bccond-best.
Method
ImageNet
A
V2
R
Sketch
Average
OOD Average
CLIP ViT-B/11 B/16
Zero-Shot [30]
66.73
47.87
60.86
73,98
46.09
59.11
57.2
Ensemble [28]
68.34
49.89
61.88
77.65
48.24
61.20
59.42
CoOp [43]
71.51
49.71
64.20
75.21
47.99
61.72
59.28
TPT [32]
68.97
54.39
63.37
77.07
48.01
62.36
60.71
DT [9]
70.30
55.68
65.10
75.00
46.80
62.28
60.52
C-TPT [39]
68.53
51.14
62.13
75.66
47.37
60.97
59.08
TPS [34]
67.96
57.46
62.95
74.90
46.03
61.86
60.34
STS (Ours)
68.85
61.23
64.15
77.13
48.06
63.88
62.64
Ensembbl
70.81
64.29
64.82
80.53
50.19
66.13
64.96
MaPLe
Zero-Shot [21]
50.90
64.07
76.98
49.15
60.28
TPT [32]
58.08
64.87
78.12
48. 16
62.31
STS (Ours)
64.83
66.49
79.43
50.62
65.34
variations among visually similar classes. Furthermore, to assess the model's adaptability across diverse domains, we incorporate Aircraft [25], EuroSAT [13], Cars [22]. Food [3]. and SUN397 [38]. encompassing broad spectrum of data modalities, including aerial and satellite imagery, object-centric datasets, and scene-centric environments. For datasets, we utilize the test splits defined by Zhou al. [43], adhering to the common evaluation protocol.
Implementation Details. Following TPT [32], generate 63 augmented versions of test image using only random resized crops and horizontal flips for all datasets, unlike prior methods that use spec-spe augmentations such AugMix, resulting in batch of 64 images including the original input. To identify nigh-confidence samples, we select the 10% of batch samples with the lowest entropy and compute the marginal entropy based their predicted probability distributions. The learnable vector initialized to zero and optimized for a single step using the Adam [24] optimizer with learning rate of 5e-3 across all datasets. In method, each class prototype is initialized using the hand-crafted prompt, "a photo of a {CLASS}." " All experiments are conducted on single NVIDIA RTX8000 GPU with 45GB of memory. The presented results are average taken three distinct random seeds. Top-1 accuracy reported in all tables, unless explicitly indicated otherwise.
Baselines. We evaluate our method against zero-shot and test-time adaptation (TTA) basclines that utilize CLIP ViT-B/16 as the backbone. The TTA methods considered include TPT [32], which performs text-prompt tuning: DiffTPT [9], a variant TPT that employs diffusion models to augment visual training data; TPS [34], which optimizes shift vector for each class prototype; and C-TPT [39], an extension TPT that improves model calibration by selecting prompts based on the dispersion of textual embeddings. To ensure fair comparison, reproduce TPT, TPS, and C-TPT using a single update step and the same backbone architecture. For DiffTPT, we report results from [9]. It is important note that the DifTTP study evaluates performance subset of each dataset that contain only 1,000 test samples, which may introduce potential variability in the reported results.
system
Textual Prompts. When Ensemble is specified, we do not use datasct-specific templates. In contrast, we use the set of 7 generic templates highlighted in the official CLIP repository [28] across all datasets.
4.2 STS Results
Natural Distribution Shifts. Table presents the top-1 accuracy of our method, comparing against zero-shot and test-time adaptation (TTA) baselines using CLIP on ImageNet and its out-
7

---

Table 2: Performance comparisons fine-grained classification. The best results in each section highlighted in bold. Underline indicates second-best.
Flowers102
OxfordPets
Caltech101
EaroSAT
StanfordCars
Food101
SUN397
Average
CLIP-VIT- BLT-
Zero-Shot [10]
67 44
44.27
88.25
65.13
93.35
23.67
42.01
65.48
83.65
62.59
63.58
Ensemble [25]
66.99
45.04
86.92
65.16
93.55
23.22
50.42
66.11
$2.86
65.63
64,59
CoOp (43)
68.71
41,92
89.14
66:35
93.70
18.47
46.39
64.51
85.30
64.15
63.88
TPT [32]
68.98
47.16
$7.07
67.89
94.19
22.85
43.01
66.55
$4.67
65.47
6d.78
CTTPT
65.26
93.23
TPS [34]
66.14
45.49
36.56
66.55
93.60
24.01
37.85
66.93
$2.96
64.85
63.49
STS (Ouns)
66.
46.02
86.09
66.52
93.72
24.57
38,
67.17
34.72
64.79
63.86
S7S
07.16
46,87
$7.11
67.14
94,20
24.21
43,80
68.16
85.15
66.79
65.06
MaPLe
Zero-Shot [21]
72.23
93.53
65.57
$6,20
67.01
66.30
[S]
72.37
93.59
STS (Oun)
70.70
47.60
90.00
68.94
94.02
25.44
40.83
68.32
36.56
67.26
65.97
of-distribution (OOD) variants. The results demonstrate that steering the text prototypes with our learnable vector, leads substantial performance improvement. Specifically, our approach achieves average OOD performance gain of 7.76% over the zero-shot CLIP baseline and a 4.23% improvement over standard TPT across OOD datasets.
Furthermore, Table shows that simply learning shift vector for per-class prototypes in TPS results slight performance drop (< 0.6 points) compared to TPT, highlighting the limitations of TPS in effectively aligning text prototypes with out-of-distribution visual embeddings. Additionally, our method is significantly more efficient, as STS runs times faster than TPT (see Table 3) while still achieving superior performance. These substantial speed-ups ups make our approach highly practical for real-world applications.
Table 3: Efficiency comparison on ImageNet. We report the testing time per sample, the memory usage, the accuracy, and the performance gains compared to zero-shot CLIP.
Method
Testing Time (s)
Memory (GB)
Accuracy
Gain
Zero-Shot
0.02
0.83
66.73
TPT
0.75
17.6
68.97
+2.24
STS
0.09
1.4
70.81
+4.08
Fine-grained Classification. We further assess the generalization capabilities of STS across ten diverse image classification datasets, with results presented in Table 2. This benchmark evaluates the model's ability to adapt to datasets that may differ significantly in domain and class composition from the VLM's s pre-training data.
In this challenging scenario, our STS variant, which leverages generic CLIP templates, achieves the highest average accuracy of 65. 65.06% among all methods compared. This result surpasses the standard TPT [32] (average 64.78%) and demonstrates the effectiveness of combining STS with prompt ensembling. The standard STS (using single "a photo of a a {CLASS}" prompt) achieves an average accuracy 63. 86%, which competitive and outperforms
Zero-Shot CLIP baseline (63.58%) It also surpasses other TTA methods such C-TPT [39] (63.58%) TTL [18] (63.59%), and TPS [34] (63.49% on average. Standard STS particularly excels on datasets like Aircraft (24.57%, best in its group), StanfordCars (67.17%, second best), and Food101 (84.72%, second best). The SaSemeemble variant shows broad strength, achieving the best results on Caltech101 (94.20%), EuroSAT (43.80%), StanfordCars (68.16%) Food101 15%), and SUN397 (66.79%) This indicates that while standard TPT a strong baseline, STS, especially when combined with prompt ensembling, offers more robust generalization across these diverse datasets.
Overall, STS demonstrates robust and often superior performance fine- fne-ggrined classification. The STSEmenhle variant, in particular, sets new state-of-the-as average across the CLIP-ViT- B/16 backbone experiments. Even with stronger MaPLe [21] initializations, STS provides a more effective adaptation than TPT. These results, combined with STS's significant ad advantages in parameter

---

efficiency and computational speed (detailed in Table 3), underscore its potential as a practical and powerful approach for real-world test-time adaptation of VLMs.
Analysis and ablation
We perform ablation studies to assess the effect of key design choices on performance. For consistency, all analyses use ImageNet and ImageNet-A with the ViT-B/16 backbone. Additionally, we evaluate STS on CIFARIO-C [15] to test its robustness under challenging distribution shifts.
5.1 Computational Analysis
Trainable Parameters: Test-time tuning approaches such as TPT, DiffTPT, and C-TPT adapt textual prompts using only 2048 trainable parameters, corresponding to four tokens with d=512. However, these methods exhibit limited generalization, achieving a Top-1 accuracy of approximately 51% to 55% on ImageNet-A. In contrast, visual adaptation techniques, such as encoder tuning and layer normalization optimization, require larger number of trainable parameters. Meanwhile, STS effectively balances this trade-off with just small number of trainable parameters, achieving the highest generalization performance at 61.23% while preserving model accessibility constraints. Table 3 shows the testing time per sample and the performance gain on ImageNet for STS and the TTA baseline TPT on single RTX8000 GPU. STS is 8x faster than TPT, corresponding to order of magnitude of computational savings in time.
5.2 Effect of STS on Different Prototypes
We study the impact of the stecring vector under different prototype constructions. Specifically, compare STS against the zero-shot CLJPP using the standard prompt "a photo of a (CLASS}" and an ensemble of seven generic hand-crafted templates from the official CLIP repository. While the design of TPT does not support the use of text ensembles, STS integrates them seamlessly; we denote this variant STS As shown in Tables and adding these generic prompts further improves STS, surpassing TPT without relying on dataset-specific templates.
Furthermore, we evaluate our method using MaPLe [21] initialization, where the MaPLe prompts are learned on ImageNet a 16-shot setup following [1]. In this evaluation, we also report results for TPT applied top of MaPLe, as in [1], which refer to as MaPLe+TPT. Although MaPLe+TPT performs better than previous methods that rely on hand-crafted prompts, our STS method notably outperforms MaPLe+TPT most datasets. This demonstrates that the adaptive mechanism of STS provides consistent advantages even when initialized with optimized textual and visual prompts.
Natural Distribution Shifts. Under natural distribution shifts, our STS method consistently out- performs TPT even when the baseline initialization MaPLe, achieving an average improvement of +3.03%. We omit evaluation ImageNet in this group, as MaPLe uses it as the source dataset for model adaptation, making the comparison unfair. For completeness, Zero-Shot MaPLe attains 70.72% accuracy on ImageNet, which further improves to 72. 72.72% when adapted with STS (+2.0%).. STS with MaPLe MMPLL++TTS) demonstrates leading performance on several natural shift datasets, including ImageNet-A (64.83%), ImageNet-Ve V2 (66.49%). ImageNet-SSetch (50.62%)
Fine-grained Classification. In fine-grained classification tasks, MaPLe+TPT shows a marginal average improvement of +0.52% over MaPLe+STS. However, this difference is primarily driven by performance on a single dataset, EuroSAT, while MaPLe+STS surpasses MaPLe+TPP on roughly half of the remaining datasets by considerable margin. For the datasets where MaPLe+STS lags behind, the differences remain minimal. As discussed in [7], EuroSAT constitutes a known failure mode for many TTA methods. Its analysis suggests that the unique nature of satellite imagery demands task-specific augmentation strategies, making EuroSAT a controversial benchmark for evaluating TTA performance.
5.3 Robustness of Linear Spectrum Steering
We evaluate STS on CIFAR10-C at the highest corruption level (severity 5) following the TPT protocol (10% most confident views, learning rate 0.005, and the hand-crafted prompt "a photo of

---

Avangge lopa Aecuroy
68
Methods
69.0
68.5
Average Top-1
63.31
82
61.73
67.3
67.0
Ensemble
STS (Ensemble)
the number of augmented views
(a) CIFAR I0-C comparison.
(h) Accuracy views N.
Figure 3: (a) Comparison on CIFARIO-C (severity 5). (b) Accuracy vs. number of augmented views.
(CLASS}"). STS matches TPT within 0.05% (Figure 3a), while clearly outperforming the naive per-class shifting (TPS), confirming the effectiveness of spectrum-aware subspace steering. With seven generic CLIP templates, STS reaches 67. 244, demonstrating strong complementarity between subspace steering and prompt ensembles. Constraining adaptation to the top-k singular vectors further stabilizes learning under severe corruptions.
5.4 Balancing Inference Efficiency and Accuracy
We analyze the impact of the number of augmented views N STS efficiency. As shown in Figure 3b, accuracy increases with N and saturates around N = 64. Increasing to 128 views yields only 15.15% gain while nearly doubling time and memory. We thus adopt N 64 (as in prior TPT work) to balance performance and efficiency, while remaining substantially faster than prompt-tuning baselines.
6 Limitations
While STS demonstrates significant advantages in efficiency and effectiveness for test-time adaptation, acknowledge two limitations that warrant discussion and offer avenues for future research:
I Linearity subspace steering. The adaptation mechanism involves linear shifts (steering) within the identified SVD subspace. While this subspace captures principal linear variations, highly complex or non-linear domain shifts might not be fully addressable by such linear adaptations alone, potentially requiring more sophisticated, non-linear mapping techniques within the latent space.
Linear complexity with respect to augmented views. Finally, although STS is notably more lightweight than current state-of of- TTA strategies, its computational demand for visual process- ing scales linearly with the number of augmented views due the need for independent forward passes. An intriguing future direction is to explore latent visual space augmentation to climinate these repeated encoder computations.
By addressing these current limitations, the robustness and performance of spectrum-aware test-time adaptation strategies could be further advanced.
7 Conclusion
We propose Spectrum-Aware Test-Time Stecring (STS), a lightweight adaptation framework for vision-langee lsiiun-aagguag models like CLIP. STS exploits the spectral structure of text embeddings to define a compact semantic subspace, where learns a per-sample steering vector to improve zero-shot robustness without modifying the frozen encoders. Our experiments show that STS consistently en- hances performance across diverse benchmarks, offering an efficient and practical test-timg time adaptation method.
Notably, in addition the text prototypes, our proposed STS method can be readily applied to the visual embeddings as well. Exploring under which conditions and settings the STS should be preferred over text prototypes or visual embeddings constitutes an interesting research direction that belongs our future agenda.
10

---

Acknowledgments
We thank Konstantinos D. Polyzos for insightful discussions and for his helpful and detailed comments. We also thank the reviewers for their constructive suggestions, which have helped improve the quality of this paper. This research has been partially funded by research grants to Metaxas through NSF: 23 2310966, 2235405, 2212301, 2003874, 1951890, AFOSR 23RT0630, and NIH 2R01HL127661.
References
[1] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. "Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization", In: Advances in Neural Information Processing Systems 36 (2023), pp. 80396-80413.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. "Intrinsic dimensionality explains the effectiveness of language model tie-ttnnng".. In: arXiv preprint arXiv:2012.13255 (2020).
[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. "Food- 101 -mining discriminative components with random forests". In: Computer vision-ECCV 2014: 13th European confer- ence, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. Springer. 2014, pp. 446-461.
[4] Evgenia Tatiani Chroni, Konstantinos M Dafnis, Georgios Chantzialexiou, Eric Cosatto, and Dimitris Metaxas. "Improving Test-Time Adaptation For Histopathology Image Segmenta- Ratio Guided Feature Alignment", " In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI). IEEE. 2024, pp. 1-5.
tion:
[5] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos. Sammy Mohamed, and Andrea Vedaldi. "Describing textures in the wild". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2014, pp. 3606-3613,
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large- scale hierarchical image database". In: 2009 IEEE conference on computer vision and pattern recognition. Teee. 2009, pp. 248-255.
[7] Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. "Frustratingly easy test-time adaptation of vision-language models". In: arXiv preprint arXiv:2405. 18330 (2024).
[8] Li Fei-Fei, Rob Fergus, and Pietro Perona. "Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories". In: 2004 conference on computer vision and pattern recognition workshop. IEEE. 2004, pp. 178- 178.
[9] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. "Diverse data augmen- tation with diffusions for effective test-time prompt tuning" In: Proceedings of the IEEÉ/CVF International Conference on Computer Vision. 2023, pp. 2704-2714.
[10] Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma. Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. "Clip-adapter: Better vision-language models with feature adapters". In: International Journal of Computer Vision 132.2 (2024), pp. 581-595.
[11] Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. "Visual prompt tuning for test-time domain adaptation". In: arXiv preprint AXXv:22/0.04423 (2022).
[12] Matan Gavish and David I. Donoho. "Optimal shrinkage of singular values". In: IEEE Trans- actions Information Theory 63.4 (2017), pp. 2137-2152.
[13] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. "Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification". In: IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing 12.7 (2019),
pp. 2217-2226.
[14] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. "The many faces of robustness: A critical analysis of out-of- distributti generalization" In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 8349
[15] Dan Hendrycks and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations" . In: arXiv preprint arXiv: 12261 (2019).
11

---

[16] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. "Natural adversarial examples". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 15262-15271.
[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. "Lora: Low-rank adaptation of large language models." In: ICLR 1.2 (2022), p. 3.
[18) Raza Imam, Hanan Gani, Muhammad Huzaifa, and Karthik Nandakumar. "Test-time low rank adaptation via confidence maximization for zero-shot generalization of vision-language models". In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE. 2025, pp. 5449-5459.
[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. "Scaling up visual and vision- -annula representation learning with noisy text supervision". In: International conference on machine learning. PMLR. 2021,
pp. 4904-4916.
[20] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb EI Saddik, and Eric Xing. "Ef- ficient test-time adaptation of vision-language models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 14162-14171.
[21] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. "Maple: Multi-modal prompt learning". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, pp. 19113-19122.
[22]
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. "3d object representations for fine- grained categorization". In: Proceedings of the IEEE international conference on computer vision workshops. 2013, pp. 554-561.
[23] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. "Tt+++: When does self-supervised test-time training fail or thrive?" In: Advances in Neural Information Processing Systems 34 (2021), pp- 17808-21820.
[24] Ilya Loshchilov and Frank Hutter. "Decoupled weight decay regularization". In: arXiv preprint arXiv: .05/0/ (2017).
[25] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. "Fine- grained visual classification of aircraft", In: arXiv preprint arXiv:1306.515) (2013). [26] Muhammad Jehanzeb Mirza, Pol Jané Soneira, Wei Lin, Mateusz Kozinski, Horst Possegger, and Horst Bischof. "Actmad: Activation matching to align distributions for test-time-training" In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 24152 22111.1.
[27] Maria-Elena Nilsback and Andrew Zisserman. "Automated flower classification over large number of classes", In: 2008 Sixth Indian conference on computer vision, graphics image processing. IEEE. 2008, pp. 722-729.
[28] OpenAL. CLIP: Contrastive Language- Image Pretraining.
CLIP. 2021.
[29]
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. "Cats and dogs". In: 2012 IEEE conference on computer vision and pattern recognition. IEFE, 2012, pp. 3498- 3505.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. "Learning transferable visual models from natural language supervision". In: International conference on machine learning. PmLR. 2021, pp. 8748-8763.
[31] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. "Do imagenet classifiers generalize to imagenet?" In: International conference on machine learning. PMLR. 2019, pp. $389-5400.
[32] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. "Test-time prompt tuning for zero-shot generalization in vision-language models". In: Advances Neural Information Processing Systems 35 (2022), pp. 14274-14289,
[33] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. "UCF101: A dataset of 101 human actions classes from videos in the wild". In: arXiv preprint arXiv: mtt.://220000 (2012).
[34] Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. "Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models". In: arXiv preprint arXiv:2403. 12952 (2024).
12

---

[35] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. "Test-time training with self-supervision for generalization under distribution shifts". In: International conference machine learning. PMLR. 2020, pp. 9229-9248.
[36] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. "Tent: Fully test-time adaptation by entropy minimization". In: arXiv preprint arXiv:2006.1072 (2020).
[37] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric Xing. "Learning robust global representations by penalizing local predictive power". In: Advances in neural information processing systems 32 (2019).
[38] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. "Sun database: Large-scale scene recognition from abbey zoo" In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE. 2010, pp. 3485-3492.
[39] Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. "C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion". In: arXiv preprint arXiv:2403, (2024).
[40] Renrui Zhang. Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. "Tip-adapter: Training-free adaption of clip for few-shot classification", In: European conference computer vision. Springer. 2022, pp. 493-510.
[41] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. "Dual memory networks: A versatile adaptation approach for vision-language models". In: Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition. 2024,
pp. 287 1--2888.
[42] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. "Conditional prompt learning for vision-language models". In: Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 2022, pp. 1681 18816-20 6-16825.
[43] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. "Learning to prompt for vision-language models". In: International Journal of Computer Vision 130.9 (2022), pp. 2337- 2348.
13

---

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
Appendix
In this supplementary document, we provide additional details and experimental results to enhance understanding and insights into our method. This supplementary document is organized follows:
Broader Impact of our method in Section A.
We present an error bar analysis for the results in Table and Table 2 in the main document, in Section B.1.
We provide an analysis on the impact of varying update steps in Section B.2.
We evaluate the effect of shared versus per-class coefficients vector in Section B.3.
We present additional performance comparisons on larger-scale VLMs, specifically OpenAI CLIP [30] with ViT-L/14 backbone, in Section B.4.
We analyze the effect of the singular vector selection for the test-time latent steering, in Section C.
We provide the detailed statistics for all the utilized datasets, and the specific textual prompts that we used in Section D.
We list the license information for all used assets in Section E.
A Broader Impact
This research contributes the overarching goal of developing more dependable and effective machine learning systems by enabling large foundation models like CLIP [30] to dynamically adapt real-world operational conditions at time. Such adaptability is critical for deploying AI robustly in diverse and unpredictable environments, thereby broadening their practical applications and fostering greater system reliability. Ultimately, we hope this work stimulates and guides future studies focused on enhancing the generalization capabilities and operational robustness of pre-trained models, ensuring they can be utilized more effectively and responsibly address a wide array of societal challenges.
B Technical Appendices
B.1 Analysis on error bars
We run STS multiple times using 3 different random seeds and report the average accuracy with stan- dard deviation in Table B1. The randomness STS mainly comes from random data augmentation. Our augmentation setup is simple and only contains resized random crops and random horizontal flips, which can constitute a "zoom-in" to a random portion of the image. We did not search for the best data augmentations, but rather stuck to established setting, using the same augmentation setup for all datasets. However, the performance of STS is linked to the impact that data augmentations have how the model perceives images, and believe that this is interesting research direction to pursue. In addition, we report an error bar analysis for the results in Table and Table 2 in the main document.
14

---

Table B1: Robustness to natural distribution shifts. We report the aceuracy with error bar (standard deviation) obtained from three runs with different random seeds. The best results in each section are highlighted in bold. Underline indicates second-best.
Method
ImageNet
A
V2
R
Sketch
Average
OOD Average
CLIP- VIT-B/16B
TPT [32]
68.97 (#.00) 54.39 (+ 20) 63.37 (#.06) 77.07 (=.06) 48.01 (#.08) 62.36 (#.03)
60.71 (#.04)
C-TPT [39]
68.53 02) 51.14 (#.09) 62.13 (1.11) 75.66 (=.07)
47.37 (*.08) 60.97 (#.02)
59.08 (#.02)
TPS [34] 67.96 02) 57.46 (2.12) 62.95 (#.11) 74.90 (=.04) 46.03 (#.09) 61.86 (#.06)
60.34 (#.07)
STS (Ours) 68.85 (d:03) 61.23 (d.26) 64.15 (#.20) 77.13 (#.06) 48.06 (2.06) 63.88 (d:08)
62.64 (2.10)
STS 70.81 (1.04) 64.29 (#.09) 64.82 (4) 80.53 (3)) 50.19 (2.02) 66.13 (2:01))
64.96 (1.005)
Table B2: Performance comparisons on cross-dateset generalization from ImageNet to fine-grained classification datasets. We report the accuracy with an error bar (standard deviation) obtained from three runs with different random seeds. The best results in each section are highlighted in bold. Underline indicates second-best.
Method Flowers102
DTD
OxfordPets UCF101
Caltech101 Aircraft
EuroSAT
StunfordCars Food101
SUN399
Average
CLIP-VIT-B/1
TPT L32] 68.98 (3:13) 47.16 is 87.07 c2 67.89 (= 94.19
12) 22 85 (8.43) 43.01 (
(6.55
84,67 de 65 (+.18) 64 78
C:TPT [39]
41,93.39
65.26
38
TTL [18]
(2.22 (2.29 15.92 (..02) 86.78 ==000)
67.80 (- 93.23
(0.00)
24.00
0 36 (000
(#:24)
84:40
(b.02) 61.02 (+.00)
63.
TPS [34] 66.14 (2:13)
dt 30 86 56 (6:06)
66.53 (= 18) 93.60
(2 24.01
(#:44) 37.85
66.93 18)
82.96
(t. 09) 64 85 (2.07) 63 19 07)
STS (Ours) 66.10 46.02
86.69
66.52
93.72
24.57 (8.00)
38.26
(7
84.72
61.79 63
STSn
67.16
(1.33 46.87
(1.12) 87.11
67.14
o6) 94,20
(1.00)
(1.05)24.22
24.21
(1.05)
43.50
(1)18)
(8.16.).25
85.15
(1.00) :1.00 65.06
(1.03)
B.2 Impact of Varying Update Steps
By default, STS updates the coefficients using single step per test instance. The optimal learning rate for this single step is determined to be 0.005 on the stan- dard ImageNet validation set (not including any of the out-of-distribution data). To evaluate the impact of different numbers of update steps on overall performance, we conduct ablation experiments by varying the number of update steps from to 5 and report the resulting performance on ImageNet-A, A. For these multi-step ablations, the value of An of the regularization loss is set to 0.01, and the initial learning rate remains 0.005. Since it is optimal for single update step, all subsequent steps are subjected to a learning rate schedule, applying one-time decay factor of 0.1 to the initial rate. As shown in Table B3, the number of update steps does not significantly influence perfor- mance (in the range of 0.1 1)). Although increasing the update steps to 2 yields slight performance gain of 0.02%, it also leads to a proportional decrease in inference efficiency. Although our method is extremely efficient, given this trade-off, we adopt the single-step update as the default for balancing efficiency and performance.
Table B3: Ablation study on different update steps in learning the steering vectors. We vary the number of update steps from 1 to 5 and report the achieved performance on ImageNet-A. Results are over random seeds.
# Steps
2
3
5
Accuracy 61.23
61.25
61.22
61.20
61.20
B.3 Effect of Shared vs. Per-Class Coefficients
Test-time adaptation strategies vary in how they modily class representations. For instance, prompt tuning methods adjust a shared prompt that subsequently undergoes non-linear transformations via the text encoder. In the context of latent-space adaptation, class prototypes can be modulated either by individual, per-class coefficients or by a single, shared coefficients vector. A shared vector applies a uniform transformation, thus maintaining the relative geometric structure of the class prototypes after adaptation. This approach primarily targets global, dataset-level distribution shifts. Although per-class vectors could, in principle, offer finer control by providing more degrees of freedom to capture distinct class-level shifts within a domain gap, the practical benefits of such granularity warrant careful consideration. Our work investigates the efficacy of the shared coefficients vector
15

---

approach, as the additional complexity introduced by per-class vectors may not consistently translate into substantial performance gains over simpler, unified shift.
As shown in Table B4, learning a per-class shift yields marginal average performance increase of only 0.03%. Similarly, Table B5 indicates that per-class coefficients provide mere 0.01 % average improvement in cross-dataset generalization from ImageNet to fine- line-grained classification tasks. These minimal gains suggest that, least for single update step, learning per-class coefficients does substantially enhance model performance when encountering a domain gap.
Table B4: A performance comparison of shared versus per-class steering vectors regarding robustness to natural distribution shifts. We present the average top- accuracy (%) results over random seeds for single update step. The best performance is highlighted in bold.
Method
ImageNet
A
V2
R
Sketch
Average
OOD Average
CLIP CLIP-ViT--e W-w-V/iTBBI
Shared 68.85
61,23
64.15
77.13
48.06
63.88
62.64
Per-class
68.91
61.24
64.20
77.15
48.06
63.91
62.66
Table B5: performance comparison of shared versus per-class steering vectors on cross-dateset generalization from ImageNet to fine- fine-graing --giiaaind classification datasets. We present the average top- accuracy (%) results over random seeds for a single update step. The best performance is highlighted in bold.
Method
Flowersi02
DTD
OrdordPets
UCF101
Caltech101
Aircraft
EuroSAT
StunfordCars
Food101
SUN397
Averuge
CLIP-VIT-E//
Shared
Pen-class
66.15
45.88
86.71
66.46
93.78
24.60
38.23
67.27
84.77
64.83
63.87
B.4
Performance Comparison Larger-Scale VLMs
Our STS method can theoretically be applied to various contrastively pre-trained vision-language models such CLIP ViT-B/16 and CLIP ViT-L/14. In Table B6, we OpenAI CLIP ViT- L/14 larger-scale OpenAl CLIP model, as an example, and compare the performance of our STS method and zero-shot on robustness to natural distribution shifts. We can observe that our STS still outperforms zero-shot by a large margin on average across 5 datasets, showcasing that our method generalizes well larger-scale VLMs.
Table B6: Performance comparison on robustness to natural distribution shifts. We present top- accuracy (%) results by employing the larger-scale ViT-L/14 variant of CLIP [30]. The reported results of STS are based on a single random seed. The best performance is highlighted in bold.
Method
ImageNet
V2
R
Sketch
Average
OOD Average
CLIP ELIP-ViTH ViT-L/14
Zero-Shot
73.45
68.76
67.79
85.39
57.81
70.64
69.94
STS (Ours)
75.37
78.52
69.88
88.07
59.85
74.34
74.08
A
+1.92
+9.76
+2.09
+2.68
+2.04
+3.70
+4.14
C
Singular Vector Selection for Test-Time Latent Steering
In our test-time adaptation (TTA) approach, Singular Value Decomposition (SVD) is applied to text prototypes (e.g., photo (CLASSJ") to analyze their underlying semantic structure. The full set of singular vectors describes this structure completely. However, when adapting new out-of-distribution (OOD) domain at test time, not all components of this original structure may be equally relevant contribute equally to successful adaptation. The process of selecting subset of
16

---

singular vectors aims to distill the dimensions most pertinent for characterizing the new domain's s relationship to the class concepts, potentially leading to a more focused and effective adaptation. As indicated in the main document (Section 3), substantial portion of the variance (e.g., 90%) is often concentrated in significantly smaller subset of singular values, highlighting the potential for effective dimensionality reduction. Discarding low-variance components here means removing directions where our specific classes are textually very similar.
This section details two principled methods for selecting k2 singular vectors and presents their impact on the zero-shot performance of CLIP [30] when integrated with our TTA method. Our findings indicate that strategic selection of singular vectors significantly enhances performance, with the method yielding slightly superior results.
C.1 Energy-Based Singular Vector Selection
common heuristic for dimensionality reduction via SVD is to retain the top-k, singular values, σ1 2 02 owe, such that they capture predefined percentage of the total "energy" (sum of squared singular values). We investigate a threshold of 98% energy, selecting ky such that:
0.98
2j=1
This method aims to preserve the most dominant components variance within the text prototype manifold, assuming these capture the most salient semantic information. Based on experiments k08% is typically less than K..
C.2 Gavish-Donoho Optimal Hard Thresholding
The Gavish-Donoke method [12] offers theoretically grounded approach for selecting an optimal number of singular values k, to retain, particularly when seeking a robust low-rank representation of the data. Developed from random matrix theory, this method computes an optimal singular value threshold w* This threshold designed to effectively separate the more dominant and structured components within the singular value spectrum from those that are less influential or exhibit characteristics similar to the singular values a random matrix. The singular values 0, < w (and their corresponding singular vectors) consequently excluded, leading the determination of the rank kG. The specific threshold value depends on the aspect ratio of the matrix undergoing SVD and can be established using the properties of the singular value spectrum itself (e.g.. via the median singular value), thus providing a data-driven cutoff without requiring an explicit "noise" model. We apply this method to determine k2 for text prototype matrix, in order to identify subset of singular vectors that is the basis of adaptation.
C.3 Performance Impact of Singular Vector Selection
To demonstrate the efficacy of these selection strategies, we evaluate our TTA method on ImageNet-A dataset. Table CI presents the top-1 accuracy, comparing the zero-shot CLIP baseline [30] with our TTA approach when using singular vectors selected by the 98% energy criterion (k98%) versus the Gavish-Donoho method (k)).
Table Cl: Impact of singular vector selection on Test-Time Adaptation (TTA) performance. Average Top-1 1 accuracy (%) over 3 random seeds is reported. Both selection methods significantly improve over the zero-shot baseline, with Gavish-Donohe (GD) offering a slight further advantage.
Dataset
Zero-Shot (ViT-B/16)
STS (Ours)
STS (Ours)
(ke5% SVs)
(k, SVs)
ImageNet-A [16]
47.87
61.09 (+13.22)
61.23 (+13.36)
The results in Table clearly indicate that employing a principled selection of singular vectors substantially boosts the performance of our TTA method compared to the zero-shot baseline. Both the 98% energy criterion and the Gavish-Donoho threshold lead to significant improvements. No- tably, the Gavish-Donoho method consistently achieves slightly better performance, suggesting its
17

---

effectiveness in identifying an optimal rank for the text prototype subspace used in our adaptation process. This underscores the importance focusing the adaptation on the most informative and semantic dimensions derived from the text prototypes.
D Additional Implementation Details
D.1 Dataset Details
In Table DI, we present the detailed statistics of each dataset we used in our experiments, including the number of classes, the sizes of training, validation and testing sets, and their original tasks.
Table DI:
Detailed statisties of datasets used in experiments. Note that the 4 ImageNet variant datasets are designed for evaluation and only contain the test sets.
Dataset
Classes
Training
Validation
Testing
Task
ImageNet [6]
1,000
1.28M
50,000
Object recognition
ImageNet-A [16]
200
7,500
Robustness adversarial attack
ImageNet-V22 [31]
1,000
10,000
Robustness of collocation
ImageNNet-R [14]
200
30,000
Robustness of multi-domains
ImageNet-SSetch [37]
1,000
50,889
Robustness sketch domain
Caltech101 [8]
100
4,128
1,649
2,465
Object recognition
DTD [5]
47
2,820
1,128
1,692
Texture recognition
EuroSAT [13]
10
13,500
5,400
8,100
Satellite image recogmition
FGVCAircraft [25]
100
3,334
3,333
3,333
tnae-graind aircraft recognition
Flowers 102 [27]
102
4,093
1,633
2,463
Fine-grained flowers recognition
Food101 [3]
101
50,500
20,200
30,300
Fine-graines food recognition
OxfordPets [29]
2,944
736
3,669
Fine- gra-ggrined pets recognition
StanfordCars [22]
196
6,509
1,635
8,041
Fine-grainedl car recognition
SUN397 [38]
397
15,880
3,970
19,850
Scene recognition
UCF101 [33]
101
7,639
1,898
3,783
Action recognition
In Table D2, we detail the specific hand-crafted prompts utilized in our experiments.
Table D2: Datasets with associated textual prompts. The first prompt is applied generally, while the subsequent generic prompts (indicated by the brace) are collectively used as an ensemble for each dataset. These generic templates are highlighted in the official CLIP repository [28].
Dataset
Prompts
ImageNet [6]
ImageNet-V2 [31]
"a photo of (CLASS)."
ImageNettSSkech [37]
ImageNet-A [16]
"a bad photo of the (CLASS)."
ImageNet-R [14]
Caltech101 [8]
"a (CLASS) in video game."
DTD [5]
"a origami (CLASS)."
EuroSAT [13]
FGVCAireraft [25]
"a photo of the small (CLASS)."
Ensemble
Flowers [27]
"art of the (CLASS)."
Food101 [3]
OxfordPets [29]
"a photo of the large (CLASS}."
StanfordCars [22]
"itap of a {CLASS)."
SUN397 [38]
UCF101 [33]
18

---

E
License Information
Datasets.
We list the known license information for the datasets below:
CC BY-SA 4.0 License: OxfordPets [29].
MIT License: ImageNet-A [16], ImageNet- V2 [31], ImageNet-R [14], and ImageNet-Sketch
[37].
Research Purposes only (term of access): ImageNet [6], DTD [5], StanfordCars [22].
SUN397 [38], FGVCAircraft [25].
Source Code. In this work, also to report their results: CLIP [30], CoOp [43]. MaPLe [21], TPT [32]. The source code used this
code implementations from existing baseline methods
paper for these methods is available under the MIT License.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our contributions in Abstract and also in the first Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We believe that we have clearly introduced the proposed STS method in the main text for reproduction. We have provided all the training details, and computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to release the code after the decision.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbccccccceececcccccccc /tt.dde//oooeeemmmiiieee /Cooeeeooiissseeeeeee wwssssnollicyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We followed all the standard benchmarks for our experiments following previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We plan to report error bars in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all the details in the Section of Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / htt////eurripp.eeeee ctteeippp.ccppuuuu cctppblic//eeeececcecccecc heroolcc//hhheee htt//Ehhiissitie csGuidelinest
Answer: [Yes]
Justification: We believe that the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not believe that this work has negative social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we will report them in the supplemental material.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:his paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: ttttps:/nneurte http//rrppss.cccoeeeeeeeeeeeeeee /LLM) for what should or should not be described.
26


Reviews:
[{"review_id": "UxrufPM3Ui", "reviewer": "Reviewer_EXj6", "summary": "This paper addresses the problem of test-time adaptation of CLIP-like vision-language models, where the goal is to adapt the pretrained CLIP model on the fly for each test sample. A pioneering work that addresses the same task is [29], where the test-time adaptation is performed using prompt learning. This paper adopts the same framework, composed of image augmentation to multiple views and confident views selection. Yet, instead of tuning the prompt, which requires full backpropagation over the text encoder, it performs a singular value decomposition of the text embeddings matrix and identifies a spectral subspace with a basis formed from the largest right singular vectors. Then a vector is learned in this basis, which leads to a steering of initial class prototypes (i.e. text embeddings). The proposed method is faster and requires less memory than test-time prompt tuning, and its effectiveness is shown for out-of-distribution generalization and cross-dataset generalization.   \n\n[29] Shu et al., Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS 2022", "questions": "* The paper claims that inference is 8$\\times$ faster than test-time prompt tuning (Lines 16,17). What does inference exactly refer to in this context? Lines 51-52 in the supplementary material indicate that inference includes the optimization steps, since it claims that inference efficiency decreases with more steps. This might be confusing since inference usually refers only to the computation of a prediction from an input, using only a forward pass.\n\n* Why is the cross-dataset generalization setting called so? I might have missed something here, but this setting is referred to in few-shot adaptation [18,39] when training on ImageNet and testing the generalization ability of the adapted model on other datasets. However, in this paper, in Table 2, test-time adaptation is performed separately for each test image in each dataset. Thus, the cross-dataset generalization name might be confusing as there is no training on one dataset and testing on others.  \n\n[18] Khattak et al., Maple: Multi-modal prompt learning. CVPR 2023                                 \n[39] Zhou et al., Conditional prompt learning for vision-language models. CVPR 2022", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is written with a clear language and is easy to follow, contributions are well-conveyed and the method is well-illustrated (e.g. Figure 1). \n\n* Performing test-time adaptation of CLIP in the latent space instead of prompt tuning is a very interesting direction, especially thanks to the faster optimization and lower memory usage it induces.  \n\n**Weaknesses**\n\n*1- Missing important details* \n\nThere are several missing details about the method and experiments. For example, the actual value of $k_t$, the number of optimizable parameters for each dataset, is missing. The value of $\\lambda_R$, the regularization loss weight, is not specified as well.\n\n*2- The method*\n\nOn the method's level, the necessity of SVD decomposition is not justified enough. In other words, why adaptation isn't performed by just learning parameters $R\\in\\mathbb{R}^{C \\times D}$, initialized with zeros, on top of $Z_{T_{\\text{init}}}$ similarly to [A]? This means that there might be no justified need for optimization in a subspace, and a residual can be directly learned on top of the text prototypes. Additionally to $R\\in\\mathbb{R}^{C \\times D}$, which is class-dependent, another choice can be explored by learning $r\\in\\mathbb{R}^{D}$, a shared residual across all classes.\n\n*3- The experiments*\n\n* On the experimental level, when not using ensembling (i.e. 7 templates to encode class names), STS performance is very close to TPT except on ImageNet-A (cf. Table 1). The reasons why such a difference occurs only and specifically for this dataset are not discussed. Such discussion/interpretation is crucial to better understand insights about the effectiveness of the method, as well as estimating where using the proposed setting could be the best. \nOtherwise, when the ensembling is used, the performance gain is more significant. This is a good signal about the complementarity between using ensembling and the proposed adaptation approach, which can't be leveraged in prompt tuning methods like TPT. However, using the ensembling imposes exploring different design choices for the method. One of them could be the residual mentioned above [A]. Another choices could be simply tuning $Z_{T_{\\text{init}}}$, or using adapters [8], or tuning the projection matrix of the vision encoder [B]. All the mentioned methods allow for ensembling and do not perform prompt tuning, so no full backpropagation is performed. Such comparisons help corroborate the effectiveness of STS as design. \n\n* In Tables 1 and 2, STS is tested on top of MaPLe. This setting is a bit confusing as there are missing details about the so-called \" MaPLe initialization\". How is MaPLe trained in both tables? Is it trained in a few-shot labeled setting? On which dataset is it trained? Calling the MaPLe-initialized model \"zero-shot\" in the same tables is also confusing. Additionally, STS seems to degrade MaPLe performance in Table 2. Why does it happen in Table 2, while the same observation doesn't hold in Table 1? Interpreting this might help practical choices: deciding when to use STS on top of a method like MaPLe and when not.\n\n[A] Yu et al., Task Residual for Tuning Vision-Language Models. CVPR 2023                                          \n[8] Gao et al., Clip-adapter: Better vision-language models with feature adapters. IJCV 2024                              \n[B] Fahes et al., CLIP’s Visual Embedding Projector is a Few-shot Cornucopia. arxiv 2024\n\n\nMinor:\n\n* Line 176: Figure ?? -> Figure 2\n* Lines 247, 253: Table 1 not clickable\n* Line 256, 293: Table 3 not clickable \n* Line 306: whe -> when\n* Table 2 caption: cross-datesets -> cross-datasets", "was_revised": true, "final_justification": "I would like to thank the authors for the rebuttal. The latter partially addressed my concerns. I acknowledge having missed TPS [31] from the tables, which indeed learns shifts of class prototypes. However, regarding [B], the authors state that: ´´[B] is evaluated using full-batch adaptation and has not been tested in an episodic TTA setting´´, which seems to be incorrect, as full-batch adaptation is only for few-shot adaptation experiments while TTA is performed in [B] the same way it is performed in the proposed SPS and the TPT [29] baseline, as Appendix E of [B] indicates. \n\nThat being said, I increased my score to 4 since [B] seems to be not published yet and I would not penalize the authors for not comparing with an arxiv paper. However, I highly encourage the authors to include this baseline to their work, as it seems to achieve strong results on TTA by simply tuning the projection layer and will absolutely help understanding whether performing SVD is necessary or not. I also highly encourage the authors to include the discussion about the apparent effectiveness of methods tuning parameters in the deep layers for ImageNet-A.\n\n[29] Shu et al., Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS 2022\n\n[31] Sui et al., Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models. WACV 2025\n\n[B] Fahes et al., CLIP’s Visual Embedding Projector is a Few-shot Cornucopia. arxiv 2024", "rebuttal": "Dear Reviewer EXj6,\n\nWe greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.\n\n**W1 - Missing Details**\n\nSpectral dimension K_t: We select K_t by applying the Gavish-Donoho thresholding rule to the singular-value spectrum of the text prototypes. For our CLIP ViT-B/16 experiments with the hand-crafted prompt “a photo of a” this yields to:\n\n\n| Dataset | ImageNet | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-Sketch | Flower102 | DTD | OxfordPets | UCF101 | Caltech101 | Aircraft | EuroSAT | StanfordCars | Food101 | SUN397 |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| $k_t$ | 106 | 31 | 106 | 37 | 106 | 20 | 7 | 10 | 19 | 16 | 29 | 3 | 51 | 19 | 72 |\n\n**W2 - The method**\n\nWe thank the reviewer for prompting a deeper comparison. Three points clarify why SVD-based steering outperforms both class-specific and shared residuals:\n\na) $\\textit{Structured, low-dimensional adaptation}$: As discussed in Section 3.2 (main manuscript), pretrained deep features exhibit a low intrinsic dimension [1], meaning their essential semantics lie on a smaller manifold. By performing an SVD of the full text-prototype matrix we extract the principal singular vectors that capture the most salient axes of class variation. Adapting within this constrained subspace inherently regularizes learning (preserving the rich, frozen knowledge of the VLM) and focuses updates along directions of maximal semantic relevance, rather than allowing noisy, unconstrained shifts in the D-dimensional embedding space.\n\nb) $\\textit{Relation to TPS’s residuals}$: In Section 2 (main manuscript) we describe Test-Time Prototype Shifting (TPS) [Sui et al., NeurIPS 2024], which indeed learns either a full class-specific residual $R ∈ \\mathbb{R}^{C×D}$ or a shared residual $ r ∈ \\mathbb{R}^D $ on top of $Z_{T_{init}}$​​, avoiding backprop through encoders similar to [A]. Our STS method advances this by introducing a spectrum-aware mechanism: rather than learning arbitrary shifts, we learn compact coefficients that steer prototypes only along the top-k singular directions. TPS’s experiments show that class-specific residuals modestly outperform shared ones, but both remain unconstrained and thus prone to overfitting under scarce, unlabeled test-time data.\n\nb) $\\textit{Empirical evidence}$: As reported in Section 4 (main manuscript), STS, using just $k_t$ parameters per sample, consistently outperforms TPS across all 15 benchmarks. This demonstrates that constraining adaptation to a semantically meaningful, low-rank subspace yields better OOD accuracy and stability than learning full or shared residuals in the high-dimensional prototype space.\n\nTogether, these points show that SVD-driven subspace adaptation is not only theoretically motivated by intrinsic dimensionality, but also empirically superior to both class-dependent and shared residual approaches.\n\n**W3 - The experiments**\n\nThanks for your thoughtful feedback. We acknowledge that our STS method largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8× faster with a 12× smaller memory footprint than conventional test-time prompt tuning. Here we discuss why without ensembling, STS and TPT yield closer performance on most ImageNet variants, but diverge sharply on ImageNet-A:\n\na1) ImageNet-A is made of “border-line” images that exploit spurious, low-level cues which CLIP has not fully suppressed. ImageNet-A was intentionally filtered to keep pictures whose foreground object is hard to spot (small, partially occluded, off-center) or whose background contains misleading texture cues. Its adversarial filtering process selects images that standard models misclassify (so they lie at the extremes of semantic variation, which align closely with the principal axes of the text-embedding manifold). Because every image already triggers the right class, the model’s output entropy is high and a tiny shift in the decision boundary flips a large fraction of errors into correct predictions. STS adapts by steering directly the text prototypes along the top-k singular vectors of the text-embedding matrix (directions that capture the dominant semantic axes e.g. object vs. background, canonical vs. extreme pose). Under the extreme distortions of ImageNet-A, the optimal correction indeed lies very close to one of those principal axes, so a single spectrum-aware steering recovers a disproportionate amount of accuracy. TPT’s prompt tokens must propagate their effect through multiple transformer layers, making it harder to affect such a large, coherent shift in a single gradient step (especially when tuning must remain lightweight). The same observation occurs for TPS method since as you can see by our experiments, TPS greatly improves performance over the TPT.\n\na2) Complementarity with ensembling: The framework requires a single representation per class, but if such a representation is derived from a single prompt, the amount of information that it carries is limited to the number of input tokens that the CLIP text encoder was trained with. Hence, we easily improve the robustness of class prototypes by taking the mean of each class embeddings. This allow us  to leverage multiple prompting and retain the knowledge from these various representations while keeping the computational and memory efficiency of our method without sacrificing prototype representation strength. As we mention in the paper, the design of TPT does not allow leveraging this text ensemble (also pointed out by concurrent work [31]).\n\na3) Alternative adaptation choice: Thank you for your suggestion. We note that TPS’s class specific and shared-residual variants (which mirror two of these suggested choices [A][Z_{T_{init}}] are already compared in Table 2 and fall short of STS’s performance (confirming the value of our spectral bottleneck). Clip-adapter [8] and [B] have a prevalent assumption which is the accessibility of labeled data from the target domain, a condition that is frequently incompatible with the requirements for rapid deployment in real-world applications. In addition, a full study of every ensemble-compatible alternative is beyond our current scope. We believe the principled regularization and semantic interpretability of SVD-based adaptation make STS uniquely effective, especially in the episodic black-box TTA that we investigate.\n\nWe thank the reviewer for pointing out the ambiguity around our MaPLe initialization and for asking about the differing TPT and STS effects in Tables 1 vs 2. We follow the official MaPLe implementation of Khattak et al. (CVPR 2023), which provides three pretrained prompt sets obtained via few-shot prompt learning on ImageNet, each corresponding to a different random seed. To avoid arbitrarily favoring any single seed, we associate one of these three weight sets with each of our three independent runs. In all tables, the entry “MaPLe zero-shot” therefore means “apply the respective ImageNet-trained MaPLe prompt to the target OOD benchmark without any additional adaptation.”  We will state this explicit mapping in the revision and annotate the tables accordingly to remove any confusion.\n\nWhy TPT and STS can slightly degrade MaPLe on cross-dataset TTA (Table 2)?\n\nSpecialized base prompt: MaPLe’s prompt tokens are finely tuned to the ImageNet distribution. When they are transferred to datasets with different semantics or styles, those tokens already sit close to a local optimum for many classes. As a result, subsequent shift-based updates (TPT or STS) have limited head-room and can occasionally over-correct, producing a small drop.\n\nFor the natural ImageNet variants, the ImageNet-trained MaPLe prompt remains reasonably well aligned with the target domain. In this case STS identifies truly helpful low-rank principal directions, yielding a net gain, while TPT achieves positive results as well. We will incorporate this explanation into the manuscript to make the interaction between MaPLe, TPT, and STS clear across all benchmarks.\n\n[Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.]\n\nQuestion 1 (“inference-time” wording)\n\nThank you for pointing out the possible ambiguity. In the submission we used the term “inference” to denote the entire test-time adaptation pipeline:\n\nThe time it takes to produce a first prediction for a previously unseen image, including any optimization / update steps that the TTA method performs on that very image. We agree that “inference” is often reserved for a single forward pass through a frozen model. To avoid confusion, we will replace “inference time” with “end-to-end adaptation latency” throughout the paper.\n\nQuestion 2 (“cross-dataset generalization”)\n\nThank you for drawing our attention to the possible confusion. In recent TTA literature the expression has been used in the following way:\n\nThis experiment was first designed in the TPT paper to compare few-shot prompt learning vs instance-specific prompt learning (without supervision).\n\nIn our case, there is no few-shot learning, so this would not apply. To avoid confusion with the few-shot prompt learning, we will rename this setting to fine-grained generalization in Table 2.\n\nTypos & citation links.\n\nAll figure/table references and minor errors (Lines 176, 247, 253, 256, 306) will be corrected."}, {"review_id": "TJIbCJ1FTM", "reviewer": "Reviewer_VSFB", "summary": "This paper tackles the well-known efficiency challenge of test-time prompt tuning (TPT) by proposing Spectrum-Aware Test-Time Steering (STS). The key idea behind STS is to selectively tune parameters at the output layer of the model. Specifically, it extracts a spectral subspace from the textual embeddings and learns to adapt a small number of per-sample shift parameters in a spectrum-aware manner. Empirical results demonstrate that the proposed method accelerates TPT by up to 8x and reduces GPU memory usage by up to 12x, all without compromising performance.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. This paper addresses an important issue in test-time prompt tuning, enhancing the practicality of this test-time technique.\n2. The empirical results are promising, significantly reducing computational costs while preserving performance.\n3. The writing is clear, and the paper is well-structured and easy to follow.\n\n### Weaknesses\n\n1. **Limited Scope of Evaluation**: While STS is presented as a plug-and-play technique applicable to various TPT methods, the experiments are restricted to TPT alone. Evaluating STS on additional methods, such as DiffTPT [1] and PromptSRC [2], would provide stronger evidence of its generalizability and effectiveness.\n2. **Missing Baselines**: The paper omits several important baselines that would help contextualize the contributions of STS. For example: 1) Directly ensembling results with negative entropy, which incurs no additional computational cost but can boost performance. 2) Updating only the last transformer layer, a potentially effective and efficient alternative. 3) Adding an identity matrix to the text features and optimizing its parameters. Including these intuitive baselines and demonstrating the gains of STS over them would significantly strengthen the paper.\n3. **More Performance-Efficiency Comparisons**: As the paper focuses on addressing efficiency, it would be valuable to compare STS with test-time training-free techniques for zero-shot image classification tasks, such as [3, 4]. These approaches demonstrate strong zero-shot performance while maintaining efficiency. Incorporating such comparisons and discussing the implications would elevate the paper's contributions and provide a more comprehensive analysis.\n\n [1] Self-regulating Prompts: Foundational Model Adaptation without Forgetting. NeurIPS 2023.\n [2] Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning. ICCV 2023.\n [3] A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models. ICML 2023\n [4] AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation. NeurIPS 2024", "was_revised": true, "final_justification": "The rebuttal addresses my concerns regarding the baseline experiments. However, the plug-and-play generality remains largely conceptual and is not supported by experimental results. I recommend that the authors include additional experiments to substantiate this aspect in their revision. Additionally, the authors declined to compare their approach with training-free baselines, due to their reliance on LLMs. However, the use of LLMs in downstream tasks is increasingly widespread and, in some cases, even essential (e.g., for never-seen classes in CLIP). Furthermore, test-time adaptation methods are known to be computationally intensive, which can limit their practical applicability. Therefore, a comprehensive comparison with alternative approaches, e.g., training-free methods, would be valuable. Based on these considerations, I keep my borderline acceptance and strongly encourage the authors to include the suggested experiments.", "rebuttal": "Dear Reviewer VSFB,\n\nThank you for your comments and positive recommendation of our work. We provide point-by-point responses to address your concerns below.\n\nWe would like to clarify that STS does **not** perform prompt tuning; instead, it **steers** the frozen text prototypes directly in a low-dimensional spectral subspace.\n\n**W1 - Limited Scope of Evaluation:** Thank you for the suggestion to evaluate STS alongside other adaptation pipelines. A few points address this:\n\na. **STS vs. DiffTPT as baseline**  \n   In Table 1, DiffTPT appears as an independent baseline; its diffusion-based augmentations are not combined with STS. While STS is fully compatible with any view generation scheme (including DiffTPT’s), our core goal is **ultra-efficient**, episodic black-box TTA: adopting DiffTPT’s augmentations would multiply per-sample latency and GPU memory by 5–10×, undermining STS’s real-time, low-footprint advantage.\n\nb. **Plug-and-play generality**  \n   Conceptually, STS can replace the gradient-through-prompt step in **any** TPT-style framework with a single spectral steering update. If a domain demands richer augmentations, one can simply feed those views into STS’s entropy-minimization loop (no changes to the SVD are required).\n\nc. **PromptSRC’s**  \n   PromptSRC [2] is designed for **few-shot** prompt tuning with labeled examples at test time. Because STS targets **zero-shot**, **unlabeled**, per-sample black-box adaptation, combining it with a few-shot method falls outside our stated problem setting.\n\n\n**W2 - Missing Baselines:** \n\na. **Directly ensembling results with negative entropy**\n\nWe thank the reviewer for these helpful suggestions. Below we explain how each of these baselines fits into our black-box, episodic TTA setting, and point to where we already cover or will cover them in revision:\n\nWe appreciate the reviewer’s suggestion to include a test-time ensembling baseline via averaging predictions over augmented views (i.e., without adaptation). We implemented this “zero-shot ensemble” baseline and tested it on the challenging ImageNet-A dataset. We found that it provides a +8.63% absolute improvement over standard zero-shot CLIP, increasing top-1 accuracy from 47.87% to 56.50%.\n\nWhile this confirms that multi-view ensembling can modestly improve robustness, our proposed STS method still significantly outperforms it, achieving up to 61.23% with a single hand-crafted prompt and 64.29% with ensemble.\n\nThis result demonstrates that STS goes beyond passive confidence averaging. It delivers active and semantically targeted adaptation in the latent space, leading to greater gains under distribution shift than ensembling alone.\n\nb. **Updating the last transformer layer**\n\nSuch a baseline requires white-box access to CLIP’s internal weights and architecture to backpropagate into a deep layer, contradicting our design goal of black-box TTA. STS assumes only query access to the model’s logits or prototypes, never internal gradients. For real-world deployment (e.g. proprietary APIs), updating a hidden layer is infeasible, so we do not evaluate it.\n\nc.  **Identity-residual on text features**\n\nLearning an matrix shift on the D-dimensional text embeddings is exactly the shared-residual variant of Test-Time Prototype Shifting (TPS) [31], and learning a full C×D residual is TPS’s class-specific variant. In TPS paper, the authors show that the class-specific variant performs slightly better than the shared-residual variant. This class-specific variant of TPS appears in Table 1 and Table 2. Across all 15 benchmarks, STS, using only a small number of spectral coefficients, consistently outperforms TPS class-specific residuals, demonstrating that our SVD-based bottleneck is a strictly stronger, more parameter-efficient adaptation mechanism that steers the textual prototypes along the most semantically meaningful directions.\n\n\n**W3 - More Performance-Efficiency Comparisons:**\n\nWe thank the reviewer for highlighting these training-free baselines, which indeed represent an important point on the performance-efficiency frontier.\n\nPrompt-weighting method, re-weights a fixed ensemble of hand-crafted prompts by minimizing the ensemble entropy. Unfortunately, no public code or details were released, so a faithful re-implementation is infeasible. In the revised manuscript, we will incorporate a qualitative discussion of it.\n\nAWT method relies on an external LLM to craft dataset-specific text prompts from prior knowledge, then re-weights and transports multi-view image features via optimal transport (still no backprop through the VLM but with significant dependency on the LLM and multiple heavy forward passes). AWT’s use of the external LLM to craft dataset-specific prompts brings a practical caveat: it requires prior knowledge of the entire target dataset’s class taxonomy and domain characteristics in order to generate tailored descriptions. In true episodic, zero-shot TTA settings, where each sample arrives in isolation and no labeled or domain-wide metadata is available, this a priori dataset overview is typically unavailable, limiting AWT’s applicability. In contrast, STS operates purely on the model’s frozen prototypes and per-sample augmentations, with no dependence on external models or dataset-level information. These additions will clarify why, even among training-free methods, STS occupies a unique point on the performance-efficiency frontier."}, {"review_id": "USTzVsqjth", "reviewer": "Reviewer_2jDh", "summary": "This paper proposes Spectrum-Aware Test-Time Steering (STS), a lightweight method for adapting vision-language models like CLIP to distribution shifts without using labels or updating model weights. STS identifies a low-dimensional semantic subspace via SVD of text prototypes and learns a small set of coefficients to shift all class embeddings in this space. The shift is optimized at test time by minimizing the entropy of predictions across augmented views, promoting confident and consistent outputs. STS achieves better or comparable performance to existing test-time adaptation methods (e.g., TPT, TPS), while offering reasonable gains in speed and memory efficiency.", "questions": "1. How does STS perform on stronger VLMs such as CLIP-L/14, SigLIP2-SO400M or Perception Encoder L/14 & G/14? Even a small-scale evaluation (e.g., on a subset of datasets) would be highly informative. A detailed justification for why such models were not considered (e.g., computational limitations) would also be appreciated but ideally with pointers to expected behavior or work-in-progress. \n\n2. Can you provide a deeper ablation on the choice of spectral subspace, such as comparing SVD to PCA? A strong response showing that the method is either robust to subspace choice or that the current design is optimal, would strengthen confidence in the generality of STS and may improve my rating.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The following are the strengths of the paper. \n\n1. The idea of constraining prototype shifts to a spectral subspace derived via SVD is reasonable. It introduces a well-motivated alternative to existing methods like TPT and TPS by grounding adaptation in semantically meaningful directions rather than unconstrained latent shifts or prompt tuning. \n\n2. STS operates entirely in latent space without requiring any backpropagation through the encoder or modifications to the model. This makes the approach efficient, reducing memory footprint compared to prior methods. \n\n3. The paper demonstrates consistent gains over zero-shot CLIP and prior TTA methods like TPT and TPS across a wide range of distribution-shifted and cross-domain datasets. The trend remains the same with a stronger MaPLe initialization. \n\n4. The paper is well written. The method is introduced step by step and the intuition behind using SVD to define the adaptation space is well-motivated. The use of entropy minimization as a label-free optimization objective is standard but well integrated. \n\nThe following are the weaknesses of the paper. \n\n1. All experiments are conducted only on CLIP ViT-B/16 which is now considered relatively outdated. With the increasing use of larger and stronger backbones like CLIP-L/14, SigLIP-SO400M [1], SigLIP2-SO400M [2] and Perception Encoder [3], it is unclear how STS scales or performs in those settings. \n\n2. While the core method is evaluated well, the paper lacks deeper ablations. For instance, how performance varies optimization steps, sensitivity to initialization, or impact of using alternative subspaces (e.g., PCA instead of SVD). These would improve our understanding of robustness and generality. \n\n---\n[1] SigLIP (SO‑400M): Zhai, Xiaohua, et al. \"Sigmoid Loss for Language Image Pre‑Training.\" arXiv preprint arXiv:2303.15343, 2023.\n\n[2] SigLIP 2 (SO‑400M variant): Tschannen, Michael, et al. \"SigLIP 2: Multilingual Vision‑Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.\" arXiv preprint arXiv:2502.14786, 2025.\n\n[3] Perception Encoder: Bolya, Daniel, et al. \"Perception Encoder: The best visual embeddings are not at the output of the network.\" arXiv preprint arXiv:2504.13181, 2025.", "was_revised": true, "final_justification": "This paper proposes a practical and well-motivated method for test-time adaptation of vision-language models using a spectral subspace derived via SVD of text prototypes. The approach is elegant in its simplicity, avoiding any changes to model weights or architecture, and achieves strong results while being significantly more efficient than existing methods like TPT and TPS.\n\nThe authors' rebuttal addressed my key concerns. They pointed to CLIP-L/14 results in the appendix, and their explanation for choosing SVD over PCA is reasonable. While I would still like to see broader evaluations on more modern VLMs (e.g., SigLIP2, Perception Encoder) and deeper ablations on subspace design, the core contribution is solid, and I would like to increase my score to 5: Accept.", "rebuttal": "Dear Reviewer 2jDh,\n\nThank you for the positive recommendation of our work and the comments. \n\n**W1**\n\nWe appreciate the reviewer’s interest in understanding STS’s behavior on more modern backbones. In the Supplementary Material, we report results for CLIP ViT-L/14 on ImageNet and its OOD variants. As we can see, our method still outperforms zero-shot by a large margin on average across 5 datasets, showcasing that our method generalizes well to larger-scale VLMs.\n\n**W2**\n\nThank you for your interest in the spectral subspace.\n\nHere, we note that PCA approach needs to center the class prototypes (substract the mean of the C rows in a CxD matrix), then compute the covariance and then perform an eigen-decomposition on the covariance matrix and take its top-k eigenvectors. \n\nBecause CLIP’s embeddings live on the unit hypersphere, their global mean is not zero, and removing it (as PCA requires) can actually distort the geometry you care about. \n\n * CLIP optimizes ⟨zᵥ, zₜ⟩ with both zᵥ and zₜ normalized to ‖·‖₂ = 1. Those points lie on the surface of a high‑dimensional sphere, and their **mean** ⎡∑ᵢ zᵢ⎤/N sits strictly *inside* the sphere.\n\n* PCA subtracts that mean before finding principal axes, effectively “recentering” your data cloud at the origin. That shifts your spherical manifold in a way that no longer respects the original contrastive geometry.\n\n**SVD on uncentered data respects the learned hypersphere**\n\n* A raw SVD of the C×D text‑prototype matrix Zₜ finds the directions of *maximum energy*,including any offset from the sphere’s center, which often encode globally relevant semantics (e.g. a “generic image” direction).\n\n* Those directions align more naturally with CLIP’s training objective, which never removed the mean and whose singular vectors capture both mean and variation.\n\n**Empirical parity (and simplicity)**\n\n   * In practice, centering then doing PCA yields virtually the same *subspace span* as SVD on the *centered* data, but differs from SVD on the *raw* data."}, {"review_id": "WdD3HXfV4W", "reviewer": "Reviewer_QTGH", "summary": "The paper introduces Spectrum-Aware Test-Time Steering (STS), a lightweight test-time adaptation (TTA) method for vision-language models (VLMs), specifically addressing the challenge of zero-shot generalization under domain shifts. STS works by steering latent text representations within a low-dimensional spectral subspace obtained from the singular value decomposition (SVD) of initial textual embeddings. This steering optimizes a minimal set of parameters (spectral coefficients) at inference, significantly enhancing model robustness and efficiency.", "questions": "1. Although the method is simple and easy to understand, I believe it requires some theoretical justification or relevant citations to support the claims made in the paper.\n2. Is data augmentation for images necessary? Can the method perform without image augmentation?\n3. Can TTA-based methods or the method proposed in this paper be applied to a wider range of domains?\n4. Line 243: Inconsistent capitalization in the title.\n5. Line 176: Citation error.\n6. Lines 247, 253, and 256: Tables are missing citations.\n7. Line 312 in the provided document appears incomplete.", "limitations": "See questions.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of using an SVD-based spectral subspace to guide adaptation is novel and well-motivated, leveraging intrinsic low-dimensionality in text embeddings.\n2. The approach is parameter-efficient, significantly reducing computational overhead compared to existing TTA strategies.\n\nWeaknesses:\n1. The method's adaptation mechanism is linear, which may limit its effectiveness in handling more complex, nonlinear domain shifts. Such a limitation could restrict generalization performance under extreme or highly varied conditions.\n2. Although the paper presents an ablation study on the computational aspects, additional detailed ablations exploring the sensitivity to the choice of subspace dimensionality or impact of specific hyperparameters (like the number of augmented views) could further clarify robustness.\n3. The experimental results appear suboptimal on certain datasets. What could be the reasons for this?", "was_revised": true, "final_justification": "After checking authors' responses, which well addressed my concerns. Thus, I raised my rating and recommend to accept for this manuscript.", "rebuttal": "Dear Reviewer QTGH,\n\nWe sincerely thank you for recognizing the novelty and motivation behind our proposed approach. We also appreciate the acknowledgment of the method’s parameter efficiency and its ability to significantly reduce computational overhead compared to existing test-time adaptation (TTA) strategies. These aspects were central to our design goals, and we are glad they resonated with the reviewer.\n\nWe address the raised concerns and questions below:\n\n**W1 - Method's Adaptation Mechanism:**\n\nWe appreciate the reviewer’s point and in fact explicitly note this in our Limitation section (Sec 6): STS’s single linear shift in Eq 4. may under-fit highly complex transformations. In STS, however, we find that:\n\na. Across ImageNet‑A, and  ‑V2 a single linear shift in our SVD‑derived subspace recovers a significant percentage of the zero‑shot accuracy gap under moderate to severe corruptions. This suggests that, even when the visual perturbation is nonlinear, the resulting change in optimal text‐prototype geometry often lies in a roughly linear manifold.\n\nb. We also evaluated STS on CIFAR-10-C at the most severe corruption level (severity 5) and observed a +3.52 % accuracy gain over the zero-shot baseline, demonstrating that even under extreme, nonlinear corruptions, steering in our low-dimensional semantic subspace yields substantial robustness improvements.\n\n|  Method | Accuracy (%) |\n|---|---|\n| Zero-shot  |       61.12        |\n|     STS    |     64.64     |\n\nWe believe that this design reflects a deliberate trade-off that maximizes robustness and efficiency:\n\nc. Under the $\\textit{manifold hypothesis}$, pretrained text embeddings concentrate on a low-dimensional semantic manifold. An SVD of the class-prototype matrix uncovers the principal axes along which classes vary. Even complex visual corruptions often project predominantly onto these top directions in text space, so a linear shift suffices to recover the bulk of the alignment gap.\n\nd. By constraining adaptation to the top-k singular vectors, STS avoids over-fitting low-energy, noise-driven directions, an ever-present risk in richer, nonlinear adapters when only a few unlabeled views are available. This bottleneck both stabilizes and focuses learning on the most meaningful shifts.\n\nIn summary, although STS’s core update is linear, it is carefully designed to harness the most significant semantic directions, yielding a practical balance of expressivity, robustness, and efficiency.\n\n**W2 - Ablation Studies:**\n\nWe thank the reviewer for suggesting a deeper ablation on (a) the choice of spectral subspace dimension and (b) the number of augmented views. Below we report two experiments:\n\na. **Subspace dimension sensitivity**\n\nWe compare two strategies for selecting $k_t$ over three random initializations (seeds) and the performance on ImageNet-A dataset:\n\n(i) Gavish-Donoho hard thresholding, and\n(ii) 98% energy retention\n\n| Strategy                     | Seed 1 Acc (%) | Seed 2 Acc (%) | Seed 3 Acc (%) | **Mean Acc (%)** |\n| :--------------------------- | :------------: | :------------: | :------------: | :--------------: |\n| Gavish–Donoho ($k_t=31$)     |      61.07      |      61.51     |        61.13    |     **61.24**     |\n| 98 % Energy ($k_t = 183$) |      60.91      |      61.35     |      60.92   |     **61.06**     |\n\nb. **Number of Augmented Views**\n\nWe vary the number of views $N ∈ ({16,32,64,128})$ reporting accuracy on ImageNet-1K:\n\n| Num Augmentations $N$ | Accuracy (%) | \n| :-------------------: | :----------: | \n|           16         |     67.52    |    \n|           32          |     68.61    |   \n|           64          |     68.93    |   \n|          128          |     69.08    | \n\nIncreasing from 64 → 128 views gives only a +0.15 % boost, while nearly doubling time and memory. We therefore adopt \n$N=64$ (as in prior TPT work) to balance performance and efficiency, still remaining substantially faster than prompt‐tuning baselines.\n\n**W3 - Suboptimal on certain Datasets:**\n\nWe thank the reviewer for raising this point. Here, we pinpoint two possible reasons that have to do with the visual transformations (augmentations):\n\na. ** Familiarity of the semantic categories:** \n\nMany of the ImageNet‐variant benchmarks (and fine‐grained datasets like SUN397 or Caltech101) cover object classes that CLIP likely saw often during pretraining. As a result, CLIP’s predictions remain stable across simple augmentations for these “common” categories. In contrast, classes in datasets such as Flowers102 or Oxford‐Pets represent less frequent concepts, so CLIP is more sensitive to their augmented views.\n\nb. **Nature of the visual transformations:**\n\nOur augmentation pipeline uses only random resized crops and horizontal flips, essentially random “zoom‐ins.” For some domains (e.g. FGVC‐Aircraft, Stanford Cars), zooming can highlight distinctive details like logos or text, which helps CLIP recover the correct label. But on datasets like Flowers102, cropping can remove critical features (e.g. petals or stems), degrading performance.\n\nIn this work, we intentionally kept our augmentations uniform across all tasks rather than tuning them per dataset. Nevertheless, these results suggest that the choice of augmentations, and how they interact with the model’s learned visual priors, is a rich avenue for future study.\n\n**Q1**\n\nWe thank the reviewer for calling attention to the need for stronger theoretical grounding. In the revised manuscript, we will add the following justifications:\n\na. **Low intrinsic dimension of pretrained embeddings:**\n\n Prior work shows that deep‐network features—both vision and text—concentrate on a low‐dimensional manifold. In particular, Aghajanyan et al. (ICML 2020) empirically measure this “intrinsic dimension” and demonstrate that most semantic information lies in a small subspace. This motivates our choice to perform adaptation in a structured, low-dimensional basis rather than the full embedding space.\n\n[1] INTRINSIC DIMENSIONALITY EXPLAINS THE EFFECTIVENESS OF LANGUAGE MODEL FINE-TUNING, Aghajanyan et al. (ICML 2020)\n\nb. **Domain‐adaptation generalization bound:**\n\n We reference the theoretical bound of Ben-David et al. (JMLR 2010), which states that the target risk is controlled by the source risk plus a divergence term between source and target feature distributions. By minimizing distributional discrepancy in our spectral subspace (via entropy minimization), STS directly reduces this divergence and thus the target error.\n\n[2] Nonlinear Dimensionality Reduction by Locally Linear Embedding, Ben-David et al. (JMLR 2010)\n\n**Q2**\n\nSTS can indeed be applied with no augmentations (i.e. a single view, $N=1$), since the optimization in Eq. (4) does not strictly require multiple crops. However, in our experiments, we observe that:\n\nAs $N$ increases accuracy improves but computational cost grows roughly linearly.\n\n**Q3** \n\nThank you for asking about the TTA and STS's broader applicability. Our experiments cover a broad range of domains, from ImageNet variants to Satellite images, and Flowers variants. While our experiments focus on zero-shot image classification, STS is not tied to any specific modality or task, as long as two conditions hold:\n\na. We have static embeddings (e.g. text prompts, class centroids, label vectors) that represent the categories or concepts of interest.\n\nb. You can generate multiple views of each test sample (e.g. augmentations of an image, noise-perturbed audio) without requiring labels.\n\n**Q4-7**\n\nAll  minor errors and typos will be corrected."}, {"review_id": "8IZRdWm16K", "reviewer": "Reviewer_eVss", "summary": "This paper introduces STS, an efficient Test-Time Adaptation technique for VLMs such as CLIP. Inspiring by the processing pipeline of previous works such as TPT, STS treats adaptation as an alignment problem between the visual and text spaces, particularly focusing on modifying the text features. A small set of parameters are learned in order to shift the right values matrix from the Singular Value Decomposition (SVD) of text features. A marginal entropy objective (based on augmentations on a single image) is used to adapt the linear variables, alongside L2 regularization. The method obtains competitive results on popular baselines for TTA.", "questions": "1. For instance, what's the advantage of performing marginal entropy minimization instead of conditional entropy minimization on filtered samples?\n\n2. Also, are the augmentations needed? What is the impediment of using a single image? Is the strategy of augmenting an image related solely to the fact that the marginal entropy is the objective function?\n\n3. If the domain shift is on the visual side, why not applying STS on the visual features so that they align better with the class concepts embedded in the text space?\n\n4. What is the performance on corruptions? Datasets such as CIFAR-10/100-C or Imagenet-C can be included. Oftentimes important gains can be obtained.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n\n1. The paper introduces an unexplored and interesting idea: adaptation can be carried out by modifying the text features in the most significant semantic directions based on SVD decomposition. This ensures to \"steer\" the text representation towarda a better alignment with the input images without incurring in expensive backpropagation, and enforcing a black-box adaptation.\n\n2. The experiments show competitive results on several of the popular datasets used in the field, and integrates MaPLe as well as CLIP, including previous adaptation strategies from the state-of-the-art. \n\n3. The writing is clear and the explanations are easy to follow. \n\nWeaknesses:\n\n1. Although the main contribution (SVD on the text space) is interesting, the cohesion with the rest of the ideas is not entirely clear.  The paper ZERO [5] proposes some intuition, but it's not discussed here. \n\n2. Popular datasets such as Imagenet-Domains and the CLIP's zero-shot suite are used, but some of the previous (and customary) datasets from TTA are disregarded; corruptions such as CIFAR-10/100-C or Imagenet-C, simulations as VisDA-C, etc. It is a trend in recent works to ignore the previous protocols (see TENT for instance).\n\n3. The comparison against the state-of-the-art is based solely on methods based on TPT. The paper intents to reduce the adaptation complexity, but that does not invalidate other gradient-based methods such as WATT, TENT, or parameter-free methods like TDA. The only difference is that they do not learn text prompts, but instead finetune the visual encoder.", "was_revised": true, "final_justification": "The idea of the paper is interesting, and the evaluation seems correct. I believe this paper is a good effort towards new directions for TTA in VLMs. The authors have not responded to my follow-up questions. I keep my score.", "rebuttal": "Dear Reviewer  eVss,\n\nThank you for your positive assessment of our work. We’re delighted that you appreciate STS’s combination of black-box simplicity and semantic grounding for episodic test-time adaptation.\n\n**W1**\n\nWe thank the reviewer for raising this point. We agree that ZERO’s hard‐voting scheme is important and it is **not** in competition with STS but rather **complementary**. In fact, one can seamlessly replace STS’s marginal‐probability averaging over confident views with a **majority‐vote** across those same views:\n\n1. **Adapt with STS as usual.**\n   Compute the low‐dimensional shift $\\Delta z = B\\*\\gamma$ via entropy minimization (Eq. 4) and apply it to the prototypes.\n\n2. **Predict by voting instead of averaging.**\n   For each of the $N$ filtered augmentations, take the arg max over the adapted similarities to cast a “vote” for its top class, then choose the class with the most votes.\n\nWhen we tried this combined STS + ZERO voting pipeline on ImageNet-A, we observed an **additional +0.15 %** accuracy gain over STS’s standard averaging, demonstrating that ZERO’s discrete consensus step can further sharpen STS’s predictions without any extra gradient computation.\n\n**Q1**\nBy minimizing the **marginal** entropy over your filtered views, rather than the **conditional** entropy of each view, we gain several practical and robustness benefits:\n\n* **Robustness to outlier views.**\n  Conditional entropy forces *every* selected crop to be highly confident, so a single ambiguous or noisy view can pull the update in a harmful direction. Marginal entropy only requires the *aggregate* distribution (across all filtered views) to be sharp, allowing a few less‐confident or noisy views to be “overruled” by the majority.\n\n* **Encourages consensus without over-fitting.**\n  By focusing on the consensus distribution, marginal entropy guides STS to find a shift that works well *overall*, not just for each individual augmentation. This is particularly important when you have only a small number of filtered views, the fewer the views, the more you want to lean on their collective signal.\n\nIn short, marginal entropy minimization on your filtered subset yields a **cleaner**, **more robust**, and **more easily optimized** objective for a single-step, black-box adaptation, exactly what STS needs to steer the prototypes confidently without getting thrown off by individual noisy views.\n\n\n**Q2** \n\nAugmentations are related to the marginal entropy minimization and our goal to have a more robust representation for the test sample.\n\n**Q3**\n\nWe tried this approach and we found that the performance improvement of having a steering vector for both the visual and textual embeddings is marginal.\n\n\n**Q4**\n\nPlease see the response on reviewer QTGH for the results on CIFAR10-C."}]

Author rebuttal:

Meta-review: This paper has received mixed recommendations before the rebuttal as follows: 2 x Borderline Reject (`EXj6`, `QTGH`), and 3 x Borderline Accept (`VSFB`, `2jDh`, `eVss`).  

Overall the reviewers appreciated the practicality of this approach, its good performance in terms of scores, parameter and computational efficiency and the clear writing. 

Regarding the criticisms and concerns, the reviewers pointed out the lack of several computationally efficient baselines, of different CLIP encoders, of ablations and potentially limited generalization on distribution shift.

The authors submitted a rebuttal to respond to reviewer questions and criticism.
After the rebuttal and discussions reviewers generally increased their scores: `EXj6` `QTGH` to Borderline Accept and `2jDh` to Accept.
The reviewers unanimously recommend this work for acceptance, but still have some recommendations of baselines and comparisons to be made in the updated paper, in particular from `EXj6`  and `VSFB`.

After analyzing and discussing the strengths and weaknesses identified by the reviewers, the meta-reviewer agrees with the feedback and recommendations expressed by the reviewers and recommend this work for acceptance. 
We encourage the authors to take into consideration the useful advice from the reviewers towards improving this work.
Decision: Accept (poster)