NeurIPS 2025

Test-Time Spectrum-Aware Latent Steering for
Zero-Shot Generalization in Vision-Language Models
Konstantinos M. Dafnis∗
Department of Computer Science
Rutgers University
Dimitris N. Metaxas
Department of Computer Science
Rutgers University
Abstract
Vision–Language Models (VLMs) excel at zero-shot inference but often degrade
under test-time domain shifts. For this reason, episodic test-time adaptation strate-
gies have recently emerged as powerful techniques for adapting VLMs to a single
unlabeled image. However, existing adaptation strategies, such as test-time prompt
tuning, typically require backpropagating through large encoder weights or altering
core model components. In this work, we introduce Spectrum-Aware Test-Time
Steering (STS), a lightweight adaptation framework that extracts a spectral sub-
space from the textual embeddings to define principal semantic directions and learns
to steer latent representations in a spectrum-aware manner by adapting a small num-
ber of per-sample shift parameters to minimize entropy across augmented views.
STS operates entirely at inference in the latent space, without backpropagation
through or modification of the frozen encoders. Building on standard evaluation
protocols, our comprehensive experiments demonstrate that STS largely surpasses
or compares favorably against state-of-the-art test-time adaptation methods, while
introducing only a handful of additional parameters and achieving inference speeds
up to 8× faster with a 12× smaller memory footprint than conventional test-time
prompt tuning. The code is available at https://github.com/kdafnis/STS.
1
Introduction
Vision-Language Models (VLMs), such as CLIP [30], have marked a paradigm shift in artificial
intelligence, demonstrating remarkable zero-shot generalization capabilities across a multitude of
downstream visual tasks. By learning rich joint representations from vast quantities of image-text
data, these models can often perform tasks like image classification with...