Textual Steering Vectors Can Improve Visual Understanding in Multimodal
  Large Language Models
Authors: name: Woody Haosheng Gan; name: Deqing Fu; name: Julian Asilis; name: Ollie Liu; name: Dani Yogatama; name: Vatsal Sharan; name: Robin Jia; name: Willie Neiswanger
Published: 2025-05-20 08:23:08

Abstract: Steering methods have emerged as effective and targeted tools for guiding
large language models' (LLMs) behavior without modifying their parameters.
Multimodal large language models (MLLMs), however, do not currently enjoy the
same suite of techniques, due in part to their recency and architectural
diversity. Inspired by this gap, we investigate whether MLLMs can be steered
using vectors derived from their text-only LLM backbone, via sparse
autoencoders (SAEs), mean shift, and linear probing. We find that text-derived
steering consistently enhances multimodal accuracy across diverse MLLM
architectures and visual tasks. In particular, mean shift boosts spatial
relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to
+3.3%, outperforming prompting and exhibiting strong generalization to
out-of-distribution datasets. These results highlight textual steering vectors
as a powerful, efficient mechanism for enhancing grounding in MLLMs with
minimal additional data collection and computational overhead.