Title: Learning to Steer: Input-dependent Steering for Multimodal LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal large language models

large multimodal models

steering

representation steering

safety alignment

hallucinations

vision-language models
Abstract: Steering has emerged as a practical approach to enable post-hoc guidance of LLMs towards enforcing a specific behavior. 
However, it remains largely underexplored for multimodal LLMs (MLLMs); furthermore, existing steering techniques, such as \textit{mean} steering, rely on a single steering vector, applied independently of the input query. This paradigm faces limitations when the desired behavior is dependent on the example at hand. For example, a safe answer may consist in abstaining from answering when asked for an illegal activity, or may point to external resources or consultation with an expert when asked about medical advice. In this paper, we investigate a fine-grained steering that uses an input-specific linear shift. This shift is computed using contrastive input-specific prompting. However, the input-specific prompts required for this approach are not known at test time. Therefore, we propose to train a small auxiliary module to predict the input-specific steering vector. Our approach, dubbed as L2S (Learn-to-Steer), demonstrates that it reduces hallucinations and enforces safety in MLLMs, outperforming other static baselines. We will open-source our code.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the challenges of steering behavior in multimodal large language models (MLLMs), focusing on input-specific steering to improve model outputs such as safety enforcement and hallucination mitigation. Traditional methods for steering rely on static steering vectors, which apply the same shift regardless of the input. The authors argue that such approaches are ineffective for cases where the desired behavior must vary based on the context, such as when a safe response is required for illegal or harmful queries. To address this limitation, they propose a method called Learn-to-Steer (L2S), which uses an auxiliary neural network to predict input-dependent steering vectors during inference.\n\nSTRENGTHS AND WEAKNESSES:\nThe quality of the work is solid in terms of its experimental validation and theoretical contributions. The paper offers a practical solution to a well-established problem—safety and hallucination mitigation in MLLMs. The proposed L2S method is a clever extension of prior work. The authors provide a clear explanation of the approach, and their experiments show that L2S outperforms traditional methods, such as mean steering, across multiple tasks. The experimental results, especially in safety enforcement and hallucination mitigation, demonstrate a clear improvement over baseline methods. The paper also adequately addresses computational efficiency, highlighting that L2S maintains minimal overhead compared to other methods, making it practical for real-world applications.\nThe paper is well-written and clearly structured. \nWhile steering methods have been explored before in the context of language models, the focus on input-specific steering vectors and the use of contrastive prompting to guide the model's behavior is novel. The authors' approach to fine-grained steering, which is tailored to specific inputs, is a significant departure from previous methods that applied a fixed steering vector to all inputs, and this represents an important conceptual and practical contribution.\n\nIn section 4.2, the authors use \"safety\" and \"security\" interchangeably. This should be changed to consistently say \"safety\".\n\nTo improve generalizability of findings the paper could evaluate different types of MLLMs or datasets.\n\nQUESTIONS:\nI have no additional questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a lightweight auxiliary network based steering technique for Multimodal LLM's. Steering of Multimodal models is still largely unexplored making this an interesting new direction to be explored. The authors motivate their work well and substantiate the part of their proposed technique thoroughly. Their experimental results show a clear advancement over the state of the art. They also identify promising avenues for future research.

STRENGTHS AND WEAKNESSES:
Strengths 
1. Excellent literature review (Quality, Clarity)
2. Good motivation of proposed steering technique (Quality, Significance, Originality)
3. Significant improvement over the state of the art (Quality, Significance)
4. Well written paper (Quality, Clarity)

Weaknesses
1. I am not sure that the proposed technique is a significantly novel technique compared to the techniques it describes. Is the use of an auxiliary network that novel? The authors should clarify. Their technique is certainly sound and interesting.

QUESTIONS:
1. Is the use of an auxiliary network truly novel? Please clarify.
2. What would happen if you had temporal signals such as video and audio in your framework?

LIMITATIONS:
The authors discuss limitations by laying out avenues for future work which is a nice way to do it. I am satisfied with their discussion of limitations. The social impact of their work is positive by definition because steering techniques have all been developed with a view to increasing safety and accuracy.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes a fine-grained steering approach to guide MLLMs towards enforcing specific behavior for safety enforcement 
or hallucination mitigation. The proposed steering is an input-dependent linear shift. It trains a small network to predict a steering vector for an input. The experiments were conducted using safety enforcement and hallucination mitigation datasets and showed that the proposal outperformed existing steering methods that do not adapt to individual input.

STRENGTHS AND WEAKNESSES:
Strength：
1.	The paper proposes a fine-grained steering for guidance of MLLMs. Existing steering methods of MLLMs only use a single steering vector for all inputs. The proposed fine-grained steering vector changes based on every input.

2.	It proposes a small network to predict the steering vector. The network is trained in advance. Steering vector prediction is quite efficient.

3.	It did two experiments – one used a safety enforcement dataset, and the other hallucination mitigation. The results show the proposal outperforms the prior works.

4.	The paper is well-written.


Weakness:
1.	Training the small network g is not clear. 

2.	The detailed information on contrast prompts would be helpful.

3.	Failure cases are due to what?

QUESTIONS:
1.	The detailed information on contrast prompts would be helpful.
2.	Failure cases in the appendix are due to what?

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces L2S, an innovative input-dependent steering method for MLLMs that addresses critical issues of hallucination and safety. Moving beyond static steering, L2S leverages a small auxiliary network to predict input-specific steering vectors, enabling dynamic adjustments to MLLM behavior based on individual queries. This approach significantly enhances model safety and reduces undesirable outputs with minimal computational overhead, offering a highly efficient and effective solution for reliable MLLM deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1、The proposed L2S method directly tackles two paramount challenges in MLLMs: hallucination and safety.\n2、The design, particularly the use of a lightweight network to predict steering vectors, makes L2S computationally very efficient.\n\nWeaknesses:\n1、The evaluation is limited to LLaVA1.5, without sufficient coverage of other representative MLLM architectures like Qwen-VL.\n2、L2S operates by applying a linear shift in the latent space. However, the paper does not delve deeply into how this shift specifically alters the MLLM's internal states or its understanding of input concepts.\n3、The paper focuses on mitigating hallucination and enforcing safety. However, guiding the model towards these specific behaviors might affect or degrade its performance on other important dimensions such as informativeness. The authors should conduct experiments to evaluate L2S's impact on other desired qualities of MLLM responses.\n\nQUESTIONS:\n1、The core method L2S relies on ground-truth steering vectors generated by P2S. However, manually crafting high-quality contrastive prompts for each new task or behavior can be time-consuming and may require domain expertise. Can this approach realistically scale to novel or previously undefined behaviors?\n2、The authors should include an ablation study on L to better isolate and evaluate its effect.\n3、The authors adopt a lightweight 2-layer MLP as the auxiliary network, which is a reasonable design choice. However, is its representational capacity sufficient? Did the authors consider more complex architectures to potentially improve performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}