Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
Authors: Jianghao Yin
Qin Chen
Kedi Chen
Jie Zhou
Xingjiao Wu
Liang He
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision-Language Models
Hallucination
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems.
Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1) truthfulness and visual perception capabilities predominantly engage different subsets of attention heads within the model architecture; and 2) truthfulness steering vectors vary significantly across different semantic contexts. Based on these observations, we propose Dynamic Multimodal Activation Steering, a training-free approach for hallucination mitigation. Our method constructs a semantic-based truthfulness steering vector database and computes visual perception steering vectors, enabling context-aware interventions during inference by dynamically selecting the most relevant steering vectors based on input semantic similarity and applying them to the most influential attention heads. We conduct comprehensive experiments across multiple models and datasets, demonstrating that our approach significantly enhances model performance, outperforming existing state-of-the-art methods.

Published as a conference paper at ICLR 2026
DYNAMIC MULTIMODAL ACTIVATION STEERING FOR HALLUCINATION MITIGATION IN LARGE VISION- LANGUAGE MODELS
Jianghao Yin, Qin Chen', Kedi Chen, Jie Zhou', Xingjiao Wu, Liang He East China Normal University jhying stu www.:esuuuuuuuu.... ecnu. edu. en, schhen@sseccnu www..nee.....oommommoomommm en, jzhouêcs. eccu. edu. on
ABSTRACT
Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems. Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1) truthfulness and visual perception capabilities predominantly engage different subsets of at- tention heads within the model architecture; and 2) truthfulness steering vectors vary significantly across different semantic contexts. Based on these observations, we propose Dynamic Multimodal Activation Steering, a training-frea approach for hallucination mitigation. Our method constructs semantic-based truthfulness steering vector database and computes visual perception steering vectors, enabling context-aware interventions during inference by dynamically selecting the most relevant steering vectors based on input semantic similarity and applying them the most influential attention heads. We conduct comprehensive experiments across multiple models and datasets, demonstrating that our approach significantly enhances model performance, outperforming existing state-of-the-art methods.
INTRODUCTION
Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on visual question answering (VQA), image captioning. and related tasks (Liu et al.) 2023 2024c Bai et al.) 2023 Chen et al., 2024 Dai et al., 2023). However, these models suffer from significant halluci- nation phenomena (Huang et al. 2025; Chen et al.] (025b)) manifested
fabricating non-existent
objects or incorrectly describing image content (Liu et al., 2024b] Bai et al. 2024). Such halluci-
nations limit the applicability of LVLMs in downstream applications including autonomous driving
(Cui et al. 2024), robotics (Li et al.; 2024b), and other safety-critics domains.
Due to the complex architecture of LVLMs, the causes of multimodal hallucinations diverse. To address these multimodal hallucination issues, numerous approaches have been proposed (Leng al. 2024 Huang al.) 2024 An et al. 2025 Yin al., 20244] Liu al., 2024a), which can be broadly categorized into two classes: training- based and decoding- based methods. Training-based methods primarily focus on constructing less biased datasets to fine-tune LVLMs, such as LRV (Liu et al., 2024a). or employing reinforcement learning to train LVLMs, as demonstrated by RLHF-V (Yu et 2024a). The limitations of these approaches lie in their requirements for carefully curated data and substantial computational resources, as well as the need to retrain models separately for different architectures. Decoding-bee
methods, on the other hand, modify the decoding strategies
of LVLMs, such as VCD (Leng et al.] (2024) and ICD (Wang et al. (024c).. While these methods avoid the need for training, they often compromise the quality of the generated content (Yin et al. 2025).
More recently, researchers have begun investigating activation engincering (Zou et al.) 2023 Li
et al. 2023b Wang al. 2024b) as alternative approach to reduce hallucinations through tar- geted intervention in model representations. ICT (Chen et al., (025a) is an image-object cross-level trusted intervention method that mitigates model hallucinations by applying noise to both images and objects, thereby enhancing the model's attention to visual information. However, this approach
Coorreponndigg authors.

---

Published as a conference paper at ICLR 2026
primarily focuses on visual-level interventions, neglecting the multimodal characteristics of LVLMs. VTI (Liu et al. 2025b) intervenes in the hidden states both the visual encoder and large language model during inference by pre-computing steering vectors for visual and textual modalities. Never- theless, this method employs fixed steering vectors regardless of input variation, ignoring potential semantic differences across diverse contexts. The uniformly applied steering vectors fail to account for the nuanced semantic variations that exist across different inputs.
To address these challenges, we propose dynamic multimodal activation steering (DMAS), training-free approach for hallucination mitigation in LVLMs. Our method focuses on two types of attention heads in LVLMs: truthfulness-relate and visual perception-relater For truthfulness heads, we explicitly model how truthfulness steering vectors vary across semantic contexts. We cluster data semantically and create sample pairs with and without hallucinations within each clus- ter. By contrasting attention activations between factual and hallucination-pae -poone samples, we extract truthfulness steering vectors. These vectors are stored alongside their cluster embeddings in a key- value database. For visual perception, we calculate activation differences between noise-free and noisy image inputs to derive perception steering vectors that enhance visual attention. During in- ference, dynamically retrieve the most semantically relevant truthfulness steering vector for the input query and apply both truthfulness and visual perception vectors to the top-K attention heads with the largest activation differences. This dual intervention effectively reduces hallucinations. The main contributions of our paper are:
We investigate activation differences in LVLMs, revealing that truthfulness and visual percep- tion capabilities predominantly engage different subsets of attention heads, and demonstrate that truthfulness vectors vary significantly across different semantic contexts through visualization, indicating the necessity for dynamic rather than static intervention approaches.
We propose dynamic multimodal activation steering, training- faree method for hallucination mit- igation that constructs a semantic-basee truthful steering vector database and visual perception steering vector, enabling context-aware interventions during inference by dynamically selecting appropriate steering vectors based on input semantic similarity.
We conduct comprehensive experiments on multiple models across discriminative tasks and open- ended generation datasets. The experimental results demonstrate that our method achieves signif- icant improvements: increasing total scores by 94.66 on MME and reducing 20.2% hallucinations on CHAIR, outperforming existing state-of-the- methods. These results highlight the effective- ness of our approach in hallucination mitigation.
2
RELATED WORK
2.1 LARGE VISION-LANGUAGE MODELS
Large Vision-Language Models (LVLMs) have recently undergone rapid development, achieving excellent performance in image captioning and VQA tasks (Yin et al. 20233 Jin al. 2024). They typically consist of a vision encoder, a connection layer, and an LLM. As for the vision encoder, the VIT from CLIP (Radford et 202I) commonly used. For the connection layer. some mod- els use simple MLP layers for alignment, such as LLaVA (Liu et al., 2023 2024c). Shikra (Chen al. ),23), PandaGPT (Su et al. 2023), etc.; some models use Q-former for alignment, like BL.IP2 (Li et al., (023a), InstructBI.IP DDai et al.] 2023), etc.; while others design special architectures. However, these LVLMs suffer from serious hallucination problems, and effectively eliminating hal- lucinations remains a popular research topic.
2.2 HALLUCINATION MITIGATION FOR LVLMS
Recently, numerous approaches have been proposed to mitigate multimodal hallucinations (Liu cet al. 2024b Bai et al. 2024), addressing this issue across three key stages: training, inference, and post-processing. At the training stage, some research concentrates constructing better data to train models. For example, LRV (Liu et al. 2024a) constructs a high-quality instruction fine-tuning dataset containing balanced positive and negative samples, while other studies introduce reinforce- ment learning to the multimodal domain to reduce hallucination, such as RLHF-V V (Yu et al. (022a) and RLAIF-V et 2024b). These methods typically require carefully constructed training data
2

---

Published as a conference paper at ICLR 2026
(a) The difference of truthfulness at- tention head activation.
(b) The difference of visual percep- (c) Steering vectors visualiza-
tion attention head activation.
tion.
Figure Activation differences in LLaVAv1.5.
and consume substantial computational resources during training. Research on mitigating hallucina- tions the inference stage often requires no training. VCD (Leng al., 2024) uses the distribution from noise-added images and the original output distribution to jointly determine the final distribu- tion mitigate hallucination. ICD (Wang et al., (2024c) reduces hallucinations by contrasting output distributions between standard and deliberately disturbed instructions. These contrastive decoding methods often compromise the quality of the generated content (Yin al., 2025). Post-processing approaches correct the generated content from LVLMs to achieve hallucination reduction effects. For instance, LURE (Zhou al., 2024) constructs a dataset to train hallucination revisor. How- ever, these methods require the construction of complex pipeline and increase the time required to obtain final outputs. To overcome these limitations, we propose dynamic multimodal activation steering, training ffree approach to mitigate hallucination in LVLMs by dynamically intervening in attention heads during inference time.
PRELIMINARY STUDY
To understand the internal mechanisms underlying multimodal hallucinations, we conduct a system- atic analysis of attention patterns in LLaVAv1.5 (Liu et al. 2024c) across 3,000 samples from the SEED (Li et al. 2024a) and AMBER (Wang et al.] 2023) datasets. Our investigation focuses on identifying which attention heads are most sensitive to truthfulness versus visual perception.
We design two complementary experiments to isolate attention mechanisms responsible for dif- ferent aspects of multimodal processing. In the first experiment, we examine truthfulness related attention head by contrasting model activations when processing identical visual inputs paired with text prompts either with ground truth or hallucinated answers. This approach enables us iden- tify attention heads most relevant to truthfulness, we measure how each head's activation changes between truthful and hallucinated content by computing the difference: truthful activation minus hallucinated activation. In the second experiment, we investigate visual perception related attention head by comparing activations between clean images and their noise-corrupted counterparts, caleu- lating activation differences by subtracting the activation values of non-noisy inputs from those with noise. As shown in Figures Ia and [b the activation patterns differ significantly between these two experiments. For truthfulness (Figure Ia). the most active attention heads appear predominantly in layer 30. In contrast, for visual perception (Figure lb), the highest activation differences concen- trate in layer 31. These distinct activation patterns provide foundation for our targeted intervention approach that addresses both aspects simultaneously.
Furthermore, we divide the SEED and AMBER datasets into four semantic clusters and compute the activation differences for each cluster. Using t-SNE to visualize these differences in two- dimensional space (Figure we observe clear separation between clusters, with each occupying a distinct region in the projection space. This separation indicates that truthfulness direction vectors vary significantly across different semantic contexts. The heterogeneity in these patterns suggests that a static intervention approach would be insufficient, as cannot account for the semantic- dependent nature of hallucinations. This observation directly motivates our dynamic multimodal activation steering method, which can adaptively select appropriate steering vectors based on the semantic content of the input query.
3

---

Published as a conference paper at ICLR 2026
4 METHOD
In this section, we introduce dynamic multimodal activation steering. shown in Figure 2 the method has three steps: the first step is to establish dynamic truthfulness steering vector database, the second step is to calculate the steering vector the model's visual perception attention heads, and the third step is to apply dynamic interventions to different attention heads during inference.
4.1 TRUTHFULNESS STHERING VECTOR DATABASE
We select the AMBER (Wang et al. 2023) and SEED (Li al., 2024a) datasets as our data sources and divide the datasets into 4 clusters based semantics. The questions in these two datasets are in the form of multiple-choice and discriminative questions, making it easy for us to create hallucinated answers for each sample (for discriminative questions, we change the answer to the opposite; for multiple-choice questions, we randomly select an incorrect option). Each cluster C2 comprises the question prompt T, visual input V, ground truth response Ypos. and incorrect response Yneg for every sample.
We input (V, T + poss) and (V, T + Ynng) separately into LVLMs and preserve the attention head activation values of the last token at each layer, denoted as Apss and Aneg- We define the truthful- ness steering vector as the activation difference between non-hallucinated outputs and hallucinated outputs within each cluster according to Equation
D; =
(Apoe.)
Aneg.)
C.l
JEC:
IC:] represents the number of samples in cluster C,, and j indexes the samples within the cluster. Subsequently, apply principal component analysis (PCA) to D; reduce insignificant noise, thereby extracting the principal components that influence truthfulness. The magnitude of D, effec- tively quantifies the significance of each attention head in governing this specific model behavior.
Next, we construct truthful steering vector database where the average embedding representation of questions from each cluster serves as the key, with the corresponding steering vector D; as the value. During inference, our approach dynamically matches the semantic content of the input question to retrieve the most semantically similar steering vector, enabling context-appropriate interventions. We obtain key embeddings in the database and input text embeddings via sentence transformer]
Step Truthfulness Streering Vector Database
Database Sturiig Centee Embeedinn
Step 3 Dynamic Intervention for Inference
Hag
Pegg
Heg
L
Oueer
V
Prs
Hag
Pas
Index
M
Chaie
LVLM
Heg
Hag
Step 2 Visual Perception Steering Vector
Antvrtes of
LVL
there in the image?
xB
hetcchette T
BE BB
Guner catter endvlding
There five sheep image.
Iepat tot endeidingg
Figure 2: Overview of the DMAS framework.
-mppetmppttt--- base--y

---

Published as a conference paper at ICLR 2026
4.2 VISUAL PERCEPTION STEERING VECTOR
Given a visual input V and distorted visual input V (obtained by adding noise to the image following the forward diffusion process (Ho et al., 2020)) first input V into object detector YOLOv11 (Khanam & Hussain 2024) to obtain objects O present in the image, and insert them into a simple template "The image depicts {objects}' denoted as Yo. Then we randomly select an equal number of objects O' from a predefined object library within the same object category that are in O, inserting them into template, denoted Yo.. The prompt is fixed 'Please describe this image.' Next, we obtain the final inputs (V, YY) and (V' T + Yor), ), and input these two samples separately into LVLMs, preserving the attention head activation values of the last token at each layer, denoted as A. and A,r respectively. We define visual perception steering vector as the activation difference between visual input and distorted visual input according to Equation 2
De = A. Av'
(2)
Similarly, we apply PCA to D, to reduce noise, thereby extracting the principal components most relevant to visual perception.
4.3 DYNAMIC INTERVENTION FOR INFERENCE
During the inference phase, for a given text input T and visual input V, we dynamically retrieve the most appropriate steering vector by computing semantic similarity between the input and each key in database as shown in Equation 3
Dj D;, where = arg max sim [ET), Key;)
where E(T) is the embedding representation of the input text, K Key, represents the key embedding for cluster i, and sim(::) denotes the cosine similarity function. This process identifies the most relevant truthfulness steering vector for the current input.
To achieve more precise control over model behavior, rather than intervening on all attention heads, We define binary mask matrices selectively target the most influential heads in both Dy and De.WWe
My and M, as Equation
((. (S.)
f1. if (1, E TopK (f.w), K)
(4)
otherwise
where (1, h)) denotes the h-th attention head in the 1-th layer, D represents the sum activation differences for each attention head in D and TopK(D (f.w). K) returns the indices of the K largest attention heads in either D, or D., representing the most influential attention heads for truthfulness and visual perception respectively.
Building upon the standard attention mechanism, we modify the computation for layers where in- tervention is applied. Our intervention-enhanned computation is formulated as Equation 5
(I+1) x() Concat(o H) (t.n (x(1))
t M(()) ((1.))
(5)
+ 3 MyN D((AA) W((
where (!) represents the hidden states at the 1-th layer, is the number of attention heads per layer, and 33 are hyperparameters controlling the intervention strength for truthfulness and visual perception respectively. The binary masks ensure that interventions are only applied to the most influential attention heads, allowing for precise and targeted steering of the model's behavior.
5 EXPERIMENTAL SETUP
DATASETS AND EVALUATION METRICS
To comprehensively evaluate proposed approach, test method discriminative tasks, including MME (Fu et al. 2023) and POPE (Li et al. 2023c), as well as on open-ended generation tasks using CHAIR (Rohrbach et al. 2018
5

---

Published as a conference paper at ICLR 2026
Model
Method
Existencef
Count
Position
Colort
Total Scores
Regular
175.67
124.67
114.00
151.00
565.33
VCD
184.66
138.33
128.67
153.00
604.66
OPERA
180.67
133.33
123.33
155.00
592.33
VAF
195.00
158.33
128.33
155.00
636.67
1.
DECO
185.00
153.33
118.33
155.00
611.66
DAMO
195.00
150.00
143.33
165.00
653.33
AGLA
195.00
153.89
129.44
161.67
640.00
ICT
190.00
160.43
128.67
170.00
649.10
Ours
195.00
158.33
133.33
173.33
659.99
Regular
155.00
127.67
131.67
173.00
587.33
VCD
156.00
131.00
128.00
181.67
596.67
VW
VAF 165.00
155.00
133.33
175.00
628.33
ICT 180.00
145.00
108.33
173.33
606.66
Ours
170.00
145.00
133.33
185.00
633.33
Table 1: Results on MME. The best results are shown in bold.
MME (Fu et al. 2023) is comprehensive evaluation benchmark for LVLMs, comprising 14 subtasks. For questions in this dataset, models are required to respond with either 'yes' 'no'.
Following Yin et al. (2024) and Leng et al. (2024), we select 'existence'. 'count', 'position', and sets. Consistent with (2023). we adopt the sum of accuracy
'attribute' as the hallucination and accuracy+ as the evaluation metrics.
(Li et al. 2023c is a benchmark designed specifically to evaluate object hallucination. varying difficulty levels: random (randomly sampling nonexistent objects), popular (selecting frequently appearing objects), and adversarial (se- lecting objects that frequently co-occur with objects present in the image). We report Accuracy, Precision, Recall, F1 Score the evaluation metrics.
POPE
The benchmark features three sampling strategies of
CHAIR (Rohrbach et al. 2018) is an open-ended generation task. This benchmark comprises 500 images sourced from MSCOCO (Lin al. 2014). where LVLMs are required to generate captions for the images, followed by evaluation of hallucinations present in these captions at sentence level CHAIRs and image level CHAIR,
5.2 BASELINES AND IMPLEMENTATION DETAILS
We validate the effectiveness on mainstream LVLMs: LLa 7 (Liu al.
2024c) and
Qwen VL. 7B (Bai et al., 2023). and compare DMAS with tate-of-the-ane the-art methods: ICT (Chen et al.,
2025a), AGLA (An et al. 2025), VAF (Yin et al., 2025). VTI (Liu et al. (0256), DECO (Wang al,
2024a). DAMO (Wang et al. 2025), VCD (Leng et al. 2024), and OPERA (Huang et al. 2024).
Our method has three key parameters: a, 3, and a and 33 respectively regulate the intensity of interventions for truthfulness and visual perception, while K refers to the intervention on the top K most active attention heads. We set the range of and (0.5, 1, 2, 3, 4, 5, 7, 8, 9, 10), and the range of K to {32, 64, 128, 256, 512, 1024). and employ grid search to determine the parameters. In our experiments, we set the temperature to and top.p to All experiments are conducted on NVIDIA RTX 4090(48GB) GPUs.
EXPERIMENT
6.1 RESULTS MME
The results the MME (Fu et al. 2023 dataset are presented in Table Our method demonstrates significant improvements of 94.66 and 46 points compared to the baseline models LLaVAv1.5 and QwenVI., respectively. On the 1Lv model, approach outperforms the existing state-

---

Published as a conference paper at ICLR 2026
of af- method ICT (Chen et al. current state-of-the- method VAF (Ynn et al. 2023 by points. Across all subtasks, we observe notable improvements regular baselines, which be attributed to dynamic intervention mechanism that retrieves the most semantically similar steering vector for each query.
2025a) by 10.89 points, while QwenVL, it surpasses the
6.2
RESULTS ON POPE
Dataset
Setting
Method
Aceuracy
Precision
Recall
Score f
Regular
81.38
88.04
72.78
79.65
VCD
84.33
85.93
83.28
84.52
OPERA
84.21
88.23
79.79
83.72
VAF
86.90
89,43
83.77
86.47
LLv
AGLA
85.82
93.78
76.83
84.44
VTI
86.48
90.11
82.09
85.90
ICT
87.35
87.12
MSCOCO
Ours
86.81
87.23
86.57
86.79
Regular
83.71
93.30
72.69
81.70
VCD
86.67
90.66
81.94
83.04
OPERA
84.26
94.40
73.52
82.65
VL
AGLA
83.9
96.20
70.62
81.44
VTI
85.18
91.31
78.18
84.08
ICT
87.53
86.98
Ours
87.63
87.92
87.3
87.65
Regular
78.33
79.33
79.13
79.13
VCD
81.16
77.31
89.08
82.67
OPERA
80.80
83.24
LL
VAF
83.67
81,50
88.00
84.50
AGLA
84.41
84.63
84.67
84.55
ICT
85.27
85.50
GQA
Ours
85.27
83.86
87.51
85.63
Regular
77.47
81,54
71.37
76.06
VCD
82.48
81.73
83.93
82.77
OPERA
82.74
82.68
QW
AGLA
81.14
86.87
73.53
79.63
ICT
83.28
83.26
Ours
84.40
85.19
83.53
84.32
Table Results on POPE, Best results are in bold, and second-best values are underlined.
The experimental results of POPE (Li et al. 2023c are shown in Table 2 We conduct experiments on MSCOCO (Lin al. 2014) and GQA (Hudson Manning 2019) under random, popular, and adversarial settings. Table presents the average results across these three settings, with detailed experimental results provided the Appendix. Our method improves LLa performance on MSCOCO 5.43% in accuracy and 7.14% in FI score, while for Qwen VI. it achieved improve- ments of 3.92% in accuracy and 5.95% in score. On GQA, our method enhances LLa 1.v by 6.94% in accuracy and 6.5% Fl score, and improves Qwen by 6.93% accuracy and 8.26% in Fl score. Compared to existing methods, our approach achieves best results in most cases, demon- strating its significant effectiveness in mitigating object hallucination. Notably, while the ICT (Chen) et al. 2025a) method applies noise to objects in images to increase the LVLMs' attention to these objects, our method achieves superior performance most cases without such specialized design elements.
6.3 RESULTS ON CHAIR
We evaluate our method on open-ended generation tasks, with experimental results on CHAIR (Rohrbach al. 2018) presented in Table Our method reduces hallucinations by 20.2 at the

---

Published as a conference paper at ICLR 2026
90
JUAA11 Oveer//
85
F1 Score (80
Overk Bose
Boerling
Parameter Value
K Valuee
Cluster Number
(a) Impact of a and
(b) Impact of K.
(c) Impact of Clusters.
Figure 3: Impact of key hyperparameters.
sentence level (CHAIRg) and by 3.8 at the image level (CHAIR,).. Compared existing methods, our approach reduces sentence-level hallucinations by points over the state-of-the-ar method VTI (Liu al.) 2025b), and matching VTT's performance image-level hallucinations. In summary, method achieves significant improvements in hallucination mitigation on both discriminative tasks and open-ended generation tasks.
Method
CHAIRsL s
CHAIR
CHAIR
POPE
Regular
51.0
15.2
Method
VCD
51.0
14.9
Cs
C₁4
Acct
FI1
OPERA
47.0
14.6
Ours
30.8
11.4
81.70
82.47
DECO
37.8
11.1
w/o visual vector
34.2
11.7
81.67
82.42
AGLA
43.0
14.1
w/o truthfulness vector
42.4
13.2
81.40
82.01
VTI
35.8
11.1
w/o both
51.0
15.2
75.08
76.06
Ours
30.8
11.4
Table 4: Ablation studies on CHAIR and POPE. Cs repre- sents CHAIRs, C, represents CHAIR,-
Table Results on CHAIR.
6.4
FURTHER ANALYSIS
6.4.1 ABLATION STUDIES
To demonstrate the effectiveness of using both truthfulness steering vector and visual perception steering vector, we compare the results when utilizing only one intervention at a time. We con- duct experiments on LLaVAv1.5 using the CHAIR and POPE. As shown in Table 'w/o visual vector' indicates intervention with only the truthfulness steering vector, while 'w/o truthfulness vec- tor' indicates intervention with only the visual perception steering vector. We observe that even when using only one intervention method, there is a notable improvement compared to the Reg- ular baseline (w/o both). Furthermore, each intervention method exhibits hallucination mitigation effects both discriminative and generation tasks. The optimal results are achieved when the two interventions are combined.
6.4.2 EFFECT OF DYNAMIC INTERVENTION
To validate our designed strategy of dynamically invoking the truthfulness steering vector based on semantics, we compare method with combining all truthfulness steering vectors into single fixed steering vector for intervention. We conduct experiments on QwenVL and LLa using the MME, with results shown Figure 4 We observe that dynamically invoking steering vectors based on semantics achieves optimal performance across all subtasks. When using fixed steering vector, the improvement smaller than with our method, and even underperforms the original model on the Position subtask of QLe which demonstrates the necessity of our designed dy- namic invocation strategy.
6.4.3 IMPACT OF HYPERPARAMETERS
8

---

Published as a conference paper at ICLR 2026
In this section, we investigate the impact of key parameters a, B, and K on the experimental re- sults. Here, and 3 control the intervention strength, while K denotes the numher of atten- tion heads receiving intervention. Experiments conducted on QwenVL using the POPE GQA random subset are shown in Figure 3 Figure 3a illustrates the relationship between F1 score and parameters and B. When a and / are negative, we observe decrease in Fl score, which effectively represents intervention in the opposite direction, pushing activations toward hallucination. As a and B increase, Fl score exhibits an upward trend; however, when and 3 become excessively large, F1 score shows precipitous decline, indicating that the model's fundamental capabilities become impaired. Figure shows how Fl varies with the number of intervened attention heads, revealing similar patterns for both truthfulness and visual perception attention heads. Few intervened heads produce minimal impact with no significant F1 improvement. As intervention extends to more heads, F1 score in- creases, but excessive intervention causes dramatic decline in FI, indicating degradation of model performance.
QivenVL
LLaVA1.5
Eardenns
135
Couet 145.00
Owat 168 85
Coler
Oolor
655.35
858.00
Regular
Fixed
Dure
Figure 4: Effect of dynamic intervention.
6.4.4 IMPACT OF CLUSTERS
In this section, we investigate the impact of cluster quantity on the performance of our proposed method. We vary the number of clusters across {2. 3, 4, 5} and conduct experiments on both OWenVL and LLaVAv1.5 using the MME benchmark. The experimental results are presented in Figure 3c| We observe that both LLa and Qwen VL achieve optimal performance when the number of clusters is set to 4. When the cluster count insufficient, the semantic granularity becomes coarse for effective representation.
6.4.5 ANALYSIS OF GENERALITY
To verify the generalizability of our method, we tested our approach on ScienceQA (Lu et. 2022) which is subject-based VQA dataset and ViQuAE (Lerner et al. 2022) which is know knowledge-basee VQA dataset. The accuracy are shown in Table 5 Our method also achieved significant improve- ments on these datasets. These datasets are completely different from the dataset types we used to construct the steering vector, which demonstrates the generalizability of our method.
ScienceQA
ViQuAE
Method
LLaVAv1.5
QwenVL LLv QwenVL
Regular
52.75
46.41
43.38
50.09
VTI 51.46
42.29
ICT
52.95
42.47
Ours
62.27
48.04
56.00
54.08
Table 5: Generality on ScienceQA and ViQuAE.
CONCLUSION
This paper proposes dynamic multimodal activation steering, training- free approach to mitigate hallucination LVLMs by dynamically intervening in attention head activations. The experiments on multiple benchmarks demonstrate the effectiveness of our method, with LLaVAv1. 1.5 achieving a remarkable 94.66-pon 1--pointt improvement on MME and reducing 20.2% hallucinations on CHAIR, outperforming existing SOTA methods. We compare the experimental performance of our proposed semantic-de -yyamic strategy for steering vector selection against fixed steering vector approaches, demonstrating the effectiveness and necessity of semantic-dynamic steering vector selection.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research funded by the National Nature Science Foundation of China (No. 62477010, No.62577022 and No.62307028), the Natural Science Foundation of Shanghai (No. 13ZR1441B 1411800), Shanghai Science and Technology Innovation Action Plan (No. 24YF2710100 and
Qiji Zhifeng Co., Ltd. 2025-GZL-RGZN-0100 and the open-
ing funding of the State Key Laboratory of DisasterReduction in Civil Engineering (Grant No. SLDRCE24-033.
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xi- aoqin Zhang, and Shijian Lu. Mitigating object hallucinations in large lsionnlanguage models with assembly of global and local attention. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29915-29922 2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023.
hbspg//bbs/333/ 2308 2ttpp///308111666666 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: survey. arXiv preprint 24Xiv:2404.18930 2024.
Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models. In Proceedings the Computer Vision and Pattern Recognition Confer- ence, pp. 4209-4221, 2025a.
Kedi Chen, Qin Chen, Jie Zhou, Xinqi Tao, Bowen Ding, Jingwen Xie, Mingchen Xie, Peilong Li, and Zheng Feng. Enhancing uncertainty modeling with semantic graph for hallucination detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp- 23586-23594, 2025b.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint a2X02.241..02271, 2024.
Can Cui, Yunsheng Ma. Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuei-Da Da Liao, A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958-979, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards pereral vision-language models with instruction tuning, 2023. URL
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li. Xing Sun, and Rongrong Ji. MME: A comprehensive eval- uation benchmark for multimodal large language models. CoRR, abs/2306. 13394, 2023, doi: 10. 13394. UR.https: h/ttpps://d11 www.orrg...... 48550.48555110 a 2306.. 13394 2339.
10

---

Published as a conference paper at ICLR 2026
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- lisoonlanguaag models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattrn Recognition, pp. 14375- 14385, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 18:6840-6551 2020.
Lei Huang, Weijiang Yu, Weitao Ma. Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 43(2): 2025.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination multi-moda large language models via over-trust penalty and -aaecciionalllcatioon In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-123 13427, 2024.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405. 10739, 2024.
Rahima Khanam and Muhammad Hussain. Yolov 11: An overview of the key architectural enhance- ments. arXiv preprint arXiv:2410. 17725,
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and bet- ter understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408. 12637, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing- Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13872-13882, 2024.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-bass visual question answering about named entities. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp. 3108-3120, 2022.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12299-13308 2024a.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tine intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 06:41451-100 455-00 2023b.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long. Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object- centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18061-18070, 2024b.

---

Published as a conference paper at ICLR 2026
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-langure models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hal- lucination large modll models via robust instruction tuning. The Twelfth International Conference Learning Representations, 2024a.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint vrXiv:2402.00253 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 26296-26306, 2024c.
Jiazhen Liu, Yuhan Fu, Ruobing Xie. Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A chatgpt-prompted visual hallucination evaluation dataset. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 19857-19866, 2025a.
Sheng Liu, Haotian Yc, and James Zou. Reducing hallucinations in large vision-language models via latent space stecring. In The Thirteenth International Conference on Learning Representations, 2025b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-878 8763. PMLR, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035-4045, 2018
Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. Illusiovqa:: A challenging optical illusion dataset for vision language 2024.
models. arXiv preprint
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-folllw them all. In Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!, pp. 11-23, 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9568-9578, 2024.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.1 11777 2024a.
12

---

Published as a conference paper at ICLR 2026
Junyang Wang. Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An lim-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.0739, 2023.
Kaishen Wang, Hengrui Gu, Meijun Gao, and Kaixiong Zhou. Damo: Decoding by accumulating
activations momentum for mitigating hallucinations in vision-language models. In The Thirteenth International Conference on Learning Representations. 2025.
Weixuan Wang, Jingyuan Yang, and Wei Peng. conantics-adaptive activation intervention for Ilms via dynamic steering vectors. arXiv preprint arXiv:2410. 12299, 2024b.
Xintong Wang. Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15840-1585. 2024c.
Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Visual signal enhancement for object halluci- nation mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14625-14634, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. Science China Information Sciences, 67(12):220105, 2024.
Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He. Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao. Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang. Taiwen He, Xiaocheng Feng, Jun Song. Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness, 2024b. URL htt.:httt///axiiiiii wrg..i.oorroaaaeee wtt.or/g//a///.aa/a 2405. 7220
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the s International Conference Machine Learning, pp. 5773300077754 2024c.
Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhut- dinov, Louis- Qhiis-Phliippe Morency, Katia P Sycara, and Yaqi Xie. Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models. In The Thir- teenth International Conference on Learning Representations, 2025.
Jianfei Zhao, Feng Zhang, Xin Sun, and Chong Feng. Mitigate language priors in large vision- 2025.
language models by cross-images contrastive decoding. arXiv e-prints, pp. arXiv-2505, 2025.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 0479, 2025a.
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucina- tions in large vision-language models via image-biased biased decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1624-16333 2025b.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazcika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.0105,, 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 AI WRITING ASSISTANCE STATEMENT
Large language models were utilized solely for minor linguistic improvements, including enhanced phrasing and clarity. These tools played role in content generation, experimental design, data analysis, or interpretation. The authors are entirely responsible for all ideas, results, and conclusions presented in this paper.
A.2 MORE DETAILS ON CHAIR
In this paper, we report & and CHAIR, as evaluation metrics. The calculation of CHAIRs and CHAIR, shown in Equation where the maximum number of new tokens to 512 in our experiments.
|
sentences with hallucinated objects]|
CHAIR
fall sentences }
[hallucinated objects}|
CHAIR { all objects mentioned]
A.3
RESULTS ON POPE
The complete experimental results on POPE are presented in Table Our method achieves signifi- cant improvements across all three experimental settings: random, popular, and adversarial.
Dataset
Setting
Method
Accuracy f
Precision
Recall
F1 Score f
Regular
83.29
92.13
72.80
81.33
Random
VCD
87.73
91.42
83.28
87.16
Ours
90.03
90.51
90.03
90.02
Regular
81.88
88.93
72.80
80.06
MSCOCO
Popular
VCD
85.38
86.92
83.28
85,06
Ours
87.33
89.16
85.00
87.03
Regular
78.96
83.06
72.75
77.57
Adversarial
VCD
80.88
79.45
83.29
81.33
Ours
83.07
82.04
84.67
83.33
Regular
83.73
87.16
79.12
82.95
Random
VCD
86.65
84.85
89.24
86.99
Ours
89.57
88.92
90.40
89.60
Regular
78.17
77.64
79.12
78.37
GQA
Popular
VCD
80.73
76.26
89.24
82.24
Ours
84.53
83.51
86.07
84.77
Regular
75.08
73.19
79.16
76.06
Adversarial
VCD
76.09
70.83
88.75
78.78
Ours
81.70
79.15
86.07
82.47
Table 6: Results on L.v.5. The best results are shown in bold.
A.4 RESULTS ON AMBER
We conduct an evaluation LLaVA v1.5 on the AMBER (Wang et al. 2023 AMBER contains both discriminative tasks and generative tasks. The experimental results are shown in Table 7 Our method outperforms existing methods on both discriminative and generative tasks, achieving significant effects in hallucination mitigation.
14

---

Published as a conference paper at ICLR 2026
Discriminative
Generative
Method
Acct
FIt
C
Halp
AMBER SCORE
Regular
67.6 .44
71.2
11.6
47.7
79.80
VCD
Leng
2024
68.1
71.1
9.8
43.8
80.65
ICD
Wang et al.
2024c
70.3
73.4
8.8
38.7
82.3
IBD
(Zhu al.
2025b)
69.2
72.2
9.8
42.2
81.2
DeFG
Zhang et al.
2025
70.2
73.0
9.1
39.9
81.95
CICD
Zhao et al.
2025
71.1
73.1
6.6
34.8
83.25
Ours
81.9
87.2
4.9
20.9
90.01
Table 7: Results on AMBER. The best results are shown bold.
64
Original
VCD Ours
Genaraled
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Latencyls)
Figure Effect of dynamic intervention.
A.5 SCALABILITY ANALYSIS ACROSS MODEL. SIZES
To verify that our method has hallucination mitigation effects for models of different sizes, we select the discriminative task dataset MME and the generative task dataset CHAIR on 1.v 7B and 13B models. The experimental results show in Table that our model achieves significant effects for models different sizes both discriminative and generative tasks.
MME
CHAIR
Model
Method
Existencef
Count
Position:
Colort
Total Scorest
CHAIRst
CHAIR
Regular Ours
175.67
124.67
114,00
151.00
$65.33
$1.0
15.2
LLaVAv1.5 7B
195.00
158.33
133.33
173.33
659.99
30.8
11.4
Regular
185.00
131.67
95.00
175.00
$86.67
45.0
11.8
LLaVA1.5 13B
Ours
185.00
158.33
103.33
180.00
626.66
38.0
10.8
Table Scalability Analysis Across Model Sizes MME and CHAIR. The best results shown
in bold.
A.6 ANALYSIS INFERENCE SPEED
In this section, we investigate DMAS's inference speed. We use LLaVAv1.5 7B and set the genera- tion content lengths to (64, 128, 256) respectively, then compare the inference speed our method with the original model and VCD. The experimental results show in Figure | that method has faster inference speed compared to the decoding method VCD. VCD's inference latency is almost twice that of the original model, but our model achieves better hallucination mitigation effects while adding only small amount of inference time.
A.7 CASE STUDY
To intuitively demonstrate the hallucination mitigation effectiveness of our method, we conduct case studies L.a.Av1.5. We utilize cases from MME and CHAIR datasets, with results shown
15

---

Published as a conference paper at ICLR 2026
in Figure Our method effectively mitigates multimodal hallucination issues across both VQA tasks and image captioning tasks. For VQA tasks, we present various question types, demonstrating method's effectiveness in reducing hallucinations at different levels including object, attribute, relation, and count. For image captioning tasks, our method not only generates fewer hallucinations but also maintains the quality of the output content.
Question: Are thene white dish in the image? answer yes no.
Question: Is the cnicket under batter's body onswer yes or no.
No.
Original model:
Original model: Yes.
Ours:
Ours: No.
Yes.
Question: only one dog in the image? Please answer
Question:
bockpack in this image? Please onswer yes or
Original
Original model:
No.
Ours:
Ours:
Yes.
Original model:
The image features large
unique design, resembling barn
store. The room is filled with various cow statues, including large cow statue in center and several smaller cow statues scattered throughout the space, statues are positioned in different areas of the room, creating visually interesting scene.
Ours:
The image features large, open with fake ceiling that resembles green field. In this room, there are two fake cows hanging from the ceiling, creating unique eye-catching display.
also few potted plants placed around the space, adding touch of nature to the environment.
Figure 6: Case study on LLaVAv1.5
A.8 EXPERIMENTS ON MORE BENCHMARKS
To further substantiate the effectiveness of approach, we conduct extensive experiments multiple challenging benchmarks, including MMVP (Tong et al. (024). Illusio (Shahgir et al., 2024). HallusionBench (multimodal subset) (Guan al.) 2024), and MM-Vet YYu et al. 2024c), evaluating on both VAv1.. and QIenVI.. The results summarized in the Table | confirm that our method delivers strong and robust performance across these diverse and complex evaluations.
Model
Method
MMVP
MM- Vet
Illusion VQA
HallusionBench
Vanilla
30.3
38.9
32.6
50.2
LLaVAv1.5
ICT
31.0
33.4
33.1
52.7
Ours
54.3
39.3
35.2
53.7
Vanilla
41.0
32.9
31.7
42.4
VL
Ours
43.3
34.4
33.6
42.9
Table 9: Experiments more benchmarks.
16

---

Published as a conference paper at ICLR 2026
A.9 COMPREHENSIVE ANALYSIS OF THE CONSTRUCTED DATASET
In this subsection, we will explore how the dataset scale and the dataset composition for constructing steering vectors respectively affect the effectiveness of DMAS.
A.9.1 DATASET SCALE
To investigate the impact of dataset size on steering vector construction, we systematically reduce the dataset to 75%, 50%, and 25% of its original size and evaluated performance MME (Fu al. AMBER (Wang et al. 2023). and MMVP (Tong et al. 2024). The results are presented in The results show that dataset size has minimal impact on performance, with only minor fluctuations across benchmarks. This demonstrates that our method is robust to variations in dataset scale and does not require large amounts of labeled data.
2023
Table I0
MME
MMVP
AMBER
Scale
Existencef
Count f
Position
Colort
Accuracyt
Relation
Fl
100%
195.00
158.33
133.33
173.33
54.3
69.0
87.2
75%
195.00
158.33
133.33
168.33
52.0
69.0
87.2
50%
195.00
158.33
133.33
168.33
52.0
69.0
87.2
25%
195.00
158.33
133.33
168.33
53.0
69.0
87.1
Table Analysis of dataset scale for steering vectors construction.
A.9.2 DATASET COMPOSITION
To explore how different datasets affect steering vector construction, we build databases using two alternative datasets: POPE (Li al. 2023c) and PHD (Liu et al. 2025a). POPE contains only object existence questions (homogeneous), while PHD includes diverse question types similar to our orig- inal datasets. Both databases are constructed using the same scale as our main experiments, and we evaluate performance on MME (Fu al. 2023), AMBER (Wang et al. 2023), and MMVP (Tong et al. 2024). The results are presented in Table 11 Results show that the POPE-hased database underperforms on several MME subtasks and AMBER's relation subtask, which is expected given POPE's limited diversity cannot fully exploit our dynamic approach. In contrast, the PHD-based database achieves comparable results to our original method across all benchmarks. These find- ings demonstrate two important insights: (1) when using datasets with good question diversity, method achieves consistent performance regardless of the specific dataset chosen, and (2) this val- idates the necessity of dynamic retrieval mechanism for selecting appropriate steering vectors based on semantic context.
MME
MMVP
AMBER
Scale
Existencef
Count f
Positiont
Colort
Accuracyt
Relation
Fl f
Ours
195.00
158.33
133.33
173.33
54.3
69.0
87.2
POPE
195.00
158.33
123.33
153.33
52.3
65.1
86.0
PHD
195.00
158.33
133.33
173.33
53.7
68.3
86.9
Table 11: Analysis of dataset composition for steering vectors construction.
A.10 EXPERIMENTS DIFFERENT DECODING STRATEGIES
To explore the effectiveness of our method under different decoding strategies, including beam search (beam=2). nucleus sampling (temperature=0.5, topp=0.7) and greedy search, the results on MME and MMVP benchmarks are presented in the Table 12 Our method consistently demon- strates effective hallucination mitigation across all decoding settings, confirming the robustness of approach.
17

---

Published as a conference paper at ICLR 2026
MME
MMVP
Decoding Strategies
Method
Existencef
Count f
Position
Colort
Total Scorest
Accuracy1
Vanilla
190.00
153.33
113.33
148.33
604.99
29.7
Beam
Ours
191.67
156.66
135.00
171.66
654.99
55.1
Vanilla
190.00
147.78
123.89
150.55
612.22
32.3
Nucleus
Ours
190.00
150.00
130.00
159.44
629.44
53.1
Vanilla
190.00
153.33 158.33
113.33
148.33
604.99
30.3
Greedy
Ours
195.00
133.33
173.33
659.99
54.3
Table 12: Experiments on different decoding strategics.
A.11 EXPERIMENTS MORE MODELS
In this subsection, we test the effectiveness of our method on several advanced LVLMs. We select Qwen2.5-VL (Bai et al. 2025), Intern (Zhu et al.) 2025a). Mantis (Jiang et al.), and Idefics3 MME and CHAIR using greedy search. The experimental results are shown Table T3 Our method remains effective for these advanced LVLMs with different architectures, demonstrating hallucination mitigation effects on both discrim- inative tasks and open-ended generation tasks, which confirms the scalability of our approach.
[Laurençon et al. 2024). and conduct experiments
MME
CHAIR
Model
Method
Total Scores
CHAIRSL
CHAIR
Vanilla
698.33
47.4
10
Qwen2.5-VL
Ours
718.33
43.2
9.5
Vanilla
698.33
28.4
7.6
Intern VL3
Ours
713.33
26.4
6.9
Vanilla
653.33
44.6
12.7
Mantis
Ours
668.33
34.8
9.7
Vanilla
665.00
49.9
8.8
Idefics3
Ours
678.33
45.2
8.2
Table 13: Experiments on more models.
A.12 ABLATION STUDIES ON HALLUCINATION OBJECTS SELECTION STRATEGIES
For visual perception attention heads, we compare two hallucination objects selection strategies: (1) random selection across categories (e.g., people - panda), and (2) hard negatives within the same category (e.g., runner - swimmer). As shown the Table 14 experiments on MME and CHAIR benchmarks demonstrate that both strategies achieve comparable performance with no significant difference. This indicates that our visual steering vector approach robust to the choice of negative selection strategy.
MME
CHAIR
Strategy
Existencet
Count
Position?
Colort
Total Scorest
CHAIRs
CHAIR,
Random
195.00
158.33 158.33
133.33 133.33
173.33 173.33
659.99 659.99
31.2 30.8
10.9
Hard Negative
195.00
11.4
Table 14: Ablation studies on hallucination objects selection strategies.
18


Reviews:
[{"review_id": "t6LGcxuC2I", "reviewer": "Reviewer_kS9v", "summary": "This paper introduces Dynamic Multimodal Activation Steering (DMAS), a training-free and plug-and-play method designed to mitigate hallucinations in Large Vision-Language Models (LVLMs). The core idea is to dynamically intervene in attention head activations by separating intervention into two components: a \"truthfulness steering vector\" and a \"visual perception steering vector.\" The truthfulness vector is learned via contrastive activation differences across semantically clustered data and stored in a dynamic database, while the visual perception vector is derived from clean vs. noisy image inputs. During inference, DMAS retrieves the most context-relevant truthfulness vector and applies both to the top-K most influential attention heads.", "questions": "I think the authors do not use a correct citation format. Citations should be with brackets when the cited papers are not the subjects. please address.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-motivated and proposes a clever approach to VLM hallcunation issue. The realization that truthfulness steering vectors vary significantly across semantic contexts and the resulting dynamic, context-aware database approach is novel within activation steering literature. In addition, the training-free nature of this work is highly preferred, especially for such kind of problems where efficiency matters.", "weaknesses": "I am giving a conditional weak reject, and I think the the following issue should be carefully addressed by authors.\n\n- A more comprehensive analysis of the constructed dataset should be provided. One of the major novelty of the method is to pre-construct a set of embeddings and select steering vectors accordingly from this dataset. As such, the property of the dataset matters a lot, but very limited analysis is given. How different the performance will be if we start from a different dataset? Will the size of the dataset matters? How random the performance is when we change the selection criteria? I think the study of dataset is largely lacking, making the understanding of DMAS very restricted.\n- The overall improvement shown in the main tables are marginal, and authors are presenting them in a somewhat misleading way (the $\\Delta$ should be with the best baselines, not the \"Regular\". This happens a lot in the paper and must be addressed). In table 2 other methods can give SOTA performance or very close performance. Without standard deviation reported, I am not convinced that this minor improvement over ICV and VTI is significant.  The authors are encouraged to include more experiments to demonstrate the effectiveness of DMAS, or otherwise, I doubt the necessity of having such a dynamic, retrival-related method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAM4xQAXFm", "reviewer": "Reviewer_o2L8", "summary": "The goal of this work is to mitigate hallucinations in LVLMs by steering internal activations in a way that respects both visual grounding and factual correctness. The first contribution is an analysis showing that truthfulness and visual perception largely rely on different subsets of attention heads, and that truthfulness directions vary with semantic context. To address this, the paper proposes a training-free method that precomputes steering vectors for the visual and textual pathways and then applies them in a context-aware manner at inference. Concretely, the method clusters prompts into four semantic groups, builds a truthfulness vector per cluster from activation differences between factual and hallucinated answers, derives a visual-perception vector from clean versus noise-corrupted images with object prompts, and then steers the most influential heads using these vectors.\n\nThe results show that the approach improves on discriminative VQA benchmarks such as POPE and MME subsets, and on open-ended captioning with CHAIR, while also showing signs of transfer beyond the construction datasets (ScienceQA, ViQuAE). The ablations explore the choice of hyperparameters and cluster size and importance of using both visual and truthfulness vectors.", "questions": "- Given that the steering database is built from AMBER/SEED (discriminative and MCQ) [lines 172-173] using label flips or random incorrect choices, clustered into four groups, I am curious to know the intuition behind how these vectors remain valid for open-ended generation. What mechanism supports transfer from discriminative supervision to generative benchmarks such as CHAIR?\n- For visual steering vector, does the selecting strategy for negative objects impact this method’s performance? e.g would selection of hard negatives or commonly hallucinated objects as negatives instead of random negative object selection done in the paper impact performance? What are the takeaways from practitioners on the sensitivity of negative object/answer selection for the database generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A training-free and context-aware method to curb hallucinations in LVLMs by nudging a small set of attention heads tied to visual grounding and factuality. It helps on both open-ended captioning (e.g., CHAIR) and VQA-style benchmarks (e.g., POPE/MME).\n- The setup is easy to follow: datasets and metrics are spelled out, baselines are sensible, and the main knobs (α, β, and the number of intervened heads K) are reported with the ranges they tried.\n- Ablations show both components (truthfulness and visual) matter, dynamic retrieval beats a single fixed vector, and you get reasonable guidance for choices like cluster count and K rather than hand-wavy defaults.", "weaknesses": "- Although training-free, the method isn’t truly plug-and-play across LVLMs: the influential-head masks and truthfulness/visual steering vectors must be recomputed for each new model (with α/β/K re-tuned), so deploying on a different backbone requires non-trivial one-time setup rather than drop-in reuse.\n- It’s encouraging to see gains without regressions and signs of cross-domain transfer (Table 5). To make the generality claim more convincing, reporting results against stronger baselines beyond Regular would make the narrative stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9T7hCHMVt", "reviewer": "Reviewer_uqR8", "summary": "This paper proposes Dynamic Multimodal Activation Steering (DMAS), a training-free and plug-and-play method to reduce hallucinations in Large Vision-Language Models (LVLMs). The authors find that truthfulness and visual perception rely on distinct attention heads and that truthfulness patterns vary with semantics. DMAS builds a semantic truthfulness steering vector database and computes visual perception steering vectors, dynamically applying them to the most relevant attention heads during inference.", "questions": "1. How does the method perform on more recent models? For example, Qwen2.5-VL-7B, which was released at the beginning of 2025.\n2. While MME and POPE are simple and well-known benchmarks, how does the method perform on other, more complex ones, such as MM-Vet and LLaVA-Bench? The effectiveness and improvements of the proposed method should be demonstrated across a broader range of benchmarks.\n3. Could the authors release the detailed hyperparameter settings? \n4. Did the authors use greedy search? \n    - If YES, I am a bit curious about the reported results.  Based on my own experiments with LLaVA-1.5-7B on the MME benchmark, the `Existence` subtask under the regular setting should achieve a score around 190, while `Count` and `Color` should be around 160 and 165, respectively.  (Note: Here, I refer to directly using the original MME benchmark rather than data from other repositories, as those may introduce different prompts.)\n    - If NOT, multiple runs should be conducted to obtain statistical results and reduce randomness or variance in the evaluation.\n5. Based on the reported results, the improvements appear to be marginal on POPE on LLaVA, with performance comparable to ICT. So how to demonstrate the effectiveness?\n6. It is interesting that the paper identifies which attention heads are most sensitive to truthfulness versus visual perception. I am curious whether these findings also hold for different models — for example, Qwen-VL, which was used in the main experiments. Furthermore, for the 13B model reported in Table 8, what are the corresponding findings? Since the 7B and 13B models have different numbers of layers, it would be insightful to know whether similar attention patterns are observed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper conducts an interesting analysis of attention patterns, revealing which attention heads are most sensitive to truthfulness versus visual perception.\n2. This paper proposes an interesting Dynamic Multimodal Activation Steering, which incorporates steering vector to mitigate hallucination.\n3. The paper demonstrates good writing quality and is easy to read.", "weaknesses": "1. Experiments were conducted on a limited set of backbones; it would be better to include experiments on more recent models.\n2. Some important hyperparameters are missing from the paper. For example, the temperature, top-p, and top-k settings are not reported.\n3. The paper lacks comparisons with recent decoding strategies, such as DECO [1] and DAMO [2]. As far as I know, they also perform well in hallucination mitigation.\n    - [1] MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation\n    - [2] DAMO: Decoding by Accumulating Activations Momentum for Mitigating Hallucinations in Vision-Language Models\n4. The layout of Table 3 and Table 4 could be improved for better readability and presentation.\n5. The formatting of captions should be consistent throughout the manuscript. For example, Figure 1, Figure 2, and Table 1 all end with a period, but Figure 3 does not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9coIWFSJWD", "reviewer": "Reviewer_KcVa", "summary": "This paper aims to address the hallucination problem in Large Vision-Language Models (LVLMs). Through an analysis of internal activation patterns, the authors report two key findings: (1) the attention heads responsible for truthfulness and visual perception are largely disjoint within the model; and (2) the truthfulness steering vectors that guide factual output vary significantly across different semantic contexts.\nBuilding on these insights, the paper proposes Dynamic Multimodal Activation Steering (DMAS) — a training-free, plug-and-play inference-time intervention technique. The method consists of three main stages:\n1.\tConstructing a Truthfulness Steering Vector Database: Semantic clustering is applied to the data, and within each cluster, activation differences between truthful and hallucinatory samples are computed. These are stored in a database where semantic embeddings serve as keys and truthfulness steering vectors as values.\n2.\tComputing the Visual Perception Steering Vector: Activation differences between the model’s responses to clean and noise-distorted images are used to derive a steering vector that enhances visual attention.\n3.\tDynamic Intervention: During inference, the method retrieves the most semantically relevant truthfulness steering vector from the database based on the input text and combines it with the visual perception steering vector. The combined signal is then applied selectively to the most influential attention heads, effectively reducing hallucinations.\nExperimental results demonstrate that DMAS achieves significant improvements over existing approaches across multiple benchmarks and model architectures.", "questions": "Counterfactual generation: For open-ended datasets (e.g., CHAIR, AMBER), how are hallucinated answers generated? Are they human-labeled, randomly perturbed, or model-generated? How sensitive are the steering vectors to this generation method?\n\nAutomatic hyperparameter selection: Is there a lightweight, label-free method to choose (α, β, K) or the number of clusters, to avoid expensive grid search for each new model or domain?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear problem orientation: The paper provides an in-depth analysis of hallucination issues in LVLMs, particularly revealing the functional tendencies of different attention heads and the semantic dependence of steering vectors through preliminary studies, which offers strong motivation for the subsequent method design.\n\nBroad experimental coverage: The paper conducts comprehensive experiments across multiple models (LLaVA v1.5, QwenVL) and multiple benchmarks (MME, POPE, CHAIR), including detailed ablation studies and case analyses, demonstrating the effectiveness of the method.\n\nSignificant performance improvement: Experimental results show that DMAS achieves state-of-the-art performance across multiple tasks, with particularly substantial gains on MME and CHAIR metrics.", "weaknesses": "Reliance on synthetic counterfactuals in database construction: The method constructs “hallucinated” answers by flipping or randomly selecting incorrect options, which is relatively easy for multiple-choice or discriminative datasets. However, this approach may not reflect realistic hallucination types in open-ended generation tasks. As a result, the learned “steering vectors” may be biased toward these synthetic patterns.\n\nDependence on specific labeled data: As noted, the main limitation of this method is that its so-called “training-free” property relies on a pre-constructed steering vector database that depends on labeled data (true/false answer pairs). This severely limits the generality and scalability of the approach. The authors should more transparently discuss this prerequisite and its associated limitations.\n\nHigh hyperparameter sensitivity: The method introduces several key hyperparameters, including intervention strengths $\\alpha$ and $\\beta$, the number of intervention heads $K$, and the number of clusters. Analysis in Figure 3 shows that model performance is highly sensitive to these parameters, and inappropriate settings can lead to a sharp drop in performance. This makes hyperparameter tuning costly in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)