The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering
Authors: Zhuowei Li
Haizhou Shi
Yunhe Gao
Di Liu
Zhenting Wang
Yuxiao Chen
Ting Liu
Long Zhao
Hao Wang
Dimitris N. Metaxas
Venue: ICML 2025 poster (2025)
Keywords: Large Vision-Language Model
Hallucination
Primary area: deep_learning->foundation_models

Abstract: Large Vision-Language Models (LVLMs) can reason effectively over both textual and visual inputs, but they tend to hallucinate syntactically coherent yet visually ungrounded contents. In this paper, we investigate the internal dynamics of hallucination by examining the tokens logits rankings throughout the generation process, revealing three key patterns in how LVLMs process information: 
(1) *gradual visual information loss* -- visually grounded tokens gradually become less favored throughout generation, and 
(2) *early excitation* -- semantically meaningful tokens achieve peak activation in the layers earlier than the final layer.
(3) *hidden genuine information* -- visually grounded tokens though not being eventually decided still retain relatively high rankings at inference.
Based on these insights, we propose **VISTA** (**V**isual **I**nformation **S**teering with **T**oken-logit **A**ugmentation), a training-free inference-time intervention framework that reduces hallucination while promoting genuine information. VISTA works by combining two complementary approaches: reinforcing visual information in activation space and leveraging early layer activations to promote semantically meaningful decoding. 
Compared to existing methods, VISTA requires no external supervision and is applicable to various decoding strategies. 
Extensive experiments show that VISTA on average reduces hallucination by about 40% on evaluated open-ended generation task, and it consistently outperforms existing methods on four benchmarks across four architectures under three decoding strategies. Code is available at https://github.com/LzVv123456/VISTA.

The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering
Zhuowei Li
Haizhou Shi"
Yunhe Gao 22 Di Liu'
Zhenting Wang' Dimitris N. Metaxas
Yuxiao Chen
Ting Liu
Long Zhao?
Hao Wang'
Abstract
1. Introduction
Large Vision-Language Models (LVLMs) can rea- son effectively over both textual and visual in- puts, but they tend to hallucinate syntactically coherent yet visually ungrounded contents. In this paper, investigate the internal dynamics of hallucination by examining the tokens logits ranking throughout the generation process, re- vealing three key patterns in how LVLMs pro- cess information: (1) gradual visual informa- tion loss visually grounded tokens gradually become less favored throughout generation, and (2) early excitation semantically meaningful to- kens achieve peak activation in the layers earlier than the final layer. (3) hidden genuine informa- tion - visually grounded tokens though not be- ing eventually decoded still retain relatively high rankings at inference. Based on these insights, we propose VISTA (Visual Information Steering with Token-logit Augmentation). a training- frree inference-ting -iie intervention framework that re- duces hallucination while promoting genuine in- formation. VISTA works by combining two com- plementary approaches: reinforcing visual infor- mation in activation space and leveraging early layer activations to promote semantically mean- ingful decoding. Compared to existing methods, VISTA requires no external supervision and is applicable to various decoding strategies. Ex- tensive experiments show that VISTA on aver- age reduces hallucination by about 40% on eval- uated open-ended generation task, and it con- sistently outperforms existing methods on four benchmarks across four architectures under three decoding strategies. Code is available at https: //githube www.///.////./ 23456/VISTA /ISSAA
Large Vision-Languse Models (LVLMs) (Dai et al., 2023; Bai et al., 2023; Chen et al., 2023; Zhu et al., 2023; Liu et al., 2024c) have revolutionized multimodal AI by en- abling seamless integration of visual and textual informa- tion, powering applications from interactive assistance to autonomous systems (Lin et al., 2023; Yang et al., 2024; Lai et al., 2024). However, LVLMs frequently hallucinate semantically coherent yet visually ungrounded contents, hindering their reliability in real-world applications.
Though LVLM hallucination is considered multifaceted (Liu et al., 2024d), a critical cause stems from the overwhelm- ing influence of language priors over visual contexts, and has been studied from the perspective of attention pat- terns (Huang al., 2024; Liu al., 2024g) and distribution divergence within logits space (Leng et al., 2024; Favero et al., 2024). Despite these insights, it remains unclear how hallucination emerges and propagates internally during the course of generation.
Inspecting Token Dynamics in LVLMs. In this work, we take a novel perspective by examining LVLM's generation dynamics through the lens of token logits ranking. Given image and its corresponding description (produced by an LVLM), we identify three categories of tokens (elaborated in Sec. 3.4):
Hidden Genuine Tokens - tokens that are missing in generated contents yet clearly visible from visual input; Decoded Genuine Tokens tokens that appear in contin- uation with visual groundings;
Hallucinated Tokens tokens extracted from the hallu- cinated contents within generation.
We then track each token type's corresponding logits rank- ings throughout the generation across temporal (Fig. left) and layer sequences (Fig. right). Our analysis makes three prominent observations:
Equal contribution 'Rutgers University "Stanford Univer- sity GGooge DeepMind. Correspondence to: Zhuowei Li <zhuowei.li@ ria.li@@nntgrrdd.......
(OBS-1) Gradual Visual Information Loss. As gener- ation progresses, genuine token rankings gradually de- cline while hallucinated tokens are surfaced (see Fig. I left and Fig. 2). This aligns with recent findings (Yue
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VISTA: Visual Information Steering with Token-logit Augmentation
M0c1 Barking trass Actos Stages
Men hanking frcrdd Layas
Ranking g Haris of Genulv Tiiers
2
A000
2369
Earty exchatts
Grodeal informationn
4300
500
Sclurdicau Taom
201p Barking Hallccates Tokers
6000
30,00
Caly
LiAe
21 32
Figure/ 1. Analysis token logits ranking patterns across 500 ran- domly selected images from MSCOCO dataset. Higher ranking indicates higher generation probability. Left: Average token rank- ing from the last five layers, showing temporal progression across early, mid, and late generation stages. Right: Layer-wise evolution of token rankings averaged across all time steps, demonstrating carly-excitation phenomenon.
Figure 2. Token ranking heatmaps for a representative image, demonstrating the evolution of token rankings across model layers (vertical axis) and generation steps (horizontal axis). Darker colors indicate higher ranking. The visualization reveals both gradual visual information loss and early excitation phenomena.
et al., 2024; Favero et al., 2024), exhibiting increasing hallucination in late generation phase. We hypothesize that this occurs as accumulated language priors in resid- ual streams progressively dilute visual information, lead- ing to syntactically coherent but visually ungrounded generation.
lost during generation. Building on these insights, we pro- pose VISTA, an inference-time intervention framework that incorporates two complementary modules. The effective- ness of VISTA is validated through comprehensive exper- iments across multiple architectures (e.g.. LLaVA, Shikra, MiniGPT-4, InstructBLIP) and evaluation protocols (open- ended generation, visual question answering), demonstrat- ing significant reduction in hallucination (up to around 40% for open- open-ended generation). Our approach is notably effi- cient as it requires no additional training or model modi- fications, making it readily applicable to existing LVLM deployments.
(OBS-2) Early Excitation of Semantically Mean- ingful Tokens. Semantically meaningful tokens exhibit peak excitation in penultimate layer (Fig. right) or within a window of layers preceding the fi- nal layer (Fig. 2). In contrast, the final layer prioritizes functional tokens like "this", "a", and other stop- words, suggesting that the model's decision process may overemphasize syntactic elements in its final stage. (OBS-3) Hidden Genuine Information. LVLMs may perceive more visual clues than they express. We ob- serve that hidden genuine tokens, though not eventually decoded, consistently maintain relatively high rankings (around 5K 32K vocabulary) during the course of generation (see Fig. 1).
2. Methodology
In this section, we first establish the notions and knowl- edge foundation in Sec. 2.1, followed by the claboration of token ranking analysis in Sec. 2.2. We then present the proposed VSV and SLA methods in Sec. 2.3 and Sec. 2.4, respectively.
Reducing Hallucination of LVLMs. Inspired by above findings, we propose VISTA (Visual Information Steering with Token-logit Augmentation), a simple and novel training-free framework that can be applied on top of var- ious decoding methods to reduce hallucination while pro- moting genuine information. VISTA introduces two com- plementary modules: Visual Steering Vector (VSV) that counteracts gradual visual information loss by extracting and reinforcing visual cues in activation space, and Self- Logits Augmentation (SLA) which utilizes early excitation patterns to prioritize semantically meaningful tokens. VSV and SLA work synergistically and can effectively mitigate hallucination in L.L.ss
2.1. Preliminaries
Conditional Generation of LVLMs. Suppose that an LVLM consists of vision encoder and a cross-modal in- terface that projects visual inputs into a sequence of visual tokens X,. Given input image, the complete prompt to- kens X, are constructed by concatenating system message tokens X, (can be empty), visual tokens Xy, and query to- kens X,: X, = concat (X,, X,, X₂). ) At each time step I, the model samples a new token oa according to the proba- bility distribution conditioned on both the input context X, and previously generated tokens X (I:
Technical Contributions. This study presents the first sys- tematic investigation of token dynamics in LVLMs through the lens of token logits ranking, revealing novel insights into how visual information is processed and potentially
24 ~ <<>p( Xe, X = coottaa(HH(((
(I)
where H denotes the model's head layer and hf_1 indi- cates the hidden state from the last layer L at time step t-1. Above formulation suggests that the dilution or insuffi- ciency of visual information in h can bias the generation towards hallucination.
We refer to categorized tokens semantically meaningful tokens since we primarily focus identifying objects, attributes, and relations as detailed in Appendix A.
2

---

VISTA: Visual Information Steering with Token-logit Augmentation
Residual Stream. Taking the mathematical interpretation from Elhage et al. (2021), we view layer-wise hidden states as residual streams that evolve recursively:
and layer 1. A lower rank indicates higher probability. This operation produces 2D ranking matrix for each token with vertical and horizontal axes indicating layers and time steps, respectively. We aggregate these matrices across tokens within each category to obtain category-specific ranking patterns, as shown in Fig. 2.
k1 = h + ai + m.
where/ I is the layer index, a' and m² represent the output activation of integrated multi-head attention (MHA) layer and feed-forward network (FFN), respectively. Within this framework, MHAs facilitate information fusion across dif- ferent residual streams, while FFNs access and integrate learned parametric knowledge (Geva al., 2021; Dai al., 2022). Residual stream provides natural interface for monitoring and controlling information flow, making it par- ticularly suitable for hallucination analysis and mitigation.
For temporal analysis, we quantize time sequence into three equal-sized buckets, i.e., early, mid, and late, and com- pute average rankings within each bucket across 500 ran- domly sampled images from MS COCO dataset (Lin et al., 2014). This temporal view (Fig. left) reveals that visually grounded information is gradually sinking while halluci- nated contents are surfaced. We further average rankings across all time steps within a layer to provide layer-wise perspective (Fig. right), which exhibits that semantically meaningful tokens achieve peak excitation in the penulti- mate layer. We refer readers to Appendix A.3 for a dis- cussion of why token ranking analysis is desirable and its limitations.
Logit Lens. The head layer H is by default applied on top of last layer hidden states h However, thanks to the gradual evolvement of hidden states within residual streams (Chuang et al., 2024), applying H to hidden states of earlier layers I < L remains effective, even without addi- tional training (Gurney, 2023). This practice is commonly referred to as "logit lens" and can be used to decipher inter- mediate states.
2.3. Visual Steering Vector (VSV)
Being aware of the challenge from gradual visual informa- tion loss, it is of critical importance to retain visual cues throughout the generation. A promising method involves in- creasing attention weights distributed on visual tokens (Liu et al., 2024g). Nevertheless, this operation simultaneously introduces undesired parametric priors cumulated in residual streams of visual tokens. Drawing inspiration from steering vectors in LLMs (Turner et al., 2023; Zou et al., 2023; Liu et al., 2024e; Li et al., 2024), we propose Visual Steering Vector (VSV) to steer the generation of LVLM towards the direction with visual groundings without amplifying inher- ent language biases.
2.2. Token Ranking Analysis
To systematically investigate how visual information is pro- cessed during generation, we propose a token ranking analy- sis framework that tracks the relative importance of different tokens throughout the generation process.
For each given image- description pair where the text description is generated by an LVLM (e.g., LLAVA-1.5 (Liu et al., 2024a)), we utilize gpt-4o (Hurst et al., 2024) as the oracle model to identify three categories of words referencing both visual and textual contents. A word is a
Identification of Target Tokens.
The core logic behind VSV is to extract directional vector within activation space without introducing disturbing language priors. To this end, we construct VSV via a contrastive process using paired context sequences: a "positive" context X, = concat(X, X,) containing visual tokens X,, and a "negative" counterpart X,, concat (X,, Xg) that dis- cards visual tokens while preserving other elements. Both sequences are processed by vectorization function F, which forwards the given token sequence through the LVLM and takes the residual stream from the last token. Visual steering vector (VSV) can be computed as:
VSV Construction.
decoded genuine word if it appears in the continuation and align with visual evidence;
hidden genuine word if it is visually evident but not included in the continuation;
hallucinated word if it appears in continuation but lacks visual grounding.
Collected words are then tokenized form our analysis sets. Implementation details are included in Appendix A.
Token Ranking via Logit Lens. To analyze token dynamics during generation, we apply the logit lens ((hh)) to each layer/ / and time step t. Given token we calculate its ranking position among all possible tokens according to:
V Vster = Vp - Vn = { I.
(4)
where V,P = =(X,) and V, = =[.,). Here, vieer refers to the steering vector for layer l.
R (a) = rank (mn)))(()).)))) x).
(3)
where (x) represents the position of token x in the probabillty-orddred sequence of all tokens at time step
Inference-time Intervention. During inference, we inject the visual steering vector into the residual stream at each
3

---

VISTA: Visual Information Steering with Token-logit Augmentation
Time exps
Generation: Imogr fomurr
Augmentation Logits. To elicit the rich semantic informa- tion present in the late layers, we calculate "augmentation logits" o at each decoding time step t. These are obtained by applying the head layer H to the hidden states of the w layers prior to the final layer and averaging their logits:
(1
Lapos
IIM hoad
LiMineed
Positive: <5,
aug O
H(h)).
<Image tokens? "Qurry>
W
Time
1=--
where w indicates the window size. In practice, larger w leads to improved performance (see Table 4). We hypothe- size that this is due the ever-evolving logits distributions within late layers, especially for those semantically mean- ingful tokens (Chuang et al., 2024), and applying larger smooths the distribution and provides nuanced information.
Layo
Negatives <Syarm message's <Qurryy>
Imput: -Systmm meoagee Cllnage tokems? Queary
Figure 3. Architectural overview of VISTA. VISTA introduces two complementary mechanisms: VSV extracts and reinforces visual grounding information (V,) at inference, and SLA leverages early- layer semantic information to guide token generation. Note: While three separate forward passes are shown for illustration purpose. they can be avoided in implementation.
Logits Ensemble. Augmentation logits are then integrated with the final layer logits through weighted aggregation:
o, = (1 (1-
7) of + 7 0, ang
(8)
where of H(hf) is the logits of the last layer, and y E [o, 1] is constant coefficient controlling the influence of early-layer information-when y 0, the model reduces to standard generation, while y = 1 would rely entirely on early-layer logits. The next token is then sampled from the output distribution H; ~ (aftmax(o,.1)). Notably, SLA seeks to promote the decoding of more semantically mean- ingful tokens than truthful tokens as done DoLa (Chuang et al., 2024). It works synergistically with the VSV module, enhancing overall performance (see Fig. 6).
generation step:
h 11 Xo', [1,
(5)
with À controlling the intervention strength, balancing visual fidelity against natural generation. To maintain stability, we normalize the modified hidden states:
h ht
LE [1,
(6)
3. Experiments
Unlike steering vectors in LLMs that capture abstract con- cept in an amortized fashion leveraging a group of con- trastive pairs (Zou et al., 2023), VSV computes the steering vector per image basis preserve crucial visual details unique each input image.
In this section, we empirically validate VISTA across four ar- chitectures, three decoding strategies, and four benchmarks. We first present the experimental configuration (Sec. 3. followed by an extensive evaluation hallucination- specific and general-purpose benchmarks (Sec. 3.2 and 3.3). We then analyze VISTA's effectiveness in addressing the observed phenomena (Sec. 3.4) and conclude with compre- hensive ablation studies (Sec. 3.5).
Remark on Difference with Existing Contrastive Strat- egy. Existing contrastive strategy (Li et al., 2022; Leng et al., 2024) necessitates a second negative logits distribution to contrast the final layer logits at all generation steps. VSV, on the other hand, extracts a steering vector priori using only the context tokens, and is applied to all layers (Fig. 3). In practice, formatting prompt and general negative prompt (e.g., bescribe the image in detail. can be forwarded only once and cached for future usage, making VSV remarkably efficient (see Table 5).
3.1. Experimental Setup
evaluate VISTA on four represen- tative LVLMs with distinct architectural designs: LLAVA- 1.5 (Liu et al., 2024a) and Shikra (Chen et al., 2023), which employ linear projections for visual-textual alignment, and MiniGPT-4 (Zhu et al., 2023) and InstructBLIP (Dai et al., 2023), which utilize Q-former (Li et al., 2023a) for cross- modal interaction.
Model Architectures.
2.4. Self-Logits Augmentation (SLA)
Motivated by early excitation phenomenon (Fig. right), where semantically meaningful tokens show stronger activa- tion in penultimate layer, we propose Self-Logits Augmen- tation (SLA) to promote the decoding of such tokens.
Decoding Strategies. To demonstrate VISTA's versatility as an inference-ting intervention method, we verify it across three widely used decoding protocols: (1) greedy decoding.
4

---

VISTA: Visual Information Steering with Token-logit Augmentation
Table 1. CHAIR hallucination evaluation results. We compare VISTA to statc-of-the-art training- free methods that do not rely on external supervision. Maximum new token is set to 512. Best and second best results are bolded and underlined, respectively. "." indicates the result not supported by released implementation.
Decading
Method
LLAVA-1.5 (Liu al., 202
MInGFFT-I (Zm et al., 2023)
Shikra (Chen al., 2023)
InstractLIIP (Dui et al.. 2023)
CHAIR, 4
CHAIR,
CHAIR,
CHAIR
CHAIR,
CHAIR;
CHAIR,
CHAIR, 4
Vanilla
46.4
12.1
36.2
10.7
56.8
14,8
38.0
10.2
DoLa (Chuang al., 2024)
11.9
60.0
15.1
Gruody
VCD (Leng et al,, 2024)
47.4
13.0
45.8
12.8
PAI (Liu 2024gg
22.8
29.2
10.9
40.8
11.0
VISTA (ours)
20.4
6.9
19.8
6.0
31.4
9.7
27.4
8.1
Vanilla
49,0
12.5
33.0
11.0
53.8
14.4
37.8
10.2
VCD [Leng et al., 2024)
49.8
12.4
49.2
13.7
Beam Search
OPERA (Hunng al., 202(1)
45.2
12.4
26.8
9.3
39.6
12.2
50.2
13,9
PAI (1.in et ol., 2024gg
22.3
6.8
31.6
11.2
41.6
10.4
VISTA (ours)
17.4
6.3
18.4
6.4
32.2
9.5
26,8
7.8
Vanilla
53.2
15.1
11.2
56.4
15.9
13.1
DoLa (Chsang et al., 2024)
47.2
14.0
56.6
16.3
Nacleus Sampling
VCD (Long et al., 2024)
16.2
PAI (Liu et al., 2024g)
30.2
10.3
31.8
13.2
43.2
12.0
VISTA (ours)
24.0
8,2
31.8
29.4
9.1
which selects the highest probability token at each step. (2) beam search with a beam size of 5, maintaining multiple generation hypotheses, and (3) nucleus sampling with top- p=0.9. Temperature is fixed at 1.0 for all scenarios.
provides a systematic framework for evaluating object hal- lucination in image captioning tasks. CHAIR assesses caption accuracy by comparing mentioned objects against ground-truth labels, with hallucinations defined as objects present in captions but absent from ground truth. The met- ric operates at two levels: instance-level (CHAIR)) and sentence-level (CHAIRs): CHAIR,
Baselines. Besides three vanilla decoding strategies, we compare VISTA with several SoTA hallucination mitiga- tion methods that can operate without external supervi- sion. DoLa (Chuang et al., 2024) signifies a internal (across layers) contrastive strategy within logits space, while VCD (Leng et al., 2024) presents a parallel contrastive method across time steps. OPERA (Huang et al., 2024) is powerful technique tailored for the beam search. PAI (Liu et al., 2024g) is another inference-time intervention method that is most comparable to ours. We reproduce all base- line results using identical evaluation data and settings (e.g.. prompt, lemperature). Methods without official supports for certain architectures and decoding strategies are omitted to prevent implementation bias.
[(ballucinated abject)] Robject]]
and CHAIRs = |[caption w/ hallucinated objects]]
Following es- tablished protocol (Huang et al., 2024; Liu et al., 2024g), we evaluate on 500 randomly sampled images from MSCOCO 2014 validation set, using the prompt "Please help me describe the image in detail" with maximum generation length of 512 tokens.
Reaption ]
Results in Table 1 show that VISTA significantly reduces hallucination in open-ended generation task, outperforming existing inference-tine intervention method and other con- trastive decoding methods by a substantial margin. VISTA brings around 40% relative improvement upon correspond- ing vanilla decoding methods. Notably, while PAI (Liu et al., 2024g) shows less efficacy for sampling-base based decoding, VISTA excels across all decoding strategies. We attribute this robust performance to the contrastive design and the choice of activation space steering which is not hinged with any specific decoding strategy.
We employ the following config- uration across all experiments unless stated otherwise. The VSV strength parameter À is set to 0.17 for both LLAVA-1.5 and InstructBLIPP 0.1 for MiniGPT-4, and 0.12 for Shikra. The SLA mixing coefficient 7 is consistently set to 0.3, with a window size w = 5 for aggregating early-layer logits. We search the hyperparameters of X and y holdout valida- tion set containing 100 images from MSCOCO to balance between generation quality and hallucination reduction.
Implementation Details.
POPE Evaluation. The Polling-based Object Probing Eval- uation (POPE) (Rohrbach et al., 2018) examines object hal- lucination through targeted visual questions of the form "Is there <objects in the image?". The bench- mark comprises three splits of increasing difficulty: random objects selected from a general vocabulary, frequently oc- curring objects chosen from common categories, and ad- versarially selected objects that are contextually plausible but absent from images. We evaluate on the COCO subset, reporting average accuracy and Fl scores across all splits. Since POPE evaluation is formulated as short VQA format and the response is simply Yes or No, the gradual visual
3.2. Results on Object Hallucination Benchmarks
We first evaluate VISTA on two widely adopted benchmarks that assess object hallucination: CHAIR (Rohrbach et al., 2018) for open-ended generation and POPE (Rohrbach et al., 2018) for targeted visual question answering.
CHAIR Evaluation. Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach et al., 2018)
5

---

VISTA: Visual Information Steering with Token-logit Augmentation
Table 2. Evaluation results POPE henchmark across four Ms..ss Results show averaged accuracy and F1 scores computed across random, popular, and adversarial object splits. Best and second best results are bolded and underlined, respectively.
Decoding
Method
LLA.VA-1 (I et 20244)
MiniCiPT-4 (7hu et 2023)
Shikra (Chen al. 2023)
InstractBl.. IP (Dai et al., 2023)
Avg. Accumny
Aug. F1 1
Aug. Accemcy 1
Avg. F1
Avg. Accuncy
Arg. F1
Avg. Accurcy
Arg.
F1
Vanilla
84,79
85.61
76.76
76.82
81.32
K2.01
84.36
84.65 64
Dol.a (Chusng al., 2024)
84,92
85.67
81.13
81.94
Groody
VCD (1 eng et al., 2024)
84,80
85.65
14.81
55.28
PAI al., 2024gg
85.85
75.64
77.57
81.30
80.81
VISTA (ours)
16.15
N6.29
77,06
77.80
N2.44
82.47
H4.N7
84.95
Vanilla
85.45
84,93
73.68
72.40
81.73
82.10
84.38
83.71
VCD (Leng ot ol., 2024)
95.81
85.90
84,90
Beam Search
OPERA (Huang et al., 2024)
85.68
85.83
24.81
25.42
82.18
82.49
85.31
85.51
PAI iu ol., 2024g)
M6.27
85.91
73.89
74.65
81.90
VISTA (ours)
85.83
85.95
75.96
77.17
82.54
$2.52
85.78
85.74
Vanilla
K1,26
82,40
60.56
62.04
78. 94
80.18
78.83
79. 74
DoLa (Chuang al., 2024)
82.44
29.49
90.72
Nucleus Sampling
VCD (Long et all, 2024)
81.08
82:22
79.61
80:43
PAI (Liu al., 2024g)
81.92
83.16
61.26
63.40
79.25
79.57
VISTA (ours)
85.35
85,54
66.96
68.05
81.01
$1.15
83.11
83.27
Table 3. Overall performance scores on MME full evaluation set.
(SPAT), environmental inferences (ENV), holistic descrip- tions (HOL), and others (OTHER). Unlike conventional VQA evaluations, MMHal-Bench emphasizes logical rea- soning and complex visual understanding, providing rigor- ous test of hallucination mitigation in challenging scenarios. Model responses are evaluated using GPT-4 for alignment with ground-truth answers. We compare VISTA with PAI and vanilla decoding methods for this evaluation.
Higher scores indicate better general capability across perception, reasoning, and knowledge- based tasks.
Decoding
Method
LLAVA-15
MinGGPT4
Shikra
InstrucBBLIP
Greody
Vanilla VISTA
1752.35
969.93
1101.50
1355.25
1771.87
1041,66
1256.22
1364.05
Vanilla
1749.57
869.74
1223.44
1357.02
Beum
VISTA
1763.5
1062.48
1323.25
1366.57
Vanilla
1625.22
845.30
1069.60 60
1397.71
Nucieus
VISTA
1738.56
1069.37
1254.31
1447.36
The results in Fig. 4 demonstrate VISTA's consistent effec- tiveness across all evaluated LVLMs. Compared to vanilla methods, VISTA achieves substantial improvements on average scores, with LLAVA-1.5 and InstructBLIP show- ing the most pronounced gains (~20% and ~30% rela- tive improvement, respectively). VISTA particularly ex- cels in challenging categories such as environmental in- ference (ENV), attribute perception (ATTR) and counting (COUNT), where visual grounding is crucial for accurate responses. While PAI shows competitive performance on specific architectures, VISTA maintains more consistent im- provements across both model architectures and question types. This robust performance across diverse tasks indi- cates that VISTA effectively addresses hallucination while preserving general visual-languge capabilities. Results un- der beam search and nucleus sampling (see Appendix B) exhibit similar performance patterns, confirming VISTA's robustness across different decoding strategies.
information loss is not evident. We therefore adjust VSV strength to A = 0.01 to reduce VSV's impact.
Results in Table 2 demonstrate VISTA's consistent superi- ority across models and decoding strategies. Under greedy decoding. VISTA achieves average accuracies of 86. 15%, 77.06%. 82.44%, and 84.87% for LLAVA-1.5, MiniGPT-4, Shikra, and InstructBLIP respectively, consistently outper- forming both vanilla decoding and PAI. The improvements are particularly pronounced in nucleus sampling, where VISTA achieves significant gains over vanilla decoding across all models: from 81.26% 85.35% (+4.09%) for LLAVA-1.5, 60.56% to 66.96% (+6.40%) for MiniGPT-44, and similarly substantial improvements for other architec- tures. These results highlight VISTA's particular efficacy in excavating genuine information under stochastic sampling settings, while maintaining strong performance in determin- istic decoding strategies like greedy and beam search.
MME Evaluation. To assess whether hallucination mitiga- tion affects general model capabilities, evaluate VISTA on MME (Fu et al., 2023), comprehensive benchmark en- compassing 14 distinct visual-language abilities. MME tests perception, reasoning, and knowledge integration through carefully curated image-question pairs, providing holistic view of model performance. We conduct full-set evaluation across all architectures and decoding strategies.
3.3. Results on Comprehensive Benchmarks
We further validate VISTA on MMHal-Bar Bench (Sun et al., 2023) and MME (Fu et al., 2023), two challenging bench- marks that examine diverse aspects of model behavior.
-ench Evaluation. MMHal-Bench (Sun et al., 2023) provides a specialized framework for assessing hal- lucination in LVLMs through 96 carefully designed image- question pairs. The benchmark spans eight distinct cate- gories: object attributes (ATTR), adversarial objects (ADV), comparisons (COMP), counting (COUNT), spatial relations
As shown in Table 3, VISTA. although initially crafted for addressing hallucination, demonstrates broad benefits be- yond its primary objective. Performance improvements are
6

---

VISTA: Visual Information Steering with Token-logit Augmentation
IX
D47
EW
EW
VETA
H4.
Figure 4. Performance comparison MMHal-Bende across different question categories: attributes (ATTR), adversarial objects (ADV), comparisons (COMP), counting (COUNT), spatial relations (SPAT), environmental inference (ENV), holistic descriptions (HOL), and others (OTHER). Scores computed using GPT-4 evaluation protocol.
Comper uon of Toeen Ranicings Detween Ventlle and VST
Table 4. Impact of window size on SLA performance. Layer ranges (X-31) indicate the span of layers used for logit augmentation, where X varies from 27 to 31. Cs and Ci denote CHAIRs and CHAIR, metrics, respectively.
2040
Bise Oae
2000
4:00
31-31
X531
25-31
28-31
2T-31
8000
G
01
139
762
186
125
77
668
126
774
162
133
764
85
776 03
113
87
566
753
184
144
765
174
127
773
13
721
112
114
200
13
729
137
159
482
141
48
719
784
Curyy
3.5. Ablation Study
Figure 5. Cross-stage token ranking comparison between greedy and VISTA L.AVA-1.5. VISTA effectively promotes the rank- ing of genuine tokens while depressing hallucination tokens.
nepeptiee Stawe
To thoroughly investigate the effectiveness of VISTA, we gauge the practical latency of VISTA, and analyze how different VSV strength (A) and SLA mixing ratio (7) affect the model's performance in terms of hallucination reduction (CHAIR-S and CHAIR-I metrics) and overall quality (FI score). Results are in Fig. 6. Additional ablation results are deferred to Appendix B.2.
consistent across architectures and decoding methods, with particularly striking gains in challenging scenarios. Un- der nucleus sampling, MiniGPT-4's performance improves substantially from 845.30 to 1069.37, while LLAVA-1.5 maintains strong performance with significant improvement from 1625.23 52 to 1738.56. These comprchensive gains sug- gest that VISTA's approach to maintaining visual grounding enhances fundamental visual-language integration mecha- nisms, leading to better overall model behavior.
Table S. Measure of throughput and latency on LLAVA-1.5.
Greedy decoding strategy is applied and listed as baseline.
Methods
Gready
VCD
PAI
VISTA
Latency (msltoken) 1
28.54 (x1..)
58. 58.34
(7.78(22.02)
36.30 (x1.27) 27.53 (x0.79)
Throghput (wkknns)
35.04 (x10)
(x.14
17.31 (K0.49)
Efficiency. The greedy decoding latency in Table 5 shows that VISTA is more efficient than the other test-time inter- vention strategy PAI and contrastive decoding strategy like VCD.
3.4. On Solving Gradual Visual Information Loss
Our analysis in Sec. 2.2 identifies gradual visual informa- tion loss as a substantial challenge for long sequence gen- cration. To validate VISTA's effectiveness in addressing this issue, we compare token logits rankings w/ and w/o VISTA throughout generation. According to Fig. 5, VISTA not only improves the average ranking of hidden genuine tokens throughout generation but also reverses the concern- ing trend of hallucinated tokens, reducing their prominence in mid and late stages where hallucination typically occurs. Not surprisingly, VISTA also maintains a high ranking for decoded genuine tokens since VSV captures all visual clues of an image and reinforces them at all time steps. This quan- titative evidence directly demonstrates VISTA's capability in maintaining visual grounding throughout the generation process. (see Appendix A.4 for additional results).
VSV Strength (A). As we vary injection strength À from 0.0 to 0.18, both CHAIR-S and CHAIR-I scores improve significantly. However, using inappropriate scale (see Fig. 6) will cause clear degradation in FI, suggesting overemphasis on visual features can harm generation quality.
SLA Mixing Ratio (7). The impact of 7 is studied across values from 0 to 0.4, revealing that moderate values of 7 (0.2-0.3) yield the best balance between hallucination reduction and generation quality, and higher y values (< 0.4) leads to degraded performance, hinting the importance of syntactic information.
Synergy & Robustness. As visualized in Fig. 6, there exists a synergistic relation between and 7. Moderate values of both parameters consistently outperform extreme settings of either component alone. Notably, generation
Case Study. Qualitative analysis further supports our quanti- tative findings above. We kindly refer readers to Appendix C for a collection of qualitative examples.
7

---

VISTA: Visual Information Steering with Token-logit Augmentation
ALMOWI Motron
OHArl ALRAEAM Matre
Matronk
S.AMinng be
03
4281
240
40
1350
12:20
32.21
62
32.00
1242
0.40
2710
31.00
17:00
10:0
12.50
7.00
660
3
1422
24.5 10.00
11.00
10.00
15.0 7000
01.00
23.42
$0.00
12:2
1823
1420
17.40
17
12:1
12
440
1844
10.30
13
13.10
13.30
11.00
76:10
7620
01:0
10
12
4748
12:44
1330
5.0
1574
14.31
11.44
11:00
14.00
18:0 0-11 11.31 15.14
7433
605
03
033
532
611
8.14
n.o
03*
937
6:18
985 008
01 011 931
634 e:s nis 0.17
0/6 9:0 6:13 n1z 0.15 934 n.5
kdd n1r 0.16
VSV
Sterghh 04)
VSV Strangh (1)
S2rngh 50
Figure 6. Ablation matrices for VSV strength (A) and SLA mixing ratio (7) on Shikra. Brighter color signifies the better performance. Red boxes highlight the parameter combinations we used. Fl score is included to demonstrate the overall generation quality.
quality remains stable (F1>72.0) across a broad range of parameter combinations, demonstrating the robustness of our approach. This complementarity suggests that VSV and SLA address different aspects of the hallucination problem.
ing, originally introduced in NLP (Li et al., 2022; Shi et al., 2023), has emerged as a promising approach for reducing hallucination in LVLMs. Recent adaptations of this technique have explored various contrasting strategies: VCD (Leng et al., 2024) introduces disual-specitee contrasts by crafting noisy visual tokens as negative samples, while DoLa (Chuang et al., 2024) innovates by contrasting logits distributions from different layers within the same model, us- ing divergence measurements to dynamically select contrast- ing layers. Taking a temporal perspective, M3ID (Favero et al., 2024) proposes a "horizontal" strategy that contrasts current logits with those from previous timesteps. Other approaches extend contrastive techniques to attention mech- anisms (Woo et al., 2024). While these methods primarily operate in the logits space, our VISTA takes different approach by performing contrasts in the activation space and intervening at residual streams. This earlier-stage in- tervention strategy offers an efficient alternative that can complement existing decoding methods.
Window Size in SLA (w). In Table 4, we explore win- dow size from one layer up to five layers across different mixing ratios. As shown, larger window size generally re- duces ballucination, and the optimal configuration achieves both strong hallucination reduction and the highest F1 score (78.4). Interestingly, we observe an inverse relationship between window size and optimal y values, suggesting that while broader layer spans capture richer visual information, they require more conservative mixing ratios to maintain generation stability.
4. Related Work
Hallucination Mitigation in LVLMs. Hallucination the generation of content that is irrelevant, factually incorrect, or inconsistent with visual inputs (Bai al., 2024)- - represents fundamental challenge in LVLM development. Research has identified three primary sources: limitations in visual encoder capabilities (Tong et al., 2024; Liu et al., 2024b; Shi et al., 2024), excessive reliance on learned parametric knowl- edge (Li et al., 2023b; Zhou et al., 2023; Leng et al., 2024;
5. Conclusion and Limitations
This study investigates the hidden life of tokens in Large Vision Language Models (LVLMs) and introduces VISTA (Visual Information Steering with Token-logit Augmentation), a lightweight approach to mitigate halluci- nation. Through systematic analysis, we reveal that visual information gradually attenuates during text generation, but can be effectively restored through our framework's 's visual information steering and strategic use of early-layer log- its Extensive experimentation across diverse architectures and decoding strategies demonstrates that our framework significantly reduces hallucination while preserving genera- tion quality. These findings not only illuminate the hidden dynamics of LVLM behavior but also establish visual infor- mation steering as a promising direction for enhancing the reliability of multimodal Al systems.
Huang et al., 2024), and noisy training data (Liu et al., 2023; Yu et al., 2024). Mitigation approaches span training-based solutions with refined datasets (Yue al., 2024; Jiang al., 2024), post-processing techniques including revision (Yin et al., 2023; Zhou et al., 2023) and verification (Chen et al., 2024; Sun et al., 2023), and inference-time interventions like Visual Contrastive Decoding (Leng et al., 2024) and en- hanced attention methods (Liu et al., 2024g). Recent studies revealing "text inertia" (Liu al., 2024g), where models generate similar hallucinations without visual input, high- light concerning reliance on learned text patterns. Similar to VISTA, VTI (Liu et al., 2024f) also applies representa- tion engineering technique to LVLM hallucination domain, yet with a focus on the instability issue of visual represen- tations. While prior findings advance our understanding, how hallucination propagates through model architectures remains elusive, and existing solutions often require external supervision hinged with specific decoding strategies.
Limitations. VISTA is subject to several limitations. First, while VISTA demonstrates robustness across a range of hy- perparameter values, optimal settings may vary across dif- ferent architectures. Second, the effectiveness of VSV relies on the quality of visual cues extracted by the LVLM's vision encoder models with weak visual encoding capabilities may see reduced benefits. Third, the current implementation
Contrastive Decoding in LVLMs. Contrastive decod-
8

---

VISTA: Visual Information Steering with Token-logit Augmentation
focuses on addressing hallucination in single-round tasks; adaptation to interactive scenarios like visual dialogue may require additional considerations.
Information Processing Systems, 2023. URL https:
/ /ppenrevieww net.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T. et al. mathematical framework for transformer circuits. Transformer Circuity Thread, 1:1, 2021.
Impact Statement
This research advances methods for making large vision- language models more trustworthy and reliable through mit- igating hallucination. While the proposed method demon- strates promising results, its effectiivnness is subject to the inherent capability of large vision-languag model, and im- proper usage may adversely affect model's performance. To the best of our knowledge, there are no ethical or other concerns that need be addressed.
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., and Soatto, S. Multi- modal hallucination control by visual information ground- ing. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 14303-14312, 2024.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang. M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
References
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang. P., Lin, Zhou, C., and Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint a3Xiv:2308. 12966, 2023.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011..om hwtp://2.......mmmmeemm mmm.p....mmppmmm.6........ 446..
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang. Z., and Shou, M. Z. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
URL https:// aclanthology org/2021111.2..2.. eeppleemmmmpmmmiin 446.
Chen, K., Zhang, 7., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15199 2023.
Gurney, A. lens. comm/posts//ActttBBxyaadaaltt /twp::://wwww.. https://ww..esswtrrr
Interpreting
gpt: The logit
Chen, Z., Zhao, Z., Luo, H., Yao, H., Li, B., and Zhou, J. Halc: Object hallucination reduction via adaptive focal- contrast decoding. arXiv preprint arXiv:2403.0044 2024.
Oc-
tober 2023. URL
conm/possts/Acccttaelpaaasssssss Lnterrpretinngggttttheeeee cessed: 2024-10-22.
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J. R., and He, Dola: Decoding by contrasting layers im- proves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httph:t//////. /tap::///ppeee wwtp///wee.eeeeeeeeeeppeeeee.eep www.errwwwwwewweeeeeeeeeeeeeeeee net/ forum? id Id=Th6NyL07na.
Ac-
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviat- ing hallucination in multi-modal large language models via over-trust penalty and retrospection-allocan Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 8493 0093-8502,11 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18633/1/2002.. acl-long.58) 581.. https:::: https////aaaaaaolga.... 2022. acl laracclloonn... 1ong.581.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint 2:Xh:22410.2127, 2024.
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q. Zhang, J., Huang, F., and Zhang, S. Hallucina- tion augmented contrastive learning for multimodal large language model. In Proceedings of the IEFF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 27036-27046, 2024.
Dai, W. Li, J., Li, D.,. Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards general-purge purpose lasion-language models with instruc- tion tuning. In seiethh Conference on Neural

---

VISTA: Visual Information Steering with Token-logit Augmentation
Lai, X., Tian, Z.,, Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9579- 9589, 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, Ocr, and world knowledge, January 2024b. URL
Leng. S., Zhang, H., Chen, G., Li, X., Lu, S.. Miao, C., and Bing, L.. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024c.
Liu, H., Xue, W. Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., and Peng, W. A survey on halluci- nation large lanoun-an models. arXiv preprint WrXiv:242.002555 2024d.
Li, Li, D., Savarese, S., and Hoi, S. 2 Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-1978 19742. PMLR, 2023a.
Liu, S., Ye, H., Xing, L., and Zou, J. In-context vectors: Making in context learning more effective and control- lable through latent space steering, 2024e.
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Con- trastive decoding: Open-ended text generation as opti- mization. arXiv preprini 20. 15097, 2022.
Liu, S., Ye, H., Xing. L.., and Zou, J. Reducing hallucina- tions in vision- vision-language models via latent space steering. arXiv preprint arXiv:2410.1577 15778, 2024f.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. In Bouamor, H., Pino, J., and Bali, K. (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, Singapore, December 2023b. Association for Computa- tional Linguistics. doi: 10. 10.18655/1112002.mmmmlp main. 20. URL ht https: ht//aaaaatthoogo/o org/2023. emn 1 1aallp-main... 20.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallueina- tion in Ivims. arXiv preprint 21772
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. arXiv preprint 1mXiv::809.02/5550 2018.
Shi, M., Liu, E. Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., et al. Eagle: Exploring the design space for multi- modal 1lms with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Li, Z., Xu, Z., Han, L., Gao, Y., Wen, S., Liu, D., Wang, H., and Metaxas, N. Implicit in-context learning. arXiv preprint ar arXiv:2405. 14660, 2024.
Lin, B., Zhu, B., Yc, Y.. Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representa- tion by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L.,, and Yih, W.-t. Trusting your evidence: Halluci- nate less with context-aware decoding. arXiv preprint 2305. 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.,. Dollár, P.. and Zitnick, C. L. Microsoft coco: Common objects in context. Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12 1,, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Sun, Z., Shen, S., Cao, S.. Liu, H., Li, C., Shen, Y. Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprini arXiv:2309. 14525, 2023.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
Tong, S., Liu, Z., Zhai Y., Ma, Y., LeCun, Y.. and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pp.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Turner, A. M., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization, 2023.
10

---

VISTA: Visual Information Steering with Token-logit Augmentation
Wang, L., Li, L.,, Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. Label words anchors: An in- formation flow perspective for understanding in-context learning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 9840-9855, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18655/v1/2023.emmml main. 609. URL https: aa//ccaannnt /ttpaaaaaaaaoooooooooooooooooott org emnlp-mate moo-mainn 609.
Woo, S., Kim, D., Jang, J., Choi, Y., and Kim, C. Don't miss the forest for the trees: Attentional vision calibra- tion for large vision language models. arXiv preprint 178. 2024.
Yang. R., Song, L., Li, Y., Zhao, S., Ge, Y. Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2024.
Yin, S., Fu, C., Zhao, S., Xu, T., Wang. H., Sui, D., Shen, Y., Li, K., Sun, X., and Chen, Woodpecker: Hallucination correction for multimodal large language models. arXiv 2023.
preprint arXiv:2310.
Yu, Q.. Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., and Zhuang, Hallucidoctor: Mitigating hallu- cinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 12944-12953, 2024.
Yue, Z.,, Zhang, L.,, and Jin, Q Less more: Mitigating mul- timodal hallucination from an eos decision perspective. arXiv preprint ar arXiv:2402.14540 /4545, 2024.
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, Analyzing and mitigating object hallucination in large vision-lange langun-angguage models. arXiv preprint ar Xiw:2310000075:55 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Zou, A., Phan, I., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazcika, M., Dombrowski, A.,,., Goel, S., Li, N., Byun, M. J., Wang, Z.. Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach to ai transparency, 2023.

---

VISTA: Visual Information Steering with Token-logit Augmentation
GPT-4o Prompt
You are a vision-language evaluator. Given an image and an Al-generated description, perform the following tasks:
1. List clearly visible contents in the image that are not mentioned in the description.
2. List hallucinated contents in the description that not present in the image.
3. List contents accurately described in the description that match the image.
For each task, include objects, object properties (e.g., color, count, position), and relationships between objects. You must answer each content with a single word, separating different contents by commas. If no contents apply, write "None". Make sure there is overlapping words between three tasks.
Answer [Missing contents]
Answer 2: [Hallucinated contents]
Answer 3: [Accurate contents]
Table 6. The prompt used for GPT-4o identify genuine and hallucinated words.
A. Implementation Details for Token Ranking Analysis
A.1. Prompt for GPT-4o
We employ GPT-4o as our oracle model for identifying three categories of tokens: hidden genuine tokens, decoded genuine tokens, and hallucinated tokens. The precise prompting strategy used to elicit these classifications is detailed in Table 6.
A.2. Additional Implementation Details
A detailed token analysis algorithm is provided in Algorithm 1.
Token Processing, As shown in Algorithm 1, we derive genuine and hallucinated tokens from their corresponding word-level classifications. In cases where a single word decomposes into multiple tokens under the model's 's tokenization scheme, we adopt the first token as a representative proxy for the entire word. This approach ensures consistent handling of multi-toker words while maintaining analytical tractability.
Ranking Aggregation Protocol. When computing cross-stage token rankings (visualized in Fig. 1 left), we implement focused aggregation strategy that considers only the final five layers of the model, deliberately excluding rankings from earlier layers. This methodological choice mitigates the inherent embedding disparity between the model's decoding layer and preceding layers. Since the LLM's decoding head specifically trained on final-layer hidden states, the reliability of token rankings decreases with distance from this layer. Consequently, we restrict our analysis to a window of layers proximate the final layer ensure robust and meaningful ranking estimates.
A.3. Advantages of Token Ranking Analysis
Why Not Attention? While attention matrices have been extensively studied in LVLM hallucination research understand information flow patterns and inform mitigation strategies (Huang et al., 2024; Liu al., 2024g), our token ranking methodology offers several distinct advantages. Previous work, such as PAI (Liu al., 2024g), has attributed hallucination phenomena like "text inertia" to insufficient attention allocation to visual tokens. However, this interpretation is potentially confounded by the presence of "anchor tokens" (Wang et al., 2023; Huang et al., 2024) that aggregate and redistribute information across the network. The existence of these information hubs means that reduced attention weights on visual tokens, particularly in later layers where visual information can be accessed indirectly through anchor positions, may not necessarily indicate information loss.
Token ranking analysis, by contrast, provides more direct insights into the model's processing of visual information. Through explicit tracking at the token level, this approach enables quantitative measurement of visual information preservation throughout the generation process. The methodology reveals gradual visual information degradation patterns that might be obscured in attention-basee analyses. Furthermore, token ranking analysis uncovers previously unobserved phenomena such as hidden genuine information and early excitation patterns, which are not readily distinguishable through attention analysis alone. These capabilities make token ranking analysis particularly well-suited for investigating the mechanisms underlying hallucination in L.L.Ms.
12

---

VISTA: Visual Information Steering with Token-logit Augmentation
Algorithm 1 Token Ranking Analysis Framework
Require: Image I, LVLM model M, Oracle model
Ensure: Token ranking matrices for each category
1: 1 Generate description using LVLM $/
2: D + (I)
3: 1* Classify tokens using Oracle */
4: Wdec: Whid, What t O D) {decoded, hidden, hallucinated]
5: Tdee; Thid. That f- Tokenize dee, W hid, What)
6: for each token category To (Taeo; Thid, That] do
7: for to []] do
8: for to do
9:
h4 < GetHiddenState( (M, (Eq. 3}
10:
logits <- (1)) {Bq. 4}
11: for each token x in To do
12:
(x) + rank(logiss, x) {per Eq. 4}
13: end for
end for end for
16: end for
17: 1* Compute stage-wise aggregation %/
18: for each stage in {early, mid, late) do
19:
for each category in {dec, hid, hal] do
20:
T, + Ge(TimeSteps(s) (timesteps in stage s}
I.g GetFinallayers() (final layers]
R Mean((R) t T,, I L Tel)
23:
end for
24:
end for
output R (Ranking matrices for all categories]
Limitations. Our token ranking analysis approach also pose certain limitations:
1. Oracle Model Reliability: While GPT-4o serves oracle for identifying genuine and hallucinated content, this process can introduce potential biases and uncertainties. The oracle model's 's classifications may not perfectly align with human judgments, and its own biases could influence the categorization of tokens. This is particularly challenging for nuanced cases where the distinction hetween genuine and hallucinated content is subtle.
2. Embedding Space Discrepancy: Another limitation arises from applying the LLM's decoding head to hidden states from earlier layers. Since the decoding head is specifically trained on final-layer representations, there exists an embedding space misalignment when analyzing preceding layers. This discrepancy becomes more pronounced for layers distant from the final layer, potentially leading less reliable token rankings in earlier stages of the network. While our analysis mitigates this by focusing layers proximate to the last layer, the issue remains inherent to the methodology.
Despite above limitations, our analysis provides valuable insights into LVLM's behavior and has proven effective in motivating our hallucination mitigation approach.
A.4. Additional Token Ranking Analysis
In the main text, we present token ranking analysis results for LLAVA-1.5. Here, we extend this analysis to other architectures to demonstrate the generalizability of our observations. As shown in Fig. 7, the cross-stage token ranking analysis on Shikra exhibits similar patterns to those observed in -LAVA-155, with genuine tokens experiencing gradual rank degradation while hallucinated tokens become increasingly prioritized across generation stages. The wise analysis presented in Fig, further corroborates the early excitation phenomenon, where semantic tokens achieve peak activation in layers preceding the
13

---

VISTA: Visual Information Steering with Token-logit Augmentation
Token Ranking Trends Across Stages
2,000
Token Type
Average Token Rank
3,000
Hidden Genuine Taken
Decoded Genuine Taken
4,000
Hallucinated Taken
5,000
6,000
7,000
8,000
Early
Mid Geneartion Stages
Late
Figure Cross Cross-stage token ranking on Shikra.
Token Ranking Trends Across Layers
2,000
Average Token Rank
6,000
8,000
Token Type
10,000
Hidden Genuine Token
Decoded Genuine Taken
12,000
Hallucinated Token
25
26
27
28
29
30
31
32
Layers
Figure 8. Layer-wise token rankings Shikra.
final decoding layer. Fig. 9 presents comparative analysis between vanilla decoding and VISTA (greedy-based) on Shikra. The results demonstrate VISTA's effectiveness in maintaining the ranking of genuine tokens throughout the generation process while simultaneously suppressing the promotion of hallucinated tokens. This pattern is consistent with our findings for LLAVA- suggesting that the phenomena we identified and the effectiveness of our mitigation strategy generalize across different LVLM architectures. The consistency of these patterns across architectures with distinct design choices (linear projector in Shikra versus Q-former in other models) provides strong evidence for the fundamental nature of these phenomena in LVLM generation dynamics.
B. Additional Experiments
B.1. MMHal-Bench Results For Other Decoding Strategies
We further report results of HMMHal-Bench under beam search (Fig. 10) and nucleus sampling (Fig. 11). As demonstrated in figures, VISTA consistently improves overall performance across all evaluated LVLMs under both decoding strategies. The performance trends remain consistent with those observed under greedy decoding in the main text, further validating the robustness our approach across different inference strategies. These comprehensive results demonstrate that VISTA's mechanisms for maintaining visual grounding and promoting semantic richness are effective regardless of the chosen decoding strategy.
14

---

VISTA: Visual Information Steering with Token-logit Augmentation
Comparison of Toxen Rankings Between Vanilla and VISTA
Hidden
3003
Decaaee Nou Decoded toste)
Araraee Thkan Bart.
salludrated (esiin
5833
1830
5830
19091
Generotin on Stages
Figure 9. Cross-stage token ranking comparison between greedy and VISTA (greedy-based) on Shikra. VISTA effectively promotes the ranking of genuine tokens while depressing hallucination tokens.
LLAWA-1.5
MiniGPT-4
Shikra
InstructBUI
CONP
COUNT
COUNT
ADV
ADV
ADV
SPAT
SPAE
SIAT
SPAT
0.3
1.22
1.00
ENV
1.00
3.0
132
Average
Average
Average
Average
HOL
HGL
HDL
o THER
OTHER
OTHER
OTHER
beam
PAI
VIS
Figure 10. Performance comparison on MMH Bench using beam search.
COMP
COMP
COMP
COMP
COUNT
COUNT
COUNT
COUNT
ADV
ADV
ADV
ADV
2.0
2:67
2,70
1.05
052
Average
Average
Average
Average
HOL
HOL
HGL
HDL
THih
THE
OTHER
OTHER
nucious
Figure 11. Performance comparison on MMHal-BoMne using nucleus sampling.
15

---

VISTA: Visual Information Steering with Token-logit Augmentation
CIAllo Adbation
ADMIS Martccas
11.00
8:00
1.43
1.12
2:42
2:1
4.00
4.10
71.00
10.00
N1:11
30.00
29.00
1980
01:00
11:00
13.00
9:12
8.00
8:20
8:00
6:00
9:20
0.00
3.10
4.10
10.0
91.00
75:80
10.25
1080
09:00
05:25
6215
01:00
04:01
01.55
0-
6.89
6:10
0087
-9.0
23.00
6.4
3.0
3:4
0.00
4.0
1200
10.00
1030
15:40
11.0
10.00
18:00
14.05
10.0
0100
00.00
021
04
00
0.5
kia
0/4
014
6.
027
Figure 12. Ablation matrices for VSV strength (X) and SLA mixing ratio (?) on MiniGPT-4, Brighter color significs the better performance, and red boxes highlight the parameter combinations used Table F1 score is included indicate the overall generation quality.
CHARS Aitittitiinnn Marie
CHAES
Golllie Monte
FEAARRE Martee
4000
4110
13:22
01:48
15.00
11.3
31.00
27:00
23:05
1110
1330
1430
1344
1180
12:3
11.30
11.30
20.00
9:23
8:13
11.00
7780
7120
71:00
40
0::0
14118
11:00
13.35
11.00
10.00
24:00
13:32
19:0
14th
1580
12:99
15:40
12:35
11:47
9:20
8.10
17
10:35
1730
7720
72:00
9720
1920
19:00
19:00
14:10
Le7b04 66
Leno0
Lembo 00
Figure 13. Ablation matrices for VSV strength (A) and SLA mixing ratio (7) LLAVA-1.5. Brighter color signifies the better performance, and red boxes highlight the parameter combinations used Table F1 score is included to indicate the overall generation quality.
4140
15.00
11:00
37
30/M
21:0
21
23
11.00
30.00
MNN
14:24
1:
9:14
510
8.14
1.54
M420
M:50
75
14
74.0
M:30
73
73
23.34
15.00
12:00
73
1120
11:00
4:4
1029
10:00
1130
21.0
11:0
1328
10.00
50.0
822
1:20
1050
922
640
11.00
91.0
7135
19.00
11.00
1220
11.0
615
115
614
518
258
65
<11
015
<ls
255
555
012
013
014
211
816
Figure 14. Ablation matrices for VSV strength (A) and SLA mixing ratio (7) on InstructBLIP. Brighter color signifies the better performance, and red boxes highlight the parameter combinations used in Table 1. Fl score included to indicate the overall generation quality.
16

---

VISTA: Visual Information Steering with Token-logit Augmentation
B.2. Additional Ablation Results
In addition the ablation studies presented in the main text, further provide detailed ablation results on LLAVA- 1.5, MiniGPT-4, and InstructBLIP to validate hyperparameter choices. Figure 12 presents the ablation matrices for MiniGPT-4, examining the impact of VSV strength (A) and SLA mixing ratio (7) on CHAIRs. CHAIR, and Fl scores, The results reveal similar trends to those observed in Shikra, though with a slightly lower optimal À value of 0.1, suggesting architecture-specife sensitivity to visual steering. For LLAVA-1.5 (Figure 13), the ablation matrices demonstrate particularly strong performance improvements with stronger VSV value (A = 0.17). The InstructBLIP results (Figure 14) show robust performance across broader range of parameter combinations, with optimal performance achieved at À 0.17 and ? = 0.3, matching the configuration used for LLAVA-1.5.
Across all architectures, we observe consistent pattern where moderate values of both VSV strength and SLA mixing ratio yield the best balance between hallucination reduction and generation quality. This consistency, despite architectural differences, validates the generality of our approach while highlighting the need for architecture-speciff fine-tuning of hyperparameters for optimal performance.
C. Case Study
In this section, extend qualitative evaluation across all four architectures demonstrate VISTA's effectiveness in reducing hallucination while promoting genuine information. Figures 15-18 present comparative examples between vanilla decoding and VISTA for LLAVA-1.5, MiniGPT-4, Shikra, and InstructBLIIP respectively. For each example, hallucinated content is highlighted in red, while genuine information that was previously omitted or wrongly described but successfully recovered by VISTA is marked in blue. The examples demonstrate VISTA's consistent ability to reduce hallucination across different architectures while eliciting pertinent visual details that vanilla decoding fails to capture.
17

---

VISTA: Visual Information Steering with Token-logit Augmentation
The image captures young boy in baseball uniform, standing at home plate and holding baseball bat, ready to swing, He is surrounded by crowd of people, inchuding teammates spectators,
Greedy:
are watching the game intently. There are several chairs placed around the fiekd, with some of be
The image features a red stop sign with unique message writeen The reads "Sook Sook Eee Stap." " which playful variation the standard "Stap" message. The sign is placed on pole, and it is situated in front of building. The also includes adding overall atnosphere location.
Greedy:
them occupied the spectators. A baseball glove seen on the ground, likely belonging to one the players. The is lively and filled with anticipation the young boy prepazes the ball.
background,
SOOK SO <EE <IT STOP
The image depicts baseball gume in progress with young standing home plate, holding baseball bat and preparing hit the ball. The and is the main focus of the scene. There are several people background,
VISTA:
image features message written on it. The sign is positioned on pole, and the message reads "Sook So So Stop." The siga located in street setting, with tree in the background.
VISTA:
stop sign playfill
wearing baseball uniform
them sitting benches spectators standing around field. few are holding handbags, and few are wearing hats. The anticipation the game.
scene captures
LLAVA-1.5
The image features a red and yellow unin traveling down the tracks, with large train following behind
Greedy:
The features wearing helmet and ricing a black motorcycle down street He is the main focus of the scene, with the motoreycle taking up a signlliant portion of the image. The commuting traveling down the road. In the background, there are a few other people visible, but they are not the main subject of image. The relatively empty, with no other vehicles or signifieant obstacles visible.
Greedy:
moving along several other trains visible in the background, indicating busy train station.
and
appears
enjoying lis ride, possibly
the trains, there are multiple people seattered throughout the likely waiting for their respective trains simply passing by. Some of these individuals are closer to the foreground, while others are further away, giving of depth the image.
appears
image The train silver and red color, with yellow front. The unin moving along the uncks, and there are multiple train visible in scene. micldle of the scene, and the tracks spread across the entire image.
VISTA:
traveling
image features person riding motoreycle on The person wearing helmet and backpack. cusuring salesy while riding, The motorcycle is black and occupies significant portion of the scene. The person positioned in the middle of the image, with the motoreycle taking up majority of the frame.
positioned
Figure 15. Case study LLAVA-L.S.
18

---

VISTA: Visual Information Steering with Token-logit Augmentation
The image shows hat holding temnis racket on tennis court. The woman standing on left side the court and the net in the background. There visible in the top right corner of the image. The woman wearing white shoes and her hair is pulled back into ponytail. The racket held both hands and serve the ball.
Greedy:
in blue dress and white
The image shows person riding dirt bike the air, with cloudy sky in the background. The person wearing a black helmet, black gloves, and green jumpruit. green
Greedy:
background and the
and being ridden by person in black hooss and gloves. The dir in the background brown and there are some trees in the distance.
ready
The image shows woman in blue dress and white tennis shoes an tennis court, is holding a tennis racket in her right hand and about to serve the ball. There are trees in the background.
VISTA:
image shows person green riding dirt hill, The person is wearing green helmet and green jumpsuit. The background dirt hill with few small dirt mounds on it.
MiniGPT-4
The image shows group of sheep standing in muddy field. The sheep are white appear to a similar age and size. They are standing in cirele, with their heads facing inward and their tails tied together. center the cizcle a large pile hay or strww which appears to have beea stacked up by the sheep. The sky is cloudy and there
Greedy:
Greedy:
The image shows man sitting on the hood a car with his feet the ground. He wearing white shirt and blue cell phone. the background, there
the
jeans and talking is a dark sky and some trees. The car is black sedan with hood open.
background.
The image shows group of sheep standing muddy field. The sheep all white and appear be group. There hay the foreground, trees and blue ska. The sheep appear to be in the process eating hay.
VISTA:
The image shows man sitting on car in a dark area. He wearing white shirt and blue jeans. background is black car. The bolding phone car. The image taken at night.
VISTA:
background
Figure 16. Case study MiniGPT-4.
19

---

VISTA: Visual Information Steering with Token-logit Augmentation
The image features young girls sitting together diving table, enjoying their donuts, One girl is on the left side the table, while the other is on the right. They both appear be treats. There scene, with being occupied by the girl on the left and the other chair by the girl the right. The table they are sealed at is quite long, extending across the image. There also cup placed the table the right side.
Greedy:
Greedy:
The image features the icunic Big Ben clock tower in Loadon, with cloudy sky overhead. The tower part the Parliament
quite content
in
streag against backdrop. There large tree in the foreground, partially obscuring the view of the clock. Several birds can seen flying near clock tower, with one bird perched the top left corner and two the image. The birds' preseace adds lively touch to the scene.
birds flying closer the bottom left of
The image features two young girls, one with glasses, enjoying donuts table. They both holding their respective donuts, with girl the left and the other the right. The girls are positioned in the middle of the scene, and appears they sitting
VISTA:
cloudy sky, towering building. and large clock on the side of the building. The clock is visible, along with the tower's overall sixe, making it prominent feanure scene. The sky appears shades, including gray, beige, and slightly blue, creating picturesque seting for the scene.
VISTA:
image
chair. There
chair visible in the overall
background, but the main locus atmosphere of the image is that of unjoying shared treat together.
Shikra
Greedy: The image captures a man skiing down snow- covered slope mountain. He wearing orange helinet and skiing cluster of The skier appears be enjoying the downhill run as he moves quickly through the snow. addition
The image features a man wearing black shirt and a purple He is standing room with wooden painting on
Greedy:
wearing glasses, which adds his distinguished look. In the background, there two visible. is placed on room, while other located closer to the center. Additionally, there one on the floor near the man and the other further the right. The overall setting suggests a well-decorated living space.
appears
side
backpacks visible
scene. skier, while the other is positioned closer to the middle image. presence backpacks suggests the skier might carrying some gear or supplies his skiing adventure.
located
books the
The image features with activity. There is skier, specifically, who sking down the snow-covered slope. The skier wearing an orange helmett also riding skis, making their way through the snowy landscape. The skier's 's ski poles can be seen their hand while skiing. The scene also highlights the presence trees surrounding sking arca, adding the picturesque quality the scene. The ures add to the lively aunosphere of the area, making this great spot for sking and exploring.
VISTA:
mountain slope bustling
young wearing black and purple di, He is standing confidently, with slight smile on his face. The man appears be wearing glasses, which
VISTA:
The image
middle the frame, and there is box nearby. The man's tie porple, and is tucked under bis shirt.
positioned
Figure 17. Case study Shikra.
20

---

VISTA: Visual Information Steering with Token-logit Augmentation
The image features a dog and cat playing in the ocean, dog splashing
Greedy:
Greedy: image features man standing on sandy beach, Blynng a kite in the sky, He is wearing red shirt and is holding the kite string, which is stretched out in the air. The kite is flying high above the man, creating beautiful sight against the clear blue The beach surrounded water, with body of water people the
and the watching from distance. The is positioned closer to the water, while the cat is farther away, possibly observing dog's playful behavior.
water is foamy and bubbly, indicating lively seme of the animals enjoying the occan.
visible the background. There scene, including woman and a child, who are also flying.
enjoying the
VISTA: A man is standing on beach, flying kite in the sky, wearing shirt and focused fying high up
VISTA: dogs playing ocean, with dog splashing in the water and the other dog enjoying the waves. The dogs are in the ocean, splashing and playing in the foreground
which relaxed and enjoyable setting, enjoying the sunny day and the serene atmosphere of the beach. The kite is colorful and playful addition to the leisure.
waves, one dog background, The dogs having fun time in the enjoying the water and the playful waves.
other
scene, adding
InstructBLIP
The image features person standing snow, holding snowboard, The snowbardd is positioned on the ground next the person, who weating blue jacket and pants. There are several ground
Greedy:
brown horse standing grassy field, with a white hird perched on its back. The bird positioned the horse's back, with its wings spread ont, and its body resting The
Greedy:
The image
in scene, one placed snowhoards snowhoard rack. The snowboard rack is another
both standing the field, with the bird's head slightly tumed towards the left side of the image. The grassy field filled with various plants and shrubs, creainng lush and natural setting.
located on the left side of the image, with few snowboards placed on it. The snowhoard rack likely used store organize the snowboards for easy snowboarding activities.
during
woman standing snow, wearing and white jacket, and holding snowboard her hands, She is posing for photo in winter setting, likely
There is a brown horse standing in grassy field, with white hird perched on it back. The horse and bird are both in the foreground scene, with the is positimned the middle of the field, with the bird oa its back, and the background features rolling landscape with hills and a distant horizon. The scene depicts peaceful moment in natural setting, with the horse and bird enjoying relaxing grassy field.
VISTA:
field ski slope. There are several other mowhoards visible the scene, likely belonging
resting on horse's back, The horse
other people
the background. The red and black snowboard,
snowboard in foreground and the woman is smiling as she poses for the picture.
the
Figure 18. Case study for InstructBLIP.
21


Reviews:
[{"review_id": "2i1EJ5dA7Y", "reviewer": "Reviewer_4sSv", "summary": "This paper discusses the topic of reducing hallucinations in LVLMs. The authors analyzed  LVLM’s generation dynamics through the lens of token logits ranking and proposed three types of inference issues. Then, the authors proposed a Visual Steering Vector (VSV) and a Self-Logits Augmentation (SLA) method to inhibit the hallucinations. Experiments are conducted on multiple benchmarks and demonstrated superior results over previous methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely appreciate your thoughtful review and insightful feedback. We are glad that you find our approach __\"simple yet effective\"__, our evaluation is __\"comprehensive\"__, and demonstrate __\"improved performance\"__. Below we address your questions in detail*.\n\n**Q1. Regarding novelty**\n\n>Our VISTA is `novel` in terms of both its `analysis` and `methodology`. To our best knowledge:\n\n>* VISTA's token ranking analysis is `the first work` inspecting `internal token-level dynamics` of LVLMs.\n>* VISTA systematically uncovers `three novel observations about how LVLMs store and process visual information`: (1) gradual visual information loss, (2) early excitation, and (3) hidden genuine information.\n>* VISTA resembles `the first activation-space intervention method for reducing LVLM's hallucination`, effectively addressing observed phenomena.\n\n>As supported by you, VISTA is a `flexible design` allowing integration with various architectures and decoding strategies. This versatility enables VISTA to `complement ongoing research` focused on improving vision encoders, strengthening visual-textual alignment, and developing specialized decoding methods to reduce hallucinations.\n\n**Q2. Regarding computational efficiency**\n\n>VISTA functions as an efficient de-hallucination strategy that `introduces only minimal computational overhead` while demonstrating `superior efficiency compared to other baseline approaches.` We clarify this in detail below:\n\n>* **For captioning, summarization, and open-ended generation tasks**: The textual prompt remains constant (e.g., \\\"please describe the image...\\\"). VISTA forwards the textual prompt `only once` without the image, caching activations for future use (minimal overhead). For the positive example (visual + textual tokens), VISTA performs vanilla inference by forwarding tokens visual + textual once. The only additional step is constructing the visual steering vector (VSV) using the residual stream from the last input token and cached negative activations via `simple vector arithmetics` (this cost is marginal). Generation then proceeds normally under our proposed intervention. VISTA `avoids forwarding the token sequence twice` by leveraging the KV cache from input prompt tokens. Remaining computations involve logit ensembling and a few element-wise additions, resulting in mild overhead.\n\n>* **For QA-like tasks**: Textual prompts may vary per query, potentially requiring one additional forward pass for the negative instance (i.e., textual only query). However, the `textual tokens are minimal compared to image tokens`, typically constituting less than 10% of total input token sequence. Consequently, the extra computational burden remains mild compared to vanilla inference.\n\n>* **Empirically**, VISTA demonstrates `better efficiency` compared to other comparable methods (see Table 5). We kindly refer the reviewer to the panel of Reviewer 5sMs (Q2) for an extended efficiency comparison with other methods.\n\n**Q3. Regarding hyperparameter sensitivity**\n\n>VISTA includes two hyperparameters: intervention strength $\\lambda$ and mixing ratio $\\gamma$. \n\n> * For $\\gamma$, we found $\\gamma\\approx0.3$ `consistently effective across all four architectures`. \n\n> * For $\\lambda$, optimal values can vary by model; however, a broad range of $\\lambda$ values perform effectively for each model, `as analyzed in the Synergy & Robustness subsection (p.7, line 381)` and shown in `Figs. 6, 12–14`, which demonstrate VISTA's consistent effectiveness across different $\\lambda$ configurations.", "claims_and_evidence": "Yes, they are.", "methods_and_evaluation_criteria": "Yes, appropriate evaluation criteria is applied.", "theoretical_claims": "This work focuses more on experimental analysis rather than rigorous theoretical proofs.", "experimental_designs_or_analyses": "Yes, experimental designs are soundness.", "relation_to_broader_scientific_literature": "This work is particularly related to hallucination mitigation in LLM, VLM, MLLM, etc.", "essential_references_not_discussed": "The references are satisfactory.", "other_strengths_and_weaknesses": "**Strengths**\n- The approach is simple yet effective.  It is a training-free approach that can be applied at inference time to existing models.\n- The authors test their approach across multiple architectures, decoding strategies, and benchmarks, demonstrating a comprehensive evaluation.\n- The approach demonstrates improved performance over previous methods in hallucination reduction.\n\n**Weaknesses**\n- The novelty is relatively limited. The concepts had already emerged in previous studies.\n- The proposed approach increases the computation burden.\n- The proposed approach incorporates some hyperparameters that may be sensitive to different benchmarks.", "other_comments_or_suggestions": "Kindly review the comments mentioned earlier.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNMYV48M1M", "reviewer": "Reviewer_5sMs", "summary": "Through the observation of the LVLM generation process, this paper introduces a hallucination mitigation method, VISTA, which includes a visual steering vector and logit ensemble. Experiments demonstrate that it outperforms existing methods.\n\n## Update after rebuttal \nI agree with the authors' rationale and will maintain my initial score (weak accept).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely appreciate your detailed review and constructive suggestions. We are encouraged to find our framework is being recognized as __\"Applying an effective method to each problem\"__. All __claims are evidenced__, and related work are __\"well discussed\"__.  Below we address your questions in detail*.\n\n**Q1. Regarding novelty**\n\n>Our VISTA is `novel` in terms of both its `analysis` and `methodology`. To our best knowledge:\n\n>* VISTA's token ranking analysis is `the first work` inspecting `internal token-level dynamics` of LVLMs.\n>* VISTA systematically uncovers `three novel observations about how LVLMs store and process visual information`: (1) gradual visual information loss, (2) early excitation, and (3) hidden genuine information.\n>* VISTA resembles `the first activation-space intervention method for reducing LVLM's hallucination`, effectively addressing observed phenomena.\n\n>As suggested in your second bullet, VISTA indeed `provides a new perspective` by tracking the rankings of proposed \"hallucinated\", \"genuine\", and \"hidden genuine\" tokens. As a result, VISTA unveils three novel observations, and contributing valuable new insights to the field (supported by Reviewer ypPL).\n\n**Q2. Comparison and Discussion with [1] [2]** \n\n>Below, we first compare VISTA with [1] and [2] in terms of performance and efficiency, followed by an in-depth discussion of the differences between VISTA and these methods. *The extended comparison and discussion will be included in the next revision.*\n\n* **Performance** \n\n>|CHAIR|LLAVA-1.5| MiniGPT-4|Shikra|InstructBLIP|\n|:-|:-|:-|:-|:-|\n||$C_S$/$C_I$↓|$C_S$/$C_I$↓|$C_S$/$C_I$↓|$C_S$/$C_I$↓|\n|ED [1]|43.0/14.0|-|-|-|\n|CT$^2$S (greedy) [2]|44.2/12.2|-|44.8/12.8|42.3/12.4|\n|__Ours__ (greedy)|__20.4/6.9__|__19.8/6.0__|__31.4/9.7__|__27.4/8.1__|\n|CT$^2$S (sampling) [2]|45.0/11.7|-|46.0/12.9|43.6/13.1|\n|__Ours__ (sampling)|__24.0/8.2__|__18.4/6.4__|__31.8/9.7__|__29.4/9.1__|\n\n>|POPE (avg)|LLAVA-1.5| MiniGPT-4|Shikra|InstructBLIP|\n|:-|:-|:-|:-|:-|\n||Acc/F1↑|Acc/F1↑|Acc/F1↑|Acc/F1↑|\n|ED [1]|__86.31__/85.86|-|-|-|\n|CT$^2$S (greedy) [2]|85.94/85.92|-|81.84/82.23|82.30/83.58|\n|__Ours__ (greedy)|86.15/__86.29__|__75.96/77.11__|__82.44/82.47__|__84.87/84.95__|\n|CT$^2$S (sampling) [2]|85.14/85.41|-|80.66/__81.65__|81.49/82.70|\n|__Ours__ (sampling)|__85.35/85.54__|__66.96/68.05__|__81.01__/81.15|__83.11/83.27__|\n\n>As shown above, VISTA achieves `superior hallucination reduction performance`, particularly in long-sequence generation tasks such as CHAIR.\n\n* **Efficiency** \n\n>We further compare efficiency among methods. Given different evaluation criteria, we set the cost of vanilla inference to 1 and measure relative changes. As demonstrated below, `VISTA also outperforms [1] and [2] in efficiency`.\n\n>|Method|Relative Inference Cost↓|\n|:-|:-|\n|Vanilla|1.0|\n|ED [1]|3.13|\n|FastED [1]|1.33|\n|CT$^2$S (40%) [2]|1.43|\n|CT$^2$S (10%) [2]|1.35|\n|**Ours**|**1.27**|\n\n* **Discussion** \n\n>`The rationale and implementation of logits ensembling differ significantly between VISTA and [1][2]`. Method [1] aims to reduce visual distractions through image cropping, ensembling logits from multiple subcrops. Method [2] addresses global noise by pruning visual tokens based on the attention matrix, creating auxiliary logits. In contrast, VISTA's self-logits ensembling is motivated by the observation of early excitation behavior and is conducted across layers preceding the final layer to encourage decoding semantically meaningful tokens. Moreover, the proposed self-logits augmentation demonstrates synergy with the visual steering vector (VSV), as detailed in the *Synergy & Robustness* subsection (p.7, line 381).\n\n**Q3. What if $V_n$ is directly subtracted?**\n\n>Following your insightful suggestion, we set the visual steering vector to $V_s=-V_n$ instead of $V_s = V_p-V_n$ and validated it across various intervention strengths ($\\lambda$) on LLAVA-1.5. Results below indicate that:\n\n>* Solely removing information from the negative example offers `limited benefits`.\n>* Negative-only steering is highly `sensitive to intervention strength`.\n>* Best performance of negative-only steering significantly `lags behind the original design`. \n\n>Specifically, negative-only steering collapses (F1 < 70) when $\\lambda > 0.1$. These findings highlight the critical role of preserving positive information during the intervention process.\n\n>|$\\lambda$|$V_s=-V_n$|$V_s=V_p-V_n$|\n|:-|:-|:-|\n||$C_S$↓/$C_I$↓/$F1$↑|$C_S$↓/$C_I$↓/$F1$↑|\n|0.05|56.8 / 15.2 / 75.5|**50.8** / **13.6** / **76.9**|\n|0.08|52.5 / 14.9 / 75.6|**50.4** / **14.6** / **76.5**|\n|0.1|**41.6** / 14.4 / 74.2|47.4 / **13.1** / **77.8**|\n|0.11|21.4 / 12.7 / 58.1 (collapse)|**45.6** / **12.9** / **77.4**|\n|0.12|2.6 / 23.6 / 13.7 (collapse)|**41.6** / **12.2** / **77.7**|\n|0.15|0.20 / 50 / 0.1 (collapse)|**32.4** / **9.8** / **76.9**|\n|0.17|- / - / - (collapse)|**20.4** / **6.9** / **72.8**|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions relate to MLLM hallucination mitigation.", "essential_references_not_discussed": "All related works are well discussed.", "other_strengths_and_weaknesses": "Strengths\n- Applying an effective method to each problem: a steering vector to enhance the model's focus on visual information and a logit ensemble to address the early excitation of semantically meaningful tokens.\n\nWeaknesses\n- Lack of novelty: \n-  As the authors mentioned, each observation has already been discussed in previous literature. What is being observed for the first time in this paper? Is it the identification of three types of tokens and the observation of the LVLM generation process from that perspective?\n- Also, the logit ensemble method has already been proposed in recent works [1, 2]. Please verify its superiority compared to those works.\n\n[1] Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding, ICLR 2025\n\n[2] Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, ICLR 2025", "other_comments_or_suggestions": "See the weakness section.", "questions_for_authors": "1. The design of the visual steering vector. What if $V_n$ is directly subtracted? What would the results be?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qpB534Xx88", "reviewer": "Reviewer_ypPL", "summary": "This paper introduces a inference-time framework called VISTA that aims to reduce hallucinations in LVLMs. The authors conduct a detailed study of token “logit rankings” across both layer depth and the temporal dimension in a generated sequence. Their analysis reveals three observations: (1) Gradual loss of visual grounding over the course of generation, (2) Early excitation of semantically meaningful tokens in penultimate or nearby layers, and (3) Hidden genuine information that the model “knows” but fails to include in its output. Building on these insights, they propose VISTA, which combines two complementary techniques: (1) Visual Steering Vector (VSV): a direction in the model’s activation space, computed by contrasting the residual streams obtained from the “positive” prompt containing the image tokens vs. a “negative” prompt that omits them. Injecting this vector at each decoding step enforces better retention of visual information. (2) Self-Logits Augmentation (SLA) aggregates token logits from earlier (penultimate) layers, where grounded or semantically rich tokens peak, and then mixes them with the final layer’s logits for next-token sampling. Experiments on four LVLM architectures (LLaVA-1.5, MiniGPT-4, Shikra, InstructBLIP) and three standard decoding regimes (greedy, beam, nucleus sampling) demonstrate that VISTA consistently outperforms baselines on several hallucination-specific benchmarks (CHAIR, POPE, MMHal) and general capability tests (MME).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We cordially appreciate your careful review and insightful questions. We are thrilled that you find our analysis __\"reveals new insights\"__., our method gains __\"strong performance\"__ and __\"can generalize\"__. See below for detailed replies.*\n\n**Q1. Regarding vision encoder’s quality**\n>VISTA focuses on hallucinations caused by parametric priors and `does not set extra requirements for visual encoder's quality`, as evidenced by its effectiveness across four widely adopted architectures. The quality requirement applies to standard public architectures, `consistent with existing methods` such as VCD, PAI, and OPERA. VISTA complements ongoing research focused on improving vision encoders.\n\n**Q2. Regarding hyperparameter tuning**\n>There are two hyperparameters: intervention strength $\\lambda$ and mixing ratio $\\gamma$. \n> * For $\\gamma$, we found $\\gamma\\approx0.3$ `consistently effective across all four architectures`. \n> * Optimal $\\lambda$ can vary by model; however, a broad range of $\\lambda$ values are effectively for each model, `as analyzed in the Synergy & Robustness subsection (p.7, line 381)` and shown in `Figs. 6, 12–14`, demonstrating VISTA's robustness across different $\\lambda$ configurations.\n\n**Q3. Regarding computational overhead**\n>VISTA is an efficient de-hallucination strategy that `introduces minimal computational overhead` and demonstrates `better efficiency than baseline approaches.` We clarify this in below:\n>* **For captioning, summarization, and open-ended generation tasks**: The textual prompt is static (e.g., \"please describe the image...\"). VISTA forwards the textual prompt `only once` w/o image to cache activations (minimal overhead). For positive case, VISTA acts alike vanilla inference which forwards the visual+textual tokens. The only difference is before generating new tokens, VISTA builds VSV via `simple vector arithmetic` (marginal overhead), and proceeds with intervention. VISTA `does not forward twice` the input tokens, and it can use KV cache of them during generation` (overhead of this is also small).\n>* **For QA tasks**: Textual prompts vary per query, potentially requiring an additional forward pass of textual tokens. However, textual tokens typically account for `less than 10% of total prompt length` compared to image tokens`. Thus, the additional computational overhead remains mild relative to vanilla inference.\n>* **Empirically**, VISTA demonstrates `better efficiency` compared to other comparable methods (see `Table 5`). We kindly refer the reviewer to Reviewer 5sMs (Q2) for an extended efficiency comparison.\n\n**Q4. Is there \"overstuffing\" of visual details?**\n>* We'd like to first clarify a potential caveat. The visual steering vector (VSV) does not merely amplify visual details. The positive vector $V_p$ is conditioned `jointly on visual and textual tokens, capturing relevant visual-textual relationships` via attention heads, while the negative vector ($V_n$) reduces textual-only priors. Consequently, VSV `does not overstuff visual details` that might blur the query's focus.\n>* This is empirically validated on MMHal-Bench (Fig.4), where `each query specifically targets an entity or relation within visually complex scenes` where many other entities and relations are existing, and VISTA `performs well`.\n\n**Q5. Multi-query scenario**\n>As clarified in Q4, VSV is constructed per visual-query pair and is expected be recalculated per new query. However, as analyzed in Q3, constructing VSV incurs only mild computational cost due to KV caching and the relatively short length of query tokens compared to visual/system tokens. As a result, it is still efficient to use VISTA under multi-query scenarios.\n\n**Q6. Regarding extremely cluttered images**\n>Inspired by your insightful question, we conducted a new experiment where 500 images from MSCOCO are randomly selected and divided into two groups (heavy and light) according to the degree of cluttering (GPT-4o is used to rate a cluttering score for each image). Results below (on LLAVA-1.5) demonstrate that `heavily cluttered images is inherently more challenging`; yet VISTA `significantly outperforms vanilla decoding` in both scenarios.\n\n>|Cluttering degree|$C_S$↓/$C_I$↓|\n|:-|:-|\n|Heavy (greedy)|52.0 / 13.0|\n|Heavy (ours)|**26.5 / 5.3**|\n|Light (greedy)|40.5 / 11.7|\n|Light (ours)|**22.0 / 5.6**|\n\n**Q7. Could early-layer steering fix the problem?**\n>Following your suggestion, we conduct additional experiment applying early-layer steering (first 15 layers) for heavy and light cluttered images. As shown below, `steering across all layers consistently outperforms early steering`. Interestingly, the `performance gap between early steering and steering all layers is smaller for lightly cluttered images`, suggesting early layers handle much of the visual processing for easy cases.\n\n>|Cluttering degree|$C_S$↓/$C_I$↓|\n|:-|:-|\n|Heavy (early)|37.0 / 10.2|\n|Heavy (all)|**26.5 / 5.3**|\n|Light (early)|26.0 / 7.1|\n|Light (all)|**22.0 / 5.6**|", "claims_and_evidence": "1. Hallucination emerges partly because the model’s language priors overshadow the diminishing visual signals, especially in later decoding steps.\n- The token-rank analysis across layers/time shows that visually grounded tokens steadily drop in probability ranks, while hallucinated tokens rise. Figures 1-2 illustrate this trend clearly.\n\n2. Meaningful tokens reach their maximal activation in intermediate layers (e.g., the penultimate), not the final layer – which is more biased toward function words.\n- Again, the “logit lens” technique indicates that object/attribute tokens see higher probabilities earlier than the final layer, which frequently emphasizes grammatical tokens.\n\n3. VISTA mitigates hallucinations by reinforcing the missing or eroded visual cues (via VSV) and leveraging earlier layer logits (via SLA).\n- VISTA yields consistent gains in object hallucination metrics (CHAIRS, CHAIRI) of ∼40% relative improvement across multiple models and decoding algorithms. They also show smaller improvements in broad tasks (MME) and short-answer tasks (POPE), indicating that the method can generalize.", "methods_and_evaluation_criteria": "They compute a “steering vector” from contrast (with/without image tokens) and inject it in the residual streams, plus a partial ensemble of earlier layer logits to finalize the token distribution. They measure CHAIR, POPE, etc and validated against strong existing baselines like VCD, OPERA, PAI, and DoLa.", "theoretical_claims": "This work is primarily empirical and methodological. The claims are mostly validated through data and are consistent with known ideas on how Transformers store information in intermediate hidden states. There are no deep new formal proofs, but no obvious theoretical errors either.", "experimental_designs_or_analyses": "Solid coverage of standard vision–language tasks and ablation on the synergy of the two main modules. However, the approach might be somewhat complicated in real usage: “steering vectors” are computed for each image.", "relation_to_broader_scientific_literature": "They cite relevant VLM hallucination works. The references are okay but possibly missing a deeper link to linear prompt engineering or “prefix tuning” for controlling generation.", "essential_references_not_discussed": "Might not mention older works on “residual stream editing” or “concept neurons.” Possibly not critical but could be relevant.", "other_strengths_and_weaknesses": "Strengths:\n1. Intriguing token ranking analysis that reveals new insights. Thorough token-level analysis using logit lens clarifies when/how hallucinated vs. grounded tokens appear.\n2. Strong performance gains across multiple models, decoding methods, and tasks.\n\nWeaknesses:\n1. Relying on the vision encoder’s quality. If the base model’s image embeddings are poor or incomplete, the “steering vector” might not help.\n2. The method can require hyperparameter tuning for different models. For example, the injection scale hyperparameter might be sensitive; not thoroughly studied.\n3. The VSV computation can be somewhat heavy if done for each image. It needs thorough time analysis.", "other_comments_or_suggestions": "Read above and below.", "questions_for_authors": "1. How does VISTA perform if there are many more entity mentions in an image than the user explicitly queries about? Could it lead to “overstuffing” of visual details?\n2. If the system has to maintain visual grounding over multiple user queries, do we re-inject the same VSV at each turn, or might some turn-based adaptation be needed?\n3. Do you see major differences if the image is extremely cluttered? Do you rely on the visual token embeddings to be relatively stable?\n4. Could early-layer steering alone fix the problem without discretization or advanced overshadowing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates hallucination in Large Vision-Language Models by analyzing token logit dynamics during generation and proposes a training-free, inference-time framework to mitigate these issues. After rebuttal, all reviewers voted for acceptance, and the AC acknowledges the paper's valuable contributions in both analysis and mitigation of hallucination. Concerns regarding novelty, hyperparameter tuning, and computational cost were raised, and most of them have been addressed during the rebuttal period. The Area Chair agrees that the novelty of the proposed method is somewhat limited, as the concept of steering vectors has been explored in prior works on mitigating hallucination in both LLMs and MLLMs. For example, the AC noted that the proposed Visual Steering Vector method is similar to that in [1]; Therefore, the AC suggests weak accept and please include a discussion and experimental comparison with [1] in the final version, along with addressing the other points raised by reviewers.

[1] Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering, ICLR 2025.
Decision: Accept (poster)