Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
Authors: Zhining Liu
Ziyi Chen
Hui Liu
Chen Luo
Xianfeng Tang
Suhang Wang
Jingying Zeng
Zhenwei Dai
Zhan Shi
Tianxin Wei
Hanqing Lu
Benoit Dumoulin
Hanghang Tong
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Model
Multi-modal QA
Mechanistic Interpretability
Primary area: foundation or frontier models, including LLMs
TL;DR: We show that VLMs often perceive the right evidence but fail to use it, and propose a simple inference-time method that highlights evidence to improve accuracy.

Abstract: Vision-Language Models (VLMs) achieve strong results on multimodal tasks such as visual question answering, yet they can still fail even when the correct visual evidence is present. In this work, we systematically investigate whether these failures arise from not perceiving the evidence or from not leveraging it effectively. By examining layer-wise attention dynamics, we find that shallow layers focus primarily on text, while deeper layers sparsely but reliably attend to localized evidence regions. Surprisingly, VLMs often perceive the visual evidence when outputting incorrect answers, a phenomenon we term "seeing but not believing" that widely exists in major VLM families. Building on this, we introduce an inference-time intervention that highlights deep-layer evidence regions through selective attention-based masking. It requires no training and consistently improves accuracy across multiple families, including LLaVA, Qwen, Gemma, and InternVL. These results show that VLMs encode reliable evidence internally but under-utilize it, and that making such signals explicit can bridge the gap between perception and reasoning, advancing the diagnostic understanding and reliability of VLMs.

Published as a conference paper at ICLR 2026
SEEING BUT NoT BELIEVING: PROBING THE DISCON- NECT BETWEEN VISUAL ATTENTION AND ANSWER CORRECTNESS IN VLMS
Zhining Liu', Ziyi Chen', Hui Liu", Chen Luo", Xianfeng Tang", Suhang Wang", Joy Zeng", Zhenwei Dai Zhan Shi?, Tianxin Wei', Hanqing Lu², Benoit Dumoulin", HHanhang Tong' University of Illinois Stbana-Champila 'Amazon 1iu3260 wwwioiinn........ edu
ABSTRACT
Vision-Language Models (VLMs) achieve strong results multimodal tasks such as visual question answering, yet they can still fail even when the correct visual evidence is present. In this work, systematically investigate whether these failures arise from not perceiving the evidence or from not leveraging it effectively. By examining layer-wise attention dynamics, we find that shallow layers focus primarily on text, while deeper layers sparsely but reliably attend to localized evidence regions. Surprisingly, VLMs often perceive the visual evidence when outputting incorrect answers, a phenomenon we term "seeing but not believing" that widely exists in major VLM families. Building on this, we introduce an inference-time intervention that highlights deep-layer evidence regions through selective attention based masking. It requires no training and consistently improves accuracy across multiple families, including LLaVA. Qwen, Gemma, and Intern VL. These results show that VLMs encode reliable evidence internally but under- utilize it, making such signals explicit can bridge the gap between perception and reasoning, advancing the diagnostic understanding and reliability of VLMs.
INTRODUCTION
Vision-L.anguage Models (VLMs) (Gemini et al. 2023 Achiam al.) 2023 Grattafiori et al. 2024; Bai et al. 2025) have recently achieved remarkable progress across wide spectrum of multimodal tasks that require reasoning over both images and text. Among these tasks, Visual Question Answering (VQA) has been central task for evaluating VLMss ability to integrate and reason over visual and linguistic information (Kim et al. 2025 Zhang et al. 2024).
Despite these advances, recent studies have highlighted persistent and puzzling gap between the availability of visual evidence in an image and the correctness of VLM answers (Wang et al. 2024 Kamoi al., 2024). Specifically, models often overlook, ignore, underutilize the erucial visual information, leading to errors even when the correct evidence is present in the image. This raises fundamental question: Do VLMs fail in those cases because they cannot perceive the visual information, or because they fail to effectively leverage it in reasoning and generation?
Prior works have begun probe this disconnect. Tong et al. (202)) showed that multimodal LLMs/VLMs sometimes ignore critical visual details, treating vision as secondary to language. More recent analysis goes further, suggesting that VLMs can be misled by different questioning methods and give incorrect answers to some questions even though it can understand the visual content (Liu al. 2025 Zhang et al. 2025). These findings raise an intriguing possibility: the problem may not lic solely in "blindness" to images, but rather in what happens after the model has already seen them. Notably, existing studies often attribute this issue to the model's overall lower attention to image tokens compared to text tokens (Liu et al. 2025 Chen et al. 2025), with few studies explore this phenomenon through the lens of the model 's internal mechanisms.
In this work, take systematic step toward unpacking following intriguing questions for un- derstanding VLMs' visual evidence utilization: (i) How do models balance and transition between textual and visual information across layers? (ii) Which layers most critical for grounding answers

---

Published as a conference paper at ICLR 2026
in the correct evidence? (iii) Can we design interventions help models actually use what they see? Our analysis reveals several interesting findings:
L Layer-wise transition. Shallow layers are text-focused, while deeper layers progressively shift toward images. This sequential process resembles how humans first read the question, then turn their eyes to the picture.
Deep-layer visual grounding. Deep layers do not scatter their attention broadly; instead, they concentrate on localized regions that correspond to key evidence, functioning like a spotlight that cuts through irrelevant clutter.
Seeing but not believing. Perhaps most intriguingly, deep layers often lock onto the correct evidence even when the final answer is wrong. The model sees, yet fails to believe. This paradox suggests that the bottleneck lies not only in perception but also in how perceived evidence is carried forward into reasoning and generation.
These findings motivate a practical intervention: leveraging deep-layer attention as a signal for guiding models toward more effective use of visual evidence. To this end, we propose an attention based visual augmentation that highlights the evidence regions attended to by the visual grounding layers, amplifying signals that would otherwise remain buried in its internal representations. This simple yet effective strategy requires no additional training, applies across architectures, and consistently improves answer quality. These results suggest that VLMs already possess latent capabilities for grounding answers in the right evidence, but require targeted elicitation to realize this potential.
In summary, main contributions are threefold. (i) Novel Analysis. We investigate how different attention layers in VLMs process mixed textual and visual inputs, revealing that deep-layer attention reliably identifies the correct evidence regions even when the final answer incorrect. (ii) Practical Algorithm. Building on this insight, we introduce an attention-based augmentation method that highlights the evidence regions identified by the model itself, guiding VLMs better utilize visual information for factual answering. (ii) Empirical Study, We conduct extensive experiments across multiple VLMs and tasks, demonstrating that our method consistently improves answer accuracy and validating both our analysis and the practical effectiveness of attention- augmentation.
2 DISSECTING TEXT AND VISUAL ATTENTION DYNAMICS IN VLMS
To better understand how VL.S process and utilize visual evidence, we conduct a systematic analysis of their internal attention behaviors. Our goal is to uncover not only whether VLMS perceive the evidence but also why they fail to leverage it for accurate answers. To this end, we organize the section around 4 research questions (RQs): RQI analyzes attention transitions between text and image tokens across layers: RQ2 examines layer-specific roles in grounding visual evidence; RQ3 investigates whether models can attend correct evidence when producing wrong answers; and finally, RQ4 explores reasons for the disconnect between attention to evidence and answer correctness.
2.1 RQI: How DOES ATTENTION TRANSITION BETWEEN TEXT AND VISUAL TOKENS?
Setup. Recent work has shown that VLMs, on average, assign significantly less attention visual tokens compared to textual ones (Chen et al. 2025). Here we take step further by examining how this imbalance evolves across layers. For each layer, we compute the Relative Attention per Token (RAPT), defined as the ratio of dection-average attention mass per token to the input-average value !]] This metric captures how much attention each token type receives on a per-token basis, rather than how attention is distributed as whole. The results, shown in Figure [ allow us to trace how the model shifts its focus between linguistic and visual inputs during inference.
Layer-wise Modality Attention Transition. Our analysis shows that although image tokens consis- tently receive less attention per token than text tokens, there is a clear modality shift across depth. Early layers focus overwhelmingly linguistic structure (the apparent crossing of curves in Figure iis due to differences in y-axis scaling; text attention values remain larger in magnitude than image attention throughout). As depth increases, however, the balance gradually shifts, with deeper layers allocating relatively more attention to image
the question text, reflecting the model's initial parsing of
Forr example, RAPT of 0.6 means each token in that section receives 60% of the input-average attention. Results shown here from the LLAVA-1.5-7B model, with similar trends across other models and datasets.
2

---

Published as a conference paper at ICLR 2026
Figure 1: Relative attention per token (RAPT) (y-axis) to text tokens (red) and image tokens (blue) across model layers (x-axis). While early layers strongly emphasize text, deeper layers pro- gressively increase attention to images, showing a sequential transition from linguistic parsing visual grounding within single-token infer- ence. Similar trends hold across different VLMM families, please see Appendix [ more results.
Attention Per Token (RAPT)
Attention Per Teken (RAPT)
30
Image Relative
0.6
Text Relative
0.4
20
0.2
10
10
# Layer Index
tokens. This layer-wise transition indicates that textual and visual information not processed in parallel, but rather sequentially, with vision playing a stronger role at later stages inference.
2.2 RQ2: WHICH IMAGE REGIONS DO DIFFERENT LAYERS ATTEND TO?
Setup. To further unpack the role of different layers, we analyze not only how much attention is allocated to image tokens, but also where this attention is directed. We begin with visualized case study to intuitively illustrate which regions of the image attract attention across layers. Figure 2 presents three images from VQA pairs and the corresponding attention distributions from different layers. For clarity, we highlight the ground-trui truth evidence regions with red bounding boxes.
Image
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Image
Figure 2: Visualization of image attention across different layers of LLaVA--.5-7BB Red bounding boxes denote ground-truth evidence regions. While shallow layers exhibit global weak attention, deeper layers consistently focus on localized regions that align with the relevant evidence.
Deep-layer Visual Grounding. Our observations show that the first layer distributes attention almost uniformly across all image patches, showing no particular focus. As shown in Fig. [ subsequent shallow layers (e.g., layers 6-11) allocate little attention image tokens overall, without exhibiting any distinct spatial patterns. Interestingly, deeper layers (e.g.. layers 16-26) display sparse yet highly concentrated attention, consistently highlighting regions aligned with ground-truth evidence?] This suggests that deep layers function visual grounders, filtering irrelevant content and selectively attending to the evidence necessary for answering the question.
2.3 RQ3: Do VLMS PERCEIVE VISUAL EVIDENCE WHEN GIVING WRONG ANSWERS?
Setup. Building on our findings about the critical role of deep layers in visual perception, a natural follow-up question arises: VQA errors always caused by the model's failure perceive visual evidence? To answer this, we conduct both qualitative and quantitative analyses. For the case study, examine representative VQA samples where the model produces incorrect answers. For the dataset-level study, we leverage the VisualCOT dataset (Shao et al.) 2024), which provides human-- annotated visual evidence regions, to statistically compare attention evidence versus non-evidence tokens across layers, conditioned on whether the model's predictions are correct or incorrect.
"In some images, isolated non non-evidence patches receive high attention, consistent with the attention sink phenomenon (Darcet et al. 2024 Kang al. 2025). This does not affect the overall grounding behavior, and Section 3.2 introduces sink-token filtering strategy remove such noise.
3

---

Published as a conference paper at ICLR 2026
Question: What type of meeting is It?
Question: Who manufictured thes nano recetver?
Question: What does the sign war of?
Logitech
Entering
Crossing
Figure 3: Qualitative examples of "seeing but not believing" with LLaVA-1.5-7B model. Each case shows the input image (left) and the average attention map of the late 50% layers (right). Deep layers attend to the correct evidence regions (red boxes), but the final answers are still incorrect due We provide more qualitative examples in Appendix C.7
generation failures.
Image Type
Non-Evidence
Correctness
False
LLaVA-1.5 ----h
Qwen2. 5-VL- VL-7B-Instrucct
Gemma-3-44al
Relativ Attertion
2000%
1000%
500%
1000%
500%
0%
0%
10
20
30
10
20
10
20
30
Layer Index
# Layer Index
# LLyer Index
Figure 4: Relative attention to the evidence/non-evidemee image tokens (y-axis) across the layers (X-axis) for different VLM families. Deeper layers pay much greater attention to crucial evidence (blue lines) in the context, even when VLM responds incorrectly (dashed lines). Best viewed in color.
Case Study. Figure 3 illustrates three typical error cases, including false rejection (refusing to answer despite evidence being present). hallucination (answering with non-existent content), and partially correct responses. In all these examples, the deep layers of the model correctly attend the relevant evidence regions, yet this perception fails to translate into correct final outputs. We term this phenomenon "seeing but not believing." " highlighting that attention evidence does not necessarily guarantee grounded and correct answers.
Seeing but Not Believing. To verify that this phenomenon is not limited to anecdotal cases, we perform a larger statistical analysis using VisualCOT. As shown in Figure[4] deeper layers consistently allocate more attention to ground-truth evidence than non-evidence tokens. Even when the model produces incorrect answers, its internal attention still concentrates relatively more on the correct evidence regions, albeit with weaker overall magnitude compared to correct predictions. This "seeing but not believing" phenomenon indicates that while the model often perceives the right visual cues, the subsequent reasoning and generation processes fail to effectively integrate this attended information into the final response.
2.4
RQ4: WHY DOES "SEEING BUT NOT BELIEVING" HAPPENS?
The above findings suggest that the visual bottleneck in VLMs is not only perceptual but also cognitive. Although deep layers focus on the correct evidence, models can still fail to translate this perception into factual answers. In this section, we build on our observations and recent literature to discuss two perspectives on the underlying reasons for this counter-intuitive phenomenon.
Textual Information Dominance. A growing body of work shows that VLMs frequently exhibit a strong preference for linguistic signals, placing "blind faith in text" even when such signals conflict with visual evidence (Ailin et al.) [2025] Kang-il al., [022]). One explanation points the architectural imbalance in current VLMs, where a large language model backbone is paired with a relatively small visual encoder, reinforcing the dominance textual patterns over visual grounding (Shi et al. 2024 Cong et al., (225)] (Shengbang et al., 2024). This textual bias has also been linked to multimodal hallucinations: as generation unfolds, the reliance on visual inputs diminishes and language priors increasingly dominate, producing outputs that are linguistically fluent but visually ungrounded (Alessandro et al. 2024; Lanyun et al. 2024 Nanxing et al. 2025). Notably, the second failure case in Figure Jaligns aligns with this line of evidence: the LLM backbone hallucinates connection between "logitech" " and "nano receiver". likely due to their frequent co- co-occurrence in the training corpus, while the visual input fails to override this strong textual prior.
4

---

Published as a conference paper at ICLR 2026
Visual Context Under-utilization. Another interesting perspective understand this phenomenon through studies of context under-utilization in retrieval- -agievee---gmeeeed generation (RAG), where the image in VQA can be viewed as a form of context. Recent work has shown that LLMs often fail to fully exploit retrieved information, generating incorrect answers even when the necessary facts present (Garima al.] 2024] Fei et al., 2024). This issue becomes the context contains more irrelevant information (Jirui et al., 2025 Huayang et al. 2024). while emphasizing salient evidence within the context can help models make better use of the provided context (Chang et al. 2024, Mortaheb et al. (025). In our case, we note that the deep-layer visual grounding behaviors of VLMs can naturally serve as signals for evidence highlighting, guiding the model to attend to critical visual information. This insight directly motivates our method design.
pronounced when
SIMPLE INFERENCE- TIME VISUAL EVIDENCE AUGMENTATION
Building the above analysis, propose inference- method that leverages deep-layer visual grounding to address the "seeing but not believing" problem. Given a VQA input, VEA (Visual Evidence Augmentation) extracts attention from visual grounding layers, applies denoising and smoothing to form highlighting mask, and overlays it on the original image to create an augmented input. VEA is lightweight, requires no additional training, and its overview shown in Fig.
Layer Attention Map
(D Model Profiling
"Seeing and belleving?"
User Inputs
Language Models
mimis mesting.
VEA
Lv
Trallis.
Gni what of meeting
Reshape Image Grid
Grounding Lepers
Layer Visue Exidence
"Seaing, but not believing"
Traffic.
(D$33.2) Extraction
(D-$3.2) Denolsing
Highlighting
(D
Figure 5: Overview of the proposed VEA framework. Best viewed in color.
3.1 VISUAL EVIDENCE ATTRIBUTION LAYER PROFILING
Notation. We summarize the key notations of Transformer-based VLMs used in this work, referring readers to Vaswani et al. (017) for full exposition and Appendix A.3 for implementation details. Given a VL. OP, a question q. an image I, and QA prompt TQA the generated answer is g < ((TOA (I, q)). A decoder-only Transformer processes input of tokens (text and image) and generates each new token by attending to all preceding tokens. For head in layer E, the attention is (C.h) E R"" and the layer-level vector is a ( 17 Σ (e.h), where H is the number of heads. This aggregated vector summarizes how layer distributes attention across input tokens, showing which parts of the input it considers most informative at that stage.
Layer Attention Extraction and Profiling. The first step of VEA is to identify Transformer layers with the strongest evidence attribution capability. We use a small diagnostic subset of Visual- COT (Shao et al.) 2024). which provides bounding-bor annotations of evidence regions. These annotations are aligned with the vision encoder patch space to obtain token-level evidence la- bels. For each layer, we extract visual attention vector and compute the AUROC against the ground-truth labels, measuring the attribution quality of that layer. Formally, let image I be divided into m patches PI = {PI..... .........m }. each mapped to a visual token. A binary label vector yr [y...... yml {0, 1) 1)rh is defined by setting 34 = 1 if patch p. overlaps with any annotated evidence region. Denote iatar as the index of the first visual token in T (I, q). The patch-level visual at- (E)
pm E R We compute AUROC(yz, a()) as the attribution score of layer for I. Finally, we select the top 10% of layers with the high- average f as the set of visual-grounding layers LvG. which will be used for subsequent inference-time evidence attribution. We note that this profiling is one-time, model-level cost and does not need to be repeated for each question at inference time. More details are in Appendix A.3 A.3
tention vector of layer e is then az
|a (E)
We that the diagnostic does need to be large, 100 examples sufficient obtain stable results.
5

---

Published as a conference paper at ICLR 2026
3.2 INFERENCE-TIME VISUAL EVIDENCE ATTRIBUTION
Attention Extraction. At inference time, VEA leverages the VLM's internal attention representations to highlight the most relevant evidence regions in the image. We note that, as shown in the analysis section, the evidence-reading layer's visual grounding pattern appears when generating the first answer token. Therefore, at this attention extraction step, we can perform only an efficient single- token forward pass rather than generating the full answer. Given the of gisuul-grounding layers Lvg identified in the profiling stage, we compute for each image I a patch evidence score vector (2) R
ez E R over its patch set PI (Pi,.... .......................... J. For layer & LvG, let az (f) [a am
denote its normalized patch-level visual attention vector. The evidence score e; for patch pt is then defined as the average attention weight assigned by layers in Lvg:
ez = [en, E2 ................................ Cm]
6
Σ EECve
a
I=,...,,m. m.
LvG
Attention Mask Denoising. We reshape the patch evidence score vector er E IR into a two- dimensional grid R # to obtain 2-D visual evidence mask. However, we note that (also can be observed in Fig. 2 and 3 the raw mask often contains spurious high-valued patches scattered in
regions unrelated to the true evidence. Those high-attention patches stem from the attention sink
phenomenon (Kang et al. 2025 Darcet et al.) 2024): certain visual tokens exhibit abnormally large activations in specific hidden-state state dimensions inherited from the language backbone, which causes them to attract disproportionate attention despite carrying little or no semantic content. Unlike genuine evidence regions that form spatially coherent clusters tied to the query, these sink-driven patches appear as isolated artifacts. To suppress those sink tokens, we apply an additional neighborhood-bue bassed filtering step. Let e denote the score of the patch grid location (i. and let (i. denote its 3 x 3 neighborhood excluding (i, We update ei,j by comparing it against its neighbors:
e ep.g if (L.))EN((.))
X max Ep.y
WhJI
(P.))EN((.))
(2)
Ci.i
otherwise,
where À is a multiplicative threshold controlling the strictness of noise suppression. We set À 10 in our experiment, i.e., a patch is regarded as noise and replaced by the local average if its score is more than one order of magnitude larger than all of its neighbors. This denoising step yields cleaner and more spatially coherent evidence mask, which better highlights the true evidence regions.
3.3 ATTENTION-GUIDEA VISUAL. EVIDENCE HIGHLIGHTING
Highlight Mask Smoothing. While neighborhood filtering suppresses isolated outliers, the resulting token-level mask e' often introduces unnatural, mosaic-like artifacts when applied to the pixel-level image, leading to sharp local fluctuations that may hinder the VLM's understanding of image content. To further enhance spatial coherence and ensure that contiguous evidence regions form smooth clusters, we apply Gaussian smoothing step. This operation distributes attention scores more evenly within local neighborhoods, reducing noise while preserving the overall evidence structure. Formally, let G E & be a Gaussian kernel with standard deviation a. The smoothed mask é is obtained by convolving e' with Go: C,, Σ Σ Go(p.9)ei-ns- p=-ro=-
where r = |k/2] and Ep Gp g) = 1. In our implementation, the effective or is set by multiplying a hyperparameter smooth strength a E [o, 1] with the shorter side of the image in pixels, ensuring that smoothing naturally adapts to different resolutions. Detailed experiments and discussions are provided in Section 4.2 validating the benefit of adaptive smoothing.
Visual Evidence Highlighting. Given the refined mask E I #xW, we next guide the model by directly emphasizing evidence regions. We blend the mask with the original RGB image I e x x x3 so that high-attention areas are preserved and low-attention areas visually down-weighted. This emphasizes critical evidence while reducing irrelevant content. Concretely, the augmented image I is constructed by attenuating low- oow-attention regions, controlled by E [0, 1]:
1.j. (a (1 a)ees) Injjes

---

Published as a conference paper at ICLR 2026
for each pixel (i.j) and channel c. Large é. (evidence) keeps pixel values close the original, while small ē. (non-evidence) darkens the region toward a. This simple augmentation makes evidence more salient without retraining, steering inference toward highlighted regions. We validate the effect of different highlight strength choices in Section 4.2 This concludes the description of VEA, we next evaluate its effectiveness across diverse VLMS and tasks.
EXPERIMENT RESULTS AND ANALYSIS
In this section, we conduct systematic experiments across cight VLMs from four different families with varying sizes and on four visual evidence retrieval-based VQA tasks from diverse domains to examine the following research questions (RQs): RQ1: How does VEA perform in improving VLM response quality and accuracy? RQ2: To what extent do the visual evidence regions highlighted by VEA align with the ground-truth visual evidence? RQ3: How robust is VEA under different types of image noise perturbations? RQ4: How do different parameter choices and the inclusion or removal of specific modules influence the performance of VEA?
4.1 EXPERIMENTAL SETUP
Datasets and Metrics. The primary goal of our experiments is to test whether VEA and related baselines help VLMs attend to fine- one-grained visual evidence for more accurate responses. For this purpose, adopt four siddence-based VQA datasets from the VisualCoT benchmark (Shao et al. 2024): InfoVQA (Mathew et al. 2022), DocVQA (Mathew et al. 2021), SROIE (Huang et al., 2019), and TextVQA (Singh et al. 2019). These tasks require models extract localized information from images, such as text snippets in natural scenes or documents, and provide cuman--nnnottte evidence that allows us to directly measure both QA accuracy and the quality of visual grounding. We use greedy decoding for deterministic outputs, reporting Exact Match and Token-level F1 as QA metrics, and AUROC and NDCG@al all as evidence attribution metrics. Please see more details in Appendix [A]
Models and Baselines. We evaluate the effectiveness and generality of VEA using the most recent series from four popular VLM families: ALLVV--Nex (Liu al. 2024). Qwen2.5VL (Bai et al. 2025). Gemma3 (Tam et al., 2025). and Intern VI3.5 (Wang et al. (025). We note that VAL and Intern VI.3.5 exhibit memory leakage when extracting attention, so we use Qwen2.5VL. as a delegate model for image augmentation. For comparison, we also include several inference-time text/image augmentation baselines designed to enhance visual information utilization. (i) Instructioning is a simple baseline for evaluating whether prompting alone is sufficient; it explicitly instructs the model to attend to visual evidence. (ii) CGR (Liu et al. 2025) employs a two-step reasoning process in which the model first extracts detailed information from the image and then generates answers from this intermediate representation. (iii) VAR (Liu et al.) 2025) uses attention scores from the final layer apply binary mask that highlights salient regions. (iv) AGLA (An et al. (025) adopts a ModdCAM-basse (Selvaraju et al., (017)) approach that masks irrelevant regions and ensembles outputs from the original and masked images strengthen visual grounding.
4.2 RESULTS AND ANALYSIS
RQ1: VEA consistently improves VLM's visual grounding. Table I reports the average per- formance of different visual evidence augmentation methods across VLMs from 4 latest model families. We observe that all augmentation methods yield improvements over the BASE models, confirming the effectiveness of explicitly enhancing visual grounding. Among the baselines, CGR, VAR, and AGLA consistently outperform simple prompting (INST), with AGLA showing the strongest gains overall. Nevertheless, our proposed VEA achieves the best results across both metrics and all model families. Specifically, it delivers an average improvement of +5.67 points (up to + +11.1) 1) in Exact Match and over +6.83 points (up to +17.3) in Token F1 compared to the base models, while also achieving the lowest average rank (1.12 and 1.22) among all baselines. The gains of VEA are especially pronounced on smaller-scale models (e.g.. I.L.VA-BB and Gemma-4B), suggesting that VEA is particularly effective in compensating for the weaker visual grounding capability of smaller VLMs, while still providing consistent benefits for larger models. These results demonstrate that VEA provides more robust and consistent gains across diverse VLM architectures and scales, establishing it as a strong and generalizable approach for improving visual evidence utilization.
RQ2: VEA accurately highlight visual evidence. To validate the quality of VEA evidence at- tribution, we evaluate how well the extracted evidence aligns with human annotation. Following Section 3.1 where we compute token-level ground-truth evidence label vectors yr E {0, 1 1) and
7

---

Published as a conference paper at ICLR 2026
Table I: Main results of applying visual evidence augmentation methods to 8 VLMs from 4 latest
series of LLaVA, Qwen, Gemma, and Intern VL families. Due to space limitation, we report the averaged results on 4 visual question answering tasks. For each metric, we also report the average rank of each method over all tested tasks and models. Detailed results can be found in Appendix C
LLa VA-XXT
Qwen2.5-VL
Gemma3
Intern VL3.5
Avg.
Metric
Method
7B
13B
7B
32B
4B
27B
8B
14B
Rank ())
Exact Match
BASE
38.5
49.4
73.4
69.3
56.6
69.3
79.3
79.3
5.38
INST
38.8.00
50.2
73.9
69.0kam
56.0vam
70.2
79.2.am
78.8.0
5.47
CGR
45.4 52.7ua3)
76. l626
73.0us.)
60.4.33
72.3
82.403.)
81.7G24
3.09
VAR
42.7042) 51.5ma2
76.4.50
70.4
58.0+1a)
73.4
80.2
80,4...0
3.44
AGLA
46.4w79)
53.4:-m
77. 77.9
73.8
60.4w33)
74.2
82.3
82.3
2.50
VEA
49.6
54.1
78.4.4.)
75.8
61.2.
75.3.
83.2
82.9.3.
1.12
BASE
33.3
53.8
77.7
69.8
50.5
57.7
79.6
69.4
5.53
Token F1
INST
38.4.5.)
54.4.007
77.6.0.1
71.2.
51.9.14
58,3
80.0g.4:
69.0u..4
CGR
41.0+1m)
56.8
80,9
74.8..0)
54.7c4m
61.3
84.5c-1s)
70.5
3.44
VAR
45.7
55.7uam
79.0
73.9.4.)
56.4uss
61.3
84.2
70.3
3.22
AGLA
45.1oms
58.7058
81. 4449
75.2
55.9
61.403.
83.2
70.8
2.31
VEA
50.6.1235
(5:4)
82.1.144)
76.9.7..
57.8.
62.3
85.8 (6.2)
71.7022.
1.22
Table 2: Comparison of visual evidence attribution accuracy (token-level AUROC and NDCG @ all) of different attribution methods. Methods denoted by two percentages are static baselines that use fixed span of layers compute visual token evidence scores (e.g., T-50% 100% uses the late 50% of layers as visual grounding layers). Similar to Table we report the average results of 4 tasks due to space limitation, full detailed results can be found in Appendix C
Evidence
LLaVA
VL
Qwen
Qwen
Gemma
Gemma 27B
Avg. Rank (L)
Attribution
AUROC
AUROC
AUROC
XDCG
AUROC
AUROC
Las L0%--100% 100%
75.9
47.2
76.3
47.2
68.5
41.7
57.0
33.0
59.5
35.5
61.8
36.4
4.33
4.42
L0%
68.2
43.2
73.1
46.4
59.4
34.2
51.3
31.9
56.5
34.3
55.4
34.9
5.67
5.67
La0% 100%
78.0
54,5
76.9
50.5
79.5
58.1
67.6
43.3
65.9
43.7
68.1
43.6
2.88
2.83
VAR
70.8
45.1
72.1
44.0
75.2
54.1
65.7
39.8
51.2
33.3
58.2
36.8
4,92
4,88
AGLA
80.2
57.2
81.1
56.6
77.7
55.4
75.1
51.3
68.3
44.5
73.8
49.0
2.21
2.21
VEA
83.6
63.5
84.4
63.5
85.2
68.6
79.1
58.4
80.0
59.9
81.2
60.1
1.00
1.00
measure their alignment with visual attention vectors er R" using AUROC and NDCG, Table 2 compares different attribution strategies. Static basclines that aggregate attention from fixed spans of layers show that later layers generally provide stronger visual grounding than earlier ones, as evidenced by the higher scores of L50% over 50%%-0%- However, such static choices are still suboptimal compared to adaptive approaches. Among adaptive methods, AGLA outperforms VAR, but both are consistently outperformed by proposed VEA, which achieves the best performance across all model families and metrics. Overall, the absolute accuracies are relatively high (e.g.. AUROC often exceeds 80 and NDCG surpasses 60), suggesting that the internal attention signals of VL.Ms are capable of localizing visual evidence with reasonable accuracy.
Effect of Noise Injection
Effect of Low Resolution
Effect of Random Masking
0A Accurssy 180
VEA
$28.8
VEAC (L47)
vel VEA
VOA
w/
V.VAA
VEVA CESN STO
(-178.0m)
VVA
20-
$18.6
20
20
L+226
0%
20%
40%
6d%
0%
50%
35%
00%
55
10%%
20%
50%
Noise
Resolution Reduction
Mask Rate
Figure 6: Robustness of VEA against various types of visual perturbations.
RQ3: VEA is robust to visual noise and corruptions. In real-world applications, visual inputs are often corrupted by noise, low resolution, or occlusion, making robustness critical requirement for VLMs. To validate the robustness of our method, we evaluate the effect visual evidence augmentation under three perturbation settings. (i) Noise Injection: additive Gaussian noise with varying strength, where 100% corresponds to pure noise. (ii) Low Resolution: downsampling the image by number of pixels, e.g., a 90% reduction results in 10% of the original pixels. (iii) Random
8

---

Published as a conference paper at ICLR 2026
Masking: randomly masking out 2% of visual patches. We report the exact match accuracy of LLaVA-NeXT-7B on the TextVQA dataset in Figurel6, The results show that the base model degrades sharply, while VEA consistently improves robustness across all types and levels of perturbations. Even under extreme conditions such as 60% noise or 30% random masking, it still delivers improvements of +16.4 and +55.8 points, corresponding to relative gains of over 110% and 220%. These results indicate that the evidence-driven signals elicited by VEA provide strong guidance, enabling the model maintain accurate reasoning even when raw visual inputs are severely degraded.
Exact Match
Taken F1
Table 3: Ablation study results.
0.6
0.7
0.6
0.6
0.
0.4
0.2
0.2
-05
Setting
Exact Match
Token F1
0.0
0.6
7.0
0.4
VEA
73.4
68.1
lo
To
0.0
0.0
on
Denoise
70.9 (-2.52)
(-3.12
Highighr Smongth
0.5
0.5 Highlght Smenggh
Profiling
(-2.42)
65.3 (-2.78)
1.0
0.0
0.0
(n)
Smoothing
68.3 (-5.12)
62.8 (-5.27)
Figure 7: Parameter analysis of VEA.
RQ4: Effect of different parameter choice. We analyze two parameters of VEA: the highlight strength a, which controls how strongly non-evidence regions are suppressed (larger values yield darker backgrounds), and the smooth strength a, which determines the degree of spatial smoothing applied to the evidence mask. To ensure consistent smoothing across images of different resolutions, the effective kernel size is set as a multiplied by the shorter image side. As shown in Figure 7 we observe that: (i) our method is overall robust wide range of parameter choices; (ii) overly strong highlighting removes too much visual context, while the absence of smoothing also hinders VLMs' understanding of the image. Balancing these factors, we set both and a to 0.5 in all experiments.
RQ4: Ablation Study Table 3 reports the effect of removing different components from VEA. We find that each component contributes positively to the final performance: removing the denoising step or the profiling step results in moderate drops (about - 2.5 EM and -3.0 F1), while removing the smoothing step leads to the largest degradation (-5.1 EM and - 5.3 F1). These results confirm that denoising, profiling, and smoothing are all necessary for maximizing the effectiveness of VEA.
More Results and Analysis. Due to space limitations, we provide summarized results in the main text to highlight key analyses and insights. Please refer to Appendix A for reproducibility details, Appendix B for discussions on limitations and future directions in applying our findings in VLM reasoning, and Appendix for for complete results and additional analyses, including per-model and per-dataset QA performance, evidence attribution accuracy, and layer- wisee attention dynamics, etc.
5 RELATED WORKS
Challenges Visual Evidence Utilization. Despite significant progress on multimodal tasks such as VQA (Singh et al. 2019 Mathew et al. [020 Huang et al.) 2019 Mathew et al. (022),, VLMs often fail to fully leverage visual evidence. Studies report a strong tendency to rely on linguistic priors even when they conflict with the image (Ailin al.] 2025 Kang-il et al. 2024 Shengbang
et al. 2 leading to hallucinations and visually inconsistent outputs (Alessandro et al. 2024
Lanyun al. 2024; Nanxing et al. 2025 Cong et al., 2025). This issue is further amplified by the imbalance between the language backbone and the vision encoder (Shi et al.) 2024 Shengbang al. 2024). Similar challenges arise in retrieval augmented generation, where models often under
use retrieved evidence (Garima et al., 2024 Hexiang et al., 2024 Evgenii et al., 2024 Fei et al. 2024). Recent efforts have started addressing this broader problem by refining the visual input itself through model internal signals, including attention guided cropping (Zhang et al.] 2025) and mask generation (Liu al. 20255] An al. 2025), illustrating the potential of training free visual enhancement for improving evidence utilization.
Attention and Interpretabilityy in VLMs. Attention patterns offer an informative view of multimodal processing. Prior analyses show that shallow layers focus primarily textual tokens while deeper layers begin to attend to localized image regions, sometimes aligning with ground truth evidence even when predictions remain incorrect (Liu et al. 2025 Chen et al.) [2025] Tong et al.) (2224). These observations motivate a wide spectrum of interpretabiliit and intervention techniques, including
GradCAM style attribution (Selvaraju et al. 2017 An et al. 2025), masking based strategies (Liu

---

Published as a conference paper at ICLR 2026
et al. 2025). and inference time attention modification to mitigate hallucination (Shi et al. 2024 Alessandro et al. 2024). More work ViCrop (Zhang et al.] (025) follows this direction by using layer specific attention to identify a better crop that enhances perception. Our work extends this direction by demonstrating that aggregated deep-layer attention, once denoised and smoothed, provides a stable training-fiee -ree signal for inference-time evidence highlighting, connecting analysis with practical improvements in visual grounding.
6 CONCLUSIONS
In this work, systematically investigated how vision-language models (VLMs) attend textual and visual information, and uncovered key disconnect between visual perception and answer correctness. Our analysis revealed three important findings: attention transitions from text image across layers, deep layers act visual grounders by sparsely focusing on evidence regions, and models often "seeing but not believing", perceiving the right evidence but failing to use for correct answers. Building these insights, we introduced VEA, an inference-time method that highlights deep-layer evidence and consistently improves factual answering across models and benchmarks without additional training. Our results suggest that internal attention patterns already encode reliable visual cues, and making them explicit can help bridge perception and reasoning.
ETHICS STATEMENT
While our findings reveal potential weaknesses in VLMs' reasoning and grounding, we focus on diagnostic analysis and training-fre free interventions rather than deployment-ready systems, minimizing risks of harmful misuse. Like other research focused on vision language models, our work has potential social impacts, but none of which we feel must be highlighted here. All authors have reviewed the ICLR Code of Ethics and affirm adherence to its principles throughout this work.
REPRODUCIBILITY STATEMENT
We made extensive efforts to ensure the reproducibility of our results. The experimental setup, including datasets, evaluation metrics, and implementation of our method and baselines. is fully described in Section 4.1 and Appendix A Specifically, Appendix A. details dataset processing and evaluation metrics, Appendix A.2 lists all models and their Hugging Face checkpoints, and Appendix A..3 provides implementation details of VEA and baselines. Complete per-model and per-dataset results, evidence attribution scores, and layer-wise attention dynamics provided in Appendix
ACKNOWLEDGMENTS
This work is supported by NSF (2416070). The content of the information in this document does necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ArXiv:2303.08774 2023.
Deng Ailin, Cao Tri, Chen Zhirui, and Hooi Bryan. Words or vision: Do vision-language models have blind faith in text? arXiv preprint 19X 2025. URL URL http::httt///ttttttt
prg/abs/ 2503. 02199
Favero Alessandro, Zancato Luca, Trager Matthew, Choudhary Siddharth, Perera Pramuditha, Achille Alessandro, Swaminathan Ashwin, and Soatto Stefano. Multi-modal hallucination control by visual information grounding. arXiv preprint 2xXiv:2403.14003 2024. URL https:: :ttp::::////ww/ //www
arxiv.oom www.iv.orraabb//240000
10

---

Published as a conference paper at ICLR 2026
Kenza Amara, Lukas Klein, Carsten Lüth, Paul Jäger, Hendrik Strobelt, and Mennatallah El-Assady. Why context matters in vqa and reasoning: Semantic interventions for vim input modalities. arXiv preprint arXiv:2410.01690, 2024.
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Ying Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. Mitigating object hallucinations in large vision- lasguanla models with assembly of global and local attention. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 29915-29926, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint 1392 2025.
Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, Bingjie Wang, and Chenliang
Xu. Unveiling visual perception in language models: An attention head analysis approach. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4135-4144, 2025,
Chia- Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanehu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das. et al. Main-rag: Multi-agent generation. arXiv preprint 2tti::2501000322,, 2et.000332,222 2024.
filtering
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-27087, 2024.
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vims? an attention mechanism perspective on focus areas. arXiv preprint arXiv:2503.01173: 2025.
Chen Cong, Liu Mingyu, Jing Chenchen, Zhou Yizhou, Rao Fengyun, Chen Hao, Zhang Bo, and Shen Chunhua. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. arXiv preprint arXiv:2503.0644: 2025. URL hitps: www.warxiv.ooo wwwpxxiw..or//aoooooo//////bb
2503. 06886
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 326-335. 305, 2017.
Kortukov Evgenii, Rubinstein Alexander, Nguyen Elisa, and Oh Seong, Joon. Studying large language model behaviors under context-memory conflicts with real documents. arXiv preprint 2rXiv:240441603222 2024. URL/ttts:
arwiv org/ abs 2404. 16032v2
Wang Fei, Wan Xingchen, Sun Ruoxi, Chen Jiefeng, and Anık Sercan, Ö. Astute rag: Overcoming imperfect retrieval retriveal augmentation and knowledge conflicts for large language models. arXiv preprint
wwwp:wwwwww....... arXiv:2410.07176, 2024. URL httpppp:wwwww.........t/////aaaa wwtpg////aaa/aaaa 07176
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference Computer Vision, pp. 148-166. Springer, 2024.
Agrawal Garima, Kumarage Tharindu, Alghamdi Zeyad, and Liu Huan. Mindful-rag: A study of points of failure in retrieval augmented generation. arXiv preprint arXiv:2407.12211 2024. URL
https: htt////www... arxiv wiw. org/ abs 2bb../22.22221666666
Team GeminiTeam, Rohan Anil, Sebastian Borgeaud, Jean-Baptis Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 23..v23/2../11
2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The la herd of models. arXiv preprint arXiv:2407.22 1783, 2024.

---

Published as a conference paper at ICLR 2026
Tan Hexiang, Sun Fei, Yang Wanli, Wang Yuanzhuo, Cao Qi, and Cheng Xueqi. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? arXiv preprint arXhv:2440.11111, 2024. BRLh:tts::/ /www www.v. wrg/abb...r./.g...
1401.11911
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. ledar2019 9 competition scanned receipt oer and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516-1520. IEEE, 2019.
Li Huayang, Verga Pat, Sen Priyanka, Yang Bowen, Viswanathan Vijay, Lewis Patrick, Watanabe Taro, and Su Yixuan. AIr2: A retrieve-then-reason framework for woongoooooetttt answering. arXiv preprint arXiv:2410.05227, 2024. httt:tts::
www. arxiv. wwtp:vi.orr//a... 2t1.
Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. VIm-16: Region recognition, reasoning, and refinement for enhanced multimodal chain-of-thought. arXiv preprint 1rXx::2505.1619, 2025.
Qi Jirui, Fernández Raquel, and Bisazza Arianna. On the consistency of multilingual context generation. arXiv preprint arXiv:2504.00597, 2025. URL
utilization in wwtp://wwwwgg////////...oo 00597
https://wwwt www. arxiv.
Ryo Kamoi, Yusen Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, and Rui Zhang. Visonlyqa: Large vision language models still struggle with visual perception of geometric information. arXiv preprint arXiv:2412.00944, 2024.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03.3 03.3221,...3,.. 2025.
Lee Kang-il, Kim Minbeom, Yoon Seunghyun, Kim Minsung, Lee Dongryeol, Koh Hyukhun, and Jung Kyomin. Vlind-bench: Measuring language priors in large vision-language models. arXiv preprint arXiv:2406. 000:2200..8870, 2024. URL http:: //www arxiv. wttpaa////....ooo.////////////oe 08702
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In European conference on computer vision, pp. 235-251. Springer, 2016.
Byeong Su Kim, Jieun Kim, Deokwoo Lee, and Beakcheol Jang. Visual question answering: A survey of methods, datasets, evaluation, and challenges. ACM Computing Surveys, 57(10) 2025.
Zhu Lanyun, Ji Deyi, Chen Tianrun, Xu Peng, Ye Jieping, and Liu Jun. Ibd: Alleviating hallucinations in large vision-language models via bmage- decoding. arXiv preprint arXiv:2402. 18476,
2024. URL : /www//wwww.////////////ww/// arxiv,
www.wwwwwwwww.waaaaaaro/rrrr.aaa ort/abs////ooa. 2200222222222 18476.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 26296-26306, 2024.
Yexin Liu, Zhengyang Liang, Yueze Wang, Xianfeng Wu, Feilong Tang, Muyang He, Jian Li, Zheng Liu, Harry Yang. Sernam Lim, et al. Unveiling the ignorance of milms: Seeing clearly, answering incorrectly. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9087-9097, 2025.
Haolang Lu, Bolun Chu, WeiYe Fu, Guoshun Nan, Junning Liu, Minghui Pan, Qiankun Li, Yi Yu, Hua Wang, and Kun Wang. Mitigating hallucination in multimodal reasoning via functional attention control. arXiv preprint arXiv:2510. 1...110112200 10285, 2025.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
12

---

Published as a conference paper at ICLR 2026
Matin Mortaheb, Mohammad A Amir Khojastepour, Srimat T Chakradhar, and Sennur Ulukus. Re-ranking the context for multimodal retrieval augmented generation. arXiv preprint arXiv:2501.04695, 2025.
Hu Nanxing, Duan Xiaoyue. Zhang Jinchao, and Kang Guoliang. Enhancing visual reliance in text generation: A bayesian perspective on mitigating hallucination in large vision-language models. arXiv preprint arXiv:2505.19406 2025. URL hhtps:tt/////c.... //w::://wwww... wrw.v.og...o..........oo...o.oo. /bs/ 2505.19499
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626 2017.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang. Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for thain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612-8644 8642. 2024.
Tong Shengbang, Liu Zhuang, Zhai Yuexiang, Ma Yi, LeCun Yann, and Xie Saining, Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.00200 2024. URL httts::////..... /www www.va org/
06209
Liu Shi, Zheng Kecheng, and Chen Wei. Paying more attention to image: A training-free method for alleviating hallucination in Ivlms. arXiv preprint arXiv:2407.21771, 2024. URL https
// www. arxiv. wwtg/raa/////e.e .1771
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8317-8326, 2019.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392-75421, 2024.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. /8265, 2025.
Yana Wei, Liang Zhao, Kangheng Lin, En Yu, Yuang Peng. Runpei Dong, Jianjian Sun, Haoran Wei, Zheng Ge, Xiangyu Zhang, et al. Perception in reflection. arXiv preprint arXiv:2504.07165, 2025.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Milms know where to look: Training-free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502.17422 2025.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. -ision-language models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(8):5625 5644, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A
Reproducibility Details
AA11
Datasets and Metrics
A.2
Models and Baselines
A.3
VEA and Baselines Implementation Details
B
Additional
Discussions
B.1
Usage Artifacts and AT Assistants
B.2
Limitations and Potential Solutions
B3
Future Directions in VLM (Agentic) Reasoning and Beyond
C
Additional Results and Analysis
Additional Evaluation on Multi-Turn and Multi-Image Reasoning
Evaluation on Global-Context Reasoning Benchmarks
Multi-Round Salf-Validation with VEA
When attention is reliable.
Discussion on Discrepancies with Existing Findings
Evaluation the Effect of Evidence-Layer Selection
Full Question Answering Results
Full Evidence Attribution Accuracy Results
Full Layer-wise Attention Dynamics Visualization
A REPRODUCIBILITY DETAILS
A.I DATASETS AND METRICS
Datasets. The primary goal of our experiments is to test whether VEA and related baselines enable VLMs to attend to and exploit fine-grained visual evidence for more accurate responses. To this end, the datasets should evaluate both model's ability to extract localized information from images during answer generation and its accuracy in grounding answers to visual evidence. We adopt four evidence-based VQA datasets from the VisualCoT benchmark (Shao al. 2024): InfoVQA (Mathew et al. 2022). Doc (Mathew et al., 2021]. SROIE (Huang et al. 2019). and TextVQA (Singh al. 2019 These tasks require models identify fine- inn-grained content, such as text snippets in natural scenes or documents, which directly supports our objective. Importantly, VisualCoT provides pixel-level evidence annotations the form of bounding boxes for each VQA pair, enabling quantitative evaluation of evidence attribution in addition to standard QA accuracy.
Metries. We use greedy decoding for deterministic outputs. QA performance is measured by Exact Match (EM) and Token-level FI. Let Å denote the set of predicted tokens and A the ground-truth tokens:
EM(Å, A) I(Á = A), FI(Á, 2 JÅn
[ + [A]
For evidence attribution evaluation, we first align the bounding-box annotations with the vision encoder patch space. Given an image divided into m patches P {pI...., .m.. we define the binary ground-truie evidence label vector as
yz [y ....... Vnl T E 10.
where y; = 1 if patch p, overlaps with any annotated evidence region and y, = otherwise. With predicted evidence scores Pr = [ [pi..... Pm [0, 1|mh AUROC is
AUROC(YI. PZ)
Σ
Σ
I(f.
P,),
P
VEP- jEP-
14

---

Published as a conference paper at ICLR 2026
where p- = {il y/ 1) and p = = y 0}. NDCG@all evaluates ranking quality:
2 1
DCG@all
DCG@all
NDCG@all IDCG@all
log,
where (i) is the index of the i-th ranked patch according to P:. and IDCG is the DCG of the ideal ranking. Higher EM, FI, AUROC, and NDCG @ all indicate better QA performance and stronger evidence attribution.
A.2 MODELS AND BASELINES
We evaluate the effectiveness and generality of VEA using recent models from four popular VLM families: LLaVA-Next (Liu al., 2024). Qwen2.5VL (Bai et al., (025), Gemma3 (Team al. 2025), and VL3.5 (aang et al. 2025). For reproducibility, we list all models with their publicly available checkpoints on Hugging Face All experiments are conducted on single NVIDIA A100 GPU with 80GB memory, implemented using the trans foreess and Py libraries under memory leakage when extracting raw attention maps during inference. Specifically, when using the configuration attn _ep_impllee Camplemmntaatcmmm a the 7B/8B models in these families cause out-of-memory eπrоrs on an 80GB GPU under bfloat precision. To address this issue, we employ Qwen2.5VL-7B as delegate model for VEA. This choice is motivated by its relatively efficient inference speed and stable evidence attribution quality. Concretely, we run VEA on Owen2.5VL. to perform visual evidence attribution and generate augmented images, which are then fed into the original models for answer generation.
6 mixed precision. We note that both VA-Next and Intern exhibit
eager' to obtain layer-wise attention outputs, even
For baselines, we include several inference-time augmentation methods that aim to improve visual information utilization without additional training. INST is prompting baseline that explicitly directs the model to focus on visual evidence, testing whether prompting alone suffices. CGR (Liu et al. 2025) employs two-step reasoning process where the model first extracts detailed information from the image and then generates answers based on this intermediate representation. VAR (Liu et al. 2025) applies a binary mask derived from final-layer attention scores to highlight salient regions. Finally, AGLA (An et al.) (025) adopts a GradCAM-base (Selvaraju et al., (017)) approach that masks irrelevant regions and ensembles outputs from the original and masked images reinforce visual grounding.
A.3 VEA AND BASELINES IMPLEMENTATION DETAILS
Given a VLM +, question q, image I, and QA prompt template TQA, we obtain the generated answer g for the question by combining the image and question as input: g +- D(TQA (I, q)). We use the following template as the base prompt for VQA:
Base prompt template Tox for image-based VQA
Directly answer question based image, no explanation needed. image contain any evidence. output "I cannot answer based on the given image." Image:: [1mage] Question: [question]
VEa Implementation Details. We first input the image-question pair into the model with the base prompt TQA and extract its attention maps over the input sequence. At this stage, we perform only a single-token forward pass rather than generating the full answer, making the process more efficient than captioning- based approaches such as CGR (Liu et al. 22025). As described in Section [.1] we then conduct profiling step identify the Transformer layers with the strongest evidence attribution
LLaVA-1.5-13B Law.f///waannilttas/////gggaeeea
Gemma-3-4B.
Gemma-3-27B:
/
Incem\l.3.5.44//htttps.
15

---

Published as a conference paper at ICLR 2026
capability. Specifically, use 100 examples from the TextVQA dataset diagnostic subset, compute the AUROC between the patch-level attention scores and ground-trume evidence labels, and select the top 10% of layers (rounded up) with the highest average AUROC scores. Table reports the average attribution quality of all layers and the subset of selected -isual-grounding layers LvG for representative models.
Table 4: Profiling results of visual-grounding layers LvG for representative models. We report the total number of layers, the average AUROC across all layers, and the statistics of the selected top layers with the strongest evidence attribution capability.
Full Layers
Visual Grouding Layers (Lvg)
Model ID
#laayers
Avg. AUROC
#Layers
Avg. AUROC
Layer IDs
llav--hf/llava- 1.5--b-
32
83.98
4
92.13
(14,15.17.19)
Ilav-hhflllava- 1.5---1-5---hhh
85.96
92.16
(13,14,15,16)
Qwen/Owen2.5-Vi1 --1-7-.----
80.07
89.09
[18,22,24]
(ecmn(Qwe2..-VV--22B--m
73.08
(49.500511.2.53.55566)
google/gemmma-3-4br
65.40
80.32
(17,19,21.23)
google/gemma---22 3-27bbit
84.70
(35.37740
Once the set LvG is identified, we aggregate attention signals from these layers to construct the visual evidence attribution map. To improve robustness, apply denoising and smoothing operations with hyperparameters highlight strength
0.5 and smooth strength o = 0.5, following the discussion in Section 3.3 and Section 4.2] The resulting attention map then overlaid onto the input image, highlighting the most relevant regions. Finally, we construct an augmented prompt TYEa that explicitly instructs the model to focus on highlighted evidence regions when generating answers.
Prompt template Tyex for VQA with VEA "EA-augmentee Images
Directly answer the question bused the image, no explanation needed. the image does not contain any relevant evidence, output cannot based the given image." Only words from the picture, especially those in the highlighted region, answer the question. Iragee | [mmage] Question: iquestioon) oon
Baseline Implementation Details. For baselines, we include several inference-time augmentation methods that aim to improve visual information utilization without additional training. INST is implemented by replacing the base QA prompt with an augmented version that explicitly instructs the model attend visual evidence when answering questions:
Instructioning prompt template
the imuge generating your answer. the image does not contain any relevant evidence, output "I cannot based on the given image." Image: (image) Question: (question)
Answer question
imagee.
CGR (Liu et al. 2025) follows a two-step reasoning process. In the first step, we prompt the model to produce detailed caption textual extraction of the image content using the following template:
CGR captioning prompt template
Carefully read and describe all relevant details from the image, espocially any visible or objects that may help the question. Provide concise but detailed textual description of the image content. Image: (image)
The generated description is then concatenated with the original image and question, and the model generates the final answer using the base QA prompt. VAR (Liu et al. 2025) directly uses the final-layer attention scores to construct binary mask that highlights salient image regions while suppressing less relevant ones. The original method applies the raw attention distribution without post-processing: in our implementation, we additionally apply denoising and smoothing procedure (using the same parameters as in Vea) to improve stability. AGLA (An et al. 2025) is implemented using the official codebase, which is based on GradCAM (Selvaraju al. 2017). It generates saliency map, masks out irrelevant regions, and ensembles the outputs from the original image and the masked image. We adopt the default hyperparameter settings provided in the official implementation.
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL DISCUSSIONS
B.1 USAGE OF ARTIFACTS AND AI ASSISTANTS
All models and datasets used in this study are publicly available on HuggingFace, and we adhered to their respective licenses and terms of use, limiting our work to non-commercial academic research. These models and datasets have been reviewed by their developers/creators to minimize the inclusion of personally identifiable information or offensive content and are widely adopted by the research community. We used Al tools to assist with language refinement during the writing process, the paper contains no Al-generated paragraphs. All material has been carefully reviewed ensure accuracy and adherence to ethical standards.
B.2 LIMITATIONS AND POTENTIAL SOLUTIONS.
While the proposed VEA framework demonstrates consistent improvements across diverse VLMs and tasks, several limitations remain. First, method relies on extracting attention maps from Transformer layers, which requires access to intermediate activations during inference. This may not be supported by proprietary API-only VLMs. Second, although our experiments provide strong empirical validation, attention may not always fully capture all the internal mechanisms of evidence localization, and alternative attribution signals (e.g.. gradient-based saliency or probing features) could provide complementary insights. Third, while per-model profiling proves effective in identifying visual- grounding layers, more efficient or automated profiling strategies could further improve usability, especially for large-scale or real-time applications.
B.3 FUTURE DIRECTIONS IN VLM (AGENTIC) REASONING AND BEYOND.
Attention as Active Perception. We believe that the internal attention signals of VLMs hold promising potential for broader applications in more complex multimodal reasoning scenarios. Beyond the image highlighting strategy explored this work, attention signals could be leveraged to guide targeted image manipulations that facilitate reasoning in long-horizon example, when a reasoning step requires the model to examine fine-grained details, attention could be used to dynamically crop zoom into the relevant region of the image and feed it back into the context. Such adaptive image refinement guided by the model's own signals may allow VLMs to act more like human observers, selectively allocating focus as reasoning unfolds.
agentic tasks. For
Self- Triggerigee Visual Enhancement. These mechanisms may be combined with lightweight enhancement modules, such as super-resolution or dehazing, selectively refine local regions of interest with minimal computational cost. This opens up the possibility of multi-stage visual processing pipelines that are triggered only when the model itself recognizes uncertainty or insufficient evidence. Beyond single-image reasoning, attention -baodd guidance could also support multi-hop hop visual reasoning across multiple images or documents, where the model iteratively decides which visual regions to revisit or enhance.
Toward Agentic Problem Solvers. We view these directions as promising opportunities for enabling more efficient and adaptive agentic multimodal reasoning, where models can actively exploit their own internal signals to better support complex decision-making processes. In the longer term, embedding such self-directed evidence gathering into broader agentic frameworks could help VLMs evolve from passive perception systems into active problem solvers that can plan, verify, and refine their own reasoning steps.
C ADDITIONAL RESULTS AND ANALYSIS
C.1 ADDITIONAL EVALUATION ON MULTI-TURE AND MULTI- REASONING
Beyond single-turn, single-image QA, real-world multimodal interactions often require reasoning across dialogue turns or integrating information from multiple images. These settings introduce additional challenges: the text context grows longer, grounding must remain consistent across turns, and evidence must be identified across multiple visual sources. To assess the generality of VEA under such conditions, we further evaluate it representative multi-turn and multi-image benchmarks.
17

---

Published as a conference paper at ICLR 2026
Setup. For multi-turn reasoning, we the Visual Dialog (VisDial) dataset (Das et al. 2017 where each question needs to be answered using both the associated image and the cumulative dialogue history. We integrate the historical dialog in new "dialog" section in the input prompt template. Following prior work, we report Token F1 and Exact Match (EM). VEA is applied using the attention from the first decoding step of the current turn. For multi-iman reasoning, we evaluate the Semantic Correspondence task in the BLINK benchmark (Fu et al. 2024), which requires cross-image visual grounding beyond textual descriptions. Specifically, in this task, the first image gives the reference point, and the model is asked to identify the point from 4 possible options in the second image that is semantically matched the reference point in the first image. We apply VEA on the second image and left the reference image untouched. Since the task is multiple-choice, we report Accuracy.
Multi-Tura QA 5 the results on VisDial. VEA consistently improves both metrics across all tested models, with particularly notable gains in Token F1. VisDial answers are free-form, and many visually grounded improvements may not yield exact string matches. Thus, provides a more sensitive measure of grounding quality. We observe that VEA shifts model predictions toward visually anchored content, increasing lexical overlap with ground truth even when EM remains unchanged. This indicates that VEA helps maintain consistent evidence usage across turns and reduces generic or amage-agnostee responses.
Table 5: Multi-tum QA results on VisDial dataset.
VisDial
LLaVA-1.6-7B
Qwen2.5VL-77
Gemma3-4B
Intern VL3.5-8B
Base Fl
28.5
27.5
25.0
33.2
VEA Fl
44.9 (+16.4)
47.8 (+20.3)
41.2 (+16.2)
48.6 (+15.4)
Base EM
38.4
37.6
34.0
38.5
VEA EM
40.9 (+2.5)
40.5 (+2.9)
37.5 (+3.5)
41.4 (+2.9)
Multi-Image QA. Table @ summarizes the results on the BLINK Semantic Correspondence task. VEA again improves performance across all evaluated models. Multi- image reasoning requires iden- tifying consistent semantic cues across distinct images. VEA enhances this process by suppressing irrelevant visual regions and strengthening attention to patches that share meaningful correspondences across images. Notably, even strong models such as Intern VL3.5-8B benefit from VEA, demon- strating that inference-time evidence emphasis remains complementary to advanced multimodal architectures.
Table 6: Multi-image QA results on BLINK dataset.
BLINK
LLaVA-116-7B
Qwen2.5VL-7B
Gemma3-4B
Intern VL3.5-8B
Base ACC
34.4
61.4 68.9 (+7.5)
42.2
66.4
VEA ACC
45.2 (+10.8)
50.9 (+8.7)
71.3 (+4.9)
These additional results show that VEA generalizes robustly to more complex multimodal settings. Whether the model must track evidence across dialogue turns or align visual cues across multiple images, emphasizing visual evidence consistently improves grounding quality. This highlights the broad applicability of VEA as lightweight inference-time enhancement for multimodal reasoning.
C.2 EVALUATION GL.ORAL-CONTER REASONING BENCHMARKS
As discussed in RQ4 (Fig. 7), overly aggressive masking (c.g., = 1) can remove too much background, but within a practical range (a E (0.25, 0.75)), VEA consistently enhances grounding without impairing global comprehension. We also include variant, VEA", that preserves both the original and highlighted images to further strengthen global-context access. To further examine whether VEA affects performance in scenarios where the focus is the non-text visual elements in global context, we additionally evaluate it on A12D (Kembhavi et al. 2016) and MMStar (Chen et al. 2024). These datasets emphasize scientific conceptual reasoning over fine- tine-grained perception. However, we also note that their images typically consist of large, clean graphical elements where visual evidence localization is not the primary challenge.
18

---

Published as a conference paper at ICLR 2026
Experimental Setup. AI2D focuses diagram understanding and scientific concept reasoning, while MMStar aggregates multimodal scientific and mathematical problems, including samples from MMMU and Math Since both benchmarks use multiple-choice questions. report accuracy. For VEA", , the model receives both the original image and the VEA-Iiggllihhte image as input. Additionally, include variant VEA* that retains both the original and highlighted images to check whether this can further strengthen access to global visual information.
Table 7: Results on global-context benchmarks (A12D and MMStar).
Dataset
Method
LLaVA-1.6-7B
Qwen2.5VL-7B
Gemma3-4B
InternVL3.5-88
Base
66.4
78,9
56.2
77.2
A12D
VEA
69.8 (+3.4)
81.1 (+2.2)
61.5 (+5.3)
81.8 (+4.6)
VEA*
70.2 (+3.8)
82.3 (+3.4)
63.3 (+7.1)
83.7 (+6.5)
Base
36.7
48.4
32.8
53.7
MMStar
VEA
40.5 (+3.8)
52.8 (+4.4)
37.5 (+4.7)
57.6 (+3.9)
VEA"
40.2 (+3.5)
53.3 (+4.9)
39.1 (+6.3)
58.2 (+4.5)
Results and Analysis. Table 7 summariizs the results. The improvements on A12D and MMStar confirm that VEA does not hinder global-context reasoning, though the magnitude of improvement smaller compared with datasets that rely more heavily on fine-grained visual evidence. This aligns with the fact that these reasoning tasks emphasize logical inference rather than evidence localization. The VEA* variant further boosts performance by preserving full-image information while still highlighting evidence regions, particularly benefiting models with strong multi-image reasoning abilities (e.g.. Qwen2.5- VL, Intern VL). These results support that VEA is compatible with ceassoninn-centrte benchmarks and that, within reasonable intervention strengths, foreground emphasis does not disrupt global context understanding.
C.3 MULTI-ROUND SELF- VALIDATION WITH VEA
While VEA enhances visual grounding by highlighting evidence during a single inference pass, its decision process remains static, limited by the model's initial belief. In principle, model could revisit its own outputs and revise incorrect responses, guided by the same visual attention mechanism in cascaded fashion. Motivated by this, we explore whether equipping VEA with self-validation loop, where the model reflects on and potentially corrects its previous answers, can further improve response accuracy.
which adds lightweight multi-round self-validation process. After generating an initial answer, the model receives the VEA-highllihhted image and is asked to verify whether its own answer is correct. If the model responds affirmatively (e.g., confirm this answer is correct"), we take the answer as final. Otherwise, the model is prompted to revise the answer and regenerate the attention map for the next round. We allow up to validation rounds and conduct experiments using VIL.- on four datasets.
Experimental Setup. We implement a variant,
Table 8: Performance of VEA-cascade with multi-round self-validation.
Method
TextVQA
DocVQA
SROIE
InfographicsQA
VEA EM
90.3
76.2
82.5
64.4
VHA-cascade EM
90.3 (+0.0)
76.6 (+0.4)
82.7 (+0.2)
64.4 (+0.0)
Avg. #Rounds
1.05
1.09
1.10
1.07
Results and Analysis. Taale/Bgsummrtii the results. While VEA-cascade introduces self-reflection, it leads to only marginal gains across all tasks, and the average number of rounds remains close 1. This suggests the model rarely alters its initial response, even when given the opportunity to revise. The limited improvements from VEA-cascade reveal that current VLMs tend to reaffirm their initial answers, with only minor evidence of genuine self-correction. The small number of rounds used suggests the model lacks confidence calibration or uncertainty estimation necessary for productive self-reflection. These findings align with recent studies that highlight the challenge of inducing
19

---

Published as a conference paper at ICLR 2026
effective reflection in VLMs Wei al. 2025); ): Jiang et al. (2025). Nonetheless, we believe that by
integrating VEA's attention feedback with stronger multi-step reasoning modules and more structured reflection prompts, future work could unlock richer and more adaptive visual reasoning pipelines, as discussed in Section B.3
C.4
WHEN ATTENTION IS NOT RELIABLE.
VEA relies on the model's internal attention maps locate supporting visual evidence. While prior analyses confirm that VLM attention often aligns well with ground-trute truth evidence, it remains important to examine failure modes, particularly when attention grounding inaccurate. We aim to answer three questions: (1) How frequently does attention misalign with ground truth? (2) Can VEA still improve performance in such cases? (3) What are the underlying causes of grounding failures?
Table 9: VEA performance on samples with low attention-evidence alignment AUROC<<0.5)
Metric
TextVQA
DocVQA
SROIE
InfographicsQA
Low-AUROC Ratio (%)
1.42
3.11
7.34
5.20
Base Fl
15.3
16.2
10.6
27.6
VEA F1
21.8 (+6.5)
26.4 (+10.2)
18.9 (+8.3)
35.3 (+7.7)
Results and Discussions. We identify "Jow-AUROC" samples where attention does not correlate well with labeled evidence (AUROC < 0.5) with Qwen2.5VL-7B Table 9 shows that such cases rare, ranging from 1.4% to 7.3% across datasets, and that VEA still improves QA performance even on these samples. Despite poor AUROC, VEA often leads to better answers. Upon manual inspection, find two main causes for these "false failure" cases: (i) Incomplete annotations. For many questions, the image contains multiple valid evidence regions, but only one is labeled in the ground truth. When the model correctly attends to unlabeled regions, AUROC is low despite accurate grounding VEA remains beneficial in such scenarios, as it reinforces these valid regions. (ii) Genuine grounding failure. In small number of cases, the model fails perceive any correct visual evidence and highlights irrelevant regions. These typically stem from vague or misaligned annotations. For example, questions referencing nonexistent elements or ambiguous phrasing. In such cases, both the model's attention and VEA's highlights reflect dataset limitations rather than model deficiencies.
C.5 DISCUSSION ON DISCREPANCIES WITH EXISTING FINDINGS
Several recent studies have reported seemingly different attention distribution trends. Upon closer examination, we find that these discrepancies arise mainly from differences in scope and terminology rather than fundamental disagreement. Our results in fact consistent with the attention dynamics described in earlier work once these distinctions are accounted for.
Specifically, Amara al. Amara al. (2024) do perform layer-wise breakdown but instead measure the total attention mass allocated to image tokens, concluding that image tokens overall receive more attention than text tokens. This matches our findings, as shown in our layer-averaged analysis. Lu et al. Lu et al. (2025) report that early layers show a high visual-attenten attention ratio with low concentration (broad scanning), and that middle layers maintain a high ratio while increasing concen- tration as cross-modal alignment emerges. This pattern directly corresponds to our observations: our "early" layers also exhibit diffuse visual scanning, and our "deeper" layers (roughly 50%-75% depth) align with their "middle" stages where focused visual grounding occurs. Similar layer trends also reported by Bi et al. Bi et al. (2025), supporting that our interpretation is consistent with the broader literature.
Overall, these analyses reflect complementary perspectives rather than conflicting findings. Once methodological framing is harmonized, the attention evolution patterns deseribed across studies present a coherent view of how multimodal transformers progressively shift from broad visual exploration to localized, semantically grounded alignment.
20

---

Published as conference paper at ICLR 2026
Table 10: Mean evidence attribution AUROC under different layer selection strategies.
Layers/ Method
LLaVA-7B
LL-VA-13B
Qwen-7B
Qwen-32B
Gemma-4B
Gemma-27B
L0%-1005 100%
75.9
76.3
68.5
$7.0
59.5
61.8
Lox 50%
68.2
73.1
59,4
51.3
56.5
55.4
Lao% 100%
78.0
76.9
79.5
67.6
65.9
68.1
L50% 75%
81.2
79.1
80.3
65.9
71.7
70.0
L75% 100%
74.8
74.7
78.7
70.4
60.8
66.3
VEA Profiling
83.6
84.4
85.2
79.1
80.0
81.2
Table 11: TextVQA Token F1 under different layer selection strategies.
Layers Method
LLaVA-7B
LL3VA-13B
Qwen-7B
Qwen-32B
Gemma-4B
Gemma-27B
Base
27.8
66.8
79.5
72.1
58.2
57.6
Lo% -0%-100%
58.2
72.8
86.6
78.2
64.5
61.2
I 1o% 50%
38.9
71.6
80.1
71.8
59.9
56.4
1.0% 100%
62.2
73.1
87.6
79.8
65.4
62.0
L50%-75%
66.7
74.1
87.5
78.8
66.9
62.1
L75% 100%
56.5
71.9
87.1
80.6
63.6
62.1
VEA Profiling
69.4
76.3
88.6
81.4
69.6
63.2
C.6 EVALUATION THE EFFECT EVIDENCE-LAYER SELECTION
VEA highlights image regions by manipulating attention at specific transformer layers. Since different VLMS may exhibit grounding behavior at different depths, we identify high-evidence layers for each model. This section discusses (1) what is the benefit performing profiling for each model, and (2) how sensitive VEA is to inaccurate static layer choices.
Profiling helps achieve more accurate evidence identification. Table 10 reports the average evidence attribution AUROC across four datasets (following the setup in Table 2 under different layer selection strategies. addition to the static layer choices in Table we further include two finer-graing static ranges, L50% 755% and L75%-1000 100% Across six model families, the later half of the layers 1150%-100% generally produces stronger attribution than early layers, with L5o%-75% often being the best-performing static choice. Per-model profiling further improves robustness and consistently achieves the highest AUROC, especially for models such as Gemma that exhibit non-monotonic, "periodic" grounding patterns across layer depth.
Robustness to Inaccurate Selection. We further examine VEA's sensitivity to imperfect layer identification by testing QA performance on TextVQA with intentionally statie (non-profiled) layer choices. As shown in Table IT] VEA maintains strong gains even when the chosen layer range deviates from the optimal one. Using middle-to-deep layers (L50% 100%))000)) consistently yields stable improvements, while profiling provides additional but incremental benefits. Only early-layer steering (L0%-50%% noticeably reduces performance, confirming that early layers lack visual grounding.
These results show that model-specific evidence- layer profiling is not strict requirement but an enhancement. VEA remains effective even with coarse, static layer choices, confirming its robustness to layer selection inaccuracies. Profiling, however, helps align VEA's steering with model-specific grounding depths and mitigates variance across architectures, particularly for models with irregular attention-layer behavior.
C.7 MORE VISUAL EXAMPLES
Figure 8 provides additional qualitative examples with Qwen2.5VL-7B to illustrate VEA's behavior across diverse visual question answering scenarios. These results show that VEA consistently enhances visual grounding and answer accuracy by guiding models to focus on semantically relevant regions, regardless of scene type task focus.
21

---

Published as a conference paper at ICLR 2026
Oueatan Wore tin the mager
lef. hottibiee Irape))
Eunding W/VdA the ef gren esl aggn
Optorn: Car
chaing foll dhatt car!
ATVES Oetinng MANRE
TEXIT
Guecica: rany arthway su 112297 (Stel Thoee
M9A Tiree
dasana ("eacier Neneetadl Secteet
Cunalers how yould you desurfie conditon
Ounator:: clertty
Octless
VEA
Ovestion: what UEs of the b20k
VEVER Worto.
WS VER: Men Soub
Qucctore whet priiee
VER The wor Natinna Artiinn gres..
Tro..
VEA DORT MILLIN
Oueston What coksheed
teremats!
A Taaaard
Ques How mayy tee case DWA6-D ARe
WDb VDU Pary had 2555 CRRS
Figure 8: Additional qualitative examples VEA. For diversity, upper half of the examples are drawn from the MMStar dataset, which focuses natural scenes, while the lower half are taken from the four datasets used in the main experiments, which emphasize textual or document-centric scenes.
C.8 FULL QUESTION ANSWERING RESULTS
Table [2 reports the complete results of methods across four VQA datasets and eight VLMs. We observe that VEA consistently achieves the best second-best performance in nearly all settings, often ranking first in both Exact Match (EM) and Token FI. Compared with the base models, VEa delivers substantial gains. particularly datasets that require fine-graing grained reading comprehension such as TextVQA and DocVQA. Among the baselines, AGLA and CGR occasionally achieve competitive performance, but their improvements are less stable across model families and scales. In contrast,
22

---

Published as a conference paper at ICLR 2026
Table 12:
Full results of applying visual evidence augmentation methods to 8 VLMs from 4 latest series of LLaVA, Qwen, Gemma, and Intern families. We report detailed results on four visual question answering tasks, including both Exact Match (EM) and Token FI scores. For each metric, we also report the average rank of each method over all tested tasks and models. These results complement the Table I in the paper and provide comprehensive view of performance across datasets and metrics.
Model
Method
TexVVQA
DocVQA
SROIE
InfographicsQA
Average Rank
EM
Token Fl
EM
Token F1
EM
Token F1
EM
Token Fl
EM
Token Fl
BASE
48.44
27.78
44.53
46.74
41.41
46.17
19.53
12.51
5.50
5,50
INST
43.95
46.49
41.00
45.35
19.82
17.98
5.25
5.25
CGR
66.50
41.03
45.89
51.21
42.85
52.39
26.54
19.52
3.00
3.50
7B
VAR
55.16
63.63
48.61
48.15
42.75
51.56
24.10
19.51
3,50
3,50
LLavaNext
AGLA
65.39
49.96
48.78
51.41
42.24
53.79
25.31
2.75
2.00
VEA
75.32
69.36
53.51
43.89
53.78
29.86
25.60
1,00
1.25
BASE
69.53
66.76
56.25
64.43
43.75
59.86
28.12
24.05
5,25
5.75
INST
74.08
68.30
55.30
64.55
43.38
61.54
27.86
23.39
5.75
4.75
CGR
77.39
73.85
56.45
64.44
45.76
61.11
31.39
27.61
3.00
4.00
13B
VAR
74.99
69.05
57.21
65.31
44,40
62,48
29.35
26.07
3.25
3.50
AGLA
78,06
74.40
57.12
66.17
44,44
62.61
34.01
31,74
2.25
1.75
VEA
78.40
76.25
57.06
66.21
46.34
62.65
34.50
31.63
1.50
1,25
BASE
85.94
79.49
73.44
79.71
75.78
92.53
58.59
59.03
5.50
5.25
85.31
80.26
75.06
80.02
75.63
91.24
59.50
58.88
5.25
$.25
TB
CGR
88.69
86.42
73.59
79.97
80,98
93.66
60,98
63.39
3.75
3.50
VAR
87.82
81.56
75. 70
80.81
78.79
91.64
63.42
61.83
3.25
4.00
Qwen2.5
AGLA
89.75
88.72
76.14
81.55
94.32
63.51
2.25
1.75
VEA
90.33
88.63
76.24
81.47
82.51
94.38
64.35
63,90
1,00
1.25
BASE
81.25
72.10
73.44
74.65
68.75
81.43
53,91
51.12
5.50
6.00
INST
80.58
75.23
73.48
74.79
67.95
81.77
54.18
$2,97
5.50
5.00
CGR
76.66
90.17
2.75
32B
VAR
81.64
77.05
74.50
76.42
70.77
88.36
54.71
53,88
4.00
3.50
AGLA
84.75
77.30
75.06
80.00
74.79
89.10
60.47
54.26
2.25
2.00
VEA
86.87
81.44
77.03
80.45
76.02
91.47
63.32
54.23
1.00
1,25
BASE
78.12
58.21
58.59
54.34
54.69
66.13
35.16
23.47
5.00
INST
76.93
61.76
$9.06
57.90
53.90
64.88
34.29
23.09
5.75
5.50
CGR
82.53
63.72
63.65
61.50
37.78
67.45
37.55
26.27
2.50
3.25
4B
VAR
81,57
68.29
60.41
54.96
69.83
34.95
27.09
4,25
Genmna
AGLA
83.02
67.48
64.25
60.07
57.20
70.47
37.26
25.55
2.50
3.25
VEA
83.18
69.60
64.52
63.24
58.76
71.43
38.32
27.12
1,00
1,00
BASE
85.16
$7.58
70.31
67.26
70.31
82.88
51.56
22.88
6.00
5,50
INST
86.24
57.49
71.80
66.68
70.67
83.72
51.98
25.17
5.00
5.50
CGR
87.08
62.10
72.69
68.33
73.45
85.51
55.88
29.38
3.75
3.25
27B
VAR
88.86
6.3
73.26
69.12
72.47
86.54
59.06
26.25
2.50
2.50
AGLA
91.48
63.15
72.85
67.82
74.16
86.54
28.20
2.50
3,00
86.87
29.95
1.25
BASE
83.59
79.41
85.94
88.50
84.38
93.40
63.28
$7.20
5.00
5.50
INST
83.30
80.31
86.43
89,56
83,17
92.18
63.87
57.95
5.50
5.25
CGR
86.58
84.34
87.64
91.47
84.12
92.74
71.16
69.48
3,00
3.75
8B
VAR
83.67
82.80
87.08
89.89
84.53
94.37
65.33
69.87
3.50
3,00
InternVL335
AGLA
86.21
84.78
87.44
92.47
84,30
93.69
71.27
69.95
3.00
2.25
VEA
87.57
86.47
88.42
92.57
84.66
94.23
72.12
70.07
1.00
1.25
BASE
86.72
75.06
88. 28
82.31
80.47
79.45 70.48
61.72
40.94
5.25
5.25
86.37
75.29
87.31
81.39
79.50
78.51
62.12
40.86
5.75
5.75
CGR
89.64
76.25
89.35
83.44
82.62
80.88
65.08
41.50
3.00
3.00
14B
VAR
87.45 47
77.87
90.42
82.36
80.71
79.50
62.95
41.60
3.25
3.25
AGLA
90.04
83.11
84.96
81.68
42.33
2.50
2.50
VEA
90.21
78.15
90.24
84.17
85.35
87.67
65.73
42.94
1.25
1.25
VAR benefits from leveraging attention scores but remains sensitive noisy signals, even with additional smoothing. Another key observation is that VEA provides consistent improvements across all four VLM families (LLaVA, Qwen, Gemma, and IL), The average rank results confirm this trend: VEA outperforms all other methods with the lowest ranks across metrics, demonstrating its robustness and generality inference-time augmentation strategy.
23

---

Published as a conference paper at ICLR 2026
Table 13: Full results of visual evidence attribution accuracy across four VQA datasets. We report token-level AUROC and @DCG@all for different attribution methods, including both static layer- based baselines (e.g., -0%-100% -0%-50%% 1150%-00%% and adaptive attribution approaches (VAR, AGLA, and VEA). For each dataset and model, we present results separately, and also include the average rank of each method across models. These results complement Table in the main paper.
Evidence Attribution
LLaVA 7B
LLLVV
13B SBOG
Qwen 7B
Qwen
32B
Gemma 4B
Gemma 27B
Avg. Rank ())
AUROIC
NBCG
AURIC:
AUBOC
NDCU
AUBOC
NDCG
AURIC:
ND0G
AUROC
NDUG
ALBOC
ND0G
Lox 103%
87.2
51.4
87.0
52.1
83.4
51.5
73.6
33.7
65.1
29.5
70.1
30.3
4.50
4.50
TerVVQA
Las-50R
79.9
45.0
84.6
51.1
73,4
34,6
65.1
31.3
62.6
28.1
61.4
28.5
5.67
5.67
Laars 100%
88.0
64.1
87.3
56.6
86.7
63.9
81.1
47.5
68.2
35.8
76.6
38.8
2.83
2.83
VAR
82.0
49.5
84.2
46.3
84.1
61.3
80.6
43.1
54.5
26.8
71.8
33.0
4.83
4.67
AGLA
90.2
68.4
900,4
66.8
86.2
61.0
87.0
58.0
72.4
36.8
81.9
43.9
2.17
2.33
VEA
92.2
76.5
92.4
75.3
90.7
74.2
89.8
63.9
88.0
63.2
88.0
$5.0
1.00
1.00
10%-1100%
76.6
45.2
77.7
46.3
67.2
40.9
53.2
33.2
60.2
37.8
62.0
38.8
4.33
4.33
DoeVQA
66.5
72.5
44.8
35.5
50.5
32.8
$7.2
36.5
56.4
37.2
5.67
Lagg, 100%%
79.6
53.7
79.2
50.2
80.0
59.7
66.
45.0
67.6
48.2
67.7
46.6
2.83
2.83
VAR
70.3
43.2
74.5
43.8
75.8
54.9
65.8
41.7
52.0
34.7
55.6
37.6
5.00
5.00
AGLA
81.7
56.3
83.0
56.0
77.8
56.8
73.8
54.0
69.5
49.2
73.7
52.7
2.17
2.17
VEA
85.2
63.1
85.7
62.9
86.0
70.4
76.5
60.8
78.7
61.8
81.9
64.8
1,00
1.00
Lox 100%
71.7
58.6
70.0
56.5
62.3
44.1
51.5
38.8
58.8
44.5
60.7
45.5
4.33
4.33
SROTE
Lan-50R
63.8
54,5
67.6
56.2
53.9
40.4
44.4
37.9
54.5
43.0
53.0
43.7
5.67
5.67
Loo's 100%
72.7
63.2
70.7
59.6
73.4
59.3
62.2
46.7
65.6
54.5
52.4
2.83
2.83
VAR
65.8
54.9
63.9
53,3
65.5
53.5
56.8
43.9
49.9
42.1
54.0
44,9
5,00
5,00
AGLA
75.0
65.7
75.6
65.0
71.7
57.1
68.9
$3.0
67.8
55.4
70.9
58.1
2.17
2.17
VEA
78.9
72.6
79.0
71.1
80.4
68.9
74.2
59.7
80.2
69.8
68.5
1.00
1.00
InfographVOS
Lox 10X-100%
100%
68.3
70.3
34.1
30.3
49.8
26.4
53.88
30.3
54.4
30.9
4.17
4.50
62.4
31.7
67.5
33.4
50.6
26.3
45.4
25.7
52.0
29.7
50.9
30.0
5.67
5.67
-1005-100%%
71.5
37.2
70.3
35.5
78.0
49.6
61.1
33.8
62.3
36.2
63.4
36.8
3.00
2.83
VAR
65.3
32.9
65.8
32.6
46.8
59.7
30.4
48.5
29.5
31.5
4.83
4.83
AGLA
73.8
38.4
75.5
38.5
75.1
46.8
70.6
40.3
63.6
36.6
68.9
41.2
2.33
2.17
VEA
77.9
42.0
80.6
44.5
83,9
60.9
75.7
49.2
73.0
45.0
76.4
52.0
1,00
1,00
C.9 FULL EVIDENCE ATTRIBUTION ACCURACY RESULTS
Table 13 reports the full token-level attribution results across four datasets and six representative VLMs. Several consistent patterns can be observed. First, static baselines that aggregate attention uniformly over either all layers (L0%-100% ) or early layers (L0%-50%) perform poorly, indicating that not all layers contribute equally evidence attribution. In contrast, restricting attention to later layers (100) yields substantially better AUROC and NDCG scores, confirming that deeper layers play critical role in localizing evidence. Second, adaptive attribution methods such as VAR and AGLA improve upon static baselines in many cases, with AGLA in particular showing stronger ranking quality across datasets. Finally, VEA with layer profiling consistently achieves the highest scores in both AUROC and NDCG @ all across all models and tasks, leading to the best overall ranking. These results validate that VEA more accurately identifies relevant evidence tokens compared with both static and adaptive alternatives, and demonstrate the robustness of the proposed approach across diverse datasets and model families.
1. FULL LAYER-WISE ATTENTION DYNAMICS VISUALIZATION
We also provide the completion of the layer-wise attention dynamics visualizations in Section 2
presents the layer-wise modality attention transition across multiple models and datasets. Although the detailed patterns differ across model families, the overall trend holds consistently: carly layers predominantly attend to text tokens, whereas deeper layers progressively shift their focus toward image tokens, revealing a sequential transition from linguistic parsing to visual grounding within single-token inference.
Figure 9 as an extension of Figure in Section 2.1
complementing Figure in Section 2.3 shows the relative average attention assigned to evidence versus non-evidence image tokens across layers. Consistent with our main-tex analysis, deeper layers across all models and datasets consistently allocate higher attention to evidence regions, even in cases where the model produces incorrect answers.
Figure 10
24

---

Published as a conference paper at ICLR 2026
To more directly assess the ability of different layers locate visual evidence, Figure reports the evidence attribution accuracy (AUROC and NDCG@ all) of each layer across multiple models and datasets. In line with Figure IO and prior discussions, deeper layers generally achieve higher attribution accuracy. Interestingly. however, the distribution of optimal layers varies across model families: LLaVA's best-performing layers cluster around the middle layers, Qwen's peak layers concentrate near the final output layers, while Gemma exhibits periodic pattern in which every few layers contain "good attribution layer." These diverse patterns highlight the importance and benefit of per-model profiling. We hypothesize that such differences may stem from family-specific design choices or training strategies, though deeper understanding of their underlying causes remains open for future investigation.
25

---

Published as a conference paper at ICLR 2026
[Modet Uiwe 5-78-Pf)
[Mode:: Usve 1-78 Hf DCCVD4)
[Nodst Uove 1.577 411
[Moreel: Uiee 5-78 MFFGRMMEESVOO
SADIE]
Gette Sreae
Queste Maan
[Mtedet Ulwaa11 5-13844
[Mode: Ulawe 3. 5-138-4f1
[Model Usaee 1.5ve-11.33 1-5-30447
[Medel Live- 5-138
Queston
(Mocet: Owen2 SAY 7B Instrant)
(Medel Dwend 5/3.7D morvet)
[Mode Coee2 S-I7Dimmruut
[Soodel: Owan7 SAT ivetruct
Quesue Meac
Duncke
35
30
[Modet: Dieen2 bol--iB-irerreet
(Modal: Creen Sal-2774rmtnuut
[Nodel Dewen2 S0ho32aanatttt
[Mode: Geen Sww.1B-Beeree
Danoua meak
Oaeator
Indlec
[Nodst: Gemma-3-44-0 5-4--)
Mocel: Gemmaa 48 [i] DOCVOK)
(Rodal Gemmro 3-rpp-3-40 3-48-R]
[Modet Gemma 3-8-111 REPORMM-ESVVAN
Indloc
(Mtudldd: KaattTooo---... (od-----d (541]
[Mudet Lwm.me---/// (5-4]
(Moddal Cammmaa--222
[Stddll: Dermmndi
Oenton loun
Quetao
2.4
Frdkes
tndio
Figure 9: Relative attention per token (RAPT) (y-axis) text tokens (red) and image tokens (blue) across Transformer layers for six representative VLMs (LLaVA-7B, LLaVA-13B, Qwen2.5-VL-7B, Owen2.5-VIL-22B. Gemma3-4B, Gemma3-27B)
four VQA datasets. Across all models and datasets, we observe a consistent trend: early layers attend predominantly to text tokens. whereas deeper layers gradually increase their focus on image tokens. This reveals a sequential transition from linguistic parsing to visual grounding during single-token inference. These figures complement Figure I in the main paper.
26

---

Published as a conference paper at ICLR 2026
Eideros Paichas
Naw-Eaderre Pastra
(2e innver Geneeree
- T99a
[Darsset: TEXTIVOA)
[Datssst: DOCVQA)
[Delomet:: $50 []
52000
000%
4000%
1200%
1000%
837%
2000/s
1000%-
190%
460%
100%
200%
200%
064
[Daresst: TEXTVOA)
[Delaset DOCVOA)
(Debaeet: SAORE)
[Calonal: REOGRMNHISSOO
0900/%
1200%
600%
1000%-
400%
1000%
207h
200%
0%
20
10
[ModLt 5-V1- mnstrcct)
instruct) [Dalsset DOCVOA)
[Modst CHiori2,5 (nsnsrrcct) (Delaset: SNO R]
[Mocet: Owen2 Instruct [Duteset: IASSCMM-HSSSON
[Darsset TEXTIOO
3005%
2000%
2300%
1802 10.
1000%
3300%
1500%
509%-
200%
25
[Modc: eistuutt [Dursssst: TEXTVOA))
(Mude: rntracc) [Datoset DOCV3A)
(Modal
[Dulaset: S/10 E]
[Jatoset: INFOCINNHHCSSOO
1500%
13.0%
1500%
1.0%
1002%
130%
1000%
153%
5.0%
150%
325
232%
200%
27
10
90
28
22
[Modee:: Gemema- 4B
[Nidel 3-4841) [LANiSse DOLY3A)
(Modle: 4..--.
[Model 3-4--1)
[LArssset: (EATVON)
1200%
1253%
190%
600%
1000%
133%
100%
2005
200%-
20
30
(Model: Genma 27B [Cotaset TEXTIVA))
[Nodel Gemera 27B [Cotacet DOCVOA
[Hode: Genma 278
(Modeel Gemma (Detacet INFOCRAPHRS50OM
(Dataset: 590 EI
1500%
12000
1250%
752%
20%
10
2.
40
2.
20
40
10
20
43
10
30
Figure 10: Relative attention to evidence image tokens (blue) vs. non-evidence image tokens (orange) across layers for six VLMs (LLaVA-7B/13B. Swen2.5-VL-7B/32B. Gemma3-4B/27B VQA datasets. Across models and datasets, deeper layers consistently assign higher attention to evidence regions, even when answers are incorrect (dashed lines). Best viewed in color. These figures complement Figure 4 in the main paper.
four
27

---

Published as a conference paper at ICLR 2026
(Rodihh
(Dulaweii: TEXTVOAA
[Dalisst:: DOCVQA)
[Delonen: Smore)
[Delidat: HEGGEIMM-SSSSA
AUROC
AIRCE
ROOD
FAROC
0035
022
15.
20
25
20
Laye
Loye inox
Intox
(Modee: (.333H [bulanel:: TEXEVSIA
[Mode:
Usoo L.S-13841) [Ailisee: Smote)
Ulve-- 5-139
[LMitset DOCVSA)
Diease AUDOC
g
bider ca ROOG
Polence AUROC
20
20
[Modee: Owen2 instrutt (Dalaset: TEXIVOA]
[Model 5 insrrctt [Dataset DOCVBA
[Model Ower? instruct (Batssee: SACIE)
[Mode: San Instruct [Dstaser: INFOGRA Suoal
Exderco Loca
92
10
15
25
10
(Nodet Opeen2 Saak instructt (Datssen: TEXINOA]
(Wodel Owee2 S-tt 3004nntruct
[Medet Owen2 340-120-4sttmut (Dxtboe:: SACIE)
[Mode: Gwee2 5-0-320 lnvtrrutt)
[Tatteet DOCVOAI
[Datase:: INFOGREP Supal
ALLDC
MOCS
10
10
20
[Model 3-40m (Dabaset: TEXIVOA)
[Midee: Gemma 4p--t
[Modal: Gemma-3-
[Nodel Cemma-3 44--1
[Dhttenet: DOCUDA!
[Dutaear: GAOIE)
[Datace:: INFOGRAP HCSVA
Eetro ALS0C
Extance
20
20
20
30
(Modell Gomma 2278-1 (Dalanet:: TEXTIVA)
[Modet 3-mmm-3-278-4 [Dateset: DOCUSA)
[Moce:: Gemmaa 3-2 3-278-1)
(Model: Gomma 3 278-1)
[Datuset: SROIE)
(Dotunet: IMFOGIAP HECVVAA
AMDO
1O00
HBOOC
Figure 11: wise evidence attribution accuracy (AUROC and NDCG) of six VLMs (LLaVA- 7B/13B, Qwwen.55-VL-7B/322 Gemma3-44//7B) on four VQA datasets. Consistent with Figure 10 deeper layers generally achieve higher attribution accuracy. While the optimal layers vary across models, their patterns are stable across datasets, highlighting the benefit of per-model profiling.
28


Reviews:
[{"review_id": "xyMLDdmNxr", "reviewer": "Reviewer_RdjA", "summary": "This paper focuses on the \"seeing but not believing\" phenomenon in Vision-Language Models (VLMs): VLMs often perceive correct visual evidence in Visual Question Answering (VQA) but fail to leverage it for accurate answers. Through layer-wise attention analysis, it reveals three key findings: shallow VLMs layers focus on text, deeper layers sparsely attend to localized evidence regions, and deeper layers still lock onto evidence even when outputs are wrong. To solve this, the paper proposes VEA (Visual Evidence Augmentation), a training-free inference-time method. It first identifies \"visual-grounding layers\". During inference, VEA extracts attention from these layers, applies denoising and Gaussian smoothing to create an evidence mask, and fuses the mask with the original image to highlight evidence (weakening non-evidence regions).", "questions": "1. The paper proposes to improve the network attention by casting masks on the input image. Why not apply this method to intermediate features or attention?\n2. How does this method work on multiturn conversations, multi-image QA or video?3. \n3. How does this method influence the benchmarks that rely on global context or multiple elements, such as benchmarks for general knowledge understanding or math (such as MMMU, MMStar, AI2D, MathVista)?\n4. As the author suggested, the answer from a single inference might not be accurate. The same applies to the attention map — can this method apply to the same question and image multiple times in a cascade way?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an insightful observation on VLMs’ behaviors toward images. Its visualizations and analyses further reveal how text and image interactions are modeled across different layers, showing that the encoding of semantic features first emerges in deeper layers. The inconsistency between attention maps (i.e., \"seeing but not believing\") highlights limitations of current VLM architectures, which is valuable for guiding future research.\n\n2. To address this issue, the authors propose a simple yet effective algorithm. The design is straightforward and practical: it can be applied to various VLMs with zero training cost and demonstrates effectiveness across multiple benchmarks.", "weaknesses": "1. The method improves VLM performance by overlaying a salient mask on the input image, but it does not \"fix the VLM’s attention behavior\" (as the behavior of the VLM or attention is not changed). Additionally, the design of the algorithm will introduce extra cost, and also raises the convern about multi-turn/multi-image scenarios.\n\n2. The proposed algorithm augments the brightness of different regions of the image. However, this augmentation will change the original image, causing information loss and changes (e.g., this will influence questions about brightness or color). For questions that rely on global context or beyond retrieval, the proposed mask method might cause troubles.\n\n3. Compared with visual reasoning methods that interactively retrieve key parts of the image to gather information, the proposed approach is plug-and-play, however, it also introduce limitations. This point could be further discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jpFmuAR5AS", "reviewer": "Reviewer_LvsH", "summary": "This work investigates the phenomenon of “seeing and not believing”, where vision-language models (VLMs) attend to the correct visual regions but still produce incorrect answers. To address this issue, the authors propose an inference-time method that encourages VLMs to better leverage visual information by overlaying attention-derived masks on the input image. Experiments across four VQA benchmarks and four VLM families demonstrate the effectiveness of the proposed approach.", "questions": "1. In cases where the model attends to incorrect regions but still answers correctly, what happens when the attention-derived mask is applied?\nDoes it lead to performance degradation in such examples?\n2. I would be interested to see qualitative examples that explore attention behavior in more general, real-world scenes, beyond the text-centric scenarios that are (to my understanding) the main focus of the examined datasets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors conduct a thorough investigation of how different layers in VLMs process inputs and distribute attention, providing valuable insights for the community.\n2. The proposed solution is simple yet effective, showing consistent  improvements across eight different VLMs.", "weaknesses": "1. It is important to evaluate cases where the model does not attend to the correct regions.\nHow often does the model still answer correctly in such cases?\nDoes performance degrade when highlighting regions based on incorrect attention?\n2. The motivation is somewhat similar to [1], which also identifies this phenomenon and proposes attention-based approaches.\nThis overlap reduces the novelty of the contribution, though the improvements of the method are still appreciated.\n\n\n[1] \"Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly\", Liu et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fddHTv36Ny", "reviewer": "Reviewer_8FW4", "summary": "The paper proposes a phenomenon called “seeing but not believing” by studying the internal layers of vision-language models (VLMs). Through analyses of layer-wise attention dynamics and layer-wise profiling, the authors introduce a visual evidence augmentation approach that enhances regions likely to contain stronger visual evidence. The proposed method is training-free and provides both improved interpretability and better performance for VLMs.", "questions": "Did the authors observe the attention sink phenomenon [1,2] during their experiments? If so, how did they handle these situations?\n\nReference:\n[1] See What You Are Told: Visual Attention Sink in Large Multimodal Models, ICLR 25\n[2] Vision Transformers Need Registers, ICLR 24", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow. The phenomenon of “seeing but not believing” is intriguing.\n\n2. The experiments and ablation studies are comprehensive.\n\n3. The proposed VEA approach provides both interpretability and performance improvement.", "weaknesses": "1. Some experimental details are missing. For example, in Figures 2 and 3, it is unclear which models were used for attention map visualization.\n\n2. Including more visual examples could strengthen and better support the overall narrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JaDgzRyXMl", "reviewer": "Reviewer_3b67", "summary": "The paper investigates the cause of hallucinations in Vision-Language Models, hypothesizing that these arise because models perceive visual information but fail to leverage it effectively. The authors analyze attention distributions across layers, observing that early layers predominantly attend to textual features, while deeper layers focus more on visual evidence, and this focus is not always used for reasoning. To address this, they propose a steering mechanism that explicitly increases attention to visual evidence regions, enhancing the model’s grounding on visual cues.", "questions": "The analysis appears to be based on single-token inference. Are the reported results averaged across all generated tokens for each answer, or do they correspond to specific tokens only?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper offers an insightful analysis of attention localization across layers, showing that deeper layers indeed attend more to semantically relevant image regions when forming answers.\n\n- The VEA step is methodologically well-designed, particularly with its denoising and Gaussian smoothing components to maintain spatial coherence in the attention masks. The method provides a simple yet interpretable tool for reducing hallucinations.", "weaknesses": "- The comparison between text and image RATP values (Fig. 1) is not clearly justified, given that the two modalities operate on different attention scales. It remains unclear how a small increase in image RATP can be interpreted as a “modality shift,” especially when text attention is still orders of magnitude higher (e.g., 0.2 → 0.6 for image vs. ~20 for text). This weakens the claim that “vision plays a stronger role in later inference stages.”\n\n- Prior studies (e.g., [1], [2]) have reported different attention trends, showing that image tokens already dominate attention in early layers. The authors should better situate their findings within this literature, clarifying methodological or model-related differences that explain these discrepancies.\n\n- The interpretation of Fig. 4 could be refined: while the authors claim that evidence tokens receive higher attention even in incorrect answers, the plot also shows a drop in attention between correct and incorrect predictions, which complicates that conclusion.\n\n- The assumption that deep layers consistently capture “ground-truth” evidence regions is empirically plausible but not guaranteed. The paper would benefit from a clearer discussion on the reliability of these “evidence layers” across architectures and datasets.\n\n- It is also unclear whether the identification of such evidence layers must be repeated per model, and how sensitive the steering mechanism is to inaccuracies in this identification.\n\n[1] Amara, Kenza, et al. \"Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities.\" arXiv preprint arXiv:2410.01690 (2024).\n\n[2] Lu, Haolang, et al. \"Mitigating Hallucination in Multimodal Reasoning via Functional Attention Control.\" arXiv preprint arXiv:2510.10285 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)