Mitigating Hallucination in Vision-Language Model with Depth and Spatial-aware Key-Value Refinement
Authors: Gusang Lee
Soohyun Kim
Donghoon Kim
Kyuhong Shim
Byonghyo Shim
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination in Vision Language Model
Depth and Spatial-aware key value Cache Refinement
Key-Value Cache Manipulation
Multi Modal
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large vision–language models (VLMs) deliver state-of-the-art results on a wide range of multimodal tasks, yet they remain prone to visual hallucinations, producing content that is not grounded in the input image. 
Despite progress with visual supervision, reinforcement learning, and post-hoc attention reshaping, the representational origins of hallucinations remain unclear.
Our study reveals that successful grounding emerges when adjacent visual tokens exhibit coherent alignment, while hallucinations arise when key vectors scatter isotropically, weakening cross-modal attention and blurring object boundaries. 
Building on this insight, we propose Depth and Spatial aware Cache Refinement (DSCR), a lightweight and training-free method that augments the Transformer's key-value (KV) cache with depth cues and 2D spatial proximity. 
DSCR clusters vectors within objects and separates those across surfaces, guiding attention toward relevant regions without any fine-tuning.
Comprehensive evaluations show that DSCR consistently reduces hallucinations, delivering up to 41.6\% accuracy gains across MME, POPE, RePOPE, CHAIR, and a new depth-sensitive benchmark. 
Our findings highlight KV-coherence as a core factor behind hallucinations and demonstrate a practical, model-agnostic solution for enhancing VLM reliability.

Published as a conference paper at ICLR 2026
MITIGATING HALLUCINATION IN VISION-LANGUAA MODEL WITH DEPTH AND SPATIAL- AWARE KEY- VALUE REFINEMENT
Gusang Lee', Soohyun Kim', Donghoon Kim', Kyuhong Shim Byonghyo Shim
Scoul National University "Sungkyunkwan University (gslee, soohyunkim, chkim, bshim]0 @inameiisllbb... snu. www.uaa........ ac.kr, khshim@skku.
ABSTRACT
Large vision-language models (VLMs) deliver state-of-the-ar results on a wide range of multimodal tasks, yet they remain prone to visual hallucinations, pro- ducing content that is not grounded in the input image. Despite progress with visual supervision, reinforcement learning, and post-hoc attention reshaping, the representational origins of hallucinations remain unclear. Our study reveals that successful grounding emerges when adjacent visual tokens exhibit coherent align- ment, while hallucinations arise when key vectors scatter isotropically, weakening cross-modal attention and blurring object boundaries. Building on this insight, we propose Depth and Spatial aware Cache Refinement (DSCR), a lightweight and training-free method that augments the Transformer's key-value (KV) cache with depth cues and 2D spatial proximity. DSCR clusters vectors within objects and separates those across surfaces, guiding attention toward relevant regions with- out any fine-tuning. Comprehensive evaluations show that DSCR consistently re- duces hallucinations, delivering up to 41.6% accuracy gains across MME, POPE, depth-sensitive benchmark. Our findings highlight KV-coherence as a core factor behind hallucinations and demonstrate a practical, aodel-ag solution for enhancing VLM reliability.
RePOPE, CHAIR, and
INTRODUCTION
In recent years, have witnessed remarkable advances in large vision-langue models (VLMs), such as GPT-5, Claude-4, and Gemini-2.5 (OpenAI, 2025; Anthropic, 2025; Gemini Team, 2025). VLMs widely used in vision-related tasks, including AR solutions (e.g., real-time navigation), VLM agent-driven automation (e.g.. smart shopping assistants), robot control, and visual content generation (Xiu et al., 2025; Song et al., 2025; Niu et al., 2024; Xu al., 2024; Yuan et al., 2024; Li et al., 2023b; Guo et al., 2025; Ge et al., 2025).
One well-known drawback that undermines the reliability of VLMs is visual hallucination. By visual hallucination, we mean the phenomenon of generating content that is not grounded in input visual information (Li et al., 2023c; Huang al., 2025; Sahoo al., 2024). Recent studies show that visual hallucinations emerge when the visual encoder fails to extract sufficient information from the input image answer the natural language query. In this scenario, VLM falls back on the linguistic priors it learned from large-scale text corpora rather than visual input data. As result, VLM would describe objects that do not exist, assign incorrect attributes to visible objects, or misinterpret spatial relationships within the scene.
Over the years, various approaches have been proposed mitigate visual hallucination phenomena. Prior work has explored three main strategies: augmenting training with extra visual supervision (e.g., bounding boxes or segmentation masks) to improve grounding (Jain et al., 2024; Wan et al., 2025), using reinforcement learning based reward modeling align outputs with human judgments (Sun et al., 2023), and applying lightweight, training-free techniques that reshape attention patterns filter out low-confidence predictions without touching model weights (Leng al., 2024b: Huang et al., 2024; Chen et al., 2024b; Wang et al., 2025; An et al., 2025). To some extent, these approaches
"Corresponding authors

---

Published as a conference paper at ICLR 2026
Non-Hallucin Honllcclcciitiion
Loyell
Lagerty
Layen
there fire hydrant in this image?
Conventional Methods: Yes
DSCR; Yes
Hallucination
Layer9
1ager21
Q. Find the farthest object in the image.
Conventional Methods: Bus
DSCR; Rui kling
(n) Question and Answer
(b) Image
(6) Key Vector Visualization of Conventional Methods
(4) Key Vector Visualization of DSCR (Ouns)
Figure 1: Visualization of non-hall hallucination (top) and hallucination (bottom) cases with key vector outputs. (a) Example questions and answers. (b) Input images, (c) Key vector visualizations from conventional methods. (d) Key vector visualizations after applying DSCR, showing more structured and spatially aligned features, especially in hallucination cases.
have shown benefits; however, a clear and shared explanation of why hallucinations emerge remains unresolved. In particular, it remains unclear how the model's internal representations change when exposed to a specific image and question, ultimately leading to hallucinations.
To investigate the fundamental reason why VLMs struggle retrieve query-relevant information from visual inputs, we applied principal component analysis (PCA) to the transformer's ' key vectors under both hallucinated and non-hallucinated conditions (see Section 2.4). As shown in Figure I(c) and (d), when hallucination does not occur, the key vectors of the spatially adjacent patches are well aligned. This spatial coherence becomes even stronger in deeper layers. In contrast, when hallucination occurs, the vectors scatter nearly isotropically, blurring the boundaries of the object. Our analysis suggests that both phenomena arise from a fundamental representational mechanism: loss of coherence among key vectors, which leads the vectors to scatter and blur object boundaries, thereby preventing the cross-attention mechanisms from anchoring onto visual representations. Con- versely, when key vectors maintain coherent alignment across adjacent patches, they facilitate robust cross-modal attention flows that reliably transmit visual information to the language model.
Building our PCA analysis of key vector coherence, we propose lightweight and training free mechanism that steers cross-attention to the most relevant image regions by refining the model's internal KV cache. We call this depth and spatial aware cache refinement (DSCR): injects both relative depth and 2D spatial proximity cues directly into every key vector, with no fine-tuning needed. Depth provides true 3D structure, sharpening object edges at depth discontinuities, separat- ing overlapping
gegrrounddbbckgrounng surfaces, and down- down-weighting occlusions. In parallel, 2D spatial proximity reinforces local context, ensuring that immediately neighboring patches (which of- ten share texture or semantics) remain tightly clustered in representation space. By combining these geometric and planar signals, DSCR guides key vectors on the same object to form coherent group and pushes apart those on different surfaces distant in the image plane. The result is geometry and locality-aware attention pattern that reduces spurious hallucinations without additional training-
Our contributions are summarized as follows:
Model agnostic KV refinement: We introduce blug-and-play technique that boosts the similarity of key vectors for spatially adjacent tokens belonging the same object, enabling VLMs to better capture core visual structures without any finetuning.
First analysis of KV-coherence neighboring- key similarity trigger visual hallucinations using PCA-based visualizations, attention-score diagnosties against query tokens, and end-to-end tests on focused hallucination benchmark.
hallucination: We uncover how the breaks in
depth-
Broad benchmark gains: Across MME, POPE, RePOPE, CHAIR, and novel bench- mark, DSCR delivers up to 41.6% improved accuracy. Morcover, DSCR can be seamlessly combined with existing mitigation strategies, providing complementary improvements.
2

---

Published as a conference paper at ICLR 2026
Depth and Sputial-aware Cache Refinement (DSCR)
BBMM X
Visual KV Cache
Refinol
Poasinity
KV Cache
olephonts in picture.
Casht...
Refindyy Cache
DSCR (Ours)
Currect
Depth Spaiial Information
DSCR tminingg-fax quer-aptosne mathod.
are elophante in picnure
Vision Encoder
Teat Tokenien
Wrong (Hallucination)
Query
Mono. Depth Estimation Model
"oor many siipfanns are picturs""
(a) Visual KV Cache Refinement
(b) Inference using Refined Cache
Illustration of the proposed DSCR. Extracted visual tokens first pass through the frozen VLM. Then, the corresponding KV cache is refined using the depth and spatial relationship between tokens. By this procedure, DSCR establishes a strong association between relevant visual tokens in the attention blocks, mitigating hallucinations in the VLMs. The VLMs with DSCR produce the accurate answer (blue), unlike the original VLM (red). Note that DSCR is training- ffree, model- invariant, and query-agnostes
Figure 2:
DEPTH AND SPATIAL AWARE KEY- VALUE CACHE REFINEMENT
2.1 NOTATION AND OVERVIEW
Notation. The token representation and notation used VLMs are defined as follows. Given an RGB image I E H x3 the vision encoder divides the image into non-overlapping patches and extracts N visual tokens. These tokens are passed as a sequence to the language model, which then consists of L transformer layers, each with H attention heads. Each head maintains KV cache, denoted as K = [k/..., kky and V = [v],..., v where (k, v!) E IRdh represents the key and value vectors for the i-th token, and d, is the head dimension.
Overview. Using DSCR, we can improve both the similarity between key vectors of neighboring image tokens and the model's ability to identify object boundaries. This implies that the refined model can more accurately attend to query-relevant visual regions during decoding. Figure il- lustrates where DSCR is applied in the model architecture. It refines the visual KV cache prior to decoding by leveraging auxiliary depth and spatial information. Importantly, DSCR does not re- quire model weight updates fine-tuning. It operates exclusively at inference time, increasing the similarity among visual token representations in a lightweight, modular manner.
2.2 THEORETICAL RATIONALE: DEPTH AND SPATIAL PRIORS
Our design grounded in classical image priors and recent graph-signal theory. Natural images are characterized by dominant low-frequency energy and strong covariance among neighboring patches. This observation motivates locality-aware Vision Transformer (ViT) variants such as LocalViT (Li et al., 2021) and SATA (Nikzad et al., 2025). Depth adds a complementary geometric prior: pixels with nearly identical disparity almost always lie on the same physical surface, while sharp depth dis- continuities coincide with true object edges (Tomasi & Manduchi, 1999). We encode both priors by re- re-weighting key-value pairs in the cache, yielding an edge-awar aage- attention operator that enforces local smoothness (via 2D spatial proximity) and respects object boundaries (via depth). Mathemat- ically, this acts as a graph-Laplacian smoothing term on the key-vector graph, clustering tokens on the same surface into low- now-frequency subspace and dispersing those across depth gaps. As result, suppresses high-frequenge noise and amplifies reliable local evidence.
Recent analyses show that hallucinations often arise when models over-rely global features and neglect local cues; explicitly combining global and local attention reduces such errors (Xing et al., 2024; An et al., 2025), and our depth-guided smoothing achieves this integration implicitly. Anal-
3

---

Published as a conference paper at ICLR 2026
Normalize
Patch (1,1)
Patch (H.W)
Riy
fa(d;
d
(s.
5)
Myy 1[(
Eg Ray 1(-0))
41 d2
dg
92
d5
kj Enikk
fa(-)
Xilg
X00
Depth-to-Depth Proximity
Spatial Proximity
Original Key Cache
Refined Key Cache
Figure 3: Details of the DSCR's key cache refinement process. Depth-to-depth and spatial proxim- ity maps show the difference in depth values and distances between image patch pairs, with darker shades indicating smaller differences. Using the importance weights (wyy) derived from the proxim- ity scores, refined cache entries computed as a weighted sum of the original ones.
ogously, DFormerv2 (Yin et al., 2025) inserts depth-based geometry self-attention into ViT back- bone, achieving SOTA on NYUD-v2 and SUN RGB-D datasets, demonstrating the practical impact of depth-guided coupling.
2.3 DEPTH AND SPATIAL-AWARE CACHE REFINEMENT
Depth Estimation. To extract the depth map D E H from the RGB image I, we use an off- the-shelf monotonic depth estimation (MDE) model. The depth map is then min-mas normalized to scale the depth values to [0, 1]. To match the resolution of the vision encoder's s output, we resize the depth map and obtain depth values (d1.d2, d2, dy) ). Each d, represents the depth value of the i-th image patch, which corresponds to the key-value (KV) cache entries k/ and v!.
Depth-to- Proximity. To identify the relationship between the and j-th image patches, we measure the depth-to-deppe proximity using the Gaussian kernel:
fa(d, d,) = exp ( (d, d,)3
(1)
203
where aa is a hyperparameter to control the width of the Gaussian function. One can notice that the proximity score is higher when the depth difference is small, indicating higher similarity for patches with similar depths.
Spatial Proximity.
Similarly, we compute the spatial proximity between and j-th image patches using a Gaussian function of the Euclidean distance:
fs - 5,) = exp ( (- |ss -0--511 s
(2)
20
where and sy are 2D pixel coordinates of each patch. Note that the proximity score increases if the two patches are close in an image.
Total Proximity and Importance Weight. The total proximity between i-th and j-th image patches is computed as:
Wij = fa(d, d d,)a
f s,
(3)
where a, B are hyperparameters controlling the contribution of depth and spatial proximity, respec- tively. To ensure that each token is refined based solely on its neighbors, we set the self- celf-proxmmity term رjj to zero by masking out the diagonal of the proximity matrix. We normalize this proximity score obtain the relative importance weight
Wi = wi 1/2 サ
(4)
Σn
Øτj 11 + j
where [[[] is the indicator function whose value one if the condition is satisfied, otherwise zero. For the hyperparameters, used fixed values for aa 0.6, a, 0.6, a 0.6, 3 0.8.
4

---

Published as a conference paper at ICLR 2026
closest object
Query Image
Quary
mmggs
Conventional
cloest
Skateboard
Methods
AL Food Cart
Suggect
All Token
Token
Metecydee
100
Query Stepect Iougt
(c) Depth-Halllucnntionn Mini Benchmark
Qu la the miuroycke on the right side of hes? No
A. Yos
Image
Coreedad
Layer
Vellas
(4) Stspect Visualization
(b) Comparison Attention Scores Retween
Conventonal Methods and DSCR
Figure 4: Analysis of hallucination suppression using DSCR. (a) Visualization of a ballucination case where DSCR reduces incorrect predictions by suppressing suspect tokens and enhancing the spatial structure of key vectors. (b) Layer-wise attention scores comparing Conventional Methods and DSCR across image, suspect, prompt, and tokens. (c) Accuracy comparisons from the Hallucination-Dere Depth Mini Benchmark, comparing the Conventional Methods with depth-refined KV cache. (d) Results on MME where spatial refinement applied to KV caches.
Key- Value Cache Refinement. Using the relative importance weight, we update each Key and Value entry by computing a weighted sum of all Key and Value entries, respectively. For the j-th entry, the refinement proceeds as follows:
kJ Ewyk! V Σ wiyy
The same weights are applied across the selected Transformer layers and attention heads. In practice, this weighted sum can be performed by single inter-tensor multiplication, ensuring that the DSCR computation remains highly efficient and negligible. The refined KV cache replaces the original KV cache before the text generation. Note that this modification is applied only to the KV cache entries corresponding to visual tokens.
We highlight that the entire DSCR process is vrining-frree, model-agnostic, and query-agnostic. Furthermore, our DSCR proposes the novel concept of refining KV caches, which has not been previously explored in ballucination prevention within both large language model (LLM) and VLM literature.
2.4 COMPREHENSIVE ANALYSIS OF HALLUCINATION IN LVLM
Figure 4 illustrates the superiority of DSCR from three perspectives: key vector visualization, atten- tion distribution, and henchmark-level performance gains reflecting improved utilization of depth and spatial information.
Key Vector Visualization. The visualizations shown in Figure 4(a) include the input image, the key vectors from the conventional method, and the key vectors after applying DSCR. Each key vector k; is projected into 3D via PCA for RGB visualization. The process is defined as:
RGB
Norm (PCAg(k,)) E R33
(6)
where PCAs(.) projects the key vector into a 3-dimensional space, and Norm(-) denotes channel- wise min-max normalization.
In VLMs, key vectors are not structurally aligned, and neighboring tokens often exhibit disjoint directions, particularly in hallucination cases. Notably, in the suspect token region (highlighted in red), sharp deviation in vector direction is observed (black region), which may undermine the model's 's ability to correctly interpret the visual input. In contrast, DSCR leads to smooth alignment among nearby tokens, with clearer object boundaries and the disappearance of suspect tokens. This behavior results from DSCR' use of depth and spatial proximity to align KV vectors for nearby patches and distinguish objects based relative depth differences.
5

---

Published as a conference paper at ICLR 2026
Table 1: Detailed evaluation results on MME hallucination subset. Best results in bold.
DSCR
VCD
OPERA
HALC +DSCR
DAMO
AGLA
Model
Metric
Baseline
VCD
OPERA
HALC
DAMO
AGLA
(Ours)
+DSCR
-DSCR
+DSCR
+DSCR
Existence
190.00
190.00
195.00
190.00
190.00
190.00
195.00
195.00
195.00
195.00
190.00
190.00
Count
143.33
143.33
160.00
153.33
148.33
135.00
160.00
143.33
160.00
160.00
155.00
155.00
Position
120.00
120.00
120.00
120.00
105.00
120.00
120.00
115.00
120.00
120.00
110.00
120.00
LL5
Color
165.00
165.00
165.00
170.00
160.00
165.00
175.00
170.00
175.00
175.00
165.00
175.00
OCR
117,50
117.50
117.50
125.00
117.50
117.50
140.00
110.00
140.00
140.00
117.50
140,00
Posters
156.85
156.85
157.19
141.10
151.37
162.67
135.96
155.14
135.37
135,96
148.97
151.37
Total
892.68
892.68
914.69
899.43
872.20
890.17
925.96
888.47
925.27
925,96
886.47
931.37
Existence
180.00
180.00
180.00
175.00
180.00
175.00
180.00
180,00
170.00
180.00
180,00
170.00
Count
153.33
138.33
158.33
141.67
150.00
146.67
156.67
136.67
135.00
156.67
155.00
143.33
Position
93.33
93.33
98.33
101.67
96.67
91.67
105.00
110.00
105.00
105.00
101.67
113.33
LLavA-166
Color
180.00
180.00
180.00
155.00
175.00
175.00
175.00
175.00
145.00
175.00
155.00
150.00
OCR
132.50
147,50
132.50
132.50
117.50
140.00
132.50
147.50
147.50
132.50
132.50
147.50
Posters
150.34
147.60
150.34
142.47
151.37
157.53
152.40
148.63
132.88
152.38
142.47
144.52
Total
889.51
886.77
899.51
848.30
870.54
885.87
901.56
897.80
835.38
901.56
866.63
868.69
Existence
170.00
165.00
170.00
170.00
160.00
170.00
175.00
165.00
165.00
185.00
153.33
175.00
Count
145.00
145.00
150.00
145.00
150.00
145.00
155.00
145.00
150.00
145.00
155.00
145.00 15:000
Position
98.33
98.33
98.33
98.33
108.33
93.33
103.33
98.33
101.67
108.33
121.67
96.67
Qwen VI.
Color
180.00
180.00
180,00
180,00
185.00
180.00
185.00
180.00
180.00
185,00
180,00
175.00
87,50
110.00
110.00
87.50
102.50
80.00
110.00
Posters
144.18
146.23
144.52
144.18
152.05
146.58
166.78
148.92
160.62
178.42
145.55
154.79
Total
825.01
844.57
830.35
825.01
865.39
822.41
872.61
839.81
844.78
881.76
865.55
826.46
Existence
195.00
195.00
190.00
185.00
190.00
190.00
190.00
195.00
185.00
190.00
190.00
190.00
Count
160.00
165.00
165.00
135.00
160.00
165.00
165.00
170.00
165.00
165.00
165.00
165.00
Position
160.00
160.00
160,00
153.33
160.00
165.00
155.00
160.00
155.00
155,00
155.00
165.00
Qwen2.5-VL Color
185.00
190.00
185.00
195.00
185.00
185.00
190.00
185.00
185.00
190.00
177.50
155.00
170.00
155.00
170.00
162.50
177.50
162.50
177.50
177.50
177.50
162.50
Posters
165.41
167.12
165/41
164.73
167.47
167.81
167.47
168.49
166.10
169.52
168.49
167.81
Total
1042,91
1032.12 12
1035.41
988.06
1032.47
1040.31
1039:97
1045.99
1043.60
1042.02
1040.99
1040.31
Attention Score Analysis. The effect of DSCR on attention distribution is shown in Figure 4(b). The three plots show layer-wise attention scores from the query to all image tokens, system prompt tokens, and text tokens. The model often exhibits little attention image tokens regardless of the question, instead focusing heavily on system prompts and textual priors. In contrast, DSCR con- sistently increases attention to relevant image tokens across different inputs while reducing reliance on text and prompt tokens. This indicates that DSCR shifts the model toward risual-grounde rea- soning. The bottom-left plot in Figure 4(b) shows the average attention scores assigned to image tokens at the suspect token indices for each query. Without DSCR, these positions exhibit notably low attention scores, suggesting that suspect tokens interfere with the model's ability attend to relevant visual information. In contrast, DSCR mitigates this issue, and higher attention scores are consistently maintained at those positions.
Depth & Position Hallucination Benchmark for Occlusion and Similar-Depth Scenarios. In typical datasets, occlusion and boundary pixels occupy less than 4% of the scene, making depth discontinuities relatively rare (Birchfield & Tomasi, 1999). In contrast, to demonstrate the robustness of DSCR even under challenging conditions, such as occlusion boundaries and semantically distinct objects located at similar depths, we constructed a depth hallucination meihmbench by selecting 50 images containing heavily overlapping objects with comparable depth values. For each image, we designed four questions asking about the closest, farthest, physically smallest, and physically largest objects, in order to evaluate the model's ability to reason about relative depth and real-world object size. The position hallucination mini-benchmark is built from the position subset of the MME (Fu et al., 2023) dataset to assess spatial relationship reasoning. As shown in Figure 4(c), DSCR improves accuracy on the depth hallucination benchbenchmark by 41.4%, indicating better utilization of depth information. Additionally, Figure 4(d) demonstrates 6% gain the position hallucination muni-benchharte confirming that spatial proximity contributes to more accurate spatial reasoning. See Appendix for more details about our mini-hhnnchmrk.
6

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results on POPE (GQA) and RePOPE (MSCOCO) datasets.
POPE (GQA)
RePOPE (MSCOCO)
Setting
Model
w/DSCR
Acc.
Prec.
Rec.
Fl
Acc.
Prec.
Rec.
Fl
0.90
0.92
0.87
0.89
0.92
0.93
0.88
0.90
LLaVA-1.5
0.90
0.93
0.85
0.89
0.92
0.95
0.87
0.90
Random
Qwen-VL
0.84
0.83
0.92
0,92
0.88
0,90
0.85
0.91
0.77
0.84
91
0.93
0.87
0.90
0.77
0.63
0.62
0.70
mPLUG-OW12
0.84
0.75
0.70
0.68
LLaVA-1.5
x
0.86
0.84
0.87
0.86
0.91
0.92
0.86
0.89
0.87
0.87
0.85
0.86
0.90
0.93
0.85
0.89
Popular
Qwen-VL VL
x
0.72
0.68
0.83
0.75
0.89
0.87
0.88
0.88
0.80
0.81
0.77
0.79
0.88
0.88
0.87
0.88
0.79
0.80
0.77
0.79
0.60
0.55
0.87
0.68
mPLUG-Owl2
V
0.79
0.82
0.75
0.78
0.66
0.60
0.90
0.74
0.81
0.78
0.87
0.82
0.89
0.87
0.87
0.87
LLaVA-1.5
0.82
0.80
0.85
0.82
0.89
0.89
0.85
0.87
Adversarial
0.77
0.89
Qwen-VI VL
0.77
mPLUG-OW12
0.77
0.76
0.77
0.77
0.58
0.53
0.85
0.66
0.77
0.79
0.75
0.77
0.63
0.58
0.88
0.72
3 EXPERIMENT
3.1 HALLUCINATION MITIGATION
MME Hallucination Subset. Table shows that DSCR achieves consistent performance improvements across various VLMs. The MME (Fu et al., 2023) total scores increase from 892.68 to 925.96 (+33.28 points) for LLaVA-1.5 (Liu et al., 2024c), from 889.51 901.56 (+12.05 points) for LLaVA-1.6 (Liu et al., 2024b), and from 825.01 to 872.61
Table 3: Evaluation results on the CHAIR dataset
using LLaVA-1.5.
Method
CHAIRs
CHAIR;
Recall
Ávg. Len.
Baseline
VCD
14.4
79.0
101.6
OPERA
39.2
11.6
73.5
83.9
DSCR
39.2
11.2
73.4
96.1
(+47.60 points) for Qwen-VL (Bai al., 2023).
In comparison with prior methods on LLaVA-1.55 DSCR improves the total score from 914.69 to 925.96 (+11.27 points) over VCD (Leng al., 2024b), from 872.20 to 925.96 (+53.76 points) over DAMO (Wang et al., 2025), from 899.43 to 925.96 (+26.53 points) over HALC (Chen et al., 2024b), and from 890.17 17 to 925.96 (+35.79 points) over AGLA (An et al., 2025). Furthermore, applying DSCR on top of existing inference-time methods consistently provides additional absolute gains in total score.
POPE & RePOPE. Table 2 demonstrates that DSCR consistently improves F1 scores on both the POPE (Li et al., 2023c) and RePOPE (Neuhaus Hein, 2025) in terms of absolute points.
For 1.55, DSCR improves the Fl score on POPE from 0.87 to 0.90 (+0.03) under the Random strategy, from 0.85 to 0.88 (+0.03) under the Popular strategy, and from 0.80 to 0.83 (+0.03) under the Adversarial strategy. On RePOPE, the Fl score increases from 0.72 to 0.80 (+0.08), from 0.70 0.75 (+0.05), and from 0.68 0.73 (+0.05) under the Random, Popular, and Adversarial settings, respectively. Additional POPE results on MSCOCO (Lin et al., 2014) and A-OKVQA (Schwenk al., 2022), as well as experiments applying DSCR to existing inference-time methods such as VCD and OPERA to further improve Fl scores, are provided in the Appendix.
CHAIR. Table 3 presents evaluation results on the CHAIR (Rohrbach al., 2018) benchmark, which measures object hallucination in image captioning. Compared to the baseline, VCD, and OPERA, DSCR achieves the lowest CHAIR; score (11.2) and matches the lowest CHAIRs score

---

Published as a conference paper at ICLR 2026
Table Evaluation results on AMBER dataset. A, P. R, FI indicates accuracy, precision, recall, and F1 Fl-score, respectively. Please see the Appendix for more details.
Generative
Discrtminative
Extstence
Relation
Attribute
Number
CHAIR Cover
Hal
Cog
Fi
A
B.
FI
Fl
A
P
R
Fi
Fi
A
F
R
FI
Baseline
66.0
66.0
73.3
79,9
VCD
64.3
64.3
156
89.0
78.9
DSCR
6,3
504
29.4
31
753
954
6577
77.8
71,6
100.0
714
834
714
M3.1
334
49.2
78.3
89.7
63.9
74.6
15.5
87.6
59.4
70,8
ML.9
926
08.3
79.3
Raseline Attention
Baseline Attention
Image
Depth
Attention
Image
Depth
Attention
Q. Are there three oranges in the image?
A: Yes
A: No
Q. there black container on lop of white plate?
A: No
A: Yes
Figure 5: Visual question-ane prestion-answering examples, including image, depth, and query-to-image atten- tion heatmaps before and after applying DSCR. Yellow parts indicate large attention probabilities.
(39.2). Although the recall is 73.4, DSCR demonstrates balanced performance by effectively sup- pressing hallucinations without substantially shortening the average caption length.
AMBER. Table presents evaluation results on the AMBER (Wang et al., 2023) benchmark, which evaluates hallucinations in multimodal LLMs for both generative and discriminative visual reasoning. Specifically, DSCR reduces CHAIRs by approximately 6.35% and improves overall by about 14.2% compared the baseline. Across most category-metric combinations, DSCR con- sistently outperforms both baseline and VCD, demonstrating effective hallucination mitigation and improved grounding quality.
3.2 QUALITATIVE RESULTS
Figure 5 visualizes selected pairs of image, depth map, and query-to- query-to-image attention map before and after applying DSCR. DSCR encourages the model to focus on more relevant regions guided by depth and spatial information. For instance, in the second image, before applying DSCR, the atten- tion is scattered over irrelevant background regions, leading to an incorrect answer. After applying DSCR, the attention is redirected the bicycle, resulting the correct answer. We suggest that DSCR improves attention allocation, ensuring focus on the most relevant regions for answering the query. Additional qualitative results are provided the Appendix.
ANALYSIS
In this section, we present series of ablation experiments to analyze the contribution of each com- ponent in DSCR. Unless specified, experiments conducted the MME hallucination benchmark using LLaVA-IA VA-1.5 1.5. See appendix for more ablations.
4.1 INFERENCE TIME AND RESOURCE USAGE
To evaluate the efficiency of the proposed method, we sample 10 images from the MME validation set and run the full DSCR pipeline (visual encoding, language decoding, cache refinement, and Depth-Anything pr-pprcesssng) five times per image to compute the average inference time. Peak GPU memory is measured across five independent runs and averaged. As shown Table 5, DSCR achieves the fastest per-image runtime among all previous methods while maintaining GPU memory usage on par with alternatives. Furthermore, when DSCR applied on top existing techniques, it delivers clear performance gains with minimal extra overhead. This efficiency arises because DSCR requires only a single forward pass and one-time KV refinement. whereas several alternative methods necessitate multiple inferences for single query (Leng al., 2024b).
8

---

Published as a conference paper at ICLR 2026
Table Inference time and GPU memory consump- tion per image for various methods.
Table COCO Image captioning perfor-
mance with and without DSCR.
Method
Time (sec/img)
GPU Mem. (MiB)
Method
BLEU-
CIDEr
SPICE
Baseline
9.35
29950.3
Baseline
0.122
0.529
0,162
DSCR (Ours)
11.06
32813.7
DSCR
0.235
0.909
0.193
VCD
15.13
29979.6
OPERA
39.37
37717.3
HALC
31.47
32890.7
DAMO
11.80
29965.2
AGLA
23.97
33448.1
Table Hyperparameter settings used across
all models and datasets.
VCD DSCR
16.31
32841.1
OPERA DSCR
42.47
40569.8
HALC DSCR
34.53
36704.4
H.Params
0
B
Layers
DAMO DSCR
13.48
32828.6
Value
0.6
0.6
0.6
0.8
10-39
AGLA + DSCR
25.08
36306.0
Table 8: GPU memory, per-image inference time, and performance comparisons of DSCR using different depth estimators.
Depth Model
(MiB)
Time (sec/img)
OCR
Color
Count
Existence
Position
Posters
Depth-Any Angth-Anythhnn--22 (Yang et al., 2024b)
2134.1
1.34
132.5
175.0
160.0
195.0
120.0
140.48
MiDaS-Lic Lite (Ranlll et al., 2020)
1264.2
1.15
125.0
180.0
160.0
195.0
111.67
132.65
DPT-Lite (Ranftl et al., 2021)
526.1
1.54
125.0
180.0
160.0
195.0
111.67
132.65
4.2 GENERALIZATION TO GENERAL VL TASKS
While many hallucinationn-iitiatiio techniques require retraining or introduce trade-off's that com- promise overall task performance, DSCR is entirely training- ffree and even slightly improves the model's core captioning ability. On the COCO image captioning task (Table 6), DSCR shows im- provements of +0.1 13 in BLEU-4, +0.380 in CIDEr, and +0.031 in SPICE compared to the LLaVA- 1.5 baseline. These results show that DSCR effectively suppresses hallucinations while maintaining and even enhancing overall text generation quality for standard vision-language benchmarks.
4.3 DEPTH ESTIMATION MODELS AND HYPERPARAMETER SENSITIVITY
In Table 8, we evaluate DSCR with three monocular depth estimators (i.e., Depth-Anything v2, MiDaS-Lite, and DPT-Lite). Despite variations in GPU memory usage (526-2134 MiB) and infer- ence time (1.15-1.54 per image), the key metrics remained within +7%. This minimal performance fluctuation across different depth-estimation models demonstrates that DSCR consistently mitigates hallucinations even when using noisy depth maps
extremely lightweight model.
For all experiments using DSCR, we adopt the fixed hyperparameter settings listed in Table This justified by the observation that method is not sensitive to hyperparameter choices, showing less than 5% performance variation across alternative settings. For detailed results on all tested hyperparametee combinations, please refer the Appendix.
CONCLUSION
In this paper, we proposed DSCR, a novel training-free KV cache refinement method guided by depth and spatial cues. Our method mitigates hallucinations vision-language models by reallo- cating KV vectors based on geometric and spatial consistency. To validate its effectiveness, we conducted comprehensive experiments on five ballucination benchmarks (MME, POPE, RePOPE, CHAIR, and AMBER), achieving up to 41.6% accuracy improvements, and can be integrated on top of existing methods. We further introduced a depth confusion mini-benchmark, specifically de- signed to evaluate cases where multiple objects overlap share similar depths. Even when depth models produced inaccurate predictions, DSCR consistently improved performance by guiding at- tention semantically meaningful regions. To the best of our knowledge, DSCR is the first to refine the KV cache using auxiliary geometric cues (i.e., depth and position), making it a practical and generalizable plug-in for hallucination mitigation.

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea governmen((MSIT) (RS-2024-00398157) and by In- stitute of Information & communications Technology Planning & Evaluation (ITTP) under Next- generation Cloud-native Cellular Network Leadership Program QUIT--2022-RS-2224-00011133 grant funded by the Korea government(MSIT).
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nic, Haonan Lin, Qianying Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. Mitigating object hallucinations in large vision-language models with assembly of global and local attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29915-299288 June 2025.
Anthropic. System Card: Claude Opus 4 and Claude Sonnet 4. https: http:://wwwwatthrpppo.o com/news/cll wtwps///laade-ia 2025. Accessed: May 2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint (tx.ix:308..2966 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Yc, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar 2502 13923, 2025.
Stan Birchfield and Carlo Tomasi. Depth discontinuities by pixel-to-pixel stereo. International Journal of Computer Vision, 35(3):269-293, 1999.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2023.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial rea- soning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35, 2025.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.0044 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-241999 2024c.
Beomsik Cho and Jachyung Kim. Revisit what you see: Disclose language prior in vision tokens for efficient guided decoding of lvims. arXiv preprint arXiv:2506.09522 2025.
10

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge. Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-gra comprehension and generation, 2025. URL https::////... arriv org/ abs 14396.
Google Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL abs/2507 06261.
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving Vision-Language-Attnnn model with online reinforcement learning. IEEE In- ternational Conference on Robotics and Automation, pp. 15665-15672. 5 2025.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang. Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2), January 2025. ISSN 1046-8188.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in modli large language models via over-trust penalty and caarspectiionaalocattin on Computer Vision and Puttern Recognition, pp. 73418-13427, 2024.
Proceedings of the IEEE/CVF Conference
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27992 2024.
Schoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W Mahoney, al. Full stack optimization of transformer inference. In Architecture and System Support for Transformer Models (ASSYST@ ISCA 2023), 2023.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13872-13882 2024a.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations large vision- lanoualanguaae models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 18872138822 2024b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742. PMLR, 2023a.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang. Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023b.
Yawei Li, Jay Pang, Lu Yuan, and Yunchao Li. Localvit: Analyzing locality in vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallu- cination in large vision- language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023c.
11

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:// llava-vl. vll. git.bb........... ho/blog///// 2024 10224-00 01-30-----00000000000
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2024c. URL https: // //openzeview....
Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Efficient inference of vision mnstruction-folloiing models with elastic cache. arXiv preprint arXiv:2407.1812) 2024d.
Yannic Neuhaus and Matthias Hein. Repope: Impact of annotation errors on the pope benchmark, 2025. URL https: //axivv org/ abs 22004 15707.
Payam Nikzad, Xinyu Chen, Arjun Kumar, and Others. Sata: Spatial autocorrelation token analy- sis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. ScreenAgent: a Vision Language model-driven computer control agent. Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, pp. 6433-6441, 2024.
OpenAL. GPT-5 System Card. https:/ /t////nnnnoopnn. cwm/gpt 5- system- card. pdf, 2025. Accessed: 13..-08-13.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- lions pattern analysis and machine intelligence, 44(3):1623-16) 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense predic- tion. Proceedings of the IEEE/CVF international conference on computer vision, 10 2021. doi: 10. (0.10000ccc44922222000011 196. URL http://///oo...//1111101000/ 2021. 01196.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 4035-4045, October 2018
Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha A comprehensive survey of hallucination in large language, image, video and audio foundation models. In Findings of the Association Computational Linguistics: EMNLP 2024, pp. 11709- 11724, 2024.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Haozhan Shen, Tiancheng Zhao, Mingwei Zhu, and Jianwei Yin. Groundvip: Harnessing zero-shot visual grounding from Aision-language pre- voo-rriinig and open- vpca-vocyy object detection. arXiv preprint ar 2rxiv:23/2. 2023.
Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, and Liang Lin. Towards long- horizon vision- Languaea navigation: Platform, benchmark and method. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2025.
12

---

Published as a conference paper at ICLR 2026
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 2301:22309. 2023.
OpenAl team. Gpt-4o system card, 2024. URL https: /tt:://aaxxiv. http:///wwwww...r///////.......
Carlo Tomasi and Roberto Manduchi. Depth discontinuities by pixel-to-pixel stereo. International Journal of Computer Vision, 35(3):269-284, 1999.
Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. Vl-cache: Sparsity and modality- aware kv cache compression for vision-language model inference acceleration. arXiv preprint arXiv:2410.23317, 2024.
David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 198 215, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72286-7
Zhongwei Wan. Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. LOOK-M: Look-once optimization in KV cache for efficient multimodal long-context inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Asso- ciation for Computational Linguistics: EMNLP 2024, pp. 4065
4065-4078, 4078, 2024.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimmensonal benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07397 2023.
Kaishen Wang, Hengrui Gu, Meijun Gao, and Kaixiong Zhou. DAMO: Decoding by accumulating activations momentum for mitigating hallucinations in vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Rep- resentations, 2024.
Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concen- tric causal attention. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yanming Xiu, Tim Scargill, and Maria Gorlatova. ViDDAR: Vision language model-based task- detrimental content detection for augmented reality. IEEE Transactions on Visualization and Computer Graphics, 3145)3319433203, 2025.
Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, Chen Luo, Qingyu Yin, Bing Yin, Long Chen, and Yangqiu Song. MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding. Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7800-7815, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.094 09000009444,... 2024b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owll ow12: Revolutionizing multi-modal large language model with modality collabo- ration. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Bing- Wei Yin, Li Zhang, Hao Sun, and Others. Dformerv2: Geometry self-attention for rgb-d semantic segmentation. In International Conference Learning Representations (ICLR), 2025.
13

---

Published as a conference paper at ICLR 2026
Rumpeng Yu, Weihao Yu, and Xinchao Wang. Api: Attention prompting image for large vision- language models. In European Conference on Computer Vision, 2024.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: vision-language model for spatial affordance prediction for robotics. arXiv preprini arXiv:2406. 10721, 2024.
Yang Zhan, Yuan Yuan, and Zhitong Xiong. Mono3dvg: 3d visual grounding in monocular images. arXiv preprint arXiv:2312.0801 2023.
Shuai Zheng, Sadeep Jayasumana, Bermardino Comera-Paredess Vibhav Vinect, Zhizhong Su, Da- long Du, Chang Huang, and Philip HS Torr. Conditional random fields recurrent neural net- works. In Proceedings of the IEEE international conference on computer vision, pp. 1529-1537, 2015.
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2026
A PRELIMINARIES
Internal Self-Attention Mechanism. Self-attention (SA) module within each Transformer layer the core component that incorporates past context to produce features for the current input. The operation of SA for each attention head can be simplified as:
KK KKIK*, K*; K*
Ct
[r..
Softmax
vdh
where K- and K- represent the key caches corresponding to question and current answer tokens, respectively. Similarly, VX and V denote the associated value caches. This operation can be di- vided into three steps: 1) calculate the similarity between the current token's query (4y (Rdh) and the keys of all previous tokens, 2) apply the softmax function the similarity scores to obtain atten- tion probabilities, and 3) aggregate the values of all previous tokens by taking a weighted sum based on the attention probabilities. In essence, modification of K affects the attention probabilities, while that of V changes the output of the self-attention operation.
Text Generation Process. Similar to L.MMs, VLMS operate in an autoregressive manner, perform- ing the next-token prediction to generate responses. Consider visual question-answering setup where a VLM generates an answer Y = .. ...,.'" .yr To generate t-th token yt of the answer, VLM computes the probability of the next token as:
to the provided query {x1,--. ".,,"",,M,
I, = P 11 K v(.N).
(8)
This indicates that the generated text depends the question X and KV cache derived from the image tokens. Notably, once the KV cache is computed, users can change the query to ask different questions about the same image without recomputing the KV cache (see Figure 2 (b)). This property referred to as query-agnostic behavior.
B RELATED WORK
B.1 HALLUCINATION IN VLMS
Visual hallucination in VLMs refers to the generation of content that is not grounded in the vi- sual input, such as nonexistent objects or incorrect attributes. To mitigate this problem, various strategies have been proposed across model components, including improved pretraining data (Zhou et al., 2024), larger more expressive vision encoders (Liu al., 2024a: Chen et al., 2024c), and decoding-level interventions (Yu et al., 2024; Leng et al., 2024a). Among the latter, OPERA (Yu al., 2024) penalizes overconfident attention weights during decoding suppress ungrounded re- sponses, while VCD (Leng et al., 2024a) filters hallucinations by comparing outputs from perturbed and original images through contrastive decoding.
B.2 KEY-VALUE CACIIE MANIPULATION
As context length increases significantly, LLM suffers from the memory bottleneck (Kim et al., 2023). order to alleviate memory overhead and reduce inference time, various techniques have been proposed to optimize the KV cache (Bolya et al., 2023; Chen et al., 2025; Wan et al., 2024; Liu et al., 2024d; Tu et al., 2024). For example, sparsity- cache allocation (Tu et al., 2024), attention based pruning (Wan et al., 2024). prioritizing recent tokens (Liu et al., 2024d), and differentiating between visual and text tokens (Tu et al., 2024) have been explored. While these methods have successfully addressed issues involved in efficiency, they have predominantly focused on compressing the cache itself rather than enhancing the representation within the cache. To our knowledge, there has been no prior work in either vision or language domains that integrates addi- tional information into the KV cache to improve its representational strength.
QQesstinn tokens also generate corresponding KV cache in practical systems, but omit the details here. See I.IM inference literature (Kim al., 2023).
15

---

Published as a conference paper at ICLR 2026
Absolute Depth Perception
Q. Fiind the closest object in the image. Pronee Please sollet the connect answer (oniy give the key such lat, (0t 1r").
Q Find the firthest objectt the image.
Prosutt: Please solect the conrect touly give the key such as W. 'b; le, W').
Glass
Hund
DESBEAT
shaker
Dessert sign
Piemi
Phone
Perspective-Awe Aware Size Perception
Q Which object would largest in real life if placed at the some distanec?
Q. object would the
real life placed the same distunce!
Promptt i Please select the correct answer (owdl give the key such Vi, W, (').. Cup
Prompe Please
connect
(onily give the key such as 'al, 1o Person
Menu bound
Glass
d. Salt shicke
Salt shikk
The left shows the input image, and the right displays four sample questions grouped by reasoning type. Top: Questions assessing Absolute Depth Perception (e.g., identifying the closest or farthest object in the scene). Bottom: Questions assessing PerspectiveAAwae Size Perception, which test whether the model can infer real-world object size based on perspective. Correct answers are highlighted in green.
Figure 6: Example from the Depth Hallucination
B.3 VISUAL GROUNDING IN VISTON LANGUALA MODELS
Strengthening visual grounding is a central objective for VLMs, and a variety of approaches have been proposed toward this goal. One line of work performs spatial and geometric grounding by tying language to 3D scene structure or monocular 3D object extents. For example, SpatialVLM (Chen al., 2024a) trains vision-language model on large-scale synthetic spatial question and answer data generated from real images and metric 3D representations, which improves quantitative reason- ing about distances, sizes, and relative positions. Mono3DVG (Zhan et al., 2023) defines 3D visual grounding in monocular RGB images and proposes transformer-based model that jointly exploits appearance features, a dedicated depth predictor, and geometry-aware text embeddings to localize the full 3D extent of referred objects. Another line of work enhances grounding by aligning textual tokens or phrases with image regions and explicitly reusing vision tokens as evidence when scoring outputs. GroundVLPP (Shen al., 2023) fuses Grad-CAM heatmaps from vision-language back- bone with region proposals from open- vecabvlcayulary detectors to achieve zero-shot phrase grounding without task sse-iipcciic grounding annotations. Re ReVisi (Cho & Kim, 2025) introduces training-fre free decoding strategy that projects vision tokens into the text-token distribution space, dynamically se- lects the most relevant vision token at each decoding step, and uses it refine the output distribution, thereby reducing over-reliance on language priors and improving visual grounding. These methods share the common aim of tightly coupling linguistic expressions with concrete visual entities, typi- cally through additional training modified decoding. In contrast, DSCR does not explicitly align language with individual objects or regions and does not introduce new training or decoding stages, but instead refines internal visual key-value representations at inference time using depth and spa- tial priors so that hallucinations are mitigated while the backbone parameters and decoding pipeline remain unchanged.
C EXPERIMENT SETUP
C.1 DEPTH HALLUCINATION MINI- HINI-BENCHMARK
To evaluate the depth perception capabilities of vision-language models, we construct a new dataset called the Depth Hallucination Mini-Benchhark. The benchmark is designed to assess two key evaluation objectives. First, it tests whether the model can reason about absolute distances be- tween objects; and second, whether it can perceive relative depth under perspective while preserving knowledge of object identity. The dataset comprises 50 images manually selected from 500 candi- dates in the COCO val2014 dataset (Lin al., 2014). Images were chosen based on two criteria: they contain at least two distinct objects, and they exhibit clear perspective information, allowing for meaningful depth comparison.
16

---

Published as a conference paper at ICLR 2026
For each image, generate four depth reethhrelated questions: identifying the closest object, the farthest object, the largest object in actual size, and the smallest object in actual size. We use GPT-4o (team, 2024) to generate multiple-choice answers for each question, including correct answer and three distractors. All generated choices are subsequently verified by humans to ensure that the objects exist in the image and that the correct answers indeed valid.
We categorize the questions into two types based on their intended depth reasoning objective.
Absolute Depth Perception includes questions about the closest and farthest objects. The purpose of this category evaluate whether the model can identify objects based on their actual distance from the camera.
Perspective-Aware Size Perception includes questions about the largest and smallest ob- jects in actual size. This category assesses whether the model can reason about visual scale and preserve knowledge of object identity under perspective dSstortion.See Figure for an example from the dataset.)
C.2 DATASET
MME. The MME (Fu et al., 2023) dataset is a comprehensive benchmark designed to evaluate the performance of VLMs across various aspects including fine- ine-ggrined visual cognition, visual percep- tion, and OCR. Since our work focuses on mitigating hallucinations, we utilize subsets of MME. Specifically, we employ the existence and count subsets evaluate object-level hallucinations, and use position and color subsets to assess attribute-leve hallucinations, Each subset consists of "Yes- or-No" questions, providing a straightforward assessment of VLM's ability to recognize objects and their attributes.
POPE RePOPE. The POPE (Li et al., 2023c) benchmark is specifically crafted evaluate the hallucination in VLMs. The evaluation targets object-level visual hallucinations, particularly focusing on the existence of objects in a visual scene. The dataset consists of binary classification questions for the target object, which may or may not appear in the image. Targets are selected based on three distinct sampling settings: random, popular, and adversarial. In the random setting, non-existent objects selected randomly. In the popular setting, non-existent objects chosen from pool of frequently appearing objects in the dataset. In the adversarial setting, objects that commonly co-appear but do not actually exist in image selected to challenge the model's perception.
RePOPE (Neuhaus & Hein, 2025) is relaheled version of the POPE benchmark that corrects anno- tation errors in the MSCOCO dataset. The POPE and RePOPE datasets each comprise 500 images, with six associated questions per image. The evaluation metrics include accuracy, precision, recall, and Fl-score.
CHAIR. The Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach al., 2018) is a metric for evaluating object hallucination in image captioning. It measures the extent to which generated captions refer objects that are not present in the ground-truth annotations of the corresponding image. CHAIR consists of two components: CHAIRs, which evaluates hallucination the sentence level, and CHAIRi, which measures Lower scores on both components reflect improved grounding of the caption to visual content. For evaluation, 500 images are randomly sampled from the COCO 2014 validation set. Captions are generated using various VLMs prompted with "Please describe this image in detail," with fixed maximum token limit to ensure fair comparison.
the instance level across multiple captions.
We conduct evaluation on the AMBER (Wang et al., 2023) benchmark, recently pro- posed dataset designed to precisely assess hallucination in vasion-language models. AMBER con- sists of natural image-question pairs across five task types: object existence, counting, positional reasoning, color recognition, and text reading. Each question constructed such that the correct answer is verifiable based on the visual content alone, and any incorrect answer lacking visual grounding is regarded as a hallucination. The dataset includes a diverse range of image sources such real-world photos, diagrams, and scene renderings, allowing for comprehensive evaluation
AMBER.
17

---

Published as a conference paper at ICLR 2026
Table 9; Evaluation results on our Depth Hallucination Mini-Benchhare
Method
Abs. Depth Score
Persp.-Aware Size Score
Average Score
Baseline
60.0
51.0
53.5
VCD (Leng el al., 2024b)
73.0
68.0
70.5
OPERA (Huang et al., 2024)
78.0
73.0
75.5
DSCR (Ours)
82.0
75.0
78.5
VCD Ours
77.0
67.0
72.0
OPERA + Ours
80.0
72.0
76.0
across different visual domains. The primary evaluation metric is accuracy, which directly reflects model's ability to generate visually grounded responses and avoid hallucinated outputs.
C.3 COMPUTATIONAL RESOURCES
All experiments were conducted on machine equipped with three NVIDIA A6000 GPUs (48GB each), an Intel Xeon Gold 6526Y processor with 32 threads (16 physical cores), and 754GB of RAM. Our method was evaluated solely in the inference setting, with no additional training fine-tuning.
C.4 MODELS
We employ the ktate-of-the-ar MDE model. Depth-Anything-v? (Yang et al., 2024b), due to its efficient inference cost and accurate depth predictions across a wide range of images. As for base- lines, we popularly used VLMs including LLaVA-1.5 (Liu al., 2024a), LLaVA-1.6 (Liu al., (Ye et al., 2024), Qwen-VL (Bai et al., 2023), and Qwen2.5-VL (Bai et al., 2025). All three models follow the common "vision emcoder-interffce-lamm language model" framework, which utilizes a pre-trained visual encoder to extract visual tokens. All experiments were conducted with fixed hyperparameters de 0.6, o. 0.6, a 0.6, $ 0.8. and layers 10-39.
C.5 IMPLEMENTATION DETAILS
We reproduce both OPERA and VCD using their official implementations and verify that our re- implementation matches the original results. During reproduction, we encountered inconsisten- cies due version differences in the Hugging Face Transformers library. To ensure compatibility and consistency, standardize all experiments, including DSCR, using Transformezs version 1.3,.0, Our DSCR implementation applies cache refinement across all layers simultaneously using optimized tensor operations, which eliminate additional overhead. As result, inference is efficient and takes approximately to 3 seconds per image.
D ADDITIONAL EXPERIMENTS
D.1 RESULTS ON DEPTH HALLUCINATION MINI- BENCHMARK
As shown in Table 9, DSCR outperforms the baseline on the Depth Hallucination Mini-Benchmatt improves the absolute depth perception score by 36.7% and the perspective-aware size perception score by 47.1%. 1%, resulting in a total score improvement of 41 1.4%. DSCR also surpasses VCD (Leng al., 2024b) and OPERA (Huang et al., 2024) by 8.0% and 3.0%, respectively. When used as an add-on, DSCR further increases the total score of VCD by 1.5% and OPERA by 0.5%. From the experimental results, we confirm that DSCR effective both as standalone method and as cascade component that enhances existing hallucination mitigation approaches.
D.2 RESULTS ON POPE
To further evaluate the generalizability of our method, conduct experiments on the POPE bench- mark, which measures robustness to visual hallucinations across MSCOCO (Lin et al., 2014), A- OKVQA (Schwenk et al., 2022), and GQA (Hudson & Manning, 2019) datasets. We apply DSCR
18

---

Published as a conference paper at ICLR 2026
Table 10: Evaluation results on the CHAIR dataset using mPLUG-Owl2 model.
Method
CHAIRs 1
CHAIR
Recall
Avg. Len.
Baseline
38.6
11.6
73.0
88.5
VCD
45.4
13.6
68.6
88.9
OPERA
35.6
11.6
71.8
86.3
DSCR
35.6
11.2
69.3
77.0
Table 11: Evaluation results the VQAv2 dataset using LLaVA-1.5 and Qwen2.5-VL.
1a-15
Qwen2.5-VL
Metric
Baseline
DSCR
Baseline
DSCR
Overall Accuracy
79.87
80.13
84.27
84.73
three vision- languaee models: LLaVA-1.5, Qwen-VL, and mPLUG-Ow12, under Random, Pop- ular, and Adversarial settings.
As shown in Table 19, DSCR consistently improves all evaluation metries, including accuracy, preci- sion, recall, and Fi-score, across different models and settings. For example, in the Random setting on the MSCOCO dataset, LLaVA-1.5 improves Fl-score from 0.82 to 0.85, Qwen-VI. from 0.80 to 0.84, and mPLUG-Ow12 from 0.83 to 0.86. Similar gains are observed under the Popular and Adversarial conditions, demonstrating that DSCR robustly enhances grounding across diverse hal- lucination scenarios. These results confirm the effectiveness of DSCR in reducing hallucinations and improving the reliability of vision-language models across various architectures and conditions.
D.3 COMPARISON TO OTHER METHODS
To validate the effectiveness and Blug-and-yy applicability DSCR, we compare it with existing methods for mitigating visual hallucinations on the POPE benchmark. Tables 20, 21, and 22 report the results of recent approaches, including VCD, OPERA, and ours, terms of accuracy, precision, recall, and Fl-score across various settings.
Overall, DSCR consistently improves performance across all vision-language models and evalua- tion scenarios when applied to either VCD or OPERA. These improvements are especially notable challenging conditions such as the Adversarial setting, where hallucination risk is elevated due to co-occurrence biases. Notably, on the MSCOCO dataset under the Random setting, applying DSCR to OPERA on mPLUG-OWl improves the Fi-score from 0.81 to 0.87, reflecting a relative improvement of approximately 7.4%. These results highlight the effectiveness of DSCR a gen- eral and model-agnostic refinement strategy that can be seamlessly integrated into existing methods, improving hallucination robustness without modifying model parameters.
D.4 RESULT ON CHAIR
Table 10 presents evaluation results the CHAIR benchmark using the mPLUG-Ow12 model. Consistent with the results in Table 3 using the LLaVA-1.5 model, DSCR achieves the lowest hal- lucination scores (CHAIRs: 34.6, CHAIR;: 10.3). Although the recall (70.3) is slightly lower than that of OPERA, DSCR effectively suppresses hallucinations while maintaining competitive recall performance. These results suggest that the effectiveness of DSCR in reducing hallucinations gen- cralizes across different backbone models.
D.5 RESULT ON VQAv2
To investigate the effect of DSCR on general VL tasks, we conduct additional experiments on question-answenin task.ca Table 11 shows evaluation results on the VQAv2 dataset. We randomly sample 500 examples from the VQAv2 validation set and evaluate the overall accuracy of base- line models and DSCR on this subset. In both LLaVa-1.5 and Qwen-2.5-VI. DSCR preserves the
19

---

Published as a conference paper at ICLR 2026
Table 12: Ablation study for varying the size of depth estimation model, conducted on the MME dataset using LLaVA-1.5 model.
Object-level
Attribute-level
Model
Type (Size)
Total
Existence
Count
Position
Color
173.33
116.66
113.33
123.33
526.66
Depth-Anything-v2
Small (0.03B)
190.00
153.33
120.00
170.00
633.33
Depth-Anything-v2
Basc (0.1B)
190.00
160.00
120.00
170.00
640.00
Deptt-Annylhing-22
Large (0.3B)
195.00
160.00
120.00
175.00
650.00
Table 13: Ablation study on the effect of o, the size of the Gaussian kernel.
Table 14: Ablation study on the selection of Key and Value Cache.
Object-level
Attribute-level
Sigma
Total
Object-level
Attribute-level
Existence
Count
Position
Color
Setting
Total
Existence
Count
Position
Color
0.2
190.00
153.33
120.00
170.00
633.33
Value-only
190.00
160.00
120.00
170.00
640.00
190.00
155.00
120.00
170.00
635.00
Key-Valus
195.00
160.00
120.00
175.00
650.00
160.00
Key-only
175.00
190.00
170.00
Table 16: Ablation study on the position of layers Table 15: Ablation study on the combination of to apply DSCR. external sources.
Object-level
Attribute-level
Layers
Total
Setup
Object-level
Atteiblut-level
Total
Existence
Count
Position
Color
Existence
Count
Position
Color
10-20
190.00
158.33
120.00
170.00
638.35
Depth (D)
195.00
160.00
120.00
170.00
645.00
Spatial (S)
195.00
160.00
120.00
170.00
645.00
20-30
190.00
153.33
120.00
170.00
633.33
& S
195.00
160.00
120.00
175.00
650.00
20-40
190.00
160.00
120.00
170.00
640.00
10-39
195.00
160.00
120.00
175.00
650.00
overall accuracy compared the baseline models. These results demonstrate that DSCR mitigates hallucination without degrading the models' general VL capabilities.
E
ABLATION STUDIES
E.
DEPTH MODEL. SIZE
We conduct ablation study to analyze the effect of the quality of estimated depth by varying the size of the monocular depth estimation model. Table 12 presents the results for different versions of the Depth-Anything--22 (Yang et al., 2024b) model, ranging from Small (0.03B) Large (0.3B) size models. The results indicate that depth information has considerable impact on reducing hallucinations VLMs. Notably, even the smallest depth model provides substantial performance boost, demonstrating depth information's 's critical role in enhancing visual representations.
E.1.1 DEPTH VS. SPATIAL PROXIMITY
Table 15 reports results for depth-only, spatial-only, and combined depth&spatial weighting. The combined "D&S" setting yields the best total score of 650.00, confirming that depth cues and 2D proximity complement each other.
E.2
HYPERPARAMETER SETTING
We conduct ablation studies focusing on four key aspects: the scale of the Gaussian function for determining adjustment sensitivity, the strategy for cache refinement (modifying the Key, Value, or
20

---

Published as a conference paper at ICLR 2026
Table 17: The architecture comparison for the VLMs and the MDE model used in the experiments.
Vision Encoder
LLM
Model
Type
Size
Input
Type
Size
LLaVA-1.5 al., 2024c)
VIT-L/14
0.3B
336x336
Vicuna
Qwen-VI. (Bai al., 2023)
VIT-G/14
1.9B
336x336
Qwen
7B
mPLUG-Owl2 (Ye et al., 2024)
VIT-L/14
0.3B
336x336
LLaMA
7B
Depth-Anything-12 (Yang et al., 2024b)
VIT-L/14
0.3B
518x518
both), and the selection of Transformer layer positions for modification. All subsequent experiments utilize LLaVA-1.5 on the MME hallucination benchmark.
E.3 GAUSSIAN KERNEL SIZE
We first evaluate the effect of the Gaussian scale parameters da and a, from Eq. (1) and Eq. (2), setting = 03 for simplicity. Here, controls how broadly neighboring tokens influence each other in the cache refinement. As shown in Table 13, a=0.6 provides the best balance between local detail sensitivity and cache smoothness.
E.4 KEY-V REFINEMENT STRATEGY
Next, we compare three strategies: Value-only, Key-only, and Key+Value, as shown in Table 14. While all three settings improve performance, Key-only refinement achieves the highest score, slightly outperforming the Vey+Value variant. This implies that key vectors play a more critical role in mitigating hallucinations, and that updating only the Key can provide an efficient trade-off' between performance and computational cost.
E.5 REFINEMENT LAYER RANGE
Finally, we investigate which layers benefit most from cache refinement. As shown in Table 16, applying KV cache refinement to layers 10-39 provides the best performance, suggesting that mid- to-high Transformer layers are most effective targets for reducing hallucinations.
E.6 QUALITATIVE ANALYSIS OF FAILURE AND SUCCESS CASES
To better understand the behavior of DSCR under challenging scenarios, present qualitative analyses of three samples using various depth estimators (Depth-Anything v2, DPT-Lite, MiDaS- Lite), shown in Figure
In the first case, all models fail to predict the correct answer, including DSCR, Although the depth- based refinement encourages attention toward text regions, the baseline model lacks sufficient lan- guage capability to correctly interpret the text content. This highlights limitation of DSCR- it can guide attention semantically relevant regions, but cannot compensate for fundamental weaknesses in the underlying language reasoning.
In contrast, the second and third examples show how DSCR successfully mitigates hallucinations. Even when lightweight depth models like DPT-Lite and MiDaS-Lite generate noisy low-quality depth maps, DSCR still improves prediction by redirecting focus to plausible object regions. When more accurate depth models (e.g., Depth-Anything v2) used, performance further improves.
Importantly, DSCR remains compatible with lightweight depth estimators, enabling deployment in real-world applications where compute resources limited. Moreover, in practical scenarios involving depth cameras (e.g., AR glasses or mobile devices), depth can be directly obtained without requiring additional prediction model--ptenntialyy leading even more reliable results.
21

---

Published as a conference paper at ICLR 2026
Q: Is this movic titled predator (1987)?
Please answer yes or no.
Baseline:
image
Depti-Anythingy2
Depth-Anything22 Answer: No
DPT-Lite Answer: No
MiDaS-Lite Answer: No
GT Answer: Yes
DPT-Lite
MiDaS-Lite
Bus Times 6363
0131
Q: phone number the picture "0131 555 6363"? Please answer yes or no.?
555
Baseline: No
Image
Deplh-Anythingv2
Depth-Anythingy2 Answer: Yes
DPT-Litc Answer: No
MiDaS-Lite Answer: No
GT Answer: Yes
DPT-Lite
MiDaS-Lite
WORE BILLESS DOLISO BOS
Q: Is this movie directed by oliver stone?
Please answer yes no.?
WALL STRKET
Baseline:
Image
Depth-Anythingv2
Depth-Anythingy2 Answer: Yes
DPT-Lite Answer: No
MiDaS-Lite Answer No
GT Answer: Yes
DPT-Lite
MiDaS-Lite
Figure 7: Qualitative examples show the effect DSCR across different depth estimation condi- tions. (Top) All models, including DSCR, fail due to the baseline's inability to interpret the text despite depth-guided attention. (Middle, Bottom) DSCR correctly answers the question by focusing on relevant regions, even when using noisy depth maps from lightweight models like DPT-Lite or MiDaS-Lite. Performance further improves with more accurate depths (e.g., Depth-Anything v2).
E.7 ADDITIONAL COST OF DEPTH ESTIMATION
We show that the additional cost of DSCR is minimal. As detailed in Table 17, Depth-Anything-v2 model employed for depth estimation contains 0.3B parameters, comparable to the vision encoders and significantly smaller than LLMSs used in VLMs.. Additionally, the depth estimation process
22

---

Published as a conference paper at ICLR 2026
Hallucination
Non- Hon-HHllucination
Conventional Methods
DSCR (Ours)
Conventional Methods
DSCR (Ours)
Image
Image
Figure 8: Key vector visualizations for hallucinated (left) and non aonhhallucinttd (right) cases. Each group shows the input image, key vector visualization from conventional methods, and key vector visualization from DSCR (ours). In hallucinated cases, conventional methods produce disorganized key vectors, while DSCR yields more semantically aligned representations. In non-hallucinated examples, both methods generate coherent vectors, but DSCR maintains sharper object boundaries and spatial structure.
with the MDE model is a one-time operation per image, requiring only a few tens of milliseconds on GPU (Yang et al., 2024a). This implies that integrating DSCR into existing VLM frameworks introduces negligible computational overhead, thereby maintaining the overall system's performance and scalability.
E.8 MORE QUALITATIVE RESULTS
Figure extends the key vector visualizations shown in Figure to broader set of examples. The left half shows hallucinated cases, while the right half displays non-hallucinated ones. Each row presents the input image, the key vector visualization from conventional methods, and the corre- sponding visualization after applying DSCR. In hallucinated cases, conventional methods exhibit disorganized and noisy vector patterns, whereas DSCR produces more structured representations with clearer object boundaries and spatial organization. These results qualitatively support the ef- fectiveness of DSCR enhancing visual coherence and improving object-centric reasoning.
Figure 9 visualizes additional examples. Results show that DSCR successfully identifies the im- portant objects and encourages attention mechanism to focus more on the relevant regions. For example, in the first and last rows of the Figure 9, we observe that object (dog, bicycle) region stands out after applying DSCR; in contrast, the original model pays more attention to background and unrelated image patches.
We empirically observe that attention is biased towards first several image tokens, represented as highlighted upper regions in the attention heatmap. We assume that this is partially because LLM processes image tokens sequentially. Please note that similar phenomenon, termed Attention Sink, has been also discovered in text-only LLMSs (Xiao et al., 2024).
23

---

Published as a conference paper at ICLR 2026
Baseline Attention
DSCR Attention
Baseline
DSCR Attention
Image
Depth
Image
Depth
Attention
Qu Is there cat sitting on top suitcase?
A: No
Q. Is the bus white blue?
A: Yes
A: No
PEBSREH PICA
Q. this poster for Primal Rage?
A: No
A: Yes
Is there large sign that says "PENARTH PIER"?
A: No
A: Yes
MOTEL
Q. Is the name of this movie
A: Yes
A: No
Q. there sign that suys "SeaBrecze MOTEL"?
No
Yes
2 Fast 2 Furious?
Figure 9: VQA examples, including image, depth, and query-to-image attention heatmaps before and after applying DSCR.
Hallucination Layer 15
Image
Layer
Layer
Layer 10
Layer
Layer
Layer
Layer
Layer
Conventional Methods
DSCR
Hon-Hllllulnnnionn
Image
Layer
s
Layer
Layer
Layer
2.5
Layer 30
Layer 35
Layer
Conventional Methods
DSCR
Figure 10: PCA visualizations of key vectors across layers for a hallucination example (top) and non-hallucination example (bottom). each block, the top row is the baseline model and the bottom row is DSCR. The visualization shows that DSCR produces more object-aligned patterns and clearer separation between foreground and background, especially in middle and upper layers.
Figure shows how DSCR changes the key vectors across layers. For hallucination-occumm and non-hallucinated examples, project visual key vectors at layers 0, 5, 10. 15, 20, 25, 30, 35, and 39 of the baseline model and from DSCR. Since DSCR is applied to layers 10-39, the visualizations at layers 0 and are identical in both rows. From layer 10 onward, the bascline shows stripe-like like or speckled patterns without a clear object shape, especially in the hallucination case. In contrast, DSCR shows interpretable patterns where object contours and the background are clearly separated
24

---

Published as a conference paper at ICLR 2026
Table 18: Boundary Contrast (BC) statistics on the MME hallucination subsets and the Depth Hal- lucination Minii-ennchmrk. For the LLaVA-1.5 model. DSCR increases BC and decreases BC on hallucination--i fix cases (i.e., corrected by DSCR), which leads to a larger overall BC difference compared to the baseline. There is no cases such that DSCR reverted the correct answer.
Split
Method
BC
BC
BC
Baseline DSCR
0.71
0.64
0.07
Baseline wrong, DSCR correct
0.87
0.57
0.30
Baseline
0.69
0,63
0.06
Both wrong
DSCR
0.73
0.60
0.13
Bascline
0.82
0.61
0.21
Both correct
DSCR
0.88
0.60
0.28
Bascline
n/a
n/a
n/a
Baseline correct, DSCR wrong
DSCR
n/a
n/a
n/a
across multiple layers. In the non-hallueination case, the baseline already reveals a rough object structure. DSCR further sharpens the object silhouette and suppresses background variation. These layer-wise visualizations support our claim that DSCR keeps early visual representations unchanged and restores coherent, object-centered key structure in the mid-to-deep layers where it is applied.
E.9 RELATIONSHIP TO CONDITIONAL. RANDOM FIELD
The proposed DSCR method shares similarities with the Fully-Connected Conditional Random Field (FC-CRF) algorithm, widely used in vision applications such as image segmentation (Zheng et al., 2015; Chen et al., 2017). In short, FC-CRF also utilizes Gaussian functions to convert dis- tances into proximity scores. However, while FC-CRF utilizes label prediction probabilities and pixel intensity for proximity computation, DSCR leverages depth information to guide cache refine- ment. Moreover, FC-CRF requires multiple iterations and heavy computations to achieve optimal results without changing the model's internal representation. In contrast, DSCR accomplishes ef- fective cache refinement with a single computation step across all target layers.
E.10 BOUNDARY CONTRAST ANALYSIS
To quantitatively investigate whether DSCR reflects object boundaries in the key vector space, we introduce depth-defined Boundary Contrast (BC) metric. This metric compensates the previous PCA-based qualitative evaluation. essence, we leverage depth maps as a proxy for identifying ob- ject boundaries in the image plane. Large depth difference between neighboring patches implies the existence of object boundary. We then measure whether key vectors along these proxy boundaries exhibit stronger contrast between same-object neighbors and different-object neighbors.
Depth-based boundary tokens. For each image, we first resize the depth map to the 24 x 24 grid that corresponds to the visual tokens used by the VLM. Let index a grid location and let d, denote the depth value at p. We define the four-connected neighborhood of p as
N(p) {q y is the up, down, left, or right neighbor of p}.
(9)
We then compute simple depth gradient magnitude
Gp
max Id, - dgl- gEN(p)
(10)
Tokens with large Gp are likely to lie on object boundaries. We define the boundary token set B by thresholding Gp with percentile on the image-specifer distribution
B (p Gp Thoundary):
(11)
where Tooundary is chosen as, for example, the 90-th percentile of (Gp)p in each image. This per- centile scheme adapts to the dynamic range of each depth map and yields a consistent number of boundary tokens across images.
25

---

Published as a conference paper at ICLR 2026
Inner and outer neighbors the boundary. For each boundary token p E B, further split its local neighbors into inner and outer sets based on depth similarity. We use per-image thresholds Tin and Tour that are also defined as percentiles of the distribution of Id, dgl over all (p, q) pairs for q E (p). Concretely,
(r) = (q E N(v) -dp-dp dgl S 7
(12)
Nowr (p) - {a E (D) Idg dg 2 Tout
(13)
Intuitively, (p) contains neighbors at similar depth that are likely to belong to the same object, while (p) contains neighbors with large depth jumps that are likely cross an object boundary. We only keep boundary tokens for which both sets are non-empty.
Boundary Contrast of key vectors. Let k.))) denote the key vector location p in layer l, after averaging over attention heads and lz fz-normalizing the vector. For a boundary token in layer e, we define its Boundary Contrast
BC
cos(kip) @E (p)
Σ
cost (() p
R(O).
(14)
Nin
(P)
(En()))))))
The first term measures how strongly the key vector at p aligns with depth-similar neighbors inside the same object. The second term measures its vector similarity to depth-distan neighbors that are likely to belong to other objects.
For each layer f, we aggregate the BC values over boundary tokens:
BC(
1
(p)
(p),
BC in
1
Σ
cos( ki(t) 1.(0).
(15)
B
B
Nu
PEB
PEB
gE.N.(p)
BC(()
cos A/ MO)),
BC
BC!
BC!
(16)
B
(p)
out
PEB
geNoul(p)
Finally, we report image-level BC by averaging over set of layers L where DSCR is applied:
BC = |C|
1
4E EEC
BC in
BC = C1
1
BCL OULY
BC
1
ΣBC REC
(17)
|
REC
Setup and findings. We compute (BCin: BCou. BC) for both the baseline VLM and its DSCR- augmented variant. Following the main experiments, we focus on the MME hallucination subsets and further split examples into four groups according to answer correctness: baseline wrong vs. DSCR correct, both wrong, both correct, and baseline correct vs. DSCR wrong. Table 18 summa- rizes the results.
Across the hallucination-fis group where the baseline hallucinates but DSCR produces the correct answer, DSCR consistently increases BCi, and decreases BCout. which leads larger overall BC compared to the baseline. In contrast, for examples where both models already answer correctly, BC remains nearly unchanged. Taken together, these trends show that DSCR specifically sharpens key vector boundaries along depth depth-defined object contours on challenging hallucination cases, rather than globally distorting the internal representation geometry. This quantitative evidence comple- ments our PCA visualizations and directly supports the mechanism that DSCR restores local object consistency in the key space.
F LIMITATIONS AND FUTURE WORKS
Despite its advantages, DSCR presents certain limitations that offer avenues for future research. Cur- rently, DSCR primarily applicable to VL.M architectures similar to LLaVA (Liu et al., 2024c), and its effectiveness with other models, such as BLII-like VLMs (Li et al., 2023a) that utilize Q-formers, remains unexplored. Extending DSCR to these architectures could significantly broaden its appli- cability. Additionally, while DSCR operates in query-agnostic manner, incorporating query-aware processing or adopting few-shot in-context learning techniques may further enhance its performance and adaptability to diverse tasks. Furthermore, although DSCR employs lightweight depth estima- tion model, integrating it more tightly with VLMs through joint training during instruction tuning could potentially improve the overall performance.
26

---

Published as a conference paper at ICLR 2026
Table 19: Evaluation results of POPE benchmark.
Dataset
Setting
Model
w/DSCR
Accuracy
Precision
Recall
F1-Score
0.88
0.97
0.79
0.87
1LaVA-1.5
0.87
0.97
0.79
0,86
Random
Qwen-VL VL
0.87
0,94
0.80
0.86
0.87
0,98
0.76
0.86
0.86
0.95
0.76
0.85
MSCOCO (Lin al. 2014)
mPLUG-OW1
0.85
0.96
0.74
0.83
0.87
0,94
0.79
0.86
LLaVA-1.5
V
0.86
0.94
0.77
0.8.5
0.87
0.94
0.80
0.86
Popular
Qwen-VL
V
0.86
0.94
0.76
0.85
mPLUG-Owl2
X
0.84
0.91
0.76
0.83
0.84
0.93
0.74
0.82
LLaVA-155
X
0.84
0.89
0.79
0.84
V
0.84
0.89
0.77
0.83
Adversarial
Qwen-VL
X
0.84
0.88
0.80
0.84
0.84
0.91
0.76
0.83
0.82
0.86
0.76
0.81
mPLUG-OW12
0.82
0.88
0.73
0.80
LLaVA-1.5
x
0.79
0.73
0.91
0.81
0.85
0.78
0.96
0.86
Random
Qwen-VL
X
0.86
0.94
0.76
A-OKVQA (Schwenk al. 2022)
0.96
0.77
0.80
0.74
0.90
mPLUG-Ow12
0.85
0.82
0.91
0.86
LLVVA-155
X
0.74
0.68
0.91
0.78
0.79
0.72
0.96
0.85
0.93
0.76
0.83
Popular
Qwen-VL
0.86
0.95
0.76
0.84
0.72
0.67
0.90
0.76
mPL.G-OW12
0.81
0.76
0.91
0.82
0.67
0.62
0.91
0.74
LLaVA-1.5
0.69
0.63
0.96
0.76
0.80
0.82
0.76
0.79
Adversarial
Qwen-VL
0.81
0.84
0.77
0.80
0.68
0.62
0.90
0.74
mPL.U--Ow22
0.71
0.66
0.91
0.76
0.90
0.92
0.87
0.89
LLaVA-155
0.90
0.93
0.85
0.89
Random
Qwen-VL. VL.
x
0.84
0.84
0.83
0.84
V
0.85
0.91
0.77
0.84
GQA (Hudson Manning, 2019)
mPLUG-Ow12
X
0.85
0.91
0.77
0.84
0.84
0.93
0.75
0.83
LLaVA-155
X
0.86
0.84
0.87
0.86
V
0.87
0.87
0.85
0.86
Popular
Qwen-VI.
X
0.72
0.68
0.83
0.75
0.80
0.81
0.77
0.79
X 0.79
0.80
0.77
mPLUG-OW12
0.82
0.75
LLaVA-1.5
X
0.81
0.78
0.87
0.82
0.79
0.72
0.77
Adversarial
Qwen-VL VL
0.78
0.79
0.77
0.78
0.77
0.76
0.77
0.77
mPLUG-Ow12
0.77
0.79
0.75
0.77
27

---

Published as a conference paper at ICLR 2026
Table 20: Evaluation results of POPE on MSCOCO across different settings.
Setting
Model
Method
w/DSCR
Accuracy
Precision
Recall
F1-Score
0.89
0.93
0.86
0.89
VCD
0.89
0.93
0.84
0.89
1L5
OPERA
X
0.88
0.97
0.79
0.87
0.87
0.97
0.76
0,85
0.89
0.93
0.84
0.88
VCD
Random
0.88
0.93
0.83
0.88
Qwen-VI.
0.88
0.98
0.84
0.82
OPERA
0.88
0.98
0.83
0.81
0.86
0.95
0.76
0.85
VCD
0.85
0.96
0.74
0.83
mPLUG-Ow12
0.86
0.95
0.75
0.83
OPERA
0.85
0.96
0.74
0.83
0.87
0.94
VCD
0.86
0.94
0.77
0.85
LL.VA-1.5
0.87
0.94
0.79
0.86
OPERA
0.86
0.94
0.76
0.84
VCD
Popular
Qwen-VL
0.87
0.91
0.83
0.87
0.87
0.84
OPERA
0.87
0.92
0.82
0.86
VCD
0.84
0.94
0.72
mPLUG-Ow12
0.90
0.76
OPERA
0.84
0.92
0.73
0.82
VCD
0,83
0.84
1LLVA-155
0.79
OPBRA
VCD
x
0.85
0.84
Adversarial
0.85
0,86
0.83
Qwen-VL. VL
0.84
OPERA
X 0.82
0.86
0.76
VCD
0.82
0.88
mPLUG-Ow12
OPERA
X
0.82
28

---

Published as a conference paper at ICLR 2026
Table 21: Evaluation results POPE on A-OKVQA across different settings.
Setting
Model
Method
w/DSCR
Accuracy
Precision
Recall
F1-Score
0.87
0.87
0.87
0.87
VCD
0.87
0.88
0.86
0.87
LL
0.88
0.86
0.92
0.89
OPERA
0.90
0.89
0.96
0.91
VCD
Random
Qwen-VL
0.88
0.89
0.86
0.95
0.77
0.85
OPERA
0.87
0.96
0.78
0.86
VCD
0.86
0.94
mPLUG-OW12
OPERA
0.87
0.86
0.89
0.87
VCD
X
0.83
0.87
0.84
0.86
LLaVA-1.5
0.78
OPERA
0.86
0.82
0.96
0.87
VCD
X
0.87
0.86
0.89
Popular
X 0.87
0,87
0.88
Qwen-VL
0.77
OPERA
0.87
0.78
X 0.83
0.84
VCD
0.82
0.88
0.76
mPLUG-OW12
X 0.81
0.77
0.82
OPHRA
0.82
0.79
0.89
VCD
x
0.78
0.73
0.87
0.80
0.78
0.74
0.86
LL
OPERA
X
0.74
0.68
0.92
01.88
0.73
0.72
0.90
VCD
X
0.81
0.76
0.89
0.82
Adversarial
0.81
0.77
0.88
0.82
Qwen-VL
OPERA
X
0.81
0.83
0.77
01.00
0.82
0.84
0.78
VCD
X
0.76
0.74
0.81
0.77
0.77
0.77
0.76
0.77
mPLUG-Ow12
OPERA
x
0.71
0.66
0.88
0.75
0.73
0.68
0.89
0.77
29

---

Published as a conference paper at ICLR 2026
Table 22: Evaluation results POPE on GQA across different settings.
Setting
Model
Method
w/DSCR
Accuracy
Precision
Recall
F1-Score
0.88
0.87
0.90
0.88
VCD
0.88
0.88
0.89
0.89
LL
0.88
0.85
0.89
0.88
OPERA
0.87
0.87
0.88
0.88
VCD
Random
Qwen-VL
0.85
0.84
0.84
0.84
0.86
0,84
OPERA
0.84
0.85
0.86
0.85
0.77
VCD
0.84
0.93
mPLUG-OW12
OPERA
0.84
0.93
0.74
0.84
VCD
X
0.90
0.84
0.82
0.89
LLaVA-1.5
OPERA
0.84
0.79
0.89
0.84
VCD
X
0.77
0.72
0.87
Popular
0.76
0.72
0.86
Qwen-VL
OPERA
X
0.72
0.77
0.73
0.86
VCD
X
0.79
0.80
0.77
0.79
mPLUG-OW12
0.79
0.83
0.73
OPHRA
X
0.79
0.83
0.74
0.79
VCD
x
0.80
0.75
0.90
0.82
0.80
0.76
LLAVA-155
OPERA
X
0.80
0.75
0.90
0.82
0.80
0.76
0.89
0.82
VCD
X
0.77
0.73
0.86
0.79
Adversarial
0.77
0.73
0.86
0..79
Qwen-VL
OPERA
X
0.78
0.72
0.86
0.79
0.77
0.73
0.86
0.79
VCD
X
0.77
0.77
0.77
0.77
0.77
0.80
0.73
0.76
mPLUG-Ow12
OPERA
X
0.77
0.79
0.77
0.76
0.77
0.80
0.74
0.76
30


Reviews:
[{"review_id": "GRJL0nus6F", "reviewer": "Reviewer_nKMD", "summary": "This paper attempts to mitigate hallucination in VLMs by first studying the key vector distribution in model's transformer layers. The authors discovered that key vector distributions exhibit distinct object boundary patterns when models are faithful, while showing blurred object borders when hallucinating. This phenomena motivated the authors to propose DSCR, a training-free method that corrects key and value vectors using depth and spatial similarity maps as guidance. Under DSCR, a separate depth estimator model is used to generate a depth map. Coupled with spatial proximity, weightage is calculated to rebalance the key and value vectors during inference. Experimental results show that DSCR achieves high performance on various hallucination benchmarks. In addition, DSCR is also complementary with other hallucination mitigation methods, further boosting the evaluation performance on multiple open source VLM models. \n\nThe paper contributes to the research community by introducing a new correlation between key vector similarity and hallucination. The training free method can be applied on top of many existing works to further reduce VLM hallucination and improve model performance.", "questions": "1. In figure 3 different shades represent different values. However, why does the distribution pattern look like this? Which 2 image patches are compared?  \n2. Based on the ablation experiment table 14, it shows that the best performance is achieved when DSCR is applied to layer 10–39. Why does the final method apply to all layers instead of following the finding from this ablation study?  \n3. For the key vector PCA visualisation in figure 1, does the pattern stay the same across different layers? Or is the distinct object boundary only specific to certain attention layers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, the paper is well written and the concepts are easy to follow. The authors also offer a new direction to study the underlying cause of hallucination. The proposed DSCR method shows good performance and generalisability. It is an efficient design that is complementary with many existing hallucination mitigation methods. The validity of DSCR design is sufficiently supported by many experiments. The author conducted extensive analysis experiments including attention score value comparison and key vector distribution visualisation for models with and without DSCR correction. The authors also conducted comprehensive ablation experiments such as depth only, spatial only and depth with spatial weightage calculation as shown in appendix.", "weaknesses": "It is insightful for the authors to reveal the different key vector PCA visualisations for hallucinating and non-hallucinating VLM inference scenarios. However, this qualitative analysis can be sensitive to different factors such as object size, object position, difficulty in text query, etc. This slightly undermines the reliability of this discovery and thus the motivation of the method design. The authors could provide more evidence, such as quantitative experimental results, to show that the phenomena is universal, that blurring of object boundaries key vectors is common for different images and query types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gc8PhvJ5TW", "reviewer": "Reviewer_qhDk", "summary": "The paper introduces Depth and Spatial-aware Cache Refinement (DSCR) — a lightweight, training-free method to suppress visual hallucinations in LVLMs. The authors identify that hallucinations stem from incoherent alignment among key vectors (KVs) within the Transformer’s attention mechanism, which disrupts the cross-modal grounding between visual and textual inputs. DSCR addresses this by refining the Transformer’s KV cache using geometric and spatial priors derived from monocular depth estimation. Through integrating 3D depth cues and 2D spatial proximity, DSCR enforces coherence among tokens representing the same object and separates tokens across different surfaces, thereby improving visual grounding and reducing false object generation. Extensive experiments over multiple benchmarks demonstrate up to 23% accuracy improvement.", "questions": "1. Does DSCR take the user’s input query into account? If it only considers image features while ignoring textual features, will this lead to performance degradation in handling more complex queries?\n\n2. Why does an isotropic distribution of key vectors tend to produce hallucinations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a novel and elegant perspective on the origin of hallucinations by analyzing the internal coherence of key vectors within multimodal Transformers.\n\n2. The DSCR method is training-free, model-agnostic, and computationally efficient.\n\n3. The experimental evaluation is comprehensive and rigorous, and the writing is clear.", "weaknesses": "1. As shown on the right side of Figure 5, while DSCR leverages pre-computed depth maps, its performance inevitably depends on the quality of depth estimation, which may introduce inaccuracies in complex lighting or occlusion scenarios.\n\n2. Why would the misalignment of key vectors weaken the model’s ability to correctly interpret visual inputs? I believe there is no essential connection. I hope the authors can provide further clarification.\n\n3. This paper focuses on optimizing the static reasoning phase, without exploring joint training or adaptive tuning of deep cues. I believe that the misalignment between visual and textual modality features is also an important factor contributing to hallucination. Should additional training be incorporated into DSCR to enhance alignment with textual vectors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Z3EEy0fJn", "reviewer": "Reviewer_UhZY", "summary": "This paper studies the representational origin of visual hallucinations in vision–language models (VLMs) and proposes a training-free method, Depth and Spatial-aware Cache Refinement (DSCR), that modifies the visual key–value (KV) cache before decoding. The central empirical observation is that successful grounding correlates with coherent alignment of neighboring key vectors, while hallucination correlates with isotropically scattered keys that blur object boundaries. DSCR injects depth cues and 2D spatial proximity into the KV cache by forming a proximity-weighted refinement of keys (optionally values), which clusters vectors within objects and separates vectors across surfaces. The method is model- and query-agnostic, adds negligible overhead, and shows gains on multiple hallucination benchmarks. The paper also introduces a hallucination benchmark to occlusions and similar-depth confounders.", "questions": "- Since DSCR leaves VLM weights frozen, could the lack of adaptation to DSCR-refined keys limit the attainable gains? Have you tried enabling fine-tuning with DSCR active, and if so, did you observe larger improvements or any shift in preferred settings (e.g., Key-only vs. Key+Value, layer ranges)?\n- DSCR modifies only the visual branch of VLMs. Can it mitigate hallucinations that arise when the language model misinterprets visual tokens (i.e., unsatisfied vision-language alignment)? Do you have controlled analyses or case studies indicating whether DSCR helps in such language model-driven error modes?\n- Semantic richness vs. key scattering. The paper describes hallucination cases where keys scatter and object boundaries blur. Could similar patterns also appear in images with genuinely rich, heterogeneous semantics? After applying DSCR, is there any measurable loss of semantic diversity that could harm performance on VL tasks requiring fine-grained distinctions or rare attributes? \n- Minor presentation. In Figure 1(a), the non-hallucination examples show “X” and “O” even though there is no wrong answer, which is confusing. Also, the “A.” prefix for answers can be misread as an option label. Clarifying the notation would improve readability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Clear mechanistic story grounded in the model’s internal representations. The paper ties hallucination to a loss of neighboring-key coherence, and supports this with PCA-based visualizations across layers and attention diagnostics that show increased attention to image tokens when DSCR is applied.\n- Simple, training-free intervention with broad applicability. DSCR only modifies the KV cache at inference, without changing weights. The reported overhead is small and the method improves several VLM backbones.\n- Consistent gains on hallucination evaluations. Across MME, POPE/RePOPE, CHAIR, and AMBER, the method shows improvements; the depth hallucination mini-benchmark highlights gains in occluded and similar-depth cases.", "weaknesses": "- The link between “neighboring-key coherence” and hallucination is mainly supported by PCA visualizations and attention trends. The paper describes that, in hallucination cases, keys scatter and object boundaries blur, but it does not define a quantitative measure of “neighboring-key similarity” or “key-vector dispersion,” nor does it report large-scale correlations with hallucination/error rates. To strengthen the claim, it is benificial to introduce simple, layer-wise metrics (for example, average cosine similarity with spatial neighbors, or a local PCA explained-variance ratio) and report correlation/predictive power with confidence intervals.\n\n- Comparisons to alternative geometry-injection routes are missing. Since DSCR effectively injects geometric priors, it would be informative to compare against baselines that (i) use a stronger vision encoder that already has geometric priors or (ii) concat depth features with visual tokens as inputs for VLM without KV rewriting. This would clarify the unique benefits of operating in KV cache.\n\n- Evidence for depth–spatial complementarity is limited. In Table 13, depth-only and spatial-only achieve the same total score (645), while the combined setting is only modestly higher (650), with improvements concentrated in one submetric. Figure 4(c,d) validates each component on its targeted subset but does not establish that combining both consistently outperforms either alone across the same benchmarks. More side-by-side results with multiple runs, error bars, and paired tests, and stratified by scene attributes (e.g., occlusion density, depth discontinuities, similar-depth distractors), are helpful to show that the combination truly helps.\n\n- Breadth beyond hallucination-centric tasks is limited. The positive COCO captioning result is helpful, but a broader suite (e.g., additional VQA settings) would better establish that there is no negative transfer to general VL capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bpfPU8Qw4e", "reviewer": "Reviewer_gacf", "summary": "This paper proposes a training-free and lightweight method to mitigate LVLM hallucinations by encoding spatial and depth priors. The core idea is to re-weight the key–value pairs of visual tokens in the cache to enhance grounding, without additional training.", "questions": "1. I saw the ablation results in Tables 10–14. Could you provide more intuition on why key-only reweighting performs better, and why selecting layers 10–39 (deep layers) works best?\n2. To confirm: my understanding is that you apply re-weighting only to the visual token cache, not the text tokens. If that’s correct, it would help to make this explicit throughout sec 2.3.\n3. A broader question: do you envision future LVLMs incorporating 3D vision encoders directly to enhance spatial reasoning?\n\nwould be happy to raise my score if my concerns are addressed", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-motivated and supported by insightful visualizations. The method is novel and achieves consistent improvements on both standard object hallucination benchmarks and attribute/spatial hallucination benchmarks.\n2. The paper is overall well written and easy to follow.", "weaknesses": "1. Since both MME and POPE are yes/no questions, it would strengthen the evaluation to extend Table 3 to include detailed CHAIR scores across LVLMs and baselines.\n2. The paper provides spatial and depth priors but does not provide enough discussion of related work on grounding visual information.\n\n**Clarification**\n1. (Major) In Table 1, Qwen-VL Count row: your method is not the best (+155) but is still bolded. Similarly, for Qwen-VL Poster row, your method (165.99) is not the best. Please correct this.\n2. In Figure 1, the Non-Hallucination subfigure is a bit confusing. why are there two “A. Yes” labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)