Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
Authors: Shiqi Chen
Tongyao Zhu
Ruochen Zhou
Jinghan Zhang
Siyang Gao
Juan Carlos Niebles
Mor Geva
Junxian He
Jiajun Wu
Manling Li
Venue: ICML 2025 poster (2025)
Keywords: Interpretability
Attention mechanism
VLM
Spatial Reasoning
Primary area: deep_learning->large_language_models
TL;DR: We find that the attention pattern in VLMs can be adjusted adaptively to modify the focus area without requiring additional training.

Abstract: Large Vision Language Models (VLMs) have long struggled with spatial reasoning tasks. Surprisingly, even simple spatial reasoning tasks, such as recognizing “under” or “behind” relationships between only two objects, pose significant challenges for current VLMs. We believe it is crucial to use the lens of mechanism interpretability, opening up the model and diving into model’s internal states to examine the interactions between image and text tokens during spatial reasoning. Our analysis of attention behaviors reveals significant differences in how VLMs allocate attention to image versus text. By tracing the areas of images that receive the highest attention scores throughout intermediate layers, we observe a notable pattern: errors often coincide with attention being misdirected towards irrelevant objects within the image. Moreover, such attention patterns exhibit substantial differences between familiar (e.g., “on the left side of ”) and unfamiliar (e.g.,“in front of ”) spatial relationships. Motivated by these findings, we propose ADAPTVIS based on inference-time confidence scores to sharpen the attention on highly relevant regions when the model exhibits high confidence, while smoothing and broadening the attention window to consider a wider context when confidence is lower. This training-free decoding method shows significant improvement (e.g., up to a 50 absolute point improvement) on spatial reasoning benchmarks such as WhatsUp and VSR with negligible additional cost.

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
Shiqi Chen
Tongyao Zhu 2 Ruochen Zhou' ! Jinghan Zhang² Siyang Gao Juan Carlos Niebles 45 Mor Geva 6
Junxian He Jiajun Wu Manling Li47
Abstract
1. Introduction
Despite rapid advancements in Large Vision-Language Mod- els (VLMs), a significant deficiency persists, i.e., their strug- gle with vision-centric abilities (Gao et al., 2023; Kamath et al., 2023; Tong et al., 2024a; Chen et al., 2024a). This lim- itation is particularly notable in spatial reasoning given the simplicity of the task. Spatial reasoning involves inferring basic relationships between just two objects, such as "leff", "right" "above", "below", "behind", or "front", as shown in Figure 1. For example, given the image with book "behind" the candle in Figure 1, VLMSs describe the book as being "leff" of the candle. This error is not an isolated incident but a frequent, recurring pattern, highlighting a fundamental bottleneck in how VI.Ms process visual-centric information.
Large Vision Language Models (VLMs) have long struggled with spatial reasoning tasks, Sur- prisingly, even simple spatial reasoning tasks, such as recognizing "under" or "behind" relation- ships between only two objects, pose significant challenges for current VLMs. In this work, we study the spatial reasoning challenge from the lens of mechanistic interpretability. diving into the model's internal states to examine the inter- actions between image and text tokens. By trac- ing attention distribution over the image through- out intermediate layers, we observe that success- ful spatial reasoning correlates strongly with the model's ability to align its attention distribution with actual object locations, particularly differ- ing between familiar and unfamiliar spatial rela- tionships. Motivated by these findings, we pro- pose ADAPTVIS based on inference-time confi- dence scores to sharpen the attention on highly relevant regions when confident, while smooth- ing and broadening the attention window con- sider a wider context when confidence is lower. This training-free decoding method shows sig- nificant improvement (e.g., up to a 50 absolute point improvement) on spatial reasoning bench- marks such as WhatsUp and VSR with negligi- ble cost. We make code and data publicly avail- able for research purposes at battps://githi com/shiqqe chenl1 7 Adaptvis.
Recent studies have probed the limitations of vision en- coders like CLIP (Radford et al., 2021) in VLMs' vision processing (Tong et al., 2024b;a), yet critical aspect re- mains underexplored: how vision and text tokens interact within the model's 's internal states to construct geometric understanding. Specifically, the model must simultancously identify objects while maintaining awareness of the broader geometric context, grasping the complex geometric relation- ships between them. This geometric understanding funda- mentally manifests in how models distribute their attention across the visual tokens. This unique challenge makes spa- tial reasoning an ideal lens for studying how VLMs inter- nally process vision-centric information.
Therefore, we open up the black box of VLMs and examine their internal mechanisms through a suite of carefully de- signed spatial reasoning tasks. By looking into the attention distributions, we can systematically investigate how vision and text tokens interact to construct, or fail to construct, ac- curate spatial understanding. Our investigation begins with a key observation: despite image tokens comprising around 90% of the input sequence, they receive only about 10% of the model's attention. This significant imbalance sug- gests that textual priors often overshadow visual evidence, explaining VLMs' struggles with vision-centric tasks.
'City University of Hong Kong "National University of Singapore 'Hong Kong University of Science and Technol- ogy "Stanford University "Salesforce Research "Tel Aviv Uni- versity "Northwesterm University. Conresonnencce to: Shiqi Manling Li <man-
Chen eae.:n4488-@mmm.tyy...c... miieii@@otweestm..dds
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
While a straightforward solution might seem to be simply increasing attention to visual tokens, our deeper analysis re-

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
book
Attention image
relation to the candle?
Smoothen
Unfamiliar!
Rehind
Lock other patches!
Left
Confidence:0.22
VLM
(L.v))
Sharpert
Confdence:0.33
Look closely!
Right
Familiar!
Attention image
relation to the person?
Figure 1. The framework of ADAPTVIS. We adaptively intervene the temperature of the attention logits of the image tokens. Top: For generations with low confidence, we smoothen the attention distribution to broaden the context window for better concentration on the correct objects. Bottom: For generations with high confidence, we trust the attention pattern and sharpen attention distribution.
veals that the challenge lies not just in the quantity but in the geometric distribution of visual attention. By examining at- tention patterns across model layers, we observe pattern in VLMs' attention behavior: their initial attention distribution often reflects learned priors that may or may not align with the actual geometric distribution over the image. This led us to a key insight: rather than accepting this initial distri- bution, we can dynamically intervene based on the model's self-belief, which is measured using confidence score. As shown Figure 1, when the model exhibits high confidence in its spatial reasoning (as measured generation probabil- ity), sharpen its attention distribution strengthen the focus on its current beliefs. Conversely, when confidence is low, we smooth the attention distribution to encourage the exploration of alternative spatial relationships.
2. Preliminary on VLMs
We center our analysis on investigating how VLMs dis- tribute their attention over image tokens, aiming to gain deeper insights into spatial reasoning errors.
Notation Large VLMs like LLaVA (Liu al., 2024a) con- sist of three components: a visual encoder like CLIP (Rad- ford et al., 2021), pretrained language model, and a pro- jector to connect them. The visual encoder functions as perception tool to "see" the image, while the image in- formation is processed through a projector to mapped into the token space. The LLM is often based on the trans- former architecture, consisting of L layers stacked together. Each layer consists of two major components: Multi- Head Attention (MHA) module, followed by feed-forward network. For each layer I, given the input X E IRaxd (where n is the number of tokens and d is the embedding dimension), MHA performs self-attention in each head NA. The output concatenation of all heads' outputs: MHA Concat (N((1.)) ....,
We call this confidence- guniddd attention intervention AD- PATVIS, which proves remarkably effective while remaining computationally efficient. In experiments across diverse spa- tial reasoning benchmarks, including WhatsUp (Kamath et al., 2023) and VSR (Liu et al., 2023), our approach achieves substantial improvements of up to 50% points. These gains are observed across both synthetic datasets with clean backgrounds and real-world images with complex scenes, demonstrating the robustness of our method.
(l.I)) (..").......))))) W.. Here is the number of heads; N₂, is the output of head N₁:
(1.1)
Our visualization of this intervention strategy reveals its underlying mechanism: by dynamically adjusting attention patterns, we effectively guide the model's focus to better align with actual object locations and their spatial relation- ships. This suggests that successful spatial reasoning isn't 't just about having the right attention mechanism, but about having the right confidence to know when trust or ques- tion one's initial spatial understanding.
(QK /db
((.))) Softmax CafftaxxAAlte (A..))VV Softmax
M
V, (1)
where attention logits A (t.h) are computed via Q = XW gh. K XW V = XW and W q& WK, W E dddx are learnable projection matrix of the head N₁. A causal mask My = if 2 j. and -o0 otherwise, prevents tokens from attending to future tokens.
2

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
For Controlled and COCO Option
For VG option
Dataset
Num of Options
Num of Objects
Source
options
options
object
objects
Syn
Real
Cont A
USER: Where is the beer bottle relation to the armchair?
<image>
USER: Where the beer bottle in relation the armchair?
<image>
Cont_B
VG one
Answer with left, right, under, front behind.
Answer with left, right, under.
VG.two
COCO one COCO_two
Figure Left: Choice counts, object counts and data source by subset WhatsUp ("Syn" = Synthetic, "Real" = Real). Right: Evaluation prompts use in evaluation.
Image Attention Text Attention
Image Length Text Length
is paired with a ground truth caption describing the spatial relationship of two objects.
10.7%
11.7%
We reformat the original (image, caption) setting into a gen- erative question-answeringg answering setting (image, question, spa- rial label), enabling evaluation of generative models like VLMMs and tracing of internal states. Questions are gener- ated using GPT-4 (OpenAI, 2024). Details in each subset with prompts are shown Figure 2. This QA task asks about positions in multiple-choice format, with possible answers like "left", "right", "on" and so on.
89.3%
88.3%
Figure striking imbalance between visual and textual atten- tion: while image tokens take approximately 90% of the sequence length, they receive only about 10% of the model's total attention on WhatsUp. This severe disparity in attention allocation suggests that VLMs fundamentally underutilize visual information.
For evaluation, we use LLaVA-1.5 (Liu et al., 2024a) throughout the analysis part (Section and 4). For metric, we apply accuracy of exact match as the primary metrie. To maintain consistency in the label space across datasets, we use four-option setting (left,, right, on, under) for the Con- trolled_Image and COCO subsets, and six-option setting (left, right, on, under, behind. fehind,front) for the VG as it con- tains additional spatial annotations.
3. Text- Vision Attention Interactions
We hypothesize that the failure on spatial reasoning stems from issues in the text-vision interactions, specifically due to insufficient attention being allocated to the image tokens. In this section, we systematically investigate the impact of the absolute values of attention logits the image tokens during spatial reasoning.
To better understand model performance, we analyze the label distribution, which provides insight into the represen- tation of different spatial relationships within the dataset. In Controlled Image, labels uniformly distributed across categories (e.g., equal number of samples for "lesf", "right", "on", "under"). Another interesting feature of this dataset, which contributes to our choice to use it, is its contrastive setting. Controlled _mmage includes pairs of images with same objects in both "left" and "right" positions, and sets of same objects exhibiting "left", "right", "on", and "under" re- lationships. It enables us further assess model performance using pair accuracy and set accuracy, requiring correct identification of all relationships within pair or set, de- fined by Kamath et al. (2023). It provides a comprehensive evaluation of spatial relationships.
Experiment settings. We select a widely-used spatial rea- soning benchmark WhatsUp (Kamath et al., 2023) since it contains both synthetic data and realistic data. The synthetic data (Controlled Image) features clean backgrounds with two objects, as shown in the upper example of Figure 1. It comprises two subsets: Controlled AA, with one large object (e.g., table) and one small object (e.g., cup), and Con- trolled_ _B with two small objects (c.g., book and plate). We use Cont A and Cont B as abbreviations in the paper. The realistic data, as shown in the lower image Figure 1, con- tains complex backgrounds with multiple objects, sourced from MS COCO (Lin et al., 2014) and Visual Genome (Kr- ishna et al., 2017) (referred to COCO and VG later, both datasets include captions involving either one object or two objects, referred to as COCO one, COCO two, VG one, and VG two, respectively). Compared with realistic im- ages, the synthetic images enables clearer observation of VLMs's inner workings with just two objects. Each image
3.1. VLMs allocate sparse attention to the image.
We analyze how output tokens attend to image tokens by extracting the attention logits across layers, and present the following key findings: The sum of attention scores to the image tokens is significantly lower than that all the
3

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
79
+ Cont A
4p Cont_ B
patterns are distributed spatially by mapping the 576 image tokens in LLaVA 1.5 to their 24 24 corresponcing image patches. This mapping enables us to trace and visualize the geometric flow of attention, showing us not just where the model looks, but how its attention aligns with actual spatial relationships in the image.
73
Accuracy
73.1
71.8
70.9
70.9
60.3 V
61
59.1
58.2
56.7
4.1. The model automatically focuses on the relevant entity when correctly answering questions.
55
1.0
1.2
1.5
2.0
To determine whether the model's factual inaccuracy is linked to incorrect attention behavior, we use YOLO (Red- mon, 2016) to annotate the relevant entities in the images. We compare the overlap between these annotations with the model's attention distribution and evaluate whether the overlap can serve as a metric to predict the correctness of the model's 's answers. We transform the bounding box coor- dinates obtained from YOLO annotations into image-sized index vectors, where each element is either or 1. We then normalize these index vectors along with the attention logits of the image attended by the final answer of certain layers corresponding to the image patches. We then calculate the cosine similarity of the two tensors for each sample, and analyze this metric on Cont_A. As shown in Figure 6, the AUROC a metric used to measure how well a model can distinguish between positive and negative classes.) no- tably high in the middle layers, with results for all the layers at Figure 7, Additional results on other subsets are presented in the Appendix E.5.
Figure 4. Accuracy adding image attention in logit space (which corresponds to the multiplication operation in probabil- ity space; the x-axis of the figure represents the multiplication coefficient). ADAPTVIS, on the other hand, utilizes multiplication in logit space.
input text tokens, despite the considerably higher number of image tokens. In Figure 3, we focus on the attention scores from the first generated token and sum the attention allocated to the image tokens in average for all attention heads in all samples in WhatsUp. The results reveal that image tokens receive substantially less attention, with text tokens receiving approximately nine times more. Although the image sequence has a length of 576, compared to the text sequence, which typically ranges from 30 to 40 tokens in our short question-answerte focuses on text when generating outputs. That is, image information is sparsely processed by the language model.
setting, the model predominantly
To investigate why the middle layers exhibit the most no- ticeable patterns, we analyze both attention scores and the overlap AUROC (Figure 7) to track information flow and assess each layer's contribution. In the initial layers, atten- tion to the input image is the highest compared to other layers. However, the AUROC in these layers is very low, suggesting that the model primarily captures broad, global understanding rather than focusing on local details. In con- trast, the middle layers play more critical role in refining the model's understanding. As shown on the right side of Figure 7, the overlap AUROC peaks in these layers, indi- cating that they are where the model begins to effectively "process" image information. Additionally, the left side of Figure 7 shows a modest peak in attention logits, further supporting our hypothesis.
3.2. Is the answer more accurate if the model sees the image more? Not really.
Building on earlier observations, a natural question arises: since the attention scores assigned to the image are relatively low compared to those for text, could increasing attention to the image improve the factual accuracy of the model? To investigate this, we conduct an experiment by increasing the attention weights allocated to the entire image by the final answer, intervening by adding positive constants to the image attention logits across all patches, as described by Zhang al. (2023). In Figure 4, observe that adding constant weight uniformly across all image tokens does not improve performance on spatial reasoning tasks. This observation motivates us to explore more intelligent ways to focus on key visual features.
We further separately examine the attention patterns for cor- rectly and incorrectly answered questions. Our observations reveal that hallucinations frequently occur due to two types of attention failures: (1) insufficient attention to the correct object, and (2) misplaced attention on irrelevant objects in the image. Figure 5 illustrates these findings. More exam- ples are shown in Appendix E.5. In the two correct examples on the left, the attention scores are well-aligned with the referenced entities, with sufficient focus. On the other hand,
4. Visual Attention Distribution
Our analysis of text-image attention imbalance has estab- lished that visual information is underutilized, yet simply increasing attention to image tokens fails to improve spa- tial reasoning. We hypothesize that the way attention is geometrically distributed across the image is the key factor. To investigate this hypothesis, we examine how attention
4

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Areas
Where is the bowl ? Golden: Right
Where is the plate Golden: Right
Where the plate
Where is the can Golden: Behind
Golden: On Model: Under
Model: Right
Model: Right
Model: Front
Figure 5. Attention visualization examples from the WhatsUp Dataset. The left two examples answered correctly. while the right are incorrect. For correctly answered questions, the attention scores are precisely focused on the core entities mentioned, In contrast, incorrect show attention distributed irrelevant image regions. The visualizations attention from the 17th layer, and the title in each image is an abbreviation of "Where is A in relation to B".
Overlap AUROC layer17
Overlap AUROC layor18
goal-developing method to detect the reliability of inter- nal states and enable intervention. Therefore, the primary challenge lies in devising effective strategies to adjust the attention scores intelligently, given that we have no prior information about the scores until a single inference run generates the attention map.
40.
JAUOOC- - 0494
AUROC - 5.535
20
20
12
1.0
2.2
2.4
3.5
1.8 2.0
035 3.0
24
24
4.2. SCALING VIS: Temperature scaling to image attention distribution
Figure 6. The figure illustrates the AUROC of the overlap between YOLO annotations and the attention patterns for Cont_A the 17th and 18th layers. This high AUROC suggests that attention could an effective metric for detecting answer correctness. more AUROC results are shown Appendix D.
Our observations from Section 4.1 reveal that the model often misallocates attention logits within images, leading to errors in spatial reasoning. To address this, we aim to enhance the model's ability to focus on key visual features, improving its capacity to accurately ground spatial rela- tionships, especially in complex or ambiguous scenarios. achieve this, we propose a straightforward yet effec- tive method that dynamically adjusts the image attention by modifying the temperature of the attention distributions. By adjusting the temperature, either sharper smoother, effectively altering the attention distribution. For instance, if the attention pattern is mostly correct but lacks precision, we want to sharpens Con- versely, if the attention pattern fundamentally incorrect, we want smooth out, allowing us to explore alternative regions. This intervention targets the attention of the final input token (at the n-th position) to the image tokens.
Image Attention Logits Layers
Image Attention Logits Layers
Score
of Bemt
make the distribution
Ovelke ABRDC
Layers
Figure Left: Variance image token attention across layers in Cont_A, showing sparse attention image, consistent with other subsets (Appendix E.4). Right: The Overlap's AUROC value across the layer of A. We infer from the figures that the model "sees" the image information in the early layers and "processes" this information in the internediate layers. So for study our paper, we use the intermediate layers.
aA (L.A)
if E
A (L,A)
(CA)
otherwise
where I represents the indices of all image tokens. In essence, we change the temperature of image attention dis- tribution by multiplying coefficient a. From a physical perspective, multiplying by a coefficient greater than en- courages the model to focus more on the original distribu- tion, while multiplying by a coefficient less than makes the
the two incorrect examples on the right demonstrate how the model incorrectly assigns attention, effectively "seeing" the wrong parts of the image. While these examples high- light the qualitative differences in attention patterns, they do not provide a quantitative metric that aligns with our
5

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
distribution more dispersed. In experiments, we uniformly apply this coefficient to all H heads across all L layers to avoid the need for extensive hyperparameter search.
bution or break it? In this section, we explore how to assess when to trust image attention.
5.1. Self-confidence reflects whether we can trust the model's image attention distribution.
Experiment Setting. We select two widely-used bench- marks on evaluating the model's is ability on spatial reason- ing WpatsUp (Kamath et al., 2023) (introduced in Sec- tion 2), and VSR (Liu et al., 2023), which contains con- tains 1223 image-caption pairs with boolean labels. The original VSR is designed in (image, caption) format to eval- uate encoder models without generation capabilities, To adapt it for our purposes, we utilize GPT-4o to generate questions for the VSR dataset. For evaluation, we report both accuracy and F1 scores. A small validation set is al- located for each subset to optimize the temperature based on validation performance, and the final test is conducted on the test set. For both methods, the hyperparameter a is selected from [0.5, 0.8, 1.2, 1.5.2.0.. For baselines, we adopt DoLaa (Chuang et al., 2023) as a baseline, which em- ploys the inner knowledge to calibrate the output logits by substracting the logits from intermediate layers. Addition- ally, we incorporate VCD (Leng et al., 2024), which is contrastive decoding method to contrast the logits before and after eliminating the image.
We aim to determine whether an internal metric can indicate when the model's attention pattern is trustworthy. Since we found that attention patterns often correlate with the cor- rectness of the model's generation as shown in $ 4.1, we reframe our goal as identifying a metric that helps assess the reliability of its outputs. Following previous findings of using generation logits as a measure of self-coonidence to as- sess generation reliability (Kadavath al., 2022; Chen et al., 2024b), we also adopt the logits of generated outputs to rep- resent the model's ' self- self-coonide Specifically, we adjust the coefficient for all four spatial relationships in Cont_A and Cont B, as these subsets provide a uniform distribution of different spatial relationships. We then examine how confidence and accuracy change response and present the results in Figure 8. First, when the coefficient is 1.0 (base- line), the model exhibits significantly higher confidence for "left" and "right" relationships while showing notably lower confidence for other relationship types. This suggests that the model is more familiar with "left" and "right" re- lationships and less familiar with others. Furthermore, the accuracy of predicting "left" and "right" relationships is also higher, indicating the positive correlation between the model's self-confidence and its correctness in predicting them. Second, we observe distinct response patterns across different coefficients. Increasing the coefficient improves performance for "left" and "right" relationships, whereas decreasing it enhances performance for other relationships.
Results. Our results for ScalingVis are presented in Ta- ble and Table 2. By controlling the distribution of at- tention weights, spatial reasoning performance improves significantly, with gains of up 37.2 absolute points. An interesting pattern emerges: temperature below one tends to enhance performance on synthetic data in most cases (3 out of 4), while a temperature above one benefits real image datasets across all cases. Table 1 indicates that for synthetic data, smoothing the image attention logits im- proves performance. Conversely, for real image datasets (COCO and VG), the optimal temperature is consistently above one, demonstrating that sharper attention distribu- tion helps the language model discern relationships more effectively. Intuitively, we believe that for familiar datasets and spatial relationships, the model requires a sharper at- tention distribution, as it is generally correct but may not be sufficiently precise. In contrast, for unfamiliar datasets and spatial relationships, smoother attention distribution is necessary to explore a broader context.
This suggests that the model responds differently based on the relationship type, necessitating tailored intervention methods. These insights motivate to propose adaptive intervention strategy.
5.2. ADAPTVIS
Motivated by our observation that attention misallocation leads to spatial reasoning errors and that the model exhibits varying self-confidence and correctness across different spatial relationships, we propose ADAPTVIS: Confidence- aware temperature scaling, an extension of SCALING Confidence-Based Attention Intervention.
5. Adaptively Intervening the Attention Distribution by Model Confidence
Recall from Section 5.1 that we observe two distinct pat- terns in the model's 's factuality behavior: (1) synthetic data presents more unfamiliar cases than the real data, and (2) VLMs could express uncertainty through confidence scores. These insights motivate using confidence scores as a metric for adaptive intervention in the model's internal states. Our intuition is straightforward: when confidence is low, suggest- ing that the attention pattern may be unreliable, we smooth
The findings from our study raise key question: Since sharpening the distribution can sometimes improve perfor- mance while smoothing it is preferable in other cases, can we establish a metric to determine when to make these ad- justments adaptively? In other words, can we establish a metric on whether we want to strengthen the original distri-
6

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
For low-confidence relationships: coefficient <1 improves performance. For oes coefficient > improves performance.
Accuracy v.s. Coafficient (Cont A)
Confidence v.s. Confficiant (Cont A)
Accuracy v.5 Confficiont (Cont B)
Confidence v.s. Coafficient (Cont
Right
0.50
Accuracc 15)
Card dearnee
Accuracy
60
0.45
40
Under
0.30
40
Sehind
0.40
0.45
20
20
0.35
0.40
0.30
0.5
1.0
1.2
2,4
16
155
0.5
1.0
12 1,4 Couficiint
La
1.
2.0
0.6 0.8
1.0
1,4
18
1.5
2.0
06
0.8
1.0
12
2.0
Coefficient
Confficient
Confiiiintt
Model has higher confidence for left / right than on / under / front behind, indicating that the model is more familiar with certain relationships.
Figure 8. O Change in accuracy with the coefficient on Cont_A. 2
Change in confidence with the coefficient on Cont_A. O Change
accuracy with the coefficient on Cont_B, ④ Change in confidence with coefficient on Cont_B. The baseline (greedy decode) corresponds to coefficient of 1.
Question: Where mug relation the lable?
Question: Where the remote in relation the candle?
Golden Answer: On
Golden Answer: Behind After intervention(a:0.55) Answer: Rehind
Original
After (ntervention(c:0.5)
Original Answer: Front
Answer: Under
Answer: On
Figure Attention scores on patches before and after our intervention in the 17th layer images in synthetic datasets Cont_A and _ont_B B. We employ "smoothing" intervention method to expand the context length of the model's focused area. From the figure, it is evident that model's focused position undergoes significant changes after our intervention.
Question: Where is the man in relation to the woman?
Question: Where is the bicycle in relation to the man? Golden Answer: Right
Golden Answer: Right
Original
Atter intervention(a:2.00 Answer: Right
Original
Atter intervention(a:2.00 Right
Answer: Front
Answer: Front
Figure 10. Attention scores on the patches before and after our intervention in the 17th layer for the images in real datasets COCO and VG. We utilize a "sharpening" intervention method to enhance original attention pattern. The highlighted areas remain largely consistent, with our method serving to reinforce the focus rather than significantly altering it.
the attention distribution. This encourages the model to explore a broader range of image regions, increasing the
likelihood of focusing the correct patches. Conversely, when confidence is high and attention is dispersed across

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
Anj
(1,N)
a1
(L.)) (1.b) n.
e otherwise
if 3
(3a)
Anj
(1,)
0g A (L,b mann A nn
e otherwise
if
(3b)
Controlled
Controlled
COcO.one COCO.to
VG_one
VG_two
Model
Acc
P Ace
S Acc
Acc
P Acc
S Acc
Acc
Acc
Acc
Acc
LLaVA-1.5
60.3
40.6
0.0
73.1
41.6
3.7
53.0
58.2
35,9
40.8
+VCD
61.5 11.2
39.4.11.
70.4
42.2 10.6
3.7
53.3 10.3
58.2
35.8 jo.1
42.5 11.7
+DoLa
61.2 10.9
41.6
0,0
10.4
42.2
3.7
53.7 10.7
57.5 10.7
36.2 10.3
42, 11.3
+SCALINGVIS
64.5 14.2
40.6
0.0
75.2
44.6
9.8 16.1
53.6
59.4
42.7
48. 17.3
+ADAPTVIS
84,9 124.6
61.2 120.6
30.3 33.3
83.8 110.7
55.7 114.1
114.6
53.6 10.6
59,9 11.7
42.7 16.8
48.1 17.3
LLaVA-1.6
48.2
37.6
0.0
63.0
39.1
3.7
59.7
41.8
31.6
7.3
+VCD
61.8
41.8 14.2
10.9 110
62.4
41.6
7.3 13.6
60.6
44.9 13.1
33.8
11.6 14.3
+DoLa
48.2
37.6
62.7 J0.3
39.1
3.7
59.7
41.5 J0:3
31.5 jo.1
7.3
ISAALIGGISS
97.0 148.8
76.4 138.8
54.5 154.5
73.4 110.4
48.9 19.8
15.9 r12.2
63.1 13.4
47.7 15.9
38.2 16.6
14.6 17.3
+ADAPTVIS
98.2 150.0
78.8 :41.2
57.0 -5700
73.4 110.4
48.9 19.8
15.9 112.2
63.1 13.4
47.7 15.9
35.2 13.6
17.2 19.9
Table 1. Results on Controlled A, Controlled B, COCO, and VG datasets. (Metrics in x10 Best-performing method per model and dataset are highlighted bold. P Acc and Acc represents Pair Acc and Set Acc.
Model
VSR
Results Our main results are presented in Table and Table By controlling the distribution of attention weights, observe significant improvement in spatial reason- ing ability, with gains of up to 50 absolute points. In most cases, ADAPTVIS achieves the best performance, par- ticularly for synthetic datasets like Cont_A and Cont_B, as shown in Table 1, where it significantly outperforms the generalized method SCALING These findings suggest that model performance varies considerably with the label distribution of the dataset, and smoothing the distribution (by applying a coefficient smaller than 1) enhances perfor- mance. For real-image datasets like COCO and VG, the adaptive method performs slightly better than the gener- alized approach, indicating the model's 's robustmess across di different label distributions. Example cases are shown in Figure 9 and Figure 10, illustrating the smooth effect and focus effect (ar and 1), respectively. is important to note that the ALLVA--eries models trained the COCO dataset, which makes them highly confident and fa- miliar with COCO and VG image types. Hence trusting the model's ' self-belief and sharpening image attention improves performance. Notably, for datasets containing more unfa- miliar images, the adaptive setting proves be significantly more effective.
Exact Match
F1 Score
LLaVA-1.5
62.4
51.3
+VCD
62.4
50.6
+DoLa
62.8 10.4
53.2
+SCALINGVIS
64.9 (2.5
+ADAPTVIS
65.0 22.6
62.5 -11.2
LLaVA-1.6
58.8
29.4
+VCD
58.8
29.4
+DoLa
59.3 10.5
31.2 n1.8
+SCALINGVIS
59.1 10.3
31.6
+ADAPTVIS
62.7 +3.9
39.3 -9.9
Table Results on VSR dataset (Exact Match and FI) (Metrics in x 100 x10=²). ²). Best-performing method per model and dataset are highlighted bold.
the image, we sharpen the distribution to concentrate on key objects more effectively. Specifically, we apply the targeted intervention the attention of the last input token the n-th position) directed toward the image tokens shown a large > 1 when the
in Equation 3a, 3b. Overall, Confidence C is large, which sharpens the attention distribu- tion, and the relevant objects are paid more attention to: we use small a < when the confidence C is small, which mitigates the model's excessive concentration on certain image tokens and makes the overall attention distribution smoother across the image.
6. Related Work
The first line of related work focuses on the attention pat- terns in LMs. Some studies on attention patterns in LLMs reveal biased attention across context windows, such as ineffective use of the middle context (Liu et al., 2024b) and initial token attention sinks (Xiao et al., 2023). While some approaches use fine-tuning to overcome these bi- ases (An et al., 2024), training-free methods like input-
Evaluation Settings. We employ the same setting with SCALING as shown in 4.2. For hyperparameter choice and robustness, we show in Appendix E.3.
8

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
adaptive calibration (Yu et al., 2024b) and position-specific interventions (Yu et al., 2024a) offer efficient alternatives. PASTA (Zhang et al., 2023), a closely related method, em- phasizes attention on selected segments for specific heads; we extend this to VLMMs without manual segment specifica- tion or multiple validation runs, Our work is also related to failure analysis in VLMs, VLMs have been shown to halluci- nate more in multi-object recognition tasks and rely on spu- rious correlations (Chen et al., 2024c), with systematic vi- sual limitations highlighted from a CLIP perspective (Tong et al., 2024b). Our work also connects to the decoding strategies for reducing hallucinations decoding strategies to mitigate hallucinations include contrastive decoding focus- ing on image regions (Leng et al., 2024), preference tuning through data augmentation (Wang et al., 2024), and meth- ods leveraging contrastive layers for enhanced knowledge extraction (Chuang et al., 2023), well as indivatton-basee optimal answer identification (Chen et al., 2024b).
derstand their spatial relationships may struggle with tasks like picking and placing items or navigating complex envi- ronments. Similarly, autonomous vehicles might face chal- lenges in interpreting traffic scenarios accurately, potentially compromising safety.
Our development of ADAPTVIS, a novel decoding method that dynamically adjusts attention distribution based on the model's confidence, represents a significant step forward. By enhancing VLMs' performance on spatial reasoning tasks, ADAPTVIS could unlock new possibilities in various fields. In healthcare, improved spatial reasoning could lead to more accurate interpretation of medical imaging, poten- tially improving diagnostic accuracy. In augmented reality applications, better spatial understanding could enable more immersive and interactive experiences. For assistive tech- nologies, enhanced spatial reasoning could provide more accurate and useful descriptions of environments to visually impaired individuals, significantly improving their indepen- dence and quality of life.
7. Conclusion and Future Work
Looking ahead, our work opens up new avenues for research in AI and cognitive science. The exploration of mechanism interpretability in VLMs, particularly for complex geometric structures and long-horizer spatial reasoning, could provide insights into how artificial systems process and understand spatial information. This could not only advance Al capa- bilities but also contribute to our understanding of human spatial cognition. Additionally, investigating the role of training data memorization in spatial reasoning bottlenecks could lead to more efficient and effective training methods for future Al models.
Our research uncovers the inner working mechanism of VLMs during spatial reasoning, which is a critical limita- tion in VLMS and constrains their practical utility when requiring geometric understanding of visual scenes. We identify critical insights through an in-depth study of atten- tion behaviors across layers: 1) VLMs allocate surprisingly insufficient attention to image tokens; 2) the location of attention on image tokens is more crucial than quantity; and 3) generation confidence serves as a reliable indicator of its familiarity with the image and the correctness of its attention pattern. Based on these findings, we propose ADAPTVIS, novel decoding method that dynamically adjusts attention distribution. significantly improving spatial reasoning per- formance. Future research could focus on further exploring the mechanism of VLMs on complicated geometrie struc- ture understanding, such as long-horizon spatial reasoning, and investigate other spatial reasoning bottlenecks.
In conclusion, our research not only addresses fundamen- tal limitation in current VLMs but also paves the way for more versatile and capable AI systems. As we continue to advance VLMs' capabilities in visually-driven tasks re- quiring nuanced spatial understanding, we have the poten- tial to significantly impact various sectors of society, from healthcare and assistive technologies to urban planning and environmental monitoring. The future research ahead in this field is both exciting and challenging, requiring ongoing collaboration between researchers, ethicists, and policymak- ers to ensure that these advancements benefit society as a whole.
Impact Statement
Our research into the spatial reasoning capabilities of Large Vision Language Models (VLMs) has significant implica- tions across various domains of artificial intelligence and its real-world applications. First and foremost, our find- ings highlight a critical limitation in current VLMs: while they excel at object recognition, they struggle with basic spatial relationships. This gap between recognition and spa- tial understanding has far-reaching consequences for the practical deployment of VLMs in scenarios requiring ge- ometric comprehension of visual scenes. Industries such as robotics, autonomous navigation, and assistive technolo- gies for the visually impaired are particularly affected. For instance, a robot that can identify objects but cannot un-
Acknowledgments
We thank Min Lin, Miao Xiong, Guangtao Zeng. Teng Xiao, Wei Liu, Chi Han, Chang Ma, Zhengxuan Wu, and Junlong Li for their valuable discussions and insights.
9

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
References
pp. 9161-9175, Singapore, singapoe, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/112023. emnlp-main.568. URL https: httppp/s//aaaaahhloogo.oo orq/20233 wwt.:2.eemmpmmmmmmmm waw.nnppmmiii..m5mmm 5588
An, S.. Ma, Z., Lin, Z., Zheng, N., and Lou, J.-G, Make your Ilm fully utilize the context, 2024.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zimick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: Lims trained on" b" fail to learn" is a". arXiv preprint ar arXiv:2309. /2288, 2023.
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Chen, B., Xu, Zz., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023.
Chen, S., Xiong, M., Liu, J., Wu, Z., Xiao, T., Gao, S., and He, In-context sharpness as alerts: An inner repre- sentation perspective for hallucination mitigation. arXiv preprint wwx.x:2243.015444 2024b.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.,, Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Chen, X., Ma, Z., Zhang, X., Xu, S., Qian, S., Yang, J., Fouhey, D. F., and Chai, J. Multi-object hallucina- tion in vision-language models, 2024c. URL https: //arxiv. org/ abs/2407. 06192.
Liu, F., Emerson, G. E. T., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computa- tional Linguistics, 2023.
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., and He, P. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.000 038833 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
Gao, J. Pi, R., Zhang. J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H.. Li, Z., et al. G-llava: Solving geo- metric problem with multi-modal large language model. arXiv preprint arXiv:2312. /370, 2023.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173,
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for roaldworld visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
2024b. doi: 10.116// a 00638. URL. https: /
OpenAL. .https://cd.. ctt. comppapers/ [rrV_System_aard.pde 2024. [Accessed 06-08- 2024].
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 1148-8763. PMLR, 2021.
Kamath, A., Hessel, J., and Chang, K.W. What's "up" with vision-language models? investigating their strug- gle with spatial reasoning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,
Redmon, J. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
10

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Tong, S., Brown, E., Wu, Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024a.
Tong, S., Liu, Z., Zhai, Y., Ma. Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Wang, F., Zhou, W., J. Xu, N., Zhang, S., Poon, H., and Chen, M. mdpo: Conditional preference opti- mization for multimodal large language models, 2024.
URL https::::/ //aaaxxvvvv../ org/ abs/2405 11839.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv, 2023.
Xiong, M., Hu, Z., Lu, X., LI, YY, Fu, J., He, J., and Hooi, B. Can ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, 2024.
Yu, Y., Jiang, H., Luo, X., Wu, Q. Lin, C.-Y., Li, D., Yang, Y., Huang, Y., and Qiu, L. Mitigate position bias in large language models via scaling single dimension. arXiv 2024a.
preprint ar 24xi:22466.025.
Yu, Z., Wang. Z., Fu, Y., Shi, H.. Shaikh, K., and Lin, Y. C. Unveiling and harnessing hidden attention sinks: Enhanc- ing large language models without training through atten- tion calibration. In International Conference on Machine Learning, 2024b.
Zhang, Q. Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., and Zhao, T. Tell your model where to attend: Post-hoc attention steering for llms, 2023.

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
Benchmark
Qwen2-VI.
+Attention Intervention
Appendix
VSR
78.96
81.60 12.64
Coco_one_oh
76.64
78.03 11.39
Coco two obj
75.28
76.52 11.24
VG_one_oby
74.89
75.11 10.22
A. Limitations
VG_two_obr
56.22
66.95 110.73
Firstly, our methods, SCALINGVIS and ADAPTVIS, specifi- cally address model-related spatial hallucinations and self- alignment issues but are not designed to handle errors out- side the language model's capabilities, such as the CLIP fail- ures discussed by Tong et al. (2024b). Secondly, ADAPTVIS relies on bobbuttio-bbsedd confidence to adaptively set the confidence threshold B, we also observe that the optimal a and 3 is different across different distributions and prompts. This dependence on a validation set for tuning poses a limi- tation on its applicability.
Controlled_A 98.18
98.18 10.00
Controlled.
91.73
92.97 11.24
Table 3. Performance of Qwen2-VIL before and after the attention intervention on spatial reasoning benchmarks.
Benchmark
Qwen2-VI.
+Attention Intervention
POPE-Overall
86.32
87.09 10.77
POPE-P
86.47
87.29 10.82
POPE-A
85.07
85,80 10.73
POPE-R
87.46
88.22 10.76
B. Additional Results on Qwen2-VL
GQA
62.09
62.17 10.08
TextVQa
79.18
79.26 10.08
We further expand our intervention experiments and atten- tion analysis on Qwen2-VI, the results on spatial reasoning benchmarks and on general benchmarks are shown in Ta- ble 3 and Table respectively. We intervene in the image attention distribution using our temperature- showing consistent improvements below, particularly in [GG_toooojecc where
Table 4. Performance of Qwen2-VT. before and after the attention intervention on additional general benchmarks.
method,
tionships with lower confidence, such as "Left" and "Under" tend to exhibit lower accuracy when compared to those with higher confidence, like "On" and "Right". After the intervention with AdaptVis, certain spatial relationships like "Left" show improved performance, accompanied by an in- crease in confidence. This demonstrates a pattern consistent with observations for LLaVA, depicted in Figure in Section $4.
challenging cases. For example, the baseline performance is the lowest among all bench- marks, our method yields significant improvement of 10+ absolute points. The gains observed on Qwen2-VLL VL further demonstrate the generalizability of our approach. Exper- iments on more benchmarks shown in Table 4, including POPE (Li et al., 2023), GQA (Hudson & Manning, 2019), and VeA (Singh et al., 2019), which proves that at- tention intervention can maintain the performance on more general tasks without hurting the performance. Compared with spatial reasoning tasks, general QA tasks achieve rela- tively smaller improvement. A possible reason is that such tasks are less sensitive to the geometric structured distribu- tion of image attention. For example, given a question like "Is there dog in this picture?", the model only needs to detect the presence of an object, and is therefore less likely to suffer from misallocated attention across spatial regions.
Benchmark
Text attention
Image attention
Controlled A Controlled
1.57e-02
7.59e-05
1.58c-02
7.69c-05
Coco_one_oo
1.77e-02
4.48e-04
Coco_two_oke
1.65e-02
3.50e-04
VG one obj
1.54e-02
4.75e-04
VG_two_obr
1.42e-02
4.19e-04
Table 5. Average text-token and image-token attention scores across benchmarks.
We also conduct an attention analysis to verify whether our observation of the sparsity of image attention compared to textual tokens still holds. The results shown in Ta- ble 6. Here we calculate the Qwen2-VL's attention scores below (average attention logits for single text/image tokens respectively), which matches our previous claim that image receives much less attention than the text tokens is generally valid for VLMs. And we also conduct the uncertainty ex- periments to show our uncertainty conclusion also valid. The results are shown in Table 5. It shows that spatial rela-
C. Related work
C.1. Attention Patterns in Language Models
Ongoing research has shown how large language models (LLMs) exhibit biased attention across differrntt parts of the context window. Liu et al. (2024b) find that LLMs fail to effecctiely utilize the information in the middle of a long context window. Meanwhile, Xiao et (2023) reveals an
12

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
Question: the bowi relation the tape/
Question: Where the plate in relation to the armchair?
Answer:
Golden
Original Answer: Front
Aftor intervention(a:0 5) Answer: Bchind
Original
After intervention(a:0.5
Answer:
Figure 11. Examples of fixed for 0.5.
Question: Where is the plate in relation to tape?
Question: Where is shower curtain in relation to the toilet?
Golden Answer: Behind
Golden Answer: Left
Original Answer: Left
After Intervention/aa0.55 Answer: Behind
Original Answer: Below
After Intervention/a:2.00
Answer: Left
Figure 12. Examples of our fixed case for 2.0.
attention sink at the initial tokens of the input. Besides fine- tuning methods to overcome such biases (An et al., 2024), some training-free free methods have been proposed with the benefit of their efficiency. Yu et al. (2024b) proposes to use anput-adaptive calibration to adjust the attention scores, while Yu (2024a) intervenes in position-specific hidden dimensions to alleviate the lost-in-the-middle phenomenon. closely related work to ours is PASTA (Zhang et al., 2023), which emphasizes the attention scores of specific text segments for selected attention heads. We further de- velop this motivation on vision language models. Moreover, our method does not require a manual specification of the emphasized segment or multiple validation runs to identify effective attention heads.
(2024b) analyze VLM failures from CLIP perspective, highlighting that the visual capabilities of recent VLMMs still face systematic shortcomings, partly due to CLIP's limita- tions in specific cases.
C.3. Decoding Strategies for Reducing Hallucinations
This work is also connected to various decoding and tun- ing strategies aimed at mitigating hallucinations in VLMs. Leng et al. (2024) introduce a contrastive decoding method that emphasizes certain image regions. Wang et (2024) propose data-augmentation approach to create image- intensive datasets, followed by preference tuning on this enhanced data. Furthermore, knowledge extraction tech- niques such as the method proposed by Chuang et al. (2023) improve decoding by leveraging contrastive layers for better knowledge extraction. Similarly, Activation De- coding (Chen et al., 2024b) identifies optimal answers as those with the highest activation values within the context.
C.2. Failure Analysis of Vision-Language Models
Our work relates to research on hallucination detection in VLMs. Chen al. (2024c) examine multi-object recogni- tion tasks, observing that VLMs exhibit more hallucinations when dealing with multiple objects compared to single- object scenarios. They also note similar phenomenon to our findings: the distribution of tested object classes impacts hallucination behaviors, suggesting that VLMS may rely on shortcuts and spurious correlations. Additionally, Tong et al.
D. Case Study
We show more case we could fix in Figure 11 and Figure 12, We show more attention examples at Figure 13.
13

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Areas
Benchmark
Qwen2-VL Unc.
+ScalingVis Unc.
Qwen2-VIL VL Acc
+Scaling Vis Acc
Left
0.4408
0.4474
70.11
74.71
On
0.5758
0.5668
100
100
Right
0,5982
0.5911
100
100
Under
0.5554
0,5418
98.82
98.82
Table
Uncertainty (Une.) and accuracy (Ace) on spatial-relation categories before and after applying Scaling Vis.
Where is the can
Where is the cup Golden: Behind Model: Front
Where is the pot ?
Where is the plate Golden: On Model: Under
Golden: Right
Golden: Under
Model: Right
Model: Under
Pigure 13, Examples of image attentions. Visualization here adopted from 14th layers' attention scores.
E. Further Analysis
Notably, the attention distribution is more concentrated in synthetic datasets than real images. attribute this to the fact that synthetic images tend to contain fewer ob- jects, resulting sharper attention distribution. However, this concentration does not provide reliable metric for measuring familiarity. Another possible metric is skewness, Another possible metric is skewness, which captures the asymmetry of the attention distribution. A high skewness suggests that the attention is predominantly focused on few positions, while a low skewness indicates a more balanced spread across multiple regions. By examining skewness, we aim to identify whether the attention is being dispropor- tionately allocated to particular image features, which could provide additional insights into how familiarity is expressed through attention patterns. We could see from Figure 14 that Synthetic datasets show higher skewness. However, it also related with the object distribution, which not the real factor behinds the difference.
E.1. Other metries to distinguish the distributions
We further explored additional metrics to distinguish the distributions and uncover their underlying characteristies during our preliminary experiments. Specifically, we exam- ined entropy and skewness. Entropy was selected based on hypothesis that parameter differences may stem from the familiarity of attention patterns in real images, which generally correct, versus synthetic images, where these patterns tend to be incorrect. We posit that the model can ex- press "familiarity" through certain metrics derived from the attention scores. For example, we hypothesize that the en- tropy of attention will be lower when the model encounters familiar cases.
E
(1..)) = Σ
P
(A
log P (A n.j
(4)
1=1
(A 1 -E
In Equation (1.h) (1.A denotes the attention scores assigned by the h-th head in the /-th layer to the j-th token in se- quence n. The summation runs over j = to f, where the total number tokens considered for this attention distribution. P. (1.AA) is the normalized probability dis- tribution of these attention scores. This entropy measures the uncertainty spread of the attention distribution across tokens.
03
(5)
The summation runs over j to I, where is the num- ber tokens considered in the attention distribution. P, denotes the probability assigned the j-th token in the nor- malized attention distribution. The term μld is the mean of the distribution, and a is its standard deviation. The skew- ness is calculated as the normalized third central moment, which measures the asymmetry of the attention distribution: positive value indicates a distribution skewed to the right, while negative value indicates a skew to the left.
Our experimental results in Figure 14 indicate that the at- tention distribution is heavily influenced by image features.
14

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
Stawness Da Distrbuutin Comparioon Across Dacasets
Entropy Datr butlon Compar son Across Data
0000, d6, peo (k) Gontooaad Itaget
Cortriled mapes,A Contrulied, mapo,8
14
Denskr
12-
Sknwnods
Entropy
Figure 14. The skewness and entropy distribution comparison between different subsets. Here we use Controlled Images and COCO datasets due to all of them are in four option label space, which enables us to eliminate the influence of prompts.
Relationship Types
performs better on confident relationships. For example, in the first figure on the left, when the coefficient is set to I (the baseline), the ground truth samples labeled as "left" and "right" exhibit significantly better performance compared to those labeled as "on" and "under". This observation aligns with our intuition that model tends to be more confident when performs well and less confident when it struggles. This finding is also consistent with previous work showing that models can convey their uncertainty through confidence scores (Kadavath et al., 2022; Xiong et al., 2024). Moti- vated by this, we propose using confidence as a metric to gauge the model's familiarity with spatial relationships in images.
Dataset
Right
Left
On
Under
Behind
Front
Controlled_A
92
92
130
92
0
0
Controlled B
102
102
0
0
102
102
VG_one
376
342
192
198
2
0
VG.lwo
137
127
3
5
19
COCO_one Lone
564
576
363
744
0
COCO two
129
150
86
75
0
Table Gold Answer Frequency by Spatial Relation WhatsUp datasct.
E.2. Label statistics in WhatsUp
We present the golden labels' distribution in Table 7. We can see that the label space in synthetic datasets is balanced while the real image datasets are imbalanced with more "left" and "right" and fewer other relationships.
Statistics of LLaVA training data To categorize six different spatial relationships, we check whether specific phrases appear in the "Ilava v1 5 mix665k.json', whishh is which the training data of LLaVA and increment corresponding counters. For the left relationship, we look for phrases such as "left side," "left of." "to the left," and "on the left." Simi- larly, for the right relationship, we detect phrases like "right side," "right of," "to the right," and "on the right." The on relationship is identified using phrases like "are on the," "is on the," and "located on," ensuring they do not co-occur with "on the left" or "on the right." To detect the under rela- tionship, we check for phrases such as "under the," "beneath the," " or "below the." For the front relationship, we look for phrases like "are in front of," "is in front of," and "locate in front of." Finally, the behind relationship is identified by the phrase "behind the.". The results are shown in Figure 15,
Label Distribution and the familarity To determine whether to focus more the image or the text, we begin by analyzing existing datasets to identify potential patterns. Our investigation starts with an examination of label dis- tributions across different subsets of the WpatsUp dataset. As shown in Figure 15, there a clear label imbalance in the real-image datasets including COCO.tw two and VG_tw vww, i... only a small portion of the samples have the relation of "behind" and "front". This may be attributed to anno- tation bias, where left and right are easier to distinguish than other relationships. In contrast, the synthetic datasets, Cont_A and _ont_B, which are carefully curated, display more balanced label distributions. Additionally, we evalu- ate the model's confidence scores across the ground-truth spatial relations by following the approach of Kadavath al. (2022), where the probability of the generated output is used to compute confidence. Figure 15 reveals that the model is unconfident with specific spatial relationships, such as "on" and "under" where it shows a colder color at Figure 15, while demonstrating higher confidence in recognizing more common relationships where it shows a warmer color. In Figure& 8, we observe that the model
E.3. Hyperparameter choice and robustness
For ADAPTVIS We use the same hyperparameters for SCALINGVIS in Section 4.2. For ADAPTVIS, we op- timize 01, 022 and B using the validation set (randomly sampled 20% from each subset) from each distribution. We also show our robustness for hyperparameters in Appendix. Notably, we find that model performance is robust across a range of these hyperparameters, generalizing effectively to
15

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
2
Left
Right
On
Under
Behind
Front
Spottal Relationship LLaVA Training Data
Confidence for Different Relationships
Corr_A
22.7%
22.7%
12000
Core A.
0.36
0.54
0.36
0.69
Datasets
Cort B
0.38
0.48
0.33
0.55
200
23.00 0%
34.0%
Coco one
0.43
0.48
0.44
0.48
vG_owc
+7.0%
44.0%
2:00
Coco too
0.43
0.46
0.42
0.50
0.25
0.5
0.75
Pas
Fart
Reletionships
Inder
1db
Lift--ight rilationships are more prevalent real data than other types
The training data for LLaVA primarily consists of real data and also contains blases
Model are more confidint with the "loft" and "right"
Figure 15. Label distributions across subsets capturing relationships between object pairs in WhatsUp. For real datasets like COCO and VG, severe label imbalance evident, with "left" and "right" being more common than other relationships 2 show the statistics in LLaVA training data. We could imbalance problem still holds. 4 The model's average confidence in different golden spatial relationships within these four-option subsets in WhatsUp. "On etc" includes "on, above, top". while "Under etc" includes "under, below, bottom".. The red heatmap box highlights instances where the model is confident in its generation, while the blue box indicates the opposite. We can observe that for familiar relationships, such as "left" and "right" (highlighted with two red boxes), the model shows higher confidence.
other subsets within the same distribution (as demonstrated in Table 8). We maintain consistency by using the same range of values for both methods. For B8 we adjust per dataset: for UP, we select values from [0.3, 0.65] for LLaVA-1.6 and [0.2, 0.55] for LLa with grid size of 0.05 (this higher range is due to VLa-- generally exhibiting higher confidence than LLaVA-1.5); for VSR, we take the mean value of the average confidence scores corresponding to the two labels.
Image Attention Logits by Layers
1.0
0.8
Sun Imaa's Alamion
Sun of lut's Alanban
0.4
0.2
0.0
0
4
8
10
14 16 Layers
18
22 24 28
Out-of- Ailoof-DDmain Test of Hyperparameters. We evalu- ated the generalizability of common hyperparameters across datasets. To this end, we applied the same set of four-option prompts to Controlled Images and COCO subsets. Results in Table 8 indicate that ADAPTVIS consistently performs well across all subsets, confirming its generalizability.
Figure 16. The variance of image token's 's attention scores through the layers in Cont_B benchmark.
Image Attention Across Layers
1.0
COCO, OA one _ojj (mean=0 1139]
0.5
CoCO two (mesn=0.1052)
VG, DA oel.oeeecee one_oty (meen=0.11411)
VG QA two ob) (m15n-0.3154)
Model
Cont_A
Cont_B
Coco_one Coco_two
0.4
Acc
P-Ace
S-Ace
Ace
P-Ace
S-Ace
Acc
Acc
ILAVA
60.3 40.6
0.0
73.1
41.6
3.7
53.0
58.2
0.2
+Ours
60.3
41.8n1
n4
48.3
13.5 ws. 53.6
19.4
B
0.5
0g 122 B - 0.3
26
Table 8. OOD test results on WhatsUp (Metrics in x x10= for LLaVA1.5. Arrows show growth over baseline. P-Ace and S-Ace are Pair and Set Accuracy.
Figure 17. Image attention COCO and VG dataset.
Figure 17 illustrates the attention pattern for real images. Consistently, we observe that image attention remains sparse across all subsets.
E.4. More Attention Analysis
The sparcity To determine whether our observation about the sparse attention pattern holds across benchmarks, we examine the attention patterns in other subsets of WhatsUp. Figure 16 presents the attention pattern for Cont B, while
AUROC Analysis of attention score and confidence we conduct a calibration experiment using two statistical ap-
16

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
100
Attention-Max Attention-eaan Confidence acc parformance
80
50.6
70
80
60.7
677
C8.
69
58.8
90.8
40
20
Cont_A
Cont_6
COCO_one
COCO_tw
Figure 18. AUROC of attention scores 17th layer relative to model's confidence.
proaches. The first approach sums the attention scores as- signed to the image as a metric, while the second extracts highest attention score among the image tokens. These metrics are then evaluated assess their effectiveness in distinguishing between correct and incorrect generations. However, as shown in Figure 18, the AUROC score using attention (the yellow and green bar) is consistently lower than the AUROC score of the model's self-confidence (the yellow bar), which we measure by the probability of the output tokens. Additionally, we observe that the maximum attention score provides better calibration than the average attention score, suggesting that key information aligns more closely with maximal attention values. This suggests that the assumption "the more attention the model pays to the image, the more accurate the results" holds only partially true.
E.5. AUROC anlysis of the overlap between YOLO and the attention scores
We present the AUROC of the overlap between YOLO an- notations and attention scores on the Controlled dataset in Figure and Figure 20. The results demonstrate that the AUROC remarkably high in the middle-to-high layers, indicating that the attention pattern serve reliable metric for detecting factual errors.
However, in some cases, this relationship is not as appar- ent, as shown in Figures 21, 22, 23, and 24. In these in- stances, the AUROC remains relatively modest, which can largely be attributed to errors in YOLO's annotations. We identified four primary error types leading mismatches between YOLO annotations and actual object instances: Missed Detection, Misclassification, Bounding Box Error, and Ambiguous Refer. These enror patterns are illustrated in Figure 25.
17

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
Figure 19. AUROC of the overlap between YOLO annotations the attention patterns on all layers Cont. A benchmark.
Figure AUROC overlap hetween YOLO annotations the attention patterns on layers Cont henchmark.
18

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
Figure 21. AUROC of the overlap hetween YOLO annotations and the attention patterns all layers in COCO benchmark.
24
Figure 22. AUROC of the overlap between YOLO annotations and the atlention patterns all layers in COCO benchmark.
19

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
car
Figure 23. AUROC of the overlap hetween YOLO annotations and the attention patterns all layers in VG benchmark.
Figure 24. AUROC of the overlap between YOLO annotations and the atlention patterns all layers in VG.two benchmark.
20

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
Bento #297
Where is the sandwich in the photo? sandwich Missed Detection
Where the in relation the refrigerator? Fail to dieecct refrigerator Missed Detection
COCO_one _one
COCO_two
phone
Where is the man in the photo?
Recognize the elephent Misclassification
VG_one
relation lady? Two ladies detected on both directions Ambiguous Refer
Where relation women?
Phone bounding box too large Bounding Error
Figure 25. Error cases in YOLO annotation. Under each image present the related question, the error in YOLO annotation and its type, and the source where the image and question comes from.
21

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Arcas
E.6. What changes after our intervention?
Baseline
ScalingVis
AdaptVis
How the confidence changes by using different as? To illustrate the impact of coefficients greater less than 1 on different relationships, Figure 8 shows accuracy and confi- dence variance for various ground- tround-truth relationships. Re- sults indicate that for familiar relationships like "left" (red) and "right" (blue), coefficients greater than boost accuracy and confidence. Conversely, for less familiar relationships like "under" (green) and "on" (yellow), coefficients less than improve accuracy and confidence.
Time (set baseline
1.00
1.02
1.64
Table 9. Inference-time statistics. It shows that Scaling Vis intro- duces negligible computation compared to the baseline.
Dataset
POPE-A
POPE.P
POPE-R
POPE-AVG
GQ4
VQAr2
Baseline Scaling
81.0
86.4
88.4
85.3
56.1
74.0
81.
86.6
88.6
85.6
56.3
74.2
Table 10. Results on QA benchmarks. Adversarial. POPE-P is POPP-Popular. POPE-R is POPE-random. POPE-AVG the is average score of the three subsets.
POPE-A is POPE-
How do the absolute values of attention scores vary be- fore and after intervention? Figure 26 visualizes the at- tention logits for two-option Cont A (a=0.5) and six-option VG (a=2). An a of 0.5 decreases the absolute value of logits across layers, while an a of 2 increases them as lay- ers progress. This indicates that an a larger than I could strengthen the model's orginal attention pattern.
AdaptVis incurs higher computational overhead.
E.10. Results on More Benchmarks
We further evaluate our method on several Question An- swering benchmarks, including POPE (Li et al., 2023), GQA (Hudson & Manning, 2019), and VQ (Antol et al., 2015). As shown in Table 10, our method consistently out- performs the baseline across all benchmarks, demonstrating its strong generalization capability in the general Question Answering setting.
E.7. Prompt Sensitivity Analysis
To assess the robustness our method, we varied the num- ber of options in prompts for specific WhatsUp subsets. For Cont_A and Cont_B, reduce the number of options to two, simplifying the task. Conversely, for COCO subsets, we increase the options to six, making it more challenging- Table 27 shows that our SCALINGVIS method maintains consistent performance across all cases, demonstrating ro- bustness.
F. URLs of Code and Data
We provide the code and data at /aaps/////nn/mmuuu.u. 4open. wtttt//seeinnneeeceeeeeeee Bttpaa/eee//aaattta--B5553
The running arguments are configured as follows: dataset specifies the evaluation dataset (e.g., ContzolLed..magessuu model-name selects the model (e.g., 1laval 5); method determines the evaluation ap- proach (scal lecaling_vi ing.vis s or adapt_vis) For scal scaling.vis, we weight1 can be set to values in [0.5, 0.8, 1.2, 1.5, 2.0]. For adapt.vis, weight1 ranges in [0.5, 0.8] and woight2 in [1.2, 1.5, 2.0], with threshold controlling the adaptation sensitivity. The --download flag enables automatic dataset downloading.
E.8. Reverse curse?
Additionally, we observe that the model exhibits a "reverse curse" phenomenon similar to that has been seen in language models (Berglund al., 2023).
When we reverse the order of the entities in Cont_A (e.g., asking the model, "Where is the armchair in relation to the beer bottle?" instead of "Where is the beer bottle in relation to the ammchair?"),, there is significant drop in per- formance. As shown in Figure 28, the model's performance declines dramatically from a high score to an exceptionally low one. This reveals that the existing VLMs's attention pat- tern and generation results could be significantly impacted by the prompt. Our methods, however, consistently improve the model's performance. It suggests that adaptively in- tervening the attention score is a generalizable method for different prompts.
E.9. Efficiency
We evaluate inference times for different decoding methods in Table 9. ScalingVin Vis introduces only negligible additional computation time compared to the baseline (greedy decod- ing). However, due to the need for computing the threshold,
22

---

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective Focus Arcas
logits Distribution
Attention logits Distribution
stp Attention logits Distribution Vis si,re, on, waghht Skooe
2227
Attention logits Distribution
Concolist_ veigesdo
Cintisike bet & magr1356
Concot meigh110
lut nagniLi
Inc wnight2.0
vn wrife (hre))
Alentoo Logil
Alien
Allor.
Figure 26, Attention logit distribution hefore (blue) and after (pink) intervention in the 14th layer (a randomly chosen middle layer). From left right, the plots represent: and attention values across heads for Cont_A, and mean and max attention for Cont_ B. Red dots mark cases corrected by our intervention. We could see that a of 0.5 shifts the line left, while a of 2 shifts it right.
case
Raw
Ours
CouLA
Cont_
Coco
Coco. two
Model
80
76.4
LLaVA-15
764
43.0
4.8
74.6
41.0
1.2
30.8
42.6
+0urrs
864
110.0
118.2
27.9
123.1
878
59.3
1183
23.0
120.8
36.0
15.2
48.6
77.3
38.8
35:2
Best
0.5
0.5
Figure
Results on WhatsUp (Metries when changing prompts for other option numbers. Arrows show improvement over greedy decoding.
Figure 28. Performance comparison before and after SCALINGVIS intervention (a 0.5).
23


Reviews:
[{"review_id": "ygWYwUm8TM", "reviewer": "Reviewer_3Fcp", "summary": "This paper introduces ADAPTVIS, an adaptive attention mechanism designed to enhance spatial reasoning in vision-language models (VLMs). By analyzing attention distributions, the authors identify that errors often arise when models focus on irrelevant image regions and that attention patterns differ between familiar and unfamiliar spatial relationships. ADAPTVIS addresses these issues by dynamically re-weighting attention maps based on model confidence during inference, improving geometric understanding with minimal computational cost. Evaluations on benchmarks like WhatsUp and Visual Spatial Reasoning (VSR) demonstrate significant accuracy gains, highlighting ADAPTVIS’s effectiveness in refining spatial reasoning in large vision-language models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and advice, We address your concerns below.\n\n> it is unclear whether the findings would generalize to other VLMs.\n\n- We agree generalizability is important so we have especially included different variances of LLaVA-series models, with LLaVA-1.5 (224×224 visual encoder, MLP projection layer) and LLaVA-1.6 (336×336 visual encoder, resampler projection layer), since they are the most widely-used open-source architecture but with different model features like confidence, as mentioned in line 742. \n- To assess the generalizability of our findings, we extend our analysis to Qwen2-VL, a SOTA VLM with different architecture. We show our findings for VLMs’ attention patterns are still consistent [see response for Reviewer 3nwP Q1].\n- To further assess the generalizability of our intervention approach, we extend experiments to Qwen2-VL. We intervene in the image attention distribution using our temperature-scaling method, showing consistent improvements below, particularly in challenging cases. For example, on `VG_two_object`, where the baseline performance is the lowest among all benchmarks, our method yields a significant improvement of **10+ absolute points**. The gains observed on Qwen2-VL further demonstrate the generalizability of our approach.\n\n| **Benchmark**|**Qwen2-VL**| **Qwen2-VL + Attention Intervention** | \n| :-: | :-: | :-: | \n|VSR|78.96|81.60 (↑ 2.64)| \n|Coco_one_obj|76.64| 78.03 (↑ 1.39) |\n|Coco_two_obj|75.28| 76.52 (↑ 1.24) |\n|VG_one_obj |74.89| 75.11 (↑ 0.22)|\n|VG_two_obj |56.22|66.95 (↑ 10.73) |\n|Controlled_A|98.18 |98.18 (↑ 0.00)  |\n|Controlled_B|91.73|92.97 (↑ 1.24) |\n\n- We also evaluate more generic benchmarks for generalizability, including POPE, GQA, and TextVQA, which proves that attention intervention can maintain the performance on more general tasks without hurting the performance. Compared with spatial reasoning tasks, general QA tasks achieve relatively smaller improvement. This may be because such tasks are less sensitive to the geometric structure of image attention. For example, when asked `Is there a dog in this picture?`, the model only needs to detect the object's presence, and is therefore less likely to suffer from misallocated attention across spatial regions.\n\n| **Benchmark**|**Qwen2-VL**|**Qwen2-VL + Attention Intervention** |\n| :-: | :-: | :-: | \n|POPE-Overall| 86.32| 87.09 (↑ 0.77)|\n|POPE-P |86.47| 87.29 (↑ 0.82)|\n|POPE-A |85.07| 85.80 (↑ 0.73)|\n|POPE-R |87.46| 88.22 (↑ 0.76)|\n|GQA |62.09|62.17 (↑ 0.08)|\n|TextVQA|79.18|79.26 (↑ 0.08)|\n\n- We believe this assumption is theoretically generalizable. The only assumption we have is the different information density between image and text. In VLMs, attention is distributed across both textual and visual tokens. Since key information in images tends to be more sparse compared to text, the attention distribution over visual tokens can be more difficult to allocate accurately. \n\n>The paper claims that the model's confidence is correlated with attention correctness…More robust, quantitative analysis to support the correlation between attention distribution and model confidence.\n\n- As detailed in Lines 184-251 of Section 4.1, we use YOLO to annotate the relevant entities in the images, and conduct quantitative analysis on AUROC of the overlap between YOLO annotations and the attention patterns , as well as performance are shown in Figure 7 in the main paper and, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22, Figure 23, Figure 24 in the Appendix. \n- In Figure 25 in the Appendix, we perform quantitative analysis on **confidence and accuracy and their correlation with the coefficient**, where we can see the confidence and accuracy follow similar trends as the temperature coefficient varies, supporting the use of confidence as a reliable proxy for performance.\n- From the first and third subfigures: for low-confidence relationships, applying a small coefficient (<1) improves performance; for high-confidence relationships, a coefficient (>1) improves performance.\n- We will move the additional quantitative evidence (from Appendix) into Section 4.2 to make this reasoning clearer in the paper.\n\n> A more adaptive approach that adjusts thresholds dynamically based on the specific task or dataset...\n\n- We believe there may be a misunderstanding. As described in lines 323, we adaptively adjust the coefficient using a small held-out subset (20% of the whole subset as the validation set) from the specific target task or dataset, rather than pre-defining hyperparameters. \n- Additionally, we propose a more adaptive variant **ScalingVis** with only a single hyperparameter $\\alpha$ to modulate the attention distribution based on model confidence. This design maintains both simplicity and adaptability, contributing to the method’s robustness and generalizability across tasks.", "claims_and_evidence": "1. The paper claims that the model's confidence is correlated with attention correctness and that high confidence leads to sharper attention, while low confidence allows the model to focus more diffusely on the image. However, there is limited empirical evidence to substantiate this claim. The analysis in Section 4, especially regarding the correlation between attention distribution and model confidence, is not comprehensive and largely based on qualitative examples. A more robust, quantitative analysis would strengthen this claim and make the argument more convincing.", "methods_and_evaluation_criteria": "1. The method of modifying attention during inference based on model confidence is conceptually sound, especially as it directly targets improving spatial reasoning in VLMs. However, the method's reliance on pre-defined threshold values (alpha1, alpha2, beta) for adjusting attention is a significant drawback. A more adaptive approach that adjusts thresholds dynamically based on the specific task or dataset could improve the method’s robustness and generalizability. Additionally, more clarity on how these hyperparameters interact with the model and their impact across different VLMs would be beneficial.\n2. The evaluation is conducted on several benchmarks, such as WhatsUp and Visual Spatial Reasoning (VSR). While these benchmarks are relevant for testing spatial reasoning, the datasets used are somewhat limited in scope. The authors should consider evaluating ADAPTVIS on more commonly used, real-world datasets, such as GQA or VQA-v2, to assess the method's generalizability and performance across a broader set of tasks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. One important concern with the current experiments is that they are solely based on the LLaVA-series models (LLaVA-1.5, LLaVA-1.6). While these models are a useful starting point, it is unclear whether the findings would generalize to other VLMs.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on prior work in spatial reasoning for VLMs by introducing a novel attention adjustment mechanism based on model confidence, addressing issues identified in earlier studies on attention misalignment and spatial reasoning failures in VLMs. This approach is closely related to efforts exploring the role of attention mechanisms in improving VLM performance on tasks requiring geometric and spatial understanding.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. Some axis titles are missing in the figures (Figures 5 and 7), which could make them harder to interpret.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RCZAlwCyTU", "reviewer": "Reviewer_3nwP", "summary": "The paper examines the attention patterns in the visual-language models, and finds patterns which might explain why spatial reasoning can be hard for VLMs. Specifically, they first find that a large chunk of the attention is focussed on the text stream, even though the number of visual tokens are more. However, they further find that manually making the attentions larger on visual tokens do not help performance. Then, they find that when the visual attention is on the right objects, it typically leads to the right answer; and vice versa. They also find that self-confidence is a good proxy for model's confidence -- when the model is confident, it is usually looking at the right location, and thus the attention map can be sharpened. When the model is less confident, it helps to make the attention map more diverse to let it look at more region of the visual image. This proposed technique of modifying attention maps based on confidence helps improve performance on several benchmarks that they evaluate on.\n\n## Update after rebuttal\nMy initial score for this paper was 4 -- with main concerns based on the datasets considered and that only one single VLM was evaluated (both concerns shared by other reviewers as well). I thank the authors for their rebuttal. I read the reviews of other reviewers as well as author's response: I revised my score from 4 to 3 because I think the new results show that the proposed interventions are barely effective on new (and more standard) benchmarks; and the performance improvement is less pronounced with Qwen2-VL (except in VG_two_obj). Nevertheless, the exploration done in this paper is solid and might be useful to foster research. \n\nI am recommending this paper for acceptance with an implicit assumption (and belief) that the authors will include the rebuttal experiments in their main paper -- without these results, the papers' results might be misleading and over-promising. I asked the authors whether they plan to do this, and they didn't reply, so I am not sure if the authors indeed want to include these results. I will leave this to AC.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and detailed comments! Your suggestions will be helpful in improving the paper. We address your concerns below.\n\n\n\n\nQ1: \n>whether the claims of this paper are generally valid for VLMs or is it specifically valid for LLAVA models that this paper tests on.\n\n\n- The claims are generally valid for VLMs. \n- The first claim **image receives much less attention than the text tokens** is generally valid for VLMs: we calculate the Qwen2-VL’s attention scores below (average attention logits for single text/image tokens respectively), which matches our previous claims.\n\n\n\n\n| **Benchmark**  | **Text attention scores** |  **Image attention scores**    |\n|:-----:|:----:|:----:|\n| Controlled_A  | 1.57e-02 | 7.59e-05  |\n| Controlled_B  | 1.58e-02 | 7.69e-05 |\n| Coco_one_obj  | 1.77e-02 | 4.48e-04  |\n| Coco_two_obj  | 1.65e-02  | 3.50e-04 |\n| VG_one_obj  | 1.54e-02| 4.75e-04 |\n| VG_two_obj  | 1.42e-02| 4.19e-04|\n\n\n\n\n- The second important claim is still valid about **confidence could serve as a proxy for the model performance** as below: \n   - Spatial relationships with lower confidence, such as \"Left\" and \"Under\" tend to exhibit lower accuracy when compared to those with higher confidence, like \"On\" and \"Right\". \n   - After the intervention with AdaptVis, certain spatial relationships like “Left\" show improved performance, accompanied by an increase in confidence. This demonstrates a pattern consistent with our observations for LLaVA, as depicted in Figure 25 in the Appendix. \n\n\n\n\n| **Benchmark**       | **Qwen2-VL Uncertainty** |  **Qwen2-VL+ScalingVis Uncertainty**  |**Qwen2-VL Acc** |   **Qwen2-VL+ScalingVis Acc**   |\n|:---:|:----:|:----:|:----:|:-----:|\n| Left  |  0.4408  | 0.4474  |70.11  | 74.71 |\n| On | 0.5758    | 0.5668 |100  | 100|\n| Right  | 0.5982   | 0.5911 |  100  | 100  |\n| Under  | 0.5554 | 0.5418 |98.82  | 98.82 |\n\n\n\n\nQ2:\n> more experiments on other VLM and other benchmarks.\n\n\n\n\n- Experiments on Qwen2-VL, a SOTA VLM with different architecture. We intervene in the image attention distribution using our temperature-scaling method, showing consistent improvements below, particularly in challenging cases. For example, on `VG_two_object`, where the baseline performance is the lowest among all benchmarks, our method yields a significant improvement of **10+ absolute points**. The gains observed on Qwen2-VL further demonstrate the generalizability of our approach.\n\n\n\n\n| **Benchmark**  | **Qwen2-VL** | **Qwen2-VL + Attention Intervention**  | \n| :-: | :-: | :-: | \n| VSR  | 78.96  | 81.60 (↑ 2.64)  | \n| Coco_one_obj | 76.64  | 78.03 (↑ 1.39)  |\n| Coco_two_obj | 75.28  | 76.52 (↑ 1.24)  |\n| VG_one_obj | 74.89  | 75.11 (↑ 0.22)|\n| VG_two_obj | 56.22  | 66.95 (↑ 10.73)  |\n| Controlled_A  | 98.18  | 98.18 (↑ 0.00)  |\n| Controlled_B  | 91.73  | 92.97 (↑ 1.24)  |\n\n\n\n\n- Experiments on more benchmarks, including POPE, GQA, and TextVQA, which proves that attention intervention can maintain the performance on more general tasks without hurting the performance. Compared with spatial reasoning tasks, general QA tasks achieve relatively smaller improvement. A possible reason is that such tasks are less sensitive to the geometric structured distribution of image attention. For example, given a question like `Is there a dog in this picture?`, the model only needs to detect the presence of an object, and is therefore less likely to suffer from misallocated attention across spatial regions.\n\n\n\n\n\n\n\n\n| **Benchmark**  | **Qwen2-VL** | **Qwen2-VL + Attention Intervention** |\n| :---: | :---: | :---: | \n| POPE-Overall  | 86.32    | 87.09 (↑ 0.77) |\n| POPE-P | 86.47| 87.29 (↑ 0.82) |\n| POPE-A  | 85.07| 85.80 (↑ 0.73)  |\n| POPE-R | 87.46| 88.22 (↑ 0.76) |\n| GQA | 62.09| 62.17 (↑ 0.08)  |\n| TextVQA  | 79.18| 79.26 (↑ 0.08) |", "claims_and_evidence": "Yes, mostly the claims are well supported. The hypothesis of various attention patterns are well tested and the resulting technique of adaptively changing the attention patterns works well on the datasets that they are tested on. \n\nHowever, it is unclear whether the claims of this paper are generally valid for VLMs or is it specifically valid for LLAVA models that this paper tests on. For eg. the claim about the model focusing more on the language tokens, compared to visual tokens could be true for LLAVA like models because only a small adapter layer is trained to add visual modality to the LLM (in addition to final small LORA on the whole base LLM). It could also be something specific to a training data or methodology detail of LLAVA models. Testing on other VLMs would make the analysis and claims more stronger. \n\nSecondly, I would recommend testing on more popular spatial reasoning datasets (like RefCOCO family / BLINK) etc -- I am not particularly tied to the datasets I mentioned, but some datasets which are commonly used for evaluating VLMs.", "methods_and_evaluation_criteria": "Yes -- see some concerns regarding the specific VLM used and evaluation datasets in claims and evidence.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes -- they are sound and valid", "relation_to_broader_scientific_literature": "It is a well-known fact that VLMs struggle with spatial reasoning. Getting insights on why that might be, and ways to fix them zero-shot are relevant to the scientific community.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is very well-written and is a joy to read -- all the hypothesis and conclusions naturally flow from each other.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- I would like to see some analysis on different VLMs (or some reasoning on why that might not be important)\n- I am not expecting the authors to test on new datasets, but explaining why that was not considered would help", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rvVQeX3Yl", "reviewer": "Reviewer_p97N", "summary": "This paper investigates the Visual Attention Distribution in VLM and finds that it affects vision-centric understanding. Based on such observation, paired with the confidence score of the model when generating tokens, the authors propose ADAPTVIS, a temperature scaling mechanism for the attention scores that effectively helps with better understanding spatial relations.\n\n\n## update after rebuttal\nThank the author for the point-by-point response to all of my raised issue. I appreciate the attention scores analysis as well as the additional experiments. While it is true that the proposed method is less effective on the new model, it is still acknowledged that the overall structure of the paper is well-established and can potentially inspire future works. Therefore I'll maintain my positive score 4 (Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer p97N for the encouraging comments and thoughtful feedback. Below, we address the concerns raised in detail.\n\n> I'm wondering how well the proposed method works for other VLMs.\n\n- Experiments on Qwen2-VL, a SOTA VLM with different architecture. We intervene in the image attention distribution using our temperature-scaling method, showing consistent improvements below, particularly in challenging cases. For example, on `VG_two_object`, where the baseline performance is the lowest among all benchmarks, our method yields a significant improvement of **10+ absolute points**. The gains observed on Qwen2-VL further demonstrate the generalizability of our approach.\n\n| **Benchmark**  | **Qwen2-VL** | **Qwen2-VL + Attention Intervention**     | \n| :-----: | :-----: | :-----: | \n| VSR  | 78.96    | 81.60 (↑ 2.64)  | \n| Coco_one_obj | 76.64    | 78.03 (↑ 1.39)  |\n| Coco_two_obj | 75.28    | 76.52 (↑ 1.24)  |\n| VG_one_obj | 74.89    | 75.11 (↑ 0.22)|\n| VG_two_obj | 56.22    | 66.95 (↑ 10.73)   |\n| Controlled_A  | 98.18    | 98.18 (↑ 0.00)  |\n| Controlled_B  | 91.73    | 92.97 (↑ 1.24)   |\n\n- Experiments on more benchmarks, including POPE, GQA, and TextVQA, which proves that attention intervention can maintain the performance on more general tasks without hurting the performance. Compared with spatial reasoning tasks, general QA tasks achieve relatively smaller improvement. A possible reason is that such tasks are less sensitive to the geometric structured distribution of image attention. For example, given a question like “Is there a dog in this picture?”, the model only needs to detect the presence of an object, and is therefore less likely to suffer from misallocated attention across spatial regions.\n\n| **Benchmark**         | **Qwen2-VL** | **Qwen2-VL + Attention Intervention**     |\n| :-----: | :-----: | :-----: | \n| POPE-Overall      | 86.32    | 87.09 (↑ 0.77)          |\n| POPE-P            | 86.47    | 87.29 (↑ 0.82)          |\n| POPE-A            | 85.07    | 85.80 (↑ 0.73)          |\n| POPE-R            | 87.46    | 88.22 (↑ 0.76)          |\n| GQA           | 62.09   | 62.17 (↑ 0.08)          |\n| TextVQA          | 79.18   | 79.26 (↑ 0.08)          |\n\n> provide more results (analytical and experimental) using other VLMs\n\nWe present the additional experimental results above. For the analytical results, we also extend our analysis to the Qwen2-VL model and find that our insights generally hold across VLMs. \n- The first claim **image receives much less attention than the text tokens** is generally valid for VLMs: we calculate the Qwen2-VL’s attention scores below (average attention logits for single text/image tokens respectively), which matches our previous claims.\n| **Benchmark**  | **Text attention scores** |  **Image attention scores**    |\n|:-----:|:----:|:----:|\n| Controlled_A  | 1.57e-02 | 7.59e-05  |\n| Controlled_B  | 1.58e-02 | 7.69e-05 |\n| Coco_one_obj  | 1.77e-02 | 4.48e-04  |\n| Coco_two_obj  | 1.65e-02  | 3.50e-04 |\n| VG_one_obj  | 1.54e-02| 4.75e-04 |\n| VG_two_obj  | 1.42e-02| 4.19e-04|\n- The second important claim is still valid about **confidence could serve as a proxy for the model performance** as below: \n   - Spatial relationships with lower confidence, such as \"Left\" and \"Under\" tend to exhibit lower accuracy when compared to those with higher confidence, like \"On\" and \"Right\".\n   - After the intervention with AdaptVis, certain spatial relationships like “Left: show improved performance, accompanied by an increase in confidence. This demonstrates a pattern consistent with our observations for LLaVA, as depicted in Figure 25 in the Appendix. \n| **Benchmark**       | **Qwen2-VL Uncertainty** |  **Qwen2-VL+ScalingVis Uncertainty**  |**Qwen2-VL Acc** |   **Qwen2-VL+ScalingVis Acc**   |\n|:---:|:----:|:----:|:----:|:-----:|\n| Left  |  0.4408  | 0.4474  |70.11  | 74.71 |\n| On | 0.5758    | 0.5668 |100  | 100|\n| Right  | 0.5982   | 0.5911 |  100  | 100  |\n| Under  | 0.5554 | 0.5418 |98.82  | 98.82 |\n\nQ1:  \n>What is the major difference between a spatial-relation understanding task and a standard QA task that makes the proposed method less effective on standard QA task?\n\n\n- That’s a great question. In fact, our initial and primary motivation was to gain deeper insights into the spatial reasoning capabilities of VLMs. Through our analysis, we observed that the **geometric distribution of image attention** plays a critical role in this reasoning process. Based on this observation, we proposed two **temperature scaling techniques** to adjust the attention distribution and address the problem.\n\n\n- As for the relatively smaller improvement on general QA tasks, we believe this may be because such tasks are **less sensitive to the geometric structure** of image attention. For example, given a question like *“Is there a dog in this picture?”*, the model only needs to detect the presence of an object, and is therefore **less likely to suffer from misallocated attention** across spatial regions.", "claims_and_evidence": "The claims in the submission is mainly on investigation of the attention mechanism and its relation with the model's output. I'm wondering have the author tried other VLMs to see if such an observation is universal.", "methods_and_evaluation_criteria": "The proposed methods make sense based on the thorough study of the attention mechanism. The evaluation criteria is standard.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The method and experimental design are straightforward and easy to understand. One concern about the design is that the proposed method is only applied to LLaVA. I'm wondering how well the proposed method works for other VLMs.", "relation_to_broader_scientific_literature": "The key contribution of the paper including the investigation of attention mechanism for spatial understanding would help with ideas in the VLM hallucination domain. Such a design would help with addressing the inefficiency of contrastive decoding method, which usually requires more than once inference of the model.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The idea is straightforward, intuitive, easy-to-understand and easy-to-implement. The simple-yet-effective approach make it possible for this work to have great impact in the VLM-related area.\n2. The writing is easy to follow, clearly stating the observations / the relations between each part.\n\nWeaknesses:\n1. As mentioned above, it'll be appreciated if the authors can provide more results (analytical and experimental) using other VLMs.", "other_comments_or_suggestions": "1. Typo: Ln. 375, right col, \"we emply ...\" should be \"we employ ...\"", "questions_for_authors": "As mentioned in the supplementary material, the proposed method could also work on general VQA benchmark. What is the major difference between a spatial-relation understanding task and a standard QA task that makes the proposed method less effective on standard QA task?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "REF3pDoivJ", "reviewer": "Reviewer_3qjz", "summary": "The authors explore how vision-language models’ struggle with spatial reasoning, focussed on how misdirected attention (i.e. to irrelevant parts of image) within transformer blocks contributes to such behavior. They analyze attention patterns and report how attention prioritizes text tokens over image tokens. They also note that attention to wrong parts of an image is an issue and how model logit probability can be a proxy for model confidence. Using these ideas, they propose an approach to adjust attention based on confidence levels: sharpening or smoothing attention weights of image tokens based on model confidence per sample. They evaluate their model on two datasets, WhatsUp and VSR. These datasets contain mixtures of natural and synthetic images, with mostly synthetic captions. The results show promise of their proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3qjz for the comments. However, we are concerned that some comments suggest a disconnect or oversight of our paper. We respond to each point below, and respectfully encourage the reviewer to revisit our work.\n> Essential References Not Discussed\n\nAmong the four references mentioned, **[1] and [3] are precisely the benchmarks we work on**, and our experiments are **exactly built on these two papers** with the same dataset names WhatsUp and VSR **cited and mentioned throughout the paper**. Both are benchmark papers that present datasets and baselines like prompting, without proposing novel methods or addressing **mechanistic interpretability (MI)** .\n\n[2] and [4] are: (1) about **LLMs rather than VLMs**, while we mentioned in the title that we work on VLMs; (2) neither about MI, while we highlighted in abstract (Lines 18-20), introduction (Lines 43-45), conclusion (Lines 422-423) that our paper is to **open up VLMs and observe their internal behavior to understand their failures in spatial reasoning**. \n\nWe believe the reviewer has a **misunderstanding** of the paper that we are NOT proposing a new model to better leverage object grounding to beat SOTA models, but rather conducting MI for VLMs, ​​by analyzing attention patterns and applying targeted interventions to understand their behavior. So the suggestions to compare with other object-grounding methods reflect the misunderstanding, detailed in W2. \n\n(MI is a well-established area to reverse-engineer complex systems, offering a more granular and causal understanding of how they process information. [5,6,7])\n\n[5] A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models  \n[6] Mechanistic Interpretability for AI Safety: A Review  \n[7] awesome-mechanistic-interpretability-lm-papers in Github  \n\nW1:\n > datasets being small and primarily synthetic\n- Benchmarks are not primarily synthetic, with 85% natural images and all human annotated captions (reformatted to generative QA settings better suited for SOTA decoding-only VLM architectures, Lines 143-149).\n| |Cont_A|Cont_B|Coco_one|Coco_two|VG_one|VG_two|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Synthetic image|yes|yes|no|no|no|no|\n|Synthetic caption|no|no|no|no|no|no|\n|samples|406|408|1160|291|2240|440|\n- MI research requires clean datasets to observe model mechanisms. Existing MI studies use smaller, clean datasets rather than larger, noisy ones [5, 6, 7]. We have made every effort to include all available datasets with clean, high-quality spatial relationship annotations.\n\n W2: \n> do not compare to any prior work in their paper.\n- We respectfully disagree. With a focus on MI of VLMs, **our experiments on attention intervention aims to give conclusions about how VLMs work**. Thus, **experiments are expected to be between w/ and w/o intervention**, rather than comparing with other object-grounding methods. \n- We did compare our methods with two prior decoding methods: Dola [8] and VCD [9] (Line 327). Like DoLa, we leverage internal signals but tailor them to VLMs using visual features. While sharing VCD’s goal of improving visual grounding through decoding, we go further by introducing fine-grained attention interventions for more precise calibration.\n- Again, we believe it reflects a misunderstanding of the paper, especially given the context that other three reviewers do not have such misunderstandings.\n\n[8] DoLa: Decoding by Contrasting Layers Improves Factuality in LLMs.  \n[9] VCD: Mitigating Object Hallucinations in Large VLMs through Visual Contrastive Decoding.\n\nW3: \n> Does not use established benchmarks.\n- We use established spatial-reasoning benchmarks WhatsUp [1] and VSR [3], and as detailed in Lines 143-149, we have to reformat the dataset from classification settings to QA settings to observe the attention behavior and information flow across layers of Transformers when VLMs generate correct or wrong answers, which is also a standard setting of MI studies [5,6,7] for observing the attention behavior. \n- We also include general VQA benchmarks in Table 6 at Appendix.\n\nW4:\n>Unclear visualizations in Figure 9 & 10\n\nThanks for the helpful suggestions! We will add more annotations to these figures.\n\nW5:\n> backing \"attention modification\" claims with some numbers\n\nThanks for the helpful feedback. We extend our experiments to test YOLO overlap w/ and w/o intervention in middle layers, where our observations (line 196) identify as key for processing spatial information. We evaluate two representative subsets: Controlled_A (synthetic images) and VG_two_obj (real images). \nAfter intervention, overlap rates improve on both. Attention–YOLO similarity increases by 2.9% on Controlled_A and 8.2% on VG_two_obj at layer 16, with similar trends across other middle layers.\n\nWhile it can support our claims, it is not a golden metric since attention map is influenced by the prompt, and YOLO may fail to detect objects in complex images (Figure 24). We will include additional discussions in our paper.", "claims_and_evidence": "No.\n\n1) They evaluate on two small datasets (that contain sythetic data). Results on these datasets are insufficient to back their claims.\n2) They explore two outdated VLMs to highlight spatial reasoning drawbacks. The analysis on more modern VLMs would make the claims stronger. Most newer VLMs show stronger spatial reasoning abilities simply by training on more data and augmenting their training data / tasks.", "methods_and_evaluation_criteria": "No, see above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experiments on 2 mostly synthetic datasets are insufficient. The focus on only older VLMs question the applicability of method to newer VLMs.", "relation_to_broader_scientific_literature": "Overlap with several existing prior works.", "essential_references_not_discussed": "Consider comparing with the baselines from these prior works. And discussing these methods more explicitly in related work to highlight how authors method differs. \n\n\n[1] What’s “up” with vision-language models? Investigating their struggle with spatial reasoning, EMNLP 2023\n\n[2] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs, CVPR 2024\n\n[3] Liu, Fangyu et al. “Visual Spatial Reasoning.” Transactions of the Association for Computational Linguistics 11 (2022): 635-651.\n\n[4] Hsu, Joy et al. “What's Left? Concept Grounding with Logic-Enhanced Foundation Models.” NeurIPS 2023.", "other_strengths_and_weaknesses": "**Strengths**\n1. The authors' innovative approach of adjusting attention at test-time for individual samples, using the model's output probability as a confidence proxy, is both clever and compelling.​\n2. Despite the datasets being small and primarily synthetic, the observed performance enhancements are promising.​\n3. The concept that generation confidence serves as a reliable indicator of attention accuracy is intriguing and opens new avenues for exploration in the vision-language domain.\n\n\n**Weaknesses**\n1. Insufficient experimentation:\n    * Results only on two small, primarily synthetic dataset\n    * Results only over two older variants of the same VLM\n   This raises concerns on applicability to real-world datasets and newer, stronger VLMs. \n\n2. Results difficult to compare:\n    * The authors do not compare to any prior work in their paper (i.e. no results from existing works are used as baselines)\n    * Even for baselines they provide, the authors use different dataset splits making it difficult to directly compare with number on the original papers. \n\n3. Does not use established benchmarks \n    * See Table 5 in [1], where results on standard VQA datasets focussed on spatial reasoning are provided (including numbers directly from prior works, making the results easy to compare and validate). Please include at least one such results table. \n    * There is one Table 6 (buried in appendix) that reports on some such datasets. There are no details on the baseline reported on it, and no discussion regarding the results in that table. \n\n4. Unclear visualizations in Figure 9 & 10\n    * The authors show pairs of images, but there is not apparent improvement visible in them.\n    * Consider explicitly highlighting (on the image) what changes the authors are talking about. \n    * Consider making the visualizations overlays (the original images seem to be shaded, making it difficult to see color details in them)\n\n5. Consider backing \"attention modification\" claims with some numbers\n    * In introduction, authors claim that \"by dynamically adjusting attention patterns, we effectively guide the model’s focus to better align with actual object locations and their spatial relationships.\" It is simple to test this on some object detection dataset - calculate attention mIoU with objects against a baseline. This will reveal if authors' method focusses attention more on object regions. This is a common form of evaluation in prior works. \n\n\n[1] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs, CVPR 2024", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors introduce ADAPTVIS, an adaptive attention mechanism designed to perform intervention to understand, and subsequently enhance spatial reasoning in vision-language models (VLMs). By analyzing attention distributions and their correlation with model confidence, ADAPTVIS dynamically re-weights attention maps during inference to improve geometric understanding with minimal computational cost. This approach is framed through the concept of Mechanistic Interpretability (MI), aiming to understand VLM behavior related to spatial tasks. Authors evaluate their method on spatial reasoning benchmarks like WhatsUp and VSR, along with additional experiments on standard VQA datasets and the Qwen2 VLM, demonstrating accuracy gains.

The reviewers reached a final consensus of one Accept and three Weak Accepts. Reviewers particularly highlighted the importance of addressing VLM spatial reasoning, the novelty and intuition of the MI-driven approach using confidence to modulate attention, the method's simplicity and low overhead, and the potential for impact.

Some concerns were also raised initially, about the generalizability of the findings beyond LLaVA models and the specific spatial datasets used, the need for more quantitative evidence linking confidence and attention patterns, and most importantly, a certain amount of confusion about the evaluation context, particularly regarding baseline comparisons and the way it manifested in the dataset format (raised by Reviewer 3qjz). The authors engaged actively in the rebuttal and reviewer discussion to alleviate the concerns: they provided additional experiments, including results on the Qwen2-VL architecture and standard benchmarks like GQA, TextVQA, and POPE, supporting the generalizability claim. There was also some analysis provided on attention modification and its link to confidence - and justification on why reformatting datasets was necessary to support the mechanistic interpretability framework. This clarification to a final positive consensus among all reviewers.

While the performance gains observed on the newer VLM (Qwen2-VL) and general VQA benchmarks were noted by reviewers as being more modest compared to the original LLaVA/spatial task results (and also noting that more recent, larger VLMs have largely improved upon spatial grounding, object pointing etc.), the AC feels that the core contribution lies in the MI analysis and the proposed intervention technique itself and provides valuable insights (insufficient attention image tokens for example). Overall, given the strengths of the paper, the novel MI-based contribution to understanding VLM spatial reasoning, and the positive final consensus from all the reviewers, I recommend the paper be accepted. That said, significant revisions are recommended in the final version to incorporate the clarifications, and ensure the MI focus and evaluation context are made explicitly clear to the reader, to mitigate any confusion as highlighted by the review process.
Decision: Accept (poster)