Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models
Authors: Raphi Kang
Hongqiao Chen
Georgia Gkioxari
Pietro Perona
Venue: ICLR 2026 Poster (2026)
Keywords: interpretability
vision
VLMs
visual reasoning
spatial understanding
temporal understanding
video
Primary area: interpretability and explainable AI
TL;DR: We identify linear spatiotemporal ID vectors as the mechanism behind visual reasoning circuits in VLMs, and extend this insight to improve VLMs downstream

Abstract: Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial structure must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally explain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding \textit{spatial IDs to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions we demonstrate that these IDs, which are ubiquitous across the model, can systematically mediate model beliefs at intermediate VLM layers. Additionally, we find that spatial IDs serve as a diagnostic tool for identifying limitations and bottlenecks in existing VLMs. We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotemporal ID mechanism, we elucidate a previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models.

Published as a conference paper at ICLR 2026
LINEAR MECHANISMS FOR SPATIOTEMPORAL REA- SONING IN VISION LANGUAGE MODELS
Raphi Kang", Hongqiao Chen', Georgia Gkioxari, Pietro Perona California Institute of Technology (rkang, harrychen, georgia, perona) calleccccllli aetch.eeee
ABSTRACT
Spatio-temporal reasoning is remarkable capability of Vision Language Models (VLMss,, but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial struc- ture must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally ex- plain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding sputial IDs to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions uitous across the model, can systematically mediate model beliefs at interme- diagnos-
demonstrate that these IDs, which ubiq-
diate VLM layers. Additionally, find that spatial IDs serve tic tool for identifying limitations existing VLMs, and as a valuable learn- ing signal. We extend analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotempo- ral ID mechanism, we elucidate previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models. We release our code for reproducibility: https: / thub. com RaphRoaphoooooooaao /1 roaar-mest maaer--mch-lsss
During Training
Language and Projections concatonatee
VLM
Right.
LO
Lend
Image Encoder
logits
"doog
"ving Lie
('dag" r.patial All (hiight))> "dog" spaniel kd-tiieft
"Q: Is the dog right of the cal?
Text
Embedding
Integration:
Linguistic Reasoning:
Reasoning happans among taxt tokens supplemented by spatiotemporal 11s
spatfotemporal IDg
Figure I: Hypothesis for spatiotemporal visual reasoning. The VLM linearly binds spatiotempo- ral localization to object word activations in early layers. Subsequent linguistic reasoning about the object is informed by its location in space and time per the spatiotemporal ID.
INTRODUCTION
Reasoning about visual input with textual queries is key challenge behind vision-language models (VLMs). Consider a typical visual question answering (VQA) prompt: "Is the dog to the left or right of the cat?". To succeed this, one field, assess spatial relationships, and make a categorical decision. Though complex capabilities spatial or temporal reasoning are still far from being fully understood reliably engineered (Stogiannidis et al.) 2025; Chen et al. [2025] Tong et al.) (024), SoTA VLMs have seen steady progress simple visual reasoning without explicit guidance. So how do they it?
resolve linguistic references, locate entities in the visual
denotes equal contribution

---

Published as a conference paper at ICLR 2026
Belief Change after Intervention with Spatial IDs vs. Noise
Model Latents altered with Spatial IDs
A
.butt Juui 1u.i
Relnww
Gemma3 48
5-338
LL.V4155778
LLaMM3.2-125 2-110
Internkt 3-148
3-8
Is above or below B?
Above
InternV1.3110
Internkl.328
Model Latents attered with Noise
Quen2-28
Uwen2.b-7U
Original Model
Qwen2.5-38
20
80
Results from Targeted Intervention ($3). Median binary belief swap due spatial ID steering is 64.4%, and 29.5% for noise. Spatial IDs have 43.6% above-chance influence on average. We conclude that spatial IDs mediate models' beliefs about objects' locations in space.
Figure 2:
Attention- based analyses in VLMs have shown various structural properties emerge in VLM inter- nals during VQA (Jiang et al.) 2025b Neo et al. 22224, Zhang et al. (2024a). Relatedly, mechanistic interpretability LLMs has uncovered linear circuits for relational linguistic reasoning (Park et al.) Feng & Steinhardi, 2024, Hernandez et al. 2024). Might such linear processes also be driving visual reasoning in VLMs, and if so, how exactly? This leads us to ask: Q1. Can we linearly model emergent structured reasoning processes that drive spatial reasoning in VLM internals?
2024
The typical VLM architecture utilizes a vision encoder which projects the input image to em- beddings that are prepended to text token embeddings. This is then processed by downstream vision-aligned LLM. Popular model families using this paradigm are LLaVA et al.] 2023),
(LaMe(Dubey al. 2024), Qwen (Bai al. 2025). VL (Chen al.) 2024b), and Gemma
2024). A growing body of work aims to improve spatial reasoning capacities in VLMs Team et al.
2024a Fan al., 2025) and temporal reasoning in video models (Xiao al. 2024 Li et al. 2024b). Identification of the internal mechanism by which SoTA VLMs do spatial VQA can
(Chen al.
empower engineers to identify current architectural components leading to VQA failure modes in 3D reasoning or simple VQA. To this end, we ask: Q2. Given our linear model of spatial reasoning in model activations, how do we use it to understand and improve SOTA VLMs?
Similar training paradigms to image- based VLMs yield video models such as LLaVA- VAVVA-Vide//Zangg jet al. 2024b), VideoLLaMAS. (Zhng et al. 2025), and Qwen2.5 (Bai et al.) 2025), among others. Given our theory for the mechanisms underlying spatial reasoning in VLMs, we ask: Q3. Do video models utilize analogous linear mechanisms for temporal reasoning?
To address these questions, conduct mechanistic analysis of autoregressive VLMs and con- struct a linear model for spatiotemporal reasoning. We show that VLMs decompose a visual rea- soning task by first binding spatial information about visual objects object word activations, in the form of linear components we term spatial IDs, answering Q1 (Fig. D. We then extract these IDs and demonstrate their mediative capacity model output through targeted belief steering in text activations (Fig. 2 We further find that spatial IDs provide insight on VLMs' struggle with depth reasoning, and incorrect spatial IDs result of weak vision encoder or poor modality inte- gration leads to failures in LLaVA and LLaMA. This answers Q2. Finally. we show that temporal IDs similarly mediate video models, answering Q3. In summary, our novel contributions are:
We propose a linear model of spatial reasoning in VLMs, called spatial IDs. These are text-anchored latent structures that bind visual elements to object tokens thus enabling linguistic reasoning about space ($2.1) We emprically extract them from SOTA VLMs for characterization (...).
Spatial ID Model Formulation:
Analytical and Empirical Proof of Causality: We show model belief can be manipulated by perturbing only the spatial IDs, demonstrating their causal role in reasoning ($3). and
provide theoretical intuition for the emergence of spatial IDs in VLMs ( (.3).
SoTA VLM Analysis with Spatial IDs: Through targeted intervention, we identify limi- tations in depth expression ($4.1) and systematic failure modes in LLaMA/LLaVA ($4.2)
and find models can be effectively finetuned with spatial ID guidance ($33
Extension to Temporal IDs in Video Models: We perform our extraction and characteri-
zation analysis on SoTA video models and show that linear temporal IDs, like spatial IDs,
can drive temporal reasoning in VLMs (55)
2

---

Published as a conference paper at ICLR 2026
Input
Hidden States
"Is the.dog
IAM
1
to the left right of the
Left
the the right of the cor?
case
Righ
three swap cases:
Edbe
stseet
case
"Is codd
Tall lmnm
swap
Right
culf"
Figure 3: Mirror swapping experiment ($2.1) Activations from case and are partially swapped at a select layer. in one of three arrangements. Computations continue normally after this point.
2 EMERGENT STRUCTURE IN SPATIAL VISUAL. REASONING
In this section, we characterize the spatial reasoning circuits in SoTA VLMs and isolate any linearly separable components used to communicate spatial information. Towards this end, we track infor- mation flow in VLMs and identify important junctions for spatial information transfer across token sequences. Then we empirically extract linear spatial IDs, and analytically derive how they arise.
2.1 TRACKING INFORMATION FLOW DURING REASONING
To uncover whether VLMS engage in structured visual reasoning, i.e., isolating and propagating spatial information across layers, we intervene internal activations during inference.
Mirror Swapping Experiment. Our goal is to compare the model's output when presented with two distinct images and the same text query. If the model uses localized intermediate representations to reason about spatial relationships, then swapping activations between spatially distinct inputs at key layers and sequence indices should disrupt its final belief about spatial orientation, while swaps between spatially equal but attribute-wist different inputs shouldn't 't have strong effect.
Concretely, we run inference on plain and mirrored image-text pairs, extract their representations at an intermediate layer L, then replace subset Q of activations in the original zz with activations from the mirrored counterpart yz. The modified representation äL is passed through the remaining layers. We conduct interventions with three variants of Q: (1) all text tokens (2) all image patches (3) object-word tokens only. If information critical to spatial reasoning is concentrated in any of these, the model's 's belief will change when that region is overwritten. As control, we concurrently perform "attribute swapping", which follows the same steps but instead of mirroring the input image for the intervention case, changes its colors. The intervention procedure is visualized in Fig. 3 and formally defined in Alg. Further implementation specifics are deferred to Appendix $A.1
Algorithm 1 Swapping Intermediate Activations in Mirrored Images
L. UL +- fo fi(x), fL o...o f1()) L + [ (
>> x.y: [seq dim, embed dim] D Lt [seq embed_dim], Q: [num_of inddsss] Tout, L, Your fimms ( fiams o... fL+1
D P ("GT"):
Here, Q denotes the selected indices in the input sequence to swap, and is all other indices. We use the COCO-SPATIAL benchmark (Kamath et al. (023) for the mirrored images, which is curated subset of COCO (Lin et al. 2014) annotated with spatial language. To quantify belief shift caused by the intervention, compute the fraction of the mirror induued change that can be attributed to the swapped activations at layer L. For the ground truth logit "GT", this quantity is derived as:
belief shift,
Pron ('GTT) P ('GT) Paun (OTT) 5- Pann ("0T"
(1)
Results from Mirror Swapping are shown in Fig. 4A. Through mirror swapping, we observe a layer-specific effect for intervention effect across modalities. Intervening on visual patch tokens has a strong effect in early layers but fades with depth. Conversely, interventions on text tokens increasingly affect final outputs in later layers. This is corroborated by observations that middle layers have a modality switching effect in VLMs (Jiang et al.) (0255). Notably, swapping only the object-word tokens alters spatial belief specifically within narrow band of intermediate layers.
3

---

Published as a conference paper at ICLR 2026
(A) Swapped
Swapped All Imgs
Swapped Obj
Swapped Text
Swapped All Imgs
Swapped
ort orthggn
EmA47.5-20
Laval
16 15 25
30
15
10
30 35 30
14
15
20
Figure 4: Ratio change in log probability for logits "left" and "right" from mirror swap (A) and attribute swap (B) interventions. (A) shows distinct binary belief swaps, where text tokens have an influence after middle layers. Image patches stop having an influence after that point, and object word tokens only have an influence these middle layers. The control, (B), is noisy.
Attribute swapping results (Fig. 4B) indicate that mirror swapping is strong experimental setup for assessing spatial information flow in isolation from spurious visual factors. For the belief shift metric, a value of 0.0 on the y axis indicates model belief in the intervened case is equivalent to case (original query), while 1.0 indicates the belief is equivalent to case (mirrored/changed query). Mirror swapping results in distinct and strong binary belief swaps whereas attribute swapping yields mostly noise, to the point belief shift magnitudes are -20~20x that of the original belief difference.
These results suggest that VLMs extract and encode spatial facts from the image into object word tokens' activations, then operate over them in text-space. We term the latent structures holding visual spatial information spatial ids. Inspired by latest mechanistic interpretability findings (discussed in So) we hypothesize that the manner of spatial information storage is approximately linear.
2.2 EMPIRICAL DERIVATION OF SPATIAL IDS
If spatial IDs are indeed linearly bound object word activations, we should be able ex- tract them by averaging out object-related semantics from text activations. Below we out- line the process of their extraction. In $ we will test if these IDs causally mediate model beliefs, to validate whether the spatial reasoning mechanism in VLMs is indeed linear.
Extraction Preliminaries. We first set up some formalisms to derive spatial IDs. Let O {or, (01,02......p oN) denote a set of object categories. For each object E O, we have set of images { where the object positioned at spatial co- ordinates (i.j) in x grid. Then let (()) be a natural language query containing the token corre- sponding to object o, such as "Is there an o?". We define (o; I(o) T(o) E IRd the embedding of the text token corresponding to object o, extracted from layer I. of the VIM when input= ((..)) T())). The mean embedding for object at layer L is then:
Obi
Loc
A orright
LLaVA1.5-7B L13
Llama3.2-118 L14
Figure 5: Spatial IDs in grid. Color and saturation of markers represent the location of the object when spatial ID was extracted. and y axes are coefficients of ID projections onto h2 and vy. L, R represent "left", "right" textual activations.
m-1
Σ
Σ
Φn (6))) T(o))
L
==0
Yielding the opject-specific spatial ID location
(i,j) for object or
| (i,j) Φr (1.)) T(l) (/0
(3)
From this we can derive the universal spatial ID at location (i.j), averaged over N objects.
(n - - (o.)
1
(4)
To extract canonical horizontal and vertical directions from the universal spatial IDs (i, e Rd we compute average difference vectors across grid-aligned coordinate pairs. The vertical and horizontal direction vectors Uz, h2 E Rdd corresponding to increasing and j. are computed based on the spatial IDs. Eq. shows the derivation for vz, and hr derived in an analogous manner.
1
UL
(A(iih)) (
(5)
m
1=0 プイ
4

---

Published as a conference paper at ICLR 2026
Empirical Extraction. For our study, we extract spatial IDs from 11 SoTA VLMs, with synthetic images created from open-source OBJAVERSE (Deitke et al., 2023) objects. The object renders are paired onto grid of
on top of random natural backgrounds. We provide further extraction details in Appendix | along with ablations showing extracted spatial IDs are invariant to chosen images and counterfactual studies confirming that spatial IDs reside in object words, and spatial axes are orthogonal $C) Fig. [ shows two example spatial ID grids projected onto their respective spatial vectors. IDs from more models are shown SB) We see that these extracted IDs arrange in an approximate m x m grid at modality binding layers. Also projected are activations for spatial words, where we find that "left" closer to leftmost spatial IDs, and "right" vice versa.
2.3 THEORETICAL SKETCH OF SPATIAL IDs
We now offer quick, highly minimal analytical intuition for how the emergence of spatial IDs can be ubiquitous across many different models. Let p = (i.j) be some coordinate on a m m grid. Then for some query to VLM, let the input sequence contain projected visual tokens {mp} } for all p, and the query text tokens include an object token o. The residual update to o by one head is:
To f To + W out Σ Got Up:
Coepp X exp
la Kg
Up = Wv
(6)
vd
PEP
With cross-modal alignment, attention peaks at the true object patch p", giving or, Pe W Wyp- Decompose each patch as Tp = 3p + Pb(p) + Ep. where 8p encodes content, (p)) E I& is shared positional basis (e.g. RoPE learned 2D embeddings), P maps positional features into model space, and Ep is small. We can now substitute (r T(o)) detailed derivation is in $2.2 but in summary we get:
+ 8ropp" into Eq. 3 A
(p = (i. 2 Wou
(v(1.)
- m
((D) P
(7)
M (fixed per model)
Thus, spatial IDs are approximately a linear transformation of a universal positional basis written into the object token by attention. Spatial logits are thus approximately linear readouts:
(LLET) - ((RIGHT) & (WEErT - WRIGHT) ) (iijj)
(8)
so if (WiRET - WRIGHT) T M aligns with the z-coordinate in sb, the model prefers "left." Empirically, a low-rank linear fit from positional encodings & to spatial IDs Ar. explains most variance (e.g. rank-3 gives R² 2 0.85, see $E.2. Table 1). A more detailed derivation for (i, for the multihead case is shown in Appendix SE This is a particularly simplified setting, and real reasoning circuits in VLMs will involve a lot more noise and nonlincarities. The main takeaway is that VLM designs like Fig. encourage models to endow text tokens with visual information, followed by lingustic reasoning. This information transfer, in its most simplified linear form, is in the form of spatial IDs.
In practice, the finegrained circuit employed by VLMs may be much more varied, distributed, and nonlinear. The spatial ID framework could capture just one component of a more complex system. But per Ockham's 's Razor, spatial IDs powerful due to their simplicity. In following sections, we demonstrate the mediative influence of this simple spatial ID model on final VLM outputs, and further show how spatial IDs can be leveraged to improve existing models and build stronger ones.
3 SPATIAL IDs MEDIATE MODEL BELIEFS
If spatial IDs capture the causal mechanisms behind spatial reasoning, we should be able to linearly subtract or add arbitrary IDs to object word activations and change the model's belief about object location. In this section, we design and perform experiments real naturalistic images to test that empirically derived spatiotemporal IDs have causal effects on model outputs on spatial VQA.
Steering with Arbitrary IDs Experiment. For some layer L, we denote the model residuals corre- sponding to the entire input sequence after that layer as L. and perturb its token activation at some index g to observe any effects on the output belief. Alg. iillsstates the process.
Here we scale the norm of A (i,j to be a[æ][/]] and Ár(i,) Ar(m -im-i-- i 1,j). This approximately preserves the norm of IL- a = 5 is some scaling constant set after grid searching for stable intervention. We take 100 COCO images where one object is to the left or right of another, per labels from COCO-SPATIAL, and ask queries of the form "Is x to the left/right of ""? We measure the log probability of "left" and "right" tokens in the final output logits to assess steering effects.
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Intervention at Layer L via Residual Modification JL f- ft 0.... fi(z)
x: [ sem, embed_dimi D A (i. [embed_diml > P Pooee ("GT"): [1]
L 5 zzl: q) + [ + Az(i,) (t(1,) + x 1
Tout -- fmmax o (#1)
(A)
Belief Steering on Single Sample Across layens
(B)
Avg Belief Steering Across Coco-Spatial
Queryl
Original: Telt
Orig Right"
02-
Lopento Tight"
Log
03
Q: Is the polted plant the left or right the bottle? A:
-04
-22
08.00
01
G
Intorvention Layer
1.5--
Lana 15-7
(C)
Stowring Deliative Distance (neur/or)
(D)
Steering 3-way Relationship (A bel and C)
+6 13.00
6 KV
9.8 0l 3.20
Nastmooice CAR0
"QUSTTION the SuBt
11.0) 2.51 13.11 D.
10.11 2.D 0.1
Figure 6: Effect of spatial steering on real images on one sample across different intervention layers (A) and across a dataset for one layer (B). In (A), dotted and solid lines indicate answer probabilities for "left" or "right". Different colors indicate intervention (blue), steering the bottle to the left and plant to the right (pink), and the reverse steering (green). Blue lines are flat and show that the unintervened model incorrectly assigns higher log probability to "left". Pink lines show intervention on intermediate layers results in overwriting initial incorrect beliefs. (B) shows the shift in log probability for "'left" vs. "right" as a result of spatial steering on the subject word token. (C) and (D) show shifts in log probability for "near" vs. "far", and "yes" to an object being sandwiched between two others, vs. "no".
Results from Arbitrary Steering. Fig. @hswws the effects of model belief steering on real images and videos. Fig. [A shows that steering impact is greatest at modality alignment layers as expected per the mirror swapping analysis, and Fig 66 shows that intervening with the rightmost spatial ID largely enhances model belief that the object is to the right, and vice versa for the leftmost ID for leftward belief. The axes show changes in log probability for those binary directions for the whole dataset, and axes show the different ID locations. Regardless of whether the answer to the original query was "left" or "right", subplot trends are the same.
We repeat the analysis for queries about relative distance and three-way relationships where one object is sandwiched in beiween two others. Again, we find that when the object is to the left, altering the spatial ID of the subject towards the right increases the likelihood of "far" and decreases that of "near", and vice versa if the object is to the right. Similarly, we find that bringing a subject closer and closer be surrounded by two objects increases the model's belief that the subject is in between the objects.
Ad Adversarial Steering Experiment. If spatial IDs are indeed ubiquitous across models, interven- tions on internal activations should change the resultant model beliefs across many SoTA models. To confirm this, we evaluate the log probability of the correct answer ("GT"" and its opposite ("-GT") for all samples in COCO-SPATIAL on 11 SoTA models. Then, we repeat this measurement after in- tervention with spatial IDs most likely to reverse their original beliefs. More detailed experimental procedure is provided in 5A.5 In addition to targeted adversarial steering, we perform steering with noise vectors of the same norm as the spatial IDs, to evaluate chance belief swaps.
Adversarial Steering Results. We report % binary belief swaps on COCO-SPATIAL from the spatial ID vs. noise steering case in Fig. 2 Steering with spatial IDs yields a median 64.6% swap in beliefs, versus 29.5% with noise. This indicates activation intervention has nonzero chance influence on model output, but there is clear above-chance average of 43.6% increase with spatial IDs. Here, model's belief on one sample is considered "swapped" if the relative likelihood of the ground truth

---

Published as a conference paper at ICLR 2026
and its opposite answer has changed. For example, if P("left") P("right") before intervention, but after intervention we see P(-left") < P("right", the intervention has swapped the model belief. Thus conclude that spatial ID mechanisms mediate model belief in the models considered.
SPATIAL IDs FOR UNDERSTANDING AND IMPROVING IMAGE VLMS
With the existence and causal nature of spatial IDs established, we explore two ways leverage them towards stronger VLMs. First, we aim to understand why 3D reasoning fails in SoTA VLMs. Second, we use spatial IDs to diagnose architectural bottlenecks of SoTA VLMs in VQA.
4.1 DEPTH REPRESENTATION IMAGE VLMS
Spatial IDs suggest that VLMs represent visual space within a 2D grid. What might this mean for depth? We hypothesize that the language model must reason about depth related queries using the 2D localization in context. To verify whether this is the case, we look at the resulting belief changes in the depth axis when the LLaVA1.5 7B model is steered with spatial IDs varying in height. Fig- 7 shows the results. The same spatial IDs increasing the likelihood for "above" and decreasing "bllow,, also drive up "front" and drive down "behind" in LLaVA.
Further, projection of these word embeddings onto spatial vectors reveals that "above"/"behind"
and "below"/'froff map to
overlapping locations, indicat-
(A)
(B)
0.2
ing their functional relation-
68-
ships with spatial IDs are sim-
Changets
ilar. These results may be due
-02
-0.65
biases in training, or innate shortcomings in the VLM archi- tecture. They certainly high- light the need for better depth- handling mechanisms, whether that be through improved train- ing data or tooling.
API buthend
-0.10
00.20
interennion
Figure 7: Depth and height are strongly correlated in LLaVA. (A) Steering results for IDs varying in y-dim and their im- pact beliefs about height depth. (B) Projection of spatial words onto a spatial ID grid. Embeddings for "above"/ "front" and"behind"""belbl are nearly identical.
4.2 DIAGNOSING VLMS
When a VLM fails at a spatial task, how do we pinpoint the reason it failed? Referring back to Fig. D VLM failure points can roughly divided into modality encoding, crossmodal information integration, or linguistic reasoning stages. Knowing what part of VLM's architecture must be improved to reduce failures is paramount to efficient model engineering.
Per-sample analysis of spatial IDs provides unique ability to identify model's bottleneck. Con- sider an evaluation set K = (ki, kg...x)), }. where each i = (image, query). An imperfect VLM will fail at some samples. In this section, perform two experiments identify the architectural component which causes for the distribution of Kwrnng to be statistically distinct from Korrrea-
An example diagnosis process may look like this. If model exhibits incorrect spatial ID binding, and that incorrect output produced is faithful with the spatial ID, then the language-only reasoning stage is likely not fault. From there, a model exhibits sensitivity to masking the correct object region for Karong but not for Koorrect, the vision encoder is the likely bottleneck. If there is no distinct sensitivity difference, the errors are likely taking place after the vision encoder, but before the linguistic reasoning. If model accuracy seems independent of both spatial ID correctness and image recognition capacity, the language model layers beyond spatial ID binding are likely the biggest bottleneck. Note that it is possible for incorrect spatial IDs to be correlated to wrong answers, but still have some model inaccuracies be resultant from factors other than spatial IDs, such erroneous priors during LM readout (Leng et al. 2024 Ramakrishnan et al. 2018). In this case, it is still valuable to find if models can benefit from stronger spatial representations through this diagnosis process, and minimize avenues for failure. For the deseribed analyses, need sufficient K Kurong subset. As their Kwrong are biggest on COCO-SPATIAL, we select LLaVA1.5 7B and LLaMA3.2VL 1IB as model organisms for this section.
Ground Truth Spatial ID Deviation Experiment. First, we want to identify if models predict incorrect spatial IDs for the samples they get wrong. If the answer is yes, then it is likely that the
7

---

Published as a conference paper at ICLR 2026
(A)
LLM94S.5 70
(8)
LLAMRRI 2NL 131 WAARDDOOOOE
C12
CR
Euclidean drffrence from Ground Truth sppnal margin
Dffin Model Model ACC between masking bbok random loc
Figure 8: Contrasting density histograms show incorrect spatial IDs drive bad predictions. (A) shows deviation of model spatial IDs from g.t., and (B) the difference in model accuracy when masking objects vs. random locations in images. Histograms are samples VLMs got right (blue) or wrong (red). LLaVA shows faulty object detection with wrong answers, while LLaMA does not.
downstream language model is not the performance bottleneck, since it is faithful to the spatial information received. To compute the deviation of the model's believed spatial ID to the ground truth (g.t.), we compute the g.t. spatial ID by projecting the word activation onto the spatial axes:
(0) (iz)) VV (o Tlo))
I(o)..
V =1u,,ha
For spatial query like "Is the o to the left right of a o?", we can thus compute A (f, j)gt and A (0) (i, The model's assigned spatial IDs to the objects are computed per Eq. 9 for A (0) (i, eu
and A (0) (i, Then the g.t. ID margin deviation for some object o is:
ID deviation margin <ext
Egt, where Egt
(0) gt ggt Fext
(()
eext
(0)
eext
(a)
(10)
Here, a negative margin indicates that the model's extracted spatial IDs oppose the ground truth.
ID Deviation Results. From Fig. S we see that deviation from ground truth in extracted spatial ID margin is highly correlated with model mistakes. In other words, for LLa and LLaMA, wrong spatial IDs in object word activations led to wrong model answers, so linguistic reasoning was not the reason these failures occurred. Each subplot shows two density histograms overlaid in the same grid, where the x axis is 6ex Egt- The red histogram represents the density of ID deviations for K Kwrong: and the blue histogram shows the same for Keorrett- The red distribution is visibly skewed to the negatives compared to the blue. Quantitatively, we perform the Mann- Whitney U test (McKnight& & Najab 2010) calculate the p-value for the hypothesis that the two distributions (red and blue) are non-identical. Now we ask, is this failure mode stemming from the vision encoder level, or does it occur during the spatial ID binding across modalities?
Image Masking Experiment. Altering the raw image input at the pixel level can help us understand whether is a faulty vision encoder or faulty crossmodal information integration that has led the failures. If the model's beliefs on Keorreat are more sensitive to masking the image raw input at the g.t. location of o, while beliefs on K Kerong change more with masking elsewhere, we can conclude that the vision encoder of this VLM is doing a poor job at object detection, leading to observed failures. If we do not observe this the case, the failure may arise from the crossmodal information integration stage. In other words, the language model is doing a poor job appending binding IDs, despite the vision encoder having the necessary object recognition capacity.
We design obfuscation paradigm inspired by methods like D-RISE (Petsiuk al. 2021). where we either blur the bounding box of o, or A other locations in the image that do not intersect with the bboxes for ō. We then measure model belief change when masking the object vs. elsewhere:
bbox sensitivity = -
(P("GT")-minn (P("GT r), r R))
(11)
Image Masking Results. Fig. BB shows overlaid histograms for bounding box masking sensi-
tivities of Koorrect and Kwrong. Here, a negative value indicates greater sensitivity to raw pixel masking of random scenes, suggesting poor object detection. For LLaVA, there is statistically sig- nificant p-value for the hypothesis that Kwrong is shifted more negative than Keorrect, indicating its vision encoder fails at object detection when it answers incorrectly. In contrast, Kurong: Keorrect in LLaMA are agnostic to image obfuscation. This suggests that its failure modes likely stem after the vision encoder. These insights could be attributed to how LLaVA uses out-of-the-boo VIT that
8

---

Published as a conference paper at ICLR 2026
was text-aligned massive scale. hence not being tuned for finegrained detection, while LLaMA has a trained in-house ViT whose image-text alignment may be less robust.
Diagnosis Conclusion. With spatial IDs, explore the causes for failure in few model VLMs. We find that for both LLaMA and LLaVA, the linguistic reasoning stage faithful to spatial IDs. LLaVA's vision encoder is likely creating wrong spatial IDs from poor object detection, while LLaMA's weak point appears to be information integration across the image patch activations to the text tokens. These conclusions are preliminary and do not suggest that all of a model's fail- ures stem from one architectural component, but can serve to guide finetuning stage choices when resources are scarce, or provide intuition for future model designs.
4.3 IMPROVING VLMS
Spatial IDs and Model Performance. To understand if spatial IDs could be valuable learning signal, we first evaluate whether stronger stccrability from spatial IDs is correlated to stronger mod- els. Fig. shows the results of this analysis, where indeed we see that models with higher zero-shot accuracy on COCO-spatial also exhibit greater belief changes with spatial ID interventions.
Accuracy COCO-spatial
We define "steerability" as the difference between the change of be- lief resultant from steering with opposing spatial IDs versus noise. The layers of intervention are chosen as the middle third of all lay- for that model. Each point shows the model's mean steerability (on x) against its accuracy on COCO-spatial with no spatial inter- vention. Dotted lines connect models within a family.
Spatial Loss Module. Fig. / shows spatial IDs signal stronger model performance. This suggests that the strength of spatial IDs could be valuable learning signal for VLMs to learn principled spatial reasoning. To validate this intuition, we finetune Qwen2- 2B on a synthetic dataset similar to the one used to extract spatial IDs, and evaluate on COCO-Spatial. We introduce an additional loss module at layer 11 that computes the cosine similarity between the predicted and ground-truth spatial ID at that layer. We provide detailed explanations for this process in $A.9 This spatial ID loss is added to the standard language modeling objective, providing extra superviston. We perform a control training without the spatial ID loss. As the training, and thus the spatial alignment, was performed simplistic synthetic data, both models saturate and start overfit after reaching peak validation accuracy around 90%. But we see that explicit spatial ID loss helps the model generalize faster, reaching 91% accuracy on COCO-spatial at 3.2k steps, net 6% accuracy gain over the control case under the same number of training steps.
Steerability
Figure Accuracy vs. Steerability. Models with higher accuracy can be better steered with spatial IDs.
Num Steps
0
800
1600
2400
3200
LM Loss (L)
3.30
0.05
<0.01
<0.01
<0.01
Control
COCO Val Accuracy (f)
0.77
0.83
0.84
0.85
0.85
LM Loss (L)
2.71
0.04
<0.01
<0.01
<0.01
With Spatial Loss
Spatial ID Loss (L)
0.75
0.58
0.41
0.36
0.33
COCO Val Accuracy (t)
0.77
0.83
0.84
0.88
0.91
5 TEMPORAL IDs IN VIDEO MODELS
Thus far, we have characterized spatial IDs as a causal model for spatial visual reasoning in VLMs. Could we find a similar linear paradigm for the temporal axis? In this section, we repeat the ex- periments in 3 for the temporal dimension in video models, with the goal of identifying linearly separable temporal markers on object words. For space, experimental procedures are described briefly here, and in greater detail in Appendix $A.
5.1 MIRRORING, EXTRACTING, AND STEERING ACROSS THE TEMPORAL AXTS
Temporal Mirror Swapping. We validate that there exist modality alignment layers with object- level visual information transfer in video models. For mirrored videos, we take the Scene QA subset

---

Published as a conference paper at ICLR 2026
of MVBENCH (Li et al. 2024a) and swap the order of frames from back to front. Following Alg. I we show results for swapping text tokens, image patches, and object words in Fig. [OA. While the error bound is noisier than spatial LLaVA, likely LLaVA- follows response formats less well, we see the expected bump around middle layers for crossmodal integration.
Temporal ID Extraction. Derivation of temporal IDs and the temporal vector ty follows Eq. with synthetic 8-frame videos of OBJAVERSE renders. Results are shown in Fig. [OB. We again see that the text activation for "before" projects closer to earlier frames, than the activation for "after".
Causality of Temporal IDs. Finally, to con- firm controllability with arbitrary temporal IDs, we perform the steering experiment per Alg. _ on MVBENCH videos. Results are shown in Fig. LOC. On these real, naturalistic videos, we see that later temporal IDs steer the model belief towards "after", and earlier IDs towards "before", as expected.
Text Tok Swisp
Img Patchas Swap
Cbj Toks Swap
(A)
11
30
18
28
(8)
Temporal IDs from LLaVA-Video, L12
5.2 EMERGENCE OF TEMPORAL IDs
framc0
frame?
Fig. 10 shows summary results on LLa but we include temporal IDs from VideoL.LaMA3 and Qwen2.5 in Appendix $B.2 LLaVA- and VideoLLaMA3 use textual description of the video length and number of frames to indicate timestamps preceding the visual input, while Qwen uses explicit MRoPE time IDs. This suggests that spatiotemporal IDs can emerge without explicit positional encoding, beyond the simple mechanism derived in Eq. 7
(C)
Layer 12
Orggial 'eebw
Layer 13 Originl Whe
grosctre)
Figure 10: Temporal ID Results. Mirror Swapping on videos (A), Temporal ID grid (B), and temporal ID steering on model be- liefs (C) with LLaVA- Video
6 RELATED WORK
Mechanistic interpretability is a growing field uncovering the inner workings of large models, pop- ularizing techniques such as circuit tracing (Elhage et al. [202;; (mmeisen et al. (0055), Sparse Au- toencoders (Cunningham et al. 2023), linear probing (Alain & Bengio 2016), and others. The Linear Representation Hypothesis posits that concepts are linearly encoded in LLM latents (Park) cet al. 2024). and activation patching supports that linear changes in activations drive model belief (Meng et al. 2022; Zhang& & Nanda 2023). Internal in-context reasoning mechanisms such as linear binding IDs (Feng & Steinhardt 2024 Feng et al. (0224) have been identified in LLMs,, along with other evidence for linear multi-hop reasoning YYngg et al. 2024), in-context task vectors (Hendel 2023) and linear relational embeddings (Hernandez et al. 2024) during reasoning.
ct al.
Linearity of embeddings have also been discovered in VLM latent spaces (Trager et al., 2023 Jiang
et al. 2025a) to some degree. Previous work showed that VLMs separate VQA into image-focused then text-focused stages (Tiang al. 20256), and others have extended L.M interpretability tech-
niques like logit lens (Neo et al. (2024) or attention tracking (Zhang et al. 2024a; Yu & Ananiadou
2025) to VLMs unearth internal circuits. In our work, we mechanistically capture spatiotemporal
information flow from image patches to text tokens in VLMs, via the spatial ID mechanism.
7 CONCLUSION, LIMITATIONS, & FUTURE WORK
We propose spatiotemporal IDs as linear model for visual reasoning about space and time in VLMs. With a series of causal analyses, we show these IDs can be obtained in many SoTA models, and that they closely mediate models' beliefs about visual objects' location in space and time. We further offer ways to extend this mechanistic insight to improving existing VLMs. For tractability, our work is currently limited to analyses in simple spatial queries or appearance-base temporal queries. Experimental design for more complex, open-ended queries will enhance our understanding of how VLMs utilize rudimentary concepts like spatial IDs in more nuanced settings. Further, we only extract and steer models of sizes up to 14B parameters. Investigation into whether the spatial ID circuit plays a similarly prominent role in larger models will reveal whether VLMs of varying capacities follow analogous methods for visual reasoning, while we show several potential ways to leverage spatial IDs for VLM diagnostics or finetuning, future work could include expanded use cases such explicit temporal guidance at large scale.
employ distinct measures. Lastly,
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank Aadarsh Sahoo, Jiahai Feng, Michael Hobley, and Brian Cheung for valuable discussions and proofreading. Raphi Kang is funded by the NSF Graduate Research Fellowship. Georgia Gkioxari is supported by the Hurt Scholar program, Meta and Google. Honggiao Chen was supported by the Caltech SURF program.
REPRODUCIBILITY
We provide finegrained details for all experiments in ŠA of the Appendix, and results on all the models considered in $B, We include experimental details, results from various ab- lation analyses and counterfactual trials in SC-D. Code for all experiments can be found at: https//gittubb.cm//appoo////enee mech-vims
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, arXiv preprint 1rXiv:1610.01644, 2016.
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDoougal, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https::: pulo 2025 awtriittiioonggeeeeeeee
//transformer-cian
wtt./ttthhhheee thods html
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VV Technical Report, Febru- ary 2025. URLhttp:: /ttpttttpp///i....t//p org/ abs 2502 13923 arXiv:2502.1399 [cs].
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 14455-14465, 2024a.
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He. Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vlms? an attention mechanism perspective on focus areas. arXiv preprint arXiv:2503.01773, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visuai-Linguistic Tasks, January 2024b. URL httt::///axxi// org/ abs/ 2212 14238 arXiv:23 14238 [cs].
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600) 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Farhadi. Objaverse: universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 11142-1113,, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
11

---

Published as a conference paper at ICLR 2026
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Caafield-Doddss Danny Hernandez, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. pttps//transformer-crcccuiii /ttp:///////aaowoot/idee...//
Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, and Xin Eric Wang. GRIT: Teaching MI.LMs to Think with Images, May 2025. URL http::ttpt///iii
org. abs / 2505. 15879 arXiv:2505. 15577999..199 [cs].
Jiahai Feng and Jacob Steinhardt. How do Language Models Bind Entities in Context?, May 2024. URL
org. httpg//aab/22333 1t1/222111.11111 17191 arXiv:2310.17199 [cs].
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring Latent World States in Language
Models with Propositional Probes, December 2024. URL http:://arziv..or//bb/////
19501 arXiv:2406. 19501 [cs].
Roee Hendel, Mor Geva, and Amir Globerson. In-Context Learning Creates Task Vectors, October arXiv:2310.15914 [cs].
2023. URRp::t:///axx wttp://o/aab....1//////
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of Relation Decoding in Transformer Language Models, February 2024. URL [es].
09122 arXiv:2308.091
Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations, February 2025a. URL http://
arxiv.org/abs//44000 02762 arXiv:2410.0276 [cs].
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallu- cinations via Attention Lens, April 2025b.
arXiv:2411.16724 [cs].
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? in- vestigating their struggle with spatial reasoning, arXiv preprint arXiv:2310.1978,, 2023.
Raphi Kang, Yue Song, Georgia Gkioxari, and Pietro Perona. Is clip ideal? no. can we fix it? yes! arXiv preprint arXiv:2503.0877 2025.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13872-13882, 2024.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22195-22206. Seattle, WA, USA, June 2024a. IEEE. ISBN 9798350353006. doi: 10. 10111999CVPP2223..224400000 weteeerrgg/dccmmen///ee
URL
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. arXiv preprint arXiv:2410.06166, 2024b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, December 2023. httptttp:://///// httt:://arxive www.wvi.ooraarreeeeee / /204. 08485 arXiv:2304.0845 [cs].
12

---

Published as a conference paper at ICLR 2026
Patrick E McKnghht and Julius Najab. Mann-whitme test. The Corsini encyclopedia of psychology, pp. 1-1, 2010.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 01117399-177777 2022.
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards In-
terpreting Visual Information Processing in Vision-Language Models, October 2024. URL
httt:://ariii org/ abs 2410 07149 arXiv:2410.07149 [cs].
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Represstttptttttttttttoooon Hypothssis and the Geom- etry of Large Language Models, July 2024. UKL_ttttp//aarz3.orr////////1111 arXiv:2311 Sttiv:2311.00600 1.03658 [es].
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad Morariu, Ashutosh Mchra, Vicente Ordonez, and Kate Saenko. Black-box explanation of object detectors via saliency maps. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1
2021.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. Advances in neural information pro- cessing systems, 31, 2018.
Bin Ren, Yahui Liu, Yue Song, Wei Bi. Rita Cucchiara, Nicu Sebe, and Wei Wang. Masked jigsaw puzzle: A versatile position embedding for vision transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 20382-20391, 2023.
Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision- laaiuag models. arXiv preprint arXiv:2503. 19707, 2025.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 9568-9578, 2024.
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste- fano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15395-15404, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems (NeurIPS), 2017. URL httt:pss//aaxiiv.cr///aaaa wwwpeiii.orgg/aaoo.o
007777660377800 03762.
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13204-13214, 2024.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do Large Lan- guage Models Latently Perform Multi-Hop Reasoning?, February 2024. URL http://arxxx.
2402. 16837 arXiv:2402.16837 [cs].
Zeping Yu and Sophia Ananiadou. Understanding Multimodal LLMs: the Mechanistic Interpretabil- ity of Llava in Visual Question Answering, January 2025. URL http:////wwt..oog//////o
2411. 10950
arXiv:2411. 10950 [cs].
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Under- standing, January 2025. URL | / www.wv..orggaaaa... [es].
wttp///////a/aao.a//..//oaeeee/a abs/ 13106. arXiv:2501.13106
13

---

Published as a conference paper at ICLR 2026
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309. 16042, 2023.
Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From Redundancy Relevance: Informa- tion Flow in LVLMs Across Reasoning Tasks, October 2024a. URL http:://rxiv org/
abs/2406. 06579 arXiv:2406.06579 [cs].
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video Instruction Tuning With Synthetic Data, October 2024b. URL http::/aaxiv..rrtt org/ab// 2410. 02713 arXiv:2410.02713 [es].
14


Reviews:
[{"review_id": "QpCFpgVicX", "reviewer": "Reviewer_rYNZ", "summary": "The paper studies how VLMs do spatial/temporal reasoning by identifying spatial/temporal IDs, which are approximately linear signals written into object-word activations at intermediate layers, and validating them with targeted interventions like mirror swapping and steering. It could also shed light on 3D directions and temporal reasoning in videos.", "questions": "Please refer to the weaknesses. I'm glad to raise my score if you cover them well.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The problem the paper focus on, spatial reasoning of VLM, is clearly defined and important. \n2. The experiments especially in section 2 are clear and well-designed. \n3. The discovered spatial IDs show that models confuse \"up/down\" with \"front/back\", which helps explain real errors in 3D cases and gives a useful diagnostic. The temporal ID result is similarly clear and interesting.\n4. The discovered spatial IDs exist among different VLMs though different backbones and training procedure.", "weaknesses": "1. The paper explains how models work in spatial reasoning but does not show that using these IDs can improve benchmark accuracy in a training-free manner or help train better models.\n2. The authors could test whether stronger or clearer IDs mean better spatial reasoning ability. For example, by checking the correlation between an \"ID strength score\" or something similar, and model accuracy on a spatial reasoning benchmark. This would make the finding more practical and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpSvmH6jwP", "reviewer": "Reviewer_zD2N", "summary": "This paper identifies that VLMs encode spatial relationships through linear features called \"spatial IDs\" bound to object token representations. Through extraction and causal intervention experiments, the authors demonstrate that these spatial IDs can be manipulated via representation steering to control model outputs on spatial reasoning tasks. The analysis extends to video models, revealing similar linear temporal ID mechanisms for temporal reasoning.", "questions": "What causes the depth-vertical conflation? Section 4.1 identifies depth-height conflation but does not determine whether this stems from training data biases (e.g., perspective projection correlating height with distance), architectural limitations of 2D vision encoders, or the lack of 3D positional encodings. Without understanding the cause, no solutions are proposed or tested. Given that depth reasoning is critical for real-world applications, this represents a significant missed opportunity to translate mechanistic insights into actionable improvements such as 3D spatial IDs, depth-aware training objectives, or multi-view architectures.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting mechanistic discovery: Identifying spatial IDs as linear features that mediate spatial reasoning in VLMs is novel and insightful, revealing how models bind location information to object tokens for subsequent linguistic processing.\n2. Robust experimental validation: The paper provides rigorous evidence through well-controlled experiments (mirror-swapping with controls, causal interventions across 11 models achieving 64.4% vs 29.5% belief swap rates) and validates the framework across multiple datasets and model types.", "weaknesses": "1. Limited spatial relation coverage: The analysis focuses primarily on simple binary spatial queries (\"left/right\", \"up/down\"), while more complex and diverse spatial relationships like \"near\", \"far\", \"between\", or \"surrounded by\" remain unexplored. The generalizability of the linear spatial ID framework to these richer spatial concepts is unclear, limiting the scope of the findings.\n2. Insufficient guidance for model improvement: While the paper offers valuable diagnostic insights, it lacks actionable strategies for improving VLM training or architecture. The identified issues (e.g., depth-height conflation in Section 4.1) are not accompanied by proposed solutions or experimental validation of potential fixes. Demonstrating how spatial ID insights can inform better training objectives, architectural modifications, or data curation would significantly enhance the practical impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QXfgQQs8h", "reviewer": "Reviewer_4Gni", "summary": "This paper investigates how and where VLMs combine spatial visual information with textual representations to facilitate capabilities such as spatiotemporal reasoning.  The authors derive and identify spatial IDs, which encode object locations in an $m\\times m$ grid.  Spatial IDs are then used for interventions such as steering, as well as a variety of insightful analyses such as the lack of depth representation in VLMs.  The paper also investigates Temporal IDs, rounding out their investigation of spatiotemporal reasoning in VLMS.", "questions": "For the mean embedding of object $o$, is this computed across a batch of examples of the particular object?  For example is it averaged across many different apple photos (positioned in different ($i, j$) per image)?\n\nIs the $(i, j)$ spatial coordinate for an object just one singular coordinate?  If the object takes up a lot of space in the image you can imagine it encompassing many coordinates in the $m \\times m$ grid.  In such a case, is only the centroid of the object used as the spatial position?  What about objects that span many discretized squares where the centroid may not actually be the position of the object (e.g. a bike; the centroid may be empty air)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the work include its discovery and derivation of spatial IDs, and its thorough utilization for a variety of insights.  The paper defended spatial IDs thoroughly, through adversarial steering as well as how deviations from the ground truth spatial ID results in worse predictions.", "weaknesses": "Whereas the Spatial ID was thoroughly explored, much less attention was put on Temporal IDs.  Furthermore, the \"before\" and \"after\" evaluation is a bit more simplistic than spatial reasoning.  One expects a smoother interpolation; but in Figure 9(C) for 'After', both changes in logprob decrease past frame 5.\n\nFurthermore, the queries are quite limited to simple position-based analysis.  General reasoning should utilize other attributes beyond position, such as properties of the model (e.g. an ice cream in sunlight should most likely melt - reasoning about this change over time requires less positional information).  For such settings, it is not clear how the current approach can provide insight into how the model is performing its reasoning.  It would be interesting to see if object-specific attributes can also be encoded within or alongside spatial IDs for more complete isolation of where the binding with text is performed; or if the authors can provide insight into where such other attributes may be found and where they are bound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Qz1sKJHh0", "reviewer": "Reviewer_dAtd", "summary": "The paper proposes a linear ID mechanism for spatiotemporal reasoning in VLMs. At intermediate layers, the model encodes an object’s spatial or temporal position as an approximately linear Spatial or Temporal ID within the activation of the corresponding object token, after which most reasoning proceeds in the language channel. The authors also provide an empirical procedure for extracting Spatial IDs. Targeted interventions on these IDs systematically alter the model’s judgments about spatial terms such as left and right, and enabling diagnosis of failure points across architectures, for example vision encoding versus cross modal integration.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The work unifies the write–then–reason pathway for spatial and temporal information under a linear ID framework, and introduces actionable intervention and diagnosis paradigms, including mirror swapping, arbitrary ID steering, and error attribution for failure cases.\n\n2. The paper reports consistent phenomena and flip rates across 11 VLMs, with noise controls indicating that the effect is not an idiosyncrasy of any single model.\n\n3. The same IDs both explain observed behavior and localize bottlenecks, separating issues in vision encoding, cross-modal binding, and language-stage reasoning, thereby offering concrete guidance for model engineering.\n\n4. The study reproduces the same linearly steerable structure in video models as Temporal IDs, aligning interventions with before and after relations and suggesting a more general mechanism.", "weaknesses": "1. The Mirror Swapping design localizes key bottlenecks and supports the claim that Spatial IDs are bound to object tokens by replacing only the object tokens. However, it omits a crucial control condition that would replace non-object tokens under an otherwise identical setup, which is necessary to more directly test whether the observed effects indeed hinge on object token binding rather than on broader sequence perturbations.\n\n2. The causal attribution in §4.2 across the stages of vision encoding, cross modal binding, and language readout is not fully convincing. The inference that Spatial ID errors imply the language stage is not the bottleneck lacks necessity level causal testing; in other words, a deviation in the Spatial ID does not rule out the possibility that the language decoder exists erroneous priors during readout, a phenomenon that has been reported as language side priors or language bias in VQA tasks[1][2]. In addition, using sensitivity to occluding the ground truth box to distinguish vision encoding from cross modal binding conflates misidentification or localization failure with miswriting into the object token, making it difficult to uniquely localize the failing component. Overall, the current Spatial ID instrumentation does not cleanly isolate the faulty segment of the architecture.\n\n3. The paper largely remains at the level of revealing and validating the Spatial ID mechanism, without translating this mechanism into concrete corrective or editing procedures for existing models and without empirical evaluation of such interventions in a remediation setting.\n\n[1] Ramakrishnan, Sainandan, Aishwarya Agrawal, and Stefan Lee. \"Overcoming language priors in visual question answering with adversarial regularization.\" Advances in neural information processing systems 31 (2018).\n[2] Leng, Sicong, et al. \"Mitigating object hallucinations in large vision-language models through visual contrastive decoding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)