Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
Authors: Guanlin Wu
Boyan Su
Yang Zhao
Pu Wang
Yichen Lin
Hao Frank Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal LLM
VLM
Spatial Intelligence
Autonomous Driving
Primary area: deep_learning

Abstract: How to integrate and verify spatial intelligence in foundation models remains an open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): a structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As a complementary channel to text, SIG provides a faithful, compositional representation of scene structure for foundation-model reasoning. Building on SIG, we derive SIG-informed evaluation metrics that quantify a model’s intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of-the-art multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as a data-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-driving scenarios.

Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
Guanlin Wu" Boyan Su*
Yang Zhao Pu Wang Yichen Lin
Hao Frank Yang
Johns Hopkins University
{gwu32, haofrankyang)@@11 jhu.ee edu
Project page: inwu123...111.... wttpuu.......tt...... so/siggenchhhh
Abstract
How to integrate and verify spatial intelligence in foundation models remains open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As complementary channel to text, SIG provides faithful, compositional representation of scene structure for foundation-mode reasoning, Building on SIG, we derive SIG-informed evaluation metrics that quantify a model's 's intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of -fte--of-hhha-- multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as data mata-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-drvvng scenarios.
Introduction
Currently, Visual Question Answering (VQA) is recognized as natural test of visual-spatial intel- ligence (VSI), as it requires interpreting images, understanding object relationships, and inferring context to produce correct answers using text [11.) In computer vision, most researchers agree that VSI involves accurately perceiving, manipulating, and reasoning about visual and spatial information, such as size, location, and correlations given an input image, using textual description [20]. In a state- of-the-an study, three VQA questions appear the first page that illustrate their VSI understanding: "What is the distance between the keyboard and the TV? How many cabinets are in the room? And how height the stool is ?" (13) And correct answers indicate effective VSI learning by the model.
However, are VQA tasks ideal for evaluating spatial intelligence? For human beings. is defined by Dr. Gardner and Dr. Lohman as "the ability to generate, retain, retrieve, and transform well- structured visual images" " [21]. In computer vision, compared to other well-explored vision problems, VSI's core challenge lies in extracting 3D geometric insights from 2D images videos and then presenting them textually 1 22 VQA inherently interweaves linguistic proficiency with spatial reasoning. However, even without visual input, humans can form mental map of their surroundings by relying on auditory cues, like the way sound reverberates in a space [23]. Some researchers argue that VQA's reliance on textual replics may not fully capture the underlying spatial intelligence (74
Equal contribution, 'Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

They advocate integrating additional physical priors or alternative more spatial -ike representations to reflect spatial reasoning, like how human beings think.
Figure I: Examples of Human VSI Painting, Abraham Bosse, French artist and theorist illustrates a systematic, grid-based method for achieving visual-spatial correlations in rendering three- sree-dimensional space on a two two-twwmeeee do-ddmiiiionl canvas (left) (28). and he employed this grid-based visual scheme in portrait painting (middle) (2 30) Procedures of drawing cast with graphical priors from scratch (right). 31
Historical art practices offer an inspiring perspective: grid-canvas is very helpful for humans to perceive the spatial priors. During the Renaissance, artists like Albrecht Dürer. Leonardo da Vinci and Vincent van Gogh famously used grid-based "drawing machines," " as often depicted in historical artworks 36) By imposing visual observations onto structured grids (see Fig. D artists could systematically decompose a 3D scene into spatial regions, then record or interpret its geometry and graphical correlations. As illustrated in Fig. [ middle, portrait painters often proceeded from these organized graphical representations to finely detailed textual descriptions, mirroring how human VSI naturally moves from structured visual framework to richer linguistic content. Consequently, once trained to perceive spatial relationships accurately for painters, the specific subject matter becomes less significant: any subject, regardless of its nature, is ultimately perceived unique combination of graphs (nodes, shapes, edges, and color notes) on these grids. Collectively, these visual graphs define the appearance of objects such cast, flower, or human head [37]
Can graphical priors on a grid- canvas be used as machine representations of VSI? Like painter's 's trained eye, machines with grid graph- based abstractions gain a structured view that captures spatial relationships and hierarchies. This simplifies scene decomposition and supports richer interpretation and language grounding. From our perspective, ideal VSI representation is a combination of a structured conceptual model of a scene that encodes geometric and topological relationships, typically through grid- or graph-formattigg priors. In this framework, visual elements (e.g. shapes, edges, etc.) are mapped onto discrete spatial partitions and connected to capture both local and global information. By systematically organizing visual data into these relational structures, such an ideal VSI representation supports robust reasoning, efficient spatial manipulation, and natural pathway from pure visual analysis higher-level semantic or linguistic descriptions.
Building upon these insights, we propose a novel VSI representation format, called spatial intelligence grid (SIG), and then conduct foundational testing on existing Multimodal Large Language Models (MLLMs) in one of the most critical VSI application tasks: autonomous driving (AD) B8 42 AD demands real-time perception, precise modeling of multiple dynamic objects, and high level of situational awareness in diverse environments. The ability to efficiently attend to elements locations, correlations and movements in the driving scene, such other vehicles, pedestrians and road signs is essential for safe and effective operation. Given the complexity. flexibility, high stakes, and the potential impacts, AD serves an ideal stress test for evaluating and refining SIG-based VSI frameworks. To facilitate this, we introduce a novel benchmark, SIGBench, and systematically investigate three fundamental tasks: 1) SIG-based VSI Evaluation of MLLMs and Metrics: Evaluate whether existing MLLMs can answer spatial queries based on SIG representations, and propose novel evaluation metrics for SIG based on graph similarity theory; 2) SIG #GG-Empoweeed In- Context Learning (ICL) for VQA: Investigate how SIG-informed VSI features enhance spatial intelligence in VQA tasks through few-shot ICL, and 3) Human v.s. Machine VSI Attention based on SIG: Examine how human VSI attention differs from nachine-based approaches within SIG, and explore methods to achieve more human-like SIG order to improve human-machine interactions. this research, our contribution can be summarized as follows:
2

---

What represents Spatial Intelligence the best?
Physical Constralints
Cognitive Priors
Reasoring
Adaptationn
sign
Right
Buskiehi
Trant (2
Back
Abstraction
Froed SoP. 1
Frrmet S..0
Spottes Irtnkgencr
Hurton Sice Accendinn Map
Mamul Amnetation
truck
Homogggphh Tranv/omation
Visud Crounding
Care Attention
maman Spotial trod
Human like special Rilstion Grash
Trarfi Saan
Trutfclath
vitich
Textualization
Kftea Sed Regia hoo
Spetiaa Relistio Paragriph
Directonnal Beitters:
Presknal Ralatiore
Havk
The uil Le lace from Nlac teach
The souk movil iiut the of the light
The gigt ti treet right al He--ige.
"he fight in cinue torti-g righ
theadt's
"Th infoo enle from giv.1
Ego Vebicle
Theageii a1lha Eust brt black succ
Human Driver
Figure 2: Overview of Human-like SIG in AD Scenario. In the left,
SIG to represent the spatial relation the image. We apply homographic transformation
of traffic sign, traffic lights, vehicles and self (ego-vehickl to convert human gaze attention from image to SIG size in the right. Combining them, we get human-like SIG, which can then he extracted to human-like SRG and SRP in middle part. The order denotes the rank of an object of the same category in the image from left to right (e.g. black truck is the left-most object among vehicles).
We introduce SIG, a novel grid-based representation for VSI in AD scenario that integrates physical priors and human-like attention cues, together with a suite of structured evaluation metrics for precise spatial reasoning assessment.
We demonstrate SIG's efficacy as data schema for enhancing VSI through few-shot ICL on MILMs. Compared ICL using VQA-style representation, SIG-based ICL yields larger, more stable and more comprehensive improvements of model's VSI.
We release SIGBench, a benchmark containing 1.4K frames annotated with ground -tuut SIG and human gaze attention map, enabling evaluation of both grid-based machine VSI and himan-like attention-driven VSI with grid under our proposed evaluation metrics.
2 Related Work
General VSI in MILMs. MILMS have demonstrated promising performance across a variety of visual tasks such as object detection, segmentation, and image captioning by virtue of their unified vision-language representations and strong reasoning capabilities [43 Building on these strengths, recent work has extended MLLMs to tackle visual-spatial reasoning queries such as which object is at the left-most position in this image, through architectural adaptations and fine-tuning strategies designed to emphasize spatial relations 19 50. To provide accurate and in-depth evaluation of model's general VSI, several comprehensive visual-task benchmarks incorporate dedicated visual-spatial reasoning sections I while others focus exclusively on assessing visual- spatial reasoning ability over images 2 16 or videos (3315] in VQA-style. Although these benchmarks yield clear right-wrong scores, they often conflate failures of visual grounding (e.g. object detection errors) with genuine visual-spatial reasoning mistakes, particularly in scenes containing many similar entities (e.g. multiple vehicles with different brands, colors and shapes).
VSI with Scenario- Physical Constraints. In world applications such as robotics (62464] AR/VR 655 and AD 60, the implementation of VSI must respect domain-speciffc physical, environmental and regulatory constraints. For AD in particular, models must not only interpret scene geometry but also adhere traffic rules and kinematic feasibility across perception 76). planning [68 77/80). and control modules 88). A suite of VQA-style benchmarks has emerged to evaluate visual-spatial reasoning under these constraints [58-61 86 87], leveraging large-scale open-source AD datasets [8] shown in Tab. [ However, by framing questions solely as text- image queries, these benchmarks may overlook the rich spatial structure exposed by bird's widd's-eye-view
(BEV) representation that are widely used in AD perception and planning pipclines.
3

---

Table 1: Comparison between existing VSI and AD benchmarks. Comp. is comprehensive. MC and AM means multiple-choice and attention map, respectively. Size means the total number of words in cach dataset. Ann. and Rep. means annotate and repurpose (the dataset is compiled from prior datasets), respectively. S.R.R R. means spatial relation representation. Text and Graph means whether the calculation of evaluation metrics based on text (e.g. MC, sentence) or graph comparison. H-L means human- like and Auto means there are predefined metrics and can evaluate the answer automatically.
Type
Benchmark
Benchmark Overview
Evaluation Metrics
Size
Source
#Tasks
Answer
S.R.R.
Text
Graph
H-L
A/G Eval.
SEED
I
1.682M
Ann.
12
MC
Text
X
Auto
MMBench 3
0.096M
Rep.
20
MC
Text
/
X
x
GPT
Comp.
MM-Vet 5
0.003M
Rep.
MC
Text
X
GPT
MMMU 6
0.324M
Ann.
30
MC/Open
Text
/
x
x
Auto
Only
VSI
GQA [57
VSR [
24.42M
0.094M
Rep.
Rep.
Open
T/F
Text
Text
x
X
Auto
Auto
SRBench 10
0.068M
Rep.
MC
Text
/
x
X
Auto
Comp.
NuScenes-QA 58
6.592M
Rep.
Open
Text
X
Auto
AD
NuScenes-MOA 59
43.43M
Rep.
Open
Text
/
X
Auto
NuPlanQA 60
16.89M
Rep.
Open
Text
x
Auto
JuScenes-Spatiall)
78.75M
Rep.
MC/Open
Auto
AD
SIGBench
71.35M
Ann.
5
MC/SIG/AM
SIG
Auto
Human-Like VSI. Human scene understanding is guided by both overt gaze patterns (where look) and covert cognitive maps (how we represent spatial relations). Gaze or saliency prediction models, which estimate pixel-level attention maps, have provided deep insights into visual prioritization in generic images [73] and driving scenarios 1981101. Complementing this, the notion of cognitive map captures the internal spatial layout that humans to reason about object relations [02
and has recently been integrated into MILM frameworks for video understanding [13] In our approach, leverage an human-like SIG emulate both gaze-driven saliency and structured spatial representations, thereby enabling a human- like evaluation of visual-spatial reasoning, particularly within complex AD environments.
3 Methods
3.1 Grid-based Visual-Spatial Intelligence
We introduce SIG, a grid-based representation format for VSI, as illustrating in Fig. 2 scenarios, roadways will be physical constraints and the primary entities we focus on are the ego-vehicle and other traffic nodes, scuh as vehicles, traffic signs, signal lights and traffic lanes. Based on SIG, we can extract a directed spatial relation graph (SRG) that describes the spatial relation (direction-dissance in grid) of each object and spatial relation paragraph (SRP) that describes spatial relation of each object within a text manner. To quantitatively assess model's VSI, we propose three novel evaluation metrics: multi-level spatial matching (MLSM). spatial relation graph similarity (SRGS) and semantic relational distance (SRD). MLSM compares object positions directly within the SIG representation, capturing absolute localization accuracy. SRGS measures both node-wisc and edge-wise correspondence between predicted and ground truth (GT) SRG, emphasizing relation classification and structure. SRD computes a semantic relational distance between predicted and GT prepositions in SRP, evaluating the fidelity of both directional and proximal relations. To isolate core spatial reasoning, our evaluation focuses exclusively on ego-vehicle (self), other vehicles, traffic lights, and traffic signs, omitting traffic lanes from the quantitative metrics because many scenes have unreliable lane cues such as faded or blurred markings and no markings at intersection or rural roads (we provide evaluation metrics including traffic lanes for additional subset with clear traffic lanes). Before evaluating predicted SIG using GT SIG, we will firstly align the position of self in predicted SIG to which in GT SIG. Next, we apply the same offset for all other objects in predicted SIG.
In AD
Multi-level Spatial Matching. To evaluate SiG-based spatial representations independently of visual- grounding noise, we introduce multi-threshol graph matching protocol inspired by tracking metrics such as MOTA [00]] and HOTA [104] MLSM first performs bipartite matching between predicted and GT objects (vehicles, traffic signs, traffic lights) using cost function c, for vehicles and cat for traffic signs and traffic lights, which can be calculated by
Cy = d We # Wo Wes
Cet d Wo
(1)

---

Mullii Mhali--Levll Spotial Matching
Exampie: evalusting matched truck GT Fred SiCs
Orjjctt Ripwrtte Alatching
Pred
Distance
And). return Eypnisr) sypeceent)
black truck
blue truck
der Level Jipt, prody: resuira return Senel_igh, pred) ordentgt) ander Conee)
Frod:
track
1ghh1
light
Livt,_ pood): resaire Jom baw, onder couar
Spscuea Staie meds
Whatiteen Requienano Proudboode
palms
alcha in
a ststance trveshodd
4% pood paires
-cascrittr algas:
Ligh. pndd:
bewil zigh, And 19.2
Soutis dlseee wete thrasteldo. and aux artiihet onant see
Egm13
PC -1
- 10
Spetisl bhians throteddle But led gurd soottes 17 cast core.
Mulr Sparial Macchrggoooohhoetoooooooooo
Pred SG
Absoracton
Spotial Relation Graph Similority
Example: evalatting GT and Prod ERCs
Absrertio
#AgeEPAiNE sape www.sstttooeesaiieoo (ainn at-andd sige pecre
GT medel polr pull
and case cal. cocr for aneeual gtoaed sades Lf not rodkcat aeliaann
Gi-MMeerrokkeeee par diiks aypplylle cabsiig
wil sat. cast for aocoall stouedd abard not edpe.a7 adv_Ane
Ireeite cort cpokid
add Cww.nciin..cooo
Gi Predeedge pairs lifts edge wartiau cast sppled
Laaio les
B.B
Ford 14
oad coletion cest for ieeerched goad coule resecaved nede_padrafpeed
149
Di eigig eiisin RiPied sppked
cnst c.rode. mel
black
Max
Spetial Rostion Graph Simikartty Picuslocodz
SIG (bosed crica from CT
Pred (cooed area fromm SKC)
Figure 3: Illustration Examples of MLSM and SRGS. At the start of both MLSM and SRGS, it will match the objects between the predicted and GT SIG using bipartite matching. For MLSM, we provide example for calculating TP, FP and FN for vehicles in the boxed area in upper part. For SRGS, highlight the node and edge needed for insertion and substitution and their total edit distance in lower part.
where Wes Wo. W/ means the weight for color, order and type matching, respectively. Here d is the euclidean distance between objects' position on SIG, and each weight wey Wa: Wt equals one when the attribute is unmatched and drops below one when it matches, thus granting larger spatial tolerances for objects that are matched in type, order, and/or color.
After matching, true positives (TP) are object pairs whose grid position lie within a distance threshold and object attributes are aligned. False negatives (FN) are GT objects with no predicted match; false positives (FP) are predicted objects with no GT match. We evaluate three hierarchical matching levels for vehicles: (1) same type, (2) same type + order, (3) same type + order + color, and one level for signs and lights (same order), respectively. Over a set of n thresholds a E [1,..., n), we compute precision P.. recall R,, F1-score F1,, and association accuracy ASsA, and normalize them to get overall P, R, Fl and AssA, as showing in Fig. 3 Detailed calculation are shown in Appendix A.
Let denote the total number of objects (n is used similarly in the following complexity analysis). For MLSM, constructing the cost matrix for vehicles, traffic signs, and traffic lights has complexity O(n²), bipartite matching requires O(n²) [105]. and thulti- matching takes O(n). Thus, the overall complexity of MLSM is TMLSM( (n) = O(n") + O(n²) + O(n) + ((n)).
Spatial Relation Graph Similarity. Different from MLSM, SRGS evaluates the correspondence of individual relations (edges) between a predicted SRG and its GT counterpart. We quantify this through the computation of the graph edit distance (GED) [106] which measures the number of operations needed to edit the predicted SRG to GT. Let's denote directed SRG as g = (V, V = (v:) ) denotes the set of all n nodes, where v, represents an instance (e.g. ego-vehicle, other vehicles, traffic signs, traffic lights). C V x V denotes set of edges, where directed edge eij- = (vi, E encodes spatial relation (direction distance in SIG) from i. to vyy
We further calculate the SRGS from two perspectives: node edit distance and edge edit distance. The computation is based on bipartite matching between nodes in the GT and predicted SRG. Let M be the set of all matched pairs (u; 02), where U; E V is GT node and egg E 1> is its corresponding predicted node, as determined by the bipartite matching algorithm. The node edit distance considers three types of costs: 1) Substitution cost d 6z): the cost of modifying a predicted node er to match a GT node v, with different position or attributes; 2) Deletion cost dáa(ōz): the cost of removing unmatched predicted node ōy; 3) Insertion cost UU,) the cost of adding an
5

---

unmatched GT node ey. The total node edit distance between the GT graph 9 (with node set V) and the predicted graph & (with node set D) can then be denoted as
DN (g.9)
Σ
(eu 08) +
Edns 0,ED
din (0z)
(2)
V.EV
Detailed computations of each cost function are provided in Appendix A.2. Similarly, we define the edge edit distance DE 9) with the edge substitution cost (es êv), edge deletion cost der (êp).
and edge insertion cost o (e,). Combining Dx and DE, we can calculate weighted total graph edit distance Diotal and weighted graph similarity score S by
Diotal
Deotal 7 (G, 3 DD(G,,) 9), S max (0, 1
[0, 1]
D Diaax
where y, 3 are weights and Dmax denotes worst-case distance (all nodes and edges unmatched).
For SRGS, constructing the fully connected graph requires ((n) for all nodes and (n²) for all edges. Bipartite matching again requires O(n"), and GED, involving node-to-node and edge-to- edge comparisons, takes ((n) + O(n²). Therefore, the overall complexity of SRGS is TSRGS 2(O(n) + O(n²)) + 0(n)) = O(n)).
Semantic Relational Distance. To quantify the fidelity of directional and proximal preposition predictions in SRP, we assign each preposi- tion position on a discrete scale and measure the minimal cyclic or linear separation from GT. Directional relations are arranged cycli- ordered linearly from closest to furthest: adjacent to, close to, at a distance from, far from, far away from. Here the semantic relational distance between "adjacent to" and "far from" " is 3. We compute MAE and MSE based semantic relational distance and accuracy. Detailed equations shown in Appendix A.3. For SRD, we compute pointwise distances. With 8 directional and 5 proximal relations, the time complexity is TRRD = O(8n) + O55n) + (n)).
Front
Front left
Front right
cally, illustrated as Fig. 4 Proximal relations
Left
3
Right
Back left
Back right
Back
Figure 4: Directional Relation Circle. The semantic relational distance between any two prepo- sitions is the smallest step count around the circle (e.g. between "at the back left of" and "at the front of" instead 5).
3.2 SIG-based In-context Learning for VSI
While existing methods focus improving MILM's VSI based on depth, detection bounding box (bbox) and segmentation mask and convert them into textual representation such QA for learn- ing (50-52), we investigate SIG's potential as direct representation for improving VSI using ICL. Let's denote dataset D {(rii (<ri,y:)) y:) containing N image-SIG pairs. The ICL process for outputting answer Mg of query Fgg can be formulated as
yq FM (z.;
(4)
where Fm the MILM and example prompt P = concat( (1,, V1....,,..................... TA Wk). our settings, we randomly select l image-SIG pairs where contains image with annotated bboxes of vehicles and task description prompt, and y, contains the GT SIG (content of JSON file, e.g. vehicles: [bblck truck 1: [5 3]}; traffic_sigee siiisssssss:.see {sign 1: (3]..].... and SRP derived from GT SIG, Our main insight is to let the MLLM learn the corresponding spatial relation between object position in image and SIG and we demonstrate SIG's strong generalization ability with experiments in Sec.
3.3 Human-Like VSI with Grid
In AD scenario, human decision maniin is guided by selective attention: drivers focus on a subset of objects rather than all of them in the scene. We incorporate this human-like bias by integrating gaze or saliency predictions [93] [8) into SIG. Let p, (w, Ug 1)" denote an image pixel and w; (X,, Y, 1) denote corresponding grid cell on SIG. We can calculate the the homographic matrix H = [hip]] E 3x33 by solving the equations of X; and Y, as
h114 + h12% + h13
h2g u4 + h22 + h22 h Usru; + hyzus + has
X;
Y
(5)
hag + ha22; hag"

---

Table 2: Quantitative comparison of different MILMSs SIGBench dataset for general VSI tasks. P. R. FI, AssA means precision, recall, Fl-score and Association Accuracy, respectively. S and WS means similarity
(% 8 = 1) and weighted similarity (r * B) in Eq. B Ace means accuracy.
Dark blue and light blue
indicates the best and the second best result among all models.
MLSMM
SRGS
SRD (Directional)
SRD (Proximal)
Type
Models
R1
Flf
AssA1
WS?
MAEI
MSE
Acct
MAEI
MSEL
Acc?
Human-Level
0.918
0.932
0,938
0.869
0.897
0.901
0.568
2.372
0.753
0,720
2.508
0.760
Claude-3.5
0.371
0.373
0.243
0.225
0.203
2.179
6.247
0.103
0.701
0.960
0.416
Propristary
Claude-3.7-Somee
0.511
0.427
0.450
0.306
0.299
0.299
2.285
6.712
0.092
0.691
0.928
0.420
Gemini-1 Sro
0.471
0.470
0.454
0.309
0.282
0.272
2.073
5.899
0.126
1.241
2.720
0.298
Geminii 2.0. Flash
0.472
0.472
0.456
0.311
0.283
0.270
2.344
6.923
0.083
1.092
1.835
0.246
Gernini-2 2.5-PPro
0.473
0.586
0.507
0.355
0.232
0.236
1,316
2.937
0.254
0.790
1.225
0.398
GPT-4o-min mini
0.167
0.156
0.154
0.087
0.054
0.019
2.478
7.451
0.061
0.922
1.442
0.312
GPT-4o
0.507
0.441
0.458
0.315
0.337
0.331
1.942
5.309
0.144
0,949
1.563
0.313
Open Open-sounss noen-source
InternVL33995
0.290
0.302
0.284
0.174
0.135
0.086
2.019
5.804
0.130
0.942
1.524
0.314
Intern 31.3-148
0.432
0.374
0.391
0.258
0.254
0.234
2.305
6.850
0.070
1.139
1.957
0.272
Intern V
0.386
0.383
0.369
0.239
0.220
0.189
2,006
5.752
0.135
0.940
1,470
0.310
Qwen 2..-2.
0.251
0.306
0.258
0. 156
0.
0.066
2.426
7.413
0.066
1.279
2.574
0.215
Owen VL- 3LB
0.427
0.350
0.375
0.245
0.248
0.223
2,080
6.077
0.113
1,650
3.732
0.128
How Videos
Frames with Soox on Some Whille
CT
Viaal Sewning
(8)
Data Preprocess
Tranfomm mesion
Aceeiio
Selected Frames
Imoge Mizendinnnnnnnnn
Atbentiu Map
Spetial Intelligence Grid Creation
Spotial Intelligence Paragraph Filling
Prompt:
Prompt:
(b)
10-10 grsd, output a 190% dertonery..
bounding trapte matis light, and vffk estimating
Ghem the arhinng tome Image ths
volicles, plesse
"Taa FALL plr ofd Sigh 1") [6 (8.80
"soll, 15.01
"trof aganee (Tight 13.60
Jean've
19.31
spatsal resttoouuiis theirput imsgee...
"alock sar [arsomal naooo toon? sign
Black oar (emosimal prepesition 2....
Black (eheecionnl preposiiorg 20......"
Rleser two penegraphe Blsttecting
prosinal ('ccaocen 1o
Tot left -21
distance (roe"_]
Figure 5: (a) is the annotation pipeline of SIGBench and (b) illustrates the SIGC and SRPF tasks SIGBench.
using singular value decomposition (SVD). We then project and normalize the raw attention map Atmago with image size into the SIG attention map ASIG using
(i,)( ALmage [[o,5]] [ir
ming, Almage (u, v) maX v Atmage (u, v) min,, Atmage (u, v) s.t. (majs Biy, 1] = H '1, (6)
Thus, we can combine ASIG and SIG to create SIG with attention weight (a.k.a human-like SIG).
Human-Like Spatial Relation Graph Similarity. To weight graph edit distance by human-like perceptual importance, we scale each node-edit cost by its corresponding attention weight from ASIG- Edge-edit costs similarly weighted by the average attention weight of the two incident nodes. This yields an attention aware SRGS that penalizes errors on highly attended objects more severely.
Human-T like Semantic Relational Distance. To apply human gaze attention as weight for the calculation of human-like SRD, we multiply the SRD between predicted and GT prepositions by the mean attention of the two referenced objects. For instance, if the GT relation is black car is fat the back left of) white car 2, the predicted relation is (at the front of/. and the attention weights for black car and white car 2 are 3 and 4. The semantic directional distance in this example is 3 (shown in 10.5.
Fig. 4 and the human-like SRD for this example can be computed as 3 x 3+d
2
Experiment
4.1 SIG Benchmark
Benchmark Overview. We introduce SIGBench, a benchmark for quantifying both grid-based and human-like VSI in MLLMs within AD scenario. SIGBench comprises 1,423 frames, each annotated
7

---

with (i) SIG and human-like SIG, (ii) SRP and human-like SRP and (iii) gaze attention map in image size. The annotation pipeline is shown in Fig. (a). Let denotes the camera focal length. We create attention map by firstly estimating a circular attention radius using
f. Sa
fov
f. tan 8h
fovn 2
r= = min(rw, (h) S.t. Tw
tan
Th
where Tw, I are image width and height in pixels, Sw; 8h are the corresponding sensor dimensions in millimeters, and fov.. and fov, are the human horizontal and vertical field of view. Then we accumulate attention map from 6 consecutive frames to create GT attention map. For annotation of SIG, we annotate vehicles inside the bounding box using "color+type+order" label (e.g. black car 1), where order runs leftmost to rightmost in the image. We also annotate traffic signs/lights that are clearly visible using "type+order" (e.g. sign I, light 1). Detailed annotation pipeline and data distribution of SIGBen ch and SIGBench-ting regarding to number of objects in each sample can be found in Appendix B. In general, SIGBench contains two main task clusters: grid-based VSI tasks: spatial intelligence grid creation (SIGC) and spatial relation paragraph filling (SRPF) and human- VSI tasks: like SIGC and SRPF, and gaze prediction.
Table 3: Quantitative comparison of 3-shot ICL for general VSI tasks SIGBench-tiny. Z-S means zero-shot, ICL-MC meaning ICL using multiple-choice VQA and ICL-SIG meaning ICL using SIG. light red indicates the results that is worse than zero-shol after applying ICL on GPT-4o and Gemini-2.5 Pro.
Models
Type
MLSM
SRGS
SRD (Direction))
SRD (Proximal)
MC (Ase)
PT
R1
F11
AssAt
St
WS1
MAE]
MSE]
Acct
MAE]
MSE.
Acct
Dir.
Prox.
0.522
0.327
1.792
4.827
0.858
0.346
0.056
0.292
GPI-4o
ICL-MC
0.545
0.431
0,468
0.320
0.324
0.323
1.600
4.106
0.218
0.920
1.621
0.365
0.144
0.337
ICL-SIG
0.592
0.138
0.179
0.328
0.337
0.357
1.593
4.091
0.220
0.775
1.271
0/136
0.172
0.309
Gemini-
7-8
0.464
0.570
0.496
0.345
0.234
0.210
1,151
2.426
0.295
0.721
1.100
0.439
0.247
0.413
P.5.Pro
ICL-MC
0.477
0.617
0.524
0.366
0.185
0.187
1.174
2.667
0.325
0.845
1.387
0.384
0.172
0.348
ICI-SIG
0:556
0.008
0.565
0.406
0.305
0.307
1.126
396
0.316
0.578
0.729
0.493
0.305
0.447
Human-Leval
Gemini-2.5-Pr0
Zero-Shot
ICL-MC
ICL-SIG
Zero-Shot
ICL-MC
ICL-SIG
GPT-40
InternVL.3
Heude-3.7-Sornet
Owen-Vi-22555 5-326
MLSM-F1
MLSM-ASEA MLSH-F1
MLSM-ASSA MLSM-F1
MLSMAAsSA
MLSM-R
SRGS-S
MLSS-RR
SRGS-S
MLSM-R
75,0
70
70
50.0
50
SAGS 6
MLSM P
SRGS-WS
10
MLSM-P
SRGS.WS
10
MLSM-P
SRD-Dir
MC-
SRD-Dir
MC-PPox
SAGS-WS
Prox
SRD-Dir
SRD-Prox
MC-Dir
SRD-Prox
MC.Dir
(a) Results of Grid-based VSI Tasks
(b) ICL using GPT-40
(c) ICL using Gemini-2.5-Pro
Figure 6: Visualization of SIG-empowered VSI results. SRD-Dir and SRD-Prox denotes the Acc in SRD (Directional) and SRF (Proximal). (a) demonstrate the performance of human and different models in grid-based tasks SIGBench. Even for leading MLLMs, there is substantial gap compared to human performance in VSL (b) and (c) denotes the ICL results of GPT-40 and Gemini-2.5-pre on SIGBench- -iny.. ICL-SIG outperforms the zero-shot baseline in all VSI metrics and delivers more general improvements than ICL-MC.
Grid-based VSI Tasks. For SIGC task, the MLLM is prompted to produce a 10x 10 SIG in which every vehicle with bbox, traffic sign, traffic light, and the self (ego-vehicle) is placed according to its true world-coordinate position into a JSON file. The output SIG will be compared with corresponding GT SIG using using MLSM, SRGS. One example is shown in Fig. 5 (b) left. For SRPF task, it supplies each model with fully formed sentences that omit only the prepositional phrase between two object mentions. The model's objective is to choose the correct preposition in context. We target two complementary relation types: directional and proximal relations. For each image, we present two short paragraphs: one composed of blank slots for directional prepositions and one for proximal prepositions. One example is shown in Fig. (b) right.
Human-Like VSI with Grid Tasks. For gaze prediction task, it evaluates an MLLM's ability to predict the human gaze attention map for frame based on attention map from frames i - 5 to i as human gaze always follow a spatial-temporal format. We want to measure how well the model anticipates where driver would pay attention to given attention maps from previous frames.
8

---

Table 4: Average per-frame runtime analysis of metrics AMD Ryzen 5900X CPU with varying object numbers.
MLSM, SRGS, and SRD can execuled within sub-millisecond latency, satisfying sal-time constraint.
Number of Objects
Metries
3
7
9
11
13
15
17
19
22
MISM [s]
0.0001
0.0002
0.0002
0.0002
0.0002
0.0002
0.0002
0.0003
0.0003
0.0004
SRGS [s]
0.0001
0.0002
0.0003
0.0004
0.0004
0.0005
0.0006
0.0006
0.0008
0.0009
SRD
<0.0001
<00,0001
<0.0001
<0,0001
<0000011
<0.0001
<0.0001
<0,0001
<00,0001
<0.0001
For human-like SIGC and SRPF tasks, they are similar as grid-based VSI SIGC and SRPF, but incorporate human gaze attention into the evaluation. Each object is assigned attention weight from SIG attention map AsiG for frame i, reflecting its relative importance to human observer. These seeention-weighteer tasks are used to reveal whether MLLM can prioritize spatial relations between objects according to their importance to current scene in a human-like manner.
4.2 Model Selection
Models. We evaluate several top-tier MLLMs on SIGBench mainly from five modal families: I)
open-source models such as Intern VI. 49
and Qwen-VI. VI.
[44] 2) Proprietary models including
OpenAl GPT 43 Google Gemini model family with their complete names we used during experiment are listed in detail in Tab. 2
45 and Anthropic Claude 48 The specific models in each
Evaluation Metries. For gaze prediction tasks, we follow the widely used metrics in gaze/saliency prediction task such as Person's Correlation Coefficient (PCC), KL-Divergence and Information Gain (IG) [93 98). For grid-based SIGC and SRPF, we use MLSM, SRGS and SRD mentioned in Sec. 3.1
For human-likk SIGC and SRPF, we use numan-like SRGS and SRD mentioned in Sec. 3.3
4.3 Results and Analysis for Grid-based VSI on SIGBench
Zero-shot Inference on SICBench, showing in Tab. 2 The performance of several MLLMs in grid-based SIGC and SRPF using zero-shot inference are shown in Tab. In general, Gemini-2.5-PP achieves the best performance in both MLSM and SRD (Directional), illustrating its strong ability in spatial understanding of object position and direction. GPT-4o achieves the best in SRGS and the second best in MLSM and SRD, revealing its strong capability in figuring spatial-relation between objects. Besides, Claude-3.7-Somme between objects in text and second best in SRGS. By analyzing failure cases, we observe that small or peripheral objects are often missed or mislocalized and substantial overlap (high IoU) between objects exacerbates this by causing identity conflation and incorrect grid-cell placement.
demonstrate strong capability in understanding proximal distance
SIG-based ICL with Random Sample Selection, showing in Tab. 3 To evaluate SIG's advantage over conventional VQA-style representation for VSI, we conduct ICL on GPT-4o and Gemini-2.5-Pry whose performance are outstanding among all models in Tab. 2 We randomly select 90 samples from SIGBench as SIGBench- tiny and generate 4-8 multiple-choice questions targeting directional and proximal relations, mimicking existing VQA benchmarks for VSI for each image. In addition, randomly selected 3 images (outside SIGBench-tiny and annotated full VQA pairs covering every object in their ground truth SIGs for training. conduct a 3-shot ICL using SIG (ICL-SIG) and VQA MC) annotations as the input representation, respectively and evaluate on SIGBench-ting using our proposed SIGC and SRPF metrics alongside the accuracy of annotated VQA tasks (MC).
The key findings are: 1) the model's VSI consistently improves with ICL-SIG, Across both models, ICL-SIG improves nearly every metric relative to zero-shot baselines; 2) Even using randomly sampled images as context examples without sophisticated sampling strategy, ICL-SIG generally improves VSI over ICL-MC, especially for MLSM, SRGS and SRD (Proximal): 3) Compared to ICL-SIG, the improvements brought by ICL-MC is unstable, which result in lower performance than zero-shot in metrics such as SRGS-S in both models. To demonstrate the potential variability of SIG- based ICL with difference choice of data samples, we conduct further ablation studies in Appendix B.4. These findings highlights SIG's superior fidelity in encoding VSI compared to traditional VQA and its potential as a new representation schema for improving VSI in MLLMs.
Empirical Runtime Analysis of Evaluation Metries Runtime efficiency is crucial for real-time applications like AD. We report the empirical per-frame runtimes of our proposed evaluation metrics on SIGBench under varying object numbers, summarized in Tab.

---

Table 5: Results of zero-shot and 3-shot ICL SIG-COCO and SIG-ARKitScenes for general VSI tasks
using GPT-40. Z-S means zero-shot, ICL-SIG meaning ICL using SIG.
Benehmark
Type
MLSM
SRGS
SRD (Directional)
SRD (Proximal)
Pt
R1
F1"
AssA
St
WSt
MAE]
MSEJ
Auct
MAEL
MSEJ
Acc?
SIG.COCO
7-8
0,758
0.826
0.771
0.652
0.574
0.607
1,303
3.403
0.407
0.893
1.160
0.230
ICL-SIG
0.860
0.840
0.849
0.749
0.711
0.746
0.713
1.827
0.640
0.357
0.360
0.643
SIG-ARKiScenes
2-8
0.714
0.756
0.727
0.581
0.619
0.597
1.267
3.467
0.433
0,900
1.167
0.233
ICL-SIG
0.809
0.850
0.823
0.719
0.729
0.737
0.467
0.533
0.567
0.300
0.500
0.800
Table Quantitative comparison of different MLLMs SIGBench dataset for human-like visual-spatial
intelligence tasks. H means human-like and KL--D means -l--Divereence.
Gaze Prediction
H-SRGS
H-SRD (Directional)
H-SRD (Proximal)
Type
Models
PCC
KL-DJ
IG1
S-
WSt
MAE]
MSEL
Acct
MAE.
MSEJ
Acc1
(laude-3.5-Haiku
0.798
0.520
0.364
0.106
0.072
1.265
3.291
0.111
0.374
0.488
0.388
Proprictury
Claude Sonnoet
0.772
0.344
0.623
0.157
0.127
1.346
3.607
0.096
0.350
0.428
0.398
Semini-1.5-Pro
0.914
0,125
0,946
0.161
0.123
1.209
3.081
0.119
0.713
1.430
0.259
Demini-2.0-Flan
0.861
0.231
0.803
0,160
0.121
1.439
3.956
0.084
0.631
0.969
0.232
Gemini-2.5.Pro
0.868
0,314
0.637
0.148
0.118
0.784
1.621
0.222
0.450
0.659
0.349
GPT-4o-mini
0.838
0,514
0.504
0.010
0.003
1.509
4.165
0.064
0.484
0.693
0.324
GPT-4o
0.673
0.506
0,406
0.197
0.163
1.146
2.912
0.145
0.557
0.870
0.287
Open-sounce source
InternV1.3-99
0.427
2,932
-26622
0.059
0.030
1.303
3,450
0.105
0.525
0.669
0.250
LL3-11
0.859
0.860
-0.073
0.131
0.099
1.338
3.585
0.093
0.780
1.448
0.182
AiternV12..5 26B
0.951
0.090
0,988
0.120
0.082
1.247
3.225
0.136
0.539
0.669
0.254
Qwen- V1-22.-7B
0.950
0.068
1,028
0.038
0.018
1.492
4.200
0.081
0.958
2.033
0.152
Qwen-VL-2,5-32B
0.664
1,373
-0.908
0.123
0.087
1.302
3,433
0.102
1,231
2.875
0.081
Cross-Domain Generalizability. Although our initial motivation arises from AD, SIG as a data representation is fundamentally domain-agnostio and can be applied wherever fixed ontology of object types exists. For demonstration, we construct two proof-of-concee
benchmarks based on subsets from MS COCO [07] and ARKitScenes [088, which we denote as SIG-COCO and SIG- ARKitScenes, respectively. We conduct both zero-shot inference and ICL experiments with GPT-4o on these benchmarks, with results reported in Tab. [] ICL-SIG consistently outperforms zero-shot
inference across both benchmarks, suggesting SIG generalizes effectively beyond AD domain.
4.4 Results and Analysis for Human-L.ike VSI with Grid on SIGBench
The results for gaze prediction, human- like SIGC and SRPF are shown in Tab. 6 Surprisingly, InternVL2.5522 and Dwen-VLL-255. achieves the best performance in gaze prediction task. After looking at the their output, we found that different from other models that apply operations such as gaussian blur, edge detection after taking the average of five previous attention map. these two models only take the average (details in Appendix C.3). Model rankings under human-SRGS and human-SRD closely mirror the grid-based VSI results (Tab. 2 indicating that incorporating attention weights doesn't 't alter relative performance between MLLMs. This shows that current MLLMs still struggle to prioritize scene objects with human-like selectivity in AD settings.
5 Conclusion and Discussion
We propose a novel representation for visual-spatial intelligence (VSI) called spatial intelligence grid (SIG) and introduce suite of graph- evaluation metrics that leverage its structured topology to enable more precise and general VSI assessment. Through experiments on different MLLMs, we demonstrate that SIG-based few-shot in-context learning consistently delivers larger, more stable and more comprehensive VSI enhancement than using traditional VQA-style prompts solely, underscoring SIG's superior capacity to encode complex spatial relations. Based on SIG, we create SIGBench, a benchmark with SIGeumm gaze attention pairs, which is designed to evaluate both grid-based machine VSI and human-like attention-driven spatial reasoning under our proposed metrics. Taken together, these contributions offer principled data schema and a practical yardstick for VSI. Despite these advances, our study has two limitations remaining for future works: (i) SIGBench currently targets single-frame settings and therefore does not assess tracking or dynamic object- ooject- relations that require temporal context; and (ii) while SIG proves effective for in-context learning, SiG-based fine-tuning and reinforcement learning with human feedback remain unexplored.
10

---

6 Acknowledgement
The author team would like to share the sincere thank to Wei Zhang from U.S. Department of Transportation (USDOT) for providing the valuable U.S. Federal Highway Administration driving dataset, based on which we construct our proposed benchmark. The author team also appreciate the valuable suggestions from Junyue Jiang, Linshen Liu and Yibo Zhao during the discussion.
References
[1] B. Li, Y. Ge, Y. Ge, G. Wang, R. Wang. R. Zhang, and Y. Shan, "Seedd-ench: Benchmarking multimodal large language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 13299 299-99911 13308.
[2] B. Li, Y. Ge, Y. Chen, Y. Ge, Zhang, and Y. Shan, "Seed-bench-22plus: Benchmarking multimodal large language models with text-rich visual comprehension, arXiv preprint arXiv:2404. 1679094444.6699,
16790, 2024.
[3] Liu, Duan, Y. Zhang,
S. Zhang, W. Zhao, Yuan, J. Wang. He. Z. Liu, K. Chen, and D. Lin, "Mmbench: Is your multi-modal model an all-around player?" 2024. [Online]. Available: https://nxiv.orr/as/222..... 06288
[4] Li Y. Ge, Y. Ge, G. Wang, R. Wang, R. Zhang, and Y. Shan, "Seed--ench-22: Benchmarking multimodal large language models," arXiv preprint arXiv:2311.170 2311.1709,, 2023.
[5] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, 7. Liu, X. Wang, and L. Wang, "Mm-vet: Evaluating large multimodal models for integrated capabilities," 2024. [Online]. Available: http://wwwww.gg/b//////..oo 024990
[6] X. Yue, Y. Ni. K. Zhang. T. Zheng, R. Liu, G. Zhang. S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, Yu, R. Yuan, R. Sun, Yin. B. Zheng, Z. Yang, Y. Liu, Huang, H. Sun, Y. Su, and W. Chen, "Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi," in Proceedings of CVPR, 2024.
[7) P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, Meng. S. Huang, Y. Qiao, and P. Luo, "Lvlm-ehub: models," IEEE Transactions on Pattern
A comprehensive evaluation benchmark for large Analysis and Machine Intelligence, vol. 47, no. pp. 1877-1893, 2025.
[8] V. Balachandran. J. Chen, N. Joshi, B. Nushi, H. Palangi, E. Salinas. V. Vineet, J. Woffinden-Luey and
S. Yousefi, "Bureka: Evaluating and understanding large foundation models." 2024. [Online]. Available:
https:/aaxxi..rr//b///////
[9] F. Liu, G. E. T. Emerson, and N. Collier, "Visual spatial reasoning." Transactions of the Association for Computational Linguistics, 2023.
[10] I. Stogiannidis. S. McDonagh, and S. Tsaftaris, "Mind the gap: Benchmarking spatial reasoning in vision-language models," 2025. [Online]. Available: https://ariivvor//bb//55550
[11] J. Wang. Y. Ming, Z. Shi, Vineet, X. Wang, L.,, and Joshi, "Is picture worth a thousand words? delving into spatial reasoning for vision language models," in The Thirty- Annual Conference on Neural Information Processing Systems, 2024.
[12] C.-H. Ych, C. Wang, S. Tong, T.-Y. Cheng, R. Wang. T. Chu, Y. Zhai, Y. Chen, S. Gao, and Y. Ma, "Seeing from another perspective: Evaluating multi-view understanding in milms," 2025. [Online]. Available: httpss//arii..rr//aa2225551.1113
[13] J. Yang, S. Yang. A. Gupta, R. Han, L. Fei-Fei, and Xie, "Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces," arXiv preprint arXiv:2412.14111 2024.
[14] B. Zhao, J. Fang, Z. Dai, Z. Wang, Zha, W. Zhang, C. Gao, Y. Wang, J. Cui, X. Chen, and Y. Li, "Urbanvideo-bench:: Benchmarking vision-language models on embodied intelligence with video data urban spaces," 2025. [Online]. Available:
[15] Li, Y. Zhang, T. Lin, X. Liu, W. Cai, Z. Liu, and B. Zhao, "Sti-bench: Are milms ready for precisc spatial-temporal world understanding?" 2025. [Online]. Available: https://ariiv.orgaab//555....33
[16] S. K. Ramakrishnan, E. Wijmans, P. Kraehenbuehl, and V. Koltun, "Does spatial cognition emerge in frontier models?" in International Conference on Learning Representations, 2025. [Online]. Available:

---

[17] L. Li. M. Bigverdi. J. Gu. Z. Ma. Y. Yang, Z. Li, Y. Choi, and R. Krishna, "Unfolding spatial cognition: Evaluating multimodal models visual simulations," 2025. [Online]. Available: https//axi..orr//bs225500.....
[18] Yang. R. Xu. Y. Xie, S. Yang. M. Li J. Lin, C. Zhu, X. Chen, H. Duan, X. Yue, D. Lin, T. Wang, and J. Pang. "Mmsi-bench: A benchmark for multi-image spatial intelligence," arXiv preprint arXiv:2505. 237744 2025.
[19] R. Xu, W. Wang. H. Tang, X. Chen, X. Wang, F.J. Chu, D. Lin, M. Feiszli, and K. J. Liang, "Multi- spatialmllm: Multi-frame spatial understanding with multi-modal large language models," arXiv preprint XXi::2505.17015 2025.
[20] J. Keller and X. Wang, "Learning spatial relationships computer vision," in Proceedings of IEEE 5th International Fuzzy Systems, vol. I. IEEE, 1996, pp. 124.
[21] I. Dennis and Tapsfield, Human abilities: their nature and measurement. Psychology Press, 2013.
[22] K. Janowicz, S. Gao, G. McKenzie, Y. Hu, and B. Bhaduri, "Geoai: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond," pp. 625-636, 2020.
[23] H. Gardner, Frames of Mind: The Theory of Multiple Intelligences. Available: httips//bbooks http://boook.ggggl..coo wwwgglecco/bboloee..o Pid= IIIII
Basic Books, 2011. 1. [Online].
[24] J. Kuang, Y. Shen, J. Xie. H. Luo, Z. Xu, R. Li, Y. Li, X. Cheng. X. Lin, and Y. Han, "Natural language understanding and inference with milm in visual question answering: A survey," ACM Computing Surveys, vol. 57, no. 8, pp. 1-36, 2025.
[25] B. S. Kim, J. Kim, D. Lee, and B. Jang, "Visual question answering: A survey of methods, datasets, evaluation, and challenges," ACM Computing Surveys, vol. 57, no. 10, pp. 1-35, 2025.
[26] D. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. F. Fei-Fei, J. Wu, J. Tenenbaum al., "Learning physical graph representations from visual scenes," Advances in Neural Information Processing Systems, vol. 33, pp. 6027-6039, 2020.
[27] J. Ji. R. Krishna, L. Fei-Fei, and J. C. Niebles. "Action genome: Actions compositions of spatio- temporal scene graphs," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 236-10 247.
[28] A. Bosse, Algemeene manier van de hr. Desargues, de practijck der perspectiven, gelijck tot die der meet-kunde, met kleyne voet-maat, mitsgaders der plaatsen, en proportien van de stercke en flaawwe rakingen, of kleuren. by Dancker Danckerts7.# # woonende in de Kalver-straat, in de Danckbaarheyt#, vol. 1.
[29] K. Scott, "Screen wise, screen play: Jacques de lajoue and the ruses of rococo," Art History, vol. 36, no. 3, pp. 568 607, 2013.
[30] D. Rousar. Alberti's 's veil. [Online], Available: http::/wwww..gghhhiic.cooooooots
[31] J. C. Chase, Drawing Made Easy. EJ Clode, 1919.
[32] E. Panofsky, The life and art of Albrecht Düirr.. Princeton University Press, 2023.
[33] J. C. Smith, The" Invention" of Dürer as Renaissance Artist. Brill, 2010.
[34] A. Düirr and Dodgson, Albrecht Dürer. Medici Society, 1926.
[35] M. Kemp, Art in History, 600 BC-2000 AD: Ideas in Profile. Profile Books, 2015.
[36] Y. Yasuoka, "A new interpretation of the grid system reform in the late period," The Journal of Egyptian Archaeology, vol. 107, no. 1-2, pp. 265-280, 2021.
[37] R. Carter, "The art of romare bearden resource for teachers," National Gallery of Art, Washington, 2003.
[38] J. Ren, Sundaresan, D. Sadigh, S. Choudhury, and J. Bohg, "Motion tracks: unified representation for human-robot transfer in few-shot imitation learning." " 2025. [Online]. Available:
https://axxiv.or//bb////5 169994
[39] Z. Chen and K. Fan, "An online trajectory guidance framework via imitation learning and interactive feedback in robot-assisted surgery," Neural Nenworks, vol. 185, 107197, 2025. [Online]. Available:
12

---

[40] M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi, "A survey of imitation learning: Algorithmss recent developments, and challenges." IEEE Transactions on Cybernetics, vol. 54, no. 12, pp. 7173-7186. 2024.
[41] Y. Duan, Q. Zhang, and R. Xu, "Prompting multi-mode tokens enhance end-to-end autonomous driving imitation learning with llms"" in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 6798-6805 6805.
[42] G.C. K. Couto and E. A. Antonelo, "Hierarchical generative adversarial imitation learning with mid-level input generation for autonomous driving on urban environments," " IEEE Transactions on Intelligent Vehicles, pp. 1-14, 2024.
[43] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, Ostrow, A. Welihinda, A. Hayes, A. Radford et al., "Gpt-40 system card," arXiv preprint arXiv:2410.21276 2024.
[44] J. Bai, S. Bai, Yang, Wang. S. Tan, P. Wang. J. Lin, C. Zhou, and Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities." arXiv preprint arXiv:2308. 12966, 2023.
[45] Team, Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, Soricut, Schalkwyk. A. M. Dai, A. Hauth, K. Mil- lican et al. "Gemini: a family of highly capable multimodal models," arXiv preprint arXiv:231.11800, 2023.
[46] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten et al., "The llama 3 herd of models," arXiv preprint arXiv:2407.21783, 2024.
[47] Wu, Chen. Pan. X. Liu, W. Liu, D. Dai, Gao. Y. Ma. C. Wu. B. Wang et al., "Deepseek- v12: Mixture-of-experts vision-language models for advanced multimodal understanding." 2024. [Online]. Available: hutpss//arii..urg////2221......0
[48] Anthropic. "The claude model family: Opus, sonnet, haiku," 2024, Card Claude_ 3.pdf
[49] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong. K. Hu, J. Luo, Z. Ma et al., "How far are we to gpt-4v? closing the gap to commercial multimodal models with open open-source suites," arXiv preprint arXiv:2404. 16821, 2024.
[50] W. Cai, Y. Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao, "Spatialbot: Precise spatial understanding with vision language models," " arXiv preprini arXiv:2406. 1344243666,, 2024.
[51] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F, Xia, "Spatialylm: Endowing vision- language models with spatial reasoning capabilities." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2024. pp. 14455-144655 14565.
[52] A.-C Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang. J. Kautz, X. Wang, and S. Liu, "Spatialrgpt: Grounded spatial reasoning in vision-languse models," in NeurIPS, 2024.
[53] L. Zhong, C. Gao, Z. Ding, Y. Liao, H. Ma, S. Zhang, X. Zhou, and S. Liu, "Topy-nav: Unlocking the top-view spatial reasoning potential of mllm for zero-shot object navigation," " 2025. [Online]. Available: https://anxiv.rrg/bb///1111 16445
[54] Y.HH. Liao, R. Mahmood, S. Fidler, and D. Acuna, "Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miani, Florida, USA: Association for Computational Linguistics. Nov. 2024, pp. 17 770228-7770 047. [Online]. Available: https:/aclaathoolgg. org/ 2024.cmmlp-mminn9944
[55] Y. Tang, A. Qu. Z. Wang, D. Zhuang. Z. Wu, W. Ma. S. Wang. Y. Zheng. Z. Zhao, and J. Zhao, "Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to spatial reasoning," " 2025. [Online]. Available: http:://arriv.or///b////111..1.0
[56] N. Martorell, "From text to space: Mapping abstract spatial models in llms during grid-world navigation task," 2025. [Online]. Available:
16690
[57] D. A. Hudson and C. D. Manning, "Gqa: A new dataset for real-world visual reasoning and compositional question answering." in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, pp. 6693-6702.
[58] T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y.GG. Jiang, "Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario," arXiv preprint 23X1::2005.44830 2023.
13

---

[59] Y. Inoue, Y. Yada. K. Tanahashi, and Y. Yamaguchi, "Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations," arXiv preprint urXiv:2312.06352, 12 2023.
[60] S.-Y. Park, C. Cui, Y. Ma, Moradipari, R. Gupta, K. Han, and Z. Wang, "Nuplanqa: A large-scale dataset and benchmark for multi-view driving scene understanding multi-modal large language models," 2025. [Online]. Available: https///arii..rr//aa//2.........
[61] K. Tian, J. Mao, Zhang, J. Jiang, Y. Zhou, and Z. Tu, "Nuscenes-spatialge A spatial undcrstanding and reasoning benchmark for vision-land language models in autonomous driving," 2025. [Online]. Available: https://ariivorr/hb//550...0
[62] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al., "Rt-1: Roboties transformer for real-world control at scale," in RSS, 2023.
[63] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, Nair, K. Pertsch, L. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky, "my: A rision-language-actinn flow model for general robot control," 2024. [Online]. Available: https//arxiw.orr//as////// 10.20//
[64] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding. D. Driess, A. Dubey, C. Finn et al., "Rt-2: Issionlanguage-action models transfer web knowledge to robotic control," in CoRL, 2023.
[65] K. Chandrasegaran, A. Gupta, I., M. Hadzic, T. Kota, J. He, C. Eyzaguirre, 7. Durante, M. J. Wu, and F--.. Li, "Hourvidea: I-hour video-language understanding." in NeurlPS, vol. 37, 2024.
[66] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, Liu et al., "Ego4d: Around the world in 3,000 hours of egocentrie vidco." in CVPR, 2022.
[67] K. Mangalam, R. Akshulakov, and Malik, "Egoschema: A diagnostic henchmark for very long-form video language understanding." " NeurlPS, 2023.
[68] X. Tian, Gu, B. Li, Liu, C. Hu, Y. Wang, K. Zhan, Jia, Lang, and H. Zhao, "Drivevlm: The convergence of autonomous driving and large vision-language models," arXiv preprint arXiv:2402.12289, 2 2024.
[69] Z. Xu, Y. Zhang, F. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao "Drivegpt:: Interpretable end-to-end autonomous driving via large language model," arXiv preprini arXiv:2310001412, 2023.
[70] W. Wang. J. Xie, C. Hu, H. Zou, J. Fan, W. Tong. Y. Wen, S. Wu, H. Deng, Z. Li, H. Tian, L. Lu, X. Zhu, X. Wang. Y. Qiao, and J. Dai. "Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving." arXiv preprint ar2iv:2312.099255 12 2023.
[71] W. Cheng, J. Yin, W. Li, R. Yang, and J. Shen, "Language-guided 3d object detection in point cloud for autonomous driving." arXiv:2305. 15765, 2023.
[72] D. Wu, W. Han, T. Wang, Y. Liu. X. Zhang, and J. Shen, "Language prompt for autonomous driving." " preprint arvv 2023.
[73] D. Wu, W. Han, Wang, X. Dong, X. Zhang, and J. Shen, "Referring multi-object tracking," Pruc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 14 633-14 642.
[74] S. Peng. K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser et al., "Openscene: 3d scene understanding with open vocahularies," in Proc. of the IREF/CVF Conf. on Computer Vision and Pottern Recognition, 2023, pp. 815--24.
[75] R. Chen, Y. Liu, L. Kong, X. Zhu, Y. Ma, Y. Y. Hou, Y. Qiao, and W. Wang, "Clip2scene: Towards label-efficient 3d scene understanding by clip," in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023, pp. 7020-7030.
[76] M. Najibi, Ji, Y. Zhou, C. R. Qi, Yan, Ettinger, and Anguelov, "Unsupervised 3d perception with 2d language distillation for autonomous driving," in Proc. of IEEE/CVF Int. Conf. on Computer Vision, 2023, pp. 8602-8612.
[77] N. Sriram, T. Maniar, J. Kalyanasundaram, V. Gandhi, B. Bhowmick, and K. M. Krishna, "Talk to the vehicle: Language conditioned autonomous navigation of self driving cars." in Int. Conf. on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 5284-5504 5290.
14

---

htwp://dddngn.l.m.reees//////ooo [78] Jain, V. Chhangani, A. Tiwari, K. M. Krishna, and V. Gandhi, "Ground then navigate: Language-guided navigation in dynamic scenes," in 2023 IEEE Int. Conf. on Robotics and Automation (ICRA). IEEE, 2023, pp. 4113-4120.
[79] M. Omama, P. Inani, P. Paul, S. C. Yellapragada, K. M. Jatavallabhula, S. Chinchali, and M. Krishna, "Alt-pilot: Autonomous navigation with language augmented topometric maps," arXiv:2310.02324, 2023.
[80] A. Keysan, A. Look, E. Kosman, G. Gürsun, J. Wagner, Y. Yu, and B. Rakitsch, "Can you text what is happening? integrating pre-trained language encoders into trajectory prediction models for autonomous driving," " arXiv preprint arXiv:2309. 052822, 2023.
[81] H. Sha, Y. Mu, Y. Jiang, L.. Chen, C. Xu, P. Luo, S. E. Li, M. Tomizuka, W. Zhan, and M. Ding, "Ianguagempe: Large language models as decision makers for autonomous driving." ar00::2300003026, 2023.
[82] C. Cui, Y. Ma, X. Cao, W. Yc, and Z. Wang. "Drive as you speak: Enabling human-like intcraction with large language models in autonomous vehicles," arXiv preprint arXiv:2309, 102222080.0028 2023.
[83] P. Wang, M. Zhu, H. La, H. Zhong, X. Chen, S. Shen, X. Wang, and Y. Wang. "Bevgpt: Generative pre- trained large model for autonomous driving prediction, decision-making, and planning." " arXiv preprint arXiv:2310. 10357, 2023.
[84] L. Chen, O. Sinavski, J. Hünermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, "Driving with Ums: Fusing object-level vector modality for explainable autonomous driving," arXiv preprint
2023.
[85] L. Wen, D. Fu, X. Li, x. Cai, T. Ma, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao, "Dilu: A knowledge- driven approach to autonomous driving with large language models," arXiv preprint arXiv:2309. 16292, 2023.
[86] M. Nie, R. Peng, C. Wang, x. Cai, J. Han, H. Xu, and L. Zhang, "Reason2drive: Towards interpretable and chain- based reasoning for autonomous driving," arXiv preprint arXiv:2312.0366) 12 2023.
[87] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li, "Drivelm: Driving with graph visual question answering," arXiv preprint arXiv: 2312. 14150, 2023.
[88] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong. Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscemes: A multimodal dataset for autonomous driving," in Proc. of IEEE/CVF conf. on comp. vision and pattern recognition, 2020, pp. 11 11622-111 631.
[89] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger. M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, 7. Chen, and D. Anguelov, "Scalability in perception for autonomous driving: Waymo open dataset," in 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2443-2451.
[90] J. Mao, M. Niu, C. Jiang, H. Liang, J. Chen, X. Liang, Y. Li, C. Ye, W. Zhang, Z. Li, J. Yu, H. Xu, and C. Xu, "One million scenes for autonomous driving: Once datasct," 2021.
[91] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez. and V. Koltun, "CARLA: An open urban driving simulator," in Proceedings of the 1st Annual Conference Robot Learning, ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldherg, Eds., vol. 78. PMIR, 13-15 Nov 2017, pp. 1-16. [Online]. Available:
[92] H. Cacsar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, "Nuplan: A closed loop ml- based planning benchmark for autonomous vehicles," 2022. [Online]. Available: https::/arsxiv.or//abbaa/ 1161 11810
[93] 7. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, "What do different evaluation metrics tell us about saliency models?" arXiv preprint arXiv: 1604.03605, 2016.
[94] T. Judd, F. Durand, and A. Torralba, "A benchmark computational models of saliency to predict human fixations," in MIT Technical Report, 2012.
[95] A. Borji, D. N. Sihite, and L. Itti, "Quantitative analysis of human model agreement in visual saliency modeling: A comparative study," Image Processing, IEEE Transactions on, vol. 22, no. 1, pp. 55 69, 2013.
15

---

[96] A. Borji and L. Itti, "Cat2000: A large scale fixation dataset for boosting saliency research," CVPR 2015 workshop "Future of Datasets" 2015, arXiv preprint arXiv:1505.0358.
[97] Y. Cheng, H. Wang. Y. Bao, and F. Lu, "Appearance-based gaze estimation with deep learning: A review and benchmark," IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 7509-7528, 2024.
[98] A. Palazzi, D. Abati, E. Solera, and R. Cucchiara, "Predicting the driver's focus of attention: the dr (eye) project," IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 7, pp. 1720- 1000-1733,220 2018.
[99] Fang. D. Yan, J. Qiao, Xue, and Yu, "Dada: Driver attention prediction in driving accident scenarios," " IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4959-4971, 2022.
[100] A. Biswas, P. Gupta, S. Khurana, D. Held, and H. Admoni, "Modeling drivers' situational awareness from eye gaze for driving assistance," in Proceedings of The 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research. P. Agrawal, O. Kroemer. and W. Burgard, Eds., vol. 270. PMLR, 06-09 Nov 2025, pp. 3551-3567. [Online]. Available: https://praceeddngs..o prs.p
[101] Y. Xia, D. Zhang. J. Kim, K. Nakayama, K. Zipser, and D. Whitney, "Predicting driver attention in critical situations." " in Computer Vision ACCV 2018, Jawahar, Li, G. Mori, and K. Schindler, Eds. Cham Springer International Publishing, 2019, pp. 658-674.
[102] Epstein, Z. Patai, J. B. Julian, and H. Spicrs, "The cognitive map humans: spatial navigation and beyond," Nature Neuroscience, vol. 20, no. 11, pp. 1504-1513, 2017. [Online]. Available: https://www.nature www.aaatrer.oooommooommmmmooooae 66656
[103] K. Bernardin and R. Stiefelhagen, "Evaluating multiple object tracking performance: the clear mot metrics," J. Image Video Process, vol. 2008, Jan. 2008. [Online]. Available:
https://doi.org//00. 11tp//1..11111151111 155/2008/24631
[104] J. Luiten, A. Osep. P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, "Hota: A higher order metric for evaluating multi-ohjeet tracking." Int. J. Comput. Vision, vol. 129, no. 2, 548-578, Feh. 2021. [Online]. Available: https::/doi..rg///..00000 1007/s1 12033-1163-22222 020-0033-335-22
[105] T. H. Cormen, "Introduction to algorithms//th cormen, leiserson, rl rivest, stein]," 2009.
[106] X. Gao, B. Xiao, D. Tao, and X. Li, "A survey of graph edit distance," Pattern Anal. Appl., vol. 13, no. 1,
1 113-129, Feb. 2010.
[107] T.-Y Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference computer vision. Springer, 2014, pp. 740-755.
[108] G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz al., "Arkitscenes: A diverse roal-world dataset for 3d indoor scene understanding using mobile rgh-d data," arXiv preprint
2021.
[109] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Chen, F. Yan, 7. Zeng, H. Zhang, J. Yang. H. Li, Jiang and I.. Zhang, "Grounded sam: Assembling open-world models for diverse visual tasks," 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec. and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. and Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include any theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: our project page, we provide the link of github repo of our code and link of hugging face for our proposed benchmark.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: our project page, we provide the link of github repo of our code and link of hugging face for our proposed benchmark.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
httegguuiddes/ooeSuubisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[No
Justification: Our experiment includes many proprietary models for several evaluation metrics, which would be costly to run several times for all tasks and models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aurrrcccccoooacc//hccaaa
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets from creators original owners of assets correctly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer Sec. 4.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There is no crowdsourcing in our paper nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no crowdsourcing in our paper nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are only used for writing, editing and formatting purpose and does not impact the core methodology this paper.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
Supplementary Material
Contents
Evaluation Metrics
A.I
Multi-level Spatial Matching
A.2
Spatial Relation Graph Similarity
A.3
Semantic Relational Distance
SIG
Benchmark
B.
Overview
Benchmark Annotation
B.3
Models for Experiment
Data Distribution and Ablation Study
SIGBench Task Examples
C.1
Spatial Intelligence Grid Creation
C22
Spatial Relation Paragraph Filling
C
Gaze Prediction
C.4
Human-like Spatial Intelligence Grid Creation
Human-like, Spatial Relation Paragraph Filling
13

---

A Evaluation Metrics
A.I Multi-level Spatial Matching
Let E [1, [.] denote thresholds based euclidean distance, we calculate
TP.
TP.
TPo
P. TPa + FPa Ra TP. FN,
AssA,
(8)
TPa + FP.. + FNo.
Then, we normalize each of them through thresholds
2PR
1 - Σ n
p= Pa. R
Σ
Ro:
F1
AssA
AssAa
(9)
P
R
A.2 Spatial Relation Graph Similarity
For the calculation of spatial relation graph similarity, we have two main parts: the node edit distance (G 9) and edge edit distance (g 9). The node edit distance Dx (G, 9) includes substitution cost dsub (v, 0p), deletion cost (1x) and insertion cost ( Let d det.,, denotes the euclidean distance between the position of node o and by on SIG and AN as the total edit distance for the unmatch of node attributes (e.g. color, type and order).
The detailed formulas for each cost can be shown as
d muryg + Ax II att)())) & attr(ú;)]. if (vs By) E M.
Saub (14, 0pt)
(10)
0,
otherwise.
if $ (u (0, E EM),
(14)
(11)
0,
otherwise.
Vins
if $ (vl M),
8ins (uz)
(12)
0,
otherwise.
(13)
where Vaet Tiis are the cost for node insertion and deletion. For the edge edit distance De(G, G), it includes edge substitution cost 8% êv), edge deletion cost o (é)) and edge insertion cost _ (ez). Let di de.ee denotes the length between edge e; and ey and Xg as the total edit distance for the unmatch of edge attributes (e.g. direction).
di + +x I[attree)))) & attr((:)], (e, êx) E Me,
Bub (ei, ex)
(14)
0,
otherwise.
Mdel
if $ (el Me).
(c:)
(15)
0,
otherwise.
E
if & (el é)
Me},
E (e²)
(16)
0.
otherwise.
E E where Naen are the cost for edge insertion and deletion and MM is the set of matched edges based M.
A.3 Semantic Relational Distance
Let de denote the signed distance (in steps) between the predicted and ground-truth preposi- tion for the i-th example. We can calculate MAE and MSE by
n
MAE
d.1
MSE
Σ
12..
(17)

---

B SIG Benchmark
B.1 Overview
We introduce SICBench, a benchmark for quantifying both grid-based and human-like VSI in MLLMs within AD scenario. SIGBench comprises 1,423 frames, each annotated with (i) SIG and human-like SIG, (ii) SRP and human-like SRP and (iii) a corresponding gaze attention map in image size. All image sequences and raw gaze data are drawn from the U.S. Federal Highway Administration driving dataset, which includes 49 driving sessions ( 65 minutes each at 25 FPS) recorded from 25 drivers, with gaze tracked by SmartEye Pro5. SIGBench contains two main parts: grid-based VSI, containing tasks such as spatial intelligence grid creation (SIGC) and spatial relation paragraph filling (SRPF) and human-like VSI, containing human-like SIGC and SRPF, and gaze prediction.
B.2 Benchmark Annotation
We begin by filtering out frames with missing or erratic gaze data, retaining only those that have valid gaze points in the current frame and the previous five frames. For each selected frame, we estimate a circular attention radius based on
f. tan
fov
Th f. tan
fov 2
r= min(rn Th) s.t. Tw
(18)
Sw
2
where f is the camera focal length, I Jn are image width and height in pixels, Sws Sp are the corresponding sensor dimensions in millimeters, and fov.. and fov, are the human horizontal and vertical field of view. We then build the per-frame attention map Almage by accumulating attention map from frame i to i. Next, detect vehicles in each selected frame using Grounded-SAM-2 [109] with the prompts "vehicles" and "trucks" with thresholds including: confidence 2 0.28, bounding box area 1900 and text 0.25. We then input the original image and vehicle bboxes into Gemini-22.0Flaast and let it output JSON type files containing SIG. The prompt spurious or missing labels, human annotators check and refine all annotated SIG. For vehicles, we only keep those with valid bounding boxes and categorize each as of four types-truck, bus, car, or van-and assign a "color+type+order" label (e.g. black car 1), where order runs leftmost rightmost the image. All traffic signs and lights that clearly visible (i.e. not too small or distant) are likewise annotated with "type+order" (e.g. sign light 1).
shown in Prompt. B.1 To correct
ICL-SIG
SIGBanch
200
SIGBench-tiny
Number Data Samp es
0.8
150-
Scores
0.6
100
50-
0.2-
0.0
0
22111
MLSE
MLOM
ALSMM
MLSM.
sans
5RG5-
PMo MCARE
uC MCADD
Number Objects
Acr
Acc
DirT
maxt
(b)
Figure 7: (a) Data distribution of SIGBench and SIGBench-ting with object numbers. (h) Error bar statistics of multi-run ICI. on SIGBench-tiny using Gemini-2.5-PPh with random data selection. The hars show the range of ja + f and whiskers indicate min and max each metrics. Across metrics, ICL.SIG generally yields higher means/medians than ICL-MC.
B.3 Models for Experiment
For experiment on SIGBench, the detailed models we used are listed here. For open- source models: In family (In.emnV InternVL--3-9 VLternVLL-3-14B) [49],
(wen-VVL-2.5-320 [44] LLaMA family (LLaMA- 3.2- 1-1BBBW Vision Instruct) 46 and DeepSeek- VI. family (DeepSeek-V) VI., DeepSeek- VI.-
Qwen-VI. family (Qwen-V1-2.5-77)
2- 2-Tiny, DeepSeck-VL-2-Smmall VL-2-Small) 47.. However, since LLaMA-3.2-- IB- Vision-Instruc
3

---

and DeepSeek-VV VL models cannot correctly output JSON type file for SIG creation after several prompt engineering, we cannot record their performance. For proprictary models, we evaluate the performance of OpenAl GPT family (GPT-4o-mini, GPT-4o) 43 Google
Gemini family (Gemini-1.5-PP) Gemini-2.0-Flann Gemini-2.5-PPr) (.ele--3.-Snneet [88].
45 and Anthropic
Claude family (Claude-3.7-HH) 3.aaakuu,
B.4 Data Distribution and Ablation Study
In this section, we provide the detailed data distribution of SIGBench and SIGBench-tiny regarding object number shown in Fig. 7 (a). To demonstrate the potential variability of SIG-based ICL with difference choice of data samples, conduct further experiments and provide the error bar statistics in Appendix B.4. These findings highlights SIG's superior fidelity in encoding VSI compared to traditional VQA and its potential representation schema for improving VSI in MILMs. To provide more insights about the impacts brought by different selection of data samples to the performance of SIG-based ICI., we conduct five independent runs with randomly selected data samples using Gemini-2.5-PP and provide the error bar statisties shown in Fig. (b). As shown in Fig. (b). SIG-based ICL shows consistent improvement than ICL using MC across almost all metrics. The results demonstrate that SIG encodes VSI with greater fidelity than traditional VQA, positioning as promising new representation schema to enhance VSI in MLLMs.

---

Prompt B.I: Initial SIG Annotation using Gemini
[Task Summary] The first image captures outdoor driving scene, and the second image is the same scene with bounding boxes on certain vehicles. Please identify vehicles, traffic lanes, traffic signs, and traffic lights within the image, and understand the spatial arrangement of these entities. Specifically, assuming the bird's eye view of the scene is represented by a 10x10 grid, please estimate the center position of these entities within this grid. The output is expected to he JSON file containing dictionary.
[Task Details] <Overall> 1. The first image captures an outdoor driving scene, and second image is the same scene but only with bounding boxes on certain vehicles. 2. Estimate the center positions of each instance within the first image, assuming the entire scene is represented by 10x10 bird's eye view grid. 3. The entities be estimated include: vehicles, traffic lanes, traffic signs, traffic lights, and the vehicle that capture the images (sclf). 4. Both the horizontal and vertical coordinates of the grid range from 0 to 9, so all estimated positions (e.g., [x_1, y_1l) must fall within this range. 5. Estimated location of cach instance should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects. 6. Please be aware of the front-backward or left-right relationship between instances, there will be partial occlusion. 7. Since it bird's eye view grid, for all instances, more away objects should placed the higher row number and the closer objects should be placed in the lower row number. <Vehicles> 1. Detect vehicles in the first image that are enclosed by bounding boxes in the second image only, and estimate the center positions of these enclosed vehicles within the grid. 2. The output is key-value pair, which is expected to be exactly like: "vehicles": { ("black 1": [x_ 1__1,y_l y_1], "gray truck 2": [x_2, y_21,, ...). where each vehicle instance is named by color, vehicle type, and order. 3. The color of vehicles are summarized into: gray, black, white, silver, blue, green, yellow, red, purple. Other colors need to be attributed to the one closest to it among these colors. 4. The type of vehicles here are summarized into four category: car(including suv), truck(including pickup truck). van, and bus. 5. The order of vehicles is decided exactly from left to right of each vehicle in the first image, the left most vehicle 1. the second left most vehicle is 2, etc. <Traffic lanes> 1. Detect all traffic lanes of the same direction of the vehicle that captures the image in the first image, and estimate the lane position within the grid. One lane can be represented multiple adjacent points
key-valuc pair, which is expected to be exactly like: "traffic lanes": ("lane 1": [[x_11, y 11], [x_ 12, 1_]] ...]. "lane 2": [[x 21, y_21], [x. 22, y y_222, .........) where each lane is named by order. 3. The order of lanes decided exactly from left to right of each lane in the first image, the left most lane is 1, the second left most lane is 2, etc. <Traffic signs> 1. Detect all traffic signs in the first image, and estimate the center positions of these traffic signs. 2. The output is key-value pair, which expected to be exactly like: "traflic signs": ("sign 1": L.y_IJ,,, "sign 2": [x 2. y 2]....,, where each sign is named by order. The order signs is decided exactly from left to right of each sign in the first image, the left most sign 1, the second left most sign is 2, ctc. 4. If multiple signs are mounted on the same horizontal pole, please treat them as separate instances. <Traffic lights> 1. Detect all traffic lights in the first image, and estimate the center positions of these traffic lights. 2. The output
is long. 2. The output
key-value pair, which is expected to be exactly like: "traffic lights": ('light 1": [x_1, y y_l "light 2": [x 2. y__], ...). where each light is named by order. 3. The order of lights is decided exactly from left to right of each light in the first image, the left most light is I, the second left most light is 2, ctc. 4. If multiple traffic lights are mounted on the same horizontal pole, please treat them Estimate the center location the vehicle that captures the image. The output is expected to exactly like: "self": [x, 0] 3. The vehicle capturing the image should be placed the point with raw index 0 and with column index depends on different images.
single light and use the midpoint between them the center. <Self> 1.
[Output] 1. Combine the key-value pairs of vehicles, traffic lanes, traffic signs and traffic lights into one dictionary. final output. The final output dictionary is expected to be exactly like:
"vehicles": {"black car 1": [x_1, y_1l, "gray truck 2": [x_2, y_2], ...),
"traffic_lames" {"lane 1": [[x_11, y-11l, [x_12, y_12], ...], "lane 2":
[[x_21, y_21], [x_22, y_221, ...], ...J, straffic_signst
{"sign 1": [x_1,
y_1], , "sign 2": [x_2, y_21, ...,, "traffic " ("light 1": [x_1, y_1),
"light 2": [x_2, y_2], ...), "self": [x, 0] } 3. Only output the final dictionary
ISON file.
5

---

SIGBench Task Examples
In this section, we provide the prompt we used for tasks including spatial intelligence grid creation (SIGC) (Sec. C.TD. Spatial Relation Paragraph Filling (SRPF) (Sec.
C.2
Gaze
prediction (Sec. C.3p human-like SIGC (Sec. [C.4) and human-like SRPF (Sec. C.5) Since
different models need slight modification to correctly output the format we want, here we only provide some of them for illustration.
C.1 Spatial Intelligence Grid Creation
Prompt C.I: SIGC Example General
Q: [SIGC task prompt] [Original Image] [Original Image with bbox of vehicles].
A:
("vehicles""""">" ("white car 1": [5,3], "grra truck 2": (4,2], "yellow
truck 3": 3":[7, 1]), ""raffic_laneee"
1": [";[[,,]] [3, "lane
2": [[4, [I, [4,7]] "lane 3"; [[5, [],
"trafficsfffffiiiieeeee {""sign
1": [3,3], "sign 2 1]3, "traffic_ ("light [" 1], "light
2": [5, 2]}, "self": [4,0))
(a) Original image
(b) Image with bbox of vehicles
Figure 8: Example of input images for SIGC task
For SIGC, the general structure of this task is shown Prompt. C.1 The detailed prompt we input is shown in Prompt. C.2 The original image and image with bbox of vehicles are shown in Fig. Sa and Fig. 8b respectively. The visualization of predicted SIG output by
GPT-4o for this image is shown in Fig. aal and GT SIG is shown in Fig. 9b
Spatial Inteneennce Grd
Spattel Inbbl ligence
bakk
8
4
DIOO
2-
(a) Visualization of predicted SIG
(b) Visualization of GT SIG
Figure 9: Visualization of predicted SIG and GT SIG

---

Prompt C.2: SIGC Prompt Example
[Task Summary] The first image captures outdoor driving scene, and the second image is the same scene with bounding boxes on certain vehicles. Please identify vehicles, traffic lanes, traffic signs, and traffic lights within the image, and understand the spatial arrangement of these entities. Specifically, assuming the bird's eye view of the scene is represented by a 10*10 grid, please estimate the center position of these entities within this grid. The output is expected to he JSON file containing dictionary.
[Task Details] <Overall> 1. The first image captures an outdoor driving scene, and second image is the same scene but only with bounding boxes on certain vehicles. 2. Estimate the center positions of each instance within the first image, assuming the entire scene is represented by 10 bird's eye view grid. 3. The entities to be estimated include: vehicles, traffic lanes, traffic signs, traffic lights, and the vehicle that capture the images (self). 4. Both the horizontal and vertical coordinates of the grid range from 0 to 9, so all estimated positions (c.g., [x_1, y_1l) must fall within this range. 5. Estimated location of cach instance should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects. 6. Please be aware of the front-backward or left-right relationship between instances, there will be partial occlusion. 7. Since it bird's eye view grid, for all instances, more far away objects should placed in the higher row number and the closer objects should be placed in the lower row number. <Vehicles> 1. Detect vehicles in the first image that are enclosed by bounding boxes in the second image only, and estimate the center positions of these enclosed vehicles within the grid. 2. The output is key-value pair, which is expected to be exactly like: "vehicles": ("black car 1": [x_1, 1_1 "gray truck 2": [x_2, y_21. "yellow bus 3": [x_3, y_3], "blue van 4": [x_4, y_4,, ...}: where each vehicle instance is named by color, vehicle type, and order. 3. The color of vehicles are summarized into: gray, black, white, silver, blue, green, yellow, red, purple. Other colors need to be attributed to the one closest to it among these colors. 4. The type of vehicles must be classified into exactly four categories: "car" (which include sedans, hatchbacks, and suvs), "truck" (including pickup trucks), "van". and "bus". Do not use "suv" or any other subcategory names directly as type. Instcad, map them into one of the four allowed categories. 5. The order of vehicles must be assigned globally, based strictly on their horizontal position in the first image. The left-most vehicle in the image must be numbered "1", the second left- vehicle must be numbered "2", and on. This numbering should apply across all vehicles, without restarting for different colors types. Vehicles usually travel within traffic lanes, so the estimated position of vehicle moving the same direction as the "self" vehicle should typically fall within the coordinates of a corresponding lane. This does not apply to vehicles traveling in the other directions or that are stationary. <Traffic lanes> 1. Detect all traffic lanes in the same driving direction the vehicle capturing the image, and estimate the lane position within the grid. One lane can be represented by multiple adjacent points as it long. 2. The output is a key-value pair, which is expected to be exactly like; "traffic_lanes". _lanes: ( ("lane 1": [[x_1] where each lane is named by order. 3. The order of traffic lanes is based strictly on their horizontal position in the first image. The left-most lane the image must be numbered "1". the second left-most lane must be numbered "2", and so on. 4. Each lane is typically straight, so the horizontal coordinates of its points should usually be the same, forming a vertical line in the grid, unless the lane is clearly turning, merging, or splitting. 5. Additionally, adjacent lanes are typically close together, so their horizontal coordinate values of adjacent lanes should usually differ by only 1. <Traffic signs> Detect all traffic signs in the first image, and estimate the center positions these traffic signs. 2. The output a key-value pair, which is expected to exactly like: "traffic_signs"" : ("sign 1": [__1,__11, "sign 2": [x_ vx_2,y_2 y_2,,__________ 2__])...). where each sign is named by order. 3. The order of traffic signs is based strictly on their horizontal position in the first image. The left-most sign in the image must be numbered "1", the second left- sign must be numbered "2", and so on. 4. If multiple signs mounted on the same horizontal pole, please treat them as separate instances. <Traffic lights> 1. Detect traffic lights in the first image, and estimate the center positions of these traffic lights. 2. The output is key-value pair, which is expected to be exactly like: traffic_lights
y_ 11. [x_ [x_12, y_12), "lane 2": [[x_21, y_ 21, [x_22, y_221,...I...
("light 1": [x_1, y_,, "light 2": [x_2, y_21, ...). where each light is named by order. 3. The order of traffic lights is based strictly on their horizontal position in the first image. The left-most light in the image must be numbered "1", the second left-most light must be numbered "2", and so on. single light and use the midpoint between them as the center. <Self> 1. Estimate the center location of the vehicle that captures the image. 2. The output is expected to be exactly like: '"self": [x. 0] 3. The vehicle capturing the image should be placed in the point with index 0 and with column index depends on different images.
multiple traffic lights are mounted on the same horizontal pole, please treat them as one
[Output] 1. Combine the key-value pairs of vehicles, traffic lanes, traffic signs and traffic lights into one dictionary, as final output. 2. The final output dictionary is expected to be exactly like: { "vehhccee":: ("black car 1": [x_1, y_1,, "gray truck 2": [x_2, y_2], "yellow hus 3": [x_3, y 2_3 "blue van 4": [x_4, y_41. ...,, traffic_lanes"
["lane 1" [[x_]1, 1__1__1]]]] 11], [x 12, y 12), ...l. "lane 2": [[x_21, y 2_1] [x 22, y_22], ........ "traffic_sign"" signs": ("sign 1": [x_1xx________ 1], "sign 2": y_2],_2 ...) "traffic "traffic_lights". ( ("light
1": [x 1. y.y_. "light 2": [x 2, 2)...... "self": [x. 0] 3. Please output the final dictionary in pure ISON format, without any additional text or explanation before or after. 4. All keys and string values in the output dictionary must be in lowercase letters only.

---

C.2 Spatial Relation Paragraph Filling
Prompt C.3: SRPF Example General
Q: [SRPF task prompt] [Original Image] [Original Image with bbox of vehicles]
[Directional Relation Paragraph Template] + [Proximal Relation Paragraph Template].
A: titpbweersddrrectoonalll
corresponding to the index of pttppp/a..0001112220012211111111 left of". "at the left of". "at the front left of". "at the front of", "at the front right of", "at the right of", "at the back right of""] and PROXIMAL LABELS BaaaSS==aaajaaaaad ["adjacent to". "close to". "at distance", "far from", "far away from"). The detailed prompt we input is shown in Prompt.
"answers_pin
For SRPF, the general structure of this task is shown as Prompt. C.S
The output answer is = ["at the back of", "at the back
The original image and image with bbox of vehicles are shown in Fig. 8a and Fig. 8b respectively.
C.4
Prompt C.4: SRPF Prompt Example
[Input Description] You will be provided with two images and two incomplete text paragraphs. The first image shows real-world driving scene. 2. The second image the same scene with bounding boxes drawn several vehicles. 3. The first paragraph is Dircctional Template that describes the directional relationships between certain vehicles, traffic signs, or traffic lights, using [directional preposition] as placcholders. Example: "Black car is [directional preposition] white truck 2. Black car is [directional preposition] sign 1." The second paragraph is a Proximal Template that describes spatial proximity relationships (e.g., near/far) between the same entities, using [proximal preposition] placeholders. Example: "Black 1 [proximal preposition] white truck 2. Black preposition] sign 1." 5. numbers of placeholders the above two paragraphs are the same, as they describe relationships for the same entities. 6. The number in each object name (e.g., black car 1, sign 2) indicates its horizontal left-to-right order in the image among entities of the same type. For vehicles, the order is based on all boxed vehicles in the second image. For signs and traffic lights, the order is based on their horizontal position in the image (from either image). [Task and Requirements] (NOT the label string) from the following label lists for each [directional preposition] and [proximal preposition] placeholder. DIRECTIONAL LABELS ["at the back of". "at the back left of", "at the left of", "at the front left of". "at the front of". "at the front right of", "at the right of". "at the back right of"] PROXIMAL MOOXIIALLLLBBLSSS LABELS ["adjacent to", "close to", a distance". "far from". ". "far away from") 2. Please output two integer lists: 'answers_ direcciooncc and "answers_ "answers_proximal where each entry is the index of the selected label from the provided label list. Output example: {( änswers directional: [4, 7.... 1J, änswers proximal' [2, .... 0] }1 The list answers. directional must contain only integer indices ranging from to (inclusive), and answers_ annserssproximiiie proximal
is [proximal
Based on the provided images and templates, predict label INDEX
a json format,
contain indices from 0 to (inclusive), as each index corresponds to valid entry from the DIRECTIONAL. L.ABLL S. and PROXIMAL, A LABELS S lists, respectively. 4. The length of the answer lists must exactly match the number of [directional preposition] or [proximal preposition] placeholders in its respective template. This number will be explicitly provided to you along with the templates for each data example. 5. Do not provide explanations. Return only the two answer lists. may no detectable entities, which case both templates will contain only a single period ".". When this occurs, you should simply return two empty lists. The following are the two incomplete text paragraphs of this inference: [Directional Template] (template diiictttoddrr)) [Proximal Template] [template proximal) The lists 'answers_ directional' and 'answers, munswers_prxximalt proximal you
some driving scenes, there
each have EXACTLY (num_pt placchhllecsss entries. 'answers directional' must contain ONLY INTEGERS FROM 0 to 7, and 'answers. __rrxxmmall'' must contain ONLY INTEGERS FROM 0 to 4. Do not use quotation marks around the numbers all entries must be returned as raw INTEGERS. NOT STRINGS.

---

C.3 Gaze Prediction
For gaze prediction, the general structure this task is shown as Prompt. C.6] The detailed
prompt input is shown in Prompt. C.5 The original image and image with bbox of
vehicles are shown in Fig. 8a and Fig. 8b respectively. The output gaze attention map and
gt attention map are shown in Fig. 10
(a) Predicted human gaze attention (b) Predicted human gaze attention (c) GT human gaze attention map map output by GPT-4o map output by Gemini-2.5-Prg
Figure 10: Example of output attention maps for gaze prediction task
Prompt C.5: Gaze Prediction Prompt Example
You are an agent designed to predict the attention distribution of a human driver in the current driving scene, based on real-time image and the gaze gray-scale from the previous 5 frames. Your prediction should reflect the driver's visual attention in the current frame.
[Task Summary] Please predict the human driver's 's attention on current driving scene frame, based on scene image and driver's 's attention on the previous five frames.
[Input] 1. The first 5 inputs are grayscale heatmaps of the driver's attention on the previous 5 frames. Each has size (1080, 1920) and is downsampled hy 16 average pooling, each 16*h 16 patch contains the same value. 2. The 6th input the full-rest full-resolution (1080, 1920) RGB image of the current driving scene. 3. Please consider the attention map from previous frames (the first five input images), the content and their potential moving patterns in the current frame (the sixth input image), to make the best prediction on the driver's ss attention on the current frame.
[Output] 1. The output array should have shape (1080, 1920), reflecting the driver's visual attention on the current driving scene frame. 2. Please note that all values in the output ar- ray will normalized into the range from to 1, via Min-Max Normalization. of outputting the array directly, please generate a Python function that creates it using the code template below. Please output exactly in this format, ONLY modify the section be- tween "# Modify here' and # End of modification'. 4. Code template: "'python import current_img" Initialize the
Instead
numpy as np import cv2 def create setaate_att___e aataattttttaaaaaaaaaaaaa attn_may np zeros (current_ingg attention map # Modify here attn, map = todo # End of modification return cv2. woo..oommaizeeeenn (attn_ [attn_map, None, alpha-0, beta=1, norm perm__ype=cv2...m NORR_MOMMAA...... dtype=cv2. CV _2F) Normalize the attention map In the code template, 'previous aaevious_attn__ _at_a_mttapp'' is np.array (with size(5, 1080, 1920)) five attention maps corresponding to the previous five frames, and 'current img' is mp.arra (with size (1080, 1920, 3)) of current driving scene image in BGR channel order.
shape [:2], dtype-object)
Min-Max Normalization
5.
[WARNING] Oulput the entire Python function INDISE A MARKDOWN CODE BLOCK, starting with "'python and ending with "* as illustrated in the code template. Otherwise, the system will fail to extract your code and your output will be considered invalid. 2. DO NOT treat lists as NumPy arrays, they DON'T have attributes like asttpe", 'sshape', 'dttpee,, or 'siize.. Convert them into np.arra if you need to apply these attribute. If violated, the output will be considered invalid. 3. When using OpenCV functions such cv2.add, Weiigheee,,, cv2.subtract, etc., MAKE SURE the input arryy have the same data type (e.g.. s.. both float32 both uint8). necessary, use .sttppe(np. (0ltppe(p..llaa32)) to convert them before the operation. Otherwise, it will cause cv2.errors.
[Important Requirements] 1. The output function can only rely the two arguments given in the code template. Please try use only methods from the NumPy and OpenCV libraries. 3. Please do not define any additional helper functions outside "create_atts aatt__mapp 4. Please ensure that the code is syntactically correct, bug-free, and ready to run without modifications; try to keep your function concise and refined. 5. Please output only the Python function, without any additional explanatory text. 6. Please do NOT use 'allency module, it may not be available. 7. Please do NOT assume any existence of external files, functions, methods.

---

Prompt c.6: Gaze Prediction Example General
Q: [Gaze Prediction Task Prompt] [Python Code Template] [Attention Map from Five Previous Frames] [Original Image].
import numpy as np import cv2 def aagatto_tttt_mmpppprriiooseee
current_img) attn_map np. zeros (current_ing shape dtype-np.float.32
Initialize the attention map Modify here, assign value for pattn_map' # Average the previous attention maps avg_previous_ axis=0) Focus on the road ahead and vehicles height, width current_img. sexreettiin.ehaape.... :] road_gray cv2.cvtcoloorroaaa croaetco cv2. Cww.CCLLLR__RRCC road_mask cv2. wtweshholddrrrdd_gaaaa 127, 1, cv2. THRESH_BINARY # Combine the averaged previous attention with the road mask attn_maph [height//2:, :] avg-previous _ttuuustttttttaeaeaa [height//2;, :] road_mast mask # End of modification return cv2. (atta map, None, alpha=0, beta=1, norm_ aarm__ype=cv2... NORM_MINMAK, dtype=cv2.CV_322 Normalize the attention map via Min-Max Normalization.
np. ppp.man www.npprevouus_atttttoeeee atteeiiiossatt_lstttt list,
road_region current_img [height//::
After close look at the results gaze prediction output by different models, we found that the widely used metrics such as person's 's correlation coefficient, KL-Divergence and information gain (IG) might not be able to fully reveal the performance of different MLLMs for gaze prediction tasks in settings (given attention map from five previous frames). The main insight for giving human gaze attention map from five previous frames is we want the model to predict the human gaze attention map for current frame. In addition, since human gaze follows high spatial-temporal property, the ground-truth (GT) human gaze attention map should be the accumulation of these human gaze attention maps from 5 previous frames and current frame. Based on the results shown in Tab. 4, Qwen-VL-225-77 and VL-2.-26 achieves the best and second best performance human gaze prediction task. However, the output code of Qwen- VL2.5-7B and Intern VL2.5-26B are always taking the average of human gaze attention map from five previous frames and do no additional operations on it, shown as Prompt. CC.7 We don't It think current metrics for gaze prediction can disclose the capability of MLLMs while based on other model's output such as using operations such as gaussian blue or edge detection is also far away from the true gaze prediction. Thus, we claim that the MILLMs we test in our experiment shows little capability in predicting human gaze attention.
Prompt C.7: Qwen-VL2.5-77 Output for Gaze Prediction Example
import numpy as np import cv2 def create_attan arrate__ttt__mapppreioooo _ist, current_imgl attn_map np. reroreecureet
shape( :2]
stype-np.floatth. here for i in range (len (previous_attn_llaee lintttt_llst):: attn_map += arevious_attn_listt
Initialize the attention map Modify
attn_map /= len (previous__ttt__lll of modification return cv2. normalimmieeeeeeeemem (attn_ None, alpha=0, beta=1, norm_ saan__ype=cc..200 MORMMMMMAAMM, stype=ov2.CV_32F) Normalize the attention map via Min-Max Normalization
# End
10

---

C.4 Human-like Spatial Intelligence Grid Creation
Prompt C.8: Human-like SIGC Example General
Q: [Human-like SIGC task prompt] [Original Image] [Original Image with bbox of vehicles].
A: {"vehicles": ("white car 1": [2,3], "gray truck 2": [5,2], "yellow
truck 3": 3 31,,,,, "traffic_lanes":: ("lane 1": 1":[[2,0] [2, 9]] "lane
2": [[6,0], [5,9]], "lane 3": 30 0] [7 "e1,,,111aafiiiii "traffic_sigas"" : (isign
1": [r: "sign 2": [],
1": [4, 1], "light
2": 2", "s,]f"
For human-like SIGC, is similar to the task SIGC (Sec. C.1 The general structure of this task is shown as Prompt. C.8 The detailed prompt input is shown in Prompt. C.9 The original image and image with bbox vehicles are shown Fig. Sa and Fig. 8b respectively. The image size attention map is shown in Fig. 11a and SIG size attention map transformed using homographic transformation is shown in Fig. 11b The attention weight of each object is the corresponding attention weight at their position SIG. During evaluation, these attention weight will be used to scale the punishment of the incorrect prediction for different objects.
(a) Image size human gaze attention map
(b) SIG size human gaze attention map
Figure 11: Example of image and SIG size human gaze attention map

---

Prompt C.9: Human-like SIGC Prompt Example
[Task Summary] The first image captures outdoor driving scene, and the second image is the same scene with bounding boxes on certain vehicles. Please identify vehicles, traffic lanes, traffic signs, and traffic lights within the image like human, and understand the spatial arrangement of these entities. Specifically, assuming the bird's 's eye view of the scene is represented by a 10*10 grid, please estimate the center position of these entities within this grid. The output is expected to be JSON file containing dictionary.
[Task Details] <Overall> 1. The first image captures an outdoor driving scene, and the second image is the same scene but only with bounding boxes on certain vehicles. 2. Estimate the center positions of each instance within the first image, assuming the entire scene is represented by 10 bird's eye view grid. 3. The entities to be estimated include: vehicles, traffic lanes, traffic signs, traffic lights, and the vehicle that capture the images (sclf). 4. Both the horizontal and vertical coordinates of the grid range from 0 to 9, so all estimated positions (c.g., [x_1, y_1l) must fall within this range. 5. Estimated location of cach instance should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects. 6. Please be aware of the front-backward or left-right relationship between instances, there will be partial occlusion. 7. Since it a bird's eye view grid, for all instances, more far away objects should placed in the higher row number and the closer objects should be placed in the lower row number. <Vehicles> 1. Detect vehicles in the first image that are enclosed by bounding boxes in the second image only, and estimate the center positions of these enclosed vehicles within the grid. 2. The output is key-value pair, which is expected to be exactly like: "vehicles": ("black car 1": [x_1, 1_1 "gray truck 2": [x_2, y_21. "yellow bus 3": [x_3, y_3], "blue van 4": [x_4, y_4,, ...}: where each vehicle instance is named by color, vehicle type, and order. 3. The color of vehicles are summarized into: gray, black, white, silver, blue, green, yellow, red, purple. Other colors need to be attributed to the one closest to it among these colors. 4. The type of vehicles must be classified into exactly four categories: "car" (which include sedans, hatchbacks, and suvs), "truck" (including pickup trucks), "van". and "bus". Do not use "suv" or any other subcategory names directly as type. Instcad, map them into one of the four allowed categories. 5. The order of vehicles must be assigned globally, based strictly on their horizontal position in the first image. The left-most vehicle in the image must be numbered "1", the second left- vehicle must be numbered "2", and on. This numbering should apply across all vehicles, without restarting for different colors types. Vehicles usually travel within traffic lanes, so the estimated position of vehicle moving in the same direction as the "self" vehicle should typically fall within the coordinates of a corresponding lane. This does not apply to vehicles traveling in the other directions or that are stationary. <Traffic lanes> 1. Detect all traffic lanes in the same driving direction the vehicle capturing the image, and estimate the lane position within the grid. One lane can be represented by multiple adjacent points as it long. 2. The output is a key-value pair, which is expected to be exactly like; "traffic_lanes". llanes: ( ("lane 1": [[x_1] where each lane is named by order. 3. The order of traffic lanes is based strictly on their horizontal position in the first image. The left-most lane the image must be numbered "1", the second left-most lane must be numbered "2", and so on. 4. Each lane is typically straight, so the horizontal coordinates of its points should usually be the same, forming a vertical line in the grid, unless the lane is clearly turning, merging, or splitting. 5. Additionally, adjacent lanes are typically close together, so their horizontal coordinate values of adjacent lanes should usually differ by only 1. <Traffic signs> Detect all traffic signs in the first image, and estimate the center positions these traffic signs. 2. The output a key-value pair, which is expected to exactly like: "traffic_signs"" : ("sign 1": [__1,__11, "sign 2": [x_ vx_2,y_2 y_2,, 2__])...). where each sign is named by order. 3. The order of traffic signs is based strictly on their horizontal position in the first image. The left-most sign in the image must he numbered "1", the second left- sign must be numbered "2", and so on. 4. If multiple signs mounted on the same horizontal pole, please treat them as separate instances. <Traffic lights> 1. Detect traffic lights in the first image, and estimate the center positions of these traffic lights. 2. The output is key-value pair, which is expected to be exactly like: traffic_lights
y_ 11. [x_ [x_12, y_12], "lane 2": [[x_21, y_ 21, [x_22, y_221,...I...
("light 1": [x_1, y_,, "light 2": [x_2, y_21, ...). where each light is named by order. 3. The order of traffic lights is based strictly on their horizontal position in the first image. The left-most light in the image must be numbered "1", the second left-most light must be numbered "2", and so on. single light and use the midpoint between them as the center. <Self> 1. Estimate the center location of the vehicle that captures the image. 2. The output is expected to be exactly like: '"self": [x. 0] 3. The vehicle capturing the image should be placed in the point with index 0 and with column index depends on different images.
multiple traffic lights are mounted on the same horizontal pole, please treat them as one
[Output] 1. Combine the key-value pairs of vehicles, traffic lanes, traffic signs and traffic lights into one dictionary, as final output. 2. The final output dictionary is expected to be exactly like: { "vvehicee":: ("black car 1": [x_1, y_1], "gray truck 2": [x_2, y_2], "yellow hus 3": [x_3, y_ 2]3 "blue van 4": [x_4, y_41. ...,, traffic_lanes" ["lane 1" [[x_]1, y_ 11], [x 12, y 12), ...l. "lane 2": [[x_21, y 2_1] [x 22, y_22], ........ "traffic_sign"" _signs": ("sign 1": [x_1,-_11 "sign 2": y_21, ...) "traffic "traffic_lights". ( ("light 1": [x 1. "light 2": [x 2, 2)...... "self": [x. 0] 3. Please output the final dictionary in pure ISON format, without any additional text or explanation before or after. 4. All keys and string values in the output dictionary must be in lowercase letters only.
12

---

C.5 Human-like Spatial Relation Paragraph Filling
Prompt C.10: Human-like SRPF Example General
Q: [SRPF task prompt] [Original Image] [Original Image with bbox of vehicles]
[Directional Relation Paragraph Template] + [Proximal Relation Paragraph Template].
A: catpbweerssdrreetionnalll
"answers_
The general structure of this task is shown as Prompt. C.T0. The detailed prompt we input is shown in Prompt. C The original image and image with bbox of vehicles are shown in Fig. [8 and Fig. [8b] respectively. The image size attention map is shown in Fig. ITa and SIG size attention map transformed using homographic transformation is shown in Fig. TTb The attention weight for each sentence between two objects would be the average of these two objects using their corresponding attention weight at their position in SIG.
For human-like SRPF, it is similar the task SRPF (Sec. C.2
Prompt C.11: Human-like SRPF Prompt Example
[Input Description] You will be provided with two images and two incomplete text paragraphs, think as human driver. 1. The first image shows real-world driving scene. 2. The second image is the same scene with bounding boxes drawn several vehicles. 3. The first paragraph is Directional Template that deseribes the directional relationships between certain vehicles, traffic signs, traffic lights, using [directional preposition] as placeholders. Example: "Black car is [directional preposition] white truck 2. Black car is [directional preposition] sign 1." 4. The second paragraph a Proximal Template that describes spatial proximity relationships (e.g., near/far) between the same entities, using [proximal preposition] as placeholders. Example: "Black is [proximal preposition] sign 1." The numbers placeholders the above two paragraphs are the same, as they describe relationships for the same entities. 6. The number in each object name (e.g.. black car sign 2) indicates its horizontal left-to-right order in the image among entities of the same type. For vehicles, the order is based all boxed vehicles in the second image. For signs and traffic lights, the order is based their horizontal position the image (from either image).
is [proximal preposition] white truck 2. Black
ITTask and Requirements] (NOT the label string) from the following label lists for each [directional preposition] and [proximal preposition] placeholder. DIRECTIONAL LABELS ["at the back of".. "at the back left of", "at the left of", "at the front left of". "at the front of". "at the front right of", "at the right of". "at the back right of"] PROXIMAL MAAXIMALLLBBBLSSS
Based on the provided images and templates, predict label INDEX
["adjacent to", "close to", a distance". "far from". ", "far away from") 2. Please output two integer lists: "answers_ dirrcciiorcccl and "answers_ "answers_proximall
a json format, where each entry is the index of the selected label from the provided label list. Output example: {( änswers directional: [4, 7.... 1J, änswers proximal' [2, .... 0] ]1 The list answers. directional must contain only integer indices ranging from to (inclusive), and answers annserssproximiiie proximal
contain indices from 0 to (inclusive), as each index corresponds to valid entry from the DIRECTIONAL. T.ABL. S and PROXIMAL, A LABELS S lists, respectively. 4. The length of the answer lists must exactly match the number of [directional preposition] or [proximal preposition] placeholders in its respective template. This number will be explicitly provided to you along with the templates for each data example. 5. Do not provide explanations. Return only the two answer lists. may be no detectable entities, which case both templates will contain only a single period ".". When this occurs, you should simply return two empty lists. The following are the two incomplete text paragraphs of this inference: [Directional Template] (template diiiitttodrrr) [Proximal Template] [template proximal) The lists 'answers_ directional' and 'answers munswers_prxximaat proximal' you
some driving scenes, there
each have EXACTLY (num_pt placcholderss entries. 'answers directional' must contain ONLY INTEGERS FROM 0 to 7, and 'answers. proximal' must contain ONLY INTEGERS FROM 0 to 4. Do not use quotation marks around the numbers all entries must be returned as raw INTEGERS. NOT STRINGS.
13


Reviews:
[{"review_id": "GpAqHzTUnk", "reviewer": "Reviewer_wMQv", "summary": "The paper proposes a novel approach and benchmark for evaluation of visual spatial intelligence of MLLMs.\nThe authors introduce spatial grid based methods  for assessing spatial relationships in Autonomous Driving scenarios and evaluate whether current MLLMs  can answer spatial queries, generate the spatial grid from images using  in-context learning. The gaze of human differers is examined in comparison with  machine based approaches.", "questions": "- Provide examples of queries and how they differ for the queries in other benchmarks such as SpatialVLM or VSR benchmarks and how they are obtained for SIG\n-  How does the approach extend to other scenarios beyond Autonomous Driving", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- novel benchmark for map style representation for assessing spatial relationships \n- novel evaluation metric, using spatial relation graph similarity, multi-level spatial matching \n\n\nWeakness \n- this is likely applicable only Autonomous driving scenarios \n- the positioning of the work with respect to related works could be improved, \n- the method is effectively a map based methods, see [1] for additional references\n- better motivation for new evaluation metrics should be provided\n- the connection to human gaze evaluation is weak\n\nThe paper is not well organized. The discussion the evaluation metrics precedes the discussion of grid computation and the benchmark, there are very few examples of prompts, so it is hard to address the complexity, strengths and weaknesses of the method. The approach suggests to use in context learning  to obtain the spatial grid json format, which is now well aligned with the benchmarks where the models reason \nabout spatial relations in Bird Eye View map. \n\nPlease look at this paper and references within, \n[1] https://arxiv.org/pdf/2410.06468\nDOES SPATIAL COGNITION EMERGE IN FRONTIER\nMODELS? Santhosh Kumar Ramakrishnan∗ Erik Wijmans Philipp Krahenb ¨ uhl Vladlen Koltun", "was_revised": true, "final_justification": "The rebuttal and the discussion improved my appreciation of the results leading to the increase of my rating. I still have some outstanding concerns about insights to MLMM's alone and the representations that are learned in the pre-training stage.", "rebuttal": "We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging the novelty of our proposed data representation Spatial Intelligence Grid (SIG) and evaluation metrics: Multi-level Spatial Matching (MLSM), Spatial Relation Graph Similarity (SRGS) and Semantic Relational Distance (SRD). We will address your questions one by one in the following.\n\n**W-1&Q-2 This is likely applicable only for AD scenarios.** Although our initial motivation arise from autonomous‑driving (AD) applications, the SIG framework is fundamentally domain‑agnostic and can be applied wherever a fixed ontology of object types exists. To demonstrate this, we have built two proof‑of‑concept benchmarks based on subsets from MS COCO [1] and ARKitScenes [2], named as SIG-COCO and SIG-ARKitScenes, respectively. We run both zero-shot inference and ICL using GPT-4o on these two benchmarks. The results are shown in the following table. Due to time limitations and workload of dataset annotation, we only provide partial results of MLSM and SRGS here, we will update the results of SRD related metrics in 3-4 days.\n\n| Benchmark           | Type    | **MLSM** |        |        |        | **SRGS** |       | \n|---------------------|---------|:--------:|:------:|:------:|:------:|:--------:|:-----:|\n|                     |         | P↑       | R↑     | F1↑    | AssA↑  | S↑       | WS↑   |\n| SIG-COCO        | Z-S     | 0.758    | 0.826  | 0.771  | 0.652  | 0.574       | 0.607     | \n|                     | ICL-SIG | 0.860    | 0.840  | 0.849  | 0.749  | 0.711        | 0.746     |\n| SIG-ARKitScenes | Z-S     | 0.714    | 0.756  | 0.727  | 0.581  | 0.619        | 0.597     |\n|                     | ICL-SIG | 0.809    | 0.850  | 0.823  | 0.719  | 0.729       | 0.737     | \n\n**W-2 Position of this work with other related works.** Thanks for drawing our attention to the related literature. In particular, the paper [3] you suggested offers valuable insights closely aligned with our work. In the revised manuscript, we will integrate a discussion of [3] alongside citations to [4], [5], [6], and [7], clearly positioning our contributions in the context of these existing studies. If you are aware of any additional papers that would further enrich our comparison, we would be grateful for your recommendations and are happy to include them.\n\n**W-3 Similar as map-based method.** While the map‑based approach in [3] bears a conceptual resemblance to our SIG, our work departs in several important ways:\n\n- Scenarios from real‑world: We annotate SIG directly on real autonomous‑driving scenes rather than synthetic or abstract maps, ensuring our benchmarks measure performance under the complex visual conditions encountered “in the wild.”\n- Richer object classes: Beyond a generic “object” class, our SIG distinguishes multiple vehicle types (cars, trucks, vans, buses), traffic signs, and lights, and further encodes intra‑class attributes such as color and order.\n- Instance‑level complexity: By including scenes with several instances of the same category—each with its own spatial relation and attribute differences—we stress‑test a model’s ability to generalize spatial understanding to densely populated environments.\n- One‑shot relational output: Our evaluation framework invites MLLMs to output a complete SIG in a single pass, mirroring how humans naturally describe spatial relations in a cohesive narrative, rather than iterative queries of VQA in [3], which has been studied by enormous of existing papers.\n\nThese distinctions underscore how SIG extends and challenges map‑based representations to better capture the visual‑spatial intelligence needed for real‑world driving applications.\n\n**W-4 Better motivation for proposed evaluation metrics.** When a driver is driving, he/she typically focus on a range containing selected objects and their spatial correlation instead of pixel-level or object-level information. To build a representation that preserve this, inspired from classical grid‑based visual schemes (e.g., Renaissance “drawing machines”) that artists used to decompose complex scenes into structured partitions, we introduce SIG that preserves fine-grained geometric and topological information in a 10×10 grid, directly encoding object positions and their spatial relations. Based on SIG, we design three complementary evaluation metrics:\n\n- MLSM: A thresholded, bipartite‑matching protocol that tolerates minor grounding noise while measuring localization precision at increasing levels of attribute matching (type, order, color).\n- SRGS: Based on graph edit distance (GED), this metric rigorously quantifies structural differences between predicted and ground‑truth scene graphs, penalizing missing or mis‐classified relations.\n- SRD: By mapping prepositions onto discrete scales (cyclical for direction, linear for proximity), SRD captures the degree of semantic error, rather than a binary correct/incorrect judgment.\n\n**W-5 Connection to human gaze evaluation is weak.** While the core contribution of this work lies in introducing SIG and SIG‑based evaluation metrics, modeling human‑like VSI via gaze serves two essential roles:\n\n- Bridge to real‑world decision making: In AD scenario, human drivers do not process every object equally. They deploy selective attention, focusing on a few critical elements such as traffic lights at an intersection—while deemphasizing others (e.g., distant vehicles). By integrating gaze‑derived saliency into SIG (Eq. 6), we weight our graph‑ and grid‑based metrics so that high‑importance objects incur greater penalty when mispredicted.\n- Human‑aligned relational evaluation: Traditional VQA or text‑only metrics treat each relation equally, ignoring that some spatial errors are more consequential than others. For instance, confusing “at the front of” versus “at the back of” a high‑attention object (e.g., a braking light) can have drastically different safety implications. Our Human‑Like SRGS and SRD scale node‑ and edge‑edit costs by gaze weights, ensuring that mistakes on visually salient entities are penalized more heavily.\n  \n\nTogether, these gaze‑weighted evaluations not only emulate how humans allocate attention in dynamic driving scenes but also align model assessment with the situational priorities of real‑world AD systems—thereby strengthening the practical relevance of our human‑like VSI benchmark.\n\n**Q-1 Examples of queries and difference from SpatialVLM and VSR.** For examples of queries, please refer to Appendix C. Below we illustrate how query formats in SpatialVLM, VSR, and SIGBench differ—both in their structure and in how they are obtained:\n\n| Benchmark  | Query Format                                                 | Evaluation Metrics                                           |\n| ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| SpatialVLM | “How far is A from B (m)?”, “Is X larger than Y?”            | Mean absolute error (m), Size‑ordering accuracy              |\n| VSR        | “Object 1 is left of Object 2.” True/False?                  | True/False classification accuracy                           |\n| SIGBench (ours)   | General‑VSI: SIG creation in JSON format (SIGC) and spatial‑relational paragraph fill‑in (SRPF); Human‑like VSI: Gaze prediction + H‑SIGC + H‑SRPF | General‑VSI: MLSM, SRGS, SRD; Human‑like VSI: attention‑based SRGS/SRD; Gaze prediction: PCC, IG, KL‑Divergence |\n\nThis direct, structured prompting—together with our ground‑truth SIG annotations—allows SIGBench to evaluate complete spatial understanding in one shot, rather than separate VQA pairs. We also provide more comprehensive results for SIG-based ICL, please refer to reviewer **BBAE W-2 and Q-4**.\n\n[1] Tsung-Yi Lin, Micheal Maire, ..., Piotr Dollar. Microsoft COCO: Common Objects in Context.\\\n[2] Gilad Baruch, Zhuoyuan Chen, ..., Elad Schulman. ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data.\\\n[3] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Krahenbuhl, Vladlen Koltun. DOES SPATIAL COGNITION EMERGE IN FRONTIER MODELS?\\\n[4] Santhosh Kumar Ramakrishnan. Predictive scene representations for embodied visual search.\\\n[5] Runsen Xu, Weiyao Wang, ..., Kevin J. Liang. Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models. \\\n[6] Sihan Yang, Runsen Xu, ..., Jiangmiao Pang. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence. \\\n[7] Linjie Li, Mahtab Bigverdi, ..., Ranjay Krishna. Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations."}, {"review_id": "DnS5YXJK9b", "reviewer": "Reviewer_wr1L", "summary": "This paper proposes a novel grid-based representation, Spatial Intelligence Grid (SIG), for encoding Visual-Spatial Intelligence (VSI) in multimodal large language models (MLLMs), particularly in the context of autonomous driving (AD). The authors argue that existing Visual Question Answering (VQA)-style representations inadequately capture spatial reasoning.\nThey introduce SIG, a structured grid-based scene representation that embeds spatial and physical priors, along with a new benchmark, SIGBench, comprising over 1,400 annotated driving frames with human gaze data and ground-truth SIG labels.\nExtensive experiments on several state-of-the-art MLLMs demonstrate that SIG-based few-shot in-context learning (ICL) improves performance in spatial reasoning tasks compared to traditional VQA-style prompts. Additionally, SIG enables human-like VSI assessment through attention-modulated metrics.", "questions": "- How well would SIG generalize to domains beyond autonomous driving? Could the authors clarify or expand on this?\n- What is the runtime overhead of the proposed graph-based metrics, and are they practical for real-time applications like autonomous driving?\n- Can the authors provide an analysis of common failure cases to better understand model limitations?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written, clearly structured, and effectively presents its main contributions. The integration of figures with the textual explanations is useful, providing clarity and improving the reader’s understanding of the proposed method.\n- SIGBench is thoughtfully designed for both grid-based and human-like VSI evaluation, incorporating gaze-based annotations and rigorous, graph-theoretic evaluation metrics. The proposed metrics (MLSM, SRGS, and SRD) are particularly well-motivated, offering a structured and interpretable means of quantifying spatial reasoning performance beyond simple right-wrong QA scoring. They allow for fine-grained assessment of relational and positional errors, which is especially valuable in complex environments like autonomous driving.\n- The authors extensively benchmark multiple proprietary and open-source MLLMs, providing strong empirical evidence that SIG consistently improves spatial reasoning in both zero-shot and in-context learning setups.\n- The discovery that current MLLMs fail to appropriately prioritize objects based on attentional salience, as humans naturally do in driving tasks, is a valuable insight. Moreover, the choice of autonomous driving as a testbed is appropriate given its real-world complexity and practical relevance.\n\nWeaknesses:\n- While autonomous driving is an appropriate and demanding testbed, the paper does not explore how SIG might generalize to other domains. While this aspect is being discussed briefly in the Appendix I think it still is worth noting.\n- The paper introduces graph-edit distance and semantic relational distance metrics, but it does not address their computational costs during large-scale inference. Given that the target application is autonomous driving, where real-time efficiency is crucial, an analysis of runtime performance of the pipeline would be valuable.\n- The paper would benefit from a qualitative or quantitative analysis of typical failure modes, such as common errors in SRGS predictions or attention misallocations. This could help identify systematic weaknesses in current models and inform future improvements.\n- The primary experimental advantage demonstrated stems from in-context learning (ICL) with SIG, while no experiments on full fine-tuning or reinforcement learning with human feedback (RLHF) are presented. This leaves open the important question of how effectively SIG could be integrated into end-to-end training pipelines for production systems.\n\nOther issues not affecting the rating:\n- Typo at line 120\n- Line 254 -> Pearson", "was_revised": true, "final_justification": "I appreciate the authors’ thoughtful rebuttal, particularly the additional analyses on runtime and failure cases, which have addressed my main concerns and strengthened the paper’s contributions. Given these improvements, I recommend Accept, under the condition that the additional analyses and clarifications provided in the rebuttal are incorporated into the camera-ready version. I trust the authors to ensure that these enhancements are faithfully integrated.", "rebuttal": "We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging our proposed evaluation metrics are well-motivated and our extensive experiments provides strong empirical evidence for Spatial Intelligence Grid (SIG)'s potential in improving visual-spatial intelligence (VSI) of MLLMs. We will address your questions one by one in the following.\n\n**W-1&Q-1 Can SIG generalize to other domains than AD.** Please refer to our response to reviewer  **wMQv W-1&Q-2**\n\n**W-2&Q-2 Analysis of runtime.** Runtime efficiency is definitely important, especially for real‑time applications in autonomous driving (AD). Below we summarize both the theoretical complexity of our metrics and empirical timing results on SIGBench. \\\nTheoretical time complexity: Let $n$ denotes the total number of objects (vehicles, traffic signs and traffic lights) per frame. \n  - For Multi-level Spatial Matching (MLSM), the time complexity for building cost matrix for vehicles, traffic signs and traffic lights is $O(n^2)$. The time complexity for bipartite matching is $O(n^3)$ [1]. For multi-threshold matching, the time complexity is $O(n)$. The overall time complexity for MLSM is $T_{MLSM}(n) = O(n^2) + O(n^3) + O(n) \\Rightarrow O(n^3)$. \n  - Similarly, for Spatial Relation Graph Similarity (SRGS), during the fully connected graph construction, we require $O(n)$ for all nodes and $O(n^2)$ for all edges. Next, the bipartite matching requires $O(n^3)$ similarly. During the graph edit distance (GED), we are performing node-to-node and edge-to-edge comparison, which requires $O(n) + O(n^2)$. Thus, the time complexity for SRGS is $T_{SRGS} = 2(O(n) + O(n^2)) + O(n^3) \\Rightarrow O(n^3)$.\n  - For Semantic Relational Distance (SRD), we are computing pointwise distance. For SRD (Directional) and SRD (Proximal) there are 8 and 5 possible relations, respectively. Thus, the time complexity for SRD is $T_{SRD} = O(8n) + O(5n) \\Rightarrow O(n)$.\n\nEmpirical timing: The dataset statistics of SIGBench is:\n| #Objects | 2   | 3   | 4   | 5   | 6   | 7   | 8   | 9   | 10  | 11  | 12  | 13  | 14  | 15  | 16  | 17  | 18  | 19  | 20  | 21  | 22  |\n|----------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| #Samples | 145 | 204 | 219 | 180 | 145 | 105 | 99  | 59  | 47  | 38  | 33  | 18  | 15  | 6   | 6   | 3   | 0   | 2   | 0   | 0   | 1   |\n\nThe average per-frame runtimes on CPU (AMD Ryzen 5900X) are:\n| #Objects  | 3      | 5      | 7      | 9      | 11     | 13     | 15     | 17     | 19     | 22     |\n|-----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| MLSM [s]  | 0.0001 | 0.0002 | 0.0002 | 0.0002 | 0.0002 | 0.0002 | 0.0002 | 0.0003 | 0.0003 | 0.0004 |\n| SRGS [s]  | 0.0001 | 0.0002 | 0.0003 | 0.0004 | 0.0004 | 0.0005 | 0.0006 | 0.0006 | 0.0008 | 0.0009 |\n| SRD [s]   | <0.0001| <0.0001| <0.0001| <0.0001| <0.0001| <0.0001| <0.0001| <0.0001| <0.0001| <0.0001|\n\nEstimated real-time performance: Consider a typical AD vehicle with six cameras at 30 FPS and an average of 11 detected objects per frame. The total estimated time cost for our metrics per second (without parallelization) is: $6 \\times 30 \\times (0.0002 + 0.0004 + 0.0001 ) = 0.126$s, which definitely meets the real‑time demands. We will include these discussion in the Appendix.\n\n**W-3&Q-3 Qualitative and quantitative analysis of typical failure modes.** To explore common failure modes, we examine zero‑shot performance of two state‑of‑the‑art MLLMs on SIGBench across scenes containing varying numbers of objects. Results are summarized in the following table. \n\n| Type               | #Objects |**MLSM**     |    |        |         | **SRGS** |       | **SRD (Directional)** |    |          |  **SRD (Proximal)**  |      |          |\n|--------------------|:--------:|:--------:|:------:|:------:|:-------:|:--------:|:-----:|:---------------------:|:------:|:--------:|:--------------------:|:------:|:--------:|\n|                    |          | P↑       | R↑     | F1↑    | AssA↑   | S↑       | WS↑   | MAE↓                  | MSE↓   | Acc↑     | MAE↓                 | MSE↓   | Acc↑     |\n| **GPT‑4o**         | 3        | 0.541    | 0.525  | 0.521  | 0.371   | 0.425    | 0.422 | 1.951                 | 5.386  | 0.152    | 0.760                | 1.061  | 0.386    |\n|                    | 7        | 0.473    | 0.346  | 0.394  | 0.255   | 0.249    | 0.238 | 1.967                 | 5.415  | 0.138    | 1.043                | 1.789  | 0.271    |\n|                    | 11       | 0.460    | 0.291  | 0.354  | 0.219   | 0.203    | 0.186 | 1.027                 | 1.602  | 0.245    | 0.383                | 0.457  | 0.654    |\n|                    | 16       | 0.429    | 0.219  | 0.288  | 0.171   | 0.114    | 0.099 | 0.433                 | 0.553  | 0.627    | 0.080                | 0.080  | 0.920    |\n|                    | 22       | 0.465    | 0.190  | 0.270  | 0.160   | 0.098    | 0.132 | 1.520                 | 3.280  | 0.200    | 0.440                | 0.440  | 0.560    |\n| **Gemini‑2.5‑Pro** | 3        | 0.482    | 0.643  | 0.538  | 0.384   | 0.212    | 0.232 | 1.219                 | 2.582  | 0.270    | 0.755                | 1.186  | 0.431    |\n|                    | 7        | 0.492    | 0.547  | 0.510  | 0.353   | 0.228    | 0.213 | 1.321                 | 2.932  | 0.253    | 0.798                | 1.216  | 0.387    |\n|                    | 11       | 0.447    | 0.503  | 0.468  | 0.311   | 0.226    | 0.181 | 0.700                 | 1.068  | 0.441    | 0.318                | 0.381  | 0.713    |\n|                    | 16       | 0.544    | 0.487  | 0.511  | 0.351   | 0.287    | 0.252 | 0.480                 | 0.667  | 0.613    | 0.060                | 0.060  | 0.940    |\n|                    | 22       | 0.565    | 0.437  | 0.493  | 0.327   | 0.407    | 0.357 | 1.000                 | 1.960  | 0.360    | 0.760                | 1.160  | 0.440    |\n\nGPT‑4o’s MLSM and SRGS scores steadily decline as object count rises, while its SRD performance fluctuates, demonstrating that graph‑based metrics are more sensitive to scene density and that raw text judgments alone can mask these trends. Gemini‑2.5‑Pro shows similar SRD variability but even improves on some graph metrics (MLSM‑P, SRGS) with more objects, underscoring the complementarity of SIG’s structured representation. \nQualitatively, we recognize two key failure scenarios: \n- Objects that are either very small or positioned at the periphery of the scene are frequently missed or mislocalized due to their low visual salience. \n- High intersection over union (IoU) between objects often cause the models to conflate object identities or assign them to incorrect grid cells, resulting in swapped or merged scene‑graph nodes. We will illustrate each of these failure modes with representative examples with visualization in the Appendix.\n\n**W-4 Full fine-tuning or RLHF based on SIG.** Thank you for this insightful suggestion. We agree that exploring end‑to‑end integration of SIG into training pipelines could substantially advance model capabilities. We provide more comprehensive results for SIG-based ICL, please refer to reviewer **BBAE W-2 and Q-4**. Based on our positive ICL results, we think two particularly promising directions are: \n- SIG‑guided supervised fine‑tuning. Rather than relying solely on in‑context examples, we could curate a labeled SIG dataset and fine‑tune an MLLM to predict grids and scene graphs directly from raw images. This supervised signal would reinforce precise spatial encoding, potentially yielding improvements not only in SIG reconstruction but also in downstream tasks like VQA or navigation with an improved visual-spatial intelligence with large generalizability.\n- RLHF via SIG. SIG outputs offer a structured, graph‑based explanation of model reasoning. By treating human judgments of SIG quality as reward feedback, e.g., ranking or correcting scene graphs, an RLHF loop could encourage models to align their spatial representations with human preferences. Such a protocol may cultivate nuanced attention patterns and more robust relational reasoning tailored to diverse driving contexts. Both avenues harness the rich structure of SIG to move beyond one‑shot prompting toward production‑ready spatial intelligence, and we will investigate these two interesting directions in future work.\n\n**Typo at line 120; Line 254 $\\rightarrow$ Pearson.** We are deeply apologize for these typos and we will correct them in the final version of this paper. \n\n[1] T. H. Cormen, C. E. Leiserson, R. L. Rivest, C. Stein, Introduction to Algorithms, 3rd ed., MIT Press, 2009, Chapter 16."}, {"review_id": "tBA7hKqc9I", "reviewer": "Reviewer_zSnK", "summary": "The present study is concerned with representing the spatial locations and relationships among static and dynamic\nobjects in Autonomous Driving (AD) environments. In the proposed graph-based representation, objects are nodes loosely\nlocated in a bird's eye view (BEV) regular grid. Relationships are encoded as edges corresponding to proximity and\nego-vehicle centric directional relations (e.g. \"car A is at the back of car B\"). Furthermore, the authors assign\nvisual saliency to objects by placing attention weights into the grid cells. \n\nThe proposed representation, termed SIG (Spatial Intelligence Graph) is employed both as ground truth and as model\noutput:\n  - **ground truth** - the authors introduce a benchmark dataset (SIGBench), consisting of approximately\n1400 frames annotated with three elements:\n    - Spatial Intelligence Graphs (SIGs)\n    - Spatial Relationship Paragraphs (SRPs) - a textual representation of the relationships in the SIGs\n    - human gaze annotations (predicted with off-the-shelf methods and back-projected from image space onto the SIGs BEV grid)\n  - **model output** - the authors prompt state-of-the-art Multimodal Large Language Models (MLMMs) with:\n    - road scenes, asking to predict SIGs\n    - road scenes, asking to perform SRP preposition completion\n    - a temporal history of frames and saliency maps, with instructions to predict the visual saliency grid values for\nthe current frame.\n\nThree metrics are used:\n - Multi-Level Spatial Matching: to capture the qualitative object localization/classification\n - Spatial Relationship Graph Similarity: to capture overall graph structure\n - Semantic Relational Distance: to capture the fidelity of particular inter-object proximal/directional relations\n\nThe authors conduct experiments involving both open-source and proprietary MLLMs. They mainly find that:\n - the selected proprietary MLLMs out-perform the open-source MLLMs on SIG an SRP prediction\n - spatial intelligence can be significantly improved by in-context learning with a random set of (image, SIG pairs) as\nprompts\n - MLLMs cannot effectively predict human visual saliency in SIG space", "questions": "- Does the SIGBench dataset also preserve object IDs? Would it allow for tracking dynamic objects across individual\nSIG frames?\n- The model and benchmark seem to rely on single-camera and single frame information. This makes depth prediction\nambiguous up to scale. Have the authors also experimented with scale-invariant matching against the BEV locations?\n- The grid resolution seems to be critical for the evaluation metrics. How do the authors decide on the value to\nuse?", "limitations": "This reviewer thinks the authors should point out that the method:\n - relies on predicted human visual saliency, hence suffers from systematic prediction errors induced by the respective methods.\n - does not consider lanes and their relationships to the objects in the driving environment, which may\n limit applicability for AD", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n  - Well written paper and clearly structured work\n  - High importance, as spatial scene understanding is an outstanding issue in MLLMs applied to AD, especially with\nthe emergence of end-to-end driving\n  - Extensive evaluation \n\nWeaknesses:\n\n - There is no human baseline for the proposed metrics (e.g. based on consistency between human subjects). It is\ndifficult to judge how far the SOTA spatial intelligence level really is compared to the gold standard of human drivers\n - The study omits lanes from the evaluation (lines 130-131). However, proper understanding of the lane and drivable\nspace structure are crucial for AD. Furthermore, by omitting lanes, the study misses on the opportunity to model\nhigher level driving scene spatial layouts, such as \"vehicle 1 on the left lane\" or \"traffic sign 1 corresponds to the\nego lane\". Such relationship are of primary importance for AD (consider overtaking or stopping for a red light).\n - Dynamic spatial relationships between objects (e.g. vehicles approaching a pedestrian crossing or accelerating\ntowards the ego vehicle) are not covered \n - The BEV grid resolution is somewhat coarse (10x10), and the authors do not provide an intuition on how to calibrate\nit\n - The saliency prediction prompt seems to ask the MLLM to generate python code for producing the attention map. It is\nnot clear to what extent this setup affects the outcome.", "was_revised": true, "final_justification": "I thank they authors for their detailed responses. Their rebuttal has addressed my concerns. In particular, I think the inclusion of lane markings (albeit on a subset of the dataset) and the generalization to other domains would greatly strengthen the contribution of the present work.\n\nIn future manuscripts, I encourage the authors to include the choice of a single-frame setup as a limitation, and to discuss their analysis on the grid size and the saliency map templating strategy in the appendix.", "rebuttal": "We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging the importance of our work in improving spatial scene understanding of MLLMs in autonomous driving (AD) scenario, along with extensive evaluation. We will address your questions one by one in the following.\n\n**W-1 No human baseline.** We agree that a human baseline is essential for contextualizing model performance. Accordingly, in Tab. 2 (first row) we report the average scores of four human subjects on our general VSI tasks. We will emphasize these results more prominently in the revised experiments section to clarify how current SOTA models compare against human-level spatial reasoning. \n\n**W-2&L-2  Omitting lanes from the evaluation.** We agree with the importance of lane and drivable‐space understanding in AD scenarios. You are absolutely right that lane assignments (e.g. “Car 1 is in the left lane”) is critical for maneuver planning. Our SIGBench dataset does include human‐annotated lane masks for images where lanes are visible. Currently, there are several mature methodologies specifically designed for traffic lane detection such as [1], [2], [3]. However, in this work we mainly focus on the spatial relationship among objects rather than spatial traffic rules. Moreover, SIGBench encompasses a diverse range of driving scenarios: some featuring faded or blurred lane markings and others (e.g., intersections or rural roads) where lane demarcations are entirely absent. In the revised manuscript, we will select a subset of images with clearly visible lane annotations and extend our evaluation metrics to incorporate lane‑based spatial relationships for this subset.\n\n**W-3 Dynamic spatial relationships between objects.** Thank you for this insightful suggestion. We agree that modeling dynamic spatial relationships, such as vehicles approaching a pedestrian crossing or accelerating toward the ego vehicle, is a critical aspect of visual‑spatial intelligence. As SIGBench currently focuses on single‑image scenarios, inferring motion or intent from a static frame can be challenging even for human observers. However, because our source data consists of video sequences, we intend to revisit these clips and introduce temporal annotations in a future extension of the benchmark. This enhancement will allow evaluation of models’ ability to reason about object trajectories and evolving spatial configurations.\n\n**W-4&Q-3 The 10x10 grid resolution is coarse.** Choosing 10×10 discretization for our Spatial Intelligence Grid (SIG) was motivated by typical U.S. highway layouts: mostly up to four lanes per direction (eight total), plus one extra column on each side for signs and lights, yielding 1+8+1=10 horizontal cells. To form a square grid, we also use 10 rows vertically. \\\nTo give an intuition for how this maps to real‑world dimensions, consider a reference plane at 27m in front of the camera. The camera used for recording has focal length 3.6mm and active area =5.4mmx4.0mm. Since our image size is $1920\\times1080$, we can get the focal length in pixels as:\n$$f_x = \\frac{3.6mm}{5.4mm} \\times 1920 = 1280\\text{px}, \\quad f_y = \\frac{3.6mm}{4.0mm} \\times 1080 = 972\\text{px} $$\nIf we divide the image into 10x10 grids, each image cell is 192px wide ($\\Delta u$) and 108px tall ($\\Delta v$). If we have $Z = 26$m, we can solve the width ($\\Delta X_{grid}$) and height ($\\Delta Y_{grid}$) of each cell in our SIG by:\n$$\n    \\Delta X_{grid} = \\frac{\\Delta u}{f_x}Z = \\frac{192\\text{px}}{1280\\text{px}}\\times 27m \\approx 4m, \\quad \\Delta Y_{grid} = \\frac{\\Delta v}{f_y}Z = \\frac{108\\text{px}}{972\\text{px}}\\times 27m \\approx 3m\n$$\nwhich means each grid in our SIG represents approximately a 4mx3m rectangle plane in real world. We also provide more comprehensive results for SIG-based ICL, please refer to reviewer **BBAE W-2 and Q-4**. Our SIG representation can also be generalized to other domains besides AD, please refer to reviewer **wMQv W-1&Q-2**.\n\n**W-5 Whether asking MLLM to generate python code for saliency prediction affects the outcome.** We choose a Python‑code prompt for saliency prediction generation using MLLMs for two main reasons:\n- Token‑limit constraints: A full 1080x1920 attention map contains 2073600 values, far exceeding the maximum output capacity of our evaluated models (e.g., 64000 tokens for GPT‑4o, which is already much larger than that of most proprietary and open-source models). If we want the GPT-4o to output the full attention map of one image, we need 2073600/64000 = 32.4 iterations. Thus, asking the model to emit code that reconstructs the map circumvents this limit by having the model produce a compact algorithm rather than raw data, which can then be executed to yield the full attention tensor.\n- Execution robustness: By prescribing a code template that references only our approved libraries and data structures, we ensure that the generated routines run correctly and consistently in our environment. This approach also allows us to validate and debug any edge cases in advance, increasing overall reliability.\n  \nThrough extensive prompt engineering and manually checked over 100 cases, we found that this code‑templating strategy does not bias the resulting attention maps compared to raw‑value generation. Instead, it simply provides a practical mechanism to deliver high‑resolution saliency outputs within existing MLLM constraints.\n\n**Q-1 Does the SIGBench dataset also preserve object IDs that supports tracking?** While SIG provides a natural schema for object identity and could facilitate cross‑frame tracking, SIGBench in its current form is limited to single‑image annotations and does not include persistent object IDs. Establishing reliable object tracks would require extensive video‑based labeling, which is beyond the scope of this initial release. We agree that integrating object IDs and temporal relationships would greatly enhance the benchmark, and we plan to explore this extension along with the dynamic object relationship in a future SIG video dataset. \n\n**Q-2  Scale-invariant matching.** We agree with you that relying on a single camera and frame can introduce scale ambiguity in depth estimation. Our choice of this setting was deliberate: by demonstrating that SIG‑based ICL improves visual‑spatial reasoning under these challenging constraints, we establish a strong foundation. Importantly, our SIG representation is agnostic to sensor modality or scale: it can be directly extended to multi‑camera rigs or LiDAR–camera fusion systems that yield metrically accurate depth.\\\nAlthough we have not yet performed explicit scale‑invariant matching against BEV ground truth, our grid‑based metrics: Multi-level Spatial Matching (MLSM) and Spatial Relation Graph Similarity (SRGS), operate on normalized cell indices rather than absolute distances, making them naturally robust to uniform scale shifts. More specifically:\n- MLSM builds a cost matrix based on the cell‑coordinate differences between predicted and ground‑truth objects; since both are expressed in the same 10×10 index space, any uniform scaling of the scene (e.g., due to camera zoom or depth ambiguity) cancels out.\n- SRGS compares the structure of scene graphs via node and edge matches. Its graph‑edit operations count mismatches in graph topology and relation labels rather than measuring literal distances in meters.\n\nAs a result, both metrics remain unaffected by global scale shifts: they focus purely on relative spatial configuration and object relationships, ensuring consistent evaluation across different calibration settings.\n\n**L-1 Predicted human visual saliency will result in systematic prediction error.** We apologize for any misunderstanding. In our experiments, ground‑truth human visual saliency was captured using the SmartEye Pro 5 eye‑tracking system. According to the device specifications and validation studies conducted by SmartEye, the gaze‑direction measurements achieve a precision of approximately $\\pm0.5^{\\circ}$ [4]. This level of accuracy is sufficient for our purposes, and we therefore consider these recordings to represent reliable ground‑truth saliency.\n\n[1] Xingang Pan, Jianping Shi, ..., Xiaoou Tang. SCNN: Spatial As Deep: A Convolutional Neural Network for Scene Segmentation of Lane Markings. ECCV 2018 \\\n[2] Lucas Tabelini, Rodrigo Berriel1, ..., Thiago Oliveira-Santos. Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection. CVPR 2021 \\\n[3] Lizhe Liu, Xiaohao Chen, Siyu Zhu, Ping Tan. CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution.  ICCV 2021\\\n[4] Smart Eye AB. SmartEye Pro 5 User Manual (Revision 249)"}, {"review_id": "McS4pbDC3V", "reviewer": "Reviewer_BBAE", "summary": "The authors introduce Spatial Intelligence Grid (SIG), a new format for representing autonomous driving scenes, where the objects as well as their geometrical spatial relationships are embedded into a grid-based data schema. They create a dataset of input scenes and annotated SIGs and define the tasks of extracting its essential components, which include the birds-eye grid view of the scene featuring the objects and their locations, and the spatial relation graph (SRG) and the spatial relation paragraph (SRP), which describe the spatial relations between objects in a numeric and linguistic terms, respectively. They also define the metrics used to evaluate the model performance on these tasks, which the authors claim to more accurately reflect the VSI capabilities of models. Finally, the authors show that these representations are more effective for few-shot in context learning than the standard VQA style representations.", "questions": "- How are gamma and beta for weighted similarity chosen?\n- Why wasn’t gaze based weighting also applied onto the MLSM?\n- Why does the fact that the relative ranking of models doesn’t change significantly with the addition of gaze based weighting point to the conclusion that the “current MLLMs are still unable to interpret the prioritization of different objects in a scene”?\n- Any insights on why Gemini-2.5-Pro is so much better than GPT-4o on ICL on SIGBench-tiny, despite not being that much better on SIGBench itself? It might have been interesting to see the performance of other models on ICL (e.g. if it turned out an open source model was competitive), given that it appears to be not all that tied to the SIGBench performance", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper introduces a novel dataset, featuring a new way of representing autonomous driving scenes, which can be valuable to the research community\n- The authors evaluate a range of state-of-the art VLMs on their tasks, simultaneously demonstrating both the utility of the resulting representations by showing them to be more effective for 3-shot in context learning than standard VQA based approaches, and showing a considerable existing performance gap to humans, motivating further research\n- Rich appendix with detailed descriptions of their data collection and annotation process, as well as examples of their proposed tasks\n\nWeaknesses:\n- The authors claim to “derive a set of SIG-based optimal evaluation metrics that rigorously quantify a model’s true VSI capabilities”, and while it does make sense that these metrics point to the VSI capabilities of models, their \"optimality\" feels overstated. In general, I do not think that the work clearly demonstrates the advantage of these evaluation metrics in terms of representing the spatial intelligence of models\n- The choice of the 3 context samples could have had some degree of impact. It would have been interesting to see if this was negligible or not\n- I feel like the results on human-like VSI offer too little insights compared to how much space in the main paper was dedicated to it. On a more minor note, in some cases, less important information, such as Eq 7 in 4.1 (which is also featured in the appendix) is presented in the main paper, while on the other hand, more important details are omitted (e.g. in 3.3 where it leaves out how the correspondences between image pixels and grid cells were established when calculating the homographic matrix)", "was_revised": true, "final_justification": "I appreciate the replies provided by the authors. I will keep the ranking from the initial review.", "rebuttal": "We want to thank the reviewer for your efforts in handling this submission and we appreciate you acknowledging novelty and utility of our proposed SIG and the rich appendix with detailed description. We will address your questions one by one in following.\n\n**W-1 Optimality is overstated.** We agree that “optimality” is stronger than what our current evidence supports. We will refine the wording to: \"derive a set of \\textbf{SIG-informed} evaluation...\". This phrasing better reflects how our metrics leverage the SIG to assess spatial reasoning without overclaiming optimality.\n\n**W-2 Choice of 3 context samples have impacts.** To answer this question, we conduct extensive experiments investigating the effect brought by the choice of training samples for ICL on GPT-4o. The results are shown in the following table. As you can see, by more carefully selecting the training samples, the performance of ICL is much higher compared to random data sample selection we mentioned in our paper. These results further demonstrate the potential of SIG as a new data schema for improving visual-spatial intelligence of MLLM. \n\n*Table. Quantitative comparison of 3-shot ICL for grid-based VSI tasks on SIGBench-tiny with data selection based on number of objects on GPT-4o. Partial-empty means the number of objects in one or two of [vehicles, traffic signs, traffic lights] equals 0, while non-empty means the number of objects in vehicles, traffic signs and traffic lights in each data sample $\\geq 1$. Results in Random is copied from the line GPT-4o ICL-SIG in Tab. 3 of our submission.*\n| Type               | #Objects | **MLSM** |     |     |        | **SRGS** |     | **SRD (Directional)** |     |        | **SRD (Proximal)**   |     |        |\n|--------------------|:--------:|:--------:|:---:|:---:|:------:|:--------:|:---:|:---------------------:|:---:|:------:|:--------------------:|:---:|:------:|\n|                    |          | P↑       | R↑  | F1↑ | AssA↑  | S↑       | WS↑ | MAE↓                 | MSE↓| Acc↑  | MAE↓                | MSE↓| Acc↑  |\n| **Random**         | Random   | 0.592    | 0.438| 0.479 | 0.328 | 0.337    | 0.357| 1.593                | 4.094| 0.220 | 0.775               | 1.271| 0.436 |\n| **Partial‑empty**  | 3        | 0.582    | 0.398| 0.459 | 0.310 | 0.303    | 0.332| 1.981                | 5.563| 0.165 | 0.775               | 1.231| 0.419 |\n|                    | 4        | 0.562    | 0.405| 0.458 | 0.314 | 0.326    | 0.341| 1.777                | 4.667| 0.180 | 0.845               | 1.314| 0.367 |\n|                    | 5        | 0.552    | 0.393| 0.446 | 0.306 | 0.306    | 0.324| 1.839                | 5.004| 0.191 | 0.832               | 1.288| 0.375 |\n|                    | 8        | 0.571    | 0.465| 0.499 | 0.349 | 0.352    | 0.365| 1.668                | 4.340| 0.203 | 0.797               | 1.244| 0.393 |\n|                    | 10       | 0.563    | 0.457| 0.487 | 0.337 | 0.326    | 0.334| 1.671                | 4.471| 0.229 | 0.757               | 1.159| 0.429 |\n| **Non‑empty**      | 3        | 0.561    | 0.382| 0.443 | 0.300 | 0.316    | 0.331| 1.980                | 5.560| 0.155 | 0.876               | 1.442| 0.368 |\n|                    | 4        | 0.584    | 0.419| 0.476 | 0.331 | 0.340    | 0.357| 1.881                | 5.164| 0.167 | 0.760               | 1.125| 0.398 |\n|                    | 5        | 0.572    | 0.417| 0.469 | 0.321 | 0.349    | 0.361| 1.921                | 5.362| 0.184 | 0.783               | 1.236| 0.412 |\n|                    | 8        | 0.543    | 0.484| 0.499 | 0.350 | 0.362    | 0.365| 1.667                | 4.359| 0.204 | 0.848               | 1.313| 0.357 |\n|                    | 10       | 0.533    | 0.460| 0.490 | 0.339 | 0.357    | 0.362| 1.690                | 4.537| 0.212 | 0.850               | 1.467| 0.403 |\n| **Combined**       | 3+4+5    | 0.567    | 0.411| 0.464 | 0.317 | 0.346    | 0.354| 1.885                | 5.143| 0.150 | 0.824               | 1.304| 0.372 |\n|                    | 3+4+8    | 0.600    | 0.442| 0.501 | 0.350 | 0.365    | 0.380| 1.849                | 5.000| 0.171 | 0.780               | 1.241| 0.421 |\n|                    | 4+5+8    | 0.571    | 0.426| 0.477 | 0.330 | 0.363    | 0.373| 1.665                | 4.335| 0.204 | 0.732               | 1.046| 0.419 |\n|                    | 4+8+10   | 0.598    | 0.455| 0.505 | 0.353 | 0.360    | 0.374| 1.566                | 4.007| 0.229 | 0.779               | 1.253| 0.430 |\n\n**W-3 Human-like VSI offers little insights.** We apologize for omitting certain implementation details due to space constraints. \\\nOur primary reason for including human‑like VSI is to capture selective attention in driving: human drivers naturally focus on a small subset of scene elements—such as a red traffic light or a pedestrian stepping off the curb, when making safety‑critical decisions, rather than processing every object equally. Incorporating gaze‑derived saliency into SIG allows us to weight model errors according to human importance, offering a more systematic evaluation of decision‑relevant understanding, a feature currently absent in existing AD benchmarks. Our attention-based metrics (SRGS, SRD) not only assess whether a model can reconstruct spatial layouts, but also whether it can prioritize the right elements, mirroring the attention mechnism of a safe driver. This human‑centered weighting transforms SIGBench from a general VSI benchmark into one that evaluates the  model’s spatial understanding in real‑world driving contexts. \\\nWe will provide detailed homographic transformation in Appendix.\n\n**Q-1 Values of gamma and beta.** We set $$ \\gamma = \\frac{2}{|\\mathcal{V}|}, \\quad \\beta = \\frac{1}{|\\mathcal{E}|} $$\nwhere $|\\mathcal{V}|$ and $|\\mathcal{E}|$ are the numbers of nodes and edges in the ground-truch SIG, respectively. This 1) bounds and scales insertion/deletion costs across graphs, and 2) weights edge operations more heavily (2:1) than node operations to reflect their inherently greater impact on GED.\n\n**Q-2 Why not attention-based MLSM?** In designing our human‑like VSI metrics, we choose to apply gaze weighting to SRGS rather than MLSM for two key reasons: \n- Metric focus: 1) MLSM evaluates the absolute placement of each object on the grid, i.e. how closely a predicted cell coordinate matches its GT; 2) SRGS assesses the relative structure of the scene graph. Human gaze naturally highlights relational importance (\"Which objects matter most in context?\") rather than precise metric distances. By weighting SRGS, we ensure that errors among highly attended object relationships incur larger penalties, while preserving MLSM’s role as an unbiased localization check.\n- Artistic analogy: Consider an artist sketching a scene: they first lay out all object contours (the \"big picture\") before adding relational details where attention is focused. In this process, they focus on all objects inside the attention range instead of pixel-level or object-level information. Thus, we think attention-based SRGS is more aligned to the evaluation of human-like VSI.\n\n**Q-3 Why does relative ranking of MLLMs doesn't change significantly point to MLLMs are still unable to prioritize objects?** Intuitively, if a model truly understood which objects garner human attention, penalizing errors on those high‑saliency objects more heavily would cause its performance to improve relative to others, and thus alter the ranking of models under gaze‑weighted metrics. In other words, a model that prioritizes the \"right\" objects would gain a bigger boost when those objects are weighted more. Because we observe that the order of models remains essentially unchanged after applying gaze‑based weights, this implies that none of the evaluated MLLMs consistently emphasize high‑attention elements over low‑attention ones. In turn, this stability in ranking indicates that current MLLMs have yet to internalize human‑like prioritization of scene objects.\n\n**Q-4 Insights on results of ICL on SIGBench-tiny and ICL using open-source models.** The zero‑shot rankings on SIGBench and SIGBench‑Tiny are consistent: Gemini‑2.5‑Pro leads on MLSM and GPT‑4o on SRGS. Their ICL performance gains diverges significantly on SIGBench‑Tiny split. Our insight for this occurrence is that if a MLLM can determine the absolute spatial position of object correctly (revealed from metrics in MLSM), it's easy for it to improve the understanding of relative spatial relationship between objects. However, to improve the localization of objects in 3D space requires further training and fine-tuning. For open-source models, we conduct the ICL using Qwen-VL-7B and Qwen-VL-32B on SIGBench-Tiny. The following results demonstrates both models are acquire significantly improvements on MLSM and SRGS metrics, highlighting SIG's strong potential for improving VSI.  Our SIG representation can also be generalized to other domains besides AD, please refer to reviewer **wMQv W-1&Q-2**.\n\n| Models             | Type    | **MLSM** |   |   |     | **SRGS** |   | **SRD (Directional)** |   |   | **SRD (Proximal)**   |   |   |\n|--------------------|---------|:--------:|:-:|:-:|:----:|:--------:|:-:|:---------------------:|:-:|:-:|:--------------------:|:-:|:-:|\n|                    |         | P↑       | R↑ | F1↑| AssA↑| S↑       | WS↑| MAE↓                  | MSE↓| Acc↑| MAE↓                 | MSE↓| Acc↑|\n| Qwen‑VL 2.5‑7B| Z‑S     | 0.273   | 0.331 | 0.283 | 0.173 | 0.104   | 0.080 | 2.129         | 6.084  | 0.093  | 1.230           | 2.461  | 0.247  |\n|                    | ICL‑SIG | 0.470   | 0.340 | 0.371 | 0.241 | 0.216   | 0.217 | 1.691         | 4.372  | 0.204  | 1.271           | 2.525  | 0.216  |\n| Qwen‑VL 2.5‑32B| Z‑S     | 0.455    |0.369|0.397|0.265|0.232     |0.207|1.767                |4.442|0.144|1.718                 |4.000|0.115|\n|                    | ICL‑SIG | 0.569    |0.468|0.499|0.351|0.328     |0.347|2.161                 |6.120|0.100|1.447                 |3.183|0.199|"}]

Author rebuttal:

Meta-review: This paper proposes the Spatial Intelligence Grid (SIG) and SIGBench as a new framework for evaluating and improving visual-spatial intelligence in multimodal LLMs. The representation is well-motivated, the metrics are rigorous, and the benchmark is thoughtfully built with human gaze data.

Experiments on both proprietary and open-source models show consistent gains over VQA-style baselines, and the rebuttal addressed concerns on generalization and efficiency. AC thinks the work is novel, timely, and has a clear impact for safety-critical domains like autonomous driving. AC recommend acceptance.
Decision: Accept (spotlight)