Title: Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal LLM

VLM

Spatial Intelligence

Autonomous Driving
Abstract: How to integrate and verify spatial intelligence in foundation models remains an open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): a structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As a complementary channel to text, SIG provides a faithful, compositional representation of scene structure for foundation-model reasoning. Building on SIG, we derive SIG-informed evaluation metrics that quantify a model’s intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of-the-art multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as a data-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-driving scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Spatial Intelligence Grid (SIG), a new format for representing autonomous driving scenes, where the objects as well as their geometrical spatial relationships are embedded into a grid-based data schema. They create a dataset of input scenes and annotated SIGs and define the tasks of extracting its essential components, which include the birds-eye grid view of the scene featuring the objects and their locations, and the spatial relation graph (SRG) and the spatial relation paragraph (SRP), which describe the spatial relations between objects in a numeric and linguistic terms, respectively. They also define the metrics used to evaluate the model performance on these tasks, which the authors claim to more accurately reflect the VSI capabilities of models. Finally, the authors show that these representations are more effective for few-shot in context learning than the standard VQA style representations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper introduces a novel dataset, featuring a new way of representing autonomous driving scenes, which can be valuable to the research community\n- The authors evaluate a range of state-of-the art VLMs on their tasks, simultaneously demonstrating both the utility of the resulting representations by showing them to be more effective for 3-shot in context learning than standard VQA based approaches, and showing a considerable existing performance gap to humans, motivating further research\n- Rich appendix with detailed descriptions of their data collection and annotation process, as well as examples of their proposed tasks\n\nWeaknesses:\n- The authors claim to “derive a set of SIG-based optimal evaluation metrics that rigorously quantify a model’s true VSI capabilities”, and while it does make sense that these metrics point to the VSI capabilities of models, their \"optimality\" feels overstated. In general, I do not think that the work clearly demonstrates the advantage of these evaluation metrics in terms of representing the spatial intelligence of models\n- The choice of the 3 context samples could have had some degree of impact. It would have been interesting to see if this was negligible or not\n- I feel like the results on human-like VSI offer too little insights compared to how much space in the main paper was dedicated to it. On a more minor note, in some cases, less important information, such as Eq 7 in 4.1 (which is also featured in the appendix) is presented in the main paper, while on the other hand, more important details are omitted (e.g. in 3.3 where it leaves out how the correspondences between image pixels and grid cells were established when calculating the homographic matrix)\n\nQUESTIONS:\n- How are gamma and beta for weighted similarity chosen?\n- Why wasn’t gaze based weighting also applied onto the MLSM?\n- Why does the fact that the relative ranking of models doesn’t change significantly with the addition of gaze based weighting point to the conclusion that the “current MLLMs are still unable to interpret the prioritization of different objects in a scene”?\n- Any insights on why Gemini-2.5-Pro is so much better than GPT-4o on ICL on SIGBench-tiny, despite not being that much better on SIGBench itself? It might have been interesting to see the performance of other models on ICL (e.g. if it turned out an open source model was competitive), given that it appears to be not all that tied to the SIGBench performance\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present study is concerned with representing the spatial locations and relationships among static and dynamic\nobjects in Autonomous Driving (AD) environments. In the proposed graph-based representation, objects are nodes loosely\nlocated in a bird's eye view (BEV) regular grid. Relationships are encoded as edges corresponding to proximity and\nego-vehicle centric directional relations (e.g. \"car A is at the back of car B\"). Furthermore, the authors assign\nvisual saliency to objects by placing attention weights into the grid cells. \n\nThe proposed representation, termed SIG (Spatial Intelligence Graph) is employed both as ground truth and as model\noutput:\n  - **ground truth** - the authors introduce a benchmark dataset (SIGBench), consisting of approximately\n1400 frames annotated with three elements:\n    - Spatial Intelligence Graphs (SIGs)\n    - Spatial Relationship Paragraphs (SRPs) - a textual representation of the relationships in the SIGs\n    - human gaze annotations (predicted with off-the-shelf methods and back-projected from image space onto the SIGs BEV grid)\n  - **model output** - the authors prompt state-of-the-art Multimodal Large Language Models (MLMMs) with:\n    - road scenes, asking to predict SIGs\n    - road scenes, asking to perform SRP preposition completion\n    - a temporal history of frames and saliency maps, with instructions to predict the visual saliency grid values for\nthe current frame.\n\nThree metrics are used:\n - Multi-Level Spatial Matching: to capture the qualitative object localization/classification\n - Spatial Relationship Graph Similarity: to capture overall graph structure\n - Semantic Relational Distance: to capture the fidelity of particular inter-object proximal/directional relations\n\nThe authors conduct experiments involving both open-source and proprietary MLLMs. They mainly find that:\n - the selected proprietary MLLMs out-perform the open-source MLLMs on SIG an SRP prediction\n - spatial intelligence can be significantly improved by in-context learning with a random set of (image, SIG pairs) as\nprompts\n - MLLMs cannot effectively predict human visual saliency in SIG space\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n  - Well written paper and clearly structured work\n  - High importance, as spatial scene understanding is an outstanding issue in MLLMs applied to AD, especially with\nthe emergence of end-to-end driving\n  - Extensive evaluation \n\nWeaknesses:\n\n - There is no human baseline for the proposed metrics (e.g. based on consistency between human subjects). It is\ndifficult to judge how far the SOTA spatial intelligence level really is compared to the gold standard of human drivers\n - The study omits lanes from the evaluation (lines 130-131). However, proper understanding of the lane and drivable\nspace structure are crucial for AD. Furthermore, by omitting lanes, the study misses on the opportunity to model\nhigher level driving scene spatial layouts, such as \"vehicle 1 on the left lane\" or \"traffic sign 1 corresponds to the\nego lane\". Such relationship are of primary importance for AD (consider overtaking or stopping for a red light).\n - Dynamic spatial relationships between objects (e.g. vehicles approaching a pedestrian crossing or accelerating\ntowards the ego vehicle) are not covered \n - The BEV grid resolution is somewhat coarse (10x10), and the authors do not provide an intuition on how to calibrate\nit\n - The saliency prediction prompt seems to ask the MLLM to generate python code for producing the attention map. It is\nnot clear to what extent this setup affects the outcome.\n\nQUESTIONS:\n- Does the SIGBench dataset also preserve object IDs? Would it allow for tracking dynamic objects across individual\nSIG frames?\n- The model and benchmark seem to rely on single-camera and single frame information. This makes depth prediction\nambiguous up to scale. Have the authors also experimented with scale-invariant matching against the BEV locations?\n- The grid resolution seems to be critical for the evaluation metrics. How do the authors decide on the value to\nuse?\n\nLIMITATIONS:\nThis reviewer thinks the authors should point out that the method:\n - relies on predicted human visual saliency, hence suffers from systematic prediction errors induced by the respective methods.\n - does not consider lanes and their relationships to the objects in the driving environment, which may\n limit applicability for AD", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel grid-based representation, Spatial Intelligence Grid (SIG), for encoding Visual-Spatial Intelligence (VSI) in multimodal large language models (MLLMs), particularly in the context of autonomous driving (AD). The authors argue that existing Visual Question Answering (VQA)-style representations inadequately capture spatial reasoning.\nThey introduce SIG, a structured grid-based scene representation that embeds spatial and physical priors, along with a new benchmark, SIGBench, comprising over 1,400 annotated driving frames with human gaze data and ground-truth SIG labels.\nExtensive experiments on several state-of-the-art MLLMs demonstrate that SIG-based few-shot in-context learning (ICL) improves performance in spatial reasoning tasks compared to traditional VQA-style prompts. Additionally, SIG enables human-like VSI assessment through attention-modulated metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written, clearly structured, and effectively presents its main contributions. The integration of figures with the textual explanations is useful, providing clarity and improving the reader’s understanding of the proposed method.\n- SIGBench is thoughtfully designed for both grid-based and human-like VSI evaluation, incorporating gaze-based annotations and rigorous, graph-theoretic evaluation metrics. The proposed metrics (MLSM, SRGS, and SRD) are particularly well-motivated, offering a structured and interpretable means of quantifying spatial reasoning performance beyond simple right-wrong QA scoring. They allow for fine-grained assessment of relational and positional errors, which is especially valuable in complex environments like autonomous driving.\n- The authors extensively benchmark multiple proprietary and open-source MLLMs, providing strong empirical evidence that SIG consistently improves spatial reasoning in both zero-shot and in-context learning setups.\n- The discovery that current MLLMs fail to appropriately prioritize objects based on attentional salience, as humans naturally do in driving tasks, is a valuable insight. Moreover, the choice of autonomous driving as a testbed is appropriate given its real-world complexity and practical relevance.\n\nWeaknesses:\n- While autonomous driving is an appropriate and demanding testbed, the paper does not explore how SIG might generalize to other domains. While this aspect is being discussed briefly in the Appendix I think it still is worth noting.\n- The paper introduces graph-edit distance and semantic relational distance metrics, but it does not address their computational costs during large-scale inference. Given that the target application is autonomous driving, where real-time efficiency is crucial, an analysis of runtime performance of the pipeline would be valuable.\n- The paper would benefit from a qualitative or quantitative analysis of typical failure modes, such as common errors in SRGS predictions or attention misallocations. This could help identify systematic weaknesses in current models and inform future improvements.\n- The primary experimental advantage demonstrated stems from in-context learning (ICL) with SIG, while no experiments on full fine-tuning or reinforcement learning with human feedback (RLHF) are presented. This leaves open the important question of how effectively SIG could be integrated into end-to-end training pipelines for production systems.\n\nOther issues not affecting the rating:\n- Typo at line 120\n- Line 254 -> Pearson\n\nQUESTIONS:\n- How well would SIG generalize to domains beyond autonomous driving? Could the authors clarify or expand on this?\n- What is the runtime overhead of the proposed graph-based metrics, and are they practical for real-time applications like autonomous driving?\n- Can the authors provide an analysis of common failure cases to better understand model limitations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach and benchmark for evaluation of visual spatial intelligence of MLLMs.\nThe authors introduce spatial grid based methods  for assessing spatial relationships in Autonomous Driving scenarios and evaluate whether current MLLMs  can answer spatial queries, generate the spatial grid from images using  in-context learning. The gaze of human differers is examined in comparison with  machine based approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- novel benchmark for map style representation for assessing spatial relationships \n- novel evaluation metric, using spatial relation graph similarity, multi-level spatial matching \n\n\nWeakness \n- this is likely applicable only Autonomous driving scenarios \n- the positioning of the work with respect to related works could be improved, \n- the method is effectively a map based methods, see [1] for additional references\n- better motivation for new evaluation metrics should be provided\n- the connection to human gaze evaluation is weak\n\nThe paper is not well organized. The discussion the evaluation metrics precedes the discussion of grid computation and the benchmark, there are very few examples of prompts, so it is hard to address the complexity, strengths and weaknesses of the method. The approach suggests to use in context learning  to obtain the spatial grid json format, which is now well aligned with the benchmarks where the models reason \nabout spatial relations in Bird Eye View map. \n\nPlease look at this paper and references within, \n[1] https://arxiv.org/pdf/2410.06468\nDOES SPATIAL COGNITION EMERGE IN FRONTIER\nMODELS? Santhosh Kumar Ramakrishnan∗ Erik Wijmans Philipp Krahenb ¨ uhl Vladlen Koltun\n\nQUESTIONS:\n- Provide examples of queries and how they differ for the queries in other benchmarks such as SpatialVLM or VSR benchmarks and how they are obtained for SIG\n-  How does the approach extend to other scenarios beyond Autonomous Driving\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}