Title: Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spatial intelligence

Multimodal large language model

3D Understanding
Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a 3D spatial encoder—initialized from the backbone of the visual geometry model—to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct a training dataset from multiple sources and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a question/answering task in video which requires 3D spatial understanding to be able to solve these questions. Their main innovation is to combine a 3D visual encoder (VGGT) with the common 2D visual encoder (Qwen2.5-VL-3B) and demonstrate that adding 3D information significantly improves performance.\nMore specifically, this paper first samples frames from a video uniformly and encodes these using a frozen VGGT encoder to get 3D features. Then they create a subselection of these frames such that the maximum number of voxels of the 3D space is covered by these features (Sec. 3.2). Given this subset, they keep corresponding 3D VGGT features and extract Gwen2.5 features from these frames. On top of this they train a 'connector' which fuses these features and they also finetune an existing LLM (text tower) on the training set for answering these questions. Additionally, they use the GRPO RL-strategy from DeepSeek to boost performance. Finally, they annotate extra data closely following the protocol of VSI-Bench [18].\nResults on VSI-Bench, ScanQA, and SQA3D convincingly demonstrate the benefits of using 3D information. Their system outperforms open-source baselines fine-tuned on the same data (e.g. Qwen2.5). They also outperform proprietary models like Gemini 1.5 Pro and GPT4o on VSI-Bench\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* Using 3D information makes sense and this is a convincing implementation.\n* Very good results on 3 different datasets.\n* Well-written\n* There is a reasonable ablation.\n\nMain weaknesses\n* Video-3D-LLM outperforms the current method when a ground-truth input map is fed to the model. What happens when this model is fed an estimate depth map from, say, DepthAnything V2?\n* Since the authors annotate extra data, I would expect a small ablation of the influence of the training data on the performance.\n\nWeaknesses (minor)\n* IMHO Spatial-MLLM-120k is rather misleading to present as a single dataset. In fact, it combines ScanQA, SQA3D, and new data closely following the VSI-Bench data curation pipeline [18]. Why not present it as three datasets and write you train on a mixture? That sets expectations more fairly.\n* The additional annotated data is for training only (which is fine, since is is also not explicitly mentioned as a contributions, but it should be more clearly communicated when it is introduced to set the expectations of the reader).\n* To me it was unclear that 'spatial intelligence' really means 3D. Many papers explicitly consider 2D spatial relations so I would advice to just call it 3D spatial intelligence to avoid any ambiguity. Or at the very minimum, 'spatial intelligence' should be defined better in the second sentence of the abstract.\n* Fig 2: please consider adding freeze icons to the 2D and 3D spatial encoder. And consider calling the 'Spatial Encoder' the '3D Spatial Encoder'.\n* The connector was a bit unclear to me. If I understand correctly the method only works if the 2D encoder and the 3D encoder extract exactly the same number of tokens from (almost) the same locations within the video. Then the combination happens on a per-token basis by projecting each token to the same number of dimensions and adding them. Please improve clarity on this section.\n* VSI-Bench also uses ScanNet. Therefore it is unclear where VSI-Bench ends and the extra data of this paper begins. This should be clarified in a final version of this manuscript.\n* The experiments on Gemini Pro result in many more frames per video to be analysed. But even though Gemini is advertised to be long-context, the extra number of frames may actually hurt performance. The authors could consider testing that.\n\nQUESTIONS:\nSee the two main weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

comments: SUMMARY:
This paper presents the Spatial-MLLM framework for visual-based spatial reasoning using purely 2D observations, aiming to enhance spatial intelligence. The authors propose a dual-encoder architecture to separately encode 2D semantic features and 3D structural features. During inference, a space-aware frame sampling strategy is employed to select spatially informative frames, improving the efficiency and accuracy of spatial understanding.

STRENGTHS AND WEAKNESSES:
Strengths:
This is a well-executed study with impressive performance results. The experiments convincingly demonstrate the effectiveness of the Spatial-MLLM-120K dataset, the proposed architecture, and the GRPO method across multiple spatial reasoning tasks.

Weaknesses:
The proposed dual-encoder architecture increases the complexity of the multi-modal large language model. Given that the vision encoder in Qwen2.5-VL already supports 3D patches, incorporating additional spatial information through VGGT further increases the computational cost, which may limit scalability or real-time applicability.

QUESTIONS:
1. The MLP connector is introduced into the pretrained Qwen-VL model; however, it is unclear whether the 120K dataset is sufficient to effectively train this module. Additionally, the trainable parameters in the overall model is not clearly reported.

2. It is not explicitly stated which variant of Qwen2.5-VL the Spatial-MLLM is built upon. The parameter size of the backbone model is also missing from the tables. Providing this information would improve the transparency and reproducibility of the work.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Spatial-MLLM, a framework designed to improve the spatial reasoning capabilities of Multimodal Large Language Models using only 2D video inputs. The core contribution is a dual-encoder architecture that combines a standard 2D visual encoder for semantic understanding with a novel spatial encoder, initialized from a visual geometry foundation model, to extract 3D structural information. These complementary features are fused by a lightweight connector before being processed by the LLM backbone. Experiments show that this approach achieves state-of-the-art performance on several visual-based spatial reasoning benchmarks, outperforming existing open-source and proprietary models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper tackles the important and challenging problem of improving spatial intelligence in MLLMs from purely 2D video inputs, a key limitation of current models that are often optimized for semantic rather than structural understanding.\n- The authors introduce a new large-scale dataset, Spatial-MLLM-120K, which is a valuable asset for the community. \n\nWeaknesses\n- The approach relies heavily on combining existing, powerful pretrained models (Qwen2.5-VL and VGGT) with a very simple fusion mechanism (MLP-based addition). While effective, the novelty lies in the combination rather than a fundamental new technique, which might be seen as an incremental contribution.\n- While the paper avoids the need for 3D data inputs, comprehensively understanding a 3D object from 2D data requires many frames. The proposed sampling strategy highlights this by pre-processing a large number of frames (128) to select a small subset (16). The paper is missing a critical analysis of the total computational load and efficiency of this approach compared to models that use 3D data directly.\n\nQUESTIONS:\nSee the weakness section above.\n\nLIMITATIONS:\nSee the weakness section above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper introduces a novel framework, Spatial-MLLM, aimed at enhancing the spatial intelligence of existing Multimodal Large Language Models (MLLMs) using purely 2D visual inputs. The paper proposes a dual-encoder architecture that integrates a 2D visual encoder for semantic feature extraction and a spatial encoder for 3D structure feature extraction. Additionally, the paper presents a space-aware frame sampling strategy to select the most spatially informative frames under input constraints.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. The paper presents a technically sound approach with a dual-encoder architecture and space-aware frame sampling strategy. The results are supported by extensive experiments.
2. The paper is well-organized and clearly written, with detailed explanations and diagrams that aid understanding.


**Weaknesses**
1. The paper lacks a detailed comparison of the computational efficiency of Spatial-MLLM versus 3D/2.5D-Input Models like Video-3D LLM and Video-Input Models. 
2. Testing on other benchmarks like MMScan QA and OpenEQA used in LLaVA-3D would better demonstrate its generalizability.
3. The space-aware frame sampling is applied after uniform sampling. Testing it directly on original video frames could show its robustness and effectiveness under different conditions.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03