GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
Authors: Zhangyang Qi
Zhixiong Zhang
Ye Fang
Jiaqi Wang
Hengshuang Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Scene Understanding
Vision Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In recent years, 2D Vision-Language Models (VLMs) have made significant strides in image-text understanding tasks. However, their performance in 3D spatial comprehension, which is critical for embodied intelligence, remains limited. Recent advances have leveraged 3D point clouds and multi-view images as inputs, yielding promising results. However, we propose exploring a purely vision-based solution inspired by human perception, which merely relies on visual cues for 3D spatial understanding. This paper empirically investigates the limitations of VLMs in 3D spatial knowledge, revealing that their primary shortcoming lies in the lack of global-local correspondence between the scene and individual frames. To address this, we introduce GPT4Scene, a novel visual prompting paradigm in VLM training and inference that helps build the global-local relationship, significantly improving the 3D spatial understanding of indoor scenes. Specifically, GPT4Scene constructs a Bird's Eye View (BEV) image from the video and marks consistent object IDs across both frames and the BEV image. The model then inputs the concatenated BEV image and video frames with markers. In zero-shot evaluations, GPT4Scene improves performance over closed-source VLMs like GPT-4o. Additionally, we prepare a processed video dataset consisting of 165K text annotation to fine-tune open-source VLMs, achieving state-of-the-art performance on all 3D understanding tasks. Surprisingly, after training with the GPT4Scene paradigm, VLMs consistently improve during inference, even without object marker prompting and BEV image as explicit correspondence. It demonstrates that the proposed paradigm helps VLMs develop an intrinsic ability to understand 3D scenes, which paves the way for a seamless approach to extending VLMs for 3D scene understanding.

Published as a conference paper at ICLR 2026
GPT4SCENE:
UNDERSTAND 3D SCENES FROM VIDEOS WITH
VISION-LANGUAGE MODELS
Zhangyang Qi', Zhixiong Zhang", Ye Fang", Jiaqi Wang"' Hengshuang Zhao
I The University of Hong Kong. *Shanghai Al Lab
Corresponding Author
(zyqi, hww.ahsahheeccoooo 3cs. hku. hk, {zhangzhixiong, fangye, wangjiaqi .rg..rrggooo cn
ABSTRACT
In recent years, 2D Vision-Language Models (VLMs) have made significant strides in image-text understanding tasks. However, their performance in 3D spa- tial comprehension, which critical for embodied intelligence, remains limited. Recent advances have leveraged 3D point clouds and multi-view images as inputs, yielding promising results. However, we propose exploring a purely vision-based solution inspired by human perception, which merely relies on visual cues for 3D spatial understanding. This paper empirically investigates the limitations VLMs in 3D spatial knowledge, revealing that their primary shortcoming lies in the lack of global-local correspondence between the scene and individual frames. To address this, we introduce GPT4Scene, novel visual prompting paradigm in VLM training and inference that helps build the global-local relationship, signif- icantly improving the 3D spatial understanding of indoor scenes. Specifically, GPT4Scene constructs a Bird's Eye View (BEV) image from the video and marks consistent object IDs across both frames and the BEV image. The model then inputs the concatenated BEV image and video frames with markers. In zero- shot evaluations, GPT4Scene improves performance over closed-source VLMM like GPT-4o. Additionally, we prepare processed video dataset consisting of 165K text annotation to fine-tune open-source VLMs, achieving state-of-the-an performance on all 3D understanding tasks. Surprisingly, after training with the GPT4Scene paradigm, VLMs consistently improve during inference, even without object marker prompting and BEV image explicit correspondence. It demon- strates that the proposed paradigm helps VLMs develop an intrinsic ability to un- derstand 3D scenes, which paves the way for seamless approach extending VLMMs for 3D scene understanding.
INTRODUCTION
3D scene understanding aims to comprehend the overall layout of the surrounding complex en- vironments and the spatial relationships between objects Azuma et al. (2022); Chen et al. (2021; 2020). It plays crucial role in applications such as embodied intelligence, virtual reality, and smart cities Huang et al. (2024c); Chen et al. (2024b); Chandrasegaran et al. (2024). With the rapid de- velopment of Large Language Models Team (2024c); Team & DeepMind (2024); OpenAI (2023); Team (2024d:b), Vision-Language Models have demonstrated impressive performance in image and video understanding Liu et al. (2023; 2024a); Li et al. (2025a); Zhu et al. (2024a); Wang et al. (2025b). Researchers have extended this paradigm to 3D perception by incorporating point clouds, aiming to improve scene understanding Hong et al. (2023); Chen et al. (2024b); Wang et al. (2025c); Fu et al. (2025b); Man al. (2024); Huang et al. (2024c;a); Kang et al. (2025a).
Recent 3D point LLMs leverage point clouds aligned with LLM features for indoor scene under- standing Chen al. (2024b); Hong et al. (2023). While combining point clouds and images Zhu et al. (2024b); Chen et al. (2024b); Huang et al. (2024a) in Point- Vision-LLMM paradigms improves performance through richer visual cues, aligning these modalities with text remains challenging.

---

Published as a conference paper at ICLR 2026
Qucstion Amseriing
Dease Caption
Cronddigg
Yisaal
Viusl (extirelje)))
olngheabjict)
calor
waskes siud apuieed woll
it Object
Object 42, 26, 2.
Large Language Model
Large Language Model
Large Language Model
Text Encoder
Rncoder
Exceder
Viezal Encoder
Poins Eacader
Visual Escoder
Text Imput
Point Cloud
Text Iegus
Malli-siw Insquee
Palne Clond
(lofernnee
Video
3D Point I.LMs
(b) 3D Point LLMs with visual encoder
(c) GPT44eene (Ours)
Figure 1: Overview of GPT4Scene. GPT4Scene understands 3D scenes and performs tasks like 3D question answering, dense captioning, and visual grounding using only vidco input. Unlike 3D point LLMs, relies solely on vision, with global context provided by BEV image.
This complexity motivates exploration of vision-only solutions, inspired by humans' natural ability to perceive 3D environments without point clouds, offering a direction for efficient scene analysis.
In this work, we aim leverage pre-trained VLMs without modifying their architecture, maximizing their vision perception capabilities. However, their effectiveness in understanding immersive 3D indoor scenes remains limited. Our analysis shows that directly inputting scene videos into VLMs fails in 3D scene understanding due to two factors: i) the lack of global scene representation, ii) misalignment between per-frame local observations and their spatial-temporaa context.
To address this, we propose GPT4Scene, framework that enhances VLMs' spatial understanding (see Figure 1). We first perform reconstruction on input videos to generate Bird's Eye View (BEV) image, offering comprehensive scene layout. Additionally, Object markers (STO-markers) both the BEV image and 2D frames. These markers maintain consistent object IDs across frames (temporal level) and align with the BEV image (spatial level), bridging the global-local relationship. Empirical results show that GPT4Scene remains robust to reconstruction quality and marker accuracy, it prioritizes learning global-local correspondences over precise geometric reconstructions.
introduce Spatial-Temporal
We first explored the effectiveness of GPT4Scene under training-free approach. Experimen- tal results revealed that it was notably effective for powerful large-scale VLMs such as Qwen2- VL-72B Wang al. (2025b), well as closed-source models like GPT-4o OpenAl (2024) and Gemini- 1.5-Pro Team (2024a), significantly enhancing their 3D scene understanding capabilities, even reaching levels comparable to previous state-of-the-aa point- point-based methods. However, the im- provements were limited on smaller- models. For smaller
vision-language models (VLMs),, we introduce ScanAlign, multimodal dataset comprising 165K aligned data pairs fea- turing 3T0-mmaker-annotteee video frames, BEV images, and textual descriptions. Fine-tuning DLe72-VL-7B on ScanAlign achieves state-of-the-at performance, showing a substantial rela- tive improvement on 3D question answering (SQA3D) and significantly surpassing the previous SOTA, Chat-scene. The model demonstrates even stronger 3D visual grounding capabilities on Multi3DRef, where it also considerably outperforms Chat-scene. These advancements underscore GPT4Scene's effectiveness in seamlessly enhancing VLMs with strong 3D spatial understanding.
Because of the page limit, the related work is shown in Section F.
2

---

Published as a conference paper at ICLR 2026
Zog
Dry
Partition
Visual Encoder & Large Language Model
Project
Teg
& Mark
Selected Fromes
Selected Frames labels y
Instonce Seg Detect
Reconstruct
Spetial Temponal
Mark
Object
3b Point
Top View
Input Video V
BEV Image ΣT
BEV Image lobels 22
The framework of GPT4Scene. seene video is processed by sampling frames, reconstructing a point cloud, and generating BEV image. Object locations are detected from the point cloud and projected onto the vidco frames. The resulting frames and BEV image, enhanced with STO-markers, inputs for Large Language Model (VLM) training and inference.
Figure 2:
Our paper makes these major contributions:
We introduce GPT4Scene, framework that enhances Vision-Lanuuage Models (VLMs) to comprehend 3D scenes directly from pure vision input.
3D BEV image with global context information, and ii) Spatial-Temporal Object markers (STO-markers) for spatial and temporal consistency across BEV image and video frames.
We introduce two techniques:
We introduce ScanAlign, a dataset comprising video frames, BEV images with STO- markers, and annotations. Fine-tuning VLMs on this dataset significantly improves 3D understanding, even with raw video inputs.
GPT4Scene demonstrates robust performance in zero-shot and fine-tuning settings, achiev- ing SOTA results across diverse 3D seene understanding tasks.
METHODOLOGY
We introduce the framework as shown Subsection 2.1, which improves 3D scene understanding using video inputs. We also explore how zero-shot prompts unlock the potential of large-scale VLMs in Subsection 2.2 and apply fine-tuning for enhanced 3D understanding in Subsection 2.3.
2.1 GPT4SCENE FRAMEWORK
Overview. Here introduce GPT4Scene's architecture. As shown in fig. our framework com- bines global scene layouts with object localization details to enhance VLMs' 3D understanding. Given an input video sequence V (11 ......... IN} captured during indoor scene traversal, first reconstruct the 3D scene from the complete video sequence, simultaneously generating a Bird's- Eye View (BEV) map and 3D instance segmentation masks. To establish ject markers (STO-markee markers), we perform uniform frame sampling by selecting n frames at indices: (I,...,,) n) yielding the sampled subset V {T_,..... ........ J. ). The 3D instance segmentation results are then projected onto both the BEV map and these keyframes to enable cross-view object localization.
Ob-
8; (i - | + 1,
Global information: 3D BEV map. Egocentric videos lack global context, which we address by reconstructing the 3D scene into point cloud and rendering bird's-eye view (BEV) image for holistic VLM understanding. Given video V {II. Ig.... In and camera extrinsics & = F1, F,,...; Ex), we use 3D reconstruction to generate meshes and point clouds:
P P R. (((Ts, N.)
3

---

Published as a conference paper at ICLR 2026
R(-) denotes the reconstruction process, assuming known camera intrinsics. We then generate BEV image from the global point cloud:
I. = T(P, Etop)
(2)
Here, Emp SE(3) represents the camera extrinsic for the top-down view, and T()) renders the BEV image. We provide global 3D information to VLMs as top-view images rather than points.
Local correspondence: STO-markers. To help VLMs focus specific objects. we introduce Spatial-Temporal Object markers (STO-markers), ensuring consistency between 2D frames and the 3D BEV image. From the 3D point cloud P reconstructed from the video V, apply 3D instance segmentation (e.g., M) to obtain instance masks M = {M1, M2....... MR) }. where K is the to- tal number of objects. For the BEV image In. we project the 3D segmentation masks M onto the ay plane and extract the center coordinates of the bounding box, C = { C2 ,.... [."".. These coordinates are then overlaid onto the BEV image. For selected egocentric video frames V the 3D instance masks M are projected onto each frame based on its camera pose. For each frame i, we extract the centroid of the 2D mask for each object as its 2D marker, C. = [C ,,2 1.2..... Crue ) where Che is the 2D marker for the k-th object in the i-th frame. The processed frames and BEV image with markers are defined as follows:
- (F CTa)) 81, 82,88,....,881 8n), F CRy)
Here, F(-) denotes the STO-marker projection operator that jointly augments both the sampled video frames y* and BEV image Ia. producing the marked versions V" and Ty respectively. The 2D markers in each frame and their corresponding 3D markers in the BEV image are spatially aligned, ensuring consistency across frames at the object level for temporal coherence.
2.2 THE EFFECTIVENESS OF GPT4SCENE IN A ZERO-SHOT SETTING
Here, 'zero-shot' refers the process where we take the original video and, through 3D reconstruction and segmentation, generate marked Bird's-Eye View (BEV) image and project these markers back onto the 2D video frames. This results in video with STO- Markers. Subsequently, the processed video is fed into the VLM to execute the Ques- tion Answering task. As shown in Table 1, approach had limited impact smaller models but significantly improved performance for large-scale ones (including the Qwen2-VL- 72B, GPT-4o, and Gemini-2.5 Pro), matching the Chat-scene Huang et al. (2024a). These findings indicate the limitations of smaller Vision-Language Models (VLMs) in 3D scene understanding with zero-shot prompting alone, which motivated us to propose the ScanAlign dataset and fine-tune open-source models for enhanced performance.
Table Video + GPT4Scene Inference without Fine-tuning.
The zero-shot capability GPT4Scene.
ROUGEGScuaQA EM- 10QAAD
Zero-shot 3D QA
VID
+4Scene
VID +5Seme
3D L.M Based Model
Pre SOTA
Pro SOTA
Chall-seznc Huang et al. (2024k)
4166
54.6
Open- counced VLM Based Model
Qwen2-VL.e IB Wang (2025h)
29.5
30.0
36.9
36.2
Qwen2. VL-BB Wang (2025h)
33.2
42.1
43.1.ra
Qwen2-VL-22B VLBB2B Wang et al. (2025b)
32.1
35.1 Amo
41.5
44.0.25
Closed soweeed VLM Bused Model
GPT-4o OpenAl (2024)
34,2
39.3
42.0
44.8 .28
Gemini-1 1.5--Po Team (2024a)
35.1
39.4...
43.5
46.0.2s
In a zero-shot setting, the model must create a global-local understanding of a 3D scene by fus- ing local 2D frame features with global BEV (Bird's-Eye View) features. Smaller models, such as Qwen2-VL-2B and Qwen2-VL-7B, are constrained by their parameter scale (typically in the low billions). Their visual encoders and cross- cross-modal fusion capabilities are consequently weaker. They struggle to fully parse the global layout logic from the BEV image (e.g., the relative positions of objects, spatial boundaries) and cannot precisely match the consistency of STO-markers across dif- ferent frames and the BEV. This leads to a breakdown in the "global-local association." For instance, Results show that the small model Qwen2-VL-2B's zero-shot score actually dropped, a clear sign of its "capability overload. In contrast, large-scale models like Qwen2-VL-72B and GPT-4o possess the architectural complexity to inherently grasp these feature associations, allowing them form preliminary 3D cognitive understanding even without any fine-tuning.
4

---

Published as a conference paper at ICLR 2026
2.3 ENHANCING VLMs WITH SCANALIGN FINE-TUNING
The analysis in the previous section indicates that while zero-shot prompting activates spatial aware- ness in large-scale models. smaller open-source VLMs require explicit fine-tuning to acquire this ca- pability. To this end, we constructed Alann a large-scale instruction-tuniie dataset designed to empower 2D VLMs with 3D understanding.
Table 2: ScanAlign: Datasets used for training GPT4Scene (Supervised Fine-Tuning Tuning),
Sourve
Data Type
Task Type
Samples
Ovenall Level
Spatiel Relationship
SaaRQA
3D QA
object Actribute
26K
Exiatence Count ing
Egocentrio Spatial
Distinct from raw data collection, ScanAlign uni- fies and transforms existing high-quality 3D anno- tations into novel visual prompting format. We aggregate 165K text-scene pairs from five main- stream benchmarks based ScanNet Dai et al. 2017a)-Scan(2 Azuma et al. (2022), SQA3D Ma al. (2023), Scan2Cap Chen al. (2021), ScanRe- fer Chen et al. (2020), and Multi3DRer Zhang et al. (2023c). While the text annotations are derived from these prior works, our core contribution lies in refor- matting these purely geometric data entries (point clouds) into a VI.--copaaible format (V* Th 7). Specifically, we augment the original labels by generating synchronized egocentric video frames and projecting mettaal-Temprrll Object (STO) markers onto both frames and BEV images.
SQA3D
QA
Navigation cb dbject Actribute
Total
52K
Object-Level
Objact Attributes
ScanZcap
3D Caption
Spatial Location
35K
State or Affordance
Combined Attributes
ScanRefer 3D Grounding
Spat.ial Relations
41K
Uniiue Attrihotes
Group Reference
MalliDDee 3D Grounding
List-paased Rater
35K
Neistional Refer
EXPERIMENTS
In this section, we primarily present the experimental results. First, we outline the training details in Subsection 3.1, and subsequently introduce the fine-tuning main results in Subsection 3.2. Finally, Subsection 3.3 details the ablation study, demonstrating the effectiveness of individual components.
3.1 IMPLEMENTATION DETAILS
Our 3D scene understanding benchmark is based the ScanNet Dai et al. (2017a) dataset and includes three tasks: 3D question answering (ScanQA Azuma et al. (2022) and SQA3D Ma et al. (2023)), 3D dense captioning (Scan2Cap Chen al. (2021)), and 3D visual grounding (ScanRe- fer Chen et al. (2020) and Multi3DRef Zhang et al. (2023c)). Our pipeline begins with 3D scene reconstruction using BundleFusion Dai et al. (2017b) generate dense point clouds, followed by 3D instance segmentation with Mask3D Schult et al. (2023) to extract object-level masks. The segmented 3D objects are then projected to BEV coordinates, while their corresponding masks are mapped to 2D image frames to establish sive processing framework enables precise cross-modal alignment essential for both dense caption- ing and visual grounding tasks. For the experiment, we sample N=32 frames per video x122499 resolution) for all models. Training is done for one epoch with a base learning rate 5e-6 and cosine annealing, completing in about 6 hours on 8 A100 GPUs.
Object (STO) markers. This comprehen-
3.2 FINE-TUNING MAIN RESULTS
We performed fine-tuning exclusively on ScanAlign: using GPT4Scene as the dataset, we fine-tuned Intern VI-3 8B Chen et al. (2024c), Qwen2-VI. 7B Wang et al. (2025b), and Qwen2.5-VL. 8B Team (2024d), and ultimately achieved state-of-tte-art (SOTA) results.
3D question answering. The 3D question answering results presented in Table 3 are categorized into three distinct groups: (1) conventional task-specific models, (2) architectures based on 3D point cloud LLMs, and (3) vision-language multimodal LLM systems. Our experimental analysis demon- strates that the baseline Qwen2-VL. VL-7B model without fine-tuning shows constrained capability in 3D QA scenarios. In terms of specific metrics, models fine-tuned using the GPT4Scene framework (based on the ScanAlign dataset) show outstanding performance: Qwen2-VL-7B (GPT4Scene) achieves BLEU-- score of 44.4 and CIDEr score of 96.3 the ScanQA dataset, along with
5

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation of 3D question answer on ScanQA Azuma et al. (2022) & SQA3D Ma et al. (2023).
Answering
Point Encoder
Vision Encoder
ScauQA (val)
SQA3D (val)
Task Specific Mode!
ScanQA Azuma al. (2022)
30.2
33.3
SQA3D (2023)
3D-VLP (2023)
3D-Vista Zhu al. (2023)
13.9
35.7
48.5
30 LLM Bosed Model
Chat-3D Wang (2025c)
Chat-3D Huang et (2024))
38.4
40.1
77.1
3D-LLM Hong et (2023)
39.3
14.5
35.7
69.4
LL3DA Chen al. (2024b)
13.5
15.9
37.3
76.8
PQ3D Zhu et al. (2024b)
47.1
LEO Huang et al. (2024c)
11.5
16.2
39.3
80.0
50.0
52.4
Groinded.3--1. -D-. Chen et all (2025)
13.4
72.7
Char-soene Huang (2024g)
43.2
14.3
18.0
41.6
87.7
54.6
57.5
ROSS3D Wang (2025a)
49.2
17.9
20.9
50.7
107.0
63.0
65.7
Vision I.M Based Model
M. Pu al. (2025h)
43.6
12.0
16.6
40.0
80.0
54.2
LLaVA-3D Zhu al. (2025)
14.5
20.7
50.1
91.7
55.6
Video-3D-LLM Zheng (2025)
47.1
16.2
19.8
102.1
58.6
LL3-8 (GPT48ene)
45.1
16.2
19.5
47.8
61.9
64.5
Qwen2: (GPT4Scene)
63.3
Qwen2.5-VL-77 (GPT4Scene)
an EM-1 score 60.6 and an EM-RI score of 63.3 on the SQA3D dataset; The more powerful Qwen2.5-VL-7B (GPT4Scene) further improves the BLEU-1 score to 49.2 and the CIDEr score to 105.7 on ScanQA, while reaching an EM-1 score of 63.5 and an EM-R1 score of 66.2 on SQA3D. These models not only significantly outperform the untuned baseline VLMs but also comprehen- sively outperform the previous SOTA models in the 3D point cloud LLM category (e.g., Chat-scene). This fully demonstrates that GPT4Scene- providing global context through BEV --yee View) images and establishing spatiotemporal consistency via STO-markers-can effectively en- hance the 3D spatial reasoning and QA capabilities of VLMs.
3D dense caption. Table presents the primary evaluation for the 3D dense captioning task on the Scan2Cap benchmark. The table contrasts the performance of various models using BLEU- 4 and ROUGE scores, evaluated at both a lenient (1oU@0.25) and strict (IoU@0.5) matching threshold. The key objective is demonstrate the efficacy of GPT4Scene framework in en- hancing the capability of Vision Language Models (VLMs) for detailed 3D scene description. Our oer4Ssene-integraaee Visual LLMs, including Qwen2-VL-7B (GPT4Scene) and Qwen2.5-VL-7B (GPT4Scene), achieve breakthrough performance using only visual inputs (video + BEV images). Specifically, Owen2.5-VL--7. (GPT4Scene) scores BLEU-4 of 45.9 and a ROUGE of 67.9 loU@0.22 and a BLEU-4 of 44.1 with ROUGE of 67.1 at loU@0.5. These results comprehen- sively outperform the other model classes. This superior performance stems from GPT4Scene's ability to provide global scene context via BEV imagery and maintain spatio-temporal consistency through STO-markers. Consequently, the VLM can precisely ground textual descriptions in 3D scene details without requiring 3D point cloud data, thereby establishing new ffute-of-the-at (SOTA) in the field.
3D visual grounding. Table 5 presents the critical evaluation for the 3D visual grounding task, which aims to accurately align textual descriptions with their corresponding objects in 3D space. Using the ScanRefer (single/multi-object grounding) and Multi3DRef (combinatorial multi-object grounding) benchmarks, this analysis validates the enhancement of spatial localization capabilities in Vision Language Models (VLMs) by GPT4Scene framework. Performance is measured by localization accuracy (Ace@0.2) 0.25 and A.c@0.5) on ScanRefer and the global Fl score (all F1 @0.25 and all F1 @0.5) Multi3DRef. Vision LLMS integrated with GPT4Scene, such Intern VI.3-

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation of 3D dense caption on Scan2Cap Chen et al. (2021). Our results outper- form those existing 3D LLM based models.
Table 5: Evaluation of 3D visual ground- ing on ScanRefer Chen et al. (2020) and Multi3DRef Zhang et al. (2023c). Our method reaches state-of state-of-the-at art performance over all methods for the 3D visual grounding task.
Dense Cuption
Methocs
BLEU- ROUGE BLEU-4 ROUGE
Tonk Specific Model
Vioaa Grouding
ScanRefer
MuliDDRef
SeadCCap Chas it al. (2021)
34.2
55.3
23.3
44.5
Methala
2: A.c all F190.22 all F1600.00
3DJCG (3022)
40.2
31.0
Tank Spooiii Model
X- Trank2ppp Youn ctall. (3022)
35.7
54.7
25.1
45.3
3D-YWTA ef dl (3023)
36.5
57.6
34.0
54.3
3DVG Trrnfomer Ziao (2021)
25.5
VoteCCup
(20236)
39.3
34.4
3DIOG
26.6
30 JJM Rosed Model
DANd Chen di (20223)
37.9
32.2
133DA
(20244g
41.4
36.8
55.1
MSDR IP Zhang al (8023))
$1.9
46.7
42.8
1K.4
PQ3D Zho al. (3024k)
36.0
30 Z.M Bradd Madel
LBO llang et al. (2024c)
36.9
57.8
Hong (3025)
Chat-scone Boung
(2004g)
38.2
36.3
Grounde LLM Cless (3025)
45.2
40.6
Gronded SD-LLM Chan et all (2025)
35.5
Char-sonne Hisang (hitaa)
55.5
902
57.1
57.4
RobanD King et (2025u)
38.4
RosdD Wing (2025a)
RosciD Wong (3225g)
596
56.3
Vaion ELM Bosed Model
Volie E2A Roood Mixdll
LLaVA-3D Zhu al. (2025)
41.1
63.4
LlaVA-3D (1UPS)
Video 3D--- Zhong ot al (2025)
41.3
Vilco-DD
(2025)
58.0
52.7
LitenVLS..8 (GPT4Seene)
44.1
41.4
Inter/V1. 3-5.. (CBPTsSaame))
61.4
57.7
65.5
A0.7
Qwen2 VL-7B (GPT4Sceme)
43.1
40.6
59.3
Qwen2 VL-7B (CPP4Semee)
62.6
$7.0
64.5
59,8
Qwen2 11225-VV 5-L--B 7B (GPT4Sorne)
45.9
67.9
46.1
67.1
Qwen2 5 VL-7B (UPT48oene)
65.6
59.5
67.3
628
8B (GPT4Scene) and Qwen2-VL-7B (GPT4Scene), demonstrate breakthrough performance using only visual inputs (video + BEV imagery). For instance, Qwen2-VL-7B (GPT4Scene) achieves an Acc@0.25 of 62.6 on ScanRefer and an all F1 @0.25 of 64.5 on Multi3DRef, surpassing the previous SOTA, Chat-scene, by 7.1 and 7.4 points, respectively. Furthermore, our improved Dwen2.5-VL-77 (GPT4Scene) model sets a new state-of-the-art, elevating these scores to 65.6 and 67.3.
GPT score: new evaluation way. We propose a novel GPT Score for 3D QA assessment, using the state-of-the-art 3D LLM Chat-scene Huang al. (2024a) the baseline. For our compari- son, we selected a total of 1,000 questions from the ScanQA dataset, distributed across six dis- tinct categories: Spatial Relationship. Object Attribute, Existence & Counting, State or Affordance, Group Reference, and Relational Refer. We then prompted GPT-4o OpenAl (2024) to compare the responses from our GPT4Scene- finetuned model against those from Chat-scene. The evalu- ation revealed a clear pattern: GPT4Scene demonstrates significant performance advantage in object-centric categories, particularly excelling in tasks related to Relational Refer and Existence & Counting. Conversely, the performance gap in the Spatial Relationship category was not signif- icant, indicating comparable capabilities in that area. These findings strongly suggest that the core strengths of GPT4Scene lie in its enhanced understanding and processing at the object level.
3.3 ABLATION STUDY
In this section, we conduct ablation studies to validate the effectiveness of GPT4Scene. First, we evaluate its robustness, including performance on small objects, followed by analyzing the robust- ness of STO- markers and reconstruction quality. Next, we perform module-wiss ablation to assess individual components. Finally, we investigate the impact of frame intervals and resolution settings.
The effectiveness during training and inference. Table 6 presents an ablation study on the GPT4Scene framework, analyzing the contribution of its components by selectively enabling them during training and inference. The study's key finding is that the fine-tuning process with GPT4Scene is the most critical driver of performance. A model trained this way maintains strong results even without the special inference components (BEV images and STO-markers), demon- strating that acquires an intrinsic 3D understanding. While training is fundamental, the inference components provide further enhancement, pushing the model to its peak performance when com- bined. This study clearly validates the dual effectiveness of the GPT4Scene framework in both empowering the model through training and enhancing its capabilities inference.
7

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on the Efficacy of GPT4Scene. Table 7: Ablation Study on BEV Reconstruction (1) on fully fine-tuned models with GPT4Scene; (2) on
Quality, The quality of BEV reconstruction has a negligible impact on QA performance, since the BEV mainly offers a global overview of the scene.
pure-video fine-tuned models: (3) in a zero-shot setting
without training. Exp conducted on Owen2-VL-77
GPT4Soene
Train
GPT4Seane
Eval
METEOR
ScarQA
ROUGE
CIDEr
BEV Reconstruction Quality
ScanOA ROUGE
SQA3D EM-1
18.9
46.5
96.3
BundieFusion (Baseline)
46.5
60.6
1866
95.4
SLAMER, 50-frame intervals
46.1
58.8
(Pure Video SFT) x
/
17.3
43.5
88.2
100-rrmme intervals
46.7
61.3
(Pure Video X
8
16.7
42.1
85.9
200 fram intervals
45.8
59.1
(No training) x
/
14.1
33.2
68.7
GS-SLAM
46.3
60.7
(No training)
SLAA
47.0
59.5
Table 8: Ablation on the size of objects.
Table 9; Ablation study on the BEV and STO-Markers. Both
Comprehension is most effective for large the BEV Image and STO-markers improve understanding. Per- objects, and diminishes for smaller ones.
formance peaks at an optimal STO-marker size before degrading.
Connector
ScanQA
ScanQA
w/f DEV Image
volo BEV Image
METEOR ROUGE CIDE
METEOR
ROUGE
CIDEr
METTER
ROUGE
CIDli
All objects Small ge.g.. alarm clock)
18.9
46.5
96,3
w/o STO-markers
17.514
43.1
90.4
16.0.29
40.0,a.s 84.1
42.8
91.5
size: 30, STO
18.2
45.3
16.7.22
42.2,43
87.5
Middlo (e.g. chair)
18.7
46.6
95.8
size: 40, STO
18.9
46.5
96.3 jo
174.,5
43.3,aa
90.1.6a
Large (e.g. cahinet)
19.3
47.3
97.5
size: STO
18.0ga
419,s
93.1g2
16.5.24
11.8,a7
86.9
Ablation Study on BEV Reconstruction Quality. This ablation study in Table 7 investigates the impact of BEV (Bird's-Eye View) reconstruction quality model performance. The central hy- pothesis is that the BEV's primary role is to provide global scene context, not precise geometric detail. The experiment compared various reconstruction methods (e.g., the baseline BundleFusion, SLAM3R at different frame rates) on the ScanQA and SQA3D benchmarks. The results show that despite variations in reconstruction quality, key performance metrics like ROUGE and EM-1 mained stable with only negligible fluctuations. This strongly confirms that the GPT4Scene frame- work is robust to the geometric precision of the BEV map, depending on it for overall layout rather than fine-grained accuracy.
Ablation on the size of objects. This ablation study in Table 8 analyzes how object size impacts the model's 'ss 3D understanding capabilities by categorizing objects into small, medium. and large. The results reveal clear positive correlation between object size and performance: the model under- stands large objects best, performs near the average for medium objects, and is least effective with small objects. This is because larger objects provide more prominent visual features and clearer spatial footprints in video frames and BEV maps, making them easier to ground. Nevertheless, the model maintains a strong baseline performance even on small objects, confirming the overall effectiveness and robustness of the GPT4Scene framework across various object scales.
Ablation study on the BEV and STO-Markers. This ablation study in Table provides a detailed analysis of GPT4Scene's 's two core components: the BEV image and STO-MarkeNs The results yield two critical conclusions. First, both the BEV image and STO-Markers are essential for performance and demonstrate clear synergistic effect. The data shows that removing either component-forr instance, removing STO-Markers while keeping the BEV image, or vice versa-cause a significant decline across all key metrics (METEOR, ROUGE, and CIDEr). This confirms that the global scene context provided by the BEV and the spatio-temporal consistency established by STO-Markers are both indispensable for high-fidelity 3D comprehension. Second, the study reveals that there is an optimal size for the STO TTO-Markers. Performance peaked when the marker size was set to 40, while smaller (30) or larger (50) sizes led to a drop in performance. This suggests that the marker scale must be properly adapted to the visual features of objects within the scene to maximize its effective- ness in linking global and local information.
Ablation on the number of input frames and image resolution. Figure 4 visualizes an ablation study that assesses the impact of two key parameters- number of input frames and image reso- lution-ov on model performance across different 3D understanding tasks. The experiment compares two core tasks: 3D Question Answering (QA) and 3D Visual Grounding. The central finding is that these parameters affect the two tasks very differently. Both higher image resolution and an increased number of input frames provide a substantial performance boost for the 3D Visual Grounding task, which relies on precise object localization. In contrast, the same parameter upgrades yield only
8

---

Published as a conference paper at ICLR 2026
Onvan--l-1 (Bead no15
Owerd (houchect
terformance Score
Owerd (oPrasarcee As
OML Count
Jun. BMC-
OR G2
9el pet.
Route
Adol Order
Figure Evaluation GPT4Scene on the VSIBench. 'S' denotes performance on ScanNet, while 'NS' refers to the ARKitScenes dataset. The results indicate that GPT4Scene enhances spatial intelligence.
Fraret
Frarus
Figure 4: Ablation on the number of input frames and image reso- Figure 5: GPT-Score evaluation. lution. We find that these parameters have the most significant impact GPT4Scene holds an advantage on on the Grounding task and the least question answering.
object-level tasks than Chat-scene.
minimal improvements for the 3D QA task, which involves more general scene comprehension. This result leads clear conclusion: for tasks like Visual Grounding, prioritizing high-resolution, multi-frame input is crucial for achieving the best results. For tasks like 3D QA, a more balanced configuration can be used optimize for efficiency without a significant loss in performance.
3.4 EALUATION ON SPATIAL INTELLIGENCE: VSIBENCH
We have completed scene understanding and conducted tests on spatial reasoning. Figure 3presents a visual analysis of the GPT4Scene framework's performance on the VSIBench, a benchmark de- signed specifically for evaluating spatial intelligence. The primary goal of this evaluation is to validate how effectively GPT4Scene enhances the spatial reasoning capabilities Vision Language Models (VLMs). The experiments ARKitScenes ("NS"), to test the framework's robustness in various types of 3D environments. The key results visualized in the chart show that models integrated with GPT4Scene consistently out- perform their baseline counterparts across both datasets. This superiority is particularly evident in tasks that demand a high degree of spatial awareness, such as judging spatial relationships between objects and understanding the overall layout of a scene. This outcome provides strong evidence for the effectiveness of GPT4Scene's 's core design principles. By providing global scene context through BEV images and establishing spatio-temporal consistency with STO-markers, the frame- work successfully empowers VLMs to overcome their previous limitations, thereby significantly strengthening their cognitive and reasoning abilities within complex 3D spaces.
conducted across two different datasets, ScanNet ("S") and
4 CONCLUSION
We propose GPT4Scene, framework enhancing VL.Ms to understand 3D scenes directly from vi- sual inputs. By reconstructing 3D point clouds for Bird's Eye View (BEV) images and aligning video frames with spatial-temporal object (STO) markers, hension. GPT4Scene achieves state-of-the-art 3D QA performance with zero-shot GPT-4o and by fine-tuning smaller VLMs using ScanAlign dataset. These fine-tuned models even excel with raw video inputs, proving effective 3D understanding. Despite relying on point cloud annotations for marker generation due benchmark constraints, aim to address this by generating STO-markers from video segmentation in future work.
enable global-local scene compre-
Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 62422606, 62441615) and Hong Kong Research Grant Council General Research Fund (No. 17213925).

---

Published as a conference paper at ICLR 2026
REFERENCES
Hsin-Ying Ahmed Abdelrcheem, Kyle Olszewski, Hsin- Ying Lee, Peter Wonka, and Panos Achlioptas. Sca- nents3d: Exploiting porase-to-3d-obyee correspondences for improved visio-linguistic models in 3d scenes. In WACV, 2024.
Panos Achlioptas, Ahmed Abdelrcheem, Fei Xia. Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In ECCV, 2020.
Man-Bappiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurIPS, 2022.
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. S.Kiv:2404.03411. 2024.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question an- swering for spatial scene understanding. In CVPR, 2022.
Eslam Bakr, Yasmeen Alsaedy, and Mohamed Elhoseiny. Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding. In NeurlPS, 2022.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. In NeurIPS, 2021.
Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djog: A unified framework for joint dense captioning and visual grounding 3d point clouds. In CVPR, 2022.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video- language understanding. In NeurIPS, 2024.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2017.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020.
Dave Zhenyu Chen, Qirui Wu, Matthias Nieβner, and Angel X Chang. D3net: speaker-listener architecture for semi-supereised dense captioning and visual grounding in rgb-d scans. In ECCV, 2022a.
Jing Chen, Wenhao Luo, Xizhou Wei, Lin Ma, and Wei Zhang. Ham: Hierarchical attention model with high performance for 3d visual grounding. arXiv:2210. 20. 12513, 2022b.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large lan- guage model as a unified interface for vasion-language multi-task learning. arXiv:2310.09478. 2023a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In NewrIPS, 2024a.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In NeurlPS, 2022c.
10

---

Published as a conference paper at ICLR 2026
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In NeurIPS, 2022d.
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In CVPR, 2023b.
Sijin Chen, Xin Chen, Chi Zhang. Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In CVPR, 2024b.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv:2405. 10370, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, In CVPR, 2024c.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, 2021.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. wwXiv2246.007476 2024.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnner Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017a.
Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt. Bundle- fusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration. TOG, 2017b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purppss vision-language models with instruction tuning. In NeurlPS, 2023.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Erie Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. In NeurlPS, 2022.
Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language- driven open- vocabulayy 3d scene understanding. In CVPR, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He. Xingeheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Incemmin-coomposee2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv:2401. 16420, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. In CVPR, 2025a.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. In WACV, 2025b.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li. and Pheng-Ann Heng. Point-bind & point-llm: Align- ing point cloud with multi- modaittyd for 3d understanding, generation, and instruction following. arXiv:2309.0060 10.15 2023a.
Zoey Guo, Yiwen Tang. Ray Zhang, Dong Wang, Zhigang Wang. Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In ICCV, 2023b.
11

---

Published as a conference paper at ICLR 2026
Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In CoRL, 2022.
Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transre- fer3d: Entity-and-relation aware transformer for fine-grained 3d visual grounding. In ACMMM, 2021.
Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In ICCV, 2023.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In NeurlPS, 2023.
Haifeng Huang, Yilum Chen, Zehan Wang, Rongjie Huang, Runsen Xu. Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In NeurIPS, 2024a.
Haifeng Huang, Zehan Wang. Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. In NewrlPS, 2024b.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chur Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML, 2024c.
Pao-Hsiang Huang, Hsin-Hsi Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. AAAI, 2021.
Shaohan Huang. Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. In NewrIPS, 2023.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual ground- ing. In CVPR, 2022a.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual ground- ing. In CVPR, 2022b.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In ECCV, 2022.
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In ECCV, 2024.
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In CVPR, 2020.
Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3d-langeage pre-training In CVPR, 2023.
Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large language model via robust instruction tuning. In ICCV, 2025a.
Weitai Kang, Mengxue Qu, Jyoti Kini, Yunchao Wei, Mubarak Shah, and Yan Yan. Intent3d: 3d object detection in rgb-d scans based on human intention. In ICLR, 2025b.
Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In CVPR, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In CVPR, 2024.
12

---

Published as a conference paper at ICLR 2026
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. TMLR, 2025a.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao. Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning space: Multimodal visualization--f-tthuug. In ICML, 2025b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping aanuuage-image pre- training for unified vision-language understanding and generation. ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video under- standing benchmark. In CVPR, 2024a.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024b.
Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ran- jay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. In CVPR, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b. URL https: //llava-vl. wtb.gii..bb.o//ooooooo..oooooooo 22044- 00-30-1---000000000000 1lava- next.
Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. Mmbench: Is your multi-modal model an all- around player? ECCV, 2024c.
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. CVPR, 2022.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv:2306.07207, 2023.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In ICLR, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Vidco-chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liang- Yan Gui, and Yu-Xiong Wang. Lexicon3d: Probing visual foundation models for complex 3d scene understanding. In NeurIPS, 2024.
Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, and Manolis Savva. Multiscan: Scal- able rgbd scanning for 3d environments with articulated objects. In NeurIPS, 2022.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In ICCV, 2021.
OpenAl. Gpt-4 technical report. arXiv:2303.08774, 2023.
OpenAl. Gpt-4o: A large language model by openai. https://openti httplpppaaaa.ccceeeeeeeeeee www/research./ go,-4o, 2024.
13

---

Published as a conference paper at ICLR 2026
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys. Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world. In ICLR, 2024.
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
Zekun Qi, Rumpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Gc, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In ECCV, 2024a.
Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Heng- shuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In CVPR, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel Chang, al. Habitat-matterprt 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In NeurlPS, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang. and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In ICRA, 2023.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, 2024.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur- Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv:1906.05997 2019.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, 2024.
Ayça Takmaz, Flisabetta Fedele, Robert W Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. In NeurIPS, 2023.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv:2403.05530. 2024a.
Gemma Team and Google DeepMind. Gemma Improving open language models at practical size. arXiv:2408.001 18, 2024.
InternLM Team. Internim2 technical report. arXiv:2403.17297, 2024b.
Llama Team. The llama 3 herd of models. arXiv:2407.21788 2024c.
14

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen2.5 technical report. 0IX1::2412.51115222244
Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. Four ways to improve verbo- fusion for dense 3d visual grounding. In ECCV, 2024.
Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation on point clouds. In CVPR, 2022.
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d
object instance re-localization in changing indoor environments. In ICCV, 2019.
Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang
Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness. In ICCV, 2025a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. In ICLR, 2025b.
Tai Wang, Xiaohan Mao, Chemming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embod- iedscan: A holistic multi-modal 3d perception suite towards embodied ai. In CVPR, 2024a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et Cogvlm: Visual expert for pretrained language models. NeurIPS, 2024b.
Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 3drp-net: 3d relative position-awer aware network for 3d visual grounding. In EMNLP, 2023.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. In NAACL, 2025c.
Ting-Yas Yao Wu, Shih-Yang Huang, and Yung-Chen Chen Francis Wang. Dora: 3d visual grounding with order-aware referring. In WACV, 2025.
Xiaoyang WI, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. In CVPR, 2024.
Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. decoupling and dense aligmment for 3d visual grounding. In CVPR, 2023.
Eda: Explicit text-
XAI, 2024. tree/main
URL https: / http///ugggggccceeee/e/ co co/daaasets//eat Seaad-org/RRawwold.e
Runsen Xu, Zhiwei Huang, Tai Wang. Yilun Chen, Jiangmiao Pang, and Dahua Lin. VIm-grounder: A vlm agent for zero-shot 3d visual grounding. In CoRL., 2024a.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In ECCV, 2024b.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In CVPR, 2023.
Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language con- trastive learning for open-world 3d scene understanding. In CVPR, 2024.
Jihan Yang. Shusheng Yang. Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xic. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. In CVPR, 2025.
Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat 2d semantics assisted training for 3d visual grounding. In ICCV, 2021.
15

---

Published as a conference paper at ICLR 2026
Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering. TVCG, 2022.
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi- level contextual referring. In ICCV, 2021.
Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guanbin Li, Shuguang Cui, and Zhen Li. X- trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. In CVPR, 2022.
Xiaoyang Wu ct.al Yunhan Yang. Sam3d: Segment anything in 3d scenes. In ICCV Workshop, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In EMNLP, 2023a.
Renrui Zhang. Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR, 2022.
Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. In CVPR, 2023b.
Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. In ECCV, 2024.
Yiming Zhang, ZeMing Gong, and Angel Chang. Multi3drefer: Grounding text description to multiple 3d objects. In ICCV, 2023c.
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In ICCV, 2021.
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video repre- sentation for 3d scene understanding. In CVPR, 2025.
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, 2020.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. ICCV, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024a.
Ziyu Zhu, Xiaojian Ma. Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. In ICCV, 2023.
Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d dision-language understanding via promptable queries. In ECCV, 2024b.
16

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
During the preparation of this manuscript, we utilized a Large Language Model (LLM) as an as- sistive tool. The primary role of the LLM was for language polishing, which included improving grammar, enhancing the clarity and fluency of the text, and ensuring consistent use of technical ter- minology. All core research ideas, methodologies, experimental designs, results, and conclusions were conceived and formulated exclusively by the human authors. The LLM did not contribute to any of the substantive scientific aspects of this work. We have carefully reviewed and edited all text modified with the assistance of the LLM and take full responsibility for the final content of.
B PROMPTS OF CLOSED-SOURCE VLMS
Here, we present the prompts used by GPT4Scene in the closed-source VLMs (GPT-40), as illus- trated in Figure 6. The process begins with a general system prompt, which outlines the overview of two images provided as input. The first image stitched 2D view captured from video, with dimensions of 2 x 4. The second image represents a BEV (Bird's Eye View). Subsequently, perform evaluations across various tasks and benchmarks, with each benchmark associated with a specific prompt. We take the 3D question-answering answering task on ScanQA mark prompt consists of three parts:
example. The bench-
1. Important Guidelines: It clarifies that while we provide object IDs for reference, they cannot be directly used when answering questions. Additionally, specifies adapting the response style to match that of ScanQA. Since ScanQA's responses are typically short single words, we aim to keep the answers concise within the benchmark prompt, targeting 1-5 words.
2. Answer Format:
In this part, we use standardized regularized format to structure the answers, which helps improve accuracy when addressing questions.
Examples: we include two example cases.
Our zero-shot prompting process is illustrated in the bottom-left corner Figure 6. The system prompt and ScanQA prompt are used the system message. In the user message, we input the stitched image and BEV image. Finally, the query message includes the question. The responses generated through this process require further refinement, as depicted in the bottom-right corner of Figure 6. First, we remove the regularized formatting from the answers. Next, we clean the answers by addressing singular/plural forms and case sensitivity. This final step ensures that we obtain the refined answers.
C 2D MULTI-MODAL BENCHMARK
We tested the fine-tuned Qwen2-VLL VL-7B (GPT4Scene) model on 2D image and video multimodal large models. Table 10 shows the results of MVBench. As we can see, our model shows improve- ment for the object and action metrics, indicating that the model fine-tuned with ScanAlign is better at handling spatial variations and the information of objects in the scene. Table 11 presents the results on other benchmarks, where we can observe that after our fine-tuning, the model's ability to understand images and videos did not decline significantly. It demonstrates the effectiveness of ScanAlign.
D QUALITATIVE RESULTS
Figures to 10 present our qualitative results obtained from Qwen2-VL, after fine-tuning on ScanAlign. Figures 7 and demonstrate 3D question answering performance using only unan- notated video inputs as context. Our observations indicate the model's capability to effectively address queries from both ScanQA Azuma et al. (2022) and SQA3D Ma et al. (2023) benchmarks. Mcanwhile, Figures 9 and 10 exhibit the model's competence in 3D dense captioning and 3D vi- sual grounding tasks requiring annotated inputs. These outcomes substantiate that our framework generate precise responses through visual information processing alone, without dependency
17

---

Published as a conference paper at ICLR 2026
Model
Task type
Qwen2-VL
Ours
Action Sequence
85.5
82.0
Action Prediction
69.5
70.5
Action Antonym
83.0
86.0
Fine-grained Action
51.5
51.5
Unexpected Action
82.0
78.0
Object Existence
87.5
88.5
Object Interaction
82.0
81.5
Object Shuffle
41.0
45.0
Moving Direction
42.0
40.0
Action Localization
65.0
66.5
Scene Transition
93.5
94.0
Action Count
47.5
43.5
Moving Count
69.5
71.5
Moving Attribute
90.0
88.5
State Change
48.0
49.0
Fine-grained Pose
63.0
63.5
Character Order
74.5
71.0
Egocentric Navigation
39.5
41.5
Episodic Reasoning
47.0
47.0
Counterfactual Inference
62.5
65.5
Avg
66.2
66.225
Table The result MVBench al. (2024a). After fine-tuning with ScanAlign in GPT4Scene, our model shows improved understanding, particularly in object and action metrics.
Model
Benchmark
Qwen2-VI VL
Ours
MMBench-ENNwa Liu et al. (2024c)
82,4
81.2
MMBench-CN Liu et al. (2024c)
81.7
79.9
MMStar Chen al. (2024a)
60.7
57.6
RealWorldQA xAI (2024)
70.1
68.5
Video-MME Fu et al. (2025a)
59.8
58.4
Table 11: The result of 2D Multi-modal Benchmark. After fine-tuning with ScanAlign in GPT4Seene, our model's 2D understanding capabilities did not decline.
on 3D point cloud inputs. Here we emphasize that post-training with the GPT4Scene framework. the model achieves accurate QA performance using only pure video input, thereby demonstrating GPT4Scene's effectiveness in enhancing visual comprehension capabilities.
E FULL QUANTITIVE RESULTS
Here, we present the complete metrics for all five benchmarks. Table and Table 13 show results for ScanQA Azuma et al. (2022) and SQA3D Ma et al. (2023). Table 14 provides the full results for
18

---

Published as a conference paper at ICLR 2026
Methods
Language
FM-1
RI.FI-1
RIFF1-2
BERI--
RC-4
ROLCCR
METFOR
CITI::
Tesk-Spergfe Mode!
ScanQA Arrma ot (1022)
30.2
20.4
15.1
10.1
33.3
13.1
64.9
(2023)
3D Vi-Vista 2hm (1021)
13.9
35.7
30 LLM Boxed Model
SD4LIM Bong es (3023)
20.5
39.3
25.2
18.4
12.0
35.7
14.5
90.4
(20245)
76,8
LE0 Huing ct (30244)
24.5
11.5
39.3
16.2
80.0
Secue-LLM
(2025b)
26.8
80.0
Chat scene Ulunng et al (20244)
21.6
43.2
29.1
20.6
14.3
41.6
18.0
57.7
Viston LLM Based Model
InternV1. 5-88 (GPT44smme)
29.5
45.1
30.7
22.7
16.2
47.8
19.5
96.8
Qwes2-VL-78 (GPT4S
Qwen2 5-ML-B (GPT4Sceme)
31.3
49.2
38.2
28.3
50.
21.1
105.7
Table 12: Full Evaluation of 3D Question Answering ScanQA Azuma et al. (2022).
Metheds
Set
(EM-1)
EMARI
Task-Spruijie Midel
ClipBERT et (2023)
30.2
60.1
38.7
63.3
42.7
43.3
SQAID Ma (2023)
31.6
63.8
46.0
69.5
43.9
45.5
46.6
3D-VsTA cta (3023)
60.8
30 Basod Model
PQ3D Ziu (2024))
37.1
61.3
44.5
60.9
47.0
45.1
47.1
LBO [ooing (2024c)
50.0
5244
Scese-LLMM
(20250)
52.5
Chat scete Buzng (2024g)
45.4
67.0
$2.0
69.5
49.9
55.0
54.6
57.5
Z1M
lateraV33 (GPT45cene)
70.8
62.4
64.5
Qwen2-VL-78 (GPT4Secme)
50.5
68.7
Qwen2.5.VL. VB (GPT4Scemc)
60.3
72.5
35.5
71.8
56.3
64.9
63.5
66.2
Table 13: Full Evaluation of 3D Question Answering on SQA3D Ma et al. (2023).
Mathods
loUg
Tusk-Speajii Madel
Scan2Cap Chen (2021)
56.8
36.2
25.3
55.5
39.1
23.3
22.0
3DVCG Cul (2022)
64.7
40.2
27.7
39.2
49.5
31.0
24.2
50%
(2022)
DENet Chen et (2022a)
62.6
35.7
25.7
53.9
VLP Fi (2023)
70.7
41.0
59.7
54.9
32.3
24,8
Vose2Cap Ches (20236)
71.5
3933
62.6
35.7
25.7
3D-Vi-TA Zhn al (2023)
71.0
36.5
28.4
57.8
66.9
34,0
27.1
$4.3
L1,DAA (2024))
LD0 [luung
68.4
36,9
27.7
Chot-sceme Hunng et al (204g)
81.9
38.
29.0
60.6
77.2
36.3
28.0
56.1
(2025c)
Vilin LLM Bannd Model
IntemVL3 (GPT4Scene)
93.5
46.1
29.8
63.1
88.1
41.4
28.7
60.3
Qwen2 VL-7B (GPT4S (PP44)cene)
91.7
43.1
29.3
61.9
86.3
40.6
28.2
59.3
Qwen2.5
Table 14: Full Evaluation of 3D Dense Caption on Scan2Cap Chen et al. (2021).
Scan2Cap Chen al. (2021), while Table 15 and Table present the results for ScanRefer Chen et al. (2020) and Multi3DRef Zhang et al. (2023c). Our model significantly improves across all benchmarks, highlighting that only pure vision language models can understand 3D scenes effec- tively.
19

---

Published as a conference paper at ICLR 2026
Unique
Multiple
Overall
Methods
Tost Tast-Spectli Model
SeunRefer
TONN Bunng all (2021)
68.6
56.8
29.8
23.2
37.4
29.7
(2027)
37.6
44.5
30.1
InstameRRfeer Youn et (2021)
75.7
64.7
24.4
23.0
38A
31.1
3DVG-TranSoomer Zhao et (2021)
81.9
60.6
30.3
28.4
47.6
34.7
MVT Heang (2022k)
3D-SPS Lio (3022)
84.1
66.7
40.3
29.8
48.8
$7.0
VIL3DRd Chen ot al. (20220)
40.3
301.7
47.9
37.7
3DIG
D3Net Chen et (20224)
72.0
30.1
37.9
BUTD-DETR
(2022)
HAM Clies (30226)
79.2
67.9
41.5
34.0
48.8
40.6
3DRP-Ner Wang (2023)
83.1
61.7
42.1
32.0
50.1
38.9
3D-VL (2025)
EDA Wu et (2023)
85.8
68.6
49.1
37.6
54.6
42.3
Zhong (20236)
3D-VirTA Zho (2023)
81.6
75.1
43.7
39.1
50.6
ConcreteNet Unal al. (2024)
56.4
82.1
42.4
38.4
50.6
46.5
37 ZJM Rased Model
Chal-soeme Honng al. (2024g)
55.5
50.2
Robin3D King (20256)
60.8
55.1
Vision LLM Based Model
(GPT4Sceae)
Qwen2-NL-7B (GPT4Socne)
90.3
83.7
56.4
$0.9
62.6
57.0
Owen2.5:ML-7B (GPT4Seene)
91.5
85.8
59.2
53.7
65.5
59.5
Table 15: Full Evaluation of 3D Visual Grounding ScanRefer Chen et al. (2020).
Mahodk
wio
ST wio
W
ALL
P1
F1
25
P1690.2
(1100.5
2580022
F1805
F180025
F1800
Tast Speetfi Model
3DVG-Trane Zlao al. (3021)
$7.1
45.5
23.5
16.7
26.5
25.5
(Cmmnding)
(1027))
SUJCG (Gounding) Cal et (2022)
94.1
66.9
26.0
16.7
26.2
26.5
MIDR:FCCLI ct (203))
39.4
47.8
18.4
30 11M Rosed Model
Chet-szmme Huung (2044)
50.3
62.6
82.9
75.9
44.5
45.7
41.1
57.1
$2.4
Visin
lateraY1.5 V1.3V13.... (GPT48cene)
97.6
85.1
85.5
78.3
6088
559
49.2
45.3
653
60.7
Qwea2-VLL-B8 (GPT4Suene)
97.4
64.5
Quenn S-V1.78 (GPT488cme)
08.0
86.5
56.8
70.9
62.5
$7.8
51.0
47.1
67.3
60.8
Table 16: Full Evaluation of 3D Visual Grounding Multi3DRef Zhang al. (2023c).
20

---

Published as a conference paper at ICLR 2026
System Prompt
indoor
labeled 2D image and labeled Bird's View (BEV) analysis.
image captured intervais arranged in 2x4 grid
right, bottom.
Object labels numbered, numbers matching between the 2D and BEV images to indicate the same objects.
ScanQA Prompt
required provide based
Important Guidelines
1. answering questions, do not reference marks directly. These marks are only provided assist understancing layout, Your answers should rofer to specific objects the soane, not the marks.
directions position, prominent
image express spatial relationships,
Keep your answers concise possible. For questions regarding quantity,
1-5 words. questions
spatal relationships, answers can be slightly longer but should not exceed 10 words, De not provide any additional, irre rrelevant information.
Answer Format
anowers lovercase.
include any punctuation Any numbers
Please format your
follows: '##Q1## answer1, ##Q2## answer2,
Examples:
cabinet?
Question: What the left the Answer: bicycle on floor
Answer: light brown
Zero-shot Prompting
Refinement Procedures
Syetem Message: <System Prompt> <StanQA Prompt>
1. Get the answer:
#AQ1EH White board.'
User Message (image type): <url, for, frames>
2. Remove answer format:
Refinement clean "White board."
Remove singular and plural forms.
Remove unnecessary adjectives.
Remove punctuation spaces.
Remove uppercase and lowercase distinctions.
3. Query Message: 'What is the black chair in front of?'
Figure 6: Prompts of Closed-sourse VLMs. We show the prompts used for GPT-4o (GPT4Scene), which consist of system prompt and benchmark prompt. After generating responses, further refine them.
21

---

Published as a conference paper at ICLR 2026
What color the armchair?
What is under the sink?
What on the wall?
The color brown.
cabinets.
Figure 7: Qualitive Results: Question Answering. We provide vidcos without object annotations.
What hanging from the bathroom door?
The stool is in front of what?
What color is the sink above the rug?
There are three towels.
The stool is in front of bathtub.
The color beige.
Figure Qualitive Results: Question Answering. We provide videos without object annotations.
22

---

Published as a conference paper at ICLR 2026
you discribe object 33?
object window frame. is directly above the bed on the side
plastic storage rectangular shape It is on the desk next to chair and shelf.
object 2.
Figure 9: Qualitive Results: Dense Caption and Visual Grounding. provide videos with object annotations.
Can you discribe the object 28?
Give me the object ID: this is gray chair. is to right of another gray chair
This is black wheeled office chair.
object 17.
positioned
table.
Figure 10: Qualitive Results: 3D Dense Caption and Visual Grounding, We provide videos with object annotations.
23

---

Published as a conference paper at ICLR 2026
F RELATED WORK
3D indoor scene understanding. 3D indoor scene understanding allows robots to identify object positions, structures, and relationships within indoor environments, enabling guestion-and-answer interactions about the scene's content. This process combines 3D perception with large language models (LLMs). 3D perception, as a foundational component, indoor datasets Dai et al. (2017a) 7a); Baruch et al. (2021); Chang et al. (2017); Mao et al. (2022); Ramakrishnan et al. (2021); Wald et al. (2019); Straub et al. (2019); Deitke et al. (2022); Khanna et al. (2024); Zheng et al. (2020) using point clouds as input, supporting tasks like 3D object de- tection and instance segmentation Qi et al. (2019); Jiang et al. (2020); Misra et al. (2021); Schult et al. (2023); Vu et al. (2022); Wu et al. (2024); Yunhan Yang (2023). Recent advancements in 3D Vision-Language Learning (3D-VL) combine 3D perception tasks with natural language, intro- ducing diverse textual annotations
typically trained on common
datasets like ScanNet to support tasks such 3D Question Answering Azuma et al. (2022); Ma et al. (2023); Ye et al. (2022); Chen et al. (2022c), 3D Dense Captioning Chen al. (2021), and 3D Visual Grounding Chen al. (2020); Zhang al. (2023c); Achlioptas et al. (2020); Abdelreheem et al. (2024); Kang et al. (2025b). Initial studies focus on single 3D-VL tasks Yuan et al. (2022); Chen et al. (2023b); Wu al. (2023); Bakr al. (2022); He et al. (2021); Huang ct al. (2022b); Jain et al. (2022); Luo et al. (2022); Yang et al. (2021); Zhao et al. (2021); Guo al. (2023b). while recent research introduces unified models for multiple tasks Cai et al. (2022); Chen et al. (2022a). 2D vision-language pretraining (2D-VLP has driven progress in 3D visual-language learning (3D-VL) Ha & Song (2022); Hegde et al. (2023); Peng et al. (2023); Takmaz et al. (2023); Xue et al. (2023); Zhang et al. (2022; 2023b); ); Xu et al. (2024a), with recent 3D-VLP Zhu et al. (2023; 2024b); Jin et al. (2023); Ding et al. (2023); Yang et al. (2024); Jia et al. (2024); Wang al. (2024a) methods demonstrating that combining 2D visual cues with 3D point clouds enables complementary rross-modal alignment. This multimodal approach helps overcome geometric complexity and sparse annotation challenges in pure point processing.
3D Point cloud LLMs. 3D vision-language tasks aim to integrate 3D scene understanding with natural language processing. However, we aspire to go further by incorporating 3D content into large language models (LLMs) to achieve more natural human compttr interaction. Initially, this began with 3D point cloud LLMs. 3D point cloud LLMs take point clouds input, enabling natural language generation and interaction in 3D scenes. Early 3D LLMs focused on object-level geometry and appearance Guo al. (2023a); Qi et al. (2024b); Xu et al. (2024b); Qi et al. (2024a). Later, they expanded to indoor scenes, emphasizing spatial relationships among objects and overall scene features, often utilizing scene point clouds augmented with auxiliary 2D multi-view images Hong et al. (2023); Chen et al. (2024b); Wang et al. (2025c); Fu et al. (2025b); Man et al. (2024). To better capture object relationships, recent 3D LLMs decouple scene objects before feeding them into L.MS Huang et al. (2024c;a). Some approaches rely more heavily on visual inputs to determine scene context Chandrasegaran et al. (2024); Zhu et al. (2025); Liu et al. (2025). Here, we aim to explore whether pure visual inputs can better handle indoor scene understanding.
Vision Language Models (VLMs). Vision Language Models (VLMs) are multimodal models that integrate visual and language processing capabilities, enabling the understanding and generation of combined image-text information. The origin of VLMS can be traced back to CLIPP's 2D image-text pair pretraining Radford et al. (2021); Changpinyo et al. (2021); Schuhmann et al. (2022), which laid the foundation for incorporating LLMs. Early VLMs used attention mechanisms or Q-Former to fuse image and text modalities before inputting them to LLMs Li et al. (2022; 2023); Dai et al. (2023); Alayrac et al. (2022); Huang et al. (2023). Later, an approach emerged that directly projects image features into the LLM using an MLP Liu et al. (2023; 2024a); Li al. (2025a): Zhu et al. (2024a); Wang et al. (2025b), achieving better performance. Building on this, VL.Ms expanded into visual grounding tasks Wang et al. (2024b); Peng et al. (2024); Rasheed et al. (2024); Lai et al. (2024); Dong et al. (2024); Sun et al. (2024); Chen et al. (2023a) and further to video understanding by using spatiotemporal compression to process information from long image sequences Zhang al. (2023a); Cheng et al. (2024); Maaz et al. (2024); Luo et al. (2023); Ataallah et al. (2024); Li et al. (2024b); Song et al. (2024); Ren et al. (2024); Liu et al. (2024b). Currently, while some studies employ VLMs for indoor scene understanding, most remain either benchmark-oriented Yang et al. (2025); Li et al. (2025b) or still rely on 3D-based methods Zhang et al. (2024); Zheng et al. (2025). This fundamentally reveals that VLMs cannot directly comprehend 3D scenes, making core mission to empower them with 3D world interpretation capabilities.
24

---

Published as a conference paper at ICLR 2026
ID: 9 Chair
ID: 20 Refrigerator
ID: 9 Round ottomon
ID: Trashbin
ID: 9 Printer
ID: 7 Toilet
ID: Printer
ID: 6 Pillow
ID: Bathroom vanity
ID: 21 Chair
Figure 11: The heatmap after about the VLM feature.
25

---

Published as a conference paper at ICLR 2026
f1
Q: Please provide the description of Object No. 46.
A: This is white water dispenser.
30
Q: Please describe Object No. 9.
This chair, currently placed under the table.
28
2V
What is the ID number of the bathroom floor drain?
A: It No. 20.
Figure 12: Failure Cases of GPT4Scene.
26


Reviews:
[{"review_id": "144sX0f0bP", "reviewer": "Reviewer_FmXB", "summary": "This paper introduces GPT4Scene, a framework designed to enhance the 3D spatial understanding abilities of Vision-Language Models (VLMs) using only visual inputs (video). The approach constructs Bird’s Eye View (BEV) images from egocentric video and overlays spatial-temporal object markers (STO-markers) to align global scene layout with local observations, enabling robust 3D comprehension without reliance on explicit 3D point cloud data. The authors provide both zero-shot and fine-tuning strategies, introduce a large aligned dataset (ScanAlign), and empirically demonstrate performance gains across multiple 3D reasoning, captioning, and grounding benchmarks, including comprehensive ablations and qualitative analyses.", "questions": "Refer to the questions above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.  I think this paper is well motivated and has a clear conceptual advance over prior VLMs for 3D. Unlike previous methods that rely heavily on point clouds, the paper proposes a vision-only approach for 3D spatial reasoning, closely mimicking human perceptual processes for scene understanding.\n2. The empirical results is solid. Across diverse 3D benchmarks (Tables 3–7, and full results in tables 12–16), GPT4Scene models consistently set new state-of-the-art or strongly outperform both point-based and previous vision-language SOTA methods in 3D question answering, dense captioning, and visual grounding.\n3. The ScanAlign dataset is a valuable resource, which represents a practical resource with 165K aligned video–BEV–text triplets, supporting reproducibility, further research, and downstream benchmarking.\n4. The proposed method is scalable and architecture-agnostic. It does not require architectural changes to VLMs and shows improvements to both large, closed-source and smaller, open-source models.", "weaknesses": "1. Since the pipeline is dependent on 3D reconstruction and instance segmentation, I have a concern about its robustness. The method assumes reliable 3D scene reconstruction and high-quality mask annotations for generating BEV images and STO-markers. While Table 7 ablation demonstrates some robustness, the reliance on Mask3D and BundleFusion or similar systems (Figure 2) acts as a performance bottleneck. Real-world deployment may face significant degradation under varied lighting, occlusions, or sensor/calibration noise.\n2. Also,  the paper lacks of exploration of failure cases or qualitative weaknesses. The paper presents strongly positive qualitative and quantitative results (Figures 7–10, Table 3–7), but is light on concrete instances where GPT4Scene underperforms (e.g., when scenes are highly cluttered, objects are partially observed, or markers are mismatched). This restricts understanding of limitations and generalization bounds.\n3. While Table 6 ablation finds that fine-tuning with BEV/STO-markers enables \"intrinsic\" 3D scene understanding (i.e., persists even if markers and BEV are not presented at inference), the paper does not fully analyze why the VLMs develop this transfer capability, nor does it probe what types of spatial relationships/generalizations are learned vs. memorized. Is this effect robust to entirely novel scenes or objects unseen in the BEV/marker format at train time?\n4. Despite advocating a vision-only solution, the proposed method requires explicit 3D information (scene geometry, camera intrinsics/extrinsics, and point cloud segmentation) for preprocessing. While the input to the VLM is images, the pipeline uses classic 3D processing to derive BEV and object correspondences. This undermines the claim that the system \"relies solely on vision\" in practice, and should be explicitly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLNcFu4zLD", "reviewer": "Reviewer_wWch", "summary": "The paper proposes a framework called GPT4Scene to enhance the spatial understanding of VLMs. Specifically given a video, the framework reconstructs it and then projects it to a BEV image. Then, it uses an off the shelf 3D instance mask segmentation network and project its predictions to the 2d frames and the BEV map, thus marking them. This becomes the input to the VLM. The paper first shows that with this input, bigger VLMs improve performance while smaller VLMs do not benefit much or slightly decrease in performance. Then, the paper proposes to fine-tune the VLMs with such marked representation of videos, and evaluates the models’ performance on 3D VQA, captioning and grounding. The paper shows that their model achieves similar or better performance than prior SOTA methods.", "questions": "I am not particularly excited about this paper, especially due to several errors in the manuscript -- however, they can probably be fixed in the revision. Answers to the discrepancies pointed out in weakness section might help increase my rating.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper tackles an interesting and important problem of enabling 3D understanding using pre-trained 2D VLMs\n- The ablations are thorough and support the main design choices of the paper", "weaknesses": "The paper appears to have several wrong claims or statements, and formatting errors.\n\n- L199-200: “our method… matching or surpassing the Chat-Scene Chat-scene” in Table-1. However, from Table-1, it seems that the version with GPT4Scene never surpasses ChatScene. \n- From the introduction, it appears that a new datasets is being created and released i.e. ScanAlign. However, it is essentially a combination of existing (and popular) 3D grounding and captioning datasets with the addition of STO markers and the BEV images (which is nice). It would be nice to be more clear and explicit about this in introduction and section-2.3.\n- The description of experimental results and results themselves are misaligned again in section 3.2: There are phrases like “outperform prior SOTA like chat scene” but the table shows that the SOTA is ROSS3D and the proposed method only matches the performance of this method. Besides, since the paper do not describe any of the main baselines and details like base VLMs or training data used for the baseline, it is hard to figure out if the comparisons are fair. The section claims that the fine-tuned versions significantly improves the performance of untuned baselines VLMs, but the results are missing from the table. \n- In Table-5 task-specific model section, the baselines are significantly old. Current SOTA on ScanRefer is UniVLG (https://arxiv.org/abs/2503.10745) an the authors can check Table-1 of UniVLG for additional recent baselines. Additionally, could the authors say more on how the evaluation is conducted? This benchmarks tests for bounding box predictions — is the proposed method trained to regress bounding boxes?\n- The related work section of the paper is moved completely to the appendix and even there it reads like a bunch of citations. I understand the page limitations, but related works is an important section to contextualize the proposed work with the existing literature and highlighting the similarities and differences\n\nThe paper is riddled with formatting errors. A few examples:\n- L200 / 222: repeated chat scene and scannet words instead of proper citations\n- The bold markings in Table-7 look wrong i.e. the bolded numbers are not the best numbers in the table", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TYdNV1JUYy", "reviewer": "Reviewer_oRKD", "summary": "This work proposes GPT4Scene, an approach to improve the performance of vision-language models (VLMs) by incorporating a 3D BEV mask and spatio-temporal object (STO) markers. The authors also introduce ScanAlign, a new dataset that includes these modalities and shows performance gains when used for fine-tuning. Experimental evaluations are conducted on 3D Question Answering, 3D Captioning, and 3D Visual Grounding tasks.", "questions": "1. (See Weakness 1) Could the authors provide more details on mask quality and generalization across datasets?\n2. Why does only Qwen2.5-VL-7B surpass the baselines in the question answering task? Relatedly, why is the improvement on SQA3D (situated understanding benchmark) marginal?\n3. The proposed dataset contains data exclusively from ScanNet. Would fine-tuning on more diverse 3D environments further enhance model performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Extensive quantitative experiments are provided.\n2. A new dataset, ScanAlign, is introduced, containing video frames, BEV images with STO markers, and text annotations.\n3. The use of spatio-temporal object markers is a well-motivated and effective idea that has not been explored before and yields improvements, particularly in 3D grounding tasks.", "weaknesses": "1. For 3D instance segmentation, the authors rely on Mask3D, which is pre-trained on ScanNet scenes.\nHow well do the predicted masks generalize to other 3D environments (e.g., ARKitScenes)?\n2. While the performance improvements are appreciated, the BEV images have been explored in prior work. Thus, the main methodological contribution lies in the introduction of STO markers, which somewhat limits the overall novelty of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fuXyIOFNn", "reviewer": "Reviewer_fXcL", "summary": "This paper presents a vision-only framework for indoor scene spatial understanding, GPT4Scene. The paper introduces two main innovations: (1) feeding BEV images into the VLM to provide global scene perception, and (2) assigning consistent object-level markers across the BEV view and multiple frames to establish correspondence between global and local observations. Experiments demonstrate that this approach enhances spatial understanding in large-scale VLMs (70B) under zero-shot settings, while smaller models (2B/7B) benefit more from fine-tuning with the proposed dataset.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers an insightful perspective: the key to 3D understanding lies not in explicit 3D geometry but in maintaining global-local consistency. Point clouds merely provide geometric constraints; a VLM can achieve “pseudo-3D understanding” by learning such consistency through BEV and object markers.\n2. The method is validated across models of various scales, and a fine-tuning dataset is constructed for smaller models.\n3. Comprehensive experiments and ablations are conducted on multiple benchmarks.\n4. The paper is well written, with clear figures and extensive supplementary material, making it easy to follow.", "weaknesses": "1. The suitability of BEV as a global representation for indoor scenes is questionable. BEV works well in outdoor settings (e.g., autonomous driving) because height variation is limited, making BEV a near-complete 2D representation. However, in indoor embodied scenarios, the global viewpoint changes dynamically as the robot moves, and BEV may miss critical global information. The authors should provide further justification or relevant experiments.\n2. Beyond benchmark performance, the paper would benefit from feature visualization or attention map analysis to offer theoretical insights and interpretability.\n3. Please include a comparison of computational overhead across components to help assess the method’s practical feasibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)