SpatialLM: Training Large Language Models for Structured Indoor Modeling
Authors: Yongsen Mao
Junhao Zhong
Chuan Fang
Jia Zheng
Rui Tang
Hao Zhu
Ping Tan
Zihan Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Structured Indoor Modeling
Large Language Models
Spatial Understanding
3D Reconstruction
Primary area: applications
TL;DR: This paper presents SpatialLM, a large language model designed to process 3D point cloud data and generate structured scene descriptions.

Abstract: SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs.

To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.

SPATIALLM: Training Large Language Models for Structured Indoor Modeling
Yongsen Mao!
Junhao Zhong' Chuan Fang² Jia Zheng'
Rui Tang'
Hao Zhu'
Ping Tan
Zihan Zhou'
Manyyore Tech Inc.
"Hong Kong University of Science and Technology
https http:://mmamcmeeeeeeeeeeee research. www.areh..tthuee................ github. io no/SpatialLM
Abstract
SPATIALLM is large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs.
To train SPATIALLM, we collect large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground- truth 3D annotations, and conduct careful study various modeling and training decisions. On public benchmarks, our model gives state-of-the-ar performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.
Introduction
3D indoor environments ubiquitous in our daily lives. We spend lot of time and perform various activities in such environments every day. Therefore, long-standing goal of artificial intelligence is teach machines to perceive, reason about, and interact with 3D indoor scenes humans do. In this work, we focus on the task of structured indoor modeling, which aims to extract structured indoor scene descriptions from raw sensorial inputs (i.e., RGBD scans). Specifically, the scene description includes both the architectural layout (i.e., walls, doors, and windows) and 3D object bounding boxes in the indoor environment. Such 3D structure information has been shown to benefit wide range of real-world applications such as scene editing 44)] augmented reality 2 and robot navigation [17]
Compared 3D formats such meshes, voxels, implicit functions, the structured scene description provides a highly compact yet flexible scene representation. this paper, our first contribution to regard the structured descriptions as scripts of ceneral-purpose language (i.e., Python) and propose to predict the language in text form, as illustrated in Figure This design choice has several advantages: (i) it is human interpretable and editable; (ii) it be easily extended to incorporate any new classes without affecting existing content in the scripts; and (iii) it allows us to leverage the strong built-in coding capability of pre-trained large language models.
Therefore, our goal to implement our method by directly fine-tuning open-source LLMs. While modern (multimodal) LLMs have revolutionized fields like natural language processing, 2D image understanding and generation, few work has attempted to use LLMs for 3D structured scene modeling. This is partly due to the lack of large-scale, high-quality datasets for the task. In this paper, second contribution, we collect a new synthetic dataset, which consists of the point clouds of 12,328
"qqual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Point Cloud
Projector (MALP)
Encoder
LLM
Narto
Detect wolls, doors,
The overall pipeline of SPATIALLM. Given the point cloud input, it employs standard Mancoder-MMLPLLL.. architecture for multimodal feature alignment (left), and generates structured scene descriptions in pure text form output (middle). The reconstructed 3D structure is further overlaid on the point cloud for visualization (right).
Figure 1:
distinct scenes (54,778 rooms) paired with 3D structure information, and conduct the first empirical study on the best strategy of aligning the point cloud input with LLMs for structured scene modeling.
Building upon our findings, we introduce SPATIALLM, large language model which is capable of processing point cloud input and generating structured scene descriptions. As our third contribution, we show that, by first training on our large-scale dataset and then on smaller downstream task data, model gives competitive performance detection, respectively. Additionally, we provide proof-of-concepl zero-shot experiments which show that model can handle point clouds from diverse sources such monocular video sequences, These results suggest that SPATIALLM may serve as a base for developing future solutions for scenarios where enhanced spatial understanding and reasoning is required (e.g., embodied Al).
public benchmarks for layout estimation and 3D object
Related Work
2.1 Structured Indoor Modeling
Structured indoor modeling aims to recover structural elements (i.e., walls, doors, and windows) as well as 3D objects from unstructured point clouds. In the literature, some studies focus on room layout estimation from point clouds. Early methods [37 38 rely traditional geometric analysis algorithms (i.e., RANSAC) to detect wall planes. SceneCAD I jointly optimizes the layout estimation and
CAD model alignment of the scanned scene. RoomFormer (60) proposes to train a Transformer-bast based
network to predict corners and rooms. Meanwhile, a number of recent methods [35 46, 51 47
have advanced the state-of-the-art on 3D object detection from point clouds.
The closest work to ours is SceneScript 2 which also casts structured indoor reconstruction as sequence modeling problem. It demonstrates great flexibility the language can be tailored to different scenarios without changing the method. But unlike our method, SceneScript uses domain- specific tokens for language commands, thus requiring specialized decoder. Instead, SPATIALLM strictly adheres to the Maacoder-MLP-LAA architecture and directly fine-tuned based on modern foundation models.
2.2 Large Language Models for 3D Scene Understanding
Recently, there has been a growing interest in empowering LLMs to understand and reason about 3D spaces, as defined in public benchmarks such as ScanRefer [9] and ScanQA [3]. One line of work [53] 20 19 29 2
first uses 3D detectors to locate objects in the point cloud, then extracts features of individual objects using a pre-trained object-level encoder. In this paper, we study how such 3D detectors be learned with modern LLMs.
Other methods directly extract scene-level features from the point cloud and align the features with 10 16 3D-LLM [88 uses pre-trained image encoder to obtain 2D features from
LLMs [8
66
multi-view images and back-project onto the point cloud (2 then leverages Q-former (8 to map them to a fixed number of tokens as input to the LLM. LL3DA [10] adopts 3D visual encoder

---

adataclaas
sdataclass
adataclass
edataclass
class
Window:
Bbox:
8_x: int
vall_id: str
wall.id: str pesition_x:
class: str
a.y:
position_x: poeition_y:
position.xx int
6.8: int
position_y: int
pesition_y: int
b_x: int
position_2: int width: int
pasition_z: int
position_ int
b-y:
vidth: int
angla_z: int
b_8: int
height: int
height: int
scale_X: int
beight:
scale_y: int
scale_z: int
Figure 2: Definition of our structured representation for layouts and objects.
pre-trained ScanNet detection the scene encoder. ScanReason 666 further employs 3D point cloud encoder to obtain fine-grained geometric information for accurate object grounding. Scene- LLM [[6] tokenizes the 3D point cloud features by simply dividing the space into fixed-resolution voxel grid and aggregating features in the same voxel.
Besides point cloud- based representations, recent studies have explored other ways integrate 3D information in LLMs. SpatialVLM [8] primarily focuses on synthesizing large-scale 3D VQA data to enhance the spatial reasoning capabilities of the VLM. LLaVA-3D [67] injects 3D position embeddings into 2D CLIP features to establish a unified architecture for 2D image understanding and 3D scene understanding. SpatiaIRGPT [11] extracts 3D scene graph from single image to enhance grounded spatial reasoning in L.Ms. Video-3D LLM
63 learns representations from video frames and 3D coordinates to leverage the strong priors of pre-trained 2D Video LLMs. LSceneLLM [65] uses an adaptive self-attention module to capture fine-grained details in local areas. Our work differs from these studies in that focus on analyzing the structural information of the scene.
SPATIALLM
Given the point cloud P (typically derived from RGBD scans), our goal is to recover the 3D layout L, which consists of all walls, doors and windows, and a set of objects O in the scene. As discussed before, we represent L and as scripts of pprrroe-purpose language, and fine-tune LLMs predict the language in text form. Figure shows all the parameters we use for the layout and objects.
To the best of our knowledge, from point clouds. Thus, we first propose new dataset suitable for training the LLMs on (Section Then, with the new dataset, cloud features to use for the structured indoor modeling task (Section 3.2 (ii) How to integrate the features into LLMs (Section 3.3)
the first employ LLMs for structured scene reconstruction
out investigate the following two key questions: (i) What point
3.1 Dataset and Metries
Table provides a list of popu- lar indoor RGBD scan datasets. We can make a few observa- tions:
Table 1: Quantitative statistics of indoor RGBD scan datasets.
annotations
Dataset (year)
#scenes
layouts objects
ScanNet (2017) 13
real
1,513
First, the number of scenes in real datasets relatively small, ranging from about one hundred to few thousands. To collect such a dataset, one needs take RGBD scans in real scenes and manually an- notate 3D information, which is highly laborious process. Further, most real datasets pro- vide annotations on the objects (i.e., semantic labels and 3D boxes) only. One exception is SceneCAD (.. which manu- ally adds wall annotations to ScanNet dataset. But it only contains 1,151 rooms.
Matterport3D (2017)
real
90
3RSean (2019) 50
real
1,482
SceneCAD (2020)
real
1,151
ARKitScenes (2021)
real
5,048
MultiScan (2022)
34
real
273
ScanNet++ (2023)
59
real
1,006
HM3DSem (2023)
58
real
181
Structured3D (2020)
64
syn. 3,500
Hypersim (2021)
45
syn.
461
ProcTHOR (2022)
sym.
12,000
HSSD (2024) 114
syn. 211
ASE (2024) 2
syn.
100.000
SPATIALLM dataset (ours)
syn.
12,328

---

ProcTHOR 14
HSSD 24
ASE 2
SPATIALLM dataset
Figure 3: Dataset visual quality comparison. The layout and object placements in ProcTHOR and ASE 2 are Program-generale which exhibit noticeable differences from real-world statistics. The scenes HSSD [2 and our dataset are fully human-and auman-authored. But HSSD only has 21 scenes.
Second, for synthetic datasets, it is much cheaper to obtain ground-truth 3D annotations. But to ensure the realism of the RGBD scans, the 3D scenes must professionally constructed. As result, the diversity of 3D scenes in some datasets is still limited. For example, Hypersim [45] and HSSD (44 only have 461 and 21 scenes, respectively. On the other hand, one may create large number of scenes by employing rule-based procedural modeling pipeline, as did in ProcTHOR [4] and ASE 2 But the quality of the scenes is not as high.
SPATIALLM dataset. In this paper, we make an effort to build a dataset suitable for training LLMs for structured indoor scene reconstruction. To this end, take advantage of access to a large repository of interior designs from an online platforn the interior design industry. Most designs are created by professional designers and used for real-world production. We parse each 3D house into individual rooms and use few rules to filter the rooms. This results in 12,328 distinct scenes with 54,778 rooms.
For the objects, we keep annotations on a selected set of 59 common categories (excluding wall, door, and window), and filter small objects with side lengths all < 15cm. In the end, dataset contains 4129932 annotated object instances of 35,426 unique CAD models.
For each scene, generate photo-realistic RGBD images utilizing an industry- rendering engine. At the time of rendering, camera trajectory traversing each room simulated, based on which the images are taken at an interval of 0.5 m. Figure 3 compares compares the visual quality of our dataset to several recent synthetic datasets. Finally, the dataset is divided into 11,328/500/500 scenes for wwaiingg/alidation/eestnggg
Evaluation metrics. We evaluate our method on both layout estimation and object detection. For layout estimation, L = {4
y :==1 and )4 denote the set predicted structural elements and ground truth, respectively. Since each element is plane segment in 3D space, we compute the 2D
intersection-over-niii (loUzp) by projecting each predicted element to the plane of the ground-trume element find the optimal assignment with the Hungarian algorithm, and report the Fl-scores at 0.25 and 0.5 thresholds. For 3D object detection, truth O = { via Hungarian matching. Then, we compute the 3D intersection-over-ume (IoUnn) between each matched pair and report the Fi-scores at 0.25 and 0.5 thresholds.
match the predictions Ô {ō.) with the ground
3.2 Point Cloud Encoders
Extracting meaningful representations for a point cloud challenging because of its irregular format. Learning-base approaches process 3D point clouds can be roughly classified into three groups: Mapping-bor baseeng-based methods are inspired by the success 2D image backbones such as CLIP (44) and the DINO series [6. 39] and map pixel-aligned image features to 3D. For example, Lexicon3D 33
httpps:/www..tr http://wwwwjiaal.coo

---

Table Experiment on the encoder design.
Params (M)
F1 layout
Fl object
Encoder
IoUn @0.25
IoU @0.5
IoUnD @0.25 loUn @0.5
Voxelize (P+DINOV2)
0.0
10.8
5.5
0.1
0.0
Rand. sampling (P+DINOv2)
0.0
10.6
4.9
0.2
0.0
3DCNN enc. (P)
10.5
79,4
76.3
59.8
46.1
3DCNN enc. (P+DINOv2)
11.6
81.5
79.4
57.1
45.0
3DCNN enc. (P) + Voxelize (DINOV2)
11.9
83.8
81.6
62.9
46.7
Sonata/PTvv3 enc. (P)
108.8
84.6
82.6
65.1
49.4
projects 3D points onto the image plane to obtain their corresponding pixel features; ConceptFu- sion 21 builds multimodal 3D maps via traditional SLAM and multi-view fusion approaches. Voxel-based methods first transform irregular point clouds to regular representations via 3D vox- elization. Then, sparse convolution algorithms [12] can be readily used for efficient computation. Point-based methods, by contrast, process point clouds directly sets embedded in continuous space. Recent approaches in this category [62 54 have successfully explored the self-attention mechanism in Transformers and demonstrated superior performance on downstream applications.
Note that all methods discussed above produce one feature per point by default. Given that a scene- level point cloud consists of hundreds of thousands of points or more, these features are too much in terms of quantity for LLMs consume directly. Since most modern open-source LLMs take an "Encoder-MLP MML--MLP-LLMM architecture for multimodal feature alignment (e.g., the Llama series [2] and Qwen series [42 4 ), we can define point cloud encoder for LLMs mapping which takes set of N points as input and generates K feature embeddings, where K < N:
F= E(P) P F FERXXD
Here, each input point is 6-dimensional vector (XYZ and RGB), and D is the feature dimension.
Recently, a few attempts have been made to convert point representations into feature embeddings [2] 16]. However, this step has not been carefully investigated. In this section, we conduct experiments to study (i) the effectiveness of different encoders, and (ii) the impact K, Le., the number of output features. Unless otherwise stated, Qwen2.5-0.53 42] is employed as our base model. We use Sonata (56) as the point cloud encoder, and a two-layer MLP as the projector.
Experiment on the encoder design. We consider six possible point cloud encoders. The first group of encoders adopts a mapping- based method to fuse 2D image features 3D points. Following Lexicon3D (33), we pool pixel-level DINOv2 features to form 3D point representations. Then, one of the two routes may be taken to reduce the points into manageable length of tokens:
Voxelize (P+DINOv2): divides the space into a fixed-resolution voxel grid and aggregates all the points in the same voxel - similar to the approach used in Scenc-LLM [16];
Rand. sampling (P+DINOv2): uses Farthest Point Sampling [1 fixed number of points with their features.
randomly sample
From Table we can see, these two options result in extremely low Fl-scores. We hypothesize that this is because too much spatial information is lost during the naive down-sampling step. While this may be acceptable for certain applications, such as visual Q& it is problematic for our task, which relies heavily on the spatial information reconstruct the geometric coordinates of objects.
The second group of encoders adopts a standard 3DCNN encoder to generate pooled features. We follow SceneScript 2 to build our encoder with sparse 3D convolution library (48). Here consider three variants:
3DCNN enc. (P): takes raw points (6-dim XYZRGB values) as input;
3DCNN enc. (P+DINOv2): takes the points with the associated DINOv2 features as input;
3DCNN enc. (P)+ Voxelize (DINOv2): concatenates the output 3DCNN with aggregated DINOv2 features.
5

---

As shown in Table learned encoder is effective in keeping the necessary semantic and geometric information for reconstruction. We also find that incorporating DINOv2 features yields slightly better results, possibly due to the enhanced contextual information.
The last encoder consider is Sonata [56], variant of Point Transformer V3 (PTv3) [55] which removes the decoder and focuses on self-supervised learning on the encoder. Compared to other backbones, such an "encoder-only" design makes it quite convenient for adapting LLMs. Indeed, as shown in Table 2 with raw points as input, it already achieves the best results among all encoders.
Experiment the spatial resolution. Recall in Eq. (I) that the parameter K denotes the number of feature embed- dings visual tokens, which critical to the performance of LLMs. In this experiment, we study the effect of K.
Table 3: Experiment on the spatial resolution. Avg. (K) indicates the average value of K under the corresponding resolution.
Fl layout
Fl - object
Res.
Avg. (K)
IoU20 @0.25 IoUn @0.5
IoUn @0.25
1oU @0.5
1x
123
84.6
82,6
65.1
49.4
1.25x
197
84.8
82.6
65.3
52.2
However, fixing K is difficult because different point clouds generally produce varying K values. Instead, since the en- coders employ coarse-to-fine structure for point cloud pro- cessing,
1.5x
286
86.9
85.1
68.7
58.6
2x
510
87.4
85.3
71.6
61.1
2.5x
788
86.1 1
83.9
72.2
61.7
4x
2,000
85.8
83.7
69.9
58.9
the resolution the finest level 5em and use 5-level hierarchical structure by default. A detailed illustration of the point cloud encoder hierarchy is presented in the appendix (see Figure IT]. Then, gradually increase the resolution up 4x, and report the results in Table As can be seen, model performance steadily improves from x to x resolutions, but further increasing the resolution hurts the performance. This likely due to the excessively long token sequences.
3.3 Training Schedules
As most recent MLLMs start from pre-trained unimodal backbones, how to align the multimodal content with the language model plays a critical role in the performance of the resulting MILMs. For example,
Table 4: Experiment on training schedule. For each training stage, we use three circles to represent the three components the FEncoder-ML.P-LL.MM architecture. "o" and "*" indicate whether component is frozen or trainable, respectively.
Training stages 1st 2nd
Fl layout
F1 object
common practice adopted by many MLLMs [30 (49 is the inclu- sion of two-stage training an alignment stage that trains the randomly initialized MLP projector only, followed by (ii) fine-tuning stage that trains both projector and language model while keeping the vision encoder frozen. Other studies (e.g., (23) advocate a single-stage training protocol, suggesting that including the projector training stage is unnecessary. Furthermore, when adapting the LLMs to rich visual content such as long videos, some research [68] has shown that three-stage pipeline, which progressively unfreezes different components in different stages, improves the performance.
1o
IoUn @0.5
IoUsp @0.25
IoUpp 0..
68.4
40.2
84.6
82,6
65.1
49.4
schedule:
70.9
62.7
27.4
13.7
85.3
83.4
60.7
45.6
76.9
70.1
31.1
16.0
83.2
80.3
57.1
41.2
In this experiment, we test six possible training configurations. ranging from one-stage to three-stage, to evaluate the influence of different training strategies. As shown Table 4 we found that single-stage fine-tuning yields the best results. Adding more stages generally degrades the model performance. We also found that making parameters trainable single stage is important, especially for 3D object detection. This may indicate that, compared modern image encoders such

---

as SigLIP 61 and DINOv2 [39] current pre-trained point cloud encoders are not versatile in supporting downstream tasks yet.
4
Experiments
In this section, we compare SPATIALLM to the current state-of-the-art on scene layout estimation (Section 4.1) and 3D object detection (Section 4.2). respectively. Following our empirical study, we use Qwen2.5-0.55 --5.BBB 42 as the base model and Sonata [56] as the point cloud encoder. We set of resolution at the finest level at 2.5cm and employ single-stage training on our dataset.
Besides testing on established RGBD benchmarks, we also investigate whether our model can handle inputs from other sources, such point clouds reconstructed from RGB videos (Section Other extensions and more technical details can be found in the appendix.
4.1 Layout Estimation
Layout estimation focuses on predicting architectural clements, i..,, walls, doors, and windows, within an indoor scene. In this experiment, we include results from two representative basclines, namely RoomFormer [60] and SceneScript [2]. As the state-of-the-arr specialist model for layout estimation, RoomFormer employs highly specialized network architecture, using two-level queries to predict room polygons and their corners, respectively. SceneScript, on the other hand, casts 3D structure estimation as an auto-regressive tnext-token-pee pokken-reddcctonn problem. However, unlike our method, it uses specialized structured language commands to describe the scene and trains Transformer decoder from scratch.
Following the setting of RoomFormer, we perform evaluation on the Structured3D benchmark [64]. which contains 3,500 residential houses with diverse floorplans. We use the original data split of respectively. For RoomFormer, we directly download
3000/250/250 for the model checkpoint from its website" for inference. Note that RoomFormer takes in 2D density maps projected from point clouds and outputs 2D layouts, which are converted to 3D by extruding based on the lowest/highest points in the point cloud. For SceneScript, we download the pre- me-trained model and c and fine-tune it on Structured3D with the default hyperparameter settings.
For our model, include three variants: (i) trained Structured3D only, (ii) trained on our dataset only, and (iii) first trained on our dataset, then fine-tuned on Structured3D. As can be seen Table 5 directly training LLMs smaller Structured3D dataset leads to poor performance. In contrast, by first training on our new dataset and then on Structured3D, our model outperforms both baselines for this task. This demonstrates the benefits of our new dataset.
Table 5: Experiment on layout estimation.
Params (M)
Structured3D loUnn IoU @0.5
Method
RoomFormer
42.2
83.4
81.4
SceneScript
29.9
90.4
89.2
SPATIALLM (ft. Structure33D)
603.5
32.6
18.1
SPATIAL.LM (ft. Ours)
603.5
59.9
44.7
SPATIAL.LM (ft. Ours + Structured3D)
603.5
94.3
93.5
Figure provides some qualitative comparisons. Since RoomFormer detects room elements inde- pendently, we can see that some doors and windows are not attached to any wall. Auto-regressive methods like SceneScript and our method can preserve the relationships between structures by design. But we observe that SceneScript tends to miss some elements in its output.
4.2 3D Object Detection
Next, we compare SPATIALLM to existing methods 3D object detection. We include results from two representative baselines, namely V-DETR 47 and SceneSeript. As the otate-of-the-art
7

---

Ground truth
RoomFrrmer 660
SceneScript 2
SPATIALLM
Figure Qualitative results on Structured3D dataset.
specialist model for the task, V-DETR builds upon the DETR framework (5] and makes several task- specific improvements, including 3D vertex relative position encoding (3DV-RPE), boject-based normalization, among others. Following prior practice, we perform evaluation on ScanNet [I], which contains 1,513 3D indoor scans with annotations of 18 object categories. The training set is composed of 1,201 scenes, while 312 scenes are used for validation/testing. Since auto-regressive models (i.e., SPATIALLM and SceneScript) do not produce confidence scores, report Fl scores instead of mean Average Precision (mAP), as suggested
For V-DETR, we directly use the model checkpoint down- loaded from its w A con- fidence threshold of 0.5 is ap- plied for both objectness and semantic predictions, followed by non-maximum suppression (NMS). For SceneScript, we fine-tune the model on ScanNet with the default hyperparameter settings. During fine-tuning, apply the same data augmenta- tions as used in PTv3 [55] and V-DETR 47
Table 6: Experiment on 3D object detection. "** numbers reported on the overlapping subset of object classes between our dataset and ScanNet.
Params (M)
ScanNet IoUn @0.25
F1
Method
loUsn @0.5
V-DETR 47
.
65.1
56.8
SceneScript
29.9
49.1
36.8
SPATIALLM (ft. ScanNet)
603.5
2.9
0.7
SPATIALLM (ft. Ours)
603.5
33.8"
22.6"
SPATIALLM (ft. Ours + ScanNet)
603.5
65.6
52.6
For our model, we again include three variants: (i) trained on ScanNet only, (ii) trained on our dataset only, and (iii) first trained on our dataset, then fine-tuned on ScanNet. As can seen in Table 6 the first variant has very low Fl scores, suggesting that ScanNet itself is too small to train LLMs on. By first training on our new dataset and then on ScanNet, our model outperforms SceneScript and achieves competitive results against the state-of-the-an specialist V-DETR.
Figure 5 shows some qualitative results. Compared to SceneSeript, SPATIALLM demonstrates improved detection accuracies across categories, In comparison to V-DETR, we observe that our method has some difficulty in locating objects in "picture", "sink" and "showercurtain" classes. Notably, "picture" and "sink" are the smallest objects in ScanNet by volume (< 0. 0.1m²), whereas "showercurtain" category is not included our dataset and has the fewest occurrences in the ScanNet.
httpp://ggthuu..oo//VDDTTT

---

Ground truth
V-DETR 447
SceneScript 2
SPATIALLM
Figure 5: Qualitative results on ScanNet dataset.
Figure 6: Qualitative results of zero-shot detection on videos.
4.3 Zero-shot Detection on Videos
Although significant progress has been made in SFM and SLAM techniques over the past decades, reli- ably inferring 3D structure from videos remains highly challenging problem. Recently, DUSt3R [52] and its extensions (27 15 36 31 provide novel solution that employs deep networks for simultane- ous point matching and camera motion estimation. These techniques are shown to be remarkably robust in dealing with real-world videos unconstrained image collections.
In this experiment, we collect set of 107 videos of virtual indoor scene tours and use MASt3R- SLAM 36 to reconstruct point cloud from each video. Then, we test SPATIAL.LM in zero-shot setting on the reconstructed point clouds. Unlike RGBD scans, these reconstructions have sigmificant noises, occlusions, and geometric artifacts, posing greater challenge for structured 3D understanding.
Figure 6 shows some qualitative results. Without fine-tuning, SPATIALLM demonstrates strong resilience to imperfect data and preserves consistent layout and object-level predictions. Trained on synthetic datasets with accurate bounding boxes, the model infers full object sizes and orientations from sparse or occluded inputs. In the first example, it predicts beds and nightstands extending to the floor, even when those regions are not visible. Leveraging the auto-regressive nature of LLMs,, SPATIAL.MM is also capable of generating plausible room layouts from partial observations. In the second example, the model reasonably reconstructs areas such as the balcony and dining space, filling in missing clements based contextual understanding.

---

5 Conclusion
In this study, we demonstrate the feasibility of training LLMs for structured indoor modeling tasks. We regard it meaningful step towards building future foundation models that can not only understand, but also reason about, interact with, and even create structured 3D scenes.
Limitations. SPATIALLM has several limitations. First, it falls short of delivering a universal, ofate-of-the-art model for extracting 3D structure from arbitrary point clouds. There are significant differences between point clouds from different sources, such as monocular videos, RGBD scans, and LiDAR sensors. As shown in Section 4 although our model exhibits reasonable generalizability across datasets, fine-tuning on specific dataset is still needed to achieve the best performance. Further scaling up the data and model size could be promising future direction. Second, in this paper, we focus on training LLM for structured indoor modeling only, without considering how affects the model's 's natural skills in natural language processing and reasoning. To address this, more thorough benchmarking may be performed. Third, our current approach models indoor layouts using a set of predefined object categories, limiting the LLMs in leveraging their open-ended language capabilities. Future directions include extending our work to support open- voccbubbyy object detection and 3D visual question answering (VQA), enabling more flexible and generalizable scene understanding.
Impacts. We envision that SPATIAL.LM will be used in real-world applications such as layout estimation and 3D object detection. For example, people may take our code and fine-tune the model on their own datasets. On the methodology side, we hope that our work will inspire future studies on MLLM models for 3D scene understanding, reasoning, and creation.
Acknowledgments and Disclosure of Funding
This work was partially supported by the Key R&D Program of Zhejiang Province (2025C01001) and the HKUST project 24251090T019.
References
[1] Armen Avetisyan, Tatiana Khanova, Christopher B. Choy, Denver Dash, Angela Dai, and Matthias Nießnnr. SceneCAD: Predicting object alignments and layouts RGB-D scans, In ECCV, pages 692, 2020.
[2] Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Y Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan P. Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard A. Newcombe, and Vasilcios Balntas. SceneScript: Reconstructing scenes with autoregressive structured language model. In ECCV, pages 247-263, 2024.
[3] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. ScanQA: 3D question answering for spatial scene understanding. In CVPR, pages 19107-19117, 2022.
[4] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitScenes: A diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data. In NeurIPS Datasets and Benchmarks, 2021.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self self-supervised vision transformers. In ICCV, pages 9630-9640, 2021.
[7] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias NieBner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. In 3DV, pages 667-676, 2017.
10

---

[8] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. SpatialVLM: Endowing lanioae models with spatial reasoning capabilities. In CVPR, pages 14455-14465, 2024.
[9] Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. ScanRefer: 3D object localization RGB-D scans using natural language. In ECCV, pages 202-221, 2020.
[10] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu. Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LLIDA: visual interactive instruction tuning for omni-3d understanding, reasoning, and planning. In CVPR, pages 14418-26428, 2024.
[1I] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo. Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. SpatiaIRGPT: Grounded spatial reasoning in vision-language models, In NeurIPS, pages 25002--35093, 2024.
[12] Christopher B. Choy, Jun Young Gwak, and Silvio Savarese. 4D spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, pages 9075-3084 2019.
[13] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. ScanNet: Richly-amotated 3D reconstructions of indoor scenes. In CVPR, pages 5828-5839, 2017.
[14] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-scale embodied AI using procedural generation. In NeurlPS, pages 5982-5994, 2022.
[15] Bardienus Pieter Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, and Jéröme Revaud. MASOR-SMM: a fully-integrated solution for unconstrained structure-from- motion. In 3DV, 2025.
[16] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-LLM: Extending language model for 3d visual reasoning. WACV, pages 2195-2206, 2025.
[17] Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Wang. Vision-and-language naviga- tion: A survey of tasks, methods, and future directions. In ACL, pages 7606-7623, 2022.
[18] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: Injecting the 3d world into large language models. In NeurlPS, pages 20482-20494, 2023.
[19] Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng. Yang Zhao, Jiangmiao Pang, and Zhou Zhao. Chat-Scene: Bridging 3d scene and large language models with object identifiers. NeurlPS, pages 113991-114017 2024.
[20] Jiangyong Huang, Silong Yong. Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML,, pages 20413-20451, 2024.
[21] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd. Omama, Ganesh Iyer, Soroush Saryazdi, Tao Chen, Alaa Maalouf, Shuang Li, Nikhil Varma Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. ConceptFusion Fusion: Open-set multimodal 3d mapping. In RSS, 2023.
[22] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Verse: Scaling 3d vision-language learning for grounded scene understanding. In ECCV, pages 289-310, 2024.
[23] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the design space of visually-condttioned language models. In JCML,, 2024.
[24] Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat Synthetic Scenes Dataset (HSSD-200): Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation. In CVPR, pages 16384-16393, 2024.
11

---

[25] Maksim Kolodiazhnyi. Anna Vorontsova, Matvey Skripkin, Danila Rukhovich, and Anton Konushin. UniDet3D: Multi-dataset indoor 3d object detection. In AAAI, pages 4365 4373, 2025.
[26] Black Forest Labs. Flux. http::///gggtu.oo///////oooootte
2024.
[27] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with MAS13R. In ECCV, pages 71-91, 2024.
[28] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, pages 19730-19742, 2023.
[29] Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3D-LLM: Unifying point cloud perception, generation and editing with large language models. CoRR, abs/2402.03327, 2024.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, pages 34892-34916 2023.
[31] Yuzheng Liu, Siyan Dong, Shuzhe Wang, Yanchao Yang, Qingnan Fan, and Baoquan Chen. SLAM3R: real-time dense scene reconstruction from monocular RGB videos. In CVPR, pages 16651-16662, 2025.
[32] LLama Team. The Llama 3 Herd Models. CoRR, abs/2407.2178.. 2024.
[33] Yunze Man, Shuhong Zheng. Zhipeng Bao, Martial Hebert, Liangyan Gui, and Yu-Xiong Wang. Lexicon3D: Probing visual foundation models for complex 3d scene understanding. In NeurIPS, pages 76819-76844 2024.
[34] Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel X. Chang, and Manolis Savva. MultiScan: Scalable RGBD scanning for 3d environments with articulated objects. In NeurlPS, pages 9058-9071, 2022.
[35] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. ICCV, pages 1186-2897, 2021.
[36] Riku Murai, Eric Dexheimer, and Andrew J. Davison. MASAR--L Real-time dense SLAM with 3D reconstruction priors. CVPR, pages 16695-16705, 2025.
[37] Srivathsan Murali, Pablo Speciale, Martin R. Oswald, and Marc Pollefeys. Indoor Scan2BIM: Building information models of house interiors. In IROS, pages 6126-66 6133, 2017.
[38] Sebastian Ochmann, Richard Vock, and Reinhard Klein. Automatic reconstruction of fully volumetric 3d building models from point clouds. CoRR, (bss/997.00661, 2019.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Ro-Yao Huang, Shang-War Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOV2: Learning robust visual features without supervision. TMLR, 2024.
[40] Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. Advances in data-driven analysis and synthesis of 3d indoor scenes. Comput. Graph. Forum, 43(1), 2024.
[41] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. PointNet++: Deep hierarchical feature learning on point sets in metric space. NeurIPS, pages 5099-5108, 2017.
[42] Qwen Team. Qwen2.5 Technical Report. CoRR, abs/2412. 2024.
[43] Qwen Team. Qwen2.5-VL Technical Report. CoRR, abs/2502.13922 2025.
12

---

[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., pages 8748-8763, 2021.
[45] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, pages 10912-10922, 2021.
[46] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. FCAF3D: fully convolutional anchor-free 3d object detection. In ECCV, pages 477 2022.
[47] Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, and Baining Guo. V-DETR: DETR with vertex relative position encoding for 3d object detection. In ICLR, 2024.
[48] Haotian Tang, Zhijian Liu, Xiuyu Li, Yujun Lin, and Song Han. TorchSparse: Efficient point cloud inference engine. In MLSys, pages 302-315, 2022.
[49] Peter Tong. Ellis Brown, Penghao Wu. Sanghyun Woo, Adithya Iyer, Sai Charitha Akula, Shusheng Yang. Jihan Yang. Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurlPS, pages 87310-87356, 2024.
[50] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. RIO: 3d object instance re-localization in changing indoor environments. In ICCV, pages 7657-7666, 2019.
[51] Haiyang Wang, Lihe Ding, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, and Liwei Wang. CAGroup3D: Class-aware grouping for 3d object detection point clouds. In NeurlPS. pages 199975-29988, 2022.
[52] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. DUSt3R: Geometric 3d vision made easy. In CVPR, pages 20697-20709, 2024.
[53] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, Tao Jin, and Zhou Zhao. Data- efficiently learn large language model for universal 3D scene perception. In NAACL, pages 313-333, 2025.
[54] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point Transformer V2: Grouped vector attention and partition-basee based pooling. In NeurIPS, pages 33330-33342, 2022.
[55] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang. Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point Transformer V3: Simpler, faster, stronger. CVPR, pages 4840-4851, 2024.
[56] Xiaoyang Wu, Daniel DeTone, Dunean P. Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob J. Engel, Richard A. Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: sulf-supe learning of reliable point representations. In CVPR, pages 22193-22201 2025.
[57] Jianfeng Xiang. Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In CVPR, pages 21469-21480,, 2024.
[58] Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Théophile Gervet, John M. Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, Alexander William Clegg, and Devendra Singh Chaplot. Habitat-matterprrt 3d semantics dataset. In CVPR, pages 4927-4936, 2023.
[59] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, pages 12-22, 2023.
[60] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the dots: Floorplan reconstruction using two-level queries. In CVPR, pages 445-854, 2023.
13

---

[61] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language mmgge pre-training. In ICCV, pages 11941-11952. 2023.
[62] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H. Torr, and Vladlen Koltun. Point Transformer. In ICCV, pages 16239-16248, 2021.
[63] Duo Zheng. Shijia Huang, and Liwei Wang. Video-3D LLM: Learning position-aware video representation for 3d scene understanding. CVPR, pages 8995-9006, 2025.
[64] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3ID: A large photo-realistic dataset for structured 3d modeling. ECCV, pages 519-535, 2020.
[65] Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, and Chuang Gan. LSceneLLM: Enhancing large 3d scene understanding using adaptive visual preferences. CVPR, pages 3761-3771, 2025.
[66] Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. ScanReason: Empowering 3d visual grounding with reasoning capabilities. In ECCV, pages 151-168, 2024.
[67] Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. LLaVA-3D: A simple yet effective pathway to empowering Imms with 3d-awareness. CoRR, abs/2409.18122 2024.
[68] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mchta, Tong Xiao, Philippe Estrucn Licheng Yu, Xiaofang Wang. Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, and Xide Xia. Apollo: An exploration of video understanding in large multimodal models. In CVPR, pages 18891-18901, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented the abstract and introduction are well-supported by the paper's contributions. We use figure (Figure T to illustrate the technical scope of SPATIAL In the subsequent sections, we introduce a new dataset and conduct compre- hensive experiments support these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all necessary details to enable reproduction of our experimental results, including dataset construction, evaluation metrics, and training settings in Sections
4 Data augmentation strategies, hyperparameters, and additional training configurations are included in the supplementary material. Our dataset and model will be made publicly available to support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have made our code, model, and dataset publicly available. All resources are available at
research. aerssarchh.gthhb..... 1o 1o/SpatialLM/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips /nips.cc/puu mublic/guiden
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly describe the dataset curation process and data splits in Section 3.1 The splits used for public datasets, along with details about the checkpoints for each method and evaluation settings, are provided in Sections 4.1 and 4.2 Additional information on data augmentation, hyperparameters, and training configurations included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We use well-established metrics for all experiments in this paper. Specifically, report the IoU2n at 0.25 and 0.5 thresholds for layout estimation, and loUsD 0.25 and 0.5 thresholds for 3D object detection. These metrics are calculated by averaging over all samples in the test set.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of the computational resources used are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp//////eerrpps.popubbbcccele
Answer: [Yes].
Justification: We have reviewed the Code of Ethics and confirm that our research conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

Justification: We discuss broader impacts of this work in Section 5 The authors do not perceive any negative socictal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide links the source codes used in the paper, and the online platform from which our dataset is collected. The license and terms of use are properly followed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The dataset curation process is clearly described in Section 3.1
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hatoss///earrips.c//offffeeeeree for what should or should not be described.
21

---

A SPATIALLM Dataset
SPATIALLM dataset comprises 12,328 distinct scenes with 54,778 rooms, featuring a total of 403,291 walls, 123,301 doors, and 48,887 windows, with an overall floor area of approximately 863,986 m²
Object categories. We have curated set of 59 commonly occurring object categories, organized by functional and semantic similarity, shown in Table 7 Statistics of these categories can be found in Figure
Table 7: Categories in the SPATIALLM dataset.
Super Categories
Categories
Seatings
sofa, chair, dining chair, bar har_chair, stool bed, pillow
Beddings
Cabinetry
wardrobe, nightstand, tv tv_cabinet, wine wine_cabinet, bathroom_ caarooet_eeettt,,eooee shoe thoe_cabinet, entrance_cahing decorative cetraiivveccbieet,,,. washing_ cabinet, wall_cahinet, sideboard, cupboard
Tables
coffee table, dining table, side table, dressing table, desk
Kitchen appliances
integrated_stove gas_stove range_hood, microwave_ovee sink, stove, refrigerator
Bathroom fixtures
hand_sine shower, shower_room, toilet, tub illumination, chandelier, floor_stanne snor__tanding__aar
Lighting
Decoration
wall decoration, painting, curtain, carpet, plants, potted bonsai tv, computer, air conditioner, washing machine
Electronics
Others
clothes rack, mirror, bookcase, cushion, bar, screen, combination sofa, dining table combination. leisure table Lable_and coaaiirtomiinaionn multifunctional combination bed
Figure & plots the object-to-room correlation heatmap, which reveals strong correlations between object distribution and specific room types. Additionally, each room type contains a variety of object categories, highlighting the diversity of our dataset.
Dataset visualizations. We provide visualizations of the ground-truth point cloud and layout annotations in Figure 2 showcase the quality of the photo-realistic scans and the realism of the layout. Additionally, we show rendered images of objects in Figure to illustrate the diversity and quality of the objects our dataset.
B Implementation Details of SPATIALLM
In Figure 12] shift the point cloud and the corresponding layout and object box coordinates ensure that they are all non-negative. Next, we quantize the coordinates into 1,280 bins, each with a resolution of 2.5cm. LLM predicts integer values, which are mapped back to continuous values using the inverse transformation and normalization, restoring them to the original coordinate system.
we show the full prompt and an example response of SPATIALLM. Note that we
Data augmentations. We apply number of augmentations the point cloud, which summarized in Table and explained below.
We begin with cuboid crop following V-DETR 47
This process involves randomly selecting point as the center of the cuboid and then determining random dimensions for the cuboid relative to the input point cloud size. We maintain minimum aspect ratio of 0.8 and ensure that at least 50,000 points remain after the cropping process.
Then, apply four types of jitter noises to the point cloud: (i) jitter that simulates the minor uncer- tainties inherent sensor accuracy; (2) jitter that replicates noise in areas with greater uncertainty; (3) jitter that reflects noise along the edges, typically from floating points between foreground objects and the background; and (4) jitter that represents distant floating points outside the room, often attributed to glossy windows, reflective mirrors, and points along the path to the outdoors.
22

---

Staee
5000000
meal ber
scseer
nn cas
bar
LooE parcoms
6.94
Sal
Ciee
musees coreent
Figure 7: Statistics of the SPATIALLM dataset. Top:
Number of objects in each category. Bottom:
Distribution of physical sizes in each category. We compute the object size as the diagonal length of the object's bounding box (in meters). All objects in our dataset have realistic physical sizes.
tum lãbr comb.
s0b
chair
clar
Cbar
wire
bath cal
or col
cal phb don ce
thens latee
Irigge
enceeer
Showed pm
taket
dam
plents
camputee
bostcas
lcmen conb.st sots di
Liing Denins Room
Living loon
Drinng Room
Recrram
Litdd hurm
Serior Room
sincc
Figure 8: Room object correlation of SPATIALLM dataset.
Compute resources. We train SPATIALLM for epochs with a total batch size of 64. The learning rate is set at 10-4, using cosine scheduler with warm-up ratio of 0.03. The parameters for A optimizer are as follows: adtll betal is 0.9, adam beta2 0.99, and adam adam_epsilon is 1 x 10 We utilized 32 NVIDIA H20 GPUs, and training SPATIALLM dataset takes approximately
day.
For experiments on Structured3D 64 and ScanNet we use batch size of 8. Fine-tuning on Structured3D involves 50 epochs, while keeping the other settings the same. For fine-tuning on ScanNet, we introduce an additional pre-training stage mitigate the difference in label spaces across datasets. Specifically, map the classes in our dataset to those of ScanNet, and further train model on our dataset for epochs, followed by fine-tuning on ScanNet for epochs.
C Supplementary Results for SPATIALLM Experiments
In this section, we present per-category quantitative results for (i) layout estimation, (ii) 3D object detection, and (iii zero-shot detection on videos.
23

---

Table 8: Point cloud data augmentations.
Method
Parameters
cuboid crop
mii__pontt=50.00 aspect=0.8, min_crop=0.75, max_crop=110, p=0.1
random jitter
o=0.025, clip=0.05, ratio=0.8, p=0.9
random jitter
o=0.2, clip=0.2, ratio=0.05,, p=0.8
random jitter
c=0.4,clip=14, clip=1.0, ratio=0..011,==0.7.
random jitter
o=0.5, clip=4.0, ratio=0.0005, p=0.65
elastic distort
params=[[0.2, 0.4]. [0.8, 1.6]]. p=[0.8. 0.5]
random rotate
axis=z, angle=[0, 90, 180, 270]
random scale
scale=[0.75, 1.25]
auto contrast
p=0,2
chromatic translation
p=0.75, ratio=0.1
chromatic jitter
std=0.05; p=0.8 p=0.1
color drop
C.1 Layout Estimation
Per-category results. In Table 9 we report the scores in percentages 0.25 and 0.5 thresholds of IoUp for architectural elements ie., walls, doors and windows, and the averaged score across the categories on Structured3D test set.
Table 9: Experiment results on layout estimation.
IoUn @0.25
loUn
Method
avg.
wall door
window
avg.
wall
door
window
RoomFormer 66
83.4
86.0
85.1
78.9
81.4
84.6
83.1
76.6
SceneScript [2]
90.4
92.9
92.8
85.5
89.2
91.9
92.5
83.3
SPATIALLM (ft. Structured3D)
32.6
19.4
24.3
18.1
33.5
11.5
SPATIALLM (ft. Ours)
59.9
71.5
42.7
65.6
44.7
64.8
39.7
29.6
SPATIALLM (ft. Ours Structured3D)
94.3 95.3
95.9
91.7
93.5
94.5
95.7
90.2
C.2 3D Object Detection
Per-category results. We report the Fl scores percentages at 0.25 and 0.5 thresholds of IoUsD
for 18 ScanNet categories and the averaged score across the categories in Table 10 and Table I1 respectively.
For IoUnD @0.25, we have +16.5% overall gain compared to SceneScript, and +0.5% difference compared to V-DETR. For IoUnD @0.5, have +15.8% overall gain compared to SceneScript, and -4.2% difference compared to V-DETR. The largest gaps between our model and V-DETR correspond to "pieture", "sink", and "shower_curtain" Note that these are either the smallest the least occurring objects in SeanNet.
Table 10: 3D object detection results with IoUnn @0.25. 1; trained on ScanNet only. trained on our
dataset only, "n/a" indicates the category is not included in our dataset.
bookshelf
avvrraee
bathaab
cabinet
curtain
2atNin.
refrige.
s.umain
windeww
desk
door
table
toiket
Method
V-DETR
47
65.1
77.4
80.1
48.8
444
82.5
55.6
66.1
68.0
62.3
54.0
47.4
50.8
74,0
79.5
72.7
55.3
98.0
54.9
SzeneScript
49.1
64.5
71.1
39.7
30.3
81.1
43.4
30.9
53.4
41.7
42.6
11.8
28.2
58.6
48.9
68.3
55.8
77.5
36.5
SPATIALLM!
2.9
6.5
2.5
12
4.0
23
4.8
2.5
0.3
14
2.3
0.9
4.7
5.0
3.7
SPATIALLM
33.8
57.0
49.2
24.4
12.2
50.9
15.3
48.1
n/a
n/a
23.4
m/a
3,6
38.3
31.4
75.6
m/a
SPATIALLM
65.6
80.6
52.0
40.0
86.6
66.8
62.8
67.1
55.6
29.7
$3.6
81.6
70.7
63.9
95.4
64.3
24

---

Table 11: 3D object detection results with In @0.5. trained on ScanNet only. *: trained on our
dataset only. "n/a" indicates the category is not included in our dataset.
bookshelf
averaae
batrab
cabiintt
cutuin
gubbin.
pieture
retrige.
s.urain
window
chair
deak
door
lable
tnilet
hed
Method
V-DETR
+7
56.8
68.8
75.5
50.4
39.1
76.2
43.6
54.1
52.1
50.9
47.8
42.7
49.2
61.5
61.4
65.4
50.6
88.6
43,9
SceneScript
36.8
37.3
21.2
16.9
44.5
23.3
SPATIALLMM
0.7
2.2
0.0
0.1
0.6
1.7
0.4
1.4
0.0
0.0
022
2.0
0.0
SPATIALLM"
22.6
6.8
17.7
59.0
SPATIALLM
52.6
52.6
C.3 Zero-shot Detection on Videas
Per-category results. We collect 107 virtual room-tour videos and use MAS(3R-SLAM 36 to
reconstruct point cloud from each video. Note that these videos are rendered from virtual scenes created by professional designers, thus ground-trute 3D annotations are available. To calibrate the reconstructed point cloud with ground truth 3D annotations, we align the estimated camera trajectory from MAAStR--LLAM the ground- truth camera trajectory. Then, we perform zero-shot detection with SPATIALLM on the calibrated point clouds.
In Table 12, report the quantitative results the top-20 occurring categories in the video set. Note that this is a challenging task, because the reconstructed point clouds are often noisy and highly incomplete. Furthermore, because of the differences scale and position due imperfect alignment of camera trajectories, the objects in the point cloud may not match well with the ground truth 3D annotations, leading lower Fl scores.
Table 12: Zero-shot results on layout estimation and 3D object detection with loU@0.25.
(a) Layout estimation
avg.
wall
door
window
55.7
68.2
47.4
51.4
(b) 3D object detection
mightstand
chandelier
wardrobe
diningtab.
coffectab.
average
curtain
bed
sofa
chair
cabinet
plants
Ivcab.
sidetab.
aircond.
dresser
stool
refrige. painting
sauper
/ 67.0 36.8 39.6 95.2 69.1 32,3 11.2 24.2 26.3 27.3 64.9 9.7 24.0 46.7 30.8 16.7 38.2 24.1 18.0
D Extensions of SPATIALLM
In this section, present more proof-of-concept experiment results future extensions of SPA- TIALLMM, including (i) handling point clouds from diverse sources, and (ii) adapting to various downstream tasks via language-based prompts.
D.1 Supporting Point Clouds from Diverse Sources
Point clouds offer a lightweight and flexible 3D representation that can be readily generated from variety of sources. Meanwhile, our dataset is large and diverse, enabling strong zero-shot performance over point cloud inputs in both synthetic and real-world data. this experiment, we examine the following four types of input sources:
Text-to-3D: We first generate an image via Flux-dev (6 with prompt low poly 3D living room in cartoon style", then convert the image into 3D using TRELLIS (57).
25

---

Hand-held video camera: We take real-world video using a hand-held camera and perform 3D reconstruction using MAS-R--LAA [36].
LiDAR-based reconstruction: We take point clouds reconstructed from captures using
iPhone ARKit from MultiScan [4
Synthetic mesh: We create point clouds by randomly sampling mesh surfaces from synthetic dataset HSSD
[24].
As can be seen in Figure 13 our model is robust to variations in point cloud origin, structure, and
appearance.
D.2 Task Adaptation via Language-based Prompts
A key idea of our work is to represent the 3D scene structures in pure text form. This enables SPATIALLM be conveniently adapted different downstream tasks via natural language instruction, without changing the underlying model architecture. Below, we demonstrate two such extensions.
Detection with user-specified categories. We enable the model perform object detection con- ditioned on user-specified categories by leveraging the flexibility of LLMs. During training, the input prompt is customized indicate which object categories detect. The model then learns to selectively predict oriented bounding boxes (OBBs) for instances matching the specified categories, all within unified training procedure, This experiment highlights the model's adaptability through prompt-based based control. Some qualitative results are provided in Figure 14
Semantic label completion. Besides natural language prompts, SPATIALLM also supports structured language scripts as input, enabling novel tasks such as semantic label completion. In this task, the model receives object bounding boxes and their poses written in general-purge purposs language (i.e., Python), while the object categories are left unspecified. The model then predicts the semantic label of each object based on the spatial layout and the corresponding point cloud data. Note that this problem often arises in real-world design workflows that involve 3D assets with known positions but without semantic labels.
Compared 3D object detection, this task relatively easy. Nevertheless, it demonstrates the versatility and potential use of large language models for structured 3D understanding. We show some qualitative results in Figure [5 Our model achieves an overall classification accuracy of 96.8% on the test set of our dataset.
26

---

living room
living room
living room
living room
bedroom
bedroom
bedroom
bathroom
Figure 9: Point clouds with overlaid 3D layout and object annotations in SPATIALLM dataset.
27

---

sofa
coffee table
cabinet
bed
nightstand
wardrobe
chair
table
desk
shoe cabinet
chandelier
dressing table
Figure 10: Example objects in SPATIALLM dataset. Note that each category has diverse set of objects with high quality in both geometry and appearance.
28

---

L1
L2
L3
L4
L5
Encode Block 3
Encode Block x
Encode Block x
Encode Block x 12
Encode Block x 3
Point Cloud
Serialization
Embedding
Grid Pool
Grid Pool
Grid Pool
Grid Pool
D6
D48
D48
12
D96
12
D192
4.2
D384
2
D512
Figure 11: Five-level hierarchical structure of the point cloud encoder [56]. D indicates the feature dimension at each level, and grid pool downscales the point cloud resolution by 2 times along each dimension.
Prompt:
Response:
val1_o-Wsil
(5,3,
15, 101,30,001)
vall_i-Wall
(144.15.30.144,28..3)
(a11___/a111139 185,30,128,16,88
fall_3-Wal1(12>
110,,110,10,,,,0011
val1_4-Wel1
(47,18.90.118,.88.8
val1_8-Wal1
377,18,0,,444422. (011)
val1_6-Wall 26,4, 352 26..222..00111 101)
sal1_7-7all (152, 16, 12,,,2,,00,1001
val1_8=Ws11 (47
cxl1_9-7ull 42, .78.688.000
cal1_10-7ell (1t.aaaa111111.11111a11 (138,42, 152, 30,
<point_cloud> cloud> Detect walls, doors,
9all_11=Wail (138 32,
windows, boxes. The reference code
vall_13-7aal 53, 2011,
val1_12-Wal1
is as follows:
val1_14-7all (138, 58, 30,
odaraclaso
vall_15-7all (152,111233333333 30, 170,
Wall:
val1_16=Wal1 (19 ,151,
1nc
sal1_17-2a11 (19.11))111111,,331.11
val1_18-7a11 (118,, 170,,,,,7,,,,,52. 17.,
80,
9.7: 182
sal1__9=9a1l (118, (113,,17,, 337,,30,118,.... 1111118,889 30, 30,
inc
vx11_20-2aa11
b.y: 187
doar_1=lleer 11.11_17..99,1111 189, 27,775)
dear_O-Docz (vw13_16, 111.1,1......10000
b_s: lns
height: lat
doar_2-Bocr 11111_19.118,215 20,211,,0000 ,7,76777..
Odatисlazя
class Dooz:
kall_id: atr
= (Ma)1__4,4 18.471_4.. 77.77775)
vindov_1-Vinddz [ww]l_17, 111.1__1,,111112 .17.,.11,,,,1 ,0,
1oao_00 (3, 223,20022,320. 36,, (66,,10,86)
pasition_y:
paxition_x: int.
1bor_
,2,0,,77,777
position_2:
5bax_2-Rbcx Bbcr_3--iicc(iittcccodiiiiooee
width: int
Bbox_5-Bbex (2.. 24,,
height: int.
bbax_ 4-ae__4Rbex
0datисlaия
abos__5Rboxx 2bor__-BBio.cu.5550000......
bbox_
1480,166.105.44
sall_id: aie
bbaz 7-
3a0o_88-Boexx11ooo:-eaaaaaaaad-- 116,,..........5555.........8 22,,2,,21.000
paxition_x: int
pazition_y:
position_2:
bbax, 1har__100Bhox(iiinnnn_chhhee
[aas__1--Boox(iinin___aaaaa 49, 50, 580, 77 ,7,,0,445)
height: int.
width: int.
abox_12-Bbox (ctninoooccnnnngcchhhee hhw.::22-Box(iinnnng.chhhhh::
27.255.43 480, 28.0.8..22 28.28.41
sbox_13-Bbox www.:pppainnnnnn....
.5..8.3))
class Rhox:
bbox. (ct--bbbo((hoonncabiiett 246,,246,,,0 480,622
1203
class: air
bbax 18-Bbox (dining.cheis www.chhhrr...,..... 480.,,2288
sbox. 1-----oxxtt (diming.chai: .41 .41,552, 21.252,43 (8,,0444480000 28,, 28,28..0 41)
(ewdeboadddddd... 264,44, 68,320.9900 122090,16,110)
paxition_y:
pasition_2:
angle_z: int
acale_x: int.
scale.y:
acale_z: int
Figure 12: An example conversation of SPATIALLM.
29

---

Figure 13: Qualitative results various input sources. First row: Text-to-3D. Second row: Hand-
held video camera. Third row: LiDAR-based reconstruction. Fourth row: Synthetic mesh
30

---

all objects
bed
chair
desk
layout
dining chair
IV, tv v__abinet
sofa, coffee _abe
Figure 14: Qualitative results of detection with user-specified categories. Labels below each figure indicate the category specified for detection.
Input
GT
SPATIALLM
Qualitative results of semantie label completion. Black-colored input boxes denote objects with unknown categories, which are classified by SPATIAL.LM.
Figure 15:
31


Reviews:
[{"review_id": "pv0sliZcUo", "reviewer": "Reviewer_33mG", "summary": "This paper presents SpatialLM, a VLM finetuned for structured indoor modeling tasks. The authors curated a large-scale 3D dataset in simulation and finetuned a small language model to take input from point cloud features and output Python-style text language to annotate objects and room layouts in 3D. Experiments demonstrate the feasibility of this fine-tuning paradigm and the effectiveness of SpatialLM compared with baselines.", "questions": "1. Line 167: What exactly is this 5-level structure?\n2. Can the model be potentially extended to handle dynamic scenes when objects and layout change? If so, what is needed?\n3. How would you augment the object sets for different application scenarios? Does that require a significant amount of additional training?\n4. After training, does SpatialLM still possess general language generation capabilities? Can you freely prompt the model to answer scene-related questions?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Methodology. This paper proves the feasibility of directly finetuning an LM, even a small one, for generating 3D indoor modeling, which is a meaningful augmentation to language models, especially useful for real-world applications.\n2. Dataset. The curated dataset seems to be of high quality and produces good pretraining effects (Tables 5 and 6). Details of the dataset are also presented in the appendix. If the dataset can be open-source as promised by the authors, it will be of great use to the community. Can the authors confirm the open-sourcing plan?\n3. Solid performance. Design choices are validated with experiments in Section 2. SpatialLM produces strong performance on layout estimation and 3D object detection tasks. Further finetuning on task data gives even better performance, which is a good sign for the potential usefulness of the model.\n\n**Weakness**\n1. The necessity of using a language model for the tasks. As the paper states, the model is currently confined to a predefined set of objects, which raises the question of the need for a language model. It is also a waste of MLLMs' generalizability. For such a fixed set, can a strong 3D point cloud feature + a strong segmentation model suffice for the task? Meanwhile, the paper justifies the advantage of using the Python language as the output by stating its editable and easily extendable. However, such features are not demonstrated in the paper. It would be nice to see if prompting the language model can help edit the layouts.\n2. Robustness. It seems the performance is mainly tested assuming perfect 3D inputs. Only Section 3.3, the model is tested with reconstructed inputs. Although it does show some robustness, the experiment is only qualitative. Is it possible to roll out quantitative experiments on datasets that have 3D groundtruth but use reconsrtructed 3D as input? What would the performance be like?", "was_revised": true, "final_justification": "My questions and concerns are largely addressed. I do notice the concern raised by reviewer ML3L, that the originality of this work is compromised by the prior work SceneScript. While I acknowledge the difference in technical approaches and the improved performance, I feel reviewer ML3L's concern is also valid.\n\nSet aside this concern, I highly look forward to the opensourcing of the entire dataset. I believe it will be a great contribution of this work to the community. Thank the authors for confirming the open-sourcing decision. However, this is hard to assess in the current double-blind reviewing stage. In light of this, I am willing to stay at my rating, but wish to slightly tune it down to a hypothetical 4.5 for AC's reference.", "rebuttal": "We thank the reviewer for the comments. Below is our detailed response.\n\n* [Confirmation of the open-sourcing plan.] Yes, we will open-source the dataset as promised in the paper.\n\n**Weaknesses**\n1. [Waste of MLLMs' generalizability] We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs. By limiting to a predefined set of objects, we are able to conduct a direct comparison with SOTA methods, such as Roomformer and V-DETR, on existing public benchmarks.\n\n   In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with large-scale vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [1] and function calling [2], with the understanding that the findings will later be integrated into a general-purpose foundation model.\n\n2. [Comparison to a strong 3D point cloud feature + a strong segmentation model] Indeed, several studies in the literature detect 3D objects via point cloud instance segmentation, as suggested by the reviewer. For example, in V-DETR, the authors also included some point cloud instance segmentation methods, such as 3D-MPA, as baselines. Here, we note that, as a bottom-up approach, the instance segmentation methods have several limitations. First, they can only output axis-aligned bounding boxes (AABB) without an orientation. Second, they may fail to predict the correct object sizes in the presence of holes, occlusions, and noise. In contrast, our method can infer full object sizes and orientations from sparse or occluded inputs (please see lines 284 - 291 of the paper for more discussion and some examples).\n\n    Furthermore, as we stated earlier, our long-term goal is to equip MLLMs with the capacity to make structured predictions in text form for 3D indoor modeling. Therefore, a dense prediction method for instance segmentation is not suitable for our purpose.\n\n3. [Use of Python language] Thanks for the suggestion. In Section 4.2 of the supplementary material, we provide some preliminary results on task adaptation via language-based prompts, including (i) detection with user-specified categories and (ii) semantic label completion. As for layout editing, we believe it is feasible with our current dataset by generating paired data with language instructions and editing actions in the future. But in this paper, our focus is on the reconstruction of the layout from point clouds.\n\n4. [Robustness to imperfect point clouds] Please note that in Section 3.2, we conduct experiments on ScanNet, which is a real dataset with point clouds reconstructed with imperfect RGB-D data with holes, occlusions, and noise. In Section 3.3 of the supplementary material, we also report quantitative 3D indoor modeling results with point clouds reconstructed from monocular videos (Table 6). Please refer to the supplementary results for more discussions.\n\n**Questions**\n\n5. Line 167 (5-level hierarchical structure): The 5-level hierarchical structure refers to the 5 layers of neural blocks in the 3DCNN and Sonata/PTv3 encoders. The first layer doubles the feature dimensions but keeps the point cloud resolution the same, and the following 4 layers each downscales the point cloud resolution to half. We will add a figure in the revised paper to illustrate this structure.\n\n6. Thanks for the interesting question. Potentially, we can use SpatialTrackerV2 [3] to reconstruct the 3D point cloud of both static scenes and dynamic moving objects with 3D tracking trajectories. Then, we may use a SpatialLM-like architecture to reconstruct the static layout and the moving object bounding boxes in different time steps auto-regressively. In addition, a large-scale point-cloud dataset with annotations of dynamic objects (e.g., humans) is needed.\n\n7. In this paper, we only keep the 59 common categories while filtering objects in the long-tail distribution. To improve the model’s generalizability, one may augment the labels of all objects in our dataset with VLM captions. Specifically, one can get a diverse set of annotations by performing image captioning with rendered images of each object in our dataset (as shown in Figure 4 of the supplementary material). Alternatively, researchers may use a different dataset of their own for their specific applications. In either case, a one-stage training schedule as discussed in Section 2.3 is likely required.\n\n    Besides, it is worth noting that the amount of training required also depends on the point cloud encoder. The Sonata/PTv3 point cloud encoder used in SpatialLM follows a DINO-like self-supervised training strategy. Such a strategy focuses on capturing local and global visual features, and does not consider alignment between visual features and text. A more powerful point cloud encoder with enhanced vision-language alignment will likely reduce the amount of training required, while better supporting language-based applications.\n\n8. In this paper, we focus on training LLMs to generate a structured scene description from point clouds only. This is one of the limitations of the paper (as stated in lines 335 - 337). It cannot answer scene-related questions based on the point clouds in natural language, as it is not trained to do so. And after finetuning, its general language generation capabilities may also be severely impacted. \n\n    However, as we discussed before, we believe our work is an important first step towards equipping LLMs with general 3D scene understanding and generation capacities.  And by training together with other general-purpose vision-language datasets, we can restore and improve the model’s general language generation capabilities.\n\n**References**\n\n* [1] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.\n* [2] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025.\n* [3] Xiao et al., SpatialTrackerV2: 3D Point Tracking Made Easy. In ICCV, 2025."}, {"review_id": "20U5Ytp7N3", "reviewer": "Reviewer_KAFv", "summary": "This paper presents a framework for 3D visual grounding using pretrained language models augmented with spatial inductive biases. The model takes as input a structured scene layout consisting of object labels and their associated 3D attributes (position, size, orientation), and predicts the 3D bounding box corresponding to a natural language query. Spatial reasoning capabilities are introduced into a transformer-based language model by adding spatial embeddings and modifying the attention mechanism to reflect geometric proximity. Experiments on synthetic and real-world layouts demonstrate improved grounding accuracy over baseline models that lack spatial awareness. The approach is positioned at the intersection of 3D vision and multimodal language reasoning, with contributions focused on task-specific integration rather than architectural or theoretical innovation.", "questions": "How sensitive is the model’s performance to the accuracy or sparsity of the input 3D layout? Would the method remain effective under noisy or incomplete object representations, as would be expected in real-world perception pipelines? How does the spatial inductive bias affect attention behavior or sample efficiency during training? Is there evidence that the proposed modifications improve generalization to unseen spatial configurations or object arrangements beyond the training distribution? Finally, could the same spatial reasoning be achieved by finetuning with task-specific data alone, without architectural modifications?", "limitations": "The method assumes access to structured scene layouts composed of object-centric tokens with precise 3D attributes including class, position, orientation, and size. These layouts are either synthetically generated or derived from processed ScanNet scenes, relying on accurate object detection, segmentation, and pose estimation pipelines. The performance and applicability of the model in scenarios where such structured representations are incomplete, noisy, or unavailable are not addressed. While the proposed spatial inductive bias improves performance on layout-conditioned reasoning tasks, it is unclear how the model would behave when exposed to real-world uncertainty, such as missing objects, ambiguous queries, or spatial occlusions.\nAdditionally, the approach is strictly conditioned on discrete object instances, limiting extensibility to raw sensory data such as point clouds or voxelized scenes. The spatial inductive bias is implemented via embedding injections and localized attention masking, but its impact on model interpretability or reasoning trajectories is not analyzed. There is no investigation into whether the model develops spatial priors or relational structures beyond what is encoded in the training data. Generalization to unseen object configurations, novel spatial relationships, or out-of-distribution compositions is not systematically evaluated, which constrains the scope of the method’s robustness claims.\nLastly, the model does not perform end-to-end reasoning from language to perception but instead operates in a middle-layer abstraction, making it difficult to assess its utility in embodied tasks or real-world robotic applications without substantial integration with upstream vision modules. More broadly, the research is situated within a structured-layout paradigm where perception is abstracted away, which limits its relevance to fields focused on holistic scene understanding or sensor-to-action pipelines. This constrains the potential impact of the work beyond its immediate 3D layout reasoning setting.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper addresses a well-scoped problem at the intersection of 3D visual grounding and language-based spatial reasoning. It presents a clean architectural adaptation of transformer-based language models by introducing spatial inductive bias through position-aware embeddings and attention masking strategies. The proposed formulation is modular and lightweight, requiring no changes to the core transformer mechanism, which enhances reproducibility and compatibility with existing pretrained language models.\nEmpirical evaluations span both synthetic environments and real-world datasets (ScanNet), with quantitative metrics showing consistent improvements over non-spatial baselines. The experimental design includes comparative analysis with spatial relation prediction and 3D bounding box localization, both of which validate the proposed enhancements. The use of object-centric representations as input provides computational efficiency and avoids the need for raw perceptual processing, making the system well-suited for layout-level inference tasks. Furthermore, the dataset construction, training setup, and evaluation protocol are clearly described, improving transparency and replicability.\nConceptually, the paper explores how structural priors can be aligned with the attention mechanism in LLMs to support grounded reasoning. This contributes to ongoing discourse in multimodal learning on how domain-specific biases can be embedded into general-purpose models.\n\nWeaknesses:\nDespite practical performance gains, the architectural contributions are modest. The proposed spatial embedding and attention mask mechanisms mirror existing practices in relational modeling and transformer locality bias, offering no new theoretical or algorithmic innovations. The model’s dependence on clean, discrete scene layouts means it cannot operate in realistic settings where perception is partial, noisy, or continuous.\nThe spatial bias mechanism is treated as a black-box utility without detailed analysis of its effect on attention dynamics, training stability, or generalization. There is no investigation into whether the model internalizes spatial hierarchies or relational patterns, nor whether the improvements stem from inductive bias or training data regularity.\nMoreover, the model is evaluated only in bounded task settings involving synthetic or preprocessed layouts, and its performance under distributional shift, ambiguous language input, or sparse spatial cues is not tested. These omissions weaken claims of general-purpose spatial reasoning.\nFrom a broader research positioning, the work contributes primarily to task-specific improvements within the domain of 3D vision and grounding. It does not generalize to open-vocabulary grounding, sensor-based reasoning, or joint perception-language planning. As such, the contribution is best viewed as an applied extension of language modeling for layout-conditioned reasoning, rather than a conceptual advance in foundational representation learning.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. Below is our detailed response.\n\n**Weakness**\n1. [Architectural contributions of our work] There seems to be a misunderstanding about the contributions of our work. In this paper, we use existing point cloud encoders such as Sonata and connect it to LLM with the standard “Encoder-MLP-LLM” architecture. We do not propose any new architectural modifications, such as “modifying the attention mechanism to reflect geometric proximity\", as suggested by the reviewer. \n\n    Instead, our primary goal is to train LLMs for structural indoor modeling tasks (i.e., layout estimation and 3D object detection). As stated in Section 1, our main contributions are\n    - (1) We regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form,\n    - (2) We propose a new synthetic dataset for the task, and conduct the first empirical study on the best strategy of aligning the point cloud input with LLMs for structured scene modeling, and\n    - (3) We empirically show that, by first training on our large-scale dataset and then on smaller downstream task data, our model gives competitive performance on public benchmarks.\n\n2. [Model’s dependency on clean, discrete scene layouts] There seems to be a misunderstanding of the SpatialLM’s input. For example, the reviewer wrote “The model takes as input a structured scene layout consisting of object labels and their associated 3D attributes (position, size, orientation), and predicts the 3D bounding box corresponding to a natural language query.” But SpatialLM actually takes 3D point cloud as input and generate structured scene descriptions as output. In other words, the “structured scene layout” is the output of our model, not the input.\n\n    Similarly, the reviewer wrote that “The method assumes access to structured scene layouts composed of object-centric tokens with precise 3D attributes including class, position, orientation, and size.” This statement is not accurate for the same reason.\n\n    In terms of handling incomplete or noisy point cloud inputs, please note that in Section 3.2, we conduct experiments on ScanNet, which is a real dataset with point clouds reconstructed with imperfect RGB-D data with holes, occlusions, and noise. In Section 3.3, we also report 3D indoor modeling results with point clouds reconstructed from monocular videos. These experiments demonstrate the robustness of our model to real-world inputs.\n\n3. [Investigation into the spatial bias mechanism] We point out that SpatialLM follows the standard “Encoder-MLP-LLM” architecture for MLLMs. We leverage existing networks such as Qwen2.5-0.5B as the base model and Sonata as the point cloud encoder. We respectfully refer the reviewer to the Sonata paper [1] for a detailed study about how the point cloud encoder captures spatial context and priors.\n\n4. [Generalization under distributional shift, ambiguous language input, or sparse spatial cues] In Section 3.3 (Zero-shot Detection on Videos), we show that our model exhibits reasonable generalizability across datasets. However, as we acknowledged in Section 4, fine-tuning SpatialLM on a task-specific dataset is still needed to achieve the best performance. Further scaling up the data and model size could be a promising future direction.\n\n5. [Limitation to task-specific improvements] We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs. \n\n    In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with large-scale vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [2] and function calling [3], with the understanding that the findings will later be integrated into a general-purpose foundation model.\n\n**Questions**\n\n6. [Model’s sensitivity to the accuracy and sparsity of 3D layout] Please refer to item #2 above.\n\n7. [Analysis on spatial inductive bias] Please refer to item #3 above.\n\n8. [Generalization to unseen spatial configurations] Please refer to items #4 above.\n\n9. [Achieving spatial reasoning by finetuning with task-specific data alone, without architectural modifications] Please note that we do not propose any architectural modifications in this paper. Instead, we follow the standard “Encoder-MLP-LLM” architecture for MLLMs, and leverage existing networks such as Qwen2.5-0.5B as the base model and Sonata as the point cloud encoder.\n\n**Limitations**\n\nWe respectfully refer the reviewer to our response to the paper's weaknesses above for discussions about the contributions of our work, the input of SpatialLM, investigation on the spatial inductive bias, generalization to unseen object configurations, and potential impact on 3D scene understanding.\n\n**References**\n* [1] Wu et al., Sonata: Self-Supervised Learning of Reliable Point Representations. In CVPR, 2025.\n* [2] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.\n* [3] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025."}, {"review_id": "8dN3W6nKFH", "reviewer": "Reviewer_dWYu", "summary": "This paper presents a LLM tailored for structured understanding. SpatialLM predicts architectural elements (walls, doors, windows), and oriented object boxes with semantic categories from 3D point clouds. It follows a standard “Encoder-MLP-LLM” architecture and is jointly trained for each downstream task. SpatialLM is evaluated on two downstream tasks: (1) layout estimation on Structured3D dataset, where SpatialLM outperforms state-of-the-art models RoomFormer and SceneScript, and (2) 3D object detection on ScanNet, where SpatialLM outperforms SceneScript and is competitive with V-DETR.", "questions": "- What’s the difference between the claimed 1st contribution from SceneScript?\n- Structured3D is multi-room dataset while ScanNet is single-room dataset. Does SpatialLM do inference on multi-room directly or can only do inference in single-room manner?\n- Section 2.2 explored different point cloud encoders. However, eventually the point cloud encoder still needs to be trained jointly for each downstream task (tab. 4). Also L198-199 says “*current pre-trained point cloud encoders are not as versatile in supporting downstream tasks yet.*” This motivates the question: is pretrained point cloud encoder really necessary? How about just initializing the encoder (e.g. SPTv3) with random weights and then jointly trained with MLP and LLM?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper presents a large-scale (*12,328 scenes*) synthetic SpatialLM dataset with layout and object annotations. The visualization shows that this dataset is of high quality in terms of realism.\n- The paper presents a unified model that can accept language prompts to solve different 3D structured understanding tasks. The architecture follows standard multimodal LLM and can leverage the rich-knowledge encoded from pretrained LLMs.\n- There are some interesting ablations and empirical study on point cloud encoders and training schedules.\n- Strong performance: SpatialLM achieves state-of-the-art performance on layout estimation of 3D scenes on Structured3D dataset and competitive performance on 3D object detection on ScanNet.\n- The paper is clearly written and very easy to follow. There are also many nice visualizations, making the results more convincing.\n\n**Weaknesses:**\n\n- Even though SpatialLM uses the same architecture for different tasks, it is not universal:\n    - It requires dedicated training for different tasks and datasets.\n    - And the training involves training all the three components of the “Encoder-MLP-LLM” architecture for best performance (tab 4), which defeats the purpose of large and universal models.\n- This paper aims for large language models and one important consideration should be model complexity and training data during evaluation. For tab. 2, 5, 6, please add a column of number of parameters so that readers know how model complexity affects results. For tab. 5, 6, please also add a column of training data to make the data factor more clear.\n- Some statement is not precise or wrong:\n    - L24-26: the paper states that the first contribution is that “*regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form*”. However, I think this is what SceneScript already does. What’s the difference between this contribution from SceneScript?\n    - L297: “*RoomFormer [58] proposes to train a two-stage Transformer-based network to predict corners and rooms.*” I think RoomFormer is a single-stage network using two-level queries, not two-stage.\n- Organization of the paper: I feel the naming is misleading. To me, Section 2 “An Empirical Study on Point Cloud Feature Alignment” mainly discusses SpatialLM, while section 3 “SpatialLM” is actually the experiment section. Therefore, I suggest to rename section 2 as “SpatialLM” and section 3 as “Experiments”, or something indicative.", "was_revised": true, "final_justification": "Thanks the authors for the rebuttal. Initially, my major concern lies in the difference between this work and SceneScript. The authors' rebuttal helped address this concern. Some of my other questions/concerns are also addressed. Thus I would like to increase my rating from \"Borderline accept\" to \"Accept\". However, I have some suggestions for the authors to consider when revising the paper - see my comments.", "rebuttal": "We thank the reviewer for the comments. Below is our detailed response.\n\n**Weaknesses**\n1. Thanks for the thoughtful comments. We acknowledge that SpatialLM, in its current form, is a task-specific model for structured indoor modeling. Our plan is to focus on this well-established task first, and to show a feasible path to tackle 3D indoor modeling tasks (i.e., layout estimation and 3D object detection) with LLMs. \n\n    In future work, we can improve its generalizability by generating open-vocabulary annotations with MLLMs and training them together with general-purpose vision-language datasets. In fact, it is not uncommon for researchers to focus on one capacity of MLLMs at a time, such as video understanding [1] and function calling [2], with the understanding that the findings will later be integrated into a general-purpose foundation model.\n\n2. Thanks for the suggestion. In the revised paper, we will add columns to the tables to make the model complexity and data factors clearer.\n\n3. [Lines 24 - 26: difference from SceneScript] Thanks for the question. Indeed, our work is inspired by SceneScript. However, a key difference is that SceneScript proposes a task-specific tokenizer, which is not compatible with the text tokenizers used in modern LLMs. And it trains the tokenizer with a task-specific decoder from scratch (please refer to Sections 4.2 and 4.3 of SceneScript paper for details). In other words, a new domain-specific language with the associated tokenizer and decoder is proposed in Scenescript. This is also noted in Section 4.1 (lines 302 - 303) of our paper.\n\n    In contrast, SpatialLM directly exploits modern LLMs' capacity for Python language generation, without designing any new tokenizer or decoder. Hence, the first contribution \"regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form\".\n\n    Furthermore, by building upon existing LLMs, our experiment in Section 3 also reveals other differences from SceneScript. For example, as shown in Tables 5 and 6, while training on existing datasets like Structured3D and ScanNet may be sufficient for SceneScript to achieve competitive performance, these datasets are too small for SpatialLM. Instead, by first training our new dataset, SpatialLM outperforms SceneScript on both tasks.\n\n4. [Line 297: RoomFormer is a single-stage network using two-level queries] Thanks for pointing this out. We will fix it in the revised paper.\n\n5. [Organization of the paper] Thanks for the suggestion. We agree that Section 2 primarily discusses the design choices made in SpatialLM through a series of ablation studies, whereas Section 3 compares SpatialLM to existing methods on various benchmarks. We will adjust the section titles to better reflect their content in the revised paper.\n\n**Questions**\n\n6. Please refer to item #3 above.\n\n7. SpatialLM is trained on our new dataset with a mix of single-room layouts and multi-room layouts. Therefore, it can do inference in both single-room and multi-room manners.\n\n8. For SPTv3, we follow common practice to leverage a pre-trained model as a starting point, rather than training it from scratch. Please note that our claim in lines 198 - 199 is more of a comparison of existing 3D point cloud encoders to powerful 2D image encoders, SigLIP and DINOv2, where one can freeze the image encoders during MLLM training and still obtain good results.\n\n**References**\n\n- [1] Zohar et al., Apollo: An Exploration of Video Understanding in Large Multimodal Models. In CVPR, 2025.\n- [2] Zhang et al., xLAM: A Family of Large Action Models to Empower AI Agent Systems. In NAACL, 2025."}, {"review_id": "82Cp5C1qha", "reviewer": "Reviewer_ML3L", "summary": "This paper introduces SpatialLM, a 3D point cloud multimodal large language model for structured indoor modeling. SpatialLM regards 3D modeling as a scripting task, thereby leveraging the outstanding programming capabilities of large language models. The authors created a large synthetic 3D dataset, the SpatialLM dataset, which surpasses existing datasets in both quantity and scale. They conducted extensive experiments to demonstrate the effectiveness and generalizability of their model.", "questions": "1. What are the advantages of scripts over box texts (like \\box{x, y, z, w, h, l} or [x, y, z, w, h, l])?\n2. What is the incremental novelty of SpatialLM over SceneScript?  I believe that the format of the output script is not a problem for large language models.\n3. What are your rules for selecting scenes/categories? Are there any special considerations to ensure diversity and avoid a long-tail distribution?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n1. The introduced synthetic dataset of 12,328 indoor scenes is a substantial contribution, enabling extensive training and benchmarking.\n2. The author conducted extensive ablation experiments on the design of the point encoder, the selection of spatial resolution, and the training schedule.\n\nWeaknesses:\n1. The motivation and approach of SpatialLM are similar to SceneScript, raising concerns about incremental novelty.\n2. The improvements of SpatialLM over SceneScript in the layout estimation task are modest (+1.3%/+0.8% at IoU@0.25/0.5). Additionally, SpatialLM does not surpass V-DETR, which does not use LLMs, in 3D object detection tasks.\n3. There are some structural issues in writing. The related work section could be presented earlier.", "was_revised": true, "final_justification": "This paper proposes a framework built upon LLM to perform 3D perception and reconstruction tasks. The idea is similar to SceneScript, and the improvement is demonstrated with experiments. However, the limited application and relatively incremental performance improvement do not support the relatively minor technical contribution of this work. The experimental results are also fixed during the discussion phase, which may mean the results may not be solid during submission.\n\nOverall, I think it might be a promising direction, but it needs more experiments or applications to support. Therefore, I vote for borderline rejection at this stage.", "rebuttal": "We thank the reviewer for the comments. Below is our detailed response.\n\n**Weaknesses**\n1. Thanks for the question. Indeed, our work is inspired by SceneScript. However, a key difference is that SceneScript proposes a task-specific tokenizer, which is not compatible with the text tokenizers used in modern LLMs. And it trains the tokenizer with a task-specific decoder from scratch (please refer to Sections 4.2 and 4.3 of SceneScript paper for details). In other words, a new domain-specific language with the associated tokenizer and decoder is proposed in Scenescript. This is also noted in Section 4.1 (lines 302 - 303) of our paper.\n\n    In contrast, SpatialLM directly exploits modern LLMs’ capacity for Python language generation, without designing any new tokenizer or decoder. Thus, much of our work is focused on how to align point cloud features to existing LLMs with a standard “Encoder-MLP-LLM” architecture (Section 2), which has not been carefully studied before.\n\n    Furthermore, by building upon existing LLMs, our experiment in Section 3 also reveals other differences from SceneScript. For example, as shown in Tables 5 and 6, while training on existing datasets like Structured3D and ScanNet may be sufficient for SceneScript to achieve competitive performance, these datasets are too small for SpatialLM. Instead, by first training our new dataset, SpatialLM outperforms SceneScript on both tasks.\n\n    To summarize, our work is the first to employ modern LLMs for structured scene reconstruction (i.e., layout estimation and 3D object detection) from point clouds. Our contributions include (i) a new large-scale dataset and (ii) thorough ablation studies on aligning point cloud features with LLMs. Furthermore, we show that SpatialLM can achieve competitive performances against existing task-specific models. We believe this is an important step towards equipping LLMs with general 3D scene understanding and generation capacities.\n\n2. Thanks for the comment. In fact, after the paper submission, we found a bug in our implementation related to using Fourier encoding of voxel indices as positional embeddings to the output Sonata features. After fixing the bug, the performance of SpatialLM on both layout estimation and 3D object detection further improved. The new results are given below.\n\n    **Table 5.** Experiment on layout estimation.\n    | Method  | F1 (IoU$_\\textrm{2D}$@0.25) | F1 (IoU$_\\textrm{2D}$@0.5) |\n    |---|---|---|\n    | RoomFormer | 70.4 | 67.2 |\n    | SceneScript | 83.1 | 80.8 |\n    | SpatialLM (ft. Structured3D)  | 32.8 | 17.9 |\n    | SpatialLM (ft. Ours) | 51.2 | 38.3 |\n    | SpatialLM (ft. Ours -> Structured3D) | **86.5** | **84.6** |\n\n    **Table 6.** Experiment on 3D object detection.\n    |Method  | F1 (IoU$_\\textrm{3D}$@0.25) | F1 (IoU$_\\textrm{3D}$@0.5) |\n    |---|---|---|\n    |V-DETR  | 65.1 | **56.8** |\n    |SceneScript  | 49.1 | 36.8 |\n    |SpatialLM (ft. ScanNet) | 2.9 | 0.7 |\n    |SpatialLM (ft. Ours) | 33.8* | 22.6*  |\n    |SpatialLM (ft. Ours -> ScanNet)  | **65.6**  | 52.6 |\n\n    As one can see, for layout estimation, SpatialLM consistently outperforms other baselines, with +3.4 IoU2D@0.25, +3.8 IoU2D@0.5 over SceneScript. For 3D object detection, we have +16.5 IoU3D@0.25, +15.8 IoU3D@0.25 over SceneScript, and +0.5% IoU3D@0.25, -4.2 IoU3D@0.5 compared to V-DETR.\n\n    Finally, we note that our primary goal of the work is not merely competing for the highest scores on existing benchmarks. Instead, with the experiments, we aim to show a feasible path to tackle 3D scene understanding tasks with the MLLM architecture.\n\n3. Thanks for the suggestion. We will revise the paper to present the related work section earlier.\n\n**Questions**\n\n4. Compared to domain-specific box texts, the Python script is more general and flexible. For example, one may specify various relationships, such as associating door/window with a wall, by passing the necessary parameters. It can also be easily extended to include an arbitrary number of object classes or other properties. Further, since LLMs have already seen a large amount of Python code data, it is easier to align the scene descriptions with other LLM capacities, such as tool calling.\n\n5. Please refer to item #1 above.\n\n6. To ensure the diversity of our dataset, we mainly resort to our access to a large repository of interior designs from a leading platform in the interior design industry. As described in Section 2.1, most 3D scene designs in our dataset are created by professional designers and used for real-world production. This way, we ensure that the distribution of 3D scenes in our dataset closely resembles that of real-world houses. Please refer to Section 1 of the supplementary material for statistics of the rooms and objects in our dataset.\n\n    During dataset curation, we select designs (i.e., scenes) by considering the following factors: (i) ratings by the professional designers, (ii) the number of renderings generated by the design, (iii) total floor area > 20 m$^2$, and (iv) number of unique objects > 35. For objects, we pick the 59 common object categories based on occurrence while filtering objects in the long-tail distribution. And as mentioned in Section 2.1, we filter out small objects with a side length < 15cm."}]

Author rebuttal:

Meta-review: The paper presents SpatialLM, a framework for adapting large language models to structured indoor scene modeling, supported by a large synthetic dataset. The method improves spatial reasoning and shows promising results on layout understanding tasks.

Reviewer scores are mixed (two positive, two negative). While concerns remain about incremental novelty and overlap with SceneScript, the rebuttal clarifies differences, addresses technical questions, and commits to stronger evaluations in the camera-ready version. 

Overall, the contribution is considered valuable, especially with the dataset release and the demonstrated improvements in structured indoor modeling tasks. The AC recommends acceptance, with the expectation that the camera-ready version further strengthens comparisons to prior work, clarifies methodological novelty, and incorporates the promised analyses.
Decision: Accept (poster)